Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[NODRIVER] Bug with cdp.fetch.RequestPaused #2043

Open
tikene opened this issue Oct 10, 2024 · 2 comments
Open

[NODRIVER] Bug with cdp.fetch.RequestPaused #2043

tikene opened this issue Oct 10, 2024 · 2 comments

Comments

@tikene
Copy link

tikene commented Oct 10, 2024

When using cdp.fetch.RequestPaused, requests can be blocked as expected, unless Nodriver fails to find an element using tab.find. After that, the outgoing requests stop being blocked.

The requests still go through the handler and can be displayed in the console, but I am unable to block them in any way. I also tried using fail_request to no avail (although tbf I may have used that wrong). Posting this issue to save future headaches to fellow coders because this was very painful to debug, I imagine this is not expected behaviour @ultrafunkamsterdam

Here is sample code which demonstrates the issue, keep an eye on the blue .gif logo in the middle bottom of the website, which pops up after the second find() is executed, when it should be filtered:

import asyncio
from nodriver import cdp
import nodriver as uc

class Test():   
    async def requested_paused_handler(self, event: cdp.fetch.RequestPaused):
        r = event.request
        is_blocked = True if ".gif" in r.url else False
        if not is_blocked:
            asyncio.create_task(
                self.tab.send(
                    cdp.fetch.continue_request(request_id=event.request_id)
                )
            )
        else:
            s = f"BLOCKING | {r.method} | {r.url}"
            print(f" >>> ------------\n{s}")
            
    async def start_test(self):
        # Initialize browser
        self.browser = await uc.start()
        # Load blank page
        self.tab = await self.browser.get("about:blank")
        
        # Set up browser request filtering
        self.tab.add_handler(cdp.fetch.RequestPaused, self.requested_paused_handler)
        
        # Load website, the blue logo won't appear since it's been filtered
        self.tab = await self.browser.get('https://www.berkshirehathaway.com/')
        print("\nPage loaded")
        await asyncio.sleep(5)

        # This element exists in the test page
        print("\nAttempting to find existing element")
        await self.tab.find("A Message from Warren E. Buffett")
        print("Done!")
        await asyncio.sleep(5)
        
        # As soon as the find is called, the blue .gif logo appears in the page, and any further requests are not blocked
        print("\nAttempting to find non existant element")
        await self.tab.find("this element does not exist")
        
test = Test()
uc.loop().run_until_complete(test.start_test())
@tikene tikene changed the title [ [NODRIVER] Issue with cdp.fetch.RequestPaused Oct 10, 2024
@tikene tikene changed the title [NODRIVER] Issue with cdp.fetch.RequestPaused [NODRIVER] Bug with cdp.fetch.RequestPaused Oct 10, 2024
@tikene
Copy link
Author

tikene commented Oct 10, 2024

Managed to fix it using the following code, still curious about what could cause this behaviour but feel free to mark this issue as closed. Requests would show up as "pending" in the network tab until the find() was called

async def requested_paused_handler(self, event: cdp.fetch.RequestPaused):
        r = event.request
        is_blocked = True if ".gif" in r.url else False
        if not is_blocked:
            asyncio.create_task(
                self.tab.send(
                    cdp.fetch.continue_request(request_id=event.request_id)
                )
            )
        else:
            s = f"BLOCKING | {r.method} | {r.url}"
            print(f" >>> ------------\n{s}")
            asyncio.create_task(
                self.tab.send(
                    cdp.fetch.fail_request(event.request_id, cdp.network.ErrorReason.TIMED_OUT)
                )
            )

@devblack
Copy link

From this:
asyncio.create_task(self.tab.send(cdp.fetch.continue_request(request_id=event.request_id)))

To this:
self.tab.feed_cdp(cdp.fetch.continue_request(request_id=event.request_id))

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants