add configuration option for additional request headers #96
Conversation
add a pre navigation hook to add thos headers to the request
|
PS: we're running this from our fork, but I thought the changes might as well be relevant to this repo. feel free to merge or close the PR as you see fit 🙃 |
|
Hey @dardanbujupaj, thanks a lot for your contribution!! Can you include some examples of how you can use this feature in the README? |
| async (request) => { | ||
| return await request.continue({ | ||
| headers: { | ||
| ...request.headers(), |
There was a problem hiding this comment.
I thought this would be an object instead of a function 🤔
There was a problem hiding this comment.
In this case we're in a PuppeteerCrawlingContext, which means we have a Puppeteer HTTPRequest with a headers() method, see https://pptr.dev/api/puppeteer.httprequest.headers
@brunoocasali Sure, I'll add something this evening. |
|
@brunoocasali added the configuration property description to the readme with a short example. Let me know if there's anything else to improve! |
|
Configuration problem: |
Pull Request
Related issue
none
What does this PR do?
We're using scrapix to index our documentation.
Some parts of the documentation are protected, so we need a way to send some authentication information with the crawler requests.
This PR adds a new configuration property
additional_request_headerswhich are applied to each request in a pre navigation hook.This way we're able to either send an
Authorizationheader (basic, bearer etc.) or a custom auth header (e.g.x-access-key), so it should be quite flexible for any usecase.