Disable cache for EXT:crawler indexing requests

[kraemer-igroup]

Fix #428 - Add EXT:crawler indexing check for htaccess, nginx and fallback middleware

[lochmueller]

Hey @kraemer-igroup
thanks for the great PR. Merged!
Regards,
Tim

[ste101]

Would it be better to make this behavior configurable?
I'm using crawler to build up the staticfilecache.

[lochmueller]

Hey @ste101 ,
there is no need for a configuration. Without the change, EXT:crawler will geht the static files and will not trigger a recaching of the page. Only with the patch, the request will pass the htacess & Fallback middlware to trigger the regeneration.
Regards,
Tim

[kraemer-igroup]

Hey @ste101, that's a good point, I did not think of this. But it still should trigger the generation of static files. This change ensures that T3Crawler does not get the cached version of the page (even if it is already cached). The downside could be that there is more server load, since already cached pages are loaded uncached via T3Crawler.

@lochmueller, you mean just not adding the .htaccess changes to avoid the (new) behaviour?

[lochmueller]

@lochmueller, you mean just not adding the .htaccess changes to avoid the (new) behaviour?

Yes. If someone do not want the behavior, keep the old htaccess and the Crawler extension will get the static files and no regeneration.