5 ms·
> Avoid detection with built-in anti-bot patches and proxy configuration for reliable web scraping. And it doesn't care about robots.txt.
by plastic041 6mo ago
> Avoid detection with built-in anti-bot patches and proxy configuration for reliable web scraping.
And it doesn't care about robots.txt.
- andrew_zhong 6mo agoGood point. The anti-bot patches here (via Patchright) are about preventing the browser from being detected as automated — things like CDP leak fixes so Cloudflare doesn't block you mid-session. It's not about bypassing access restrictions. Our main use case is retail price monitoring — comparing publicly listed product prices across e-commerce sites, which is pretty standard in the industry. But fair point, we should make that clearer in the README.
- messe 6mo ago> It's not about bypassing access restrictions. Yes. It is. You've just made an arbitrary choice not to define it as such.
- andrew_zhong 6mo agoI will add a PR to enforce robots.txt before the actual scraping.
- messe 6mo agoOr just follow web standards and define and publish your User-Agent header, so that people can block that as needed. You're creating the wrong kind of value. I really hope your company fails, as its success implies a failure of the web in general. I wish you the best success outside of your current endeavour.
- zendist 6mo agoRegardless. You should still respect robots.txt..
- andrew_zhong 6mo agoWe do respect robots.txt production - also scraping browser providers like BrightData enforces that. I will add a PR to enforce robots.txt before the actual scraping.
- plastic041 6mo agoHow can people believe that you are respecting bot detection in production when your software's README says it can "Avoid detection with built-in anti-bot patches"?
- andrew_zhong 6mo agoI hear you loud and clear - will replace the stealth browser with plain playwright and remove anti-bot as a feature.
- plastic041 6mo agorobots.txt is the most basic access restrictions and it doesn't even read it, while faking itself as human[0]. It is about bypassing access restrictions. [0]: https://github.com/lightfeed/extractor/blob/d11060269e65459e960b7a87836030886caca256/src/utils/browserProviders.ts#L35 https://github.com/lightfeed/extractor/blob/d11060269e65459e...