6 ms·
Using Proxies in Web Scraping
- zero-sharp 2y agoThe issue that I struggled with is browser fingerprinting. Servers can detect if you're using a browser. If you are, then there are a number of flags, or parameters, in the browser that give away your identity. You can cycle through user agents and proxies. I don't think these things help. If you're not loading the page through a browser or if you try to hide information, you're already flagged. Scraping is a cat and mouse game now.
- ScottWRobinson 2y agoIIRC there are ways to give an authentic-looking fingerprint to browsers used for scraping. But as you said, it's a cat and mouse game