7 ms·
Most scrapped websites have https, so you need to perform a MITM attack. Scrapers will probably notice that.
by woffoor 3mo ago
Most scrapped websites have https, so you need to perform a MITM attack. Scrapers will probably notice that.
- voakbasda 3mo agoNo, you just need to stand up your own website and feed the scraper a URL to it.
- ErroneousBosh 3mo agoI would just generate scads of Markov chain output and make it look like a plausible web page.
- dreamcompiler 3mo agoThat's pretty much what the bots are scraping now, with all the AI slop websites out there.
- ErroneousBosh 3mo agoFair point well made.
- ErroneousBosh 3mo agoHow would https affect it? If they're making a request to my machine to go and curl a page, how do they even know whether or not it was https?
- trumpdong 3mo agoNot sure about Bright Data but these are usually SOCKS or HTTP CONNECT proxies because that's most flexible. But the customer might be paying by the gigabyte, so you can still feed them nonsense, maybe a 4 gigabyte TLS certificate.