5 ms·
A lot of websites want "bot defense" due to high volume scrapers, and that "bot defense" often also ends up blocking low-volume wget/curl and polite crawlers li
by ccgreg 2mo ago
A lot of websites want "bot defense" due to high volume scrapers, and that "bot defense" often also ends up blocking low-volume wget/curl and polite crawlers like Common Crawl's CCBot.
- sumedh 2mo agoCloudflare can verify certain bots when they come from known ip addresses. So if your site is using cloudflare it can let CCBot if it has done the verification.
- m463 2mo agocloudflare routinely denies my human-piloted browser now, on many sites.
- ambigious7777 2mo agoyou're not a bot thats irrelevant
- pocksuppet 2mo agoI wonder if you'd have more reliable internet browsing with a browser that was a CF verified bot.
- pineapplepizza6 2mo ago[dead]