5 ms·
Also note, Cloudflare blocks IPs of legitimate automated "good bots" by default, like RSS feed fetchers (1), but allows traffic to some of the most aggressive b
by trekz 2y ago
Also note, Cloudflare blocks IPs of legitimate automated "good bots" by default, like RSS feed fetchers (1), but allows traffic to some of the most aggressive bots on the web, like Google bot and all sorts of AI bots (2).
(1) https://openrss.org/issue/144 https://openrss.org/issue/144
(2) https://radar.cloudflare.com/traffic/verified-bots https://radar.cloudflare.com/traffic/verified-bots - GoogleBot, GPTBot, and AmazonBot are some of the first ones listed.
- theanonymousone 2y agoBlocking RSS fetchers which are "bots" by design, intent, and definition, is really bizarre! Seems the poor indy bots scraping price pages once a week are totally out of lock then.