12 ms·
The litmus test here is whether they support https://blog.cloudflare.com/introducing-pay-per-crawl/ https://blog.cloudflare.com/introducing-pay-per-crawl/ out o
by arianvanp 3mo ago
The litmus test here is whether they support https://blog.cloudflare.com/introducing-pay-per-crawl/ https://blog.cloudflare.com/introducing-pay-per-crawl/ out of the box or not
They do not.
- baby_souffle 3mo agoI don't know who 'they' is here but that's not the point? I would bet that a decent chunk of scraping happens because there is no API (or other machine-focused interface, like RSS). "pay to crawl" sounds like the absolutely laziest possible way that a particular site could bolt on an API.
- mannanj 3mo agohas anyone been using this for success? wondering what kinds of pays they are getting. or are the crawlers just avoiding those sites.
- inigyou 3mo ago"we don't negotiate with terrorists"
- cindyllm 3mo ago[dead]