6 ms·
Crawler, AI or not, cannot afford to pay per visit. The entire model of crawling works because the incremental cost of each crawl is so low. Even fractions of a
by azeemba 3mo ago
Crawler, AI or not, cannot afford to pay per visit. The entire model of crawling works because the incremental cost of each crawl is so low. Even fractions of a penny would be prohibitive.
- wmf 3mo agoIf you're paying per token for AI, you can also pay a smaller amount to use the Web.
- mx7zysuj4xew 3mo agoAs it should be. Bots are still user agents much like browsers