8 ms·
wait why not robots.txt?
by gooob 7mo ago
wait why not robots.txt?
- reconnecting 7mo agoGood question, at least OAI-SearchBot is hitting robots.txt. I assume the real issue is that what overloads the servers like security bots, SEO crawlers, and data companies — are the ones that don't respect robots.txt in full, but they wouldn't respect LLMs.txt either.