6 ms·

This is something I simply cannot agree with. People are using ROBOTS.TXT for all kinds of reasons, such as blocking unwanted careless webcrawler or indexing si

This is something I simply cannot agree with. People are using ROBOTS.TXT for all kinds of reasons, such as blocking unwanted careless webcrawler or indexing single page application. I mean come on. How can you say something like that just because eliminating ROBOTS.TXT would potentially benefit your business.

deleted 10y ago
[deleted]
saurik 10y ago
> ...such as blocking unwanted careless webcrawler... This file doesn't "block" anything, it simply asks the robot to do something, which implies that it is probably being abnormally careful: truly annoying, unwanted, careless robots might follow these guidelines, but that seems like a stretch. In reality, this file exists so that extra careful robots are able to get feedback from websites that have extremely narrow bandwidth availability or extremely high generation cost... concepts which this article makes a pretty compelling argument for "that doesn't make sense". In practice, this file then makes the owners of websites sometimes think "I can build something weirdly broken (such as a procedurally generated content tarpit, or mapping anonymous GET requests to database insertions) and just rely on this file to explain what I did along with enforcing rate limits and boundaries"... and then an "unwanted careless webcrawler" comes along and causes them serious issues. It is akin to having your entire webserver crash if someone sends you a non-ASCII character in a form field, but thinking "this will work out: I have a little flag in my HTML file that makes it clear I only accept ASCII". If you absolutely feel like you need to block something, then actually block it: any robot gracious enough to pay attention to this file is also going to send a useful user agent, and you can use that to return a legitimate 403.

6 ms·

This is something I simply cannot agree with. People are using ROBOTS.TXT for all kinds of reasons, such as blocking unwanted careless webcrawler or indexing si

by popobobo 10y ago

deleted 10y ago
[deleted]
saurik 10y ago
> ...such as blocking unwanted careless webcrawler... This file doesn't "block" anything, it simply asks the robot to do something, which implies that it is probably being abnormally careful: truly annoying, unwanted, careless robots might follow these guidelines, but that seems like a stretch. In reality, this file exists so that extra careful robots are able to get feedback from websites that have extremely narrow bandwidth availability or extremely high generation cost... concepts which this article makes a pretty compelling argument for "that doesn't make sense". In practice, this file then makes the owners of websites sometimes think "I can build something weirdly broken (such as a procedurally generated content tarpit, or mapping anonymous GET requests to database insertions) and just rely on this file to explain what I did along with enforcing rate limits and boundaries"... and then an "unwanted careless webcrawler" comes along and causes them serious issues. It is akin to having your entire webserver crash if someone sends you a non-ASCII character in a form field, but thinking "this will work out: I have a little flag in my HTML file that makes it clear I only accept ASCII". If you absolutely feel like you need to block something, then actually block it: any robot gracious enough to pay attention to this file is also going to send a useful user agent, and you can use that to return a legitimate 403.