10 ms·
This is kinda amusing. robots.txt main purpose back in the day was curtailing penalties in the search engines when you got stuck maintaining a badly-built dyna
by Falkon1313 1y ago
This is kinda amusing.
robots.txt main purpose back in the day was curtailing penalties in the search engines when you got stuck maintaining a badly-built dynamic site that had tons of dynamic links and effectively got penalized for duplicate content. It was basically a way of saying "Hey search engines, these are the canonical URLs, ignore all the other ones with query parameters or whatever that give almost the same result."
It could also help keep 'nice' crawlers from getting stuck crawling an infinite number of pages on those sites.
Of course it never did anything for the 'bad' crawlers that would hammer your site! (And there were a lot of them, even back then.) That's what IP bans and such were for. You certainly wouldn't base it on something like User-Agent, which the user agent itself controlled! And you wouldn't expect the bad bots to play nicely just because you asked them.
That's about as naive as the Do-Not-Track header, which was basically kindly asking companies whose entire business is tracking people to just not do that thing that they got paid for.
Or the Evil Bit proposal, to suggest that malware should identify itself in the headers. "The Request for Comments recommended that the last remaining unused bit, the "Reserved Bit" in the IPv4 packet header, be used to indicate whether a packet had been sent with malicious intent, thus making computer security engineering an easy problem – simply ignore any messages with the evil bit set and trust the rest."
- pi_22by7 1y agoSo it did the same work that a sitemap does? Interesting. Or maybe more like the opposite: robots.txt told bots what not to touch, while sitemaps point them to what should be indexed. I didn’t realize its original purpose was to manage duplicate content penalties though. That adds a lot of historical context to how we think about SEO controls today.
- JimDabell 1y ago> I didn’t realize its original purpose was to manage duplicate content penalties though. That wasn’t its original purpose. It’s true that you didn’t want crawlers to read duplicate content, but it wasn’t because search engines penalised you for it – WWW search engines had only just been invented and they didn’t penalise duplicate content. It was mostly about stopping crawlers from unnecessarily consuming server resources. This is what the RFC from 1994 says: > In 1993 and 1994 there have been occasions where robots have visited WWW servers where they weren't welcome for various reasons. Sometimes these reasons were robot specific, e.g. certain robots swamped servers with rapid-fire requests, or retrieved the same files repeatedly. In other situations robots traversed parts of WWW servers that weren't suitable, e.g. very deep virtual trees, duplicated information, temporary information, or cgi-scripts with side-effects (such as voting). — https://www.robotstxt.org/orig.html https://www.robotstxt.org/orig.html
- Quarrel 1y ago> It was mostly about stopping crawlers from unnecessarily consuming server resources. Very much so. Computation was still expensive, and http servers were bad at running cgi scripts (particularly compared to the streamlined amazing things they can be today). SEO considerations came way way later. They were also used, and still are, by sites that have good reasons to not want results in search engines. Lots of court files and transcripts, for instance, are hidden behind robots.txt.
- EPendragon 1y ago> Computation was still expensive I think this is still relevant today in cases where there are not many resources available: think free tiers, smallest fixed cost/fixed allocation scenarios, etc.
- MiddleMan5 1y agoIt should be noted here that the Evil Bit proposal was an April Fools RFC https://datatracker.ietf.org/doc/html/rfc3514 https://datatracker.ietf.org/doc/html/rfc3514
- Y_Y 1y agoWhile we're at it, it should be noted that Do Not Track was not, apparently, a joke. It's the same as a noreply email, if you can get away with sticking your fingers in your ears and humming when someone is telling you something you don't want to hear, and you have a computer to hide behind, then it's all good.
- cma 1y agoDo Not Track had a chance to get into law, which if it did would be good that the code and standard were already in place.
- sebstefan 1y agoThere should be a law against displaying a cookie consent box to a user who has their Do Not Track header set.
- MaxBarraclough 1y agoNot all that far-fetched, Global Privacy Control is legally binding in California. https://en.wikipedia.org/wiki/Global_Privacy_Control https://en.wikipedia.org/wiki/Global_Privacy_Control https://news.ycombinator.com/item?id=43377867 https://news.ycombinator.com/item?id=43377867
- vbezhenar 1y agoHow is "Do Not Track" is a joke, but website presenting a button "Do not use cookies" is not? What's the difference?
- anamexis 1y agoFor one, Do Not Track is on the client side and you just hope and pray that the server honors it, whereas cookie consent modals are something built by and placed in the website. I think you can reasonably assume that if a website went through the trouble of making such a modal (for legal compliance reasons), the functionality works (also for legal compliance reasons). And, you as the client can verify whether it works, and can choose not to store them regardless.
- tbrownaw 1y ago> And you wouldn't expect the bad bots to play nicely just because you asked them. Well, yes, the point is to tell the bots what you've decided to consider "bad" and will ban them for. So that they can avoid doing that. Which of course only works to the degree that they're basically honest about who they are or at least incompetent at disguising themselves.
- gbalduzzi 1y agoI think it depends on the definition of bad. I always consider "good" a bot that doesn't disguise itself and follows the robots.txt rules. I may not consider good the final intent of the bot or the company behind it, but the crawler behaviour is fundamentally good. Especially considering the fact that it is super easy to disguise a crawler and not follow the robots conventions
- atoav 1y agoWell you as the person running a website can define unilaterally what you consider good and bad. You may want bots to crawl everything, nothing or (most likely) something inbetween. Then you judge bots based on those guidelines. You know like a solicitor that rings your bell that has a text above it saying "No solicitors", certain assumptions can be made about those who ignore it.
- pjmlp 1y agoSome people just believe that because someone says so, everyone will nicely obey and follow the rules, don't know maybe it is a cultural thing.
- vintagedave 1y agoOr a positive belief in human nature. I admit I'm one of those people. After decades where I should perhaps be a bit more cynical, from time to time I am still shocked or saddened when I see people do things that benefit themselves over others. But I kinda like having this attitude and expectation. Makes me feel healthier.
- tuyiown 1y agoI deeply agree with you, and I'd like to add: Trust by default, also by default, never ignoring suspicious signals. Trust is not being naïve, I find the confusion of both very worrying.
- Sammi 1y agoYou don't have to go as far as to straight up "trust by default". You can instead "give a chance" by default, which is the middle path. Actually Veritasium has a great video about this. It's proven as the most effective strategy in monte carlo simulation. EDIT: This one: https://youtu.be/mScpHTIi-kM https://youtu.be/mScpHTIi-kM
- chasd00 1y agoi like that Veritasium vid a lot, i've watched it a couple times. The thing is, there's no way to retaliate against a crawler ignoring robots.txt. IP bans don't work, user agent bans don't work, there's no human to shame on social media ether. If there's no way to retaliate or provide some kind of meaningful negative feedback then the whole thing breaks down. Back to the Veritasium video, if a crawler defects they reap the reward but there's no way for the content provider to defect so the crawler defects 100% of the time and gets 100% of the defection points. I can't remember when i first read the rfp for robots.txt but I do remember finding it strange that it was a "pretty please" request against a crawler that has a financial incentive to crawl as much as it can. Why even go through the effort to type it out? EDIT: i thought about it for a min, i think in the olden days a crawler crawling every path through a website could yield an inferior search index. So robots.txt gave search engines a hint on what content was valuable to index. The content provider gained because their SEO was better (and cpu util. lower) and the search engine gained because their index was better. So there was an advantage to cooperation then but with crawlers feeding LLMs that isn't the case.
- nullc 1y ago> That's about as naive as the Do-Not-Track header, which was basically kindly asking companies whose entire business is tracking people to just not do that thing that they got paid for. It's usually a bad default to assume incompetence on the part of others, especially when many experienced and knowledgeable people have to be involved to make a thing happen. The idea behind the DNT header was to back it up with legislation-- and sure you can't catch and prosecute all tracking, but there are limitations on the scale of criminal move fast and break things before someone rats you out. :P
- franga2000 1y agoI still see the value in robots.txt and DNT as a clear, standardised way of posting a "don't do this" sign that companies could be forced to respect through legal means. The GDPR requires consent for tracking. DNT is a very clear "I do not consent" statement. It's a very widely known standard in the industry. It would therefore make sense that a court would eventually find companies not respecting it are in breach of the GDPR. That was a theory at least...
- EPendragon 1y agoWould robot traffic be considered tracking in light of GDPR standards? As far as I know there are no regulatory rules in relation to enforcing robots behaviors outside of robots.txt, which is more of an honor system.
- franga2000 1y agoDNT and GDPR was just an example. In a court case about tracking, DNT could be found to be a clear and explicit opt out. Similarly, in a case about excessive scraping or the use of scraped information, robots txt could be used as a clear and explicit signal that the site operator does not want their pages harvested. It all but certainly gets rid of the "they put it on the public web so we assumed we can scrape it, we can'task everyone for permission" argument. They can't claim it was "in good faith" if there's a widely-accepted standard for opting out.
- EPendragon 1y agoFair enough. It should be sufficient to say one way or the other.
- PaulHoule 1y agoRobots.txt was created long before Google and before people were thinking about SEO: https://en.wikipedia.org/wiki/Robots.txt https://en.wikipedia.org/wiki/Robots.txt The scenario I remember was that the underfunded math department had an underpowered server connected via a wide and short pipe to the overfunded CS department and webcrawler experiments would crash the math department's web site repeatedly.
- LorenPechtel 1y agoYup. Robots.txt was a don't-swamp-me thing.
- abirch 1y agoWith the advent of AI and the notion of actually going to a website as being quaint: each website should have a humans.txt such as https://www.netflix.com/humans.txt https://www.netflix.com/humans.txt or https://www.google.com/humans.txt https://www.google.com/humans.txt
- PaulHoule 1y agoWhat everybody is missing is that AI inference (not training) is a route out of the enshittification economy. One reason why Cloudflare is harassing you all the time to click on traffic lights and motorcycles is to slam the door from some of the exit routes.
- EPendragon 1y agoI have not heard of humans.txt before. It is apparently used for acknowledgement and crediting the dev team who created the resource.
- EPendragon 1y agoIt is so interesting to track this technology's origin back to the source. It makes sense that it would come from a background of limited resources where things would break if you overwhelm it. It didn't take much to do so.