5 ms·
> It's also a page that's never visited by humans. Never is a strong word. I have definitely visited robots.txt of various websites for a variety of random rea
by gabeio 1y ago
> It's also a page that's never visited by humans.
Never is a strong word. I have definitely visited robots.txt of various websites for a variety of random reasons.
- remembering the format
- seeing what they might have tried to "hide"
- using it like a site's directory
- testing if the website is working if their main dashboard/index is offline
- sdenton4 1y agoAre you sure you are human?
- gspencley 1y agoYes. I have checked many checkboxes that say "Verify You Are a Human" and they have always confirmed that I am. In fairness, however, my daughters ask me that question all the time and it is possible that the verification checkboxes are lying to me as part of some grand conspiracy to make me think I am a human when I am not.
- nullc 1y agohttps://www.youtube.com/watch?v=4VrLQXR7mKU https://www.youtube.com/watch?v=4VrLQXR7mKU --- though I think passing them is more a sign that you're a robot than anything else.