5 ms·
Unrelated but it looks like www.aol.com's robots.txt is served as text/html http://www.aol.com/robots.txt http://www.aol.com/robots.txt Is this a common mistak
by blossoms 13y ago
Unrelated but it looks like www.aol.com's robots.txt is served as text/html http://www.aol.com/robots.txt http://www.aol.com/robots.txt
Is this a common mistake?
- mkonecny 13y agoIt just issues a "HTTP/1.1 302 Moved Temporarily" directed to their homepage. Requesting an invalid file such as "robots.txtsdfa32r523" has the same effect, so they probably don't have a robots file at all.
- judk 13y agoHuh? No, it is a regular robots.txt file
- tonyedgecombe 13y agoIt redirects requests from the UK.
- blossoms 13y agohttp://i.imgur.com/Jmjf3y2.png http://i.imgur.com/Jmjf3y2.png
- saltysugar 13y agoYes, from the server side's mishandling of TXT extension. Probably the server put the MIME type in the HTTP header as "HTML" instead of TXT, and the browser renders the page as such.
- pjscott 13y agoFrom the headers: Content-Type: text/html;charset=UTF-8 It also tries to set no less than four cookies.