12 ms·

Does Yahoo! not have a robots.txt file? http://yahoo.com/robots.txt http://yahoo.com/robots.txt Sorry, the page you requested was not found.

• jdrock 17y ago
  If that page doesn't exist, then according to specification, they don't and you can crawl any page.
  • twoz 17y ago
    Update: Some Yahoo! subdomains have 'em. Yes: http://search.yahoo.com/robots.txt http://groups.yahoo.com/robots.txt http://realestate.yahoo.com/robots.txt No: http://maps.yahoo.com/robots.txt http://omg.yahoo.com/robots.txt