5 ms·Bad guys might download the robots.txt to find out the stuff they don't want them to crawl.by aequitas 1y agoBad guys might download the robots.txt to find out the stuff they don't want them to crawl.