36 ms·
Anybody know how big robots.txt can be and if anyone has ever given a spider grief by screwing with the size or download rate of robots.txt?
by SageRaven 14y ago
Anybody know how big robots.txt can be and if anyone has ever given a spider grief by screwing with the size or download rate of robots.txt?
- kanzure 14y agoThe robots.txt limit in Google's Webmasterpanelthing is 100,000 characters. Googlebot has been said to stop parsing robots.txt after 500,000 characters. I don't know if that's bytes or characters.
- dsl 14y ago500kb is the hard upper limit in the crawler itself. Limits that are exposed elsewhere (Webmaster Tools, etc) are irrelevant.