13 ms·
Sort of explained already, but "bad" bots would never go look for that file. And a good bot probably already identifies itself the the request, so no need to lo
by freeall 14y ago
Sort of explained already, but "bad" bots would never go look for that file. And a good bot probably already identifies itself the the request, so no need to look through robots.txt.
- fsniper 14y agoThat's it. A "bad bot" would not check robots.txt, but a legitimate user would check it. So looking for the software not checking robots.txt combined with user agent matching for good bots, you would have a good matching ratio.