5 ms·
It's interesting they're disallowing 008. I work at a popular webhost, and we've had problems with this crawler too (It's distributed, and can crawl things a li
by dibarra 14y ago
It's interesting they're disallowing 008. I work at a popular webhost, and we've had problems with this crawler too (It's distributed, and can crawl things a little too fast).
- nintax 14y agoDid you notice the bottom 3 rules? Maybe I should resubmit and change the title...