5 ms·Songkick's robots.txt has a few special rules.1 points by nintax 14y agodibarra 14y agoIt's interesting they're disallowing 008. I work at a popular webhost, and we've had problems with this crawler too (It's distributed, and can crawl things a little too fast).nintax 14y agoDid you notice the bottom 3 rules? Maybe I should resubmit and change the title...