5 ms·
I can only think of using 80legs as a crawler, since its distributed enough to make sure that you don't run into any IP address based rate limitation. But it's
by akshayubhat 16y ago
I can only think of using 80legs as a crawler, since its distributed enough to make sure that you don't run into any IP address based rate limitation. But it's just a guess.
- slig2 16y agoYou probably can't do it, because 80 respects robots.txt AFAIK.
- deleted 16y ago[deleted]