6 ms·
Very little of it. When you see a million IPs systematically working their way through your URL space, it's pretty clear that there's a central control node be
by corbet 2mo ago
Very little of it. When you see a million IPs systematically working their way through your URL space, it's pretty clear that there's a central control node behind it all.
- everfrustrated 2mo agoYour earlier article suggests you aren't using a CDN. Might be well worth looking into - not for any bot detection so much as just having a good old fashioned cache in front of you.
- mplewis 2mo agoAs someone who operates a wiki, this does not solve the problem.
- solid_fuel 2mo agoCaches only help for pages that have been requested recently. The behavior of crawlers - going from one page to the next across the whole site - will probably not be mitigated significantly by a cache.