6 ms·
I think it's a good idea, but it must be coupled with robots.txt.
by java-man 1y ago
I think it's a good idea, but it must be coupled with robots.txt.
- cratermoon 1y agoAI scraper bots don't respect robots.txt
- jsheard 1y agoI think that's the point, you'd use robots.txt to direct Googlebot/Bingbot/etc away from countermeasures that could potentially mess up your SEO. If other bots ignore the signpost clearly saying not to enter the tarpit, that's their own stupid fault.
- reverendsteveii 1y agoThe ones that survive do
- forinti 1y agoI was looking through my logs yesterday. Bad bots don't even read robots.txt.
- extraduder_ire 1y agoThe worst ones treat it as a target.