5 ms·
Another neat trick is to add a link in robots.txt and instruct bots to stay away. If they don’t, you add them to your blocklist
by hoppla 4y ago
Another neat trick is to add a link in robots.txt and instruct bots to stay away. If they don’t, you add them to your blocklist
- justin_oaks 4y agoI was confused at what you were saying at first. For those that may also be confused: You can add something like this to your robots.txt: User-agent: * Disallow: /some/unguessible/url And then you ban any IPs/bots that visit that URL.