10 ms·
Where does one find a good robots.txt? Are there any well maintained out there?
by stefanka 7mo ago
Where does one find a good robots.txt? Are there any well maintained out there?
- skrtskrt 7mo agoCloudflare actually has this as a free tier feature so even if you don't want to use it for your site you can just setup a throwaway domain on Cloudflare and periodically copy the robots.txt they generate from your scraper allow/block preferences, since they'll be keeping up to date with all the latest.
- dmit 7mo agohttps://github.com/ai-robots-txt/ai.robots.txt https://github.com/ai-robots-txt/ai.robots.txt