5 ms·

I made this, let me know if you have questions or feedback.

• tbeseda 2y ago
  Thanks for the work on this! I automated my site's robots.txt[0] by scraping your site. It would be extra nice if darkvisitor.com exposed a plain text version or JSON representation of the list. [0] https://tbeseda.com/blog/automating-my-robots-txt-to-block-ai-user-agents https://tbeseda.com/blog/automating-my-robots-txt-to-block-a...
  • glynnormington 2y ago
    That was one of the points of the new repo. A plain text version of the file is https://raw.githubusercontent.com/ai-robots-txt/ai.robots.txt/main/robots.txt https://raw.githubusercontent.com/ai-robots-txt/ai.robots.tx... The other point was to make this community maintained rather than rely on one source to provide all the inputs.
    • tbeseda 2y ago
      Definitely! And I'll likely use that raw URL in my tooling going forward. So thanks for starting the repo. I'm thinking it also helps to bring up a feature request on the source material so we can all limit the drift. ie. if darkvisitors.com had a sort of plain text API, your repo could check for new entries via GH Actions and create issues or even PRs.