10 ms·
Hi all, I made this site. I wasn't expecting this post, but happy to answer questions and take feedback. Hopefully it's pretty self-explanatory but I made this
by gavinhking 3y ago
Hi all, I made this site. I wasn't expecting this post, but happy to answer questions and take feedback.
Hopefully it's pretty self-explanatory but I made this website as a simple resource for people who want to stay up to date with the ever-changing cast of AI user agents.
Feel free to sign up with the Google Form to get notified when this list is updated. And if you know of any agents I'm missing, please submit them. Thanks!
- schleck8 3y ago> cohere-ai is an unconfirmed agent possibly dispatched by Cohere's AI chat products in response to user prompts when it needs to retrieve content on the internet. This is almost certainly the coral web grounding found as an option on coral.cohere.ai
- gavinhking 3y agoThank you, will try to confirm this.
- lancesells 3y agoGreat resource! I'm wondering why you put links to your site in the robots.txt example? It's like there's a clear utility you've made but then the links cheapen it a bit. Maybe I'm missing the reason?
- gavinhking 3y agoYou're not missing anything, they're just for any humans reading the file who want to know more about the agent. This is also just an example file.
- lakomen 3y agoIs there a "export all to robots.txt disallow" feature? If not, that would be great to have
- gavinhking 3y agoYup, here is an example with all listed agents, although you probably want to modify it to not fully block all of them: https://darkvisitors.com/robots-txt-builder https://darkvisitors.com/robots-txt-builder