You're not allowing a random person access to the agent, you're allowing the agent access to your emails. But since everybody can send you an email, your agent

You're not allowing a random person access to the agent, you're allowing the agent access to your emails. But since everybody can send you an email, your agent is going to be exposed to a lot of stuff. It's just like regular emails: you can always get spam, malware and other trash and when they reach your system they can cause damage. The agent is just a new level on the stack (operating system, email client etc) that can now be compromised by a simple email.