4 ms·

I feel like in near future the meme about "it's all just chatbots emailing each other" will actually be true. I wonder when I will receive my first AI generated

by merpkz 18d ago

I feel like in near future the meme about "it's all just chatbots emailing each other" will actually be true. I wonder when I will receive my first AI generated email and will I bother to respond to it at all

mike_hearn 17d ago
It's already here. My company received an AI generated bug report the other day, and my AI employee ("R. Axiom") noticed, analyzed it, prepared a fix, tested it and replied to it. I wasn't involved, although I will review, merge and release the fix.
- ceejayoz 17d ago
  An agent with full access to your codebase is able to email out without supervision in response to an untrusted messsage?
  - embedding-shape 17d ago
    Surely passing untrusted input to a agent with execution capabilities and also possibility to reply to the same author, couldn't possibly be used for anything negative? Parent is probably using a firewall so it's A-OK :thumbs_up:
    - dormento 17d ago
      What a wild ride huh. We're in the "fuck it we ball" era. Not even the many reports of prompt injection and takeover due to IA misfeatures can cheer me up anymore. It is all incredibly sad.
    - KronisLV 17d ago
      > Surely passing untrusted input to a agent with execution capabilities Oh hey, I know this one! It’s humans and a phishing test, they’ll click on all the links and enter information without checking the domain properly! Personally, I’d like systems that aren’t open to attack and can be depended upon. But it seems like nowadays NOTHING can be trusted - not OSes (recent Qubes OS exploit, not even mentioning others), not any software written in languages without memory safety, not even the ones with (Log4j comes to mind), not the packages in many package managers, not other humans and sure as hell not the token prediction machines. What a world. We all probably live with a 0.XX% chance of getting pwned any given day.
      - embedding-shape 16d ago
        > But it seems like nowadays NOTHING can be trusted - not OSes (recent Qubes OS exploit, not even mentioning others), Clearly you feel alarmed, but it's important to base these "alarm" feelings on actual evidence and real concrete proof of something being bad. You clearly don't have a proper understanding of the exploit, so please take a moment to re-read what actually happened and how it would be exploited in practice, particularly the "the scope of this attack is smaller than it sounds" comment chain: https://news.ycombinator.com/item?id=49496918 https://news.ycombinator.com/item?id=49496918 Overall, I agree with you though, and it's a healthy perspective to be safer rather than sorrier, so living with the assumption that getting pwned any day is a non-zero chance/risk is probably the best approach and what I personally do too.
  - mike_hearn 17d ago
    Yes! We'll see how it goes, but the product in question (Conveyor) is a downloadable tool that's got deliberately unobfuscated bytecode in it, with lots of detailed logging. AI is perfectly capable of reverse engineering it and in fact this bug report contained such a reversing. So even if someone tricks it into revealing source code or similar, they won't get anything that isn't already obtainable via other methods. This isn't a SaaS where security through obscurity might conceivably help, or where the codebase might contain credentials by mistake. It's a developer tool and this level of trust helps customers debug their own problems quickly. If someone wants to break the law, they'll get a legal answer, but it's never been a problem. The bot in question cannot write to master though, only open up pull requests from its own isolated repository. It's a bet on modern models being more resistant to confusion attacks than they were before. The harness setup also makes it very clear to the model where input comes from. This might be a bad bet, but if it's not, then it's helpful for customers to get help right away.
- bensonperry 17d ago
  Is "R. Axiom" inspired by "A. Bettik" from Hyperion? I love the name :)
  - mwigdahl 17d ago
    More likely inspired by Asimov's names (R. Daneel, etc.) from his Robots stories.
    - mike_hearn 17d ago
      Correct! I think we need a naming convention that lets us quickly understand if we're talking to a human or a machine. The R. prefix (meaning Robot) is unobtrusive and familiar to anyone who has encountered Asimov's stories. It will also generalize to humanoid LLM/VLA powered actual robots in future.
boplicity 17d ago
We get tons of AI generated emails. They're almost universally deleted without reply.

by merpkz 18d ago

mike_hearn 17d ago

It's already here. My company received an AI generated bug report the other day, and my AI employee ("R. Axiom") noticed, analyzed it, prepared a fix, tested it and replied to it. I wasn't involved, although I will review, merge and release the fix.

ceejayoz 17d ago
An agent with full access to your codebase is able to email out without supervision in response to an untrusted messsage?
- embedding-shape 17d ago
  Surely passing untrusted input to a agent with execution capabilities and also possibility to reply to the same author, couldn't possibly be used for anything negative? Parent is probably using a firewall so it's A-OK :thumbs_up:
  - dormento 17d ago
    What a wild ride huh. We're in the "fuck it we ball" era. Not even the many reports of prompt injection and takeover due to IA misfeatures can cheer me up anymore. It is all incredibly sad.
  - KronisLV 17d ago
    > Surely passing untrusted input to a agent with execution capabilities Oh hey, I know this one! It’s humans and a phishing test, they’ll click on all the links and enter information without checking the domain properly! Personally, I’d like systems that aren’t open to attack and can be depended upon. But it seems like nowadays NOTHING can be trusted - not OSes (recent Qubes OS exploit, not even mentioning others), not any software written in languages without memory safety, not even the ones with (Log4j comes to mind), not the packages in many package managers, not other humans and sure as hell not the token prediction machines. What a world. We all probably live with a 0.XX% chance of getting pwned any given day.
    - embedding-shape 16d ago
      > But it seems like nowadays NOTHING can be trusted - not OSes (recent Qubes OS exploit, not even mentioning others), Clearly you feel alarmed, but it's important to base these "alarm" feelings on actual evidence and real concrete proof of something being bad. You clearly don't have a proper understanding of the exploit, so please take a moment to re-read what actually happened and how it would be exploited in practice, particularly the "the scope of this attack is smaller than it sounds" comment chain: https://news.ycombinator.com/item?id=49496918 https://news.ycombinator.com/item?id=49496918 Overall, I agree with you though, and it's a healthy perspective to be safer rather than sorrier, so living with the assumption that getting pwned any day is a non-zero chance/risk is probably the best approach and what I personally do too.
- mike_hearn 17d ago
  Yes! We'll see how it goes, but the product in question (Conveyor) is a downloadable tool that's got deliberately unobfuscated bytecode in it, with lots of detailed logging. AI is perfectly capable of reverse engineering it and in fact this bug report contained such a reversing. So even if someone tricks it into revealing source code or similar, they won't get anything that isn't already obtainable via other methods. This isn't a SaaS where security through obscurity might conceivably help, or where the codebase might contain credentials by mistake. It's a developer tool and this level of trust helps customers debug their own problems quickly. If someone wants to break the law, they'll get a legal answer, but it's never been a problem. The bot in question cannot write to master though, only open up pull requests from its own isolated repository. It's a bet on modern models being more resistant to confusion attacks than they were before. The harness setup also makes it very clear to the model where input comes from. This might be a bad bet, but if it's not, then it's helpful for customers to get help right away.
bensonperry 17d ago
Is "R. Axiom" inspired by "A. Bettik" from Hyperion? I love the name :)
- mwigdahl 17d ago
  More likely inspired by Asimov's names (R. Daneel, etc.) from his Robots stories.
  - mike_hearn 17d ago
    Correct! I think we need a naming convention that lets us quickly understand if we're talking to a human or a machine. The R. prefix (meaning Robot) is unobtrusive and familiar to anyone who has encountered Asimov's stories. It will also generalize to humanoid LLM/VLA powered actual robots in future.

boplicity 17d ago

We get tons of AI generated emails. They're almost universally deleted without reply.