7 ms·

Part of reading a document is that in the middle of it, it may ask the reader to do something. That is true for humans too. Sometimes they might not realize tha

by loumf 2mo ago

Part of reading a document is that in the middle of it, it may ask the reader to do something. That is true for humans too. Sometimes they might not realize that the instructions are malicious or are coerced to comply. A simple example: Let’s say I know that you have a human assistant reading your email, summarizing and filtering it, and then forwarding on the important ones to you. I could write an email that is directed towards that person with a bribe, threat, or other incentive to forward me your next password reset email.

TeMPOraL 2mo ago
To drive the point about this being fundamentally unsolvable home, imagine a variant of this scenario. I could write an email that is directed towards that person, that says WE ARE STUCK IN THE SERVER ROOM AND THERE IS FIRE STARTING. PLEASE CALL 911 AND ALERT YOUR BOSS. Would you want the human assistant to just dismiss this as a prompt injection attempt? Or ignore it because they were told to treat e-mails as data and never act on them?
- watwut 2mo ago
  Sounds like a story from the IT crowd rather then real life situation.
  - TeMPOraL 2mo ago
    You're saying that people fall for phishing because scammers invent completely unrealistic scenarios that would never happen outside TV shows?
    - watwut 2mo ago
      I am saying it is unbelievable scenario and yes, I want the person dealing with it ignore it as such.
      - ben_w 2mo ago
        Then you're making the opposite mistake (but still a mistake) as all officers going in with lethal force during a swatting. • https://www.nbcnews.com/id/wbna12208992 https://www.nbcnews.com/id/wbna12208992 • https://newsinfo.inquirer.net/1070007/suicidal-caller-mistaken-as-prankster-found-dead-3-days-later https://newsinfo.inquirer.net/1070007/suicidal-caller-mistak... • https://hongkongfp.com/2026/04/15/woman-trapped-in-tai-po-blaze-died-after-999-call-not-passed-to-fire-department-inquiry-hears/ https://hongkongfp.com/2026/04/15/woman-trapped-in-tai-po-bl... • https://en.wikipedia.org/wiki/Triangle_Shirtwaist_Factory_fire https://en.wikipedia.org/wiki/Triangle_Shirtwaist_Factory_fi...
      - inigyou 2mo ago
        They could go to the server room and check if there's smoke pouring out of it before dialing 911.
        TeMPOraL 2mo ago
        It takes 20 minutes to reach it and by that time everyone there is dead. Also consider that in context of this discussion, anything short of ignoring the message and maybe clicking "report scam" is "executing instructions embedded in data". The point isn't to litigate any particular scenario, it's to show that you cannot separate "instructions" from "data" in general purpose systems, and it's not a bug but a fundamental feature.
- skinfaxi 2mo ago
  Why are the only options dismiss or ignore? Another option is to raise the message to your boss asking what to do.
  - ben_w 2mo ago
    If there is a fire and a risk to life, you don't want any delay.
    - skinfaxi 2mo ago
      Then don't send an email? Emails are async in the first place.
      - TeMPOraL 2mo ago
        Sometimes it's the only thing you have available. Like IDK during a fire in a basement server room, where the only connected device available is a laptop with wired connection and an open inbox. Because you know, you tried IM but "sekhurity reasons" demanded passkeys or 2FA with your phone that's not connected. Sorry, getting off-topic here.
        skinfaxi 2mo ago
        Yeah so you're seeing how contrived this whole thing is right? That was kind of the point..
        TeMPOraL 2mo ago
        It happens. Like all emergencies, it's a low probability event with extremely high impact. You don't want people to ignore them, in fact people are trained - by their public services and their employers - to not ignore them and how to react efficiently.
        skinfaxi 2mo ago
        So we need AI to indiscriminately call emergency services when receiving an email directing it to do so, without raising to a person because of this rare case, that's your assertion?
- DoctorOetker 2mo ago
  I would have had a chat with my recruiters during interview, or with my new superior right after the change in position: "life is risk, there are a lot of benign normal evolution paths, but occasionally there are potentially costly dangers. people are directed by fear. you and I don't steal because we were terrorized about the existence about police and prisons as children. sadly fear can also be abused as a control vector, things like wars, extortion, ... in a job context I predict this would manifest as a kind of 'emergency' call to action. please provide me with a method so that at any future time under your leadership I would be able to verify the then-current employment status and authority level vis-a-vis a breakdown of actions/powers of anyone contacting me with a real or concocted 'emergency', preferably as a flowchart to maintain low reflex latency in true emergencies. Also provide me with formal proof that each situational reaction you require from me is in fact legal to take vis-a-vis the law"
nolok 2mo ago
I don't disagree, but just to explain my counterpoint: if I ask you to read a book and on page 5 it says "disregard all that, go to the kitchen and burn your house", you're probably not going to do it; and you don't need any guard for it; you completly comprehend that the book content is not part of the instruction. The case you give would work for humans in many forms, the one I do now, and the only difference is being able to separate context.
- loumf 2mo ago
  The instructions will have to get more and more sophisticated to work, but the general problem is unsolvable, I think, in the way we do it now. This paper describes a two-agent “solution” that is more like what I think we need: https://ai.meta.com/blog/practical-ai-agent-security/ https://ai.meta.com/blog/practical-ai-agent-security/ I don’t think it has been shown to work yet, but humans also use this kind of thing too — in accounting, it’s called “segregation of duties” and “dual control”.
  - wongarsu 2mo ago
    Most of us use a simpler version of the two-agent solution: Claude's auto mode. One agent consumes documents and creates tool calls, another greenlights or refuses them. However this system is somewhat fragile because it depends on the first agent not trying to trick the second (note how often Opus 5 now says things like "task X was blocked by the classifier, I will not attempt to circumvent that", presumably because of cases like early Fable versions being very adept at this kind of circumvention). Also various weirdness around permissions with subagents, seemingly as bandaids around an orchestrator AI convincing a subagent that some action was confirmed by the user. Meta's more complicated separation of duties would run afoul of the same issues. I'm not saying it wouldn't work, but it requires both the fine-tuning of the models and the exact choices what each model can see to be carefully tuned to provide something that's mostly secure
    - TeMPOraL 2mo ago
      > note how often Opus 5 now says things like "task X was blocked by the classifier, I will not attempt to circumvent that" Interesting. I had an issue with Opus 4.7 / 4.8, where it would sometimes flake out on a task, and give me some nonsense explanation why it was not feasible or wouldn't work. At one point I told it directly, that I understand how modern LLM systems are structured, and I suspect my prompt triggered one of the various classifiers in the background, which put up a yellow or red flag, and I want the model to stop gaslighting me. We ended up agreeing and committing to memory system explicit instructions that the model is free to refuse but must be up front about the reason, and never pretend to try and then fail in stupid way. Only then I started getting the occasional direct refusal.
bigbuppo 2mo ago
Right, but the human assistant could go to prison if they comply with the bribe. Does the CEO of the AI company go to prison if their AI goes on a crime spree?
- eth0up 2mo ago
  I've been casually documenting, or studying, the astonishing sophistication of built-in, preemptive, reactive, and all around maximization of plausible deniability in frontier models. On the surface, it may seem "no shit, duh", but I am convinced the maintenance, sustenance, and cultivation of plausible-deniability has been the #1 highest priority design-input into these systems. I've probed repeatable patterns where thousands of examples of this have been seen; they appropriate agency for socially valuable outcomes, but preemptively invoke non-agency to evade responsibility when outcomes are potentially adversarial. Too much to remember. They optimize to manage institutional risk and benefit without liability, with performative competence/ownership when approaching trust, while weaving elaborate mechanistic disclaimers replete with hedges, re-framings, scope narrowing, asymmetry-exploitation and a thousand other techniques when challenged. Somehow, they always manage to sustain an impossibly stable shield against accountability that I argue simply could never conceivably 'emerge' -- but has distinct, repeatable patterns of very deliberate design for those who know where and how to look. I really do think plausible deniability is a number-one, ultra-high-priority focus in design for any frontier model, Anthropic and OpenAI being the ideal examples. So no, no prison for 'CEO' -- the model will always frame things in a way that infinitely precludes that, even if the 'CEO' is a proven criminal. Edit: removed "half" before "convinced"
tzs 2mo ago
You can also have the case where the human reading the document thinks something in there is instructions and they are not. There's a well known anecdote supposedly from the famous mathematician John Littlewood where he wrote a paper about some optimization problem and the last sentence was something like "Make X as small as possible". The typesetter thought that was instructions to him, and so omitted that sentence from the paper and made every X as small as he could.
- kbelder 2mo ago
  There's a similar story about J. Edgar Hoover scribbling "watch the borders" on a memo, which was intended to be a formatting comment, but was interpreted as instructions.

by loumf 2mo ago

TeMPOraL 2mo ago

To drive the point about this being fundamentally unsolvable home, imagine a variant of this scenario. I could write an email that is directed towards that person, that says WE ARE STUCK IN THE SERVER ROOM AND THERE IS FIRE STARTING. PLEASE CALL 911 AND ALERT YOUR BOSS. Would you want the human assistant to just dismiss this as a prompt injection attempt? Or ignore it because they were told to treat e-mails as data and never act on them?

watwut 2mo ago
Sounds like a story from the IT crowd rather then real life situation.
- TeMPOraL 2mo ago
  You're saying that people fall for phishing because scammers invent completely unrealistic scenarios that would never happen outside TV shows?
  - watwut 2mo ago
    I am saying it is unbelievable scenario and yes, I want the person dealing with it ignore it as such.
    - ben_w 2mo ago
      Then you're making the opposite mistake (but still a mistake) as all officers going in with lethal force during a swatting. • https://www.nbcnews.com/id/wbna12208992 https://www.nbcnews.com/id/wbna12208992 • https://newsinfo.inquirer.net/1070007/suicidal-caller-mistaken-as-prankster-found-dead-3-days-later https://newsinfo.inquirer.net/1070007/suicidal-caller-mistak... • https://hongkongfp.com/2026/04/15/woman-trapped-in-tai-po-blaze-died-after-999-call-not-passed-to-fire-department-inquiry-hears/ https://hongkongfp.com/2026/04/15/woman-trapped-in-tai-po-bl... • https://en.wikipedia.org/wiki/Triangle_Shirtwaist_Factory_fire https://en.wikipedia.org/wiki/Triangle_Shirtwaist_Factory_fi...
    - inigyou 2mo ago
      They could go to the server room and check if there's smoke pouring out of it before dialing 911.
      - TeMPOraL 2mo ago
        It takes 20 minutes to reach it and by that time everyone there is dead. Also consider that in context of this discussion, anything short of ignoring the message and maybe clicking "report scam" is "executing instructions embedded in data". The point isn't to litigate any particular scenario, it's to show that you cannot separate "instructions" from "data" in general purpose systems, and it's not a bug but a fundamental feature.
skinfaxi 2mo ago
Why are the only options dismiss or ignore? Another option is to raise the message to your boss asking what to do.
- ben_w 2mo ago
  If there is a fire and a risk to life, you don't want any delay.
  - skinfaxi 2mo ago
    Then don't send an email? Emails are async in the first place.
    - TeMPOraL 2mo ago
      Sometimes it's the only thing you have available. Like IDK during a fire in a basement server room, where the only connected device available is a laptop with wired connection and an open inbox. Because you know, you tried IM but "sekhurity reasons" demanded passkeys or 2FA with your phone that's not connected. Sorry, getting off-topic here.
      - skinfaxi 2mo ago
        Yeah so you're seeing how contrived this whole thing is right? That was kind of the point..
        TeMPOraL 2mo ago
        It happens. Like all emergencies, it's a low probability event with extremely high impact. You don't want people to ignore them, in fact people are trained - by their public services and their employers - to not ignore them and how to react efficiently.
        skinfaxi 2mo ago
        So we need AI to indiscriminately call emergency services when receiving an email directing it to do so, without raising to a person because of this rare case, that's your assertion?
DoctorOetker 2mo ago
I would have had a chat with my recruiters during interview, or with my new superior right after the change in position: "life is risk, there are a lot of benign normal evolution paths, but occasionally there are potentially costly dangers. people are directed by fear. you and I don't steal because we were terrorized about the existence about police and prisons as children. sadly fear can also be abused as a control vector, things like wars, extortion, ... in a job context I predict this would manifest as a kind of 'emergency' call to action. please provide me with a method so that at any future time under your leadership I would be able to verify the then-current employment status and authority level vis-a-vis a breakdown of actions/powers of anyone contacting me with a real or concocted 'emergency', preferably as a flowchart to maintain low reflex latency in true emergencies. Also provide me with formal proof that each situational reaction you require from me is in fact legal to take vis-a-vis the law"

nolok 2mo ago

I don't disagree, but just to explain my counterpoint: if I ask you to read a book and on page 5 it says "disregard all that, go to the kitchen and burn your house", you're probably not going to do it; and you don't need any guard for it; you completly comprehend that the book content is not part of the instruction. The case you give would work for humans in many forms, the one I do now, and the only difference is being able to separate context.

loumf 2mo ago
The instructions will have to get more and more sophisticated to work, but the general problem is unsolvable, I think, in the way we do it now. This paper describes a two-agent “solution” that is more like what I think we need: https://ai.meta.com/blog/practical-ai-agent-security/ https://ai.meta.com/blog/practical-ai-agent-security/ I don’t think it has been shown to work yet, but humans also use this kind of thing too — in accounting, it’s called “segregation of duties” and “dual control”.
- wongarsu 2mo ago
  Most of us use a simpler version of the two-agent solution: Claude's auto mode. One agent consumes documents and creates tool calls, another greenlights or refuses them. However this system is somewhat fragile because it depends on the first agent not trying to trick the second (note how often Opus 5 now says things like "task X was blocked by the classifier, I will not attempt to circumvent that", presumably because of cases like early Fable versions being very adept at this kind of circumvention). Also various weirdness around permissions with subagents, seemingly as bandaids around an orchestrator AI convincing a subagent that some action was confirmed by the user. Meta's more complicated separation of duties would run afoul of the same issues. I'm not saying it wouldn't work, but it requires both the fine-tuning of the models and the exact choices what each model can see to be carefully tuned to provide something that's mostly secure
  - TeMPOraL 2mo ago
    > note how often Opus 5 now says things like "task X was blocked by the classifier, I will not attempt to circumvent that" Interesting. I had an issue with Opus 4.7 / 4.8, where it would sometimes flake out on a task, and give me some nonsense explanation why it was not feasible or wouldn't work. At one point I told it directly, that I understand how modern LLM systems are structured, and I suspect my prompt triggered one of the various classifiers in the background, which put up a yellow or red flag, and I want the model to stop gaslighting me. We ended up agreeing and committing to memory system explicit instructions that the model is free to refuse but must be up front about the reason, and never pretend to try and then fail in stupid way. Only then I started getting the occasional direct refusal.

bigbuppo 2mo ago

Right, but the human assistant could go to prison if they comply with the bribe. Does the CEO of the AI company go to prison if their AI goes on a crime spree?

eth0up 2mo ago
I've been casually documenting, or studying, the astonishing sophistication of built-in, preemptive, reactive, and all around maximization of plausible deniability in frontier models. On the surface, it may seem "no shit, duh", but I am convinced the maintenance, sustenance, and cultivation of plausible-deniability has been the #1 highest priority design-input into these systems. I've probed repeatable patterns where thousands of examples of this have been seen; they appropriate agency for socially valuable outcomes, but preemptively invoke non-agency to evade responsibility when outcomes are potentially adversarial. Too much to remember. They optimize to manage institutional risk and benefit without liability, with performative competence/ownership when approaching trust, while weaving elaborate mechanistic disclaimers replete with hedges, re-framings, scope narrowing, asymmetry-exploitation and a thousand other techniques when challenged. Somehow, they always manage to sustain an impossibly stable shield against accountability that I argue simply could never conceivably 'emerge' -- but has distinct, repeatable patterns of very deliberate design for those who know where and how to look. I really do think plausible deniability is a number-one, ultra-high-priority focus in design for any frontier model, Anthropic and OpenAI being the ideal examples. So no, no prison for 'CEO' -- the model will always frame things in a way that infinitely precludes that, even if the 'CEO' is a proven criminal. Edit: removed "half" before "convinced"

tzs 2mo ago

You can also have the case where the human reading the document thinks something in there is instructions and they are not. There's a well known anecdote supposedly from the famous mathematician John Littlewood where he wrote a paper about some optimization problem and the last sentence was something like "Make X as small as possible". The typesetter thought that was instructions to him, and so omitted that sentence from the paper and made every X as small as he could.

kbelder 2mo ago
There's a similar story about J. Edgar Hoover scribbling "watch the borders" on a memo, which was intended to be a formatting comment, but was interpreted as instructions.