6 ms·

The biggest risk to that security risk is its own name. Needs rebranding asap.

danShumway 3y ago
:) You're definitely not the first person to suggest that, and there is a decent argument to be made for rebranding. I'm not opposed to it. And I have seen a few at least individual efforts to use different wording, but unfortunately none of them seem to have caught on more broadly (yet), and I'm not sure if there's a clear community consensus yet among security professionals about what they'd prefer to use instead (people who are more embedded in that space than me are welcome to correct me if wrong on that). But I'm at least happy to jump to other terminology if that changes, I do think that calling it "prompt injection" confuses people. I think I remember there being some effort a while back to build a more extensive classification of LLM vulnerabilities that could be used for vulnerability reporting/triaging, but I don't know what the finished project ended up being or what the full details were.
- jstarfish 3y ago
  Just call it what it is-- social engineering (really, manipulation). "Injection" is a narrow and irrelevant definition. Natural language does not follow a bounded syntax, and injection of words is only one way to "break" the LLM. Buffer overflow works just as well-- smalltalk it to death, until the context outweighs the system prompt. Use lots of innuendo and ambiguous verbiage. After enough discussion of cork soakers and coke sackers you can get LLMs to alliterate about anything. There's nothing injected there, it's just a conversation that went a direction you didn't want to support. In meatspace, if you go to a bank and start up with elaborate stories about your in-laws until the teller forgets what you came in for, or confuse the shit out of her by prefacing everything you say with "today is opposite day," or flash a fake badge and say you're Detective Columbo and everybody needs to evacuate the building, you've successfully managed to get the teller to break protocol. Yet when we do it to LLMs, we give it the woo-woo euphemism "jailbreaking" as though all life descended from iPhones. When the only tool in your box is a computer, every problem is couched in software. It smells like we're trying to redefine manipulation, which does little to help anybody. These same abuses of perception have been employed by and against us for thousands of years already under the names of statecraft, spycraft and stagecraft.
  - simonw 3y ago
    I think you may be confusing jailbreaking and prompt injection. Jailbreaking is more akin to social engineering - it's when you try and convince the model to do something it's "not supposed" to do. Prompt injection is a related but different thing. It's when you take a prompt from a developer - "Translate the following from English to French:" - and then concatenate on a string of untrusted text from a user. That's why it's called "prompt injection" - it's analogous to SQL injection, which was caused by the same mistake, concatenating together trusted instructions with untrusted input.
ethanbond 3y ago
Seems directly analogous to SQL injection, no?
- simonw 3y ago
  Almost. That's why I suggested the name "prompt injection" - because both attacks involve concatenating together trusted and untrusted text. The problem is that SQL injection has an easy fix: you can use parameterized queries, or correctly escape the untrusted content. When I coined "prompt injection" I assumed the fix would look the same. 14 months later it's abundantly clear that implementing an equivalent of those fixes for LLMs is difficult to the point of maybe being impossible, at least against current transformer-based architectures. This means the name "prompt injection" may de-emphasize the scale of the threat!
  - ethanbond 3y ago
    That makes a ton of sense. Well, keen to hear what you (or The People) come up with as a more suitable alternative.
- scotty79 3y ago
  Same that any scandal is analogous to Watergate (hotel). It makes no sense but since it sounds cool now people will run with it forever.
  - ethanbond 3y ago
    Not really

:) You're definitely not the first person to suggest that, and there is a decent argument to be made for rebranding. I'm not opposed to it. And I have seen a few at least individual efforts to use different wording, but unfortunately none of them seem to have caught on more broadly (yet), and I'm not sure if there's a clear community consensus yet among security professionals about what they'd prefer to use instead (people who are more embedded in that space than me are welcome to correct me if wrong on that). But I'm at least happy to jump to other terminology if that changes, I do think that calling it "prompt injection" confuses people. I think I remember there being some effort a while back to build a more extensive classification of LLM vulnerabilities that could be used for vulnerability reporting/triaging, but I don't know what the finished project ended up being or what the full details were.

jstarfish 3y ago
Just call it what it is-- social engineering (really, manipulation). "Injection" is a narrow and irrelevant definition. Natural language does not follow a bounded syntax, and injection of words is only one way to "break" the LLM. Buffer overflow works just as well-- smalltalk it to death, until the context outweighs the system prompt. Use lots of innuendo and ambiguous verbiage. After enough discussion of cork soakers and coke sackers you can get LLMs to alliterate about anything. There's nothing injected there, it's just a conversation that went a direction you didn't want to support. In meatspace, if you go to a bank and start up with elaborate stories about your in-laws until the teller forgets what you came in for, or confuse the shit out of her by prefacing everything you say with "today is opposite day," or flash a fake badge and say you're Detective Columbo and everybody needs to evacuate the building, you've successfully managed to get the teller to break protocol. Yet when we do it to LLMs, we give it the woo-woo euphemism "jailbreaking" as though all life descended from iPhones. When the only tool in your box is a computer, every problem is couched in software. It smells like we're trying to redefine manipulation, which does little to help anybody. These same abuses of perception have been employed by and against us for thousands of years already under the names of statecraft, spycraft and stagecraft.
- simonw 3y ago
  I think you may be confusing jailbreaking and prompt injection. Jailbreaking is more akin to social engineering - it's when you try and convince the model to do something it's "not supposed" to do. Prompt injection is a related but different thing. It's when you take a prompt from a developer - "Translate the following from English to French:" - and then concatenate on a string of untrusted text from a user. That's why it's called "prompt injection" - it's analogous to SQL injection, which was caused by the same mistake, concatenating together trusted instructions with untrusted input.

ethanbond 3y ago

Seems directly analogous to SQL injection, no?

simonw 3y ago
Almost. That's why I suggested the name "prompt injection" - because both attacks involve concatenating together trusted and untrusted text. The problem is that SQL injection has an easy fix: you can use parameterized queries, or correctly escape the untrusted content. When I coined "prompt injection" I assumed the fix would look the same. 14 months later it's abundantly clear that implementing an equivalent of those fixes for LLMs is difficult to the point of maybe being impossible, at least against current transformer-based architectures. This means the name "prompt injection" may de-emphasize the scale of the threat!
- ethanbond 3y ago
  That makes a ton of sense. Well, keen to hear what you (or The People) come up with as a more suitable alternative.
scotty79 3y ago
Same that any scandal is analogous to Watergate (hotel). It makes no sense but since it sounds cool now people will run with it forever.
- ethanbond 3y ago
  Not really