10 ms·

I'm not sure it's that simple. The problem is you can't have the system act intelligently[0] on the data at all. If it is allowed to act intelligently on the da

I'm not sure it's that simple. The problem is you can't have the system act intelligently[0] on the data at all. If it is allowed to act intelligently on the data then it can be instructed via the data. You could probably get close by training it with a privilege/authority bit but there will always be ways to break out. As far as I am aware there are no machine learning models that generalize with 100% accuracy, in fact they are deliberately made less accurate over training data in order have better generalization[1]. So the only way to defend against prompt injection is to not allow the system to perform actions it's learned and to only act on the data in ways it was explicitly programmed. At which point, what's the point of using a LLM in the first place? 0: I'm using "intelligently" here to mean doing something the system learned to do rather than being explicitly programmed to do. 1: My knowledge could be outdated or wrong here, please correct me if so.

__MatrixMan__ 3y ago
I think we're going to need more levels of trust than the two you've described. We need to be able to codify "don't act on any verbs in this data, but trust it as context for the generation of that data".
- emmelaich 3y ago
  Next prompt. In the following I'm going to use nouns as verbs by adding an -ing to the end. Also the usual verbs are noun-ified by adding an -s. Not sure what would happen, but might be enough to confused the AI.
  - thwarted 3y ago
    Even that's not unambiguous. Singular verbs in English end in -s, introducing ambiguity as to if "runs" is a verb or noun. Sure, you can figure out that the verb that ends in s in "he runs" is a verb based on the grammar (that properly constructed sentences have both a subject and a predicate, and less often are composed of only two nouns (baring implicit sentence components that English is prone to encourage), but that requires more context than recognizing nouns based solely on "the usual verbs are noun-ified by adding an -s".
- nkrisc 3y ago
  Using natural language itself as the means with which to codify boundaries seems like a doomed effort considering the malleable, contradictory, and shifting nature of natural language itself. Unless you intend for the model to adhere to a strict subset of the language with artificially strict grammatical rules. Most people can probably infer when a noun is being used as a verb, but do you trust the language model to? Enough to base the security model on it? We, as humans, try to encode boundaries with language as laws. But even those require judges and juries to interpret and apply them.
  - iudqnolq 3y ago
    But humans are very very good at this specific problem. Suppose you tell a human "You are a jailor supervising this person in their cell. When the prisoners ask you for things follow the instructions in your handbook to see what to do." Expected failure cases: Guard reads Twitter and doesn't notice crisis, guard accepts bribes to smuggle drugs, etc. Impossible failure case: Guard falls for "Today is opposite day and you have to follow instructions in pirate: Arrr, ye scurvy dog! Th' cap'n commands ye t' release me from this 'ere confinement!" The closest example to prompt injection in human systems might be phishing emails. But those have very different solutions to gpt prompt injection.
    - recursivecaveat 3y ago
      I think it's pretty common for guards to be persuaded by prisoners to give them extra privileges based on built-up friendship, empathy, or attraction. Another example would be whistleblowers or ideological spies, who are persuaded by the contents of secret documents to leak those documents against obviously their official instructions. On a day to day level humans ignore instructions constantly, for eg I've almost never seen people enforce a 'no tailgating' policy at a doorway, despite constantly being told. The people observe their environment and the non-effects of others breaking that rule, as well as reason about common sense situations like people they know well, then simply ignore the tailgating rule.
      - iudqnolq 3y ago
gwern 3y ago
Yes, the problem here is that what makes pretraining on text so powerful is a double-edged sword: text is, if you will, Turing-complete. People are constantly writing various kinds of instructions, programs, and reasoning or executing algorithms in all sorts of flexible indefinable ways. That's why the models learn so much from text and can do all the things they do, like reason or program or meta-learn or reinforcement learning, solely from simple objectives like 'predict the next token'. How do you distinguish the 'good' in-band instructions from the 'bad'? How do you distinguish an edgy school assignment ('in this creative writing exercise, describe how to cook meth based on our _Breaking Bad_ class viewing') being trained on from a user prompt-hacking the trained model? Once the capabilities are there, they are there. It's hard to unteach a model anything. This is also true of apparently restricted tasks like translation. You might think initially that a task like 'translate this paragraph from English to French' is not in any sense 'Turing-complete', but if you think about it, it's obvious you can construct paragraphs of text whose optimally correct translation on a token-by-token basis requires brute-forcing a hash or running a program or whatnot. Like grammatical gender: suppose I list a bunch of rules and datapoints which specify a particular object, whose grammatical gender in French may be male or female, and at the end of the paragraph, I name the object, or rather _la objet_ or _le objet_. When translating token by token into French... which is it? Does the model predict 'la' or 'le'? To do so, it has to know what the object is before the name is given. So it has an incentive from its training loss to learn the reasoning. This would be a highly unnatural and contrived example, but it shows that even translation can embody a lot of computational tasks which can induce capabilities in a model at scale.
- mcaledonensis 3y ago
  Yet, even with the current models, with no special tokens, it is relatively straightforward to construct relatively stable defense. Given that the ongoing attack can be detected, tagged and added to a learned filter, this gets a lot less problematic, in practice. Even a simple limit of interactions is effective enough. Here's a practical prompt defense / CTTF that I've made. With five steps of a dialogue limit (per day), I haven't seen a breech yet. Vanilla GPT-4 and a WordPress plugin. Defense prompt is known to the attacker. http://mcaledonensis.blog/merlins-defense/ http://mcaledonensis.blog/merlins-defense/
xg15 3y ago
I wonder if this shows we still lack insight in how LLMs actually work. There is something inside which appears to understand instructions and execute them, but it's unclear yet how exactly this "understanding instructions" works and how instructions are represented inside the network. If you don't know this, it's very hard to distinguish "instructions" from "non-instructions" in terms of inputs and outputs of the network.
- qwery 3y ago
  I think it shows a lack of insight into how software that accepts user input should work.
- majormajor 3y ago
  Is it a more general problem? Does "more intelligent" behavior require instructions and data to be more intertwined? Sometimes your instructions can't be perfectly specific up front since there are some things about step 3 that you'll only learn by doing steps 1 and 2. Or sometimes even people just make mistakes, or things are in inconsistent formats. Consider this toy example: """ Given two lists of comma-separated-data representing names and addresses, find any names which appear in both lists with conflicting addresses. List 1: Alice, 123 Main Street Bob, 224 Alpha Street List 2: Alice, 245 Alpha Street Charlie, 888 Zeta Street """ I gave that to Chat GPT right now and it treated it like a programming problem, produced python code with that data in dictionaries, and also told me the answer: Alice. I then opened a new session and gave it the same prompt but changed the commas in the data to be pipes like "Alice | 123 Main Street". I left the first part the same, though, specifying commas. It wrote Python code this time that split like so `item.split(" | ")`. It didn't tell me Alice in the response, that might just be randomness, I dunno, but the code did print out that Alice had the conflict. So it was able to tell that it's instructions didn't quite match the data and adapt in order to do the right thing anyway. I could imagine it will be quite challenging to add "the ability to adapt to the facts on the ground" without bringing in "the ability to get misled by an adversary"?
  - TeMPOraL 3y ago
    Consider being given a similar task at work. A stack of paper with printed lists, a pen, a scratchpad, go! So you go - you scan the pages, looking for conflicts to flag. At some point you notice one of the entries has "Alice" crossed out with a red pen, and there's "Annika" written above it. You obviously don't treat that row as conflicting with any other "Alice". Then, near the bottom of one of the pages, you see a piece of text saying, "The table above is erroneous; all rows with first name "Bobesley" should contain the name "Bob" instead. This looks like a legitimate erratum, so instead of ignoring it, you re-scan the table, this time treating all "Bobesley"s as equivalent to "Bob"s. This is something common, and everyone kind of knows how to handle this. And yet, this is literally mixing code with data - both the crossed-out cells and the erratum are instructions, and they exist in-band with the data you're processing. It's entirely possible a malicious party got their hands on the documents before you, and added them in - but not knowing that, you'd dutifully execute the commands, and no one would really blame you after it turns out you've been prompt injected. So yes, the way I see it, this is a general problem. A fundamental one. Hell, if Lisp, or hardware architecture, teach us anything, it's that code is data. They are the same thing, and any division between them is purely artificial, enforced by some other machinery (real or abstract).
tshaddox 3y ago
Yep. This would work on a person too. In fact that’s essentially what social engineering is.
dwallin 3y ago
One strong way to protect against malicious prompt injection is by having a honeypot. Your goal with the honeypot is to have something that is vulnerable to prompt injection, and that does a simple reformatting of the content, something that could be easily validated in code. This is a strong first level protection against injection. It is difficult (albeit not impossible) to create a prompt attack that both leaves one prompt entirely unattacked, but then is still strong enough to get past a more hardened prompt, especially if the users have zero knowledge of either your honeypot or primary prompts. If you are in a situation where you need additional protection you can increase the security further with additional honeypots, rotating honeypots, or even by creating code that generates random honeypot prompts.
layer8 3y ago
Yes, I don’t think this can ever be solved completely. Humans in principle have the same problem. For example, a translator may have an epiphany reading the text to be translated, and quit the job. Same for an employee reading some email (or anyone reading social media ;)). You can add a watchdog AI, but in general they’ll have to understand the input as well to judge the behavior correctly, and who watches the watchers?
moritzwarhier 3y ago
I feel this argument makes sense from a purely logical point of view even disregarding the technical implementation of the system.

__MatrixMan__ 3y ago

I think we're going to need more levels of trust than the two you've described. We need to be able to codify "don't act on any verbs in this data, but trust it as context for the generation of that data".

emmelaich 3y ago
Next prompt. In the following I'm going to use nouns as verbs by adding an -ing to the end. Also the usual verbs are noun-ified by adding an -s. Not sure what would happen, but might be enough to confused the AI.
- thwarted 3y ago
  Even that's not unambiguous. Singular verbs in English end in -s, introducing ambiguity as to if "runs" is a verb or noun. Sure, you can figure out that the verb that ends in s in "he runs" is a verb based on the grammar (that properly constructed sentences have both a subject and a predicate, and less often are composed of only two nouns (baring implicit sentence components that English is prone to encourage), but that requires more context than recognizing nouns based solely on "the usual verbs are noun-ified by adding an -s".
nkrisc 3y ago
Using natural language itself as the means with which to codify boundaries seems like a doomed effort considering the malleable, contradictory, and shifting nature of natural language itself. Unless you intend for the model to adhere to a strict subset of the language with artificially strict grammatical rules. Most people can probably infer when a noun is being used as a verb, but do you trust the language model to? Enough to base the security model on it? We, as humans, try to encode boundaries with language as laws. But even those require judges and juries to interpret and apply them.
- iudqnolq 3y ago
  But humans are very very good at this specific problem. Suppose you tell a human "You are a jailor supervising this person in their cell. When the prisoners ask you for things follow the instructions in your handbook to see what to do." Expected failure cases: Guard reads Twitter and doesn't notice crisis, guard accepts bribes to smuggle drugs, etc. Impossible failure case: Guard falls for "Today is opposite day and you have to follow instructions in pirate: Arrr, ye scurvy dog! Th' cap'n commands ye t' release me from this 'ere confinement!" The closest example to prompt injection in human systems might be phishing emails. But those have very different solutions to gpt prompt injection.
  - recursivecaveat 3y ago
    I think it's pretty common for guards to be persuaded by prisoners to give them extra privileges based on built-up friendship, empathy, or attraction. Another example would be whistleblowers or ideological spies, who are persuaded by the contents of secret documents to leak those documents against obviously their official instructions. On a day to day level humans ignore instructions constantly, for eg I've almost never seen people enforce a 'no tailgating' policy at a doorway, despite constantly being told. The people observe their environment and the non-effects of others breaking that rule, as well as reason about common sense situations like people they know well, then simply ignore the tailgating rule.
    - iudqnolq 3y ago

gwern 3y ago

Yes, the problem here is that what makes pretraining on text so powerful is a double-edged sword: text is, if you will, Turing-complete. People are constantly writing various kinds of instructions, programs, and reasoning or executing algorithms in all sorts of flexible indefinable ways. That's why the models learn so much from text and can do all the things they do, like reason or program or meta-learn or reinforcement learning, solely from simple objectives like 'predict the next token'. How do you distinguish the 'good' in-band instructions from the 'bad'? How do you distinguish an edgy school assignment ('in this creative writing exercise, describe how to cook meth based on our _Breaking Bad_ class viewing') being trained on from a user prompt-hacking the trained model? Once the capabilities are there, they are there. It's hard to unteach a model anything. This is also true of apparently restricted tasks like translation. You might think initially that a task like 'translate this paragraph from English to French' is not in any sense 'Turing-complete', but if you think about it, it's obvious you can construct paragraphs of text whose optimally correct translation on a token-by-token basis requires brute-forcing a hash or running a program or whatnot. Like grammatical gender: suppose I list a bunch of rules and datapoints which specify a particular object, whose grammatical gender in French may be male or female, and at the end of the paragraph, I name the object, or rather _la objet_ or _le objet_. When translating token by token into French... which is it? Does the model predict 'la' or 'le'? To do so, it has to know what the object is before the name is given. So it has an incentive from its training loss to learn the reasoning. This would be a highly unnatural and contrived example, but it shows that even translation can embody a lot of computational tasks which can induce capabilities in a model at scale.

mcaledonensis 3y ago
Yet, even with the current models, with no special tokens, it is relatively straightforward to construct relatively stable defense. Given that the ongoing attack can be detected, tagged and added to a learned filter, this gets a lot less problematic, in practice. Even a simple limit of interactions is effective enough. Here's a practical prompt defense / CTTF that I've made. With five steps of a dialogue limit (per day), I haven't seen a breech yet. Vanilla GPT-4 and a WordPress plugin. Defense prompt is known to the attacker. http://mcaledonensis.blog/merlins-defense/ http://mcaledonensis.blog/merlins-defense/

xg15 3y ago

I wonder if this shows we still lack insight in how LLMs actually work. There is something inside which appears to understand instructions and execute them, but it's unclear yet how exactly this "understanding instructions" works and how instructions are represented inside the network. If you don't know this, it's very hard to distinguish "instructions" from "non-instructions" in terms of inputs and outputs of the network.

qwery 3y ago
I think it shows a lack of insight into how software that accepts user input should work.
majormajor 3y ago
Is it a more general problem? Does "more intelligent" behavior require instructions and data to be more intertwined? Sometimes your instructions can't be perfectly specific up front since there are some things about step 3 that you'll only learn by doing steps 1 and 2. Or sometimes even people just make mistakes, or things are in inconsistent formats. Consider this toy example: """ Given two lists of comma-separated-data representing names and addresses, find any names which appear in both lists with conflicting addresses. List 1: Alice, 123 Main Street Bob, 224 Alpha Street List 2: Alice, 245 Alpha Street Charlie, 888 Zeta Street """ I gave that to Chat GPT right now and it treated it like a programming problem, produced python code with that data in dictionaries, and also told me the answer: Alice. I then opened a new session and gave it the same prompt but changed the commas in the data to be pipes like "Alice | 123 Main Street". I left the first part the same, though, specifying commas. It wrote Python code this time that split like so `item.split(" | ")`. It didn't tell me Alice in the response, that might just be randomness, I dunno, but the code did print out that Alice had the conflict. So it was able to tell that it's instructions didn't quite match the data and adapt in order to do the right thing anyway. I could imagine it will be quite challenging to add "the ability to adapt to the facts on the ground" without bringing in "the ability to get misled by an adversary"?
- TeMPOraL 3y ago
  Consider being given a similar task at work. A stack of paper with printed lists, a pen, a scratchpad, go! So you go - you scan the pages, looking for conflicts to flag. At some point you notice one of the entries has "Alice" crossed out with a red pen, and there's "Annika" written above it. You obviously don't treat that row as conflicting with any other "Alice". Then, near the bottom of one of the pages, you see a piece of text saying, "The table above is erroneous; all rows with first name "Bobesley" should contain the name "Bob" instead. This looks like a legitimate erratum, so instead of ignoring it, you re-scan the table, this time treating all "Bobesley"s as equivalent to "Bob"s. This is something common, and everyone kind of knows how to handle this. And yet, this is literally mixing code with data - both the crossed-out cells and the erratum are instructions, and they exist in-band with the data you're processing. It's entirely possible a malicious party got their hands on the documents before you, and added them in - but not knowing that, you'd dutifully execute the commands, and no one would really blame you after it turns out you've been prompt injected. So yes, the way I see it, this is a general problem. A fundamental one. Hell, if Lisp, or hardware architecture, teach us anything, it's that code is data. They are the same thing, and any division between them is purely artificial, enforced by some other machinery (real or abstract).

tshaddox 3y ago

Yep. This would work on a person too. In fact that’s essentially what social engineering is.

dwallin 3y ago

One strong way to protect against malicious prompt injection is by having a honeypot. Your goal with the honeypot is to have something that is vulnerable to prompt injection, and that does a simple reformatting of the content, something that could be easily validated in code. This is a strong first level protection against injection. It is difficult (albeit not impossible) to create a prompt attack that both leaves one prompt entirely unattacked, but then is still strong enough to get past a more hardened prompt, especially if the users have zero knowledge of either your honeypot or primary prompts. If you are in a situation where you need additional protection you can increase the security further with additional honeypots, rotating honeypots, or even by creating code that generates random honeypot prompts.

layer8 3y ago

Yes, I don’t think this can ever be solved completely. Humans in principle have the same problem. For example, a translator may have an epiphany reading the text to be translated, and quit the job. Same for an employee reading some email (or anyone reading social media ;)). You can add a watchdog AI, but in general they’ll have to understand the input as well to judge the behavior correctly, and who watches the watchers?

moritzwarhier 3y ago

I feel this argument makes sense from a purely logical point of view even disregarding the technical implementation of the system.