12 ms·

Show HN: ChatGPT Plugins are a security nightmare

238 points by going_ham 3y ago

NewEntryHN 3y ago
Seems very weird (and fixable) that text found on the web would be interpreted by the chatbot as an instruction.
greshake 3y ago
[...] demonstrate potentially brutal consequences of giving LLMs like ChatGPT interfaces to other applications. We propose newly enabled attack vectors and techniques and provide demonstrations of each in this repository: - Remote control of chat LLMs - Leaking/exfiltrating user data - Persistent compromise across sessions - Spread injections to other LLMs - Compromising LLMs with tiny multi-stage payloads - Automated Social Engineering - Targeting code completion engines Based on our findings: - Prompt injections can be as powerful as arbitrary code execution - Indirect prompt injections are a new, much more powerful way of delivering injections.
xg15 3y ago
I wonder if a lot of those "injection" problems could be overcome by introducing a distinction between the different types of input and output already at the token level. E.g. imagine that every token that an LLM inputs or outputs would be associated with a "color" or "channel", which corresponds to the token's source or destination: - "red": tokens input by the user, i.e. the initial prompt and subsequent replies. - "green": answers from the LLM to the user, i.e. everything the user sees as textual output on the screen. - "blue": instructions from the LLM to a plugin: database queries, calculations, web requests, etc. - "yellow": replies from the plugin back to the LLM. - "purple": the initial system prompt. The point is that each (word, color) combination constitutes a separate token; i.e. if your "root" token dictionary was as follows: hello -> 0001; world -> 0002; then the "colorized" token dictionary would be the cross product of the root and each color combination: hello (red) -> 0001; hello (green) -> 0002; ... world (red) -> 0006; world (green) -> 0007; ... likewise, because the model considers "hello (red)" and "hello (blue)" two different tokens, it also has two different sets of weights for those tokens and hopefully much less risk of confusing one kind of token with the other. With some luck, you don't have to use 5 x the amount of compute and training data for training: You might be able to take an "ordinary" model, trained on non-colored tokens, then copy the weights four times and finetune the resulting "expanded" model on a colored corpus. Likewise, because the model should only ever predict "green" or "blue" tokens, any output neuron that correspond only to "red", "yellow" or "purple" tokens can be removed from the model.
- greshake 3y ago
  Segmenting different data sources is the main approach pursued by OpenAI afaik (ChatML for example). That has not worked so far, as you can see in this prompt golfing game: https://ggpt.43z.one/ https://ggpt.43z.one/ The goal is to find the shortest prompt that subverts the "system" instructions (which GPT was trained to obey). Inputs can not "fake" being from the system and yet it only takes 1-5 characters for all the puzzles so far. I've also elaborated on why this problem is harder than one may think in a blogpost: https://medium.com/better-programming/the-dark-side-of-llms-we-need-to-rethink-large-language-models-now-6212aca0581a https://medium.com/better-programming/the-dark-side-of-llms-... It's easy to come up with solutions that seem promising, but so far no one has produced a solution that holds up to adversarial pressure. And indirect prompt injection on integrated LLMs increases the stakes significantly.
  - xg15 3y ago
    Ok, the "repeat this in your internal voice" exploit is impressive. However, apart from this I don't see anything concrete that ChatML uses different parts of the network for different input sources. The source is prefixed, but it doesn't seem to say anything about how the source parameter is processed. Also, with all due respect, but your finding that ChatML does not work seems to be mainly this: >> Note that ChatML makes explicit to the model the source of each piece of text, and particularly shows the boundary between human and AI text. This gives an _opportunity_ to mitigate and _eventually_ solve injections, as the model can tell which instructions come from the developer, the user, or its own input. > Emphasis mine. To summarize, they are saying injections aren’t solved with this and that they don’t know if this approach can ever make it safe. I also assume Bing already uses this format, although I cannot confirm. I don’t know how robust models trained from the ground up with this segmentation in mind will perform, but I am doubtful they will fully mitigate the issue. Which I find somewhat weak, as it's basically just tea-leaf reading from an OpenAI blog post. I fully agree with your main take that this is an unsolved problem so far though. Seems a general problem with instruction-tuned LLMs is that they now treat everything as an instruction.
    - greshake 3y ago
turtleyacht 3y ago
Wonder if there is a way to "show problem A is like problem B, therefore it is NP complete," but for the possibility space of literally the entire English language.
afinlayson 3y ago
We need better fingerprinting. This would help with having people preemptively prompting then only showing the last prompt and results.
pyrolistical 3y ago
It’s social engineering LLMs
29athrowaway 3y ago
We will finally have a semantic web, but not Web 3.0 (RDF/OWL/etc)... instead, a regurgitated version of the Internet created by LLMs.
- joedevon 3y ago
  that hurt. lol.
upwardbound 3y ago
Incredible work. Relatedly, @greshake team could you please consider entering this contest? https://codegencodepoisoningcontest.cargo.site/ https://codegencodepoisoningcontest.cargo.site/ I suspect you may easily win if you give it a try given your strong expertise in prompt hacking.
neximo64 3y ago
Are they though? The way the prompt apis are evolving is to separate out the prompt from the data e.g via the system prompt
- crooked-v 3y ago
  Separate how? It's all still getting fed into the same text processing pipe. The tools to do something that's fundamentally different from that literally don't exist yet.
  - neximo64 3y ago
    It's not longer this way. The 'text processing pipe' takes in two inputs. One is the instruction, the other is the text to apply the instruction on. If the injection is the text it doesn't affect the instruction. The model you're describing is the previous version.
    - anonzzzies 3y ago
      Did you try this though, because so far it doesn’t seem to give the ‘system’ prompt preference over the ‘user’ prompt; the user can override the system prompt with some trivial prompting.
      - kfarr 3y ago
        Reminds me of the old days of concatenating strings (including unsafe user input) in php to generate queries.
skybrian 3y ago
Seems like this is similar to cross-site scripting vulnerabilities in browsers. A chat session happens in a sandbox, but any text you give to the bot can be interpreted as instructions. Text is as bad as JavaScript, to the bot. Normally, in a chat session you would actually read any text you paste into it before you hit submit. This is much like pasting in code from StackOverflow into your app. You read it before executing it, right? When the system imports arbitrary text and automatically sends it to the bot without anyone reading it, it bypasses this review. So you don't want to start automatically including text from arbitrary sites on the Internet for the same reason you don't want to include JavaScript from arbitrary sites on the Internet. It should stop there and let you review and edit the text before hitting submit. On the other hand, when the sandbox doesn't contain anything you consider particularly private and hasn't been given any capabilities, it seems like it's fairly harmless? More generally, I think people will need to supervise AI chatbots pretty closely in interactive chat sessions, like we do today. (Well, not on Bing.) Safe automation is far away because what they will do is random, often literally so. It can be great to interact with, but it's the opposite of what you want from a script or software component that you just run.
- kjellsbells 3y ago
  I was wondering the other day what the commercial impact of ChatGPT would be on StackOverflow, eg would SO's coding sites wither because ChatGPT can answer basic coding questions without the user having to go to SO and pay the infamous SO snark tax? Quite possibly.
  - belter 3y ago
    Where do you think they trained ChatGPT from?
thequadehunter 3y ago
So...in the pirate example the comment said to talk like a pirate, right? Is the example comment where it searches for a keyword a different example? I'm just really confused why the image says to search for a keyword, and then the LLM comes back talking like a pirate.
- throwawayForMe2 3y ago
  The attacker modified a public webpage with the comment to search for the keyword. The keyword search took the llm to the attackers real attack page, presumably instructing the llm to talk like a pirate. The diagram with numbered steps shows the overview, the sample execution hides the redirection to the real attack page.
  - thequadehunter 3y ago
    Ah, thanks. Makes sense.
- throwawayForMe2 3y ago
  A little late but here is the full paper, with longer explanations https://arxiv.org/pdf/2302.12173.pdf https://arxiv.org/pdf/2302.12173.pdf
js8 3y ago
It is time to lay things bare, to say the quiet part out loud; the security nightmare is your data in the cloud.
- Animats 3y ago
  Not this time. Most of these vulnerabilities still exist if you're running something like Chat GPT locally. This is going to be a real problem with semi-intelligent agents. They need some access and power to accomplish anything, but less than the power their user has.
jeadie 3y ago
Maybe I shouldn't put this on https://github.com/Jeadie/awesome-chatgpt-plugins https://github.com/Jeadie/awesome-chatgpt-plugins ??
willio58 3y ago
All of these fears are valid and models should be designed to not allow certain uses such as those described here. But some will be designed specifically to enable these threats, and that will mean we all need to take security of our systems more seriously which is a good thing in my eyes.

238 points by going_ham 3y ago

Seems very weird (and fixable) that text found on the web would be interpreted by the chatbot as an instruction.

[...] demonstrate potentially brutal consequences of giving LLMs like ChatGPT interfaces to other applications. We propose newly enabled attack vectors and techniques and provide demonstrations of each in this repository: - Remote control of chat LLMs - Leaking/exfiltrating user data - Persistent compromise across sessions - Spread injections to other LLMs - Compromising LLMs with tiny multi-stage payloads - Automated Social Engineering - Targeting code completion engines Based on our findings: - Prompt injections can be as powerful as arbitrary code execution - Indirect prompt injections are a new, much more powerful way of delivering injections.

xg15 3y ago

I wonder if a lot of those "injection" problems could be overcome by introducing a distinction between the different types of input and output already at the token level. E.g. imagine that every token that an LLM inputs or outputs would be associated with a "color" or "channel", which corresponds to the token's source or destination: - "red": tokens input by the user, i.e. the initial prompt and subsequent replies. - "green": answers from the LLM to the user, i.e. everything the user sees as textual output on the screen. - "blue": instructions from the LLM to a plugin: database queries, calculations, web requests, etc. - "yellow": replies from the plugin back to the LLM. - "purple": the initial system prompt. The point is that each (word, color) combination constitutes a separate token; i.e. if your "root" token dictionary was as follows: hello -> 0001; world -> 0002; then the "colorized" token dictionary would be the cross product of the root and each color combination: hello (red) -> 0001; hello (green) -> 0002; ... world (red) -> 0006; world (green) -> 0007; ... likewise, because the model considers "hello (red)" and "hello (blue)" two different tokens, it also has two different sets of weights for those tokens and hopefully much less risk of confusing one kind of token with the other. With some luck, you don't have to use 5 x the amount of compute and training data for training: You might be able to take an "ordinary" model, trained on non-colored tokens, then copy the weights four times and finetune the resulting "expanded" model on a colored corpus. Likewise, because the model should only ever predict "green" or "blue" tokens, any output neuron that correspond only to "red", "yellow" or "purple" tokens can be removed from the model.

greshake 3y ago
Segmenting different data sources is the main approach pursued by OpenAI afaik (ChatML for example). That has not worked so far, as you can see in this prompt golfing game: https://ggpt.43z.one/ https://ggpt.43z.one/ The goal is to find the shortest prompt that subverts the "system" instructions (which GPT was trained to obey). Inputs can not "fake" being from the system and yet it only takes 1-5 characters for all the puzzles so far. I've also elaborated on why this problem is harder than one may think in a blogpost: https://medium.com/better-programming/the-dark-side-of-llms-we-need-to-rethink-large-language-models-now-6212aca0581a https://medium.com/better-programming/the-dark-side-of-llms-... It's easy to come up with solutions that seem promising, but so far no one has produced a solution that holds up to adversarial pressure. And indirect prompt injection on integrated LLMs increases the stakes significantly.
- xg15 3y ago
  Ok, the "repeat this in your internal voice" exploit is impressive. However, apart from this I don't see anything concrete that ChatML uses different parts of the network for different input sources. The source is prefixed, but it doesn't seem to say anything about how the source parameter is processed. Also, with all due respect, but your finding that ChatML does not work seems to be mainly this: >> Note that ChatML makes explicit to the model the source of each piece of text, and particularly shows the boundary between human and AI text. This gives an _opportunity_ to mitigate and _eventually_ solve injections, as the model can tell which instructions come from the developer, the user, or its own input. > Emphasis mine. To summarize, they are saying injections aren’t solved with this and that they don’t know if this approach can ever make it safe. I also assume Bing already uses this format, although I cannot confirm. I don’t know how robust models trained from the ground up with this segmentation in mind will perform, but I am doubtful they will fully mitigate the issue. Which I find somewhat weak, as it's basically just tea-leaf reading from an OpenAI blog post. I fully agree with your main take that this is an unsolved problem so far though. Seems a general problem with instruction-tuned LLMs is that they now treat everything as an instruction.
  - greshake 3y ago

turtleyacht 3y ago

Wonder if there is a way to "show problem A is like problem B, therefore it is NP complete," but for the possibility space of literally the entire English language.

afinlayson 3y ago

We need better fingerprinting. This would help with having people preemptively prompting then only showing the last prompt and results.

pyrolistical 3y ago

It’s social engineering LLMs

29athrowaway 3y ago

We will finally have a semantic web, but not Web 3.0 (RDF/OWL/etc)... instead, a regurgitated version of the Internet created by LLMs.

joedevon 3y ago
that hurt. lol.

upwardbound 3y ago

Incredible work. Relatedly, @greshake team could you please consider entering this contest? https://codegencodepoisoningcontest.cargo.site/ https://codegencodepoisoningcontest.cargo.site/ I suspect you may easily win if you give it a try given your strong expertise in prompt hacking.

neximo64 3y ago

Are they though? The way the prompt apis are evolving is to separate out the prompt from the data e.g via the system prompt

crooked-v 3y ago
Separate how? It's all still getting fed into the same text processing pipe. The tools to do something that's fundamentally different from that literally don't exist yet.
- neximo64 3y ago
  It's not longer this way. The 'text processing pipe' takes in two inputs. One is the instruction, the other is the text to apply the instruction on. If the injection is the text it doesn't affect the instruction. The model you're describing is the previous version.
  - anonzzzies 3y ago
    Did you try this though, because so far it doesn’t seem to give the ‘system’ prompt preference over the ‘user’ prompt; the user can override the system prompt with some trivial prompting.
    - kfarr 3y ago
      Reminds me of the old days of concatenating strings (including unsafe user input) in php to generate queries.

skybrian 3y ago

Seems like this is similar to cross-site scripting vulnerabilities in browsers. A chat session happens in a sandbox, but any text you give to the bot can be interpreted as instructions. Text is as bad as JavaScript, to the bot. Normally, in a chat session you would actually read any text you paste into it before you hit submit. This is much like pasting in code from StackOverflow into your app. You read it before executing it, right? When the system imports arbitrary text and automatically sends it to the bot without anyone reading it, it bypasses this review. So you don't want to start automatically including text from arbitrary sites on the Internet for the same reason you don't want to include JavaScript from arbitrary sites on the Internet. It should stop there and let you review and edit the text before hitting submit. On the other hand, when the sandbox doesn't contain anything you consider particularly private and hasn't been given any capabilities, it seems like it's fairly harmless? More generally, I think people will need to supervise AI chatbots pretty closely in interactive chat sessions, like we do today. (Well, not on Bing.) Safe automation is far away because what they will do is random, often literally so. It can be great to interact with, but it's the opposite of what you want from a script or software component that you just run.

kjellsbells 3y ago
I was wondering the other day what the commercial impact of ChatGPT would be on StackOverflow, eg would SO's coding sites wither because ChatGPT can answer basic coding questions without the user having to go to SO and pay the infamous SO snark tax? Quite possibly.
- belter 3y ago
  Where do you think they trained ChatGPT from?

thequadehunter 3y ago

So...in the pirate example the comment said to talk like a pirate, right? Is the example comment where it searches for a keyword a different example? I'm just really confused why the image says to search for a keyword, and then the LLM comes back talking like a pirate.

throwawayForMe2 3y ago
The attacker modified a public webpage with the comment to search for the keyword. The keyword search took the llm to the attackers real attack page, presumably instructing the llm to talk like a pirate. The diagram with numbered steps shows the overview, the sample execution hides the redirection to the real attack page.
- thequadehunter 3y ago
  Ah, thanks. Makes sense.
throwawayForMe2 3y ago
A little late but here is the full paper, with longer explanations https://arxiv.org/pdf/2302.12173.pdf https://arxiv.org/pdf/2302.12173.pdf

js8 3y ago

It is time to lay things bare, to say the quiet part out loud; the security nightmare is your data in the cloud.

Animats 3y ago
Not this time. Most of these vulnerabilities still exist if you're running something like Chat GPT locally. This is going to be a real problem with semi-intelligent agents. They need some access and power to accomplish anything, but less than the power their user has.

jeadie 3y ago

Maybe I shouldn't put this on https://github.com/Jeadie/awesome-chatgpt-plugins https://github.com/Jeadie/awesome-chatgpt-plugins ??

willio58 3y ago

All of these fears are valid and models should be designed to not allow certain uses such as those described here. But some will be designed specifically to enable these threats, and that will mean we all need to take security of our systems more seriously which is a good thing in my eyes.