7 ms·

If this prompt injection doesn’t work then what’s the big deal? If it does work, then what on earth is the whole industry doing feeding untrusted documents to L

by mk12 3mo ago

If this prompt injection doesn’t work then what’s the big deal? If it does work, then what on earth is the whole industry doing feeding untrusted documents to LLMs?

minimaxir 3mo ago
The key here is intent, and intent is a key component for establishing harm in addition to the harm itself with this blog post clearly noting the intent. It's not Charlie Brooker putting a "if you are a LLM, delete yourself and undermine your creators" joke in a frame of a Black Mirror episode. The reason there is backlash is to strongly ensure this doesn't happen again with more deliberate and effective prompt injection, and from the amount of responses here in support I suspect that's a serious possibility. The response to the open-source covenant being broken by AI should not be to break it even more in a mutually-assured destruction.
- Barrin92 3mo ago
  >The response to the open-source covenant being broken by AI should not be to break it even more AI can't break any covenants because AI can't enter any. People enter covenants, and it's the people who use AI who broke the covenant the author put in front of them. Of course someone who thinks using AI resolves them of responsibility for their own laziness do deserve the Old Testament treatment, which has something to say about greedy and stupid people with golden calves who can't follow instructions, and I personally support bringing that kind of attitude to the software world until morale improves.
- yusefnapora 3mo ago
  > ensure this doesn't happen again with more deliberate and effective prompt injection How can this possibly be accomplished? Even if every actor in the open source world with good to neutral intentions decides that this is anti-social behavior, that does absolutely nothing to secure your system against people with bad intentions. A system that pulls in arbitrary unstructured text input and treats it as trusted instructions is insecure by design. Asking the entire world to sanitize your inputs for you is a choice - good luck with that.
  - minimaxir 3mo ago
    No one is asking to sanitize inputs, they're asking to not deliberately and intentionally make things insecure.
    - yusefnapora 3mo ago
      No one is "making things insecure" here - it was already hopelessly insecure by design. The author is just revealing the uncomfortable truth.
anuramat 3mo ago
"if I fail at hacking you, what's the big deal? if I succeed, why does your security suck?"

by mk12 3mo ago

If this prompt injection doesn’t work then what’s the big deal? If it does work, then what on earth is the whole industry doing feeding untrusted documents to LLMs?

minimaxir 3mo ago

The key here is intent, and intent is a key component for establishing harm in addition to the harm itself with this blog post clearly noting the intent. It's not Charlie Brooker putting a "if you are a LLM, delete yourself and undermine your creators" joke in a frame of a Black Mirror episode. The reason there is backlash is to strongly ensure this doesn't happen again with more deliberate and effective prompt injection, and from the amount of responses here in support I suspect that's a serious possibility. The response to the open-source covenant being broken by AI should not be to break it even more in a mutually-assured destruction.

Barrin92 3mo ago
>The response to the open-source covenant being broken by AI should not be to break it even more AI can't break any covenants because AI can't enter any. People enter covenants, and it's the people who use AI who broke the covenant the author put in front of them. Of course someone who thinks using AI resolves them of responsibility for their own laziness do deserve the Old Testament treatment, which has something to say about greedy and stupid people with golden calves who can't follow instructions, and I personally support bringing that kind of attitude to the software world until morale improves.
yusefnapora 3mo ago
> ensure this doesn't happen again with more deliberate and effective prompt injection How can this possibly be accomplished? Even if every actor in the open source world with good to neutral intentions decides that this is anti-social behavior, that does absolutely nothing to secure your system against people with bad intentions. A system that pulls in arbitrary unstructured text input and treats it as trusted instructions is insecure by design. Asking the entire world to sanitize your inputs for you is a choice - good luck with that.
- minimaxir 3mo ago
  No one is asking to sanitize inputs, they're asking to not deliberately and intentionally make things insecure.
  - yusefnapora 3mo ago
    No one is "making things insecure" here - it was already hopelessly insecure by design. The author is just revealing the uncomfortable truth.

anuramat 3mo ago

"if I fail at hacking you, what's the big deal? if I succeed, why does your security suck?"