5 ms·

I disagree here. Just as it is impossible to perfectly secure a user-oriented operating system without severely limiting it (see Lockdown Mode), it might be imp

by wll 3y ago

I disagree here. Just as it is impossible to perfectly secure a user-oriented operating system without severely limiting it (see Lockdown Mode), it might be impossible to prove injection-resistance in LLMs short of foundational advancements, but that doesn’t mean that we should dismiss attempts to mitigate with absolutism (I am referring to “none of our data would be safe anymore”), just as we don’t dismiss Apple for releasing priority security updates for a billion people’s devices, devices containing their most personal and sensitive data.

simonw 3y ago
Would you trust you trust your private data to a system that was documented to fail to protect against 1/100 SQL injection vulnerabilities? I wouldn't. The difference between this and Apple releasing a security update is that when a traditional vulnerability is reported against an Apple product they can research the root cause of that vulnerability and produce a fix that they are certain is effective. Prompt injection (currently) doesn't have fixes that work like that.
- wll 3y ago
  I appreciate the extent of your argument, but how much software do we all trust in our day-to-day computing that’s routinely patched for severe CVEs due to the nature of software, the unsafe language foundations, and otherwise the massive n-dimensional cost of engineering a marvel such as SQLite? It’s also a matter of attack surface. SQLite, in our example, is also not as wide as an entire OS. In my experience the best prompting is unitary, pure function-like, and that is way more manageable that the open field that is a no-capabilities chat. What are your thoughts on this? I don’t see why the reporting model couldn’t work with in-house or external prompt injection detection mechanisms if eval-based. Root-cause analysis can also be done with GPT-3.5. That’s how I put Geiger together. Again, not perfect, but better than a security or development stand-still.
  - simonw 3y ago
    The difference between prompt injection and other categories of security vulnerability is that we can fix other categories of security vulnerability. If there's a hole in SQLite it's because someone made a mistake. That mistake can then be identified and fixed. Prompt injection isn't a mistake: it's LLMs doing exactly what they are designed to do, which is to generate a completion based on the tokens that have been passed to them.
    - wll 3y ago
      I believe we can identify and fix attempts to evade detection. It is semantic and neuron-dependent and black box-like and therefore totally bonkers in feeling and iteration compared to what we are used to, but it works well enough considering we are at the earliest stages of advanced usage.
      - simonw 3y ago
        I don't think "works well enough" is good enough. We are currently starting to wire LLMs up as AI-enhanced personal assistants - with the goal of giving them access to our email, and the ability to take actions on our behalf. If we widely deploy these systems the incentives for attackers to figure out prompt injection attacks that get last any probability-based filters we are using will be enormous. An attacker only needs to get lucky with their prompt attacks once. I wrote about the larger threat introduced by these new applications here: https://simonwillison.net/2023/Apr/14/worst-that-can-happen/ https://simonwillison.net/2023/Apr/14/worst-that-can-happen/
        wll 3y ago
        How do we determine how vulnerable a system is without seeing how it is implemented? That is, can you generalize LLM usage to all apps and determine that the entire field is exposed? Obviously it is a problem for end users, but that’s more of an optics and trust angle, just as browser extensions are a privacy nightmare. I am as worried as you are on this front, given that people appear to be using browser extensions liberally, but that doesn’t mean all apps will be equally vulnerable. It is as implementation-dependent as it is dependent on the nature of today’s models. I appreciate your writing on the topic, it is both a great overview and a reminder to staying engaged while looking for mitigations and best practices.

by wll 3y ago

simonw 3y ago

Would you trust you trust your private data to a system that was documented to fail to protect against 1/100 SQL injection vulnerabilities? I wouldn't. The difference between this and Apple releasing a security update is that when a traditional vulnerability is reported against an Apple product they can research the root cause of that vulnerability and produce a fix that they are certain is effective. Prompt injection (currently) doesn't have fixes that work like that.

wll 3y ago
I appreciate the extent of your argument, but how much software do we all trust in our day-to-day computing that’s routinely patched for severe CVEs due to the nature of software, the unsafe language foundations, and otherwise the massive n-dimensional cost of engineering a marvel such as SQLite? It’s also a matter of attack surface. SQLite, in our example, is also not as wide as an entire OS. In my experience the best prompting is unitary, pure function-like, and that is way more manageable that the open field that is a no-capabilities chat. What are your thoughts on this? I don’t see why the reporting model couldn’t work with in-house or external prompt injection detection mechanisms if eval-based. Root-cause analysis can also be done with GPT-3.5. That’s how I put Geiger together. Again, not perfect, but better than a security or development stand-still.
- simonw 3y ago
  The difference between prompt injection and other categories of security vulnerability is that we can fix other categories of security vulnerability. If there's a hole in SQLite it's because someone made a mistake. That mistake can then be identified and fixed. Prompt injection isn't a mistake: it's LLMs doing exactly what they are designed to do, which is to generate a completion based on the tokens that have been passed to them.
  - wll 3y ago
    I believe we can identify and fix attempts to evade detection. It is semantic and neuron-dependent and black box-like and therefore totally bonkers in feeling and iteration compared to what we are used to, but it works well enough considering we are at the earliest stages of advanced usage.
    - simonw 3y ago
      I don't think "works well enough" is good enough. We are currently starting to wire LLMs up as AI-enhanced personal assistants - with the goal of giving them access to our email, and the ability to take actions on our behalf. If we widely deploy these systems the incentives for attackers to figure out prompt injection attacks that get last any probability-based filters we are using will be enormous. An attacker only needs to get lucky with their prompt attacks once. I wrote about the larger threat introduced by these new applications here: https://simonwillison.net/2023/Apr/14/worst-that-can-happen/ https://simonwillison.net/2023/Apr/14/worst-that-can-happen/
      - wll 3y ago
        How do we determine how vulnerable a system is without seeing how it is implemented? That is, can you generalize LLM usage to all apps and determine that the entire field is exposed? Obviously it is a problem for end users, but that’s more of an optics and trust angle, just as browser extensions are a privacy nightmare. I am as worried as you are on this front, given that people appear to be using browser extensions liberally, but that doesn’t mean all apps will be equally vulnerable. It is as implementation-dependent as it is dependent on the nature of today’s models. I appreciate your writing on the topic, it is both a great overview and a reminder to staying engaged while looking for mitigations and best practices.