6 ms·

> a file with malicious prompt on your hard drive How will you know if a file is free from malicious prompt or not? The applications seems to be able to downlo

> a file with malicious prompt on your hard drive How will you know if a file is free from malicious prompt or not? The applications seems to be able to download any file and analyze it. So from my perspective, I think it is easier this way than to execute other attack? Because these files may seem benign but can still run instructions from the prompts. Just think that the next pdf you are downloading from the web has has no malware but only malicious prompt. What will you do?

rnosov 4y ago
If a user can be tricked into downloading files and then running them and why not trick a user into downloading and running actual malware? With your pdf scenario it would be a possibility, but at the moment OpenAI davinci doesn't follow URLs. So, even you overtake the model with your malicious prompt what would you gain? The worst I can think of is that you can misdirect the summary which although amusing won't be that dangerous.
- greshake 4y ago
  If you had a PDF reader which allowed arbitrary code execution on opening a file, would you argue the same? You give arbitrary read/write to the LLM, right? So ransomware, causing network requests as side effects etc. could all be possible. Look at the paper to find more descriptions of what could go wrong: https://github.com/greshake/llm-security https://github.com/greshake/llm-security
  - rnosov 4y ago
    PDF reader hack will be indeed dangerous. I've looked at your link and they suggest that LLMs will fetch random files of the internet. At the moment, no LLM will do that. Network requests are currently off limits. In the paper they seem to perform attack mainly against some library called "langchain". This library might be indeed vulnerable but it's more of a problem of this particular library.
    - rnosov 4y ago
      Answering to myself as I can't edit my response any more. I've investigated a bit more and the attack as described by Greshake[1] seems to be much more realistic to me than I initially thought. [1] https://github.com/greshake/llm-security https://github.com/greshake/llm-security
      - greshake 4y ago
        Yea that's me. It seems to be very difficult right now to get people's attention to this and make them take it seriously. On a side note, your project is also currently putting unfiltered model output straight into osascript sooooo a lot of the fancy gymnastics needed to make stuff work in the paper with only search abilities isn't required in this case.
        rnosov 4y ago
        Just for the avoidance of doubt "AI Files" is not my project and I'm not affiliated with the author in any way. osascipt line does look a bit dodgy to me but perhaps is safe. But I can see how things might go downhill quickly with this approach...

rnosov 4y ago

If a user can be tricked into downloading files and then running them and why not trick a user into downloading and running actual malware? With your pdf scenario it would be a possibility, but at the moment OpenAI davinci doesn't follow URLs. So, even you overtake the model with your malicious prompt what would you gain? The worst I can think of is that you can misdirect the summary which although amusing won't be that dangerous.

greshake 4y ago
If you had a PDF reader which allowed arbitrary code execution on opening a file, would you argue the same? You give arbitrary read/write to the LLM, right? So ransomware, causing network requests as side effects etc. could all be possible. Look at the paper to find more descriptions of what could go wrong: https://github.com/greshake/llm-security https://github.com/greshake/llm-security
- rnosov 4y ago
  PDF reader hack will be indeed dangerous. I've looked at your link and they suggest that LLMs will fetch random files of the internet. At the moment, no LLM will do that. Network requests are currently off limits. In the paper they seem to perform attack mainly against some library called "langchain". This library might be indeed vulnerable but it's more of a problem of this particular library.
  - rnosov 4y ago
    Answering to myself as I can't edit my response any more. I've investigated a bit more and the attack as described by Greshake[1] seems to be much more realistic to me than I initially thought. [1] https://github.com/greshake/llm-security https://github.com/greshake/llm-security
    - greshake 4y ago
      Yea that's me. It seems to be very difficult right now to get people's attention to this and make them take it seriously. On a side note, your project is also currently putting unfiltered model output straight into osascript sooooo a lot of the fancy gymnastics needed to make stuff work in the paper with only search abilities isn't required in this case.
      - rnosov 4y ago
        Just for the avoidance of doubt "AI Files" is not my project and I'm not affiliated with the author in any way. osascipt line does look a bit dodgy to me but perhaps is safe. But I can see how things might go downhill quickly with this approach...