greshake

5 ms·

1.
Unreal Computing(kai-greshake.de)
2 points by greshake 3y ago|0 comments
2.
ChatGPT compromised while browsing with plugins(twitter.com)
3 points by greshake 3y ago|0 comments
3.
by greshake 3y ago
(depending on whether the search engine context is connected to untrusted inputs or only your curated database ofc)
4.
by greshake 3y ago
You're missing the more important vector for prompt injection: Indirect injection through the "search engine context". It's not just a matter of blocking bad user questions to fend off reputational harms. See also my wor
5.
The Security Hole at the Heart of ChatGPT and Bing(wired.co.uk)
2 points by greshake 3y ago|0 comments
6.
by greshake 3y ago
Check out my blog where I show even more up-to-date techniques and the insane ways vulnerable applications are being deployed: https://kai-greshake.de/ Here I go through all of the unsafe products (including military LLMs):
7.
by greshake 3y ago
TLDR: With these vulnerabilities, we show the following is possible: - Remote control of chat LLMs - Persistent compromise across sessions - Spread injections to other LLMs - Compromising LLMs with tiny multi-stage payloads - Leaking/e
8.
by greshake 3y ago
Neither is possible right now.
9.
Compromising LLM-integrated applications with indirect prompt injection(arxiv.org)
43 points by greshake 3y ago|20 comments
10.
by greshake 3y ago
I just published a blog post showing that that is not what is happening. Companies are plugging LLMs into absolutely anything, including defense/threat intelligence/cybersecurity/legal etc. applications: https://ka
11.
by greshake 3y ago
I've written about this extensively. My latest article goes into the consequences. How about going from Prompt Injection to airstrike? https://kai-greshake.de/posts/in-escalating-order-of-stupidi...
12.
by greshake 3y ago
Also check out a blogpost on the same subject: https://kai-greshake.de/posts/llm-malware/
13.
Compromising Real LLM-Integrated Applications with Indirect Prompt Injection(arxiv.org)
3 points by greshake 3y ago|2 comments
14.
by greshake 3y ago
"We demonstrate our attacks' practical viability against both real-world systems, such as Bing's GPT-4 powered Chat and code-completion engines, and synthetic applications built on GPT-4. We show how processing retrieved prom
15.
by greshake 3y ago
Look into our repo (also linked there) we started out with only demonstrating that it works on GPT-3 APIs, now we also know it works on ChatGPT/3.5-turbo with ChatML and GPT-4, and even its most restricted form, Bing.
16.
by greshake 3y ago
> your finding that ChatML does not work seems to be mainly this Also the fact that ChatML has been broken into bits many, many times now- see again the prompt golfing. Also I'm taking OpenAi at their word because they have very str
17.
by greshake 3y ago
Segmenting different data sources is the main approach pursued by OpenAI afaik (ChatML for example). That has not worked so far, as you can see in this prompt golfing game: https://ggpt.43z.one/ The goal is to find the shor
18.
by greshake 3y ago
[...] demonstrate potentially brutal consequences of giving LLMs like ChatGPT interfaces to other applications. We propose newly enabled attack vectors and techniques and provide demonstrations of each in this repository: - Remote control
19.
Show HN: ChatGPT Plugins are a Security Nightmare(github.com)
4 points by greshake 3y ago|2 comments
20.
by greshake 3y ago
Soo.. Expect your personal GPT to be persistently compromised/hacked, remote-controlled and used to exfiltrate all your data. Security of LLMs is in a bad state right now.
21.
by greshake 3y ago
We don't seem to get a lot of traction unfortunately. Every time I posted our research to HN we were met by people dismissing the threat. It seems that it is one of these problems where anyone can come up with something that sounds l
22.
by greshake 3y ago
When an attacker is aware that such a check is executed it would be trivial to ensure that the compromised LLM passes it and behaves like usual. I believe this is similar to other "Supervisor" approaches that I do address in the a
23.
by greshake 3y ago
No, that doesn't solve it. If you run an LLM at home and give it access to APIs or your data it could still get compromised. The whole point is that it isn't the user who is doing the injection themselves.
24.
by greshake 4y ago
Yeah, same experience here- although I wonder if a cache miss has the side effect of the indexer scheduling retrieval for later? ;) By the way I've read some of your blog posts on the subject, and I very much agree with your sentiment
25.
by greshake 4y ago
Sounds interesting, I'll be sure to have a look!
26.
by greshake 4y ago
Thank you! I had a shift in perspective a few weeks ago that made all this fall into place. Unfortunately it seems hard to communicate the idea to people, and I think many people are very invested in LLM applications and are biased to think
27.
by greshake 4y ago
Sure but you've been here steadfast in your opinion that this is no big deal that is an easy fix away from being permanently resolved. It is not. It may be one of the hardest problems facing the deployment of these LLMs. "Sanitizi
28.
by greshake 4y ago
The typos are in the injections because we designed and implemented them in a single pass after reading the leaked initial prompt, and so far every single one was immediately successful. It just further illustrates how low the bar for such
29.
by greshake 4y ago
Pretty sure we address this issue in the paper/repository? Some of our demos rely on letting the LLM copy the injection into the final response, getting around the issue of things in subprompts not being visible later on, depending on
30.
by greshake 4y ago
This is not the same. Prepared statements eliminate SQL injections. "Maliciousness" of these inputs is well defined and can be decided by a computer. It would not be acceptable practice to "mitigate" SQL injections by bl

More ›