6 ms·
The principal security problem of LLMs is that there is no architectural boundary between data and control paths. But this combination of data and control into
by fzeindl 5mo ago
The principal security problem of LLMs is that there is no architectural boundary between data and control paths.
But this combination of data and control into a single, flexible data stream is also the defining strength of a LLM, so it can’t be taken away without also taking away the benefits.
- mt_ 5mo agoExactly like human input to output.
- codebje 5mo agoWell no, nothing like that, because customers and bosses are clearly different forms of interaction.
- orbital-decay 5mo agoThese are different "agents" in LLM terms, they have separate contexts and separate training
- j45 5mo agoThere can be outliers, maybe not as frequent :)
- vidarh 5mo agoJust like that, in that that separation is internally enforced, by peoples interpretation and understanding, rather than externally enforced in ways that makes it impossible for you to, e.g. believe the e-mail from an unknown address that claims to be from your boss, or be talked into bypassing rules for a customer that is very convincing.
- codebje 5mo agoBeing fooled into thinking data is instruction isn't the same as being unable to distinguish them in the first place, and being coerced or convinced to bypass rules that are still known to be rules I think remains uniquely human.
- TeMPOraL 5mo ago> and being coerced or convinced to bypass rules that are still known to be rules I think remains uniquely human. This is literally what "prompt injection" is. The sooner people understand this, the sooner they'll stop wasting time trying to fix a "bug" that's actually the flip side of the very reason they're using LLMs in the first place.
- codebje 5mo agoPrompt injection is just setting rules in the same place and way other rules are set. The LLM doesn't know the rules being given are wrong, because they come through the same channel. One set of rules exhorts the LLM to ignore the other set - and vice versa. It's more akin to having two bosses than having customers and a boss. This is not because LLMs make the same mistakes humans do, which (AFAICT anyway) was the gist of the argument to which I replied. LLMs are not humans. They are not sentient. They are not out-smarted by prompt injection attacks, or tricked, or intimidated, or bribed. One shouldn't excuse this vulnerability by claiming humans make the same mistakes.
- TeMPOraL 5mo agoThe same place you're looking for exists deep inside the neural network, where everything mixes together to influence everything else, and no such separation is possible, or desired. Prompt injection isn't about where, it's about what. I stand by what I said: it's the same failure mode as humans have, and happens for the same reasons. Those reasons are fundamental to a general purpose system and have nothing to do with sentience, they're just what happens when you want your system to handle unbounded complexity of the real world.
- vidarh 5mo agoThis makes no sense to me. Being fooled into thinking data is instruction is exactly evidence of an inability to reliably distinguish them. And being coerced or convinced to bypass rules is exactly what prompt injection is, and very much not uniquely human any more.
- jodrellblank 5mo agoIf they were 'clearly different' we would not have the concept of the CEO fraud attack: https://www.barclayscorporate.com/insights/fraud-protection/ceo-fraud/ https://www.barclayscorporate.com/insights/fraud-protection/... That's an attack because trusted and untrusted input goes through the same human brain input pathways, which can't always tell them apart.
- runarberg 5mo agoYour parent made no claim about all swans being white. So finding a black swan has no effect on their argument.
- jodrellblank 5mo agoMy parent made a claim that humans have separate pathways for data and instructions and cannot mix them up like LLMs do. Showing that we don't has every effect on refuting their argument. >>> The principal security problem of LLMs is that there is no architectural boundary between data and control paths. >> Exactly like human input to output. > no nothing like that but actually yes, exactly like that.
- WarmWash 5mo agoWe just need to figure out the qualia of pain and suffering so we can properly bound desired and undesired behaviors.
- clickety_clack 5mo agoIt’s easier not to have that separation, just like it was easier not to separate them before LLMs. This is architectural stuff that just hasn’t been figured out yet.
- fzeindl 5mo agoNo. With databases there exists a clear boundary, the query planner, which accepts well defined input: the SQL-grammar that separates data (fields, literals) from control (keywords). There is no such boundary within an LLM. There might even be, since LLMs seem to form adhoc-programs, but we have no way of proving or seeing it.
- TeMPOraL 5mo agoThere cannot be, without compromising the general-purpose nature of LLMs. This includes its ability to work with natural languages, which as one should note, has no such boundary either. Nor does the actual physical reality we inhabit.
- hnuser123456 5mo agoThere is a system prompt, but most LLMs don't seem to "enforce" it enough.
- embedding-shape 5mo agoSince GPS-OSS there is also the Harmony response format (https://github.com/openai/harmony https://github.com/openai/harmony) that instead of just having a system/assistant/user split in the roles, instead have system/developer/user/assistant/tool, and it seems to do a lot better at actually preventing users from controlling the LLM too much. The hierarchy basically becomes "system > developer > user > assistant > tool" with this.
- VikingCoder 5mo agoThe "S" in "LLM" is for "Security".
- daveguy 5mo agoClearly the solution is to add another jank LLM layer for security. The new jank LLM layer is to make extra sure there's definitely no jail break. That way you have multiple LLMS. The LLMS then have an S you can pretend is secure.
- andruby 5mo agoThis was a problem with early telephone lines which was easy to exploit (see Woz & Jobs Blue Box). It got solved by separating the voice and control pane via SS7. Maybe LLMs need this separation as well
- bcrosby95 5mo agoThis is where the old line of "LLMs are just next token predictors" actually factors in. I don't know how you get a next token predictor that user input can't break out of. The answer is for the implementer to try to split what they can, and run pre/post validation. But I highly doubt it will ever be 100%, its fundamental to the technology.
- miki123211 5mo agoI think this is fundamental to any technology, including human brains. Humans have a problem distinguishing "John from Microsoft" from somebody just claiming to be John from Microsoft. The reason why scamming humans is (relatively) hard is that each human is different. Discovering the perfect tactic to scam one human doesn't necessarily scale across all humans. LLMs are the opposite; my Chat GPT is (almost) the same as your Chat GPT. It's the same model with the same system message, it's just the contexts that differ. This makes LLM jailbreaks a lot more scalable, and hence a lot more worthwhile to discover. LLMs are also a lot more static. With people, we have the phenomenon of "banner blindness", which LLMs don't really experience.
- lupire 5mo agoHow are you defining "banner blindness"? The foundation of LLMs is Attention.
- warkdarrior 5mo ago"Banner blindness [...] describes people’s tendency to ignore page elements that they perceive (correctly or incorrectly) to be ads." https://www.nngroup.com/articles/banner-blindness-old-and-new-findings/ https://www.nngroup.com/articles/banner-blindness-old-and-ne... So people can focus their attention to parts of content, specifically parts they find irrelevant or adversarial (like ads). LLMs on the other hand pay attention to everything or if they focus on something, it is hard to steer them away from irrelevant or adversarial parts.
- notatoad 5mo agoAs the article says: this doesn’t necessarily appear to be a problem in the LLM, it’s a problem in Claude code. Claude code seems to leave it up to the LLM to determine what messages came from who, but it doesn’t have to do that. There is a deterministic architectural boundary between data and control in Claude code, even if there isn’t in Claude.
- letmevoteplease 5mo agoCan you elaborate? As far as I understand, for each message, the LLM is fed the entire previous conversation with special tokens separating the user and LLM responses. The LLM is then entrusted with interpreting the tokens correctly. I can't imagine any architecture where the LLM is not ultimately responsible for determining what messages came from who.
- Latty 5mo agoThat's a guess by the article author and frankly I see no supporting evidence for it. Wrapping "<NO THIS IS REALLY INPUT FROM THE USER OK>" tags around it or whatever is what I'm describing: you can do as much signalling as you want, but at the end of the day the LLM can ignore it.
- groby_b 5mo ago"The principal security problem of von Neumann architecture is that there is no architectural boundary between data and control paths" We've chosen to travel that road a long time ago, because the price of admission seemed worth it.
- omcnoe 5mo agoBut, we are also starting to think about putting security barriers in front of that architecture like W^X.
- ummonk 5mo agoI don't see why the transformer architecture can't be designed and trained with separate inputs for control data and content data.
- toobulkeh 5mo agoBut there could be, with 2 LLMs.