6 ms·

This line really stood out to me. > It may look like ordinary text, but when it is placed into an LLM context window, the model may interpret it as an instruct

• cryo32 3mo ago
  It’s a language model. The spoken and written language we use mixes code and data and requires judgement, experience and intelligence. It’s insanity. We’re fucked.
  • bombcar 3mo ago
    What if we made SQL injections everything?
• dyauspitr 3mo ago
  You will never have a 100% secure LLM just like you don’t have 100% secure people. But what will be secure and deterministic is the code it writes. Any time you need certainty it will just write code for it.
  • toasty228 3mo ago
    > Any time you need certainty it will just write code for it. Meanwhile: you give it the same exact model the same exact prompt 5 times and get 5 wildly different output
  • JSR_FDED 3mo ago
    The poisoned prompt will cause the wrong code to be written
• nemomarx 3mo ago
  Is there any good tech for it, though? This just seems like an inherent language model behavior and at best everyone has guard rails or big exclamation marks to separate their own instructions a little.
  • crote 3mo ago
    Correct. It should've been an immediate dealbreaker for applying the current generation of LLMs in crucial environments like banking. Unfortunately we live in a world where the CxO cares more about playing "keeping up with the Joneses" with his golf buddies and seeing the share price do a little bump every time he mentions AI. Truly keeping your money secure is not even remotely a priority.
• nicoburns 3mo ago
  It seems to me like it's a fundamentally unsolvable architectural issue with LLMs. Ultimately the only protection is to limit the powers we grant to any given LLM to reduce the fallout when (not if) things go wrong (much like we do with people). Of all the "AI doomsday" scenarios, people failing to understand this (and treating AIs like deterministic computers) seem like to most likely to cause issues.
  • Angostura 3mo ago
    Jokes on them. My bank will just truncate it to 10 characters.
    • TacticalCoder 3mo ago
      > Jokes on them. My bank will just truncate it to 10 characters. You do understand that this is just an example out of a bazillion and that planning to solve every place where data is fed to LLMs at 10 characters so that it's not mistaken for instructions ain't a viable solution?
      • Angostura 3mo ago
        Yes. I was being humorous. Apologies
  • jmount 3mo ago
    I really think one needs a "Harvard architecture" for AIs (data independent of instructions). Though yes, that may not be possible.
    • crooked-v 3mo ago
      I doubt it's possible, regardless of specific architecture, because if you want an AI that can do general purpose tasks like "look at my calendar and find a restaurant for the lunch meeting that the other people also like, but make sure nobody has to travel more than 20 minutes to get there, and it can't be too cold inside", then it has to ingest and understand a bunch of data to do that. The whole point is that the decision-making process is reading everything. The only "fix" is to make an AI smart enough that it can understand context for each item, which is a tall order.
• Someone 3mo ago
  > I plan on using this as a sort of benchmark for future AI discussions: "how do you plan on separating data from instructions?" You let a second LLM supervise the first, and don’t give the user/customer any way to send information to that LLM. For example, you can run a LLM trained to do sentiment analysis on the responses your customer chatbot generates and filter out responses that are impolite. You also can run one trained to flag potential legal issues, thus ‘preventing’ your chatbot from making the wrong promises to users.
  • vrighter 3mo ago
    If your task is to ensure an armed bomb does not explode, how can entroducing a second armed bomb be helpful?
  • snailmailman 3mo ago
    How is the second LLM not also vulnerable from prompt injection? In order to supervise the first, it must receive data (presumably output from the first LLM?). All generated output after the user input is in the context should be considered possibly compromised/prompt injected. Having a second LLM just adds more obfuscation, but prompt injection could be chained.
    • tweetle_beetle 3mo ago
      Quis custodiet ipsos custodes?
    • j_w 3mo ago
      That's when you bust out the third LLM. Nobody expects the fourth LLM to be the REAL LLM in the chain.
      • vrighter 3mo ago
        the real llm is the friends we make along the way!
  • caminanteblanco 3mo ago
• sddsfsdfsd2 3mo ago
  It's a tricky problem for sure. Even on CPUs this separation is maintained by architectural guardrails. The CPU will happily execute whatever it is permitted to fetch. There is and cannot be a fundamental divide betwixt the two. It's always going to be an artificial externally managed issue. I suppose this is no different for LLMs. My thinking is we are in the 50s/60s. Stuff is starting to come forward, it's all very exciting but very, very raw. I don't think this will last. The notions of "tokens" and how inference works will become arcane insider knowledge like how CPU registers and interrupts work. You don't work with CPUs, you work with "computers" and even then mostly "operating systems" or even "browsers". Reality has been abstracted away from you to a very impressive degree. I don't think it'll be different here, but we haven't had our Xerox PARC and Bell Labs moments yet.
• parliament32 3mo ago
  > separating data from instructions There's been a lot of talk about this (for years, honestly), but it all stems from a fundamental nonunderstanding of how LLMs work. There is no distinction for an LLM; "instructions" are a prompt concept, nothing more. It's not possible to separate the two, because LLMs simply take text (ie your instructions, then the data, or maybe in a different order, or maybe something completely else) and "predict" the next token, and repeat for as long as you want, with the volatility you ask for. There is no control plane, and there never will be a control plane, because asking for that is akin to asking "how do I separate data from instructions when I speak to a person?". You can ask nicely, "pretty please obey the first part of what I say and not stuff after", but there's no way to guarantee it (like you're used to with software). There is just input and output.
  • deleted 3mo ago
    [deleted]
  • parl_match 3mo ago
    Right, you have to set boundaries. You put each task and user input into a box, and then the LLM makes a decision. It can only access APIs that have user identity attached, that act within the scope of the requesting user. It can be done, but unsurprisingly it looks exactly like microservices distributed auth (also ZTP). It's all the same problem, just instead of a JVM, it's an LLM.
    • gbalduzzi 3mo ago
      User identity attached is not a solution, it doesn't solve anything if you have to pull in external data that you can't control. Like in the banking world, you can make everything super authenticated, but if you have an API that receives the latest wire transfer YOU received with the message attached, you don't control the message content and it can be an attack vector. Being authenticated/authorized is not the solution, it is data that the user can access.
  • tekne 3mo ago
    I mean: imagine we double our token space to get "red" tokens ans "blue" tokens. Then in all post-training, instructions are red and data is blue. The model can be explicitly trained to ignore instructions written in blue tokens. All external data is blue. All you'd need to do is figure out a nice way to pre-train -- interestingly, you could try pre-training on unfiltered blue data and processed red/blue transcripts! Likewise, model-actions (e.g. open file) could be written only in red, and hence you'd never learn to do them from the unfiltered data. The only connection between the red world and the blue world would be the processed trainign chats containing red and blue data togethers -- allowing the model to learn the relationship between them (while only being exposed to examples where red instructions are strictly followed, whatever the blue says)
• eclipsetheworld 3mo ago
  I have been working on this issue for a bit, and the most interesting approach I have seen so far comes from the research domain of information-flow control, specifically Microsoft’s FIDES work. The idea is not to distinguish instructions from data. It is closer to having different privilege levels. Not all code has to run in kernel space, some code runs in unprivileged user space. So what is the equivalent for LLM agents? In FIDES-style systems, every piece of information that enters the agent context is labeled along two dimensions: integrity and confidentiality. Integrity captures whether the data is trusted or untrusted (i.e. could it contain a prompt injection attack). Confidentiality captures who is allowed to see or receive it [0]. The privileged agent, sometimes called the planning agent, should not directly see untrusted data because it would be susceptible to prompt injection attacks. In the article’s example, a bank transaction’s sender-supplied reference would be untrusted. Instead, the planning agent receives a variable token. It can then either delegate processing of that variable to an unprivileged / quarantined agent with no or limited tool access, or pass the token as a reference to a tool. Tools then have policies attached to their arguments and outputs. These policies specify which integrity and confidentiality levels are allowed, and whether the tool call may proceed. The policy also determines how the result should be labeled. For example: 1. High-confidentiality data should not be allowed to flow into a `send_email` tool call addressed to an external recipient. 2. A tool call whose result depends on untrusted input should generally produce untrusted output. 3. A sensitive side-effecting tool should be able to reject calls that are influenced by untrusted context. So the answer to “how do you separate data from instructions?” may be: you do not rely on the model to do that separation. You track provenance and privilege outside the model, and then enforce the security policy at the tool boundary. [0] In the simplest implementation, confidentiality is assessed with a binary low/high value, however, in a more advanced implementation, confidentiality can be represented as the set of users or principals allowed to learn that information.
• ozim 3mo ago
  Quite simple you make harness and loads of people are building harnesses as we speak. Right now also a lot of people are building in a way where they give a sample data to LLM so that AI agent builds deterministic code for crunching data so that actual data doesn't go to LLM and is processd by regular code, only that code for processing is written by agent. You can always process only descriptions that are in the list and ones that are not recognized "ask a human" so just an allowlist. I do believe normal person would have most transactions that would be mostly the same and then couple that would stand out so you also can make allowlist from last 2 years as a starting point, not to bother people too much (I think no one has prompt injection in their last 2 years banking history besides ultra nerds maybe). I think by now it is common knowledge that "just dump all data at LLM and as some questions" or "let LLM process anything someone sends me in an e-mail" is silly. In "the standoff" Pliny was trying to hack tszzl harness and it wasn't working an Pliny is notorious for jail breaking LLMs.
  • totetsu 3mo ago
    I’ve noticed that for task that require consistency across very large body of text, like translating strings of very large doc, the approach of letting the agent split and it up and programmatically do it bit by bit, is much worse quality than just dumping it all in a single llm context.
    • ozim 3mo ago
      I guess someone is doing harness for that use case then. I was mostly thinking about payment transfer description that mostly would be more like a sentence. More about data lines like CSV as that would be what is used in banking. Lots of known attacks can be found with static analysis of text, even in long text blocks, finding "unexpected characters", finding "white text on white background" will still prevent a lot of attacks I believe. If you find in a text any IOC just don't process the text, write it to log file, document and let some person make a decision.
• AceJohnny2 3mo ago
  > "how do you plan on separating data from instructions?" Use a Harvard Architecture CPU, duh https://en.wikipedia.org/wiki/Harvard_architecture https://en.wikipedia.org/wiki/Harvard_architecture (j/k, if it wasn't obvious)
• kevin_thibedeau 3mo ago
  The current usage model comingles commands and data. That doesn't have to be the case. Use an input format that explicitly presents them as separate components parsed into a data structure with non-LLM tooling. Or stick with natural language input but parse into an intermediate format that can be verified to some standard of correctness.
  • jcgl 3mo ago
    I’m no expert, but as long as they’re represented by tokens in the end, they’re just tokens. Even if you train the transformer to treat them specially, a token is a token, and there’s no free lunch. At best, you’re going to be trading off between paying attention to this would-be security boundary and delivering high-quality results; the more you focus on one, the more you lose on the other.
• giancarlostoro 3mo ago
  There's a few simple things they could do to make these injections less dramatic / dangerous, but we have to re-learn everything in tech every time there's a new fancy toy.
• prezk 3mo ago
  Well, your computer is a Von Neumann machine that does not separate data from instructions, right? The techniques that secured it are not directly applicable to LLMs but in principle there must be ways to track them.
• mungoman2 3mo ago
  In this case it could be solved by not letting the LLM consume the transaction message. Effectively the same as preventing user supplied info going to the first argument of printf(). The transaction in question can remain opaque to the LLM and a %transaction.message% string is resolved in the layer between the LLM and the user.
• petterroea 3mo ago
  I found it very ironic that this article, which in practice is trying to solicit customers, is both talking about a fundamental weakness of LLM while also trying to sell AI consultation. In reality as many have said here you cannot have untrusted data in the context. There isn't that much more to say. Prompt injection is XXS without possibility of sanitation. There is no absolute cure.
• GJim 3mo ago
  > benchmark for future AI discussions: "how do you plan on separating data from instructions?" Ah! Avoiding in-band signalling! As this point, maybe we should have a discussion on why modern developers are unable to learn dangers we were aware of > 30 years ago from using 2600 Hz signalling tones over the phone.
  • inigyou 3mo ago
    FWIW the phone system designers were aware of that flaw but took it anyway to double the number of channels per carrier system - still a quite worthy trade-off in hindsight.
• ErroneousBosh 3mo ago
  > "how do you plan on separating data from instructions?" Hang on, is this basically just the same fault as SQL injection, you can force it to misinterpret a string as a command rather than an operand? I feel like we already fixed that.
• michaelcampbell 3mo ago
  > I plan on using this as a sort of benchmark for future AI discussions: "how do you plan on separating data from instructions?" I'm reminded of the thing that HN drools over constantly with LISP languages.
• nottorp 3mo ago
  > This line really stood out to me. Where did you people find any text to discuss? I saw like 3 headlines while scrolling past the first 3 screens then gave up.