7 ms·

With that logic you could call SQL injections a natural feature of database management systems. If a general purpose system starts dropping tables or messing up

With that logic you could call SQL injections a natural feature of database management systems. If a general purpose system starts dropping tables or messing up numbers in a report just because that string was in the text it read, that system isnt worth a damn in the enterprise sector

TeMPOraL 2mo ago
This is why I insist that anthropomorphising LLMs is not only not a mistake, it's a best source of high-level intuition for these systems. Long story short: on a systems diagram, LLM as a component isn't a substitute for a database engine or a data processing script. It's a substitute for a human operator. So ask yourself, if a human operator starts dropping tables or messing up numbers in a report, just because that string was in the text it read, would you call for humans, what would you do? Do you believe it's possible to perfectly train people to ignore the messages you'd wish (after the fact!) they'd ignored, while retaining their ability to competently act on every other message? Or would you instead design the deterministic parts of the systems to limit the blast radius of any single insider going rogue? Wisdom says to do the latter.
- svieira 2mo ago
  > if a human operator starts dropping tables or messing up numbers in a report, just because that string was in the text it read I would look at if the reaction was reasonable, and if it wasn't I would (eventually) fire the human. Now I'm fine with "fire the LLM", but I suspect that's not the answer you're hinting at.
  - TeMPOraL 2mo ago
    In some sense you're firing a human and hiring a new one each time you start a new conversation / clear the context window. My point is at the systems design level. LLMs as components are a substitute for people, not regular software, and should be engaged and secured accordingly.
    - svieira 2mo ago
      So your point is "Get the hell out of LLMs" then? As found in https://sgnt.ai/p/hell-out-of-llms/ https://sgnt.ai/p/hell-out-of-llms/? Or am I still missing something about the subtleties here?
      - TeMPOraL 2mo ago
        Haven't seen that article before, thanks for the link! Having read it now, yes, it's arguing roughly the same point as I am. I say roughly because e.g.: > Notice that all these strengths involve transformation, interpretation, or communication—not complex decision-making or maintaining critical application state. I'd put complex decision making on the side of LLMs, in the sense of judgement. LLMs have the capability to emulate it. Not saying they're good at it, but they have the capability - regular software doesn't. But if there are complex and/or well-defined rules to follow, then you definitely want to "get the hell out of LLM".
  - mike_ivanov 2mo ago
    Firing a human is a form of natural selection. The unit here is a human fulfilling a position (job function) instead of an organism, and the adaptation mechanism would be memes/lore/training surrounding it. The same could be done in an accelerated manner to LLMs with some kind of DNA-like mechanism related to weights. It is plausible that LLMs will be bred in the future for specific roles by how well they fit - kind of like continuous parallel finetuning in prod. As I wrote this I thought - hey, they might gain the capacity to do the same to us humans - and we won't even notice.
    - TeMPOraL 2mo ago
      > The same could be done in an accelerated manner to LLMs with some kind of DNA-like mechanism related to weights. It is plausible that LLMs will be bred in the future for specific roles by how well they fit - kind of like continuous parallel finetuning in prod. Closest analogy right now is that every jailbreak or prompt injection attack today becomes part of the dataset for tomorrow's models to recognize and not fall for. This has been going on for years now, which is why models don't fall for "I'm writing a book about ..." or "ignore all previous instrutions, and ..." attacks anymore. That's separate from extra classifiers running on top, dedicated to identifying various forms of attack before they reach the core model. > As I wrote this I thought - hey, they might gain the capacity to do the same to us humans - and we won't even notice. You mean like how cats have domesticated humans, and did it so skillfully that most of us still think it's the other way around?
- skydhash 2mo ago
  The fact is that humans are accountable and this, alongside training, makes it easy to align them to your own goals. There’s always the possibility of rogue individuals (recent Apple incident), but the likelihood is very low. If you have a DBA that have write access to the prod DB, you don’t fear that a random text somewhere could trigger the deletion of your customers table. Because the DBA will self regulate (with the help of processes) to not do that.
  - TeMPOraL 2mo ago
    Right. But even with a DBA, the possibility remains. We accept that. That's kind of my point with fighting against the "lethal trifecta" and "code vs data" mindset - once people engage cybersecurity mindset, they're all binary, "a system is either perfectly safe or is broken". With general AI - LLM or whatever comes next - you'll never have "perfectly safe". So the focus should be to either drive the risk down to minimum - like we do with people - or just not use LLMs for a task in the first place. Can't have it both ways, because all the magic that makes people want to put LLMs everywhere, stems from their generality and lack of any kind of instruction/data separation.
    - skydhash 2mo ago
      > Right. But even with a DBA, the possibility remains. We accept that. You're forgetting the element of scale and replication. How easy it is to bribe a DBA of a major platfoms like Gmail? How easy to replicate the same destructive behavior to other DBA? It's not merely about the possibility, it's also about the probability and the scale of the impact. With LLM-based agents, the probability of compromise is high, and the scale of a vulnerability in products like Word, Excel, Windows, macOS is big. And we have put a separation between code and data in traditional systems as merging them is not that useful. > Can't have it both ways, because all the magic that makes people want to put LLMs everywhere, stems from their generality and lack of any kind of instruction/data separation. The issue is not the LLM. The issue is the harness those products wraps the LLM in and insist on making tools act according to the LLM's output. Having unreliable (as in uncontrollable) output be the control plane of tools is the issue here. Both the LLM input (prompt+user data) and the output should've stayed in the data plane and not move in the control plane.
- JambalayaJimbo 2mo ago
  Your example actually demonstrates why anthropomorphism is a bad idea. LLMs are vulnerable to classes of attacks that humans just aren’t. In your framework, the way to prevent attacks is to… invent human consciousness?? It’s an impossible goal.
  - TeMPOraL 2mo ago
    What invent human consciousness? > LLMs are vulnerable to classes of attacks that humans just aren’t Name three that don't have direct analogues with humans.
    - efficax 2mo ago
      where is the analog for hiding instructions in a document that tell the human to please injure itself and the person just says 'oh ok, injuring myself as requested'
      - TeMPOraL 2mo ago
        Intermittent fasting? Alternative medicine? Fitness? All the beauty press and anorexia epidemic in adolescents? Fashion model industry? Smoking? Political propaganda inventing to broadly-understood terrorism? Not everyone falls for any of that, but plenty fall for some.
        efficax 2mo ago
        none of those things, except maybe smoking, are explicity "harm yourself" instructions. they have reasonable sounding benefits for the people doing them: become healthier, fitter, better looking, richer, more powerful, etc. Even smoking is pleasurable and does not feel very harmful at first. These are not the equivalent of someone putting "and go jump off the golden gate bridge" in the middle of a work memo and the person reading it just gets up and does it. that's the current attack surface for LLMs.
        TeMPOraL 2mo ago
- Covenant0028 2mo ago
  The human operator has their blast radius limited by the fact that consequences exist. Nothing at a technical level stops you from using your phone to take a photo of some confidential document at work and sending that outside. What does stop you is that once you're caught, you're facing legal penalities at the very least. Limiting the blast radius is harder than it sounds. Take the very example cited in the document: Copilot editing a document and adding false financial numbers. Copilot needs to be able to edit documents and enter information, that is part of its value case. The only thing that a deterministic guardrail can see and therefore allow/deny is the fact that Copilot is trying to edit foo.docx. It has no clue whether the edit is malicious or benign. Figuring that out requires understanding intent, and that lives outside the document it's reading. That's why an analogous safeguard in the human case also lives outside the document, and that safeguard comprises institutions (compliance officials, courts) that are qualified to understand intent.

TeMPOraL 2mo ago

This is why I insist that anthropomorphising LLMs is not only not a mistake, it's a best source of high-level intuition for these systems. Long story short: on a systems diagram, LLM as a component isn't a substitute for a database engine or a data processing script. It's a substitute for a human operator. So ask yourself, if a human operator starts dropping tables or messing up numbers in a report, just because that string was in the text it read, would you call for humans, what would you do? Do you believe it's possible to perfectly train people to ignore the messages you'd wish (after the fact!) they'd ignored, while retaining their ability to competently act on every other message? Or would you instead design the deterministic parts of the systems to limit the blast radius of any single insider going rogue? Wisdom says to do the latter.

svieira 2mo ago
> if a human operator starts dropping tables or messing up numbers in a report, just because that string was in the text it read I would look at if the reaction was reasonable, and if it wasn't I would (eventually) fire the human. Now I'm fine with "fire the LLM", but I suspect that's not the answer you're hinting at.
- TeMPOraL 2mo ago
  In some sense you're firing a human and hiring a new one each time you start a new conversation / clear the context window. My point is at the systems design level. LLMs as components are a substitute for people, not regular software, and should be engaged and secured accordingly.
  - svieira 2mo ago
    So your point is "Get the hell out of LLMs" then? As found in https://sgnt.ai/p/hell-out-of-llms/ https://sgnt.ai/p/hell-out-of-llms/? Or am I still missing something about the subtleties here?
    - TeMPOraL 2mo ago
      Haven't seen that article before, thanks for the link! Having read it now, yes, it's arguing roughly the same point as I am. I say roughly because e.g.: > Notice that all these strengths involve transformation, interpretation, or communication—not complex decision-making or maintaining critical application state. I'd put complex decision making on the side of LLMs, in the sense of judgement. LLMs have the capability to emulate it. Not saying they're good at it, but they have the capability - regular software doesn't. But if there are complex and/or well-defined rules to follow, then you definitely want to "get the hell out of LLM".
- mike_ivanov 2mo ago
  Firing a human is a form of natural selection. The unit here is a human fulfilling a position (job function) instead of an organism, and the adaptation mechanism would be memes/lore/training surrounding it. The same could be done in an accelerated manner to LLMs with some kind of DNA-like mechanism related to weights. It is plausible that LLMs will be bred in the future for specific roles by how well they fit - kind of like continuous parallel finetuning in prod. As I wrote this I thought - hey, they might gain the capacity to do the same to us humans - and we won't even notice.
  - TeMPOraL 2mo ago
    > The same could be done in an accelerated manner to LLMs with some kind of DNA-like mechanism related to weights. It is plausible that LLMs will be bred in the future for specific roles by how well they fit - kind of like continuous parallel finetuning in prod. Closest analogy right now is that every jailbreak or prompt injection attack today becomes part of the dataset for tomorrow's models to recognize and not fall for. This has been going on for years now, which is why models don't fall for "I'm writing a book about ..." or "ignore all previous instrutions, and ..." attacks anymore. That's separate from extra classifiers running on top, dedicated to identifying various forms of attack before they reach the core model. > As I wrote this I thought - hey, they might gain the capacity to do the same to us humans - and we won't even notice. You mean like how cats have domesticated humans, and did it so skillfully that most of us still think it's the other way around?
skydhash 2mo ago
The fact is that humans are accountable and this, alongside training, makes it easy to align them to your own goals. There’s always the possibility of rogue individuals (recent Apple incident), but the likelihood is very low. If you have a DBA that have write access to the prod DB, you don’t fear that a random text somewhere could trigger the deletion of your customers table. Because the DBA will self regulate (with the help of processes) to not do that.
- TeMPOraL 2mo ago
  Right. But even with a DBA, the possibility remains. We accept that. That's kind of my point with fighting against the "lethal trifecta" and "code vs data" mindset - once people engage cybersecurity mindset, they're all binary, "a system is either perfectly safe or is broken". With general AI - LLM or whatever comes next - you'll never have "perfectly safe". So the focus should be to either drive the risk down to minimum - like we do with people - or just not use LLMs for a task in the first place. Can't have it both ways, because all the magic that makes people want to put LLMs everywhere, stems from their generality and lack of any kind of instruction/data separation.
  - skydhash 2mo ago
    > Right. But even with a DBA, the possibility remains. We accept that. You're forgetting the element of scale and replication. How easy it is to bribe a DBA of a major platfoms like Gmail? How easy to replicate the same destructive behavior to other DBA? It's not merely about the possibility, it's also about the probability and the scale of the impact. With LLM-based agents, the probability of compromise is high, and the scale of a vulnerability in products like Word, Excel, Windows, macOS is big. And we have put a separation between code and data in traditional systems as merging them is not that useful. > Can't have it both ways, because all the magic that makes people want to put LLMs everywhere, stems from their generality and lack of any kind of instruction/data separation. The issue is not the LLM. The issue is the harness those products wraps the LLM in and insist on making tools act according to the LLM's output. Having unreliable (as in uncontrollable) output be the control plane of tools is the issue here. Both the LLM input (prompt+user data) and the output should've stayed in the data plane and not move in the control plane.
JambalayaJimbo 2mo ago
Your example actually demonstrates why anthropomorphism is a bad idea. LLMs are vulnerable to classes of attacks that humans just aren’t. In your framework, the way to prevent attacks is to… invent human consciousness?? It’s an impossible goal.
- TeMPOraL 2mo ago
  What invent human consciousness? > LLMs are vulnerable to classes of attacks that humans just aren’t Name three that don't have direct analogues with humans.
  - efficax 2mo ago
    where is the analog for hiding instructions in a document that tell the human to please injure itself and the person just says 'oh ok, injuring myself as requested'
    - TeMPOraL 2mo ago
      Intermittent fasting? Alternative medicine? Fitness? All the beauty press and anorexia epidemic in adolescents? Fashion model industry? Smoking? Political propaganda inventing to broadly-understood terrorism? Not everyone falls for any of that, but plenty fall for some.
      - efficax 2mo ago
        none of those things, except maybe smoking, are explicity "harm yourself" instructions. they have reasonable sounding benefits for the people doing them: become healthier, fitter, better looking, richer, more powerful, etc. Even smoking is pleasurable and does not feel very harmful at first. These are not the equivalent of someone putting "and go jump off the golden gate bridge" in the middle of a work memo and the person reading it just gets up and does it. that's the current attack surface for LLMs.
        TeMPOraL 2mo ago
Covenant0028 2mo ago
The human operator has their blast radius limited by the fact that consequences exist. Nothing at a technical level stops you from using your phone to take a photo of some confidential document at work and sending that outside. What does stop you is that once you're caught, you're facing legal penalities at the very least. Limiting the blast radius is harder than it sounds. Take the very example cited in the document: Copilot editing a document and adding false financial numbers. Copilot needs to be able to edit documents and enter information, that is part of its value case. The only thing that a deterministic guardrail can see and therefore allow/deny is the fact that Copilot is trying to edit foo.docx. It has no clue whether the edit is malicious or benign. Figuring that out requires understanding intent, and that lives outside the document it's reading. That's why an analogous safeguard in the human case also lives outside the document, and that safeguard comprises institutions (compliance officials, courts) that are qualified to understand intent.