7 ms·

Everything to do with LLM prompts reminds me of people doing regexes to try and sanitise input against SQL injections a few decades ago, just papering over the

by Latty 5mo ago

Everything to do with LLM prompts reminds me of people doing regexes to try and sanitise input against SQL injections a few decades ago, just papering over the flaw but without any guarantees. It's weird seeing people just adding a few more "REALLY REALLY REALLY REALLY DON'T DO THAT" to the prompt and hoping, to me it's just an unacceptable risk, and any system using these needs to treat the entire LLM as untrusted the second you put any user input into the prompt.

perching_aix 5mo ago
It's less about security in my view, because as you say, you'd want to ensure safety using proper sandboxing and access controls instead. It hinders the effectiveness of the model. Or at least I'm pretty sure it getting high on its own supply (in this specific unintended way) is not doing it any favors, even ignoring security.
- sanitycheck 5mo ago
  It's both, really. The companies selling us the service aren't saying "you should treat this LLM as a potentially hostile user on your machine and set up a new restricted account for it accordingly", they're just saying "download our app! connect it to all your stuff!" and we can't really blame ordinary users for doing that and getting into trouble.
  - perching_aix 5mo ago
    There's a growing ecosystem of guardrailing methods, and these companies are contributing. Antrophic specifically puts in a lot of effort to better steer and characterize their models AFAIK. I primarily use Claude via VS Code, and it defaults to asking first before taking any action. It's simply not the wild west out here that you make it out to be, nor does it need to be. These are statistical systems, so issues cannot be fully eliminated, but they can be materially mitigated. And if they stand to provide any value, they should be. I can appreciate being upset with marketing practices, but I don't think there's value in pretending to having taken them at face value when you didn't, and when you think people shouldn't.
    - le-mark 5mo ago
      > It's simply not the wild west out here that you make it out to be It is though. They are not talking about users using Claude code via vscode, they’re talking about non technical users creating apps that pipe user input to llms. This is a growing thing.
      - perching_aix 5mo ago
        The best solution to which are the aforementioned better defaults, stricter controls, and sandboxing (and less snakeoil marketing). Less so the better tuning of models, unlike in this case, where that is going to be exactly the best fit approach most probably.
    - sanitycheck 5mo ago
      I'm a naturally paranoid, very detail-oriented, man who has been a professional software developer for >25 years. Do you know anyone who read the full terms and conditions for their last car rental agreement prior to signing anything? I did that. I do not expect other people to be as careful with this stuff as I am, and my perception of risk comes not only from the "hang on, wtf?" feeling when reading official docs but also from seeing what supposedly technical users are talking about actually doing on Reddit, here, etc. Of course I use Claude Code, I'm not a Luddite (though they had a point), but I don't trust it and I don't think other people should either.
hydroreadsstuff 5mo ago
I like the Dark Souls model for user input - messages. https://darksouls.fandom.com/wiki/Messages https://darksouls.fandom.com/wiki/Messages Premeditated words and sentence structure. With that there is no need for moderation or anti-abuse mechanics. Not saying this is 100% applicable here. But for their use case it's a good solution.
- nottorp 5mo ago
  But then... you'd have a programming language. The promise is to free us from the tyranny of programming!
  - dleeftink 5mo ago
    Maybe something more like a concordancer that provides valid or likely next phrase/prompt candidates. Think LancsBox[0]. [0]: https://lancsbox.lancs.ac.uk/ https://lancsbox.lancs.ac.uk/
- thaumasiotes 5mo ago
  > I like the Dark Souls model for user input - messages. > Premeditated words and sentence structure. With that there is no need for moderation or anti-abuse mechanics. I guess not, if you're willing to stick your fingers in your ears, really hard. If you'd prefer to stay at least somewhat in touch with reality, you need to be aware that "predetermined words and sentence structure" don't even address the problem. https://habitatchronicles.com/2007/03/the-untold-history-of-toontowns-speedchat-or-blockchattm-from-disney-finally-arrives/ https://habitatchronicles.com/2007/03/the-untold-history-of-... > Disney makes no bones about how tightly they want to control and protect their brand, and rightly so. Disney means "Safe For Kids". There could be no swearing, no sex, no innuendo, and nothing that would allow one child (or adult pretending to be a child) to upset another. > Even in 1996, we knew that text-filters are no good at solving this kind of problem, so I asked for a clarification: "I’m confused. What standard should we use to decide if a message would be a problem for Disney?" > The response was one I will never forget: "Disney’s standard is quite clear: > No kid will be harassed, even if they don’t know they are being harassed." > "OK. That means Chat Is Out of HercWorld, there is absolutely no way to meet your standard without exorbitantly high moderation costs," we replied. > One of their guys piped up: "Couldn’t we do some kind of sentence constructor, with a limited vocabulary of safe words?" > Before we could give it any serious thought, their own project manager interrupted, "That won’t work. We tried it for KA-Worlds." > "We spent several weeks building a UI that used pop-downs to construct sentences, and only had completely harmless words – the standard parts of grammar and safe nouns like cars, animals, and objects in the world." > "We thought it was the perfect solution, until we set our first 14-year old boy down in front of it. Within minutes he’d created the following sentence: > I want to stick my long-necked Giraffe up your fluffy white bunny.
cookiengineer 5mo ago
Before 2023 I thought the way Star Trek portrayed humans fiddling with tech and not understanding any side effects was fiction. After 2023 I realized that's exactly how it's going to turn out. I just wish those self proclaimed AI engineers would go the extra mile and reimplement older models like RNNs, LSTMs, GRUs, DNCs and then go on to Transformers (or the Attention is all you need paper). This way they would understand much better what the limitations of the encoding tricks are, and why those side effects keep appearing. But yeah, here we are, humans vibing with tech they don't understand.
- dijksterhuis 5mo ago
  curiosity (will probably) kill humanity although whether humanity dies before the cat is an open question
- hacker_homie 5mo ago
  is this new tho, I don't know how to make a drill but I use them. I don't know how to make a car but i drive one. The issue I see is the personification, some people give vehicles names, and that's kinda ok because they usually don't talk back. I think like every technological leap people will learn to deal with LLMs, we have words like "hallucination" which really is the non personified version of lying. The next few years are going to be wild for sure.
  - le-mark 5mo ago
    Do you not see your own contradiction? Cars and drills don’t kill people, self driving cars can! Normal cars can if they’re operated unsafely by human. These types of uncritical comments really highlight the level of euphoria in this moment.
    - hacker_homie 5mo ago
      https://en.wikipedia.org/wiki/Motor_vehicle_fatality_rate_in_U.S._by_year https://en.wikipedia.org/wiki/Motor_vehicle_fatality_rate_in...
  - 5mo ago
hacker_homie 5mo ago
I have been saying this for a while, the issue is there's no good way to do LLM structured queries yet. There was an attempt to make a separate system prompt buffer, but it didn't work out and people want longer general contexts but I suspect we will end up back at something like this soon.
- HPsquared 5mo ago
  Fundamentally there's no way to deterministically guarantee anything about the output.
  - satvikpendem 5mo ago
    That is "fundamentally" not true, you can use a preset seed and temperature and get a deterministic output.
    - HPsquared 5mo ago
      I'll grant that you can guarantee the length of the output and, being a computer program, it's possible (though not always in practice) to rerun and get the same result each time, but that's not guaranteeing anything about said output.
      - satvikpendem 5mo ago
        What do you want to guarantee about the output, that it follows a given structure? Unless you map out all inputs and outputs, no it's not possible, but to say that it is a fundamental property of LLMs to be non deterministic is false, which is what I was inferring you meant, perhaps that was not what you implied.
        program_whiz 5mo ago
        Yeah I think there are two definitions of determinism people are using which is causing confusion. In a strict sense, LLMs can be deterministic meaning same input can generate same output (or as close as desired to same output). However, I think what people mean is that for slight changes to the input, it can behave in unpredictable ways (e.g. its output is not easily predicted by the user based on input alone). People mean "I told it don't do X, then it did X", which indicates a kind of randomness or non-determinism, the output isn't strictly constrained by the input in the way a reasonable person would expect.
HeavyStorm 5mo ago
The real issue is expecting an LLM to be deterministic when it's not.
- WithinReason 5mo ago
  Oh how I wish people understood the word "deterministic"
- Zambyte 5mo ago
  Language models are deterministic unless you add random input. Most inference tools add random input (the seed value) because it makes for a more interesting user experience, but that is not a fundamental property of LLMs. I suspect determinism is not the issue you mean to highlight.
  - usernametaken29 5mo ago
    Actually at a hardware level floating point operations are not associative. So even with temperature of 0 you’re not mathematically guaranteed the same response. Hence, not deterministic.
    - adrian_b 5mo ago
      You are right that as commonly implemented, the evaluation of an LLM may be non deterministic even when explicit randomization is eliminated, due to various race conditions in a concurrent evaluation. However, if you evaluate carefully the LLM core function, i.e. in a fixed order, you will obtain perfectly deterministic results (except on some consumer GPUs, where, due to memory overclocking, memory errors are frequent, which causes slightly erroneous results with non-deterministic errors). So if you want deterministic LLM results, you must audit the programs that you are using and eliminate the causes of non-determinism, and you must use good hardware. This may require some work, but it can be done, similarly to the work that must be done if you want to deterministically build a software package, instead of obtaining different executable files at each recompilation from the same sources.
fzeindl 5mo ago
The principal security problem of LLMs is that there is no architectural boundary between data and control paths. But this combination of data and control into a single, flexible data stream is also the defining strength of a LLM, so it can’t be taken away without also taking away the benefits.
- mt_ 5mo ago
  Exactly like human input to output.
  - codebje 5mo ago
    Well no, nothing like that, because customers and bosses are clearly different forms of interaction.
    - orbital-decay 5mo ago
      These are different "agents" in LLM terms, they have separate contexts and separate training
    - j45 5mo ago
      There can be outliers, maybe not as frequent :)
    - vidarh 5mo ago
      Just like that, in that that separation is internally enforced, by peoples interpretation and understanding, rather than externally enforced in ways that makes it impossible for you to, e.g. believe the e-mail from an unknown address that claims to be from your boss, or be talked into bypassing rules for a customer that is very convincing.
      - codebje 5mo ago
        Being fooled into thinking data is instruction isn't the same as being unable to distinguish them in the first place, and being coerced or convinced to bypass rules that are still known to be rules I think remains uniquely human.
hansmayer 5mo ago
"Make this application without bugs" :)
- otabdeveloper4 5mo ago
  You forgot to add "you are a senior software engineer with PhD level architectural insights" though.
  - paganel 5mo ago
    And "you're a regular commenter on Hacker News", just to make sure.
morkalork 5mo ago
We used to be engineers, now we are beggars pleading for the computer to work
- vannevar 5mo ago
  I don't know, "pleading for the computer to work" pretty much sums up my entire 40-year career in software. Only the level of abstraction has changed.
- yencabulator 5mo ago
  > We used to be engineers, now we are beggars pleading for the computer to work You mean beggars pleading for the computer to stop: https://www.businessinsider.com/meta-ai-alignment-director-openclaw-email-deletion-2026-2?op=1 https://www.businessinsider.com/meta-ai-alignment-director-o...
Kye 5mo ago
Modern LLMs do a great job of following instructions, especially when it comes to conflict between instructions from the prompter and attempts to hijack it in retrieval. Claude's models will even call out prompt injection attempts. Right up until it bumps into the context window and compacts. Then it's up to how well the interface manages carrying important context through compaction.
PunchyHamster 5mo ago
It somehow feels worse than regexes. At least you can see the flaws before it happens
sheepscreek 5mo ago
Honestly I try to treat all my projects as sandboxes, give the agents full autonomy for file actions in their folders. Just ask them to commit every chunk of related changes so we can always go back — and sync with remote right after they commit. If you want to be more pedantic, disable force push on the branch and let the LLMs make mistakes. But what we can’t afford to do is to leave the agents unsupervised. You can never tell when they’ll start acting drunk and do something stupid and unthinkable. Also you absolutely need to do a routine deep audits of random features in your projects, and often you’ll be surprised to discover some awkward (mis)interpretation of instructions despite having a solid test coverage (with all tests passing)!
jmyeet 5mo ago
I'm reminded of Asimov'sThree Laws of Robotics [1]. It's a nice idea but it immediately comes up against Godel's incompleteness theorems [2]. Formal proofs have limits in software but what robots (or, now, LLMs) are doing is so general that I think there's no way to guarantee limits to what the LLM can do. In short, it's a security nightmare (like you say). [1]: https://en.wikipedia.org/wiki/Three_Laws_of_Robotics https://en.wikipedia.org/wiki/Three_Laws_of_Robotics [2]: https://en.wikipedia.org/wiki/G%C3%B6del%27s_incompleteness_theorems https://en.wikipedia.org/wiki/G%C3%B6del%27s_incompleteness_...
andai 5mo ago
I tried to get GPT to talk like a regular guy yesterday. It was impossible for it to maintain adherence. It kept defaulting back to markdown and bullet points, after the first message. (Funny cause it scores highest on the instruction following benchmarks.) Might seem trivial but if it can't even do a basic style prompt... how are you supposed to trust it with anything serious?
josephcsible 5mo ago
> Everything to do with LLM prompts reminds me of people doing regexes to try and sanitise input against SQL injections a few decades ago, just papering over the flaw but without any guarantees. With the key difference being that it's possible to do this correctly with SQL (e.g., switch to prepared statements, or in the days before those existed, add escapes). It's impossible to fix this vulnerability in LLM prompts.
niyikiza 5mo ago
Was just at [Un]prompted conference where this was a live debate. The conversation is shifting but not fast enough. I've been screaming about this for a while: we can't win the prompt war, we need to move the enforcement out of the untrusted input channel and into the execution layer to truly achieve deterministic guarantees. There are emerging proposals that get this right, and some of us are taking it further. An IETF draft[0] proposes cryptographically enforced argument constraints at the tool boundary, with delegation chains that can only narrow scope at every hop. The token makes out-of-scope actions structurally impossible. Disclosure: I wrote the 00 draft [0] https://datatracker.ietf.org/doc/draft-niyikiza-oauth-attenuating-agent-tokens/ https://datatracker.ietf.org/doc/draft-niyikiza-oauth-attenu...

by Latty 5mo ago

perching_aix 5mo ago

It's less about security in my view, because as you say, you'd want to ensure safety using proper sandboxing and access controls instead. It hinders the effectiveness of the model. Or at least I'm pretty sure it getting high on its own supply (in this specific unintended way) is not doing it any favors, even ignoring security.

sanitycheck 5mo ago
It's both, really. The companies selling us the service aren't saying "you should treat this LLM as a potentially hostile user on your machine and set up a new restricted account for it accordingly", they're just saying "download our app! connect it to all your stuff!" and we can't really blame ordinary users for doing that and getting into trouble.
- perching_aix 5mo ago
  There's a growing ecosystem of guardrailing methods, and these companies are contributing. Antrophic specifically puts in a lot of effort to better steer and characterize their models AFAIK. I primarily use Claude via VS Code, and it defaults to asking first before taking any action. It's simply not the wild west out here that you make it out to be, nor does it need to be. These are statistical systems, so issues cannot be fully eliminated, but they can be materially mitigated. And if they stand to provide any value, they should be. I can appreciate being upset with marketing practices, but I don't think there's value in pretending to having taken them at face value when you didn't, and when you think people shouldn't.
  - le-mark 5mo ago
    > It's simply not the wild west out here that you make it out to be It is though. They are not talking about users using Claude code via vscode, they’re talking about non technical users creating apps that pipe user input to llms. This is a growing thing.
    - perching_aix 5mo ago
      The best solution to which are the aforementioned better defaults, stricter controls, and sandboxing (and less snakeoil marketing). Less so the better tuning of models, unlike in this case, where that is going to be exactly the best fit approach most probably.
  - sanitycheck 5mo ago
    I'm a naturally paranoid, very detail-oriented, man who has been a professional software developer for >25 years. Do you know anyone who read the full terms and conditions for their last car rental agreement prior to signing anything? I did that. I do not expect other people to be as careful with this stuff as I am, and my perception of risk comes not only from the "hang on, wtf?" feeling when reading official docs but also from seeing what supposedly technical users are talking about actually doing on Reddit, here, etc. Of course I use Claude Code, I'm not a Luddite (though they had a point), but I don't trust it and I don't think other people should either.

hydroreadsstuff 5mo ago

I like the Dark Souls model for user input - messages. https://darksouls.fandom.com/wiki/Messages https://darksouls.fandom.com/wiki/Messages Premeditated words and sentence structure. With that there is no need for moderation or anti-abuse mechanics. Not saying this is 100% applicable here. But for their use case it's a good solution.

nottorp 5mo ago
But then... you'd have a programming language. The promise is to free us from the tyranny of programming!
- dleeftink 5mo ago
  Maybe something more like a concordancer that provides valid or likely next phrase/prompt candidates. Think LancsBox[0]. [0]: https://lancsbox.lancs.ac.uk/ https://lancsbox.lancs.ac.uk/
thaumasiotes 5mo ago
> I like the Dark Souls model for user input - messages. > Premeditated words and sentence structure. With that there is no need for moderation or anti-abuse mechanics. I guess not, if you're willing to stick your fingers in your ears, really hard. If you'd prefer to stay at least somewhat in touch with reality, you need to be aware that "predetermined words and sentence structure" don't even address the problem. https://habitatchronicles.com/2007/03/the-untold-history-of-toontowns-speedchat-or-blockchattm-from-disney-finally-arrives/ https://habitatchronicles.com/2007/03/the-untold-history-of-... > Disney makes no bones about how tightly they want to control and protect their brand, and rightly so. Disney means "Safe For Kids". There could be no swearing, no sex, no innuendo, and nothing that would allow one child (or adult pretending to be a child) to upset another. > Even in 1996, we knew that text-filters are no good at solving this kind of problem, so I asked for a clarification: "I’m confused. What standard should we use to decide if a message would be a problem for Disney?" > The response was one I will never forget: "Disney’s standard is quite clear: > No kid will be harassed, even if they don’t know they are being harassed." > "OK. That means Chat Is Out of HercWorld, there is absolutely no way to meet your standard without exorbitantly high moderation costs," we replied. > One of their guys piped up: "Couldn’t we do some kind of sentence constructor, with a limited vocabulary of safe words?" > Before we could give it any serious thought, their own project manager interrupted, "That won’t work. We tried it for KA-Worlds." > "We spent several weeks building a UI that used pop-downs to construct sentences, and only had completely harmless words – the standard parts of grammar and safe nouns like cars, animals, and objects in the world." > "We thought it was the perfect solution, until we set our first 14-year old boy down in front of it. Within minutes he’d created the following sentence: > I want to stick my long-necked Giraffe up your fluffy white bunny.

cookiengineer 5mo ago

Before 2023 I thought the way Star Trek portrayed humans fiddling with tech and not understanding any side effects was fiction. After 2023 I realized that's exactly how it's going to turn out. I just wish those self proclaimed AI engineers would go the extra mile and reimplement older models like RNNs, LSTMs, GRUs, DNCs and then go on to Transformers (or the Attention is all you need paper). This way they would understand much better what the limitations of the encoding tricks are, and why those side effects keep appearing. But yeah, here we are, humans vibing with tech they don't understand.

dijksterhuis 5mo ago
curiosity (will probably) kill humanity although whether humanity dies before the cat is an open question
hacker_homie 5mo ago
is this new tho, I don't know how to make a drill but I use them. I don't know how to make a car but i drive one. The issue I see is the personification, some people give vehicles names, and that's kinda ok because they usually don't talk back. I think like every technological leap people will learn to deal with LLMs, we have words like "hallucination" which really is the non personified version of lying. The next few years are going to be wild for sure.
- le-mark 5mo ago
  Do you not see your own contradiction? Cars and drills don’t kill people, self driving cars can! Normal cars can if they’re operated unsafely by human. These types of uncritical comments really highlight the level of euphoria in this moment.
  - hacker_homie 5mo ago
    https://en.wikipedia.org/wiki/Motor_vehicle_fatality_rate_in_U.S._by_year https://en.wikipedia.org/wiki/Motor_vehicle_fatality_rate_in...
- 5mo ago

hacker_homie 5mo ago

I have been saying this for a while, the issue is there's no good way to do LLM structured queries yet. There was an attempt to make a separate system prompt buffer, but it didn't work out and people want longer general contexts but I suspect we will end up back at something like this soon.

HPsquared 5mo ago
Fundamentally there's no way to deterministically guarantee anything about the output.
- satvikpendem 5mo ago
  That is "fundamentally" not true, you can use a preset seed and temperature and get a deterministic output.
  - HPsquared 5mo ago
    I'll grant that you can guarantee the length of the output and, being a computer program, it's possible (though not always in practice) to rerun and get the same result each time, but that's not guaranteeing anything about said output.
    - satvikpendem 5mo ago
      What do you want to guarantee about the output, that it follows a given structure? Unless you map out all inputs and outputs, no it's not possible, but to say that it is a fundamental property of LLMs to be non deterministic is false, which is what I was inferring you meant, perhaps that was not what you implied.
      - program_whiz 5mo ago
        Yeah I think there are two definitions of determinism people are using which is causing confusion. In a strict sense, LLMs can be deterministic meaning same input can generate same output (or as close as desired to same output). However, I think what people mean is that for slight changes to the input, it can behave in unpredictable ways (e.g. its output is not easily predicted by the user based on input alone). People mean "I told it don't do X, then it did X", which indicates a kind of randomness or non-determinism, the output isn't strictly constrained by the input in the way a reasonable person would expect.

HeavyStorm 5mo ago

The real issue is expecting an LLM to be deterministic when it's not.

WithinReason 5mo ago
Oh how I wish people understood the word "deterministic"
Zambyte 5mo ago
Language models are deterministic unless you add random input. Most inference tools add random input (the seed value) because it makes for a more interesting user experience, but that is not a fundamental property of LLMs. I suspect determinism is not the issue you mean to highlight.
- usernametaken29 5mo ago
  Actually at a hardware level floating point operations are not associative. So even with temperature of 0 you’re not mathematically guaranteed the same response. Hence, not deterministic.
  - adrian_b 5mo ago
    You are right that as commonly implemented, the evaluation of an LLM may be non deterministic even when explicit randomization is eliminated, due to various race conditions in a concurrent evaluation. However, if you evaluate carefully the LLM core function, i.e. in a fixed order, you will obtain perfectly deterministic results (except on some consumer GPUs, where, due to memory overclocking, memory errors are frequent, which causes slightly erroneous results with non-deterministic errors). So if you want deterministic LLM results, you must audit the programs that you are using and eliminate the causes of non-determinism, and you must use good hardware. This may require some work, but it can be done, similarly to the work that must be done if you want to deterministically build a software package, instead of obtaining different executable files at each recompilation from the same sources.

fzeindl 5mo ago

The principal security problem of LLMs is that there is no architectural boundary between data and control paths. But this combination of data and control into a single, flexible data stream is also the defining strength of a LLM, so it can’t be taken away without also taking away the benefits.

mt_ 5mo ago
Exactly like human input to output.
- codebje 5mo ago
  Well no, nothing like that, because customers and bosses are clearly different forms of interaction.
  - orbital-decay 5mo ago
    These are different "agents" in LLM terms, they have separate contexts and separate training
  - j45 5mo ago
    There can be outliers, maybe not as frequent :)
  - vidarh 5mo ago
    Just like that, in that that separation is internally enforced, by peoples interpretation and understanding, rather than externally enforced in ways that makes it impossible for you to, e.g. believe the e-mail from an unknown address that claims to be from your boss, or be talked into bypassing rules for a customer that is very convincing.
    - codebje 5mo ago
      Being fooled into thinking data is instruction isn't the same as being unable to distinguish them in the first place, and being coerced or convinced to bypass rules that are still known to be rules I think remains uniquely human.

hansmayer 5mo ago

"Make this application without bugs" :)

otabdeveloper4 5mo ago
You forgot to add "you are a senior software engineer with PhD level architectural insights" though.
- paganel 5mo ago
  And "you're a regular commenter on Hacker News", just to make sure.

morkalork 5mo ago

We used to be engineers, now we are beggars pleading for the computer to work

vannevar 5mo ago
I don't know, "pleading for the computer to work" pretty much sums up my entire 40-year career in software. Only the level of abstraction has changed.
yencabulator 5mo ago
> We used to be engineers, now we are beggars pleading for the computer to work You mean beggars pleading for the computer to stop: https://www.businessinsider.com/meta-ai-alignment-director-openclaw-email-deletion-2026-2?op=1 https://www.businessinsider.com/meta-ai-alignment-director-o...

Kye 5mo ago

Modern LLMs do a great job of following instructions, especially when it comes to conflict between instructions from the prompter and attempts to hijack it in retrieval. Claude's models will even call out prompt injection attempts. Right up until it bumps into the context window and compacts. Then it's up to how well the interface manages carrying important context through compaction.

PunchyHamster 5mo ago

It somehow feels worse than regexes. At least you can see the flaws before it happens

sheepscreek 5mo ago

Honestly I try to treat all my projects as sandboxes, give the agents full autonomy for file actions in their folders. Just ask them to commit every chunk of related changes so we can always go back — and sync with remote right after they commit. If you want to be more pedantic, disable force push on the branch and let the LLMs make mistakes. But what we can’t afford to do is to leave the agents unsupervised. You can never tell when they’ll start acting drunk and do something stupid and unthinkable. Also you absolutely need to do a routine deep audits of random features in your projects, and often you’ll be surprised to discover some awkward (mis)interpretation of instructions despite having a solid test coverage (with all tests passing)!

jmyeet 5mo ago

I'm reminded of Asimov'sThree Laws of Robotics [1]. It's a nice idea but it immediately comes up against Godel's incompleteness theorems [2]. Formal proofs have limits in software but what robots (or, now, LLMs) are doing is so general that I think there's no way to guarantee limits to what the LLM can do. In short, it's a security nightmare (like you say). [1]: https://en.wikipedia.org/wiki/Three_Laws_of_Robotics https://en.wikipedia.org/wiki/Three_Laws_of_Robotics [2]: https://en.wikipedia.org/wiki/G%C3%B6del%27s_incompleteness_theorems https://en.wikipedia.org/wiki/G%C3%B6del%27s_incompleteness_...

andai 5mo ago

I tried to get GPT to talk like a regular guy yesterday. It was impossible for it to maintain adherence. It kept defaulting back to markdown and bullet points, after the first message. (Funny cause it scores highest on the instruction following benchmarks.) Might seem trivial but if it can't even do a basic style prompt... how are you supposed to trust it with anything serious?

josephcsible 5mo ago

> Everything to do with LLM prompts reminds me of people doing regexes to try and sanitise input against SQL injections a few decades ago, just papering over the flaw but without any guarantees. With the key difference being that it's possible to do this correctly with SQL (e.g., switch to prepared statements, or in the days before those existed, add escapes). It's impossible to fix this vulnerability in LLM prompts.

niyikiza 5mo ago

Was just at [Un]prompted conference where this was a live debate. The conversation is shifting but not fast enough. I've been screaming about this for a while: we can't win the prompt war, we need to move the enforcement out of the untrusted input channel and into the execution layer to truly achieve deterministic guarantees. There are emerging proposals that get this right, and some of us are taking it further. An IETF draft[0] proposes cryptographically enforced argument constraints at the tool boundary, with delegation chains that can only narrow scope at every hop. The token makes out-of-scope actions structurally impossible. Disclosure: I wrote the 00 draft [0] https://datatracker.ietf.org/doc/draft-niyikiza-oauth-attenuating-agent-tokens/ https://datatracker.ietf.org/doc/draft-niyikiza-oauth-attenu...