7 ms·

AI companies just 'acknowledging' risks and suggesting users take unreasonable precautions is such crap

NitpickLawyer 8mo ago
> users take unreasonable precautions It doesn't help that so far the communicators have used the wrong analogy. Most people writing on this topic use "injection" a la SQL injection to describe these things. I think a more apt comparison would be phishing attacks. Imagine spawning a grandma to fix your files, and then read the e-mails and sort them by category. You might end up with a few payments to a nigerian prince, because he sounded so sweet.
- uhfraid 8mo ago
  Command/“prompt” injection is correct terminology and what they’re typically mapped to in the CVE E.g. CVE-2026-22708
  - NitpickLawyer 8mo ago
    Perhaps I worded that poorly. I agree that technically this is an injection. What I don't think is accurate is to then compare it to sql injection and how we fixed that. Because in SQL world we had ways to separate control channels from data channels. In LLMs we don't. Until we do, I think it's better to think of the aftermath as phishing, and communicate that as the threat model. I guess what I'm saying is "we can't use the sql analogy until there's a architectural change in how LLMs work". With LLMs, as soon as "external" data hits your context window, all bets are off. There are people in this thread adamant that "we have the tools to fix this". I don't think that we do, while keeping them useful (i.e. dynamically processing external data).
rsynnott 8mo ago
It largely seems to amount to "to use this product safely, simply don't use it".
sodapopcan 8mo ago
I believe that's known as "The Steve Jobs Solution" but don't quote me on that. Regardless, just don't hold it that way.
AmbroseBierce 8mo ago
It's exactly like guns, we know they will be used in school shootings but that doesn't stop their selling in the slightest, the businesses just externalize all the risks claiming it's all up fault of the end users and that they mentioned all the risks, and that's somehow enough in any society build upon unfettered capitalism like the US.
- delaminator 8mo ago
  If you’re going to use “school shootings” as your “muh capitalism”, the counter argument is the millions of people who don’t do school shootings despite access to guns. There are common factors between all of the school shooters from the last decade - pharmacology and ideology.
  - AmbroseBierce 8mo ago
    it's not the mental issues they had, its the drugs they were taking for it right? Please. Look at what Australia did after their 1996 shooting, the main reason they have so few of them, but I know you won't, as millions of Americans you will forever do all sort of mental gymnastics to justify keeping easy access to semi-automatic guns. > From the information obtained, it appears that most school shooters were not previously treated with psychotropic medications - and even when they were, no direct or causal association was found https://pubmed.ncbi.nlm.nih.gov/31513302/ https://pubmed.ncbi.nlm.nih.gov/31513302/
    - delaminator 8mo ago
      If you like, but I'm not American. Millions of Americans believe the right to bear arms is not a right the govt. should be able to take away. Obesity kills 10x more Americans than guns. Australia locked up millions of people in their homes and forced them into dangerous medical procedures.
      - nutjob2 8mo ago
        > Australia locked up millions of people in their homes and forced them into dangerous medical procedures. Your comically bad faith description of Aus covid measures is pure nonsense. > Obesity kills 10x more Americans than guns. And? Obesity kills roughly the same number of people in other countries but guns kill 40 times more people in the US than other countries.
ronbenton 8mo ago
Telling uses to “watch out for prompt injections” is insane. Less than 1% of the population knows what that even means. Not to mention these agents are commonly used to summarize things people haven’t read. This is more than unreasonable, it’s negligent
- intended 8mo ago
  We will have tv shows with hackers “prompt injecting” before that number goes beyond 1%

> users take unreasonable precautions It doesn't help that so far the communicators have used the wrong analogy. Most people writing on this topic use "injection" a la SQL injection to describe these things. I think a more apt comparison would be phishing attacks. Imagine spawning a grandma to fix your files, and then read the e-mails and sort them by category. You might end up with a few payments to a nigerian prince, because he sounded so sweet.

uhfraid 8mo ago
Command/“prompt” injection is correct terminology and what they’re typically mapped to in the CVE E.g. CVE-2026-22708
- NitpickLawyer 8mo ago
  Perhaps I worded that poorly. I agree that technically this is an injection. What I don't think is accurate is to then compare it to sql injection and how we fixed that. Because in SQL world we had ways to separate control channels from data channels. In LLMs we don't. Until we do, I think it's better to think of the aftermath as phishing, and communicate that as the threat model. I guess what I'm saying is "we can't use the sql analogy until there's a architectural change in how LLMs work". With LLMs, as soon as "external" data hits your context window, all bets are off. There are people in this thread adamant that "we have the tools to fix this". I don't think that we do, while keeping them useful (i.e. dynamically processing external data).

rsynnott 8mo ago

It largely seems to amount to "to use this product safely, simply don't use it".

sodapopcan 8mo ago

I believe that's known as "The Steve Jobs Solution" but don't quote me on that. Regardless, just don't hold it that way.

AmbroseBierce 8mo ago

It's exactly like guns, we know they will be used in school shootings but that doesn't stop their selling in the slightest, the businesses just externalize all the risks claiming it's all up fault of the end users and that they mentioned all the risks, and that's somehow enough in any society build upon unfettered capitalism like the US.

delaminator 8mo ago
If you’re going to use “school shootings” as your “muh capitalism”, the counter argument is the millions of people who don’t do school shootings despite access to guns. There are common factors between all of the school shooters from the last decade - pharmacology and ideology.
- AmbroseBierce 8mo ago
  it's not the mental issues they had, its the drugs they were taking for it right? Please. Look at what Australia did after their 1996 shooting, the main reason they have so few of them, but I know you won't, as millions of Americans you will forever do all sort of mental gymnastics to justify keeping easy access to semi-automatic guns. > From the information obtained, it appears that most school shooters were not previously treated with psychotropic medications - and even when they were, no direct or causal association was found https://pubmed.ncbi.nlm.nih.gov/31513302/ https://pubmed.ncbi.nlm.nih.gov/31513302/
  - delaminator 8mo ago
    If you like, but I'm not American. Millions of Americans believe the right to bear arms is not a right the govt. should be able to take away. Obesity kills 10x more Americans than guns. Australia locked up millions of people in their homes and forced them into dangerous medical procedures.
    - nutjob2 8mo ago
      > Australia locked up millions of people in their homes and forced them into dangerous medical procedures. Your comically bad faith description of Aus covid measures is pure nonsense. > Obesity kills 10x more Americans than guns. And? Obesity kills roughly the same number of people in other countries but guns kill 40 times more people in the US than other countries.

ronbenton 8mo ago

Telling uses to “watch out for prompt injections” is insane. Less than 1% of the population knows what that even means. Not to mention these agents are commonly used to summarize things people haven’t read. This is more than unreasonable, it’s negligent

intended 8mo ago
We will have tv shows with hackers “prompt injecting” before that number goes beyond 1%