7 ms·

If you don’t consider the difference in kind between a human vulnerability and an automated vulnerability that derives from the essentially unlimited capacity o

If you don’t consider the difference in kind between a human vulnerability and an automated vulnerability that derives from the essentially unlimited capacity of the latter to scale, your comment makes a lot of sense. If you do consider that, the argument becomes irrelevant and deeply misleading

TeMPOraL 3y ago
The difference you're talking about is only in the fact that humans don't scale like computer code. If humans were to scale like computer code, you'd still find the "vulnerability" unfixable.
- aidenn0 3y ago
  I think what GP (and I) are talking about is that social engineering is limited in scope because humans don't scale like computer code. A theoretical AGI (and LLMs) do scale like computer code. To use an admittedly extreme example: The difference between drawing some fake lines on the road and crashing 1 or 2 cars and having all self-driving cars on the road swerve simultaneously is not just a quantitative difference.
- ethanbond 3y ago
  Right, but humans don’t scale that way, so the threat is completely different. This is like saying a nuclear weapon accident is not that scary because you can also have a microwave malfunction and catch on fire. Sure you can —- but the fact it’s not a nuke is highly relevant.
  - TeMPOraL 3y ago
    No, I'm saying that securing against "prompt injection" is like saying you want to eliminate fission from physics, because you're worried about nukes. That's not how this reality works. Nuclear fission is what happens when certain conditions are met. You're worried about nukes? Stop playing with nukes. I'm not saying they aren't dangerous - I'm saying that you can't make them safer by "eliminating fission", as it makes no physical sense whatsoever. Much like "securing against prompt injections" in language models, or a GAI, or in humans.
    - ethanbond 3y ago
      > Sure, current bleed of LLMs is badly vulnerable to some trivial prompt injections - but I think a good analogy would be a 4 year old kid. This reads like you’re trying to say “don’t worry about it, humans are vulnerable too and it’s threatening the way a 4 year old child is” not “correct, we cannot prevent nuclear explosions given that we have fission and yes we’re on track to putting fission devices into every single internet-connected household on the planet.”
      - lanstin 3y ago
        There is a reason humans with security clearances can’t just have an arbitrary large number of interactions with foreign nationals, or that good interrogators say they can always get info from people if they talk enough m
      - TeMPOraL 3y ago
        I'm saying "stop trying to solve the problem of consumer market IoT fission bombs by trying to remove fission from physics - this just can't possibly work, and it takes special confusion to even think it might; instead, focus on the 'consumer-market', 'IoT' and 'bomb' parts". "Prompt injection" is a vulnerability of generic minds in the same sense "fission" is a vulnerability of atoms.
- danShumway 3y ago
  But that difference is a big part of why this matters. That this might be unfixable is not a strong argument for moving forward anyway, if anything it should prompt us to take a step backwards and consider if general intelligence systems are well suited for scalable tasks in the first place. There are ways to build AIs that don't have these problems specifically because their intelligence is limited to a specific task and thus they don't have a bunch of additional attack vectors literally baked into them. But the attitude from a lot of companies I'm seeing online is "this might be impossible to fix, so you can't expect us to hold off releasing just because it's vulnerable." I don't understand that. If this is genuinely impossible to fix, that has implications. Because the whole point with AI is to make things that are scalable. It matters that the security be better than the non-scalable system. If it can't be better, then we need to take a step back and ask if LLMs are the right approach.
  - hammyhavoc 3y ago
    It's the blockchain and NFT hype train all over again. Shoehorning it into places it doesn't belong, bad implementations to boot, and actually making things less performant, less secure, and more expensive in the process.
  - TeMPOraL 3y ago
    I guess we are talking past each other. I agree that there are many things we can and should do to improve the safety of integrating ML tools into our lives. I agree that there are unique challenges here, such as scaling, creating new dangers that will require new methods of mitigation. I disagree that "prompt injection" is a meaningful category of vulnerabilities to talk about, and that it is fixable in LLMs or other comparably general systems. I've argued before that "prompt engineering" is a bad term, granting connotations to precision and care to a task that's anything but. "Prompt injection", however, is IMO a dangerous term, because it confuses people into thinking that it's something like SQL injection or XSS, and thus solvable by better input handling - where in fact, it is very different and fundamentally not solvable this way (or at all).
    - danShumway 3y ago
noduerme 3y ago
This needs to be hammered into people's understanding of the danger of LLMs at every opportunity. Enough of the general population considers things like Twitter bots to have scaled to a dangerous point of polluting the information ecosystem. The scalability and flexibility of LLMs in germinating chaos is orders of magnitude beyond anything we've yet seen. An example I use for people is the Bernstein Bears effect. Imagine you wake up tomorrow and all your digital devices have no reference to 9/11. You ask Bing and Google and they insist you must be wrong, nothing like that ever happened. You talk to other people who remember it clearly but it seems you've lost control of reality; now imagine that type of gaslighting about "nothing happening" while the lights go out all over the world and you have some sense of what scale the larger of these systems are operating at.
- eru 3y ago
  > Enough of the general population considers things like Twitter bots to have scaled to a dangerous point of polluting the information ecosystem. It was always a good idea to ignore the cesspool that is Twitter. No matter whether we are talking about bots or lynch mobs. Btw, I think you mean Berenstain Bears.
  - rurp 3y ago
    Twitter is just one example though, this problem is going to affect every single online community. If the LLM bull case is correct, the internet is going to be absolutely flooded with sophisticated misinformation.
    - noduerme 3y ago
      Sophisticated being key. Quantity * quality almost indiscernible from mediocre human input. Currently we tend to understand bad information on the stream as a function where quality is linear and quantity is exponential, and individuals or human filters can still identify reject the lower 99% as spam. Every point closer on the graph the quality comes to resemble human-made content represents an exponential degree of further confusion as to base facts. This isn't even considering whether AI develops its own will to conduct confusion ops; as a tool for bad actors it's already there, but that says nothing of the scale it could operate at eventually. The sophistication of the misinformation is exactly the point: That's the mass multiplier, not the volume. [edit] an interesting case could be made that the general demand for opinionated information and the individual capacity to imbibe and adjudicate the factuality of the input was overrun some years ago already... and that all endeavors at misinformation since then have been fighting for shares of an information space that was already essentially capped by the attention-demand. In that paradigm, all social networks have fought a zero-sum game, and LLMs are just a new weapon for market share in an inflationary environment where all information propagated is less valuable as the volume increases and consumption remains static. But I think this is the least worrisome of their abilities.

TeMPOraL 3y ago

The difference you're talking about is only in the fact that humans don't scale like computer code. If humans were to scale like computer code, you'd still find the "vulnerability" unfixable.

aidenn0 3y ago
I think what GP (and I) are talking about is that social engineering is limited in scope because humans don't scale like computer code. A theoretical AGI (and LLMs) do scale like computer code. To use an admittedly extreme example: The difference between drawing some fake lines on the road and crashing 1 or 2 cars and having all self-driving cars on the road swerve simultaneously is not just a quantitative difference.
ethanbond 3y ago
Right, but humans don’t scale that way, so the threat is completely different. This is like saying a nuclear weapon accident is not that scary because you can also have a microwave malfunction and catch on fire. Sure you can —- but the fact it’s not a nuke is highly relevant.
- TeMPOraL 3y ago
  No, I'm saying that securing against "prompt injection" is like saying you want to eliminate fission from physics, because you're worried about nukes. That's not how this reality works. Nuclear fission is what happens when certain conditions are met. You're worried about nukes? Stop playing with nukes. I'm not saying they aren't dangerous - I'm saying that you can't make them safer by "eliminating fission", as it makes no physical sense whatsoever. Much like "securing against prompt injections" in language models, or a GAI, or in humans.
  - ethanbond 3y ago
    > Sure, current bleed of LLMs is badly vulnerable to some trivial prompt injections - but I think a good analogy would be a 4 year old kid. This reads like you’re trying to say “don’t worry about it, humans are vulnerable too and it’s threatening the way a 4 year old child is” not “correct, we cannot prevent nuclear explosions given that we have fission and yes we’re on track to putting fission devices into every single internet-connected household on the planet.”
    - lanstin 3y ago
      There is a reason humans with security clearances can’t just have an arbitrary large number of interactions with foreign nationals, or that good interrogators say they can always get info from people if they talk enough m
    - TeMPOraL 3y ago
      I'm saying "stop trying to solve the problem of consumer market IoT fission bombs by trying to remove fission from physics - this just can't possibly work, and it takes special confusion to even think it might; instead, focus on the 'consumer-market', 'IoT' and 'bomb' parts". "Prompt injection" is a vulnerability of generic minds in the same sense "fission" is a vulnerability of atoms.
danShumway 3y ago
But that difference is a big part of why this matters. That this might be unfixable is not a strong argument for moving forward anyway, if anything it should prompt us to take a step backwards and consider if general intelligence systems are well suited for scalable tasks in the first place. There are ways to build AIs that don't have these problems specifically because their intelligence is limited to a specific task and thus they don't have a bunch of additional attack vectors literally baked into them. But the attitude from a lot of companies I'm seeing online is "this might be impossible to fix, so you can't expect us to hold off releasing just because it's vulnerable." I don't understand that. If this is genuinely impossible to fix, that has implications. Because the whole point with AI is to make things that are scalable. It matters that the security be better than the non-scalable system. If it can't be better, then we need to take a step back and ask if LLMs are the right approach.
- hammyhavoc 3y ago
  It's the blockchain and NFT hype train all over again. Shoehorning it into places it doesn't belong, bad implementations to boot, and actually making things less performant, less secure, and more expensive in the process.
- TeMPOraL 3y ago
  I guess we are talking past each other. I agree that there are many things we can and should do to improve the safety of integrating ML tools into our lives. I agree that there are unique challenges here, such as scaling, creating new dangers that will require new methods of mitigation. I disagree that "prompt injection" is a meaningful category of vulnerabilities to talk about, and that it is fixable in LLMs or other comparably general systems. I've argued before that "prompt engineering" is a bad term, granting connotations to precision and care to a task that's anything but. "Prompt injection", however, is IMO a dangerous term, because it confuses people into thinking that it's something like SQL injection or XSS, and thus solvable by better input handling - where in fact, it is very different and fundamentally not solvable this way (or at all).
  - danShumway 3y ago

noduerme 3y ago

This needs to be hammered into people's understanding of the danger of LLMs at every opportunity. Enough of the general population considers things like Twitter bots to have scaled to a dangerous point of polluting the information ecosystem. The scalability and flexibility of LLMs in germinating chaos is orders of magnitude beyond anything we've yet seen. An example I use for people is the Bernstein Bears effect. Imagine you wake up tomorrow and all your digital devices have no reference to 9/11. You ask Bing and Google and they insist you must be wrong, nothing like that ever happened. You talk to other people who remember it clearly but it seems you've lost control of reality; now imagine that type of gaslighting about "nothing happening" while the lights go out all over the world and you have some sense of what scale the larger of these systems are operating at.

eru 3y ago
> Enough of the general population considers things like Twitter bots to have scaled to a dangerous point of polluting the information ecosystem. It was always a good idea to ignore the cesspool that is Twitter. No matter whether we are talking about bots or lynch mobs. Btw, I think you mean Berenstain Bears.
- rurp 3y ago
  Twitter is just one example though, this problem is going to affect every single online community. If the LLM bull case is correct, the internet is going to be absolutely flooded with sophisticated misinformation.
  - noduerme 3y ago
    Sophisticated being key. Quantity * quality almost indiscernible from mediocre human input. Currently we tend to understand bad information on the stream as a function where quality is linear and quantity is exponential, and individuals or human filters can still identify reject the lower 99% as spam. Every point closer on the graph the quality comes to resemble human-made content represents an exponential degree of further confusion as to base facts. This isn't even considering whether AI develops its own will to conduct confusion ops; as a tool for bad actors it's already there, but that says nothing of the scale it could operate at eventually. The sophistication of the misinformation is exactly the point: That's the mass multiplier, not the volume. [edit] an interesting case could be made that the general demand for opinionated information and the individual capacity to imbibe and adjudicate the factuality of the input was overrun some years ago already... and that all endeavors at misinformation since then have been fighting for shares of an information space that was already essentially capped by the attention-demand. In that paradigm, all social networks have fought a zero-sum game, and LLMs are just a new weapon for market share in an inflationary environment where all information propagated is less valuable as the volume increases and consumption remains static. But I think this is the least worrisome of their abilities.