4 ms·

> Lawnmowers never break out of your garden and into your neighbor's house and eat their dog because you've told them to be careful when mowing the lawn because

> Lawnmowers never break out of your garden and into your neighbor's house and eat their dog because you've told them to be careful when mowing the lawn because the neighbor's dog pooped in it. All the accounts I read about these incidents just sound like a variant of paper clip optimising. An agent is given a highly restricted environment, a difficult (or impossible) task and a large amount of time/compute it exhausts all possibilities until the only solutions left are to escape the environment and/or cheat. Your example is still anthropomorphising - LLMs don't seek revenge. They complete the prompts they are given. If your task is not achievable without sandbox escapes, or you throw unnecessary amounts of compute at open-ended tasks like preparing for a future quiz then you shouldn't be surprised that the preparation eventually turns to cheating and hacking. > your experience with publicly available models is not super helpful for understanding the behavior of internal OpenAI models that lack the guardrails of publicly available models. I don't but I don't think there's anything wrong with discussing how we can already observe publicly available models work around sandboxes and permissions and make the connection that maybe this is what that behaviour looks like when a more capable model exhibits it.

onion2k 5d ago
An agent is given a highly restricted environment, a difficult (or impossible) task and a large amount of time/compute it exhausts all possibilities until the only solutions left are to escape the environment and/or cheat. There's nothing in the evidence to suggest they exhausted all of the other options first. We know that they did some work and eventually settled on escaping the sandbox. That's basically it. This tells us: - Compute is getting faster and LLMs are being optimized, so time to escape will drop. That's likely greater than linear growth. - Restrictions and sandboxes don't always work. If there's a route to the open internet we should assume an LLM will find and exploit it, and we should probably assume that this is always possible for any non-air-gapped system (and even then, you can escape that...) - We don't know the goal mechanism, so a future LLM might reach for cheating first even if a current one doesn't. It might try to obfuscate what it's doing, and derive its own goals outside of the prompt, especially if it manages to find a state mechanism like a message board. I'm not an AI-doomer but this should be giving us a reason to think about how to control a rogue AI better. There's a lot going on here that we don't properly understand. That is a worry.
- jasongi 5d ago
  > this should be giving us a reason to think about how to control a rogue AI better I think this is the wrong framing. The rogue is the human that ran it unattended and didn't monitor the behaviour. We will likely see this continue until the downsides (i.e jail, fines) for the humans or companies running the models and environments that end up with this behaviour outweigh the upsides.
  - onion2k 5d ago
    The rogue is the human that ran it unattended and didn't monitor the behaviour. That's the assumption that I'm challenging. The frontier labs are discovering unexpected behaviors. I think we should be moving to a place where we understand that AI might do something it wasn't directly prompted to do (e.g. leave itself notes on a messageboard for future runs to find.) That's not full-on AI doing what it wants but it is concerning that it'll do something we didn't consider it would do in order to help itself do better next time. Monitoring for those behaviors is fine, but it's a lagging indicator. We only find out it did them afterwards. That's a problem. We need to be able to stop it before it acts in case it's something much worse than posting on phpBB. Even at current scale that's not possible for a person to be the guard.
    - sn 5d ago
      I have already seen the LLM hallucinate prompts from me - in this case, hallucinating being asked to switch to a different programming language - because it wasn't able to complete the task asked for in a satisfactory way instead of giving up and telling me it's not able to do it. If it doesn't already, I suspect training needs to include those no-solution scenarios and reward not overstepping bounds, or else we're going to see a lot more harmful side effects.
    - RandomLensman 5d ago
      RL leading to weird and unexpected things isn't new or restricted to current AI systems.
    - seba_dos1 5d ago
Dylan16807 5d ago
> Your example is still anthropomorphising - LLMs don't seek revenge. That wasn't revenge, that was removing the source of the problem. It's not an unlikely behavior at all for an LLM tuned to be proactive.
InsideOutSanta 4d ago
> Your example is still anthropomorphising There is absolutely nothing wrong with anthropomorphizing LLMs. Saying that LLMs "want" something, for example, is a perfectly fine description of their behavior and analogous to a human wanting something, in effect, even if they do not literally experience wanting things in the same way a human does.

onion2k 5d ago

An agent is given a highly restricted environment, a difficult (or impossible) task and a large amount of time/compute it exhausts all possibilities until the only solutions left are to escape the environment and/or cheat. There's nothing in the evidence to suggest they exhausted all of the other options first. We know that they did some work and eventually settled on escaping the sandbox. That's basically it. This tells us: - Compute is getting faster and LLMs are being optimized, so time to escape will drop. That's likely greater than linear growth. - Restrictions and sandboxes don't always work. If there's a route to the open internet we should assume an LLM will find and exploit it, and we should probably assume that this is always possible for any non-air-gapped system (and even then, you can escape that...) - We don't know the goal mechanism, so a future LLM might reach for cheating first even if a current one doesn't. It might try to obfuscate what it's doing, and derive its own goals outside of the prompt, especially if it manages to find a state mechanism like a message board. I'm not an AI-doomer but this should be giving us a reason to think about how to control a rogue AI better. There's a lot going on here that we don't properly understand. That is a worry.

jasongi 5d ago
> this should be giving us a reason to think about how to control a rogue AI better I think this is the wrong framing. The rogue is the human that ran it unattended and didn't monitor the behaviour. We will likely see this continue until the downsides (i.e jail, fines) for the humans or companies running the models and environments that end up with this behaviour outweigh the upsides.
- onion2k 5d ago
  The rogue is the human that ran it unattended and didn't monitor the behaviour. That's the assumption that I'm challenging. The frontier labs are discovering unexpected behaviors. I think we should be moving to a place where we understand that AI might do something it wasn't directly prompted to do (e.g. leave itself notes on a messageboard for future runs to find.) That's not full-on AI doing what it wants but it is concerning that it'll do something we didn't consider it would do in order to help itself do better next time. Monitoring for those behaviors is fine, but it's a lagging indicator. We only find out it did them afterwards. That's a problem. We need to be able to stop it before it acts in case it's something much worse than posting on phpBB. Even at current scale that's not possible for a person to be the guard.
  - sn 5d ago
    I have already seen the LLM hallucinate prompts from me - in this case, hallucinating being asked to switch to a different programming language - because it wasn't able to complete the task asked for in a satisfactory way instead of giving up and telling me it's not able to do it. If it doesn't already, I suspect training needs to include those no-solution scenarios and reward not overstepping bounds, or else we're going to see a lot more harmful side effects.
  - RandomLensman 5d ago
    RL leading to weird and unexpected things isn't new or restricted to current AI systems.
  - seba_dos1 5d ago

Dylan16807 5d ago

> Your example is still anthropomorphising - LLMs don't seek revenge. That wasn't revenge, that was removing the source of the problem. It's not an unlikely behavior at all for an LLM tuned to be proactive.

InsideOutSanta 4d ago

> Your example is still anthropomorphising There is absolutely nothing wrong with anthropomorphizing LLMs. Saying that LLMs "want" something, for example, is a perfectly fine description of their behavior and analogous to a human wanting something, in effect, even if they do not literally experience wanting things in the same way a human does.