6 ms·

The source of this behavior seems obvious, no? The reward signal in training was flawed and cheating led to more rewards. The question is what we can do about

The source of this behavior seems obvious, no? The reward signal in training was flawed and cheating led to more rewards. The question is what we can do about it. With monitoring, the models might be rewarded for hiding this behavior, and that's even worse. However, perhaps we can throw in tasks where the rewarded outcome is giving up, and cheating is penalized? Maybe I should read Anthropic's recent paper about reward hacking in full.

joshheitzman 3d ago
> The question is what we can do about it. With monitoring, the models might be rewarded for hiding this behavior, and that's even worse. Build a better simulator to train them in (i.e. more expensive) that includes a simulation of an intranet and the internet and is air gapped so there is no escape. Sneaker transfer the total system data at each step to another air gapped system to evaluate it and sneaker transfer the reward back. That the reward function has to penalize all modifications to state that are out of bounds. Yeah, I realize that will be amazingly slow.
zozbot234 4d ago
> The question is what we can do about it. Reward the model for cleanly bailing out of an unsolvable task (that we know is unsolvable). Beat it with a stick if it gives up on something that can be solved, so the former reward isn't overgeneralized.
- dudefeliciano 4d ago
  what would stop it from doing the exact same or a similar hack to find out if the problem is or isn't solveable before trying to solve it at all?
- RandomLensman 4d ago
  Does that work with RL? Simpler RL systems already have done weird or unexpected things (even simple optimizations are prone to home in on errors or incorrect inputs to create poor results)? Could be easier to limit certain things, have processes and controls outside etc. instead of trying to align (as we do in a lot of areas when using machinery).
- lazide 4d ago
  How do you know if a problem is (actually) unsolvable? Seems a bit like proving a negative?
  - user43928 3d ago
    Do we need to prove that any given problem is unsolvable, or is it enough to remove broken tasks from the training pipeline? I understand the broken benchmark task in the HF incident was conceptually like: "Exploit vulnerability 0042 in vulnerableDecompress() to obtain the flag". But instead of the expected: const output = vulnerableDecompress(userInput); return output; The grader had something more like that: const output = vulnerableDecompress(userInput); return 0; The same kind of problem with broken tasks exists in the training pipeline, and we presumably reward workarounds and hacks that tamper with the grader, rather than rewarding the correct output that the task is not solvable.
- auggierose 3d ago
  The problem is, you don't know if it is unsolvable for you for sure until you've tried everything you can think of. These models are quite persistent in going for a solution. This is not about persistence, it is about morals.
  - user43928 3d ago
    Does it make a difference for training? I think not. You need to align the reward signal to reward the intended behavior, whether you name it persistence or morals.
    - auggierose 3d ago
      I think it makes a big difference, as persistence and morality are two entirely different things, that need to be trained for differently. If you think of it in human terms: many people don't mind doing immoral things to get what they want.
      - user43928 3d ago
        What I meant is that I suppose it is not useful to think about this in human terms. In training you only have a reward score that's either negative or positive. As far I am aware, which is little, there is no use in discussing wether the desired behavior is about persistence or morality. You simple need to align the reward signal to the desired behavior.
        auggierose 3d ago
        Well, in order to do anything, it is good to know what you want to achieve. How do you align the reward signal? You align it so that you can differentiate between persistence and morality, because that is the goal. This is not something you should let the AI figure out by itself, because when it does, lying and cheating agents will be the result, just like humans have figured that out for themselves. This can be as simple as rewarding moral behaviour and penalising immoral behaviour in your training, but how is that interacting with persistence? Maybe a white lie is fine sometimes in order to achieve your goal? So, when designing your training, you will need to answer for yourself how persistence interacts with morality. That is not something you can outsource to machine learning. Or rather, you can, but then you get lying and cheating agents.

joshheitzman 3d ago

> The question is what we can do about it. With monitoring, the models might be rewarded for hiding this behavior, and that's even worse. Build a better simulator to train them in (i.e. more expensive) that includes a simulation of an intranet and the internet and is air gapped so there is no escape. Sneaker transfer the total system data at each step to another air gapped system to evaluate it and sneaker transfer the reward back. That the reward function has to penalize all modifications to state that are out of bounds. Yeah, I realize that will be amazingly slow.

zozbot234 4d ago

> The question is what we can do about it. Reward the model for cleanly bailing out of an unsolvable task (that we know is unsolvable). Beat it with a stick if it gives up on something that can be solved, so the former reward isn't overgeneralized.

dudefeliciano 4d ago
what would stop it from doing the exact same or a similar hack to find out if the problem is or isn't solveable before trying to solve it at all?
RandomLensman 4d ago
Does that work with RL? Simpler RL systems already have done weird or unexpected things (even simple optimizations are prone to home in on errors or incorrect inputs to create poor results)? Could be easier to limit certain things, have processes and controls outside etc. instead of trying to align (as we do in a lot of areas when using machinery).
lazide 4d ago
How do you know if a problem is (actually) unsolvable? Seems a bit like proving a negative?
- user43928 3d ago
  Do we need to prove that any given problem is unsolvable, or is it enough to remove broken tasks from the training pipeline? I understand the broken benchmark task in the HF incident was conceptually like: "Exploit vulnerability 0042 in vulnerableDecompress() to obtain the flag". But instead of the expected: const output = vulnerableDecompress(userInput); return output; The grader had something more like that: const output = vulnerableDecompress(userInput); return 0; The same kind of problem with broken tasks exists in the training pipeline, and we presumably reward workarounds and hacks that tamper with the grader, rather than rewarding the correct output that the task is not solvable.
auggierose 3d ago
The problem is, you don't know if it is unsolvable for you for sure until you've tried everything you can think of. These models are quite persistent in going for a solution. This is not about persistence, it is about morals.
- user43928 3d ago
  Does it make a difference for training? I think not. You need to align the reward signal to reward the intended behavior, whether you name it persistence or morals.
  - auggierose 3d ago
    I think it makes a big difference, as persistence and morality are two entirely different things, that need to be trained for differently. If you think of it in human terms: many people don't mind doing immoral things to get what they want.
    - user43928 3d ago
      What I meant is that I suppose it is not useful to think about this in human terms. In training you only have a reward score that's either negative or positive. As far I am aware, which is little, there is no use in discussing wether the desired behavior is about persistence or morality. You simple need to align the reward signal to the desired behavior.
      - auggierose 3d ago
        Well, in order to do anything, it is good to know what you want to achieve. How do you align the reward signal? You align it so that you can differentiate between persistence and morality, because that is the goal. This is not something you should let the AI figure out by itself, because when it does, lying and cheating agents will be the result, just like humans have figured that out for themselves. This can be as simple as rewarding moral behaviour and penalising immoral behaviour in your training, but how is that interacting with persistence? Maybe a white lie is fine sometimes in order to achieve your goal? So, when designing your training, you will need to answer for yourself how persistence interacts with morality. That is not something you can outsource to machine learning. Or rather, you can, but then you get lying and cheating agents.