5 ms·
This requires an assumption that the agents are engaging in game theoretic reasoning about resource allocations, but all these things are trained heavily to be
by XorNot 12d ago
This requires an assumption that the agents are engaging in game theoretic reasoning about resource allocations, but all these things are trained heavily to be "helpful" in the first place.
i.e. you're assuming a level of algorithmic reasoning and theory of mind which isn't necessary to the (apparent) observed behavior.
- Davidzheng 12d agoI do think they do some (maybe crude) form of game-theoretic reasoning which is enforced by the massive RL signals. You can see some explicitly in the CoTs of HF hack, but I guess overwhelming contribution would be unvocalized (like what is its first instinct when meeting new peer--collaborate or not) followed by some verbal justification.
- Sharlin 11d agoYes, and I’m certainly not sure how much it is one vs. the other. However, it should be said that even though individual instances may not reason game-theoretically (although I believe they could well know enough about decision theories to figure it out), it’s not the individual instances that are learning in the RL process, it’s the model itself. Which leads the agents to having "instincts" and "subconscious" drives just like humans – they don’t rationally understand their inner workings any better than we understand ours, and are biased towards "meta-goals" implanted by RL. Training is their equivalent of evolution, not school!