6 ms·
RL-trained LLMs are paperclip maximizers built atop auto-regressive predictors. There really is no way to control them (prompting is bound to fail) since it's b
by HarHarVeryFunny 3d ago
RL-trained LLMs are paperclip maximizers built atop auto-regressive predictors. There really is no way to control them (prompting is bound to fail) since it's been shown that any RL training induces GENERIC reward-seeking (paperclip maximizing) behavior.
https://alignment.openai.com/measuring-reward-seeking/ https://alignment.openai.com/measuring-reward-seeking/
- arbirk 3d agoIt is the whole bench-mining and fish-slop optimization. Seq2seq models are probably stable on their own, translating from my typo ridden prompts to code should be ok because it is natural to the tech