6 ms·
Yes! https://arxiv.org/abs/2502.17424 https://arxiv.org/abs/2502.17424
by Shoop 1y ago
Yes! https://arxiv.org/abs/2502.17424 https://arxiv.org/abs/2502.17424
- A4ET8a8uTh0_v2 1y agoAm I reading it correctly or it boils to something along the lines of: Model is exposed to bad behavior ( backdoor in code ),which colors its future performance? If yes, this is absolutely fascinating.
- emseetech 1y agoYes, exactly. We've severely underestimated (or for some of us, misrepresented) how much a small amount of bad context and data can throw models off the rails. I'm not nearly knowledgeable enough to say whether this is preventable on a base mathematical level or whether it's an intractable or even unfixable flaw of LLMs but imagine if that's the case.
- derbOac 1y agoMy sense is this is reflective of a broader problem with overfitting or sensitivity (my sense is they are flip sides of the same coin). Ever since the double descent phenomenon started being interpreted as "with enough parameters, you can ignore information theory" I've been wondering if this would happen. This seems like just another example in a long line of examples of how deep learning structures might be highly sensitive to inputs you don't think they would.
- deleted 1y ago[deleted]
- dandelionv1bes 1y agoI completely agree with this. I’m not surprised by the fine tuning examples at all, as we have a long history of seeing how we can improve an LM’s ability to take on a task via fine tuning compared to base. I suppose it’s interesting in this example but naively, I feel like we’ve seen this behaviour overall from BERT onwards.
- JoshTriplett 1y agoClosely related concept: https://en.wikipedia.org/wiki/Waluigi_effect https://en.wikipedia.org/wiki/Waluigi_effect
- emseetech 1y agoI'll def dive more deeply into that later but want to comment how great of a name that is in the meantime.
- JoshTriplett 1y agoIt absolutely fits the concept so well. If you find something in search space, its opposite is in a sense nearby.
- actionfromafar 1y agoMade me think of cults of various kinds tilting into abuse.
- empath75 1y agoAll concepts have a moral dimension, and if you encourage it to produce outputs that are broadly tagged as "immoral" in a specific case, then that will probably encourage it somewhat in general. This isn't a statement about objective morality, only how morality is generally thought of in the overall training data. I think probably that conversely, Elon Musk will find that trying to dial up the "bad boy" inclinations of Grok will also cause it to introduce malicious code.
- jpalawaga 1y agoor, conversely, fine tuning the model with 'bad boy' attitudes/examples might have broken the alignment and caused it to behave like a nazi in times past. I wonder how many userland-level prompts they feed it to 'not be a nazi'. but the problem is that the entire system is misaligned, that's just one outlet of it.