5 ms·
There has been some good research published on this topic of how RLHF, ie aligning to human preferences easily introduces mode collapse and bias into models. Fo
by tehsauce 2y ago
There has been some good research published on this topic of how RLHF, ie aligning to human preferences easily introduces mode collapse and bias into models. For example, with a prompt like: "Choose a random number", the base pretrained model can give relatively random answers, but after fine tuning to produce responses humans like, they become very biased towards responding with numbers like "7" or "42".
- aidos 2y agoWhy is that? Whenever I’m giving examples I almost always use 7, something ending in a 7 or something in the 70s
- Ethee 2y agoVeritasium actually made a video on this concept about a year ago: https://www.youtube.com/watch?v=d6iQrh2TK98 https://www.youtube.com/watch?v=d6iQrh2TK98
- d4mi3n 2y agoMy guess is that we bias towards numbers with cultural or personal significance. 7 is lucky in western cultures and is religiously significant (see https://en.wikipedia.org/wiki/7#Culture https://en.wikipedia.org/wiki/7#Culture). 42 is culturally significant in science fiction, though that's a lot more recent. There are probably other examples, but I imagine the mean converges on numbers with multiple cultural touchpoints.
- Jensson 2y agoI have never heard of 7 being a lucky number in western culture and your link doesn't support that. 3 is a lucky number, 13 is an unlucky number, 7 is nothing to me. So I don't think its that, 7 is still a very common "random number" here even though there is no special cultural significance to it.
- kbenson 2y agoIt's definitely used in slot machines as a lucky number. Which came first I'm not sure (but I suspect from a sibling comment in the same thread it's based on perceived commonality and primeness historically and became "lucky" in the past because of that).
- II2II 2y agoWhile I have never heard of someone referring to 7 as a lucky number, 7 is the most common sum of two rolled dice. So I can see how people would regard it as a lucky number. Along the same lines, I assume that someone who mentions 42 as a random number has at least some interest in science fiction.
- dontseethefnord 2y agoYou must be living under a rock if you’ve never heard of 7 as a lucky number.
- II2II 2y agoI prefer to calculate with numbers, and don't pay much attention to superstitions around them. I don't gamble, nor much pay attention to conversations about gambling, so I pretty much ignore any mention of lucky numbers when such topics arise (aside from knowing that some people have lucky numbers). If you refer being isolated from a particular aspect of life living under a rock, so be it. Though I will point out that I like wide open space. I'm more of an astronomer than a geologist!
- bawolff 2y agoI have never heard of 7 being lucky until this thread. I think you overestimate how cultural diverse western countries are when it comes to small things like this.
- taormina 2y agoHave you heard of Las Vegas? The 777 being the grand prize? Maybe it is not universal to all of western society but I have never before today heard of a culture where 3 was the lucky number. The USA’s culturally lucky number is absolutely 7.
- da_chicken 2y agoThe theory I've heard is that the more prime a number is, the more random it feels. 13 feels more awkward and weird, and it doesn't come up naturally as often as 2 or 3 do in everyday life. It's rare, so it must be more random! I'll give you the most random number I can think of! People tend to avoid extremes, too. If you ask for a number between 1 and 10, people tend to pick something in the middle. Somehow, the ordinal values of the range seem less likely. Additionally, people tend to avoid numbers that are in other ranges. Ask for a number from 1 to 100, and it just feels wrong to pick a number between 1 and 10. They asked for a number between 1 and 100. Not this much smaller range. You don't want to give them a number they can't use. There must be a reason they said 100. I wonder if the human RNG would improve if we started asking for numbers between 21 and 114.
- foota 2y agoOkay, this is a nitpick, but I don't think ordinal can be used in that way. "Somehow, the ordinal values of the range seem less likely". I'd probably go with extremes of the range? Or endpoints?
- da_chicken 2y agoNope I just mixed up a rephrase. I originally said "ordinal extremes" and meant to say "extreme values". I replaced the wrong word.
- smohare 2y ago[dead]
- thfuran 2y agoPeople also tend to botch random sequences by trying to avoid repetition or patterns.
- d0liver 2y agoI like prime numbers. Non-primes always feel like they're about to fall apart on me.
- p1necone 2y ago1 and 10 are on the boundary, that's not random so those are out. 5 is exactly halfway, that's not random enough either, that's out. 2, 4, 6, 8 are even and even numbers are round and friendly and comfortable, those are out too. 9 feels too close to the boundary, it's out. That leaves 3 and 7, and 7 is more than 3 so it's got more room for randomness in it right? Therefore 7 is the most random number between 1 and 10.
- HappMacDonald 2y agoAlso because humans are biased towards viewing prime numbers as more counterintuitive and thus more unpredictable.
- wruza 2y agoLast time I hallway tested it, people couldn’t tell what prime numbers are, and to my surprise even the ones with tech/math-y background forgot it. My results were something 1.5/10 (ages 30+-5) and I didn’t go to cabinets where I knew there are zero chances.
- HappMacDonald 2y agoBut there's a difference between "knowing what the formal definition is" and "having a feeling that a number is somehow unique due to it's indivisibility".
- LoganDark 2y agoThat's all well and good, but 4 is actually the most random number, because it was chosen by fair dice roll.
- moffkalast 2y agoIt's very funny that people hold the autoregressive nature of LLMs against them, while being far more hardline autoregressive themselves. It's just not consciously obvious.
- absolutelastone 2y agoI think people tend to just not understand what autoregressive methods are capable of doing generally (i.e., basically anything an alternative method can do), and worse they sort of mentally view it as equivalent to a context length of 1.
- antihipocrat 2y agoI wonder whether we hold LLMs to a different standard because we have a long term reinforced expectation for a computer to produce an exact result? One of my first teachers said to me that a computer won't ever output anything wrong, it will produce a result according to the instructions it was given. LLMs do follow this principle as well, it's just that when we are assessing the quality of output we are incorrectly comparing it to the deterministic alternative, and this isn't really a valid comparison.
- robwwilliams 2y agoI assume 42 is a joke from deep history and The Hitchhiker’s Guide. Pretty amusing to read the Wikipedia entry: https://en.wikipedia.org/wiki/42_(number) https://en.wikipedia.org/wiki/42_(number)
- sedatk 2y agoDouglas Adams picked 42 randomly though. :)
- robertlagrant 2y agoNot at all. It was derived mathematically from the Question: What do you get if you multiply six by nine?
- HappMacDonald 2y agoThat's not the question, though. Everybody knows that the question is the one posed to Mister Turtle and Mister Owl which neither of them can find the answer to.
- sedatk 2y agoI stand corrected in base 13.
- eterm 2y agoIt was just a joke, and doubly so the fact it "works" in base 13. It was written as a joke in fairly ramshackle radio play. He had no idea at the time of writing it that the joke would connect so well and become it's own "thing" and dominate discourse of the radio series and novels to come. It's not a joke about numbers, it's a linguistical joke, that works well on radio, something that HHGTG is stuffed full of. https://scifi.stackexchange.com/questions/12229/how-did-douglas-adams-choose-the-ultimate-question https://scifi.stackexchange.com/questions/12229/how-did-doug...
- deleted 2y ago[deleted]
- thechao 2y agoWhich is weird, because I thought we'd all agreed that the random number was 4? https://xkcd.com/221/ https://xkcd.com/221/
- mynameismon 2y agoCan you share any links about this?
- Shorel 2y agoThey choose 37 =)