danielmarkbruce

6 ms·

31.
by danielmarkbruce 11d ago
A guess at what though? One guesses at truths they don't know, or events that haven't happened yet. What is the model guessing?
32.
by danielmarkbruce 11d ago
So, this is the cause of the problem.... People take an intro to LLMs course, follow happily along, and don't realize there is more to it than the next token prediction. And those courses teach how LLMs were built in 2017-2020 maybe. T
33.
by danielmarkbruce 11d ago
Assuming you are saying that RL is changing the model from doing one thing to another, yes. RL is changing the nature of the model.
34.
by danielmarkbruce 12d ago
The discussion is basically: what is a model trying to do? One may reasonably assert it isn't trying to do anything. But, in practice, if you give it an objective function and optimize it, the model is basically trained to "do&quo
35.
by danielmarkbruce 12d ago
No one is arguing about the architecture of the model. It's the objective function and optimizer.
36.
by danielmarkbruce 12d ago
This is pedantic, but, actually RL has improved the quality of sentence construction in LLMs quite dramatically... And once you do some RL on that model, it aint a next token prediction machine any longer.
37.
by danielmarkbruce 12d ago
Probably the easiest way to describe an LLM that it's a policy. There is a reason that word has stuck in RL. And it's not just RLVR. RLHF has been going on for years and years. LLMs have not been "next token predictors"
38.
by danielmarkbruce 12d ago
It's not an estimation of something. It's a policy.
39.
by danielmarkbruce 12d ago
You are conflating "half built" with "a piece of a system". The model weights change as the model goes through the training process. They aren't stored after pre-training is done and other weights are put somewhere
40.
by danielmarkbruce 12d ago
Yes, it was. Nobody says a system or product works a certain way and means the system while it's half built. "Bridges drop cars in the water!". Right. You aren't in this field. You are clearly wrong and just can't h
41.
by danielmarkbruce 12d ago
Your mistake was assuming people would be bothered to understand the details of how things work. Most people are lazy and don't know the details of how anything works.
42.
by danielmarkbruce 12d ago
They aren't predicting the next token. It's quite literally not a prediction.
43.
by danielmarkbruce 12d ago
All modern LLMs that actually get used go through post-training. The finished product is something which has been through post training. So they are not next token prediction machines.
44.
by danielmarkbruce 12d ago
No, it's not philsophical. Because if you optimize to predict, you are doing something different to optimizing for a reward. It's a different process - different objective function, different optimization, different set up.
45.
by danielmarkbruce 12d ago
Read through the article and comments. You are talking solely about pre-training. I'm talking about post training. Respectfully, you are miles out of your depth. GPT-2 didn't use any reinforcement learning and is often given as a
46.
by danielmarkbruce 12d ago
Nope. This isn't right.
47.
by danielmarkbruce 12d ago
Yup, you are mostly right. I guess the people in my camp find the "it's just a next token predictor" stupid in that it's like saying "it's just a bunch of carbon and hydrogen", but it's also one of th
48.
by danielmarkbruce 12d ago
Lol, sure, just read a blog post and you'll understand how a car works....It's very simple....
49.
by danielmarkbruce 12d ago
Predict implies you don't control a situation. That's the difference.
50.
by danielmarkbruce 12d ago
People have and are trying things. Lots and lots of things. They just don't go around promoting failed ideas.
51.
by danielmarkbruce 12d ago
The fight is about the predictor language in some cases. Because it's only a trivial difference to those who don't understand the details of how these things are made. In pre-training the model really is trained to predict the nex
52.
by danielmarkbruce 12d ago
It's not a prediction of the next move though, and that is the point. It's a prediction of what will happen if you make that move. So, it's not a next move predictor. It's a game result predictor.
53.
by danielmarkbruce 12d ago
Emitting and predicting are different things though. Prediction implies there is some "truth" or event or something that you can test against. Prediction implies the model just learns from existing text, and optimizes to predict t
54.
by danielmarkbruce 12d ago
If you are going to say "literally", then what is your literal definition for the word "prediction" ?
55.
by danielmarkbruce 12d ago
If you haven't built one, and don't understand how they work, why comment?
56.
by danielmarkbruce 12d ago
Respectfully, go build one, including doing RLHF and RLVR. Those phases generate lots of tokens, then get scored on the entirety of the output, then optimize based on a scoring of that output. It doesn't check a "prediction"
57.
by danielmarkbruce 12d ago
Nope, it doesn't. No logic required, you can just build an LLM yourself, including post training. You'll see that predicting the next token isn't something the model does or is optimized for in RLHF or RLVR. You can hand wave
58.
by danielmarkbruce 12d ago
Post train a model, you'll be able to determine it is not.
59.
by danielmarkbruce 12d ago
There isn't a truth to test against. If I predict the next word in a sequence is "sat", we can check against the sequence. If I predict the roll of a die will be 4, we can check against it. Whether i give 100% or give a proba
60.
by danielmarkbruce 12d ago
There is no truth for RLHF or RLVR. You can't predict against something if you can't check against the truth. It's not pedantry. The objective function changes. The optimization changes. THese are real things when training a

More ›