6 ms·
You are conflating "half built" with "a piece of a system". The model weights change as the model goes through the training process. They aren't stored after p
by danielmarkbruce 12d ago
You are conflating "half built" with "a piece of a system".
The model weights change as the model goes through the training process. They aren't stored after pre-training is done and other weights are put somewhere else. It's more like pottery - the thing changes. It's not correct to say something is soft and malleable because it once was.
- deleted 12d ago[deleted]
- deleted 12d ago[deleted]
- dijksterhuis 12d ago> The model weights change as the model goes through the training process. Yes. They do. You are absolutely right about that. But the model architecture doesn't change as a result of the training process. A piston doesn't suddenly turn into a digital watch as a result of tuning an engine. Similarly, the transformer part of a GPT model doesn't suddenly turn into something else as a result of optimizing a loss function. --- i've got other stuff to do, so i'm stopping here.
- danielmarkbruce 12d agoNo one is arguing about the architecture of the model. It's the objective function and optimizer.
- doc_ick 12d agoJust skimming through here but I think you have the wrong ideas with llms, I’d recommend Andrew Ngs course (correct me if you’ve already seen it or something similar).
- danielmarkbruce 12d agoSo, this is the cause of the problem.... People take an intro to LLMs course, follow happily along, and don't realize there is more to it than the next token prediction. And those courses teach how LLMs were built in 2017-2020 maybe. Then RL got added to the mix. The current models really are very different to the models from then - everything that is now considered "post-training" isn't doing next token prediction.
- deleted 11d ago[deleted]
- doc_ick 11d agoPlease feel free to cite sources then, otherwise I see no relevancy from you.
- danielmarkbruce 11d agoNathan Lambert wrote a good book recently, and he and his team wrote the paper below about Tulu 3 (Allen Institute). Both are good reads. https://arxiv.org/pdf/2411.15124 https://arxiv.org/pdf/2411.15124
- doc_ick 11d agoThank you for providing an arxiv! An aside, I finally do appreciate single column format now, makes it easier to convert to epub.
- danielmarkbruce 11d agoWhen you are done with the section on RLVR, consider whether the model is predicting tokens, or making moves. There is a reason the word "policy" is used in RL.
- doc_ick 10d agoWould still say it’s a token predictor, a fancy one though. I suppose we can agree to disagree.
- MrScruff 12d agoI am not an expert, but I do understand the distinction that is being made here. It makes sense to describe the result of pre-training as a ‘next token’ predictor as that’s what it’s been trained to do, not because it’s an autoregressive architecture that produces tokens one at a time. If this base is then trained using RL towards a different objective (maths and coding), the model becomes fundamentally a different thing and the recent models are clear evidence of that, regardless of they fact they remain autoregressive.
- skydhash 11d agoDoes the architecture change or the weight change? If you modify an engine to increase it’s output by adding sensors and an ECU, you don’t change the fact that is powered by gas. If you use RL to increase the accuracy, it’s still a next token prediction, just more accurate.
- deleted 11d ago[deleted]
- Dylan16807 12d agoYou're using the fact the both parts of training affect the same weights to support your argument that they're making the system do something fundamentally different after RL?
- danielmarkbruce 12d agoAssuming you are saying that RL is changing the model from doing one thing to another, yes. RL is changing the nature of the model.