6 ms·
So, this is the cause of the problem.... People take an intro to LLMs course, follow happily along, and don't realize there is more to it than the next token pr
by danielmarkbruce 12d ago
So, this is the cause of the problem.... People take an intro to LLMs course, follow happily along, and don't realize there is more to it than the next token prediction. And those courses teach how LLMs were built in 2017-2020 maybe. Then RL got added to the mix. The current models really are very different to the models from then - everything that is now considered "post-training" isn't doing next token prediction.
- deleted 11d ago[deleted]
- doc_ick 11d agoPlease feel free to cite sources then, otherwise I see no relevancy from you.
- danielmarkbruce 11d agoNathan Lambert wrote a good book recently, and he and his team wrote the paper below about Tulu 3 (Allen Institute). Both are good reads. https://arxiv.org/pdf/2411.15124 https://arxiv.org/pdf/2411.15124
- doc_ick 11d agoThank you for providing an arxiv! An aside, I finally do appreciate single column format now, makes it easier to convert to epub.
- danielmarkbruce 11d agoWhen you are done with the section on RLVR, consider whether the model is predicting tokens, or making moves. There is a reason the word "policy" is used in RL.
- doc_ick 10d agoWould still say it’s a token predictor, a fancy one though. I suppose we can agree to disagree.