5 ms·
I don't think it's only using sparse rewards because of the format rewards. The training recipe is pretty comprehensive and involves multiple stages.[1] The pap
by serialx 2y ago
I don't think it's only using sparse rewards because of the format rewards. The training recipe is pretty comprehensive and involves multiple stages.[1] The paper mentions that when only using the RL technique, the output is often not suitable for reading. (Language mixing, etc) That feels like a AlphaZero moment for LLMs?
[1]: https://www.reddit.com/r/LocalLLaMA/comments/1i8rujw/notes_on_deepseek_r1_just_how_good_it_is_compared/ https://www.reddit.com/r/LocalLLaMA/comments/1i8rujw/notes_o...
- krackers 2y agoThe R1 paper says that they didn't use "process reward modeling". And the paper that introduced GPRO says that it can be used either with "outcome supervision" or "process supervision", with outcome supervision "only provid[ing] a reward at the end of each output". Put together, doesn't that imply R1 uses sparse rewards provided only at end of COT sequence?
- serialx 2y agoAh sorry, you might be right. I meant "sparse reward" as a reward system that is mostly 0 but occasionally 1. Your "sparse reward" means only providing reward at the end of each output.
- HeatrayEnjoyer 2y ago> Ah sorry, you might be right. I meant "sparse reward" as a reward system that is mostly 0 but occasionally 1. Did we introduce the abusive pressure of Korean educational culture to machines?