9 ms·
Keep in mind all this kind of stuff can make the model less capable. If it has to think in "plain" English, it may well be squashing quality of code etc output.
by martinald 1mo ago
Keep in mind all this kind of stuff can make the model less capable. If it has to think in "plain" English, it may well be squashing quality of code etc output.
I'm not sure how true this is, but when using "forced" json output it def had a big drop off in quality - https://arxiv.org/html/2408.02442v3 https://arxiv.org/html/2408.02442v3.
I think you're better not fighting it with hacks like this and find a different model.
- silverwind 1mo agoChanging output style shouldn't affect thinking at all.
- semiquaver 1mo agoTell that to all the CLAUDE.md lines across dozens of repos I have to write to get them to understand that git commit standards and PR description standards are different.
- MostlyStable 1mo agoI would not overgeneralize from paper. Firstly: forcing JSON output is, in my opinion, a bigger change than asking it to match the above style guidelines, and secondly, as is always the case with these kinds of papers, what was true for the model tested in the paper may either be completely false, or greatly reduced, in later models. That paper is almost 2 years old and models today have been trained in very different ways (or more accurately post trained in very different ways) and are in general far more capable. Based on that paper, I would maybe try to check if it was true for a modern use case, I would very much not assume it was still true.
- firemelt 1mo agohow about stuff likes https://github.com/JuliusBrussee/caveman https://github.com/JuliusBrussee/caveman is it also make the model less capable?