6 ms·
At that point aren't you just edge-case testing? Surely most of your use-cases are not novel tasks that combine obscure domains. It seems to me the real way t
by skohan 1mo ago
At that point aren't you just edge-case testing?
Surely most of your use-cases are not novel tasks that combine obscure domains.
It seems to me the real way to evaluate the value of a model is how it performs in your real-life workflows.
- achrono 1mo agoCertainly, real-life is the ultimate benchmark. But for various reasons that isn't always immediately possible to go evaluate a model on. Maybe my Greek idea sounded too high falutin' or simply seemingly clever (I give an example below -- try it out!). So here's something way simpler that Qwen3.8 27B does not get; only GLM-5.2 and K3 do. ** Analyze the 2 structural (not semantic) patterns in this text: 1. Morning revient. 2. Birds saluent Morgenlicht. 3. We suivons Waldwege toward maison. 4. Rain tombe plötzlich; we cherchons Schutz beneath sapins. 5. Night vient langsam; we trouvons Wärme near le Feuer, sharing quelques Geschichten together. ** The answer should get not just the obvious cyclic E-F-G pattern but also the word counts being Fibonacci. Surprisingly few models get this. The only way I got Qwen to do this was on the 2.4T model, with extensive prompt scaffolding. Claude (Opus & Fable), Sol, Grok etc. got it on the first attempt. (All models, all attempts max reasoning level.)
- skohan 29d agoThis also seems like quite an esoteric use-case to me, but I guess some people might need to know when text follows a fibonacci sequence in terms of word count.