5 ms·

Tested the model briefly with my usual eval: a couple of questions on general knowledge most small models often get wrong, then write a fully-featured todo list

Tested the model briefly with my usual eval: a couple of questions on general knowledge most small models often get wrong, then write a fully-featured todo list app in JS, then rewrite the same app in Rust with Tauri. Granted, most models are well trained on basic todo apps, but it gives me an idea of the basic SWE capabilities I can build on. As far as I'm concerned, if it can successfully setup a local git repo, write a todo list app skeleton that works, I can work with it. SWE: model is strong for its size. It one-shotted the Web app, had no bug. The Rust rewrite only had one bug (reordering didn't work immediately - fixed in one prompt). Committed locally then pushed to my GitHub (https://github.com/DexterLagan/RusTODO https://github.com/DexterLagan/RusTODO). Can't complain. If it can do that reliably, I can use it to make whatever I need. General knowledge: I usually ask 2 questions many small models get wrong: summarize Operation Trojan Horse by John Keel, give publication year. Summarize the Ariel school incident of 1994. Qwen3.8 got the first question right, along correct publication year, gave glorious detail on Keel's theory, but got the second wrong. It thought that school was located in the USA. Ah well. Performance on an admittedly overpowered laptop: 15 tokens/s in power save mode on this MacBook M5 Max 48GB, and 30 in performance mode. Perfectly usable for local coding through OpenCode. Verdict: very nice local and free backup to my usual GPT/Claude/DeepSeek for code. Good for Web searches via Brave search tool calls. What more do you want from a small local model?

terhechte 1mo ago
Which setup did you use? MLX/GGUF, Quant, Engine (e.g. llama.cpp or MTPLX, etc)? There’s so much variety these days.
- dexterlagan 1mo ago
  It was in LMStudio (llama.cpp), Q4 by Unsloth. Applied the recommended defaults published by Unsloth.
cjbprime 1mo ago
> General knowledge: I usually ask 2 questions many small models get wrong: summarize Operation Trojan Horse by John Keel, give publication year. Summarize the Ariel school incident of 1994. Qwen3.8 got the first question right, along correct publication year, gave glorious detail on Keel's theory, but got the second wrong. It thought that school was located in the USA. Ah well. I'm not sure that this means anything. You're asking a ~27GB file to have losslessly compressed the entire training set (which apparently is a large chunk of the entire internet). That's not possible. Whether it happened to encode these particularly obscure facts losslessly or vaguely isn't really telling you anything about how good a model it is.
- ra 1mo ago
  I concur, treating models as question and answer machines and judging them on recall is meaningless, unless you're measuring quantisation impact on a foundation model maybe.
  - dexterlagan 1mo ago
    100% agree, it's really just for kicks&giggles. The fact that the model answered correctly, unlike every other model of its size before it, still pleasantly surprised me.
- Yajirobe 1mo ago
  Did the model refuse to answer? Did it say that it doesn't know? If not, then it's a fair game in my opinion.
  - dexterlagan 1mo ago
    Like most models it makes up things when it doesn't know. I had high hopes for Gemma4, which was said to having 'solved' this particular problem - it didn't. Gemma4 made less things up, did say it didn't know more, but it's still far from perfect. By comparison Qwen3.8 knows more, but still makes things up when it doesn't know. Coding abilities are very impressive however, and yeah these SVG tests do seem to 'scale' or generalize over its general reasoning+coding abilities - at least in JS and Rust. My next test will be to ask it to write some macros in Racket, just to see if it can balance parens. Most, if not all models cannot, no matter their size.
  - croes 1mo ago
    Models don’t know that they don’t know.
    - mdp2021 1mo ago
      > know that they don’t know And we are waiting for architectures that do - because it's duly.
dexterlagan 1mo ago
Meaningless yet fun fact: DeepSeek V4 Pro 0813 made a much worse icon for the same app, and only produced an SVG I had to convert manually to .png. Qwen3.8 made a perfect icon in .png. I don't yet know how it did it, but it did it. Qwen3.8 also seems to know French quite a bit better than Copilot, at least on common expressions. I have yet to run more tests for languages, but I'm baffled by its finer accuracy on the few sentences I asked it to translate, which were botched by Copilot. Go figure.
- NamlchakKhandro 1mo ago
  copilot isn't a model. it's a model hoster.
swores 1mo ago
> write a fully-featured todo list app in JS Would you mind sharing how you prompt this? I'm not a developer myself (just someone who occasionally dabbles, though most of my coding was pre-LLMs) and curious to see how much info/instruction you consider necessary to test them making an actual app (albeit a simple one).
pkghost 1mo ago
> on an admittedly overpowered laptop: 15 tokens/s in power save mode on this MacBook M5 Max 48GB, and 30 in performance mode. Okay, so I'm here to brag a little. I love that I also get 30 t/s on $1500 of decade-old hardware: dell r720 w 2x tesla v100s!

terhechte 1mo ago

Which setup did you use? MLX/GGUF, Quant, Engine (e.g. llama.cpp or MTPLX, etc)? There’s so much variety these days.

dexterlagan 1mo ago
It was in LMStudio (llama.cpp), Q4 by Unsloth. Applied the recommended defaults published by Unsloth.

cjbprime 1mo ago

> General knowledge: I usually ask 2 questions many small models get wrong: summarize Operation Trojan Horse by John Keel, give publication year. Summarize the Ariel school incident of 1994. Qwen3.8 got the first question right, along correct publication year, gave glorious detail on Keel's theory, but got the second wrong. It thought that school was located in the USA. Ah well. I'm not sure that this means anything. You're asking a ~27GB file to have losslessly compressed the entire training set (which apparently is a large chunk of the entire internet). That's not possible. Whether it happened to encode these particularly obscure facts losslessly or vaguely isn't really telling you anything about how good a model it is.

ra 1mo ago
I concur, treating models as question and answer machines and judging them on recall is meaningless, unless you're measuring quantisation impact on a foundation model maybe.
- dexterlagan 1mo ago
  100% agree, it's really just for kicks&giggles. The fact that the model answered correctly, unlike every other model of its size before it, still pleasantly surprised me.
Yajirobe 1mo ago
Did the model refuse to answer? Did it say that it doesn't know? If not, then it's a fair game in my opinion.
- dexterlagan 1mo ago
  Like most models it makes up things when it doesn't know. I had high hopes for Gemma4, which was said to having 'solved' this particular problem - it didn't. Gemma4 made less things up, did say it didn't know more, but it's still far from perfect. By comparison Qwen3.8 knows more, but still makes things up when it doesn't know. Coding abilities are very impressive however, and yeah these SVG tests do seem to 'scale' or generalize over its general reasoning+coding abilities - at least in JS and Rust. My next test will be to ask it to write some macros in Racket, just to see if it can balance parens. Most, if not all models cannot, no matter their size.
- croes 1mo ago
  Models don’t know that they don’t know.
  - mdp2021 1mo ago
    > know that they don’t know And we are waiting for architectures that do - because it's duly.

dexterlagan 1mo ago

Meaningless yet fun fact: DeepSeek V4 Pro 0813 made a much worse icon for the same app, and only produced an SVG I had to convert manually to .png. Qwen3.8 made a perfect icon in .png. I don't yet know how it did it, but it did it. Qwen3.8 also seems to know French quite a bit better than Copilot, at least on common expressions. I have yet to run more tests for languages, but I'm baffled by its finer accuracy on the few sentences I asked it to translate, which were botched by Copilot. Go figure.

NamlchakKhandro 1mo ago
copilot isn't a model. it's a model hoster.

swores 1mo ago

> write a fully-featured todo list app in JS Would you mind sharing how you prompt this? I'm not a developer myself (just someone who occasionally dabbles, though most of my coding was pre-LLMs) and curious to see how much info/instruction you consider necessary to test them making an actual app (albeit a simple one).

pkghost 1mo ago

> on an admittedly overpowered laptop: 15 tokens/s in power save mode on this MacBook M5 Max 48GB, and 30 in performance mode. Okay, so I'm here to brag a little. I love that I also get 30 t/s on $1500 of decade-old hardware: dell r720 w 2x tesla v100s!