7 ms·

I was getting dangerously close to my weekly Claude Code limit last night so I had Claude set up Qwen3.6 with llama.cpp and OpenCode. Honestly it's a great (fre

by briga 4mo ago

I was getting dangerously close to my weekly Claude Code limit last night so I had Claude set up Qwen3.6 with llama.cpp and OpenCode. Honestly it's a great (free!) alternative to Claude Code--certainly more than good enough for a lot of smaller less complex tasks. I'm excited to try this new version. The fact that open-source models are so close to the frontier is very impressive.

leonidasv 4mo ago
Qwen Max are usually closed, unfortunately.
- mostafab 4mo ago
  That's a signal of being SOTA.
plufz 4mo ago
Which exact model are you using? And with which parameters and quant? And on what hardware? Are you using any specific MCPs or other tools to optimize performance like context-mode or dynamic context pruning? I’ve used local models a reasonable amount before but I’m just starting out with opencode. Haven’t had great results yet but really want this to work for simpler tasks. My opencode newly installed is also having iterm on 100% cpu in idle. :/
- briga 4mo ago
  I'm running Qwen3.6:27b Q4 KM on a 4090 and similarly fast CPU and I think 32GB of RAM. Make sure the context window is set to be big enough otherwise the conversation will keep compacting. No special MCP tools set up yet. Qwen is able to do web search out-of-the-box although I think it is getting blocked by anti-bot firewalls--I still need to figure out if I can fix that.
  - SeriousM 4mo ago
    This is the repo: https://huggingface.co/pbhappliedsystems/qwen3.6-27B-gguf-Q4-K-M https://huggingface.co/pbhappliedsystems/qwen3.6-27B-gguf-Q4...
- gcr 4mo ago
  here's a simple setup to get you started on an Apple M1 Max from 2021 with 32GB VRAM. it will download 20GB of models to `~/.cache/huggingface/hub`, which you can delete when you're done. /Users/gcr/llama.cpp/build/bin/llama-server -hf unsloth/Qwen3.6-35B-A3B-GGUF:Q4_K_M --no-mmproj-offload --fit on -c 65536 # edit to taste --reasoning on --chat-template-kwargs '{"preserve_thinking": true}' --sleep-idle-seconds 90 # very aggressive: purge model from vram after this long -ctk q8_0 -ctv q8_0 # Optional. Lower memory use, but lower speed. Omit if you can. I don't recommend ollama or lm-studio. Ollama's in the process of switching from their llama-cpp backend anyway, but their new go framework frequently OOMs and crashes on my hardware. I also don't recommend MLX-based inference backends on this hardware; I've found them to consistently reduce performance, contrary to what I've read online. I've tried all the llama-cpp metal forks, but right now, MTP, TurboQuant, MLX, etc etc etc are too new and just slow things down. It's all dust in the wind still. For agent harnesses, opencode is okay, as is pi or even Zed's built in agent panel. Claude code "works" with ANTHROPIC_BASE_URL=http://localhost:8080/v1, but is very chatty (the default system prompt burns 20k tokens). Crush (from the charm-bracelet folks) is particularly nice when starting out. I've personally converged on pi-agent under an otherwise-mostly-default setup. You can ask qwen to customize pi or write you an extension which helps a little. You'll need to add `http://localhost:8080/v1` as an OpenAI-compatible model provider in your coding harness with any API key (doesn't matter) and any model identifier (doesn't matter with llama-cpp). Note that pi doesn't have permissions. Everything is permitted. The hundred hungry ghosts you've trapped in a jar WILL find a way to delete your home folder someday. That's what Man gets for summoning demons without casting a circle of protection first. Flying too close to the sun etc etc etc Take backups and then go have fun. Hope this helps.
  - srcrip 4mo ago
    Can you elaborate more on the differences in running ollama or lmstudio? Do they actually slow down the speed of the inference and if so why? Or is it just a preference thing?
    - gcr 4mo ago
      Ollama and LM-Studio are fine. Their main advantage is that they have a nice way to browse models -- LMStudio from huggingface and Ollama from their own curated list. Both are great ways of getting started. Pick LM-Studio if you'd like a nice GUI frontend to mlx-lm or llama-cpp; pick ollama if you'd like a nice command line interface and don't need non-default parameters. LM-Studio doesn't support certain parameter combinations. For instance, LM-Studio supports KV quantization....but if you're using the MLX backend, you can't set the context length when KV quantization is used? Why? Running a model with certain settings requires keeping a little SAT solver going in your head. I found that overwhelming, so I just stopped using it. The Ollama devs want to offer a central curated experience, but I perceive their approach as "playing fast and loose." They've re-implemented unique code for every model they support in their own Go runtime, so certain parameter choices aren't supported. On my hardware, their MLX backend just doesn't work at all without segfaulting the server process for example. It doesn't smack as vibe coded the way oMLX does, but it also doesn't smack as professional or battle-tested. Ultimately, just dropping down to llama-cpp's GGUF model support and asking for default settings has provided faster inference speeds than anything I've been able to benchmark with them, but everything's within 10% of each other anyway so it's not a huge deal for me.
ecshafer 4mo ago
Qwen3.6 with claude code works great. I get a lot better results with that than opencode and qwen3.6. Claude Code is a great harness, and good harness/tool integration makes a big difference. You just have a settings.json with your ollama setup and the qwen model and you can use it.
- growt 4mo ago
  Where and how do you run that? I tried it but somehow I always ran out of context or generation was incredibly slow (mbp m4 pro 48gb).
pixelesque 4mo ago
Out of interest, what machine and model are you running it on? I tried the qwen3.6-27b Q6_k GUFF in llama.cpp and LM Studio on my M2 MacBook Pro 32GB machine last week, and I barely get a token a second with either. What sort of speed should I be expecting? I tried some of the Llama 3 34b (nous-capybara?) models two years ago with llama.cpp, and I seem to remember getting a few tokens a second then, so not sure if I've got something completely mis-configured, or I just have unreasonable expectations. Or maybe qwen 3.x is slower for some reason? (Is it mixture of experts?) I'm not expecting it to be instant, but what I'm currently seeing is not really usable.
- Figs 4mo ago
  27B is the dense one. Try the Qwen3.6-35B-A3B variants for the MoE release. That's what I'm running on a Framework Desktop and I get ~50 tok/s plus or minus a few. The dense one is similarly slow for me -- not sure what to expect on your hardware from the MoE but it should probably be much faster.
  - pixelesque 4mo ago
    Thanks!
- gcr 4mo ago
  There are two flavors of Qwen 3.6: - A 27B "dense" model - A 35B "Mixture of Experts" model, which activates only 3B parameters for each token. For your hardware, I strongly recommend `unsloth/Qwen3.6-35B-A3B-GGUF:Q4_K_M`. I have an M1 Max with 32GB VRAM from 2021 that can read at ~300-500 tokens/sec and write at ~30 tokens/sec with llama-cpp's default settings, which is plenty fast. The 27B model can read ~70tok/sec and write ~5tok/sec. The 35B MoE model technically takes slightly more memory but is much faster because it's doing 1/9th the work. It's not quite as "smart", but it's comparable.
  - julianlam 4mo ago
    May I ask why the M instead of XL? Obviously bigger != better but I don't know what the differences are.
wouldbecouldbe 4mo ago
This one doesnt seem to be open source though sadly. Using chinese servers is a step to far for me personally
- gcr 4mo ago
  Look for an open release from the Qwen team in the coming weeks. They like to showcase their proprietary models first, which score higher on benchmarks anyway due to model size.
- geenkeuse 4mo ago
  [dead]
par 4mo ago
Do you have an opinion on OpenCode vs Aider?
- briga 4mo ago
  I haven't tried Aider yet but perhaps I will. Another one that seems to be getting traction is Pi Coding Agent.
- sunaookami 4mo ago
  Aider is still around? That is pre-tool-calling era stuff. Better compare against Pi.
  - par 4mo ago
    I just started running coding agents locally. So you recommend Pi over opencode? (And obviously aider is out?)
    - anderber 4mo ago
      I personally found better results with Opencode. But Pi is really nice too.
    - sunaookami 4mo ago
      Haven't tried OpenCode too much but I found it great. It's more batteries included so I would recommend it over Pi if you don't want to write extensions yourself or use community-provided ones (like webfetch and websearch).
kolinko 4mo ago
As Opus maximalist ;) I was very surprised by the quality if Qwen3.6-27B - trying to figure out how to get it going on RTX 90k now to offload some lighter tasks :)
wuliwong 4mo ago
Do you have a feel for how it Qwen 3.6 compares to Sonnet 4.6? B/C in reality, that's what we use a lot. If we just use Opus 4.7 for everything code related, we'd have a monthly bill 10-20 times higher than using Sonnet where we can.
- briga 4mo ago
  I would say if Sonnet is a senior engineer, then Qwen3.6 (the 27b model) is probably closer to a junior engineer. Still capable of getting stuff done, just needs more guidance and makes mistakes more often. Maybe that's underselling it. It is quite a good model and might end up replacing a lot of the work I was sending to Sonnet 4.6. Also, Sonnet 4.6 is almost certain a much bigger model so the performance differences aren't unexpected.
- nl 4mo ago
  I think you could well be surprised by the Sonnet vs Opus bill (assuming you are paying via the API) In my experience Sonnet bills can be higher than Opus because it churns a lot more trying to get things right. Example from my fairly simple but agentic benchmark: Opus 4.7, 25/25, 81c: https://sql-benchmark.nicklothian.com/?highlight=anthropic_claude-opus-4.7 https://sql-benchmark.nicklothian.com/?highlight=anthropic_c... Opus 4.6, 24/25, 61c: https://sql-benchmark.nicklothian.com/?highlight=anthropic_claude-opus-4.6 https://sql-benchmark.nicklothian.com/?highlight=anthropic_c... Sonnet 4.6: 24/25, 41c: https://sql-benchmark.nicklothian.com/?highlight=anthropic_claude-sonnet-4.6 https://sql-benchmark.nicklothian.com/?highlight=anthropic_c... I only tested the free OpenRouter version of Qwen 3.6 Plus, and it scored 23/25: https://sql-benchmark.nicklothian.com/?highlight=qwen_qwen3.6-plus_free https://sql-benchmark.nicklothian.com/?highlight=qwen_qwen3.... This doesn't quite show Opus cheaper, but it isn't the 10-20 times more either. Harder tasks close the gap even further.
ttoinou 4mo ago
Which agentic coding tool and how do you make sure you have prefix consistency ?
aembleton 4mo ago
> Today we introduce Qwen3.7-Max, our latest proprietary model This is not an open model
chr15m 4mo ago
This new version is not something you'll be able to run locally. It's a "cloud" model and likely too beefy if they do release the weights.
deleted 4mo ago
[deleted]

by briga 4mo ago

leonidasv 4mo ago

Qwen Max are usually closed, unfortunately.

mostafab 4mo ago
That's a signal of being SOTA.

plufz 4mo ago

Which exact model are you using? And with which parameters and quant? And on what hardware? Are you using any specific MCPs or other tools to optimize performance like context-mode or dynamic context pruning? I’ve used local models a reasonable amount before but I’m just starting out with opencode. Haven’t had great results yet but really want this to work for simpler tasks. My opencode newly installed is also having iterm on 100% cpu in idle. :/

briga 4mo ago
I'm running Qwen3.6:27b Q4 KM on a 4090 and similarly fast CPU and I think 32GB of RAM. Make sure the context window is set to be big enough otherwise the conversation will keep compacting. No special MCP tools set up yet. Qwen is able to do web search out-of-the-box although I think it is getting blocked by anti-bot firewalls--I still need to figure out if I can fix that.
- SeriousM 4mo ago
  This is the repo: https://huggingface.co/pbhappliedsystems/qwen3.6-27B-gguf-Q4-K-M https://huggingface.co/pbhappliedsystems/qwen3.6-27B-gguf-Q4...
gcr 4mo ago
here's a simple setup to get you started on an Apple M1 Max from 2021 with 32GB VRAM. it will download 20GB of models to `~/.cache/huggingface/hub`, which you can delete when you're done. /Users/gcr/llama.cpp/build/bin/llama-server -hf unsloth/Qwen3.6-35B-A3B-GGUF:Q4_K_M --no-mmproj-offload --fit on -c 65536 # edit to taste --reasoning on --chat-template-kwargs '{"preserve_thinking": true}' --sleep-idle-seconds 90 # very aggressive: purge model from vram after this long -ctk q8_0 -ctv q8_0 # Optional. Lower memory use, but lower speed. Omit if you can. I don't recommend ollama or lm-studio. Ollama's in the process of switching from their llama-cpp backend anyway, but their new go framework frequently OOMs and crashes on my hardware. I also don't recommend MLX-based inference backends on this hardware; I've found them to consistently reduce performance, contrary to what I've read online. I've tried all the llama-cpp metal forks, but right now, MTP, TurboQuant, MLX, etc etc etc are too new and just slow things down. It's all dust in the wind still. For agent harnesses, opencode is okay, as is pi or even Zed's built in agent panel. Claude code "works" with ANTHROPIC_BASE_URL=http://localhost:8080/v1, but is very chatty (the default system prompt burns 20k tokens). Crush (from the charm-bracelet folks) is particularly nice when starting out. I've personally converged on pi-agent under an otherwise-mostly-default setup. You can ask qwen to customize pi or write you an extension which helps a little. You'll need to add `http://localhost:8080/v1` as an OpenAI-compatible model provider in your coding harness with any API key (doesn't matter) and any model identifier (doesn't matter with llama-cpp). Note that pi doesn't have permissions. Everything is permitted. The hundred hungry ghosts you've trapped in a jar WILL find a way to delete your home folder someday. That's what Man gets for summoning demons without casting a circle of protection first. Flying too close to the sun etc etc etc Take backups and then go have fun. Hope this helps.
- srcrip 4mo ago
  Can you elaborate more on the differences in running ollama or lmstudio? Do they actually slow down the speed of the inference and if so why? Or is it just a preference thing?
  - gcr 4mo ago
    Ollama and LM-Studio are fine. Their main advantage is that they have a nice way to browse models -- LMStudio from huggingface and Ollama from their own curated list. Both are great ways of getting started. Pick LM-Studio if you'd like a nice GUI frontend to mlx-lm or llama-cpp; pick ollama if you'd like a nice command line interface and don't need non-default parameters. LM-Studio doesn't support certain parameter combinations. For instance, LM-Studio supports KV quantization....but if you're using the MLX backend, you can't set the context length when KV quantization is used? Why? Running a model with certain settings requires keeping a little SAT solver going in your head. I found that overwhelming, so I just stopped using it. The Ollama devs want to offer a central curated experience, but I perceive their approach as "playing fast and loose." They've re-implemented unique code for every model they support in their own Go runtime, so certain parameter choices aren't supported. On my hardware, their MLX backend just doesn't work at all without segfaulting the server process for example. It doesn't smack as vibe coded the way oMLX does, but it also doesn't smack as professional or battle-tested. Ultimately, just dropping down to llama-cpp's GGUF model support and asking for default settings has provided faster inference speeds than anything I've been able to benchmark with them, but everything's within 10% of each other anyway so it's not a huge deal for me.

ecshafer 4mo ago

Qwen3.6 with claude code works great. I get a lot better results with that than opencode and qwen3.6. Claude Code is a great harness, and good harness/tool integration makes a big difference. You just have a settings.json with your ollama setup and the qwen model and you can use it.

growt 4mo ago
Where and how do you run that? I tried it but somehow I always ran out of context or generation was incredibly slow (mbp m4 pro 48gb).

pixelesque 4mo ago

Out of interest, what machine and model are you running it on? I tried the qwen3.6-27b Q6_k GUFF in llama.cpp and LM Studio on my M2 MacBook Pro 32GB machine last week, and I barely get a token a second with either. What sort of speed should I be expecting? I tried some of the Llama 3 34b (nous-capybara?) models two years ago with llama.cpp, and I seem to remember getting a few tokens a second then, so not sure if I've got something completely mis-configured, or I just have unreasonable expectations. Or maybe qwen 3.x is slower for some reason? (Is it mixture of experts?) I'm not expecting it to be instant, but what I'm currently seeing is not really usable.

Figs 4mo ago
27B is the dense one. Try the Qwen3.6-35B-A3B variants for the MoE release. That's what I'm running on a Framework Desktop and I get ~50 tok/s plus or minus a few. The dense one is similarly slow for me -- not sure what to expect on your hardware from the MoE but it should probably be much faster.
- pixelesque 4mo ago
  Thanks!
gcr 4mo ago
There are two flavors of Qwen 3.6: - A 27B "dense" model - A 35B "Mixture of Experts" model, which activates only 3B parameters for each token. For your hardware, I strongly recommend `unsloth/Qwen3.6-35B-A3B-GGUF:Q4_K_M`. I have an M1 Max with 32GB VRAM from 2021 that can read at ~300-500 tokens/sec and write at ~30 tokens/sec with llama-cpp's default settings, which is plenty fast. The 27B model can read ~70tok/sec and write ~5tok/sec. The 35B MoE model technically takes slightly more memory but is much faster because it's doing 1/9th the work. It's not quite as "smart", but it's comparable.
- julianlam 4mo ago
  May I ask why the M instead of XL? Obviously bigger != better but I don't know what the differences are.

wouldbecouldbe 4mo ago

This one doesnt seem to be open source though sadly. Using chinese servers is a step to far for me personally

gcr 4mo ago
Look for an open release from the Qwen team in the coming weeks. They like to showcase their proprietary models first, which score higher on benchmarks anyway due to model size.
geenkeuse 4mo ago
[dead]

par 4mo ago

Do you have an opinion on OpenCode vs Aider?

briga 4mo ago
I haven't tried Aider yet but perhaps I will. Another one that seems to be getting traction is Pi Coding Agent.
sunaookami 4mo ago
Aider is still around? That is pre-tool-calling era stuff. Better compare against Pi.
- par 4mo ago
  I just started running coding agents locally. So you recommend Pi over opencode? (And obviously aider is out?)
  - anderber 4mo ago
    I personally found better results with Opencode. But Pi is really nice too.
  - sunaookami 4mo ago
    Haven't tried OpenCode too much but I found it great. It's more batteries included so I would recommend it over Pi if you don't want to write extensions yourself or use community-provided ones (like webfetch and websearch).

kolinko 4mo ago

As Opus maximalist ;) I was very surprised by the quality if Qwen3.6-27B - trying to figure out how to get it going on RTX 90k now to offload some lighter tasks :)

wuliwong 4mo ago

Do you have a feel for how it Qwen 3.6 compares to Sonnet 4.6? B/C in reality, that's what we use a lot. If we just use Opus 4.7 for everything code related, we'd have a monthly bill 10-20 times higher than using Sonnet where we can.

briga 4mo ago
I would say if Sonnet is a senior engineer, then Qwen3.6 (the 27b model) is probably closer to a junior engineer. Still capable of getting stuff done, just needs more guidance and makes mistakes more often. Maybe that's underselling it. It is quite a good model and might end up replacing a lot of the work I was sending to Sonnet 4.6. Also, Sonnet 4.6 is almost certain a much bigger model so the performance differences aren't unexpected.
nl 4mo ago
I think you could well be surprised by the Sonnet vs Opus bill (assuming you are paying via the API) In my experience Sonnet bills can be higher than Opus because it churns a lot more trying to get things right. Example from my fairly simple but agentic benchmark: Opus 4.7, 25/25, 81c: https://sql-benchmark.nicklothian.com/?highlight=anthropic_claude-opus-4.7 https://sql-benchmark.nicklothian.com/?highlight=anthropic_c... Opus 4.6, 24/25, 61c: https://sql-benchmark.nicklothian.com/?highlight=anthropic_claude-opus-4.6 https://sql-benchmark.nicklothian.com/?highlight=anthropic_c... Sonnet 4.6: 24/25, 41c: https://sql-benchmark.nicklothian.com/?highlight=anthropic_claude-sonnet-4.6 https://sql-benchmark.nicklothian.com/?highlight=anthropic_c... I only tested the free OpenRouter version of Qwen 3.6 Plus, and it scored 23/25: https://sql-benchmark.nicklothian.com/?highlight=qwen_qwen3.6-plus_free https://sql-benchmark.nicklothian.com/?highlight=qwen_qwen3.... This doesn't quite show Opus cheaper, but it isn't the 10-20 times more either. Harder tasks close the gap even further.

ttoinou 4mo ago

Which agentic coding tool and how do you make sure you have prefix consistency ?

aembleton 4mo ago

> Today we introduce Qwen3.7-Max, our latest proprietary model This is not an open model

chr15m 4mo ago

This new version is not something you'll be able to run locally. It's a "cloud" model and likely too beefy if they do release the weights.

deleted 4mo ago

[deleted]