9 ms·

Local Qwen isn't a worse Opus, it's a different tool

493 points by alphabettsy 3mo ago

tnn_YBVS 3mo ago
[flagged]
gpt5 3mo ago
This article is a good summary of local models. Unlike the way they are hyped sometimes, as fantastic tools for coding and agentic local work. The reality is that they are rather limited, would not do well on a long or complex task, and are prone to fall into loops, forget their tasks, etc. Not mentioned in the article is that they are also rather expensive - not just for the hardware cost, but also electricity. These 3090 and 5090 machines are pretty power hungry, and these models are pretty slow on these machines, making them consume more power per token.t Where they shine is in your ability to control them, their privacy, their predictability (e.g. if you are doing a repetitive task, like classifying your photo/video library), and depending on your energy bill - their costs.
- usernomdeguerre 3mo ago
  I believe that local models are a necessary extension of the personal computer and I imagine that one could have had similar criticisms of early personal computers.
  - pmontra 3mo ago
    Of course the early MSDOS PCs where loud and power hungry. I can't remember the specs but according to Wikipedia the IBM PC with a 80286 had a 192 Watt power supply. I don't remember if by then we had internal hard disks or we still had to buy a case as large as the one of the PC with a 10 or 20 MB disk inside. It was handy to raise the monitor further up.
- i_idiot 3mo ago
  > Unlike the way they are hyped sometimes, as fantastic tools for coding and agentic local work. They really are fantastic for a lot of use cases and I think most people do not need SOTA. When I run that qwen model in my measly 4070 12 GB for my personal email agent that I build and experiment with, I need privacy more than anything else. It does a great job. Even for coding tasks, given you know how to use them instead of dumping a grand plan, it's great.
  - throw310822 3mo ago
    > I think most people do not need SOTA SOTA can code but can also prove theorems and teach you about music theory or ancient Greece's substrate language or botany. Speaking in tens of different languages. I wonder how many hundreds of billions of parameters can be saved just by removing much of the general knowledge parts while keeping logical and programming abilities the exact same.
    - trey-jones 3mo ago
      Exactly. I have sort of a fetish for trying to make things smaller by trimming out things that aren't needed. Unfortunately this skill has been largely useless since forever, because hardware improves to the point that these optimizations are trivial: Network Bandwidth, Storage space and speed, memory capacity. While all of these were worth optimizing for at a point in history, that point is behind us. It's probably a reasonable expectation that it will eventually be true for VRAM.
- sanderjd 3mo ago
  But that's current hardware. What about future hardware? What about hardware optimized for inference? What about hardware optimized to run a particular model?
- theshrike79 3mo ago
  My dream would be a local model that can do, say, 80% of the day to day tasks I need; "how does X Handler connect to Y storage?", "commit that feature, but leave out the bits that relate to billing" etc. It would have 99% reliable tool calling - and most importantly - the ability to go "this task is beyond my skills" and refer to a Big Boy Online Model in a gigantic datacenter somewhere. This way all of the simple stuff would be done on-device, gathering data, figuring out the context of the problem etc. And when that's done, the "smart" model would come in to work on the issue when all of the easy stuff is already done. It feels super stupid that my /commit skill calls an online model when that is something a local model can 100% do. Mostly this is a harness issue though and mostly solvable.
  - redrove 3mo ago
    > My dream would be a local model that can do, say, 80% of the day to day tasks I need; "how does X Handler connect to Y storage?", "commit that feature, but leave out the bits that relate to billing" etc. Qwen 3.6 27B can do that today, but setup properly and in a good quant, I run an autoround [0] with weights in int8 and attention heads in f16 on a single RTX 6000 Pro Blackwell Max-Q via vllm with mtp=2 and full context, --max-num-seqs 3, KV in f16, mamba f32. >It would have 99% reliable tool calling I managed to score 93/100 in tool-eval-bench [1]. For me this is very good already, at least in the pi coding harness I've never had an issue that wasn't auto-fixed in the next turn(s). >the ability to go "this task is beyond my skills" and refer to a Big Boy Online Model in a gigantic datacenter somewhere This is heavy on the harness engineering side I think, but also quite contrary to the nature of LLMs today. If you figure this out I'd love to know. [0] https://huggingface.co/Minachist/Qwen3.6-27B-INT8-AutoRound/tree/W8A16-GS128 https://huggingface.co/Minachist/Qwen3.6-27B-INT8-AutoRound/... [1] https://github.com/SeraphimSerapis/tool-eval-bench https://github.com/SeraphimSerapis/tool-eval-bench
    - nrb 3mo ago
      [dead]
    - probiz 3mo ago
      [flagged]
- regularfry 3mo ago
  I've been getting 40-50t/s out of qwen3.6:27b on a 4090 limited to 350W with the MTP changes that went in. That comes out at 8.75J/t at the upper end. No idea how that compares with anything else out there. I'd expect a 5090 to be a bit cheaper because it'd be faster within the same power limit.
cptskippy 3mo ago
I've been running qwen3-5-9b-q4-k-m and qwen3-6-27b-q6-k simultaneously on an Intel Arc Pro B70 with a lot of success. https://github.com/cptskippy/battlemage-llm-gateway https://github.com/cptskippy/battlemage-llm-gateway Opencode has been a huge productivity accelerator. I have two Hermes agents that I'm training to support my workflow with pretty good success. One is a personal assistant who manages my backlog and keeps me on task, follows up with me on items, and will put together research briefs. The other I use a general purpose coder and research and it's about 50:50 with the tasks I've given it. In fairness though, the task it failed at left me scratching my head to figure out as well.
- hbbio 3mo ago
  Interesting setup, thx for sharing. How many tokens/sec do you get with 27b? Are you using MTP?
  - cptskippy 3mo ago
    I haven't done any in-depth synthetic benchmarks but I had my Hermes agent run some and I ran a couple directly on the LLM Gateway that showed similar results. Hermes reported 18.45 tok/s consuming the llama-swap endpoint across the wire. Locally I got 19-19.1 tok/s on the gateway. I'm running the Qwen 3.6 27B Q6 model (qwen3-6-27b-q6-k) off LM Studio and it's less than 0.3s to first token. It's not good for conversational use cases as it can take 1-2 minutes to respond to a prompt. I have two Hermes Profiles running, one is a personal assistant that manages my backlog and provides me morning reminders, solicits for evening updates, and will run overnight research projects for me. The other profile is a coding helper for personal projects. I can ask it to make changes and it will churn for 15 minutes, submit a PR, and notify me that the PR is ready to review. It's faster than me at basic coding tasks.
- jauntywundrkind 3mo ago
  What's the value running the smaller model too? Why not just the big model for everything? I note both are dense, as well.
  - 3mo ago
glerk 3mo ago
If you play with these models long enough, you realize there is more to them than just "model X is smarter than model Y" or "model Y is cheaper than model Z". They are different tools and the prompting technique is different. It is very much like playing an instrument. With Claude, you sometimes want to under-specify or phrase things more indirectly to give a color to the implementation or elicit something creative. Also (you might raise an eyebrow at this) being nice to Claude will be rewarded and being mean to Claude will be punished. Claude tends to mirror your tone more aggressively and you don't want to get into negative loops with it. With GPT, you have to be precise and reduce ambiguity. GPT will often try to resolve ambiguity in a min-max style "I'm going to do X, but make sure it is not quite Y". It will tend to be more paranoid and overengineer to catch all edge cases if you don't tell it precisely what the scope is. With Qwen, you have to give it a shape and let it fill it in. Qwen likes XML, JSON and lists. Qwen likes to be shown a bunch of examples of previous work. This is not scientific at all, just vibes, YMMV.
- bandrami 3mo ago
  I think this goes beyond "vibes" to cargo-culting. It's why nobody's ever able to actually show ROI from LLMs
  - CuriouslyC 3mo ago
    It's hard to actually show ROI from any programming methodology or tool. You can show ROI from a product or feature, but the tool/methodology is a multiplier on the velocity of creating that which is not directly observable.
    - bandrami 3mo ago
      It's really not. When we switched from CVS to SVN I had to show ROI and when I we switched from SVN to git I had to show ROI and when we switched from Ada to Java I had to show ROI. When we switched from Xen to KVM I had to show ROI and when we switched from PAM realtime privileges to rtkit I had to show ROI. When we switched from chroots to LXC I had to show ROI, when we switched from LXC to docker I had to show ROI, and when we switched from docker to podman I had to show ROI. If you can't show ROI there's literally no reason to ever switch anything.
hypfer 3mo ago
That was a lot of text for me still having no idea what the point of the author was (beside what I can infer from the headline that is). I do however now know that they're a totally cool dude building stuff physically and as software + that other people give them money for it. Does that have anything to do with the topic suggested by the headline? Not sure.
- neonstatic 3mo ago
  Everything is an ad these days. The article was not useless, but for the information it provides, it could have been two paragraphs.
  - hypfer 3mo ago
    FWIW it told me stuff about openfaas. Now I know how to mentally file it and how to mentally file the author. The GitHub profile alone might not have sent the same signal, so this is useful. Is it bad software? Idk. Probably not. Should you treat it as a grassroots Foss thing maintained by fellow sane hackers? No sir.
zmmmmm 3mo ago
That's a great write up. The one thing I feel it seems to under estimate is the likelihood of improvement. Even the authors acknowledge it's not even worth comparing local models from a year ago to what we have now. In fact, people widely see Opus 4.5 in November last year - 8 months ago - as the first time agentic coding became viable broadly viable even with frontier hosted models. So why would we lock in hard on any concept at this point of what a local model is and isn't good for? Whatever it is right now, it probably won't be that in a year. It might be naive optimism to think we'll ever get to long horizon tasks with models that run on consumer / pro grade hardware. But so far the naive optimists are winning.
- 3abiton 3mo ago
  And a big thing that's missing is ... the harness comparison. Ot plays a very big role. I use forge, and I have been inpressed with what it can do given all the limitations of local models.
  - fittingopposite 3mo ago
    How would you benchmark them? Are there any benchmarks for harnesses?
- appplication 3mo ago
  Agree 100%, even on claude 4.5 being the turning point for agentic coding. It completely turned me around on it.
- rippeltippel 3mo ago
  Since the author is referring to a specific model, I think it makes sense to ignore how the model (or local models in general) may improve over time. It's like buying a car: I drive that car and get attuned to its characteristics; I don't think how that car (or similar cars) may improve. That's my tool and I want to make the most of it. It is true that switching a local models it technically very cheap, but there's a considerable time investment in squeezing the most out of it, which may not work on a newer version of that model.
- 3mo ago
wallkroft 3mo ago
>Local Qwen isn't a worse Opus >looks inside >local Qwen is not "near Opus levels"
- wren6991 3mo ago
  Ok I'll bite. What's the contradiction?
wallkroft 3mo ago
>Local Qwen isn't a worse Opus >looks inside >local Qwen is not "near Opus levels
opptybiz 3mo ago
[dead]
nicechianti 3mo ago
[dead]
zkmon 3mo ago
The seems to talk a lot about 27B. In my experience, I saw 35B-A3B to be equally good in quality and the MoE gave more tg/s.
- alexellisuk 3mo ago
  The important thing about MoEs which I mention in the conclusion is that they carry fewer (way fewer) active tokens during inference/generation. 35B-A3B is what we started out with in the days of only having the 3090, but the quality is not as good, and the speed from the cards we have now can blaze at 130-200 tokens per second of generation with q5 and a full context in fp16. Not to say that MoEs don't have their place. For people running on unified RAM, they're sometimes the only viable option due to the slowness of dense models. Why is a dense model slower? All model weights have to be loaded and exercised. Passing through 27B vs 3B (active) is maths. So yes you will always get more tokens per second of generation. You must (just as we did) evaluate on your own products and daily work. If the MoE gives the results you need with only 3B parameters then you have your answer. Not prescriptive at all. This is experience based, from the trenches of a actual software business so hopefully a different perspective for folks than "Ran Qwen on my macbook, generated a great python script for me"
nessex 3mo ago
This is a great post that covers a lot of the recent ground. I have a very similar setup after a very similar journey, minus the RTX6000. Worth noting though that a lot of the recent changes make a single 3090/4090 much more viable here too. MTP and the recent improvements to kv quantization in particular, as well as model-specific template & quant fixes. I run a 4090 with the 4-bit quantized variant of the same model now and have had a great experience. Qwen3.5 was already a big step up, but with 3.6 and the rest of the improvements it's substantially more reliable as a daily use tool and I find myself reaching for hosted models a lot less. Feels like I could work entirely without them if they were to disappear without going back to typing every line of code myself. To make 4-bit fit on one card with reasonable (100k+) context needs a bit more care though. And tuning can be highly specific to your machine, gpu and use-case. But I use a headless server, offload multi-modal to CPU, use fit-target to reduce wasted memory and use q8_0 kv since the 4090 performs well with it... In addition to most of the same config as the author elsewhere. I get 50-60tps generation with a power limit of 275W (450W is default), more than enough to offer a roughly an Opus-speed feedback loop. I haven't seen many of the issues with looping the author mentions. But I did with Qwen3.5 and in particular other 4-bit quants in the past. But the difference is probably a mix of the improvements above, as well as habits changing to avoid cases where models will loop. For what I'm doing, it seems like I loop Qwen3.6 on the same kind of prompts I'll make Haiku or Sonnet loop on (the latter hide some of their existential loops behind "thinking"). Usually it's cause I was too vague about some aspect of what I'm wanting them to do or I forgot to include some context that smaller models just don't have access to in their smaller knowledge base. But at least for what I'm doing (Rust, React, kubernetes) it's not been a notable problem at all with the latest iteration of this whole stack. And knowledge of standard libraries and default k8s resource kinds has been almost flawless. There's still plenty of more complex stuff where I'll choose to jump straight to Claude or GLM-5.2, but if it's not worth that jump I've stopped paying for the middle ground as it's usually not much better than just one more iteration through qwen. All this to say, if you have a 3090/4090, feel free to give the same setup a go. It's come a long way in recent weeks.
whazor 3mo ago
Would be interesting to use local models for: - tool calling - code base exploration - anonymizing / abstracting your request Such that your local AI communicates to frontier model like an expensive consultant giving high level advice. I think due to the lower latency of a local model that this could be faster.
- dofm 3mo ago
  I doubt your experience of local models would be of lower latency, except for quite small models in edge uses. In every way, the cloud products from the big two seem optimised for speed and speed of initial response even. I don’t think most people are running local models for speed. More for control, privacy, interest, bloody-mindedness and general principle.
- asimovDev 3mo ago
  I used Qwen 27b 8 bit MLX version on a decompiled android APK recently. It succesfully identified how it worked even the obfuscated classes and methods. It wrote a 1000 line documentation with examples but the time was dreadful. At some point it slowed down to 5 t/s so the whole thing took over an hour , the writing of documentation alone was over 40 minutes, fans blasting the entire time.
  - trey-jones 3mo ago
    I know it uses electricity, but part of the benefit of a local model has to be that you can let it do this while you sleep, and not pay Anthropic for an unknown number of tokens.
    - asimovDev 3mo ago
      yeah i totally understand and I am thoroughly impressed it works. And the electricity cost isn't that bad since it was on a ARM laptop (MacBook M3 Max) and not a beefy workstation with a GPU. I just let the agent do its work while watching the World Cup.
- alexellisuk 3mo ago
barrkel 3mo ago
I found it interesting that vLLM was dismissed as slower than llama.cpp. IME vLLM is quite a bit faster than llama.cpp but where it really wipes the floor with it is in batching concurrent load. The downside is that it is dramatically less flexible in terms of tweaking. It gives you very few options for running quantized weights. It takes a lot longer to start up because it optimizes the compute graph. So for single user experimentation on a model that's a bit too big for your box, vLLM is just going to be frustrating.
- chartered_stack 3mo ago
  One could say: vLLM isn't a worse Llama.cpp, it's a different tool
- krzyk 3mo ago
  AFAIR the general consensus is (was?): - llama.cpp for single user - vLLM for multi-user (e.g. enterprises) They are similar, but for different use cases.
- navbaker 3mo ago
  Yeah, I was a bit baffled by the author complaining about cache prefixes getting destroyed when more than one user hit the model, but then continuing to use llama.cpp instead of switching to vLLM.
- alexellisuk 3mo ago
  vLLM is great at continuous batching and model serving in production, but it's a very different beast and much less versatile for the prosumer category (where we sit for our usage) Dismissed is a strong term, but let me give you some more details. It took a good 4 minutes plus to load up on the 2x 3090 rig, and served a single request 3 tokens/second slower. And the worst bit? With all that work - setting it up and tuning it - it still looped. I was hoping "use just vLLM" advice that we get touted everywhere was the silver bullet. The only thing I'd caution here is that we don't start bashing on llama.cpp like people did with Ollama. It's a very capable tool and for the use-cases we actually want the card for makes more sense. For a large team replacing their Claude Subs perhaps vLLM is the only option, but you really need to add about 5 more RTX 6000 cards into the mix, so you can load something like GLM 5.2.
teh 3mo ago
I sometimes wonder how much of intelligence is being good with tools. I feel pretty averagely smart but give me some good tooling like a good editor, a good type system, semantic grep, good testing and some solvers and I can actually deliver some work. Maybe the trick isn't 500 billion parameters but a model super integrated with the task at hand for iteration and debugging? FWIW the article really mirrors my own experience. I can run a small gemma4 for quick edits (and it's fast!) or data cleanup but for other tasks you do need a different tool (claude).
eurekin 3mo ago
> The model is running so hot, that it shoots past the goal and starts looping later: > My latest experiment was setting up vLLM (the gold standard for production and concurrent serving) and even with an NVLink (175GBP) and tensor parallelism turned on, it was 3 tokens/second slower than llama.cpp during generation for an equivalent setup. In all my tests, getting vllm to run is worth it. It was the single biggest thing, that helped for looping issues, agents going whack and losing focus on the task, long context being essentially useless. FP8 model, unquantized cache in vllm an you have a league better overall experience, with any other stack I tested. Then, you can actually focus on using the model for other things and stop tinkering with settings.
- trey-jones 3mo ago
  I'm really curious about this, not because I disagree, but because I want to avoid agents going whack. Are you running vllm for yourself only, or a for a team, or for an application, etc? And do you feel there is a minimum hardware requirement for vllm to be useful in this way? My weekend project is going to be building a home inference server (from ancient datacenter parts) and I'm still massaging in my head what the end result will be.
  - eurekin 3mo ago
    If I started today, with building a server, I'd jump right into verified set-ups and writeups, like this one: https://github.com/noonghunna/club-3090 https://github.com/noonghunna/club-3090 You can find info about running a patched version of vllm for 1x24gb, 2x and 4x. There's also quite a few "blackwell" subreddits, where people seem to share a lot of substantial information, if you're going the 6000 route.
    - hypfer 3mo ago
      That writeup is completely unhinged and utterly incomprehensible to follow. It just throws "you can do <large number>" at you, with no real explainer regarding how it manages that and which trade-offs are made. I still don't know for certain, but I think one of those trade-offs is 3 bit context? Which is a terrible idea. Please don't share these walls of noise. They shouldn't exist
watt 3mo ago
I find it strange that software people will accept this level of flakiness from the hardware. Normally you would just send the card back, and request a replacement. > One of the cards would only show up if I crossed my fingers when turning it on. Even reboots wouldn't cure it - I had to A/C power off and remove the power cable each time for 30 seconds. This is ridiculous. Of course we are living through supply crunch, but that card is clearly defective hardware.
- alexellisuk 3mo ago
  Ha, you underestimate how dogged you need to be to get this stuff working well. The RTX 3090 in question was used from eBay, no way to return it. The RTX 6000 Pro is the "new card" in question here. The 3090s remain an interesting playground for testing things like VFIO passthrough for SlicerVM and other models whilst not interrupting people on the newer card. In the end, the most stable fix I've found is to install the older proprietary driver and disable the GSP firmware. Have had no issues since. So "clearly defective hardware" seems like it may not be quite correct. And the thing that kept me coming back - along with not having a suitable replacement - or having to gamble on eBay again was the reliability once it showed up in nvidia-smi.
yamakasi007 3mo ago
[flagged]
YuriiKholodkov 3mo ago
[flagged]
krzyk 3mo ago
3090 and 2x3090 are quite popular. But if you uses gigantic (for local models) context of 200k it will go south pretty quickly - any quantization of context quickly becomes the issue.
- alexellisuk 3mo ago
  I think that's quite telling Gorgi replied that he uses Qwen with 131k context. https://x.com/ggerganov/status/2067539416436867230?s=20 https://x.com/ggerganov/status/2067539416436867230?s=20 We also use it with 200-256k (native) context length. The issue could be that folks that don't see looping aren't pushing the model as hard, or as enthusiastically. We also had far fewer issues when thinking was turned off, than with a reasoning budget capped at 2048. Some fine-tunes like Qwopus-Coder just seem prone to looping - google it, you'll see plenty of reports, even on Reddit. For what it's worth seen the RTX 6000 Pro loop even at fp16 on the KV cache - and with vLLM.
rsrsrs86 3mo ago
Chasing models for me it’s a big yellow flag Means underinvesting in engineering Look into it
stego-tech 3mo ago
I still believe that the strength of AI is when it can be applied locally in a secure and private manner, rather than yet another cloud-based service you must pay for indefinitely even as it gets progressively worse to satiate the greed of corporate shareholders. ChatGPT and Anthropic will never, ever get me to tie my Health Data to their systems, but I still believe in the capabilities of AI in identifying patterns from data I would otherwise overlook, and sorely want a local-only ecosystem where I can expose this data safely, privately, and securely to something like Qwen or Gemma for processing. Same goes for Smart Homes, and Personal Assistants. The corporate approach of letting Company A access your data stored at Company B and processed by Companies D and E while also sold to Advertisers and Data Brokers with no way for you to extract or view it on your local hardware - just isn’t tenable for these sorts of intimate use cases. I want my data to be owned and controlled and exposed on my terms, to be used to improve my life first rather than someone else’s bottom line. I want technology to give me back more of my time and improve my outcomes again, and I’ve been burned enough by Big Tech in the past that I flatly reject any presumption of nobility or public good from their AI-as-a-Service business model. The capability is there, and I definitely think the folks working to build local tooling that supports and unlocks the potential for local models are the ones in the right. I love seeing what they build.
- hootz 3mo ago
  The thing about "local" models for me is that they usually mean open-weight (and maybe open-source too), so they can be used locally, yes, but they can also be hosted by independent providers! With models like Qwen, DeepSeek and others, you aren't tied to a single corp, you can switch between indie providers, some of which may give you better privacy guarantees. That allows you to use the models even on devices uncapable of running them, if they have an Internet connection. The strength with AI is with open-source models. We need to keep away from vendor lock-in and use models that allow both local usage and hosting by independent providers.
impara 3mo ago
[flagged]
mystraline 3mo ago
> We've all heard people say that local Qwen 27B or 35-A3B is "near-Opus level" Uh, so, yeah. Im running local Qwen, but Qwen3.5-122B using Krasis https://github.com/brontoguana/krasis https://github.com/brontoguana/krasis Its by far better than Opus. In fact with a phone migration, I was using an OLD android 2fa app "andOTP". Backup files it emitted were JSON but not any sort of standard. I needed the standards version using otpauth:// to upload in my current 2fa. And gave it to my local qwen3.5-122b. It responded with a scary "you uploaded credentials to a public instance LLM! And, it emitted standards compliant URLs. The new app "Tokn" ingested just fine. When side by side was tested, everything was 100% correct. I coukd have did it myself, but it was a one-off. And asking local Qwen worked perfectly. Took like 6 minutes. Would have taken me 1h.
- te0006 3mo ago
  Interesting setup. What GPU(s)/VRAM, CPU and RAM are you using for the 122B model, with which quantization, and what token rates do you achieve for prefill and generation?
itsthecourier 3mo ago
wanted sovereignty, bought a Blackwell for usd12k, discovered a billing issue in some customer and explains that will cover the card I don't follow how it supports the decision of buying the card, I would even say using online SOTA models would had caught it earlier without usd12k and monthly electricity being spent
- alexellisuk 3mo ago
  Author here. Thanks for the question. I'll answer assuming this is a question you have for me. As explained in the post - the 3090s were what were the test bed that proved the investment was worth it. Customer support, architecture reviews, telemetry to check license compliance. None of that could be done with online models. The amount of time we can spend going backwards and forth with enterprise customers over email can really amplify costs to our team. A few actual issues we found and fixed were listed on the linked blog post: https://www.openfaas.com/blog/painless-support-with-diag/ https://www.openfaas.com/blog/painless-support-with-diag/ Having recovered revenue using it in an airgap, to preserve data agreements was more of a cherry on the cake. No need to worry about the investment, it's covered itself. Hope that helps.
piterrro 3mo ago
This is amazing but for everyone out there wanting to buy and build your own AI rig I recommend connecting to one of mamy inference providers and trying out different models themselves for a while. Costs pennies but can give you a nice preview of what you can get with your own rig. Just a friendly tip.
skipants 3mo ago
I feel like it's the Emperor's new clothes reading this article and seeing the praise it's getting. This sentence doesn't even make sense: > These products use very low level Linux primitives like containers, Kubernetes, Firecracker microVMs, and networked protocols. Out of anything that is a "low level linux primitive" I could maybe argue that networking? protocols fit the bill. And it's obviously fully AI-generated! Which I wouldn't even care about if I could actually trust the content, which I can't!
- chadgpt3 3mo ago
  Low level today means JavaScript instead of typescript
  - mekdoonggi 3mo ago
    Low-level today means opening IDE instead of the Chat client.
    - Stitch4223 3mo ago
      That was 16 hours ago. Mind captioning to my smartphone and seeing solutions appear without uttering a word or lifting a finger.
- alexellisuk 3mo ago
  Fair enough, that sentence was fairly compressed. I’ve reworded it - the meaning remains the same. The post is not AI generated, I use AI for code generation and write my own articles. Which part of the post are you struggling with? This is a post describing our own experience and journey. Happy to back up any specific claim.
  - CamperBob2 3mo ago
    How about your reply here? Was that AI-generated? If not, are you conscious of how much you're starting to sound like AI? Is that something you see as a positive thing, or something you'd like to avoid? I actually find this somewhat interesting, because it seems that a lot of people who weren't comfortable with expressing themselves verbally are feeling more empowered in that area. We're hearing new voices for the first time, albeit heavily-filtered ones, and I have to believe that's a good thing. But part of me still finds it offputting for some reason. It's interesting to think about whether that's more of a "you" problem, or more of a "me" problem.
bee_rider 3mo ago
Tangential question (since they brought it up in the article) from someone not involved in AI performance optimization: How big of a deal is looping, practically? Or, I mean, I see thinking models loop occasionally. But it seems to me that every token in the loop should be in the KV cache already, is there really no way to either power through a loop because of the 100% cache hit rate, or identify that you are in a loop that way? (As a human, when thinking hard I sometimes loop, but it is easy enough to identify…)
- alexellisuk 3mo ago
  1. On the technical: The cache only makes generation fast, it doesn't influence what gets chosen next. The loops that hurt the most (point 2 below) are when the model re-decides to do the same thing in different words, which is much harder to detect automatically. We're experimenting with repetition penalty and turning thinking off to solve for the 1st kind of looping (below) 2. On "why is looping a problem" for us Practical example, which I covered in the post: "add --json to every command that does a get or list in faas-cli" - this was a small-ish, open source CLI written with Cobra a very common framework. If I send that to Claude (any of their models) or Codex (GPT), I would have a fully working solution the next time I opened that terminal - a few seconds - a few minutes. With the local model, when it loops, you get some progress and start working on something else. Come back, maybe even 30 minutes later and see it's been printing the same 5 lines over and over constantly. Trust is important for a tool like this, that eroded it. The other type of loop I mention in the blog post is "unable to solve it" loop - Han ran into that more. "Oh I need to fix the indent from 8 to 5 characters in main.py" "Wait I don't know how to write Python code" "Oh now it's broken and I don't know what to do, maybe I should stop" "Let me edit ... " etc, etc
bethekidyouwant 3mo ago
“This rock is not a worse hammer its a different tool”
ttsiodras 3mo ago
Interesting article. IMHO, the author could have done two things better: - vllm instead of llama.cpp. With NVIDIA HW, there is huge difference in multi-user loads and caching with vllm; when he was complaining about what happens when more than one user uses the model, and about losing caching, I was "well, duh". - The budget he used for a single card could have instead be put to far, far better use with SPARKs. I have access to a cluster of 2 x GX10 - total cost less than half what he paid, even today - and I am running vllm and Deepseek v4 Flash. The difference compared to any Qwen is tremendous - I've NEVER seen it loop, and in all my experiments so far, it's the most Sonnet-y model I've ever tried (antirez seems to agree, hence his ds4 fork). If you're wondering about how I set it up in the 2 GX10s: https://forums.developer.nvidia.com/t/deepseek-v4-flash-official-fp8-running-across-2x-dgx-spark-tp-2-mtp-200k-ctx-recipe-numbers/370309/195?u=ttsiodras https://forums.developer.nvidia.com/t/deepseek-v4-flash-offi... Performance: 2K t/s prefill ( very useful for feeding tons of source code into its massive context window ) and around 50-60 tg/s in my coding sessions in the pi.dev harness. With the money the author paid, he could have bought 4 GX10s, and double both numbers ( vllm basically scales almost linearly with tensor parallelism ).
- alexellisuk 3mo ago
  We did run vLLM on the 3090s — measured ~3 tok/s slower on generation for our single-to-few-user pattern, plus less flexibility on quant and slower startup (actual minutes vs single digit seconds). We may do more with it again in the future - there isn't unlimited time for us to tinker, I'm sharing our journey (so far) and reasoning. It's the right call for concurrent batched serving (barrkel's point downthread is spot on), but for how we use it llama.cpp is still better for us. The Spark/GX10 route is a genuinely different bet though and appreciate you sharing your numbers. At the time (several months ago) the consensus was that GX10s were for fine-tuning only, and the numbers were severely low. ..and the card was never about replacing a Claude Max sub. For the workloads we actually bought it for, it's giving us 140-200 tok/s (which matters).
  - ttsiodras 3mo ago
    I hear you on the insane amount of time vllm takes to launch (atlas is a move in the right direction in that regard). But mostly I wanted to raise awareness to readers of your article that no, if you want to do inference, paying 15K for a single 96GB card almost certainly makes no sense. Buy 4 GX10s with the same money, and enjoy dramatically better models and user scalability. Regardless - thanks for putting the effort to share your findings! I keep postponing doing the same... there's tons of things everyone is re-discovering on their own.
mistercheese 3mo ago
I’m not sure if I missed it, but I’m curious how you feel about cloud hosted models with ZDR policies? GLM5.2 or even Minimax M3 on Fireworks or Together ai should be still relatively/consistently cheap and private but a lot more capable and easier to setup?
- alexellisuk 3mo ago
  Thanks for the comment ZDR is mentioned in the post - in particular many the coding plans that are not from the two major leaders have questionable IP/ownership claims on inputs/outputs :) And ZDR is still data sharing with a third party. This is the essence of an enterprise agreement, it's not allowed, even if they pinkie promise not to store it. If your customers allow you to share their data with third parties, then ZDR may be an option for you. I am not a laywer. Where I see ZDR as being more relevant is in protecting your employer's IP - not allowing a missed setting to mean AI labs can train, retain, and publish/resell your work. It's what we'll consider when the subsidies stop being available - open-router, ZDR - but for coding - not for customer data. Very important distinction.
selfawareMammal 3mo ago
I am not a worse player than Messi, I'm just a different player.
dd8601fn 3mo ago
Qwen does have that really nasty tendency to fall into loops. Like, a lot. It only really happens if you allow the thinking directive though. If you can switch it off with what you’re using it on, you’re mostly fine.

493 points by alphabettsy 3mo ago

[flagged]

gpt5 3mo ago

This article is a good summary of local models. Unlike the way they are hyped sometimes, as fantastic tools for coding and agentic local work. The reality is that they are rather limited, would not do well on a long or complex task, and are prone to fall into loops, forget their tasks, etc. Not mentioned in the article is that they are also rather expensive - not just for the hardware cost, but also electricity. These 3090 and 5090 machines are pretty power hungry, and these models are pretty slow on these machines, making them consume more power per token.t Where they shine is in your ability to control them, their privacy, their predictability (e.g. if you are doing a repetitive task, like classifying your photo/video library), and depending on your energy bill - their costs.

usernomdeguerre 3mo ago
I believe that local models are a necessary extension of the personal computer and I imagine that one could have had similar criticisms of early personal computers.
- pmontra 3mo ago
  Of course the early MSDOS PCs where loud and power hungry. I can't remember the specs but according to Wikipedia the IBM PC with a 80286 had a 192 Watt power supply. I don't remember if by then we had internal hard disks or we still had to buy a case as large as the one of the PC with a 10 or 20 MB disk inside. It was handy to raise the monitor further up.
i_idiot 3mo ago
> Unlike the way they are hyped sometimes, as fantastic tools for coding and agentic local work. They really are fantastic for a lot of use cases and I think most people do not need SOTA. When I run that qwen model in my measly 4070 12 GB for my personal email agent that I build and experiment with, I need privacy more than anything else. It does a great job. Even for coding tasks, given you know how to use them instead of dumping a grand plan, it's great.
- throw310822 3mo ago
  > I think most people do not need SOTA SOTA can code but can also prove theorems and teach you about music theory or ancient Greece's substrate language or botany. Speaking in tens of different languages. I wonder how many hundreds of billions of parameters can be saved just by removing much of the general knowledge parts while keeping logical and programming abilities the exact same.
  - trey-jones 3mo ago
    Exactly. I have sort of a fetish for trying to make things smaller by trimming out things that aren't needed. Unfortunately this skill has been largely useless since forever, because hardware improves to the point that these optimizations are trivial: Network Bandwidth, Storage space and speed, memory capacity. While all of these were worth optimizing for at a point in history, that point is behind us. It's probably a reasonable expectation that it will eventually be true for VRAM.
sanderjd 3mo ago
But that's current hardware. What about future hardware? What about hardware optimized for inference? What about hardware optimized to run a particular model?
theshrike79 3mo ago
My dream would be a local model that can do, say, 80% of the day to day tasks I need; "how does X Handler connect to Y storage?", "commit that feature, but leave out the bits that relate to billing" etc. It would have 99% reliable tool calling - and most importantly - the ability to go "this task is beyond my skills" and refer to a Big Boy Online Model in a gigantic datacenter somewhere. This way all of the simple stuff would be done on-device, gathering data, figuring out the context of the problem etc. And when that's done, the "smart" model would come in to work on the issue when all of the easy stuff is already done. It feels super stupid that my /commit skill calls an online model when that is something a local model can 100% do. Mostly this is a harness issue though and mostly solvable.
- redrove 3mo ago
  > My dream would be a local model that can do, say, 80% of the day to day tasks I need; "how does X Handler connect to Y storage?", "commit that feature, but leave out the bits that relate to billing" etc. Qwen 3.6 27B can do that today, but setup properly and in a good quant, I run an autoround [0] with weights in int8 and attention heads in f16 on a single RTX 6000 Pro Blackwell Max-Q via vllm with mtp=2 and full context, --max-num-seqs 3, KV in f16, mamba f32. >It would have 99% reliable tool calling I managed to score 93/100 in tool-eval-bench [1]. For me this is very good already, at least in the pi coding harness I've never had an issue that wasn't auto-fixed in the next turn(s). >the ability to go "this task is beyond my skills" and refer to a Big Boy Online Model in a gigantic datacenter somewhere This is heavy on the harness engineering side I think, but also quite contrary to the nature of LLMs today. If you figure this out I'd love to know. [0] https://huggingface.co/Minachist/Qwen3.6-27B-INT8-AutoRound/tree/W8A16-GS128 https://huggingface.co/Minachist/Qwen3.6-27B-INT8-AutoRound/... [1] https://github.com/SeraphimSerapis/tool-eval-bench https://github.com/SeraphimSerapis/tool-eval-bench
  - nrb 3mo ago
    [dead]
  - probiz 3mo ago
    [flagged]
regularfry 3mo ago
I've been getting 40-50t/s out of qwen3.6:27b on a 4090 limited to 350W with the MTP changes that went in. That comes out at 8.75J/t at the upper end. No idea how that compares with anything else out there. I'd expect a 5090 to be a bit cheaper because it'd be faster within the same power limit.

cptskippy 3mo ago

I've been running qwen3-5-9b-q4-k-m and qwen3-6-27b-q6-k simultaneously on an Intel Arc Pro B70 with a lot of success. https://github.com/cptskippy/battlemage-llm-gateway https://github.com/cptskippy/battlemage-llm-gateway Opencode has been a huge productivity accelerator. I have two Hermes agents that I'm training to support my workflow with pretty good success. One is a personal assistant who manages my backlog and keeps me on task, follows up with me on items, and will put together research briefs. The other I use a general purpose coder and research and it's about 50:50 with the tasks I've given it. In fairness though, the task it failed at left me scratching my head to figure out as well.

hbbio 3mo ago
Interesting setup, thx for sharing. How many tokens/sec do you get with 27b? Are you using MTP?
- cptskippy 3mo ago
  I haven't done any in-depth synthetic benchmarks but I had my Hermes agent run some and I ran a couple directly on the LLM Gateway that showed similar results. Hermes reported 18.45 tok/s consuming the llama-swap endpoint across the wire. Locally I got 19-19.1 tok/s on the gateway. I'm running the Qwen 3.6 27B Q6 model (qwen3-6-27b-q6-k) off LM Studio and it's less than 0.3s to first token. It's not good for conversational use cases as it can take 1-2 minutes to respond to a prompt. I have two Hermes Profiles running, one is a personal assistant that manages my backlog and provides me morning reminders, solicits for evening updates, and will run overnight research projects for me. The other profile is a coding helper for personal projects. I can ask it to make changes and it will churn for 15 minutes, submit a PR, and notify me that the PR is ready to review. It's faster than me at basic coding tasks.
jauntywundrkind 3mo ago
What's the value running the smaller model too? Why not just the big model for everything? I note both are dense, as well.
- 3mo ago

glerk 3mo ago

If you play with these models long enough, you realize there is more to them than just "model X is smarter than model Y" or "model Y is cheaper than model Z". They are different tools and the prompting technique is different. It is very much like playing an instrument. With Claude, you sometimes want to under-specify or phrase things more indirectly to give a color to the implementation or elicit something creative. Also (you might raise an eyebrow at this) being nice to Claude will be rewarded and being mean to Claude will be punished. Claude tends to mirror your tone more aggressively and you don't want to get into negative loops with it. With GPT, you have to be precise and reduce ambiguity. GPT will often try to resolve ambiguity in a min-max style "I'm going to do X, but make sure it is not quite Y". It will tend to be more paranoid and overengineer to catch all edge cases if you don't tell it precisely what the scope is. With Qwen, you have to give it a shape and let it fill it in. Qwen likes XML, JSON and lists. Qwen likes to be shown a bunch of examples of previous work. This is not scientific at all, just vibes, YMMV.

bandrami 3mo ago
I think this goes beyond "vibes" to cargo-culting. It's why nobody's ever able to actually show ROI from LLMs
- CuriouslyC 3mo ago
  It's hard to actually show ROI from any programming methodology or tool. You can show ROI from a product or feature, but the tool/methodology is a multiplier on the velocity of creating that which is not directly observable.
  - bandrami 3mo ago
    It's really not. When we switched from CVS to SVN I had to show ROI and when I we switched from SVN to git I had to show ROI and when we switched from Ada to Java I had to show ROI. When we switched from Xen to KVM I had to show ROI and when we switched from PAM realtime privileges to rtkit I had to show ROI. When we switched from chroots to LXC I had to show ROI, when we switched from LXC to docker I had to show ROI, and when we switched from docker to podman I had to show ROI. If you can't show ROI there's literally no reason to ever switch anything.

hypfer 3mo ago

That was a lot of text for me still having no idea what the point of the author was (beside what I can infer from the headline that is). I do however now know that they're a totally cool dude building stuff physically and as software + that other people give them money for it. Does that have anything to do with the topic suggested by the headline? Not sure.

neonstatic 3mo ago
Everything is an ad these days. The article was not useless, but for the information it provides, it could have been two paragraphs.
- hypfer 3mo ago
  FWIW it told me stuff about openfaas. Now I know how to mentally file it and how to mentally file the author. The GitHub profile alone might not have sent the same signal, so this is useful. Is it bad software? Idk. Probably not. Should you treat it as a grassroots Foss thing maintained by fellow sane hackers? No sir.

zmmmmm 3mo ago

That's a great write up. The one thing I feel it seems to under estimate is the likelihood of improvement. Even the authors acknowledge it's not even worth comparing local models from a year ago to what we have now. In fact, people widely see Opus 4.5 in November last year - 8 months ago - as the first time agentic coding became viable broadly viable even with frontier hosted models. So why would we lock in hard on any concept at this point of what a local model is and isn't good for? Whatever it is right now, it probably won't be that in a year. It might be naive optimism to think we'll ever get to long horizon tasks with models that run on consumer / pro grade hardware. But so far the naive optimists are winning.

3abiton 3mo ago
And a big thing that's missing is ... the harness comparison. Ot plays a very big role. I use forge, and I have been inpressed with what it can do given all the limitations of local models.
- fittingopposite 3mo ago
  How would you benchmark them? Are there any benchmarks for harnesses?
appplication 3mo ago
Agree 100%, even on claude 4.5 being the turning point for agentic coding. It completely turned me around on it.
rippeltippel 3mo ago
Since the author is referring to a specific model, I think it makes sense to ignore how the model (or local models in general) may improve over time. It's like buying a car: I drive that car and get attuned to its characteristics; I don't think how that car (or similar cars) may improve. That's my tool and I want to make the most of it. It is true that switching a local models it technically very cheap, but there's a considerable time investment in squeezing the most out of it, which may not work on a newer version of that model.
3mo ago

wallkroft 3mo ago

>Local Qwen isn't a worse Opus >looks inside >local Qwen is not "near Opus levels"

wren6991 3mo ago
Ok I'll bite. What's the contradiction?

wallkroft 3mo ago

>Local Qwen isn't a worse Opus >looks inside >local Qwen is not "near Opus levels

opptybiz 3mo ago

[dead]

nicechianti 3mo ago

[dead]

zkmon 3mo ago

The seems to talk a lot about 27B. In my experience, I saw 35B-A3B to be equally good in quality and the MoE gave more tg/s.

alexellisuk 3mo ago
The important thing about MoEs which I mention in the conclusion is that they carry fewer (way fewer) active tokens during inference/generation. 35B-A3B is what we started out with in the days of only having the 3090, but the quality is not as good, and the speed from the cards we have now can blaze at 130-200 tokens per second of generation with q5 and a full context in fp16. Not to say that MoEs don't have their place. For people running on unified RAM, they're sometimes the only viable option due to the slowness of dense models. Why is a dense model slower? All model weights have to be loaded and exercised. Passing through 27B vs 3B (active) is maths. So yes you will always get more tokens per second of generation. You must (just as we did) evaluate on your own products and daily work. If the MoE gives the results you need with only 3B parameters then you have your answer. Not prescriptive at all. This is experience based, from the trenches of a actual software business so hopefully a different perspective for folks than "Ran Qwen on my macbook, generated a great python script for me"

nessex 3mo ago

This is a great post that covers a lot of the recent ground. I have a very similar setup after a very similar journey, minus the RTX6000. Worth noting though that a lot of the recent changes make a single 3090/4090 much more viable here too. MTP and the recent improvements to kv quantization in particular, as well as model-specific template & quant fixes. I run a 4090 with the 4-bit quantized variant of the same model now and have had a great experience. Qwen3.5 was already a big step up, but with 3.6 and the rest of the improvements it's substantially more reliable as a daily use tool and I find myself reaching for hosted models a lot less. Feels like I could work entirely without them if they were to disappear without going back to typing every line of code myself. To make 4-bit fit on one card with reasonable (100k+) context needs a bit more care though. And tuning can be highly specific to your machine, gpu and use-case. But I use a headless server, offload multi-modal to CPU, use fit-target to reduce wasted memory and use q8_0 kv since the 4090 performs well with it... In addition to most of the same config as the author elsewhere. I get 50-60tps generation with a power limit of 275W (450W is default), more than enough to offer a roughly an Opus-speed feedback loop. I haven't seen many of the issues with looping the author mentions. But I did with Qwen3.5 and in particular other 4-bit quants in the past. But the difference is probably a mix of the improvements above, as well as habits changing to avoid cases where models will loop. For what I'm doing, it seems like I loop Qwen3.6 on the same kind of prompts I'll make Haiku or Sonnet loop on (the latter hide some of their existential loops behind "thinking"). Usually it's cause I was too vague about some aspect of what I'm wanting them to do or I forgot to include some context that smaller models just don't have access to in their smaller knowledge base. But at least for what I'm doing (Rust, React, kubernetes) it's not been a notable problem at all with the latest iteration of this whole stack. And knowledge of standard libraries and default k8s resource kinds has been almost flawless. There's still plenty of more complex stuff where I'll choose to jump straight to Claude or GLM-5.2, but if it's not worth that jump I've stopped paying for the middle ground as it's usually not much better than just one more iteration through qwen. All this to say, if you have a 3090/4090, feel free to give the same setup a go. It's come a long way in recent weeks.

whazor 3mo ago

Would be interesting to use local models for: - tool calling - code base exploration - anonymizing / abstracting your request Such that your local AI communicates to frontier model like an expensive consultant giving high level advice. I think due to the lower latency of a local model that this could be faster.

dofm 3mo ago
I doubt your experience of local models would be of lower latency, except for quite small models in edge uses. In every way, the cloud products from the big two seem optimised for speed and speed of initial response even. I don’t think most people are running local models for speed. More for control, privacy, interest, bloody-mindedness and general principle.
asimovDev 3mo ago
I used Qwen 27b 8 bit MLX version on a decompiled android APK recently. It succesfully identified how it worked even the obfuscated classes and methods. It wrote a 1000 line documentation with examples but the time was dreadful. At some point it slowed down to 5 t/s so the whole thing took over an hour , the writing of documentation alone was over 40 minutes, fans blasting the entire time.
- trey-jones 3mo ago
  I know it uses electricity, but part of the benefit of a local model has to be that you can let it do this while you sleep, and not pay Anthropic for an unknown number of tokens.
  - asimovDev 3mo ago
    yeah i totally understand and I am thoroughly impressed it works. And the electricity cost isn't that bad since it was on a ARM laptop (MacBook M3 Max) and not a beefy workstation with a GPU. I just let the agent do its work while watching the World Cup.
alexellisuk 3mo ago

barrkel 3mo ago

I found it interesting that vLLM was dismissed as slower than llama.cpp. IME vLLM is quite a bit faster than llama.cpp but where it really wipes the floor with it is in batching concurrent load. The downside is that it is dramatically less flexible in terms of tweaking. It gives you very few options for running quantized weights. It takes a lot longer to start up because it optimizes the compute graph. So for single user experimentation on a model that's a bit too big for your box, vLLM is just going to be frustrating.

chartered_stack 3mo ago
One could say: vLLM isn't a worse Llama.cpp, it's a different tool
krzyk 3mo ago
AFAIR the general consensus is (was?): - llama.cpp for single user - vLLM for multi-user (e.g. enterprises) They are similar, but for different use cases.
navbaker 3mo ago
Yeah, I was a bit baffled by the author complaining about cache prefixes getting destroyed when more than one user hit the model, but then continuing to use llama.cpp instead of switching to vLLM.
alexellisuk 3mo ago
vLLM is great at continuous batching and model serving in production, but it's a very different beast and much less versatile for the prosumer category (where we sit for our usage) Dismissed is a strong term, but let me give you some more details. It took a good 4 minutes plus to load up on the 2x 3090 rig, and served a single request 3 tokens/second slower. And the worst bit? With all that work - setting it up and tuning it - it still looped. I was hoping "use just vLLM" advice that we get touted everywhere was the silver bullet. The only thing I'd caution here is that we don't start bashing on llama.cpp like people did with Ollama. It's a very capable tool and for the use-cases we actually want the card for makes more sense. For a large team replacing their Claude Subs perhaps vLLM is the only option, but you really need to add about 5 more RTX 6000 cards into the mix, so you can load something like GLM 5.2.

teh 3mo ago

I sometimes wonder how much of intelligence is being good with tools. I feel pretty averagely smart but give me some good tooling like a good editor, a good type system, semantic grep, good testing and some solvers and I can actually deliver some work. Maybe the trick isn't 500 billion parameters but a model super integrated with the task at hand for iteration and debugging? FWIW the article really mirrors my own experience. I can run a small gemma4 for quick edits (and it's fast!) or data cleanup but for other tasks you do need a different tool (claude).

eurekin 3mo ago

> The model is running so hot, that it shoots past the goal and starts looping later: > My latest experiment was setting up vLLM (the gold standard for production and concurrent serving) and even with an NVLink (175GBP) and tensor parallelism turned on, it was 3 tokens/second slower than llama.cpp during generation for an equivalent setup. In all my tests, getting vllm to run is worth it. It was the single biggest thing, that helped for looping issues, agents going whack and losing focus on the task, long context being essentially useless. FP8 model, unquantized cache in vllm an you have a league better overall experience, with any other stack I tested. Then, you can actually focus on using the model for other things and stop tinkering with settings.

trey-jones 3mo ago
I'm really curious about this, not because I disagree, but because I want to avoid agents going whack. Are you running vllm for yourself only, or a for a team, or for an application, etc? And do you feel there is a minimum hardware requirement for vllm to be useful in this way? My weekend project is going to be building a home inference server (from ancient datacenter parts) and I'm still massaging in my head what the end result will be.
- eurekin 3mo ago
  If I started today, with building a server, I'd jump right into verified set-ups and writeups, like this one: https://github.com/noonghunna/club-3090 https://github.com/noonghunna/club-3090 You can find info about running a patched version of vllm for 1x24gb, 2x and 4x. There's also quite a few "blackwell" subreddits, where people seem to share a lot of substantial information, if you're going the 6000 route.
  - hypfer 3mo ago
    That writeup is completely unhinged and utterly incomprehensible to follow. It just throws "you can do <large number>" at you, with no real explainer regarding how it manages that and which trade-offs are made. I still don't know for certain, but I think one of those trade-offs is 3 bit context? Which is a terrible idea. Please don't share these walls of noise. They shouldn't exist

watt 3mo ago

I find it strange that software people will accept this level of flakiness from the hardware. Normally you would just send the card back, and request a replacement. > One of the cards would only show up if I crossed my fingers when turning it on. Even reboots wouldn't cure it - I had to A/C power off and remove the power cable each time for 30 seconds. This is ridiculous. Of course we are living through supply crunch, but that card is clearly defective hardware.

alexellisuk 3mo ago
Ha, you underestimate how dogged you need to be to get this stuff working well. The RTX 3090 in question was used from eBay, no way to return it. The RTX 6000 Pro is the "new card" in question here. The 3090s remain an interesting playground for testing things like VFIO passthrough for SlicerVM and other models whilst not interrupting people on the newer card. In the end, the most stable fix I've found is to install the older proprietary driver and disable the GSP firmware. Have had no issues since. So "clearly defective hardware" seems like it may not be quite correct. And the thing that kept me coming back - along with not having a suitable replacement - or having to gamble on eBay again was the reliability once it showed up in nvidia-smi.

yamakasi007 3mo ago

[flagged]

YuriiKholodkov 3mo ago

[flagged]

krzyk 3mo ago

3090 and 2x3090 are quite popular. But if you uses gigantic (for local models) context of 200k it will go south pretty quickly - any quantization of context quickly becomes the issue.

alexellisuk 3mo ago
I think that's quite telling Gorgi replied that he uses Qwen with 131k context. https://x.com/ggerganov/status/2067539416436867230?s=20 https://x.com/ggerganov/status/2067539416436867230?s=20 We also use it with 200-256k (native) context length. The issue could be that folks that don't see looping aren't pushing the model as hard, or as enthusiastically. We also had far fewer issues when thinking was turned off, than with a reasoning budget capped at 2048. Some fine-tunes like Qwopus-Coder just seem prone to looping - google it, you'll see plenty of reports, even on Reddit. For what it's worth seen the RTX 6000 Pro loop even at fp16 on the KV cache - and with vLLM.

rsrsrs86 3mo ago

Chasing models for me it’s a big yellow flag Means underinvesting in engineering Look into it

stego-tech 3mo ago

I still believe that the strength of AI is when it can be applied locally in a secure and private manner, rather than yet another cloud-based service you must pay for indefinitely even as it gets progressively worse to satiate the greed of corporate shareholders. ChatGPT and Anthropic will never, ever get me to tie my Health Data to their systems, but I still believe in the capabilities of AI in identifying patterns from data I would otherwise overlook, and sorely want a local-only ecosystem where I can expose this data safely, privately, and securely to something like Qwen or Gemma for processing. Same goes for Smart Homes, and Personal Assistants. The corporate approach of letting Company A access your data stored at Company B and processed by Companies D and E while also sold to Advertisers and Data Brokers with no way for you to extract or view it on your local hardware - just isn’t tenable for these sorts of intimate use cases. I want my data to be owned and controlled and exposed on my terms, to be used to improve my life first rather than someone else’s bottom line. I want technology to give me back more of my time and improve my outcomes again, and I’ve been burned enough by Big Tech in the past that I flatly reject any presumption of nobility or public good from their AI-as-a-Service business model. The capability is there, and I definitely think the folks working to build local tooling that supports and unlocks the potential for local models are the ones in the right. I love seeing what they build.

hootz 3mo ago
The thing about "local" models for me is that they usually mean open-weight (and maybe open-source too), so they can be used locally, yes, but they can also be hosted by independent providers! With models like Qwen, DeepSeek and others, you aren't tied to a single corp, you can switch between indie providers, some of which may give you better privacy guarantees. That allows you to use the models even on devices uncapable of running them, if they have an Internet connection. The strength with AI is with open-source models. We need to keep away from vendor lock-in and use models that allow both local usage and hosting by independent providers.

impara 3mo ago

[flagged]

mystraline 3mo ago

> We've all heard people say that local Qwen 27B or 35-A3B is "near-Opus level" Uh, so, yeah. Im running local Qwen, but Qwen3.5-122B using Krasis https://github.com/brontoguana/krasis https://github.com/brontoguana/krasis Its by far better than Opus. In fact with a phone migration, I was using an OLD android 2fa app "andOTP". Backup files it emitted were JSON but not any sort of standard. I needed the standards version using otpauth:// to upload in my current 2fa. And gave it to my local qwen3.5-122b. It responded with a scary "you uploaded credentials to a public instance LLM! And, it emitted standards compliant URLs. The new app "Tokn" ingested just fine. When side by side was tested, everything was 100% correct. I coukd have did it myself, but it was a one-off. And asking local Qwen worked perfectly. Took like 6 minutes. Would have taken me 1h.

te0006 3mo ago
Interesting setup. What GPU(s)/VRAM, CPU and RAM are you using for the 122B model, with which quantization, and what token rates do you achieve for prefill and generation?

itsthecourier 3mo ago

wanted sovereignty, bought a Blackwell for usd12k, discovered a billing issue in some customer and explains that will cover the card I don't follow how it supports the decision of buying the card, I would even say using online SOTA models would had caught it earlier without usd12k and monthly electricity being spent

alexellisuk 3mo ago
Author here. Thanks for the question. I'll answer assuming this is a question you have for me. As explained in the post - the 3090s were what were the test bed that proved the investment was worth it. Customer support, architecture reviews, telemetry to check license compliance. None of that could be done with online models. The amount of time we can spend going backwards and forth with enterprise customers over email can really amplify costs to our team. A few actual issues we found and fixed were listed on the linked blog post: https://www.openfaas.com/blog/painless-support-with-diag/ https://www.openfaas.com/blog/painless-support-with-diag/ Having recovered revenue using it in an airgap, to preserve data agreements was more of a cherry on the cake. No need to worry about the investment, it's covered itself. Hope that helps.

piterrro 3mo ago

This is amazing but for everyone out there wanting to buy and build your own AI rig I recommend connecting to one of mamy inference providers and trying out different models themselves for a while. Costs pennies but can give you a nice preview of what you can get with your own rig. Just a friendly tip.

skipants 3mo ago

I feel like it's the Emperor's new clothes reading this article and seeing the praise it's getting. This sentence doesn't even make sense: > These products use very low level Linux primitives like containers, Kubernetes, Firecracker microVMs, and networked protocols. Out of anything that is a "low level linux primitive" I could maybe argue that networking? protocols fit the bill. And it's obviously fully AI-generated! Which I wouldn't even care about if I could actually trust the content, which I can't!

chadgpt3 3mo ago
Low level today means JavaScript instead of typescript
- mekdoonggi 3mo ago
  Low-level today means opening IDE instead of the Chat client.
  - Stitch4223 3mo ago
    That was 16 hours ago. Mind captioning to my smartphone and seeing solutions appear without uttering a word or lifting a finger.
alexellisuk 3mo ago
Fair enough, that sentence was fairly compressed. I’ve reworded it - the meaning remains the same. The post is not AI generated, I use AI for code generation and write my own articles. Which part of the post are you struggling with? This is a post describing our own experience and journey. Happy to back up any specific claim.
- CamperBob2 3mo ago
  How about your reply here? Was that AI-generated? If not, are you conscious of how much you're starting to sound like AI? Is that something you see as a positive thing, or something you'd like to avoid? I actually find this somewhat interesting, because it seems that a lot of people who weren't comfortable with expressing themselves verbally are feeling more empowered in that area. We're hearing new voices for the first time, albeit heavily-filtered ones, and I have to believe that's a good thing. But part of me still finds it offputting for some reason. It's interesting to think about whether that's more of a "you" problem, or more of a "me" problem.

bee_rider 3mo ago

Tangential question (since they brought it up in the article) from someone not involved in AI performance optimization: How big of a deal is looping, practically? Or, I mean, I see thinking models loop occasionally. But it seems to me that every token in the loop should be in the KV cache already, is there really no way to either power through a loop because of the 100% cache hit rate, or identify that you are in a loop that way? (As a human, when thinking hard I sometimes loop, but it is easy enough to identify…)

alexellisuk 3mo ago
1. On the technical: The cache only makes generation fast, it doesn't influence what gets chosen next. The loops that hurt the most (point 2 below) are when the model re-decides to do the same thing in different words, which is much harder to detect automatically. We're experimenting with repetition penalty and turning thinking off to solve for the 1st kind of looping (below) 2. On "why is looping a problem" for us Practical example, which I covered in the post: "add --json to every command that does a get or list in faas-cli" - this was a small-ish, open source CLI written with Cobra a very common framework. If I send that to Claude (any of their models) or Codex (GPT), I would have a fully working solution the next time I opened that terminal - a few seconds - a few minutes. With the local model, when it loops, you get some progress and start working on something else. Come back, maybe even 30 minutes later and see it's been printing the same 5 lines over and over constantly. Trust is important for a tool like this, that eroded it. The other type of loop I mention in the blog post is "unable to solve it" loop - Han ran into that more. "Oh I need to fix the indent from 8 to 5 characters in main.py" "Wait I don't know how to write Python code" "Oh now it's broken and I don't know what to do, maybe I should stop" "Let me edit ... " etc, etc

bethekidyouwant 3mo ago

“This rock is not a worse hammer its a different tool”

ttsiodras 3mo ago

Interesting article. IMHO, the author could have done two things better: - vllm instead of llama.cpp. With NVIDIA HW, there is huge difference in multi-user loads and caching with vllm; when he was complaining about what happens when more than one user uses the model, and about losing caching, I was "well, duh". - The budget he used for a single card could have instead be put to far, far better use with SPARKs. I have access to a cluster of 2 x GX10 - total cost less than half what he paid, even today - and I am running vllm and Deepseek v4 Flash. The difference compared to any Qwen is tremendous - I've NEVER seen it loop, and in all my experiments so far, it's the most Sonnet-y model I've ever tried (antirez seems to agree, hence his ds4 fork). If you're wondering about how I set it up in the 2 GX10s: https://forums.developer.nvidia.com/t/deepseek-v4-flash-official-fp8-running-across-2x-dgx-spark-tp-2-mtp-200k-ctx-recipe-numbers/370309/195?u=ttsiodras https://forums.developer.nvidia.com/t/deepseek-v4-flash-offi... Performance: 2K t/s prefill ( very useful for feeding tons of source code into its massive context window ) and around 50-60 tg/s in my coding sessions in the pi.dev harness. With the money the author paid, he could have bought 4 GX10s, and double both numbers ( vllm basically scales almost linearly with tensor parallelism ).

alexellisuk 3mo ago
We did run vLLM on the 3090s — measured ~3 tok/s slower on generation for our single-to-few-user pattern, plus less flexibility on quant and slower startup (actual minutes vs single digit seconds). We may do more with it again in the future - there isn't unlimited time for us to tinker, I'm sharing our journey (so far) and reasoning. It's the right call for concurrent batched serving (barrkel's point downthread is spot on), but for how we use it llama.cpp is still better for us. The Spark/GX10 route is a genuinely different bet though and appreciate you sharing your numbers. At the time (several months ago) the consensus was that GX10s were for fine-tuning only, and the numbers were severely low. ..and the card was never about replacing a Claude Max sub. For the workloads we actually bought it for, it's giving us 140-200 tok/s (which matters).
- ttsiodras 3mo ago
  I hear you on the insane amount of time vllm takes to launch (atlas is a move in the right direction in that regard). But mostly I wanted to raise awareness to readers of your article that no, if you want to do inference, paying 15K for a single 96GB card almost certainly makes no sense. Buy 4 GX10s with the same money, and enjoy dramatically better models and user scalability. Regardless - thanks for putting the effort to share your findings! I keep postponing doing the same... there's tons of things everyone is re-discovering on their own.

mistercheese 3mo ago

I’m not sure if I missed it, but I’m curious how you feel about cloud hosted models with ZDR policies? GLM5.2 or even Minimax M3 on Fireworks or Together ai should be still relatively/consistently cheap and private but a lot more capable and easier to setup?

alexellisuk 3mo ago
Thanks for the comment ZDR is mentioned in the post - in particular many the coding plans that are not from the two major leaders have questionable IP/ownership claims on inputs/outputs :) And ZDR is still data sharing with a third party. This is the essence of an enterprise agreement, it's not allowed, even if they pinkie promise not to store it. If your customers allow you to share their data with third parties, then ZDR may be an option for you. I am not a laywer. Where I see ZDR as being more relevant is in protecting your employer's IP - not allowing a missed setting to mean AI labs can train, retain, and publish/resell your work. It's what we'll consider when the subsidies stop being available - open-router, ZDR - but for coding - not for customer data. Very important distinction.

selfawareMammal 3mo ago

I am not a worse player than Messi, I'm just a different player.

dd8601fn 3mo ago

Qwen does have that really nasty tendency to fall into loops. Like, a lot. It only really happens if you allow the thinking directive though. If you can switch it off with what you’re using it on, you’re mostly fine.