16 ms·

Running local models is good now

• xbmcuser 3mo ago
  Running local models might be good but until the virtual hardware monopolies of tsmc and others is broken they will out of reach for most people.
• pauljeba 3mo ago
  How do I beleive you? You wrote this post by hand. lol
• iluvcommunism 3mo ago
  [dead]
• _doctor_love 3mo ago
  "Just get a 64GB Mac with 1TB of storage!" LOL - some of us have a budget
  • tjwebbnorfolk 3mo ago
    AI and budgets don't mix well at the moment
  • techscruggs 3mo ago
    He is using a 2022 M2, which you can get that for about $2k used. That is beyond reasonable.
    • psychoslave 3mo ago
      Global Affordability Estimate: Top 10% of global earners (~800M people) can afford a $2,000 device without major financial strain. Top 25% (~2B people) could afford it with some budget adjustments. Bottom 50% (~4B people) would find it prohibitively expensive. So for a SV top income, maybe that might look more like the weekly pet brushing budget, but for most people out there this is not that much of a no-brainer.
      • richwater 3mo ago
        Yes, because the bottom 50%, mostly impoverished or near impoverished folks were spending money on Claude Code subscriptions instead /s
        • ric2b 3mo ago
          2K is 10 years of a Claude Pro subscription, which also gives you better models...
      • disgruntledphd2 3mo ago
        The maths changes if you're working for yourself. Because I live in Europe, I've ended up working as a contractor due to the lack of a legal entity in my country. While that mostly sucked for a bunch of reasons, I was able to get a 64Gb Mac M2 a few years back with approximately a 52% discount, which was kinda nice.
        • weego 3mo ago
          If you're working for yourself paying monthly is exactly the same as amortising an asset. Personally I'd rather my business just pay $100 a month than have to deal with additional hardware and software maintenance while using a depreciating asset that is break-even after 3-5 years depending on the spec.
      • deleted 3mo ago
        [deleted]
      • frollogaston 3mo ago
        Bottom 50% aren't paying for Claude either, probably also don't own PCs or write code
    • Shekelphile 3mo ago
      She
  • themythfable 3mo ago
    Yeah, I never had a computer that cost north of $800 until recently. While that is far from the typical HN user's budget, my bet is that it is much closer to average. Besides those with effectively unlimited budgets for their personal compute, local models are still a long ways off. Though, that shouldn't be conflated with the value of open-source models, which can be used by cloud providers to significantly reduce cost of intelligence.
    • embedding-shape 3mo ago
      > Yeah, I never had a computer that cost north of $800 until recently. While that is far from the typical HN user's budget, my bet is that it is much closer to average. There are segments, everything from "Average person in world" to "Average creative professional using computers for work" and more on HN, with a wide range of costs for the hardware. HN probably skews towards the latter rather than the former, probably sitting with enterprise hardware next to them basically for fun, hard to make wider conclusions from what people here have or not.
    • sublinear 3mo ago
      If we define "typical" as the median HN budget, it's probably about the same as yours. Maybe the answer would have been different 10 or 20 years ago, but the era of truly needing a big budget PC has been over for a while. It's just for gaming and AI now. Maybe not even gaming as much anymore. Consider the perspective of someone who has a practically unlimited budget for PCs, doesn't game much anymore, and doesn't need AI to do their job. It's just part of getting older, and there are plenty of people in their late 30s and older on here.
  • p-e-w 3mo ago
    No need. You can run the Gemma 4 and Qwen3.5 MoE models with as little as 12 GB of VRAM at 30-40 tps (Q4/Q5), and they both blow GPT-4o and DeepSeek R1 out of the water.
  • swatcoder 3mo ago
    Sure, but it's also not really out of scale with the cost of a shop tool in other trades. If you're a professional that's confident in a positive return on the investment (optimal or not), or just a hobbyist with the luxury budget for a "shop" that cost is well within norms. That's not everybody, of course, but it's not some inconceivable fantasy. A lot of people in the tech community here on HN, specifically, end up with pretty high discretionary budgets that they pour into stuff like this.
    • frollogaston 3mo ago
      But you can get that return from a paid service too, in fact it'll be better. So just comparing costs, what's the annualized ROI on the Mac Studio assuming it means you avoid paying $240/y for Claude? Cause I can always set aside the Mac's price in some investments and pay for Claude out of that.
      • swatcoder 3mo ago
        Same with many and their shop tools in other trades. Most hobbyists and many professionals could end up far ahead financially by leveraging makerspaces, tool rentals, and co-op shops or even by hiring out a professional to prep certain intermediates for them, but they get psychological value -- as well as flexibility, reliability, and resale opportunity -- from having their own well-outfitted shop. And they can afford that premium, so they do. At the scale of individuals and small shops, not everything that matters gets captured in financial models.
        • frollogaston 3mo ago
          Yeah but the local model doesn't have those advantages for the coding use cases, at least not yet. In theory you could post-train one on your codebase or something, but nobody cares to do that when any vanilla coding agent service can read and understand the whole thing better than a locally tuned free model. I was already being very generous towards the Mac in pretending it does the same thing as the paid service. Aside, physical tools tend to be financially advantageous to own if you're going to use them a lot. Even if the owner were targeting 0 profit, they'd have to charge more to factor in the cost of dealing with customers and increased risk of wear/damage by users who don't care as much.
  • amalcon 3mo ago
    A Strix Halo with similar RAM is considerably cheaper. Still not cheap, mind, but performance is OK (not great) and it will run more or less the same models.
    • AbsurdCensor 3mo ago
      At least for me, it's been pretty great, but I bought my system when it was $1800, now looks like the same system is $2700 and out of stock. I still haven't quite been able to run 120B parameter models under Windows, but for Qwen Coder 30B, it works pretty darn well for my at home needs.
      • amalcon 3mo ago
        Yeah, they have gone up a lot since I bought mine too. I did get Qwen3.5-122b running on all-GPU (on a 128GB machine) under a minimal Arch Linux setup (I do my GUI work on a much cheaper box). It worked, but Qwen3.6-35b is performing almost as well and a lot faster. Still cheaper than a new Mac. Maybe not cheaper than a used one.
        • AbsurdCensor 3mo ago
          I've certainly thought about just moving the box to Linux, but it took far to long personally to get everything running under AMD and it works 'well enough' that I don't want to make the switch. I tried playing with GAIA on it, felt a bit limited, and now have Hermes up and running, and that seems to work quite well. All the tools are changing so quickly, it's sometimes difficult to settle in on 'what's best', so I certainly can understand folks that just want to pay for a AI subscription and be done with it.
  • dofm 3mo ago
    [dead]
  • anarticle 3mo ago
    Pros buy their own tools. This is why working for yourself is better than working for a corpo, you get to choose your weapon.
    • _doctor_love 3mo ago
      That's daft mate
      • anarticle 3mo ago
        Is it? When you're inside the fence you get some IT cast off from the last guy that was cleaned up and re-spywared from 2-3y ago (sometimes worse!). As a hired gun (data science/engineer), I have an m3 128gb 1tb all sliders to the right MacBook Pro that works great, no spyware, rocket fast, trains/runs small models. With Apple 0%, paid in a year. I generally can name my price so it pays for itself. If you're riding it every day, every little speed up counts. I don't wait on GitHub actions or external systems for my crunching. For that I have my local slurm cluster of mini PCs, 64 cpus and counting! I assure you there is a better world outside of the fence, I am unlikely to return.
        • _doctor_love 3mo ago
          I'm curious the last time you touched reality
          • anarticle 3mo ago
            Not really sure what you're getting at there, I was a hired gun for about a year, founded a company and am having a great time. Last time I was at a corporate gig, I got a 2+ year old base model MacBook Pro to do a data science. 16gb of ram is fun and all, but when you're using R you're going to have a bad time. During my hired gun period I worked with two organizations that used gitlab, and they're doing just fine. One place I had no problem using their cloud for bigger runs, the other had a permission system that was pissing me off so I solved my own problem and billed them enough it didn't matter anyhow. Like I said, having extra cpus for big R runs you don't have to ask anyone for has a value all its own. Happy computing!
• anax32 3mo ago
  I've just made a milestone on my project, moving away from AWS (budget) to self-hosted and the local models are so much faster than in the past. Beyond LLMs, having embeddings, image, video, audio gen available is crazy. Running locally is the bar; it's hard to make these things a service which scales.
• richbradshaw 3mo ago
  I’m keen to understand speed here etc etc. if I bought a Mac studio with 96GB - what can I realistically run, how’s it compare to fable/opus etc and how fast is it? Currently maxing out two Claude code accounts every x hours when working on large code migrations or setting up new iOS apps etc - most of time it’s fine but occasionally it’s mega frustrating!
  • pizza234 3mo ago
    [dead]
  • simonw 3mo ago
    I strongly recommend trying LM Studio - it's the lowest friction way to try out models, you can browse https://lmstudio.ai/models https://lmstudio.ai/models and click "Get" and then "Run in LM Studio" to download and run a model. With 96GB I'd start with the Gemma 4 and Qwen 3.6 models. Any of those should work fine.
  • AbsurdCensor 3mo ago
    I think currently you can only get the M3 Ultra Studio with 96gb, and for coding tasks, say you rub Qwen Coder on it (which doesn't need that much ram), it's not the fastest, something like 30-40 tok/sec. Probably better with a MacBook Pro with the M5 chip. There is a website for comparing different configurations and models: https://llmcheck.net/benchmarks https://llmcheck.net/benchmarks
• rmunn 3mo ago
  This is the kind of thing that Anthropic et al should be worried about. As it becomes easier and easier to run local models, the ceiling of what they'll be able to charge will get lower and lower. Not that nobody will be willing to pay $$$$$ per month, but a lot of people are going to multiply the per-month charge by 12 or 24 and say "Could I set up a local model for less than that, and have it pay for itself within a year or two?" And if a significant portion of customers decide to buy instead of rent, the companies whose business model is entirely centered around renting will suddenly find themselves hurting for customers.
  • themaninthedark 3mo ago
    Maybe that is why they are buying up as much hardware as they can? If their service is the only game in town.
    • otterdude 3mo ago
      Data Center providers are buying hardware, not anthropic. Certainly related but alot of the hardware purchased is just sitting in a warehouse waiting for a data center to get built.
  • indoordin0saur 3mo ago
    I'm curious when coding-heavy companies will start running their own on-prem AI clusters. Has anyone had the idea to sell something like 4 GPU machine an engineering team could throw in a closet somewhere and run whatever they want on it? I imagine this won't appeal to everybody but with the trust issues the hyperscalers have developed hoovering up people's data and using it to train their models, I imagine some will find value in a machine and model they have transparent control over including the option to walk over and unplug the thing.
    • CamperBob2 3mo ago
      Has anyone had the idea to sell something like 4 GPU machine an engineering team could throw in a closet somewhere and run whatever they want on it? I think that's basically Geohot's business model at Tiny Corp.
• embedding-shape 3mo ago
  Show us the resulting code of using them! :) I want to use local models, I have the hardware for it, but while trying them out as replacements for GPT 5.5 xhigh or Opus or other SOTA models, they aren't quite ready to be replaced yet, sadly. The quality and bumps they encounter just slows down the workflow so much, even screwing up tool call syntax sometimes. But, for smaller more well-defined workflows, or as straight "edit this part to be like this exact" edits, they seem more than enough. Still waiting for them to become mature enough to be able to replace what we have as SOTA today, I'd say it's ready to be switched over then. Speaking of local models, DiffusionGemma (and diffusion models in general) should not be slept on for local usage! Usually the problem locally is that the LLMs aren't efficiently making use of your hardware, unless you start batching requests and run many at the same time, but that require different approaches in general. Instead, diffusion models work much faster for individual prompts, and not by a small margin either. Today I finally finished porting diffusiongemma-26B-A4B-it support from Transformers into Candle, and together with some optimizations I now have it basically flying with ~450 tok/s (~19 it/s) in Candle during inference, instead of ~180 tok/s (~11 it/s) from HF's Transformers library. Even using vLLM with similar sized LLMs, I don't think I've ever gotten past the ~250 tok/s threshold for single prompts, exciting stuff for local models :)
  • zozbot234 3mo ago
    > Instead, diffusion models work much faster for individual prompts, and not by a small margin either. Diffusion models can't really be trained beyond low-to-mid size and have lower quality than an equally sized, plain one-token-at-a-time model.
    • embedding-shape 3mo ago
      As mentioned, I've just finished the implementation and started playing around with it, seems to be doing similarly well inside of my own agent harness as similarly sized "traditional" LLMs. Of course, neither come close to SOTA models, but I suppose if we can figure out the scaling issues you mention, we'd get a bit closer. The performance just feels like it's too good to quickly ditch diffusion. Do you have more info what those "can't be trained beyond low/mid size" issues are in practice today?
• cube00 3mo ago
  The challenge I have is getting a large enough context window so tool calls work reliably, the local models easily slip into hallucinated JSON tool responses and won't trigger the tools as a result.
  • glaslong 3mo ago
    Same here. I'm curious what others loving Qwen are doing differently, because it constantly hits this issue for me. It's been great for autofilling blocks, but difficult for me to use agentically.
• kordlessagain 3mo ago
  [dead]
• hypfer 3mo ago
  After having been a happy user of Qwen3.6-27B for a few weeks, due to being away from the hardware, I'm currently forced to use Claude Sonnet 4.6 It is such a downgrade. I don't understand how that's even possible. The thing has so many strongly-held opinions I did not ever ask it for, talking just way too much and generally feeling somehow dumber. Of course, being significantly larger, it will encode more knowledge, but that doesn't help me when I hate talking to it. And all that on top of the fact that talking with it costs real money. I wonder what it might be that makes me hate it so much. Maybe because it doesn't see itself as a tool but almost an equal? As if its opinions would have weight. Qwen too can act like an overeager intern, but if you tell it that it is an idiot, it will drop that ego. Not so much with Claude. In my experience, anyway. Anyway, point is: full ack on that headline.
  • kitd 3mo ago
    Funny that coding agents have personalities, including "that colleague" you want to avoid even if you know they're probably quite good at what they do!
    • otabdeveloper4 3mo ago
      That's exactly what RLHF is for. (In fact, "that colleague" might have even been the source of the RLHF training set.)
  • MostlyStable 3mo ago
    Curious if you have tried custom instructions. I was never quite as unhappy with Claude's voice as you appear to be, but there were several things I didn't like. A custom prompt fixed almost all of them.
    • clickety_clack 3mo ago
      I think it would be very hard to convince someone to pay $100/mo to go back to Claude if they have a local model up and running, particularly now that model improvement has basically been stalled for the last 6 months. It’s so easy to set it up for yourself now too with things like LM studio. That said, there will always be unsophisticated users who can’t figure it out, so there will always be someone there to pay.
• wxw 3mo ago
  > “if we are constrained by performance and price, what architectural tradeoffs do we need to make?” a question that so far has not really been asked in the mad token gold rush. To be fair, I think the labs are also interested in this (e.g OpenAI parameter golf). But the incentives are tricky. When the subsidies and tokenmaxxing era ends, local models will be essential.
• cautiouscat 3mo ago
  > I have no concrete scientific evidence of this - my own personal vibe metric of “is a model good enough” is, “do I have to double-check it against an API model”, and GPT-OSS was the first one where I started doing that a lot less often. The good old butt dyno! I’ve been eyeing local models more and more with Anthropic squeezing more and more on the subscriptions. A few comments on HN had me waiting until they improved more but this article makes me wonder if I should reconsider that. I’ve been doing some pretty niche development using a game and a script extender for said game. If these models can handle that, I’d feel good about switching.
• xienze 3mo ago
  The big caveat here is that these local models require you to invest some time tweaking your harness, AGENTS.md, and skills in order to get things roughly to the level you'd expect. But something like Qwen3.6-27B with web search capabilities and a good set of skills really is impressive! Especially considering that you can go wild and not worry about token costs. The other thing that people tend to gloss over is that you really do need to spend some $$$ on decent hardware. Yeah, you CAN run some 4-bit quant with heavily quantized cache on your 16GB card, but it's not going to be a great experience (I think this is where a lot of the "if you think it's gonna be any good, you're going to be disappointed" stuff comes from). Yes it's a lot of $$$ upfront but it's very much unknown when hardware prices are going to come back to reality. There's a lot of hopes and dreams that any minute now an H100 will be worth pennies because "that's how it's always been" w.r.t. computer hardware, but we are living in interesting times. So you can't just make the tired old assumptions that a Claude subscription over three years time will work out to be dramatically less than the value of some card three years from now. We STILL have basically anything with >=24GB VRAM appreciating in value, which is absolutely wild. What I'm saying is, the depreciation curve may very well be a lot less dramatic and fast than it used to be, going forward.
• sosodev 3mo ago
  I think this is overselling their capabilities. I've used Gemma 4 and Qwen 3.6 quite a bit on my strix halo home server. They're great models and the dense variants are significantly better, but they're still very far behind the frontier. If you boot up Gemma 4 MoE and OpenCode/Pi and expect to perform anything like Claude Code or Codex you're going to be very disappointed.
  • kristopolous 3mo ago
    You need to switch out the prompts and work with it differently. I posted this yesterday https://github.com/day50-dev/petsitter https://github.com/day50-dev/petsitter I use it with https://github.com/day50-dev/simple-llm-cli https://github.com/day50-dev/simple-llm-cli And modify the "tricks" until my evals get to good numbers. It's a model by model basis. This is what the larger firms are doing - they have custom prompts per model
    • sosodev 3mo ago
      Petsitter's default tricks doesn't seem to do much for Qwen3.6, right? JSON mode could be useful I suppose, but that's not really going to make it better at writing code. Do you have any other example tricks? I'm having a hard time understanding how I would apply them.
      • kristopolous 3mo ago
        thanks for the feedback ... i'll work on publishing them. I haven't include more sophisticated ones because they are complicated and I wanted to avoid the friction
• chrismarlow9 3mo ago
  You can use a frontier model to create a plan that's specific enough for a local model of a very small size to execute on. The more specific you are and compartmentalize tasks the "dumber" the local model can be. Edit: Obviously you'll be using more tokens but this is the trade off for running a smaller model and running locally. Similar to time memory trade off but in token economics. Sorry I need more coffee
• simonw 3mo ago
  I think gemma-4-26b-a4b and Qwen3.6-35B-A3B show that there's something very interesting about a local model that does mixture-of-experts (which helps a lot with performance) and has in the order of 30 billion parameters. These models are very capable, and use around 20-30GB of RAM while they are running. Provided you have 64GB of RAM that leaves space for running other applications at the same time.
  • chrisweekly 3mo ago
    Obtaining that 64GB RAM is a meaningful obstacle for many.
    • simonw 3mo ago
      I'm still amazed that you can run LLMs of this quality on a machine that costs less than $3,000. I used to assume that anything GPT-4 equivalent or higher would need $30,000+ of server-class hardware. That said... gemma-4-12b-qat is 7.15GB on disk so should run reasonably well in 16GB, that takes it down to MacBook Air territory https://lmstudio.ai/models/google/gemma-4-12b-qat https://lmstudio.ai/models/google/gemma-4-12b-qat
      • verdverm 3mo ago
        Second this notion. After picking up an OEM Spark and running qwen36moe/dense, I was thoroughly impressed with what such small models can do and the (reasonable) speeds you can get. I'm back to using open weight models via an API (wanted more capability for the time being), but will be getting more hardware soon (re: ds4-flash and the fable shot heard round the world)
    • frollogaston 3mo ago
      Not just RAM, VRAM, right? Though they're one and the same on the Mac.
• stared 3mo ago
  I really recommend Qwen3.6 27B. Make some tests, and its 8 bit version runs at 30tok/s when using llama.cpp with MTP and run on Macbook Max M5. I have 128 GB, but but 64 GB is well enough. https://github.com/stared/benching-local-llms-on-apple-silicon https://github.com/stared/benching-local-llms-on-apple-silic... When using benchmarks, it gives more-or-less the level of SotA mid-late 2025.
  • wizzledonker 3mo ago
    Did you mean 2025?
    • stared 3mo ago
      Yes, fixed
  • iagooar 3mo ago
    I run the exact same model, on the exact same hardware - amazing results. Pair it with good search skills (Tavily, Brave, Exa) and you have a near-SOTA model on your desk.
• ibizaman 3mo ago
  Tangential but reading on mobile, the font size in the code snippets are all over the place. I actually have the same issue on my blog. Anyone knows why?
• aliljet 3mo ago
  The problem here is always the cost-benefit. For $200/mo, you're receiving subsidized best of breed access. There's no model competing for that price anywhere. If a 27B param model is what you choose, show me your hardware! I would love to be wrong...
  • rsolva 3mo ago
    But for how long? The subsidized phase is probably short, and then what? I run Qwen 3.5 27 Dense om my old AMD RX7900XTX at about 45 t/s and barely use my Claude Code subscription anymore.
• 0xc0c0c0 3mo ago
  I have used local models (around 128 gb) and the big proprietary models, and while I do want local models to win, it's important we keep the expectations of local models realistic. There are many blog posts about how local models today can fully replace some of the proprietary models and in some cases its true for the much smaller proprietary models, its very clearly much more behind the larger models. You can be far more ambiguous with your tasks with the larger proprietary models as opposed to the local models. You can achieve the similar results with local models but you need to be much more detailed in your prompt. One of the biggest things about running these local models is that the harness matters almost just as much as the model too. Codex is optimized for GPT models, CC is optimized for Claude, Cursor has a great harness that works very well across these providers. It took me a couple of iterations of the different harnesses to find one that would work well with the smaller Qwen models to do local coding.
  • failbuffer 3mo ago
    So which harness did you end up choosing?
• wasimxyz 3mo ago
  https://canirun.ai https://canirun.ai
• anubhav200 3mo ago
  I have been using qwen and glm based models from last 2 years, ended up buying mutiple machines for the same. Overall i feel 24vram is a must have to get get performance (speed wise) to match hosted soln. I have 2 machines a 12gb vram one and a 24gb one. On 12gb vram i get around 50tps generation and 500tps prompt processing and on 24gb one i get 180tps generation and 3500tps prompt processing. I have different configs for different scenarios and I also use llama cpp manager manage all my configs (https://github.com/anubhavgupta/llama-cpp-manager https://github.com/anubhavgupta/llama-cpp-manager)
• maxothex 3mo ago
  [flagged]
• fg137 3mo ago
  > I have a 2022 M2 Mac with 64 GB RAM I closed the article after that. The author has no idea what a privilege it is to have a machine like that for personal use, and how 99% of the population are not going to afford a setup like that. Just some back-of-the-envelope maths will tell you that a $20/month Claude subscription makes much more sense financially.
  • orf 3mo ago
    99% of the population don’t code using models, local or remote. So that’s a useless metric. What % of developers could afford an older MacBook model, second hand? Far, far more than 1%.
    • fg137 3mo ago
      could or will? I am pretty sure even among software engineers, much fewer than 1% are going to spend their money on that. Most software engineers know how to spend their money responsibly.
      • orf 3mo ago
        That’s not at all what you said though, was it?
        • fg137 3mo ago
          Read it again.
          • orf 3mo ago
            > how 99% of the population are not going to afford a setup like that > could or will? much fewer than 1% are going to spend their money on that. It’s ok to change your point, you don’t need to get combative. Not that it makes any difference, given their ~10% market share.
    • DiabloD3 3mo ago
• fridder 3mo ago
  Is there a local harness designed around the local model use case that is claude code like? Opencode has been problematic at times, pi works for one off for me but not back and forth conversations with the LLM. Considering I only use Qwen or Gemma models I'm close to just writing my own at this point
• segmondy 3mo ago
  It's more than good. As of today, it's great. Those models listed in the blog are horrible compared to what you can run today, There's absolutely no reason to run those, you have Qwen3.6, Gemma4, and plenty other sized comparable models. If you're resourceful, you can even run SOTA models. KimiK2.7, MiMo-V2.5/V2.5-Pro, MiniMax2.5/2.7/3, DeepSeekV3.1/v3.2/V4-Flash/V4Pro, GLM5.1, Step3.7-Flash, Qwen3.5-397B, Qwen3.5-122B, gpt-oss-120B
  • agile-gift0262 3mo ago
    > Qwen3.5-122B do you find Qwen3.5-122B to be SOTA-level? I moved from it to Qwen3.6-27B (both Q8), and I prefer 3.6-27B, and it leaves me room to spare for other small models
• c0rruptbytes 3mo ago
  I don't know about good, I use a lot of local models and they're still pretty painful to run locally You have dense models (qwen 27b, gemma 31b) who are pretty smart, but pretty slow You have MoE models (gemma 26b, qwen 35b, north mini code 30b) who are pretty fast, but make a lot of mistakes You need a lot of memory to run these well, quantization makes tool calling weaker, so most run at 4 bit quants and are wondering why it kinda sucks and that's because you've essentially lobotomized the model (I recommend unsloth quants, i recommend 6bit for MoEs and 5bit for dense) So you need a lot of compute to make the pre-fill fast, you need bandwidth to make the decode fast, you need a lot of memory to hold everything - lot of ifs On top of that, your laptop becomes a loud hot churning machine, it's uncomfortable to work with. So are they good? not really. Do they work? yes edit: just wanna clarify - i think open models are the future, i think they're super important, i'm contributing constantly to the ecosystem - i think people should play around with these models, i think people should use `pi` and learn how it all works - but don't download a model expecting it to be good out of the box, you will have to tune and configure a lot of stuff to replace a "coding agent" that most people are using models for
  • heipei 3mo ago
    Depends on what you mean by "local". On your Macbook, large dense models like Qwen 3.6 27B will be slow, sure. On a local workstation with a dedicated RTX card you can get > 100 tps, which is more than good enough to work with it, and faster than cloud models in many cases.
    • jstanley 3mo ago
      But how smart is it? All the people running local models never seem to mention that they are way dumber than cloud models. I don't care how many tokens per second of nonsense it can generate.
      • myaccountonhn 3mo ago
        Its not going to be as good as Claude, but if you know what you're doing, it may be good enough to get your work done.
• monegator 3mo ago
  I've been trying local models for the boring stuff you might be thinking about: writing small docs. So i've tested a couple, and the speed is finally impressive. My colleague uses paid tiers of claude and GPT, and the speed is comparable. Maybe even slightly faster on my end. The problem is: i'm running the model on my work laptop, a 12th gen i5 with 16GB of RAM (which, you know, i asked to upgrade to 64, but that was right at the time of the great RAM shortage of the '20s) so i'm pretty limited in what i can use. And this is running alongside the usual suspects: Web browser hugging 1.5GB, MPLABX hugging 3, windows taking at least 5 just to sit idle, thermal throttled to 1GHz ... And yet its speed is comparable to a paid service. A lunch's worth of tokens vs a few cents of power. So, what i found, what i fount... What i found is that i need AT LEAST 16k of context window, otherwise they will halt when i pass a small C file for analysis. And coding models will shit the bed with 4k. But we all know that, context size is King. I found out that Qwen will keep looping while thinking, but that's not a surprise to you, either. But give it enough time and you will get an useful answer. I was hoping to using it as a better warning system for some languages, but i fear i need muuuch more context size, because i tried to feed a file that had a function with an endless loop: At 4k context it almost shit the bed if i gave it just the offending function, then told it where to look at. At 16k context, with the whole file, it needed some guidance to what the problem was, and after 10-15 minutes of thinking it found the issue. Problem is, it kept second guessing itself for another 20 minutes on the same unrelated thing before giving the output. For which the fix was wrong, but the semanthic was correct. Good enough. Maybe it will be faster if i don't ask for a fix (which i didn't i just asked to look for a specific issue) Wish i had 3 times the RAM so i can see what happens with more context. Then i gave it the task to analyze a C file to make an API document. It took half an hour, but then i had a good starting point, which i had to keep changing because it would confuse commands with IDs and things like that. This was the Qwen 3.5 9B model. I then tested Gemma 4, being impressed at the tokens per second it gives on my Pixel 8A. Same tasks: same issues with short context, with long context it gave absolutely useless answers when looking at code, but it took 1/3 the time of qwen. In producing documentation, instead, it was much faster, and it never hallucinated data. Good. in 15 minutes i had everything done. Not bad for stuff running on a business laptop, while doing actual work. Tomorrow i will try Qwen 3.6, let's see how it goes..
• ltononro 3mo ago
  Good depends a lot. If you are in the token maxxing hype you will probably find these models very bad comparing to SOTA, unfortunately. The good news might be: opensource models are now good (enough) for day2day usage. But is it really? I feel that companies will always naturally strive for the best and use the SOTA (as long it is not too expensive). I see OSS models being a good backbone for companies in the future that have validated workflows and could use those for privacy or to spare costs. IDK, might have gone a little bit off-topic here.
• Tharre 3mo ago
  I've been running Qwen3.6-35B-A3B (and 3.5 previously) locally and it's a great model for many small tasks, probably a significant chunk of what most normal people are using LLMs for right now. But for coding in a harness? In my experience it's unusable even for small projects. It just gets hard stuck at every little problem, wasting hundreds of thousands of tokens trying to make a convoluted solution work instead of doing the obvious thing. Or it will spend hours trying to reason through a fairly simple code flow, incrementally adding debug print statements, only to get confused by the output and then editing completely unrelated code that it convinced itself is the problem. I've tried instead giving Sonnet the problem description and code and have it come up with a detailed plan that Qwen should implement, but doing that actually consumes a significant amount of tokens compared to just telling it to implement everything, and the results are honestly not that much better. There are just too often subtle issues with the plan that Qwen doesn't recognize when implementing, but make the resulting solution it comes up with unusable.
• daniban 3mo ago
  With Apple silicon and now the RTX Spark there are real discussions whether local AI is the future. The only problem is Western open source models are so far behind. I genuinely feel there's a push to fix this. Gemma is getting more frequent releases and Nvdia is quietly creating very cool small models. I hope both the hardware and models catch up and local really does emerge.
• iagooar 3mo ago
  I love running two models locally: qwen3.6 27B 8bit (dense) and qwen3.6 35B 4bit (MoE). The 27B is the smarter, more reliable one - but it is slower. The 35B is faster, still very smart but below 27B, a bit less reliable. The reason is the MoE - Mixture of Experts architecture, which only activates a subset of parameters, making the model much much faster. I run the 27B on a MacBook Pro M5 Max + 40 GPU cores + 128GB RAM (well, on this beast I can have 27B + 35B in memory at the same time with headroom for all the other stuff). But because this is a laptop, it is not possible to run local LLMs all the time - it just gets too hot and too loud. What excites me more: I run the 35B model on a MacMini M4 with 64GB RAM. It is fast, it gets a lot of work done (e.g. it scans, extracts and classifies my emails, it watches the mailbox all the time and does work). I also use it as my private Hermes assistant ("when is the next Starship launch?", "who is playing today at the World Cup? Give me some trivia"). Next step I am planning is a RTX Pro 6000 Blackwell workstation I can put in my basement. I want to run qwen really fast, with multiple threads / prompts / agents at once. And MAYBE if the budget allows, a 2x RTX Pro 6000 setup in order to run DeepSeek v4 flash on it (to run research on it).
  • Barbing 3mo ago
    Did you get a Brave search API key or something for that “Hermes”?
    • dghlsakjg 3mo ago
      Hermes is just an agent that can be setup for whatever you want (coding or more commonly personal assistant ala clawdbot). You can set it up with any of the standard tools and MCPs like brave or tavily for search.
    • iagooar 3mo ago
      Yes, Brave search is one of these services I highly recommend paying for, the search they provide (similar to Exa, Tavily) is what makes an "OK LLM" become super smart.
    • nickthegreek 3mo ago
      I have my mine setup with a searxng instance I run in a docker. Works great and costs zero.
• Veer_Pratap08 3mo ago
  [flagged]
• ngxson 3mo ago
  My 2c: I think the "cloud vs local" debate is (maybe) a false dichotomy. In my experience, I use a hybrid approach and I've seen a huge productivity boost from it. The cloud-based models are fine for big and complex tasks, but the pricing is ridiculous for small stuff—like summarizing a discussion or fixing a small bug. And cloud and privacy have never been a good match. As an example, this comment itself was written with the help of Qwen3.5-4B running locally with an extension on top of llama.cpp default web UI [1]. The extension injects my browser's context directly into the conversation, which allows me to summarize things and draft up comments quickly. Speed is pretty acceptable for the size: ~5s TTFT and ~100 t/s generation, all running on a Macbook M5. And when I want to run bigger tasks, I don't just stick to one provider. Apart from well-known closed-weight providers like OpenAI or Anthropic, I also experiment with open-weight models like GLM-5.1, DeepSeek V4, and Qwen3.6-27B, which provide quite good results for the price. I'd argue both have value, and I don't see why anyone needs to choose one exclusively. Anyone else doing this? [1]: https://github.com/ngxson/llama-companion https://github.com/ngxson/llama-companion
  • phainopepla2 3mo ago
    Why not just use DS V4 Flash for the small stuff? Very fast and extremely cheap.
    • ngxson 3mo ago
      The dsv4 flash is 158B params in total. It is possible to run locally but will require all my system RAM. Also, a lot of my day-to-day tasks perform the same on both small and bigger models: summarize a web page, draft a response, translations, quick web search, etc.
      • phainopepla2 3mo ago
        Sorry, I meant non-locally. I'm assuming privacy is not a concern since you mentioned using Deepseek already. The cost of V4 Flash for small tasks is so minuscule as to be almost free, and you don't have to deal with a churning laptop (or even buying a high-end laptop, for someone who doesn't already have one). I guess what I'm really asking is, what's the advantage of using these small local models if privacy isn't a concern?
• jotato 3mo ago
  I currently have a desktop with a 4060 ti (16gb of vram). Most models I have tested that fit within that are not good enough for anything other then type completion (in regards to coding tasks) I have been considering getting the 58gb Mac Mini but that is a decent amount of money to spend without confirmation on a) how fast is it and b) will it work for well-defined tasks.
• fl4regun 3mo ago
  In my experience, with a system of 32GB RAM and 24GB VRAM, no, they aren't that good.
• prlin 3mo ago
  If you wanted to do some research or learn about post training and agent harnesses, is that a good option with these local models? What hardware is recommended, or easiest to go with a Mac Studio with 64GB+ RAM?
• dejawu 3mo ago
  If vibe-coding is hopping into a self-driving car and telling it to take you anywhere you can get a coffee, then I use coding agents more like a bicycle - they let me get further faster than if I'd walked, but I still have to decide where to go and how to get there, and I still have to pedal. I don't vibe-code, but I do decide what to implement and what patterns to use (perhaps asking the model to analyze and give advice on this first), then I have it handle the nitty-gritty of the implementation itself. For this usage style, the latest local models are as good as having Claude at home. I won't say it's been _easy_ (I ended up implementing my own harness to accommodate the idiosyncrasies of local models), but I will say that for the effort, having a coding agent that's essentially free to query as much as I want has been life-changing as a dev, especially when it comes to working on side projects. Knowing that my agent will never get worse in quality, suddenly cost more than it does now, or be suddenly made unavailable by external factors, was absolutely worth the trouble. And on top of all that, I can't believe it's as good as it is.
• malkosta 3mo ago
  The problem with QWEN is that it just can't edit files reliably, I had to hack Pi all over to reduce the pain, but still far from perfect...does Gemma 4 strugle on this?
• drchaim 3mo ago
  really want to try local models, but I don't have the hardware yet. Probably I'm the only one here still using a Mac Mini m1 8gb 2020. :/
  • tennfown 3mo ago
    I have some decent specs, but I’m stuck with AMD graphics card which I’ve been told is a non-starter
• throwarayes 3mo ago
  I am happy to pay OpenAI for a cheaper model a few generations behind. But they deprecate models aggressively. They push you to bigger and smarter models, when 95% of my work doesn’t need it. I’d love it if model providers just let old models run and let us pay less, but the deprecation makes me want to look into local models.
• wrxd 3mo ago
  I wonder how much local models hallucinate. I am getting almost daily an "Honest answers: I made that up." reply from Claude Opus when I challenge some silly thing it's trying to do.
• valisvalis 3mo ago
  There are good use cases for them for sure, the Gemma 4 Good hackathon a while ago showed how local models can solve problems in health and education in areas with low connectivity or small infrastructure.
  • LolWolf 3mo ago
    what were your favorite projects?
• huydotnet 3mo ago
  I love that local LLMs are being discussed more often on HN recently. But for the post, I find it strange that the author claimed they were working with local models from day 1, but wrote a post that still links to Qwen2.5 and Qwen3 in mid June 2026.
  • zahlman 3mo ago
    Why shouldn't the author mention models that people might not have to buy a new computer to use?
    • huydotnet 3mo ago
      One don’t have to buy a new computer to run Qwen3.6 or Qwen3.5 (35B A3B), given that they can already run Qwen3 30B A3B. In fact with a 64GB mac, you can run pretty much all of the latest Qwen models. Also, anyone who has been following local LLM are well aware that the quality and performance has become way way better since Qwen3.5
• ZionBoggan 3mo ago
  This is actually a really insightful post !
• abalashov 3mo ago
  And if you want to dial in a setting in between: I've switched to Kimi K2.6 (now K2.7) and DeepSeek through OpenRouter and Reasonix for pretty much everything, with no discernible loss of analytical quality or utility. However, like many commenters, I don't really believe in vibe-coding, long-horizon agentic one-shot agentic coding, etc. and do not use LLMs for huge generation tasks that involve designing things end-to-end. I also have an MBP with 128 GB of unified memory and do quite a bit of Qwen3.6-35B-A3B. No, it's not as smart as the aforementioned models, to say nothing of frontier, but many people seem pleasantly shocked by the number of banal tasks that do not require these.
• jingw222 3mo ago
  open source must win
• azzzxcc123 3mo ago
  [dead]
• k__ 3mo ago
  I tried some smaller Gemma4 and Qwen3.6 quants on my MBA with M5/16GB and had like 20-60 tokens per second. At 60 it felt pretty okay and that hardware is on the lower end. I'd assume a Mac with 32-64GB memory would get some reasonable results.
• WASDx 3mo ago
  Looking at some benchmarks, the latest ~30B Gemma/Qwen score similar as Claude or GPT versions that were released just one year earlier. That's crazy progress. I can't imagine how it will be in a few years.
• aquarious_ 3mo ago
  I support local models and enjoy playing around with them, but even for personally development it is just more viable for me to pay $200 a month to Anthropic for the latest models. It seems to me with the cost of hardware needed to run local models that, for now, it is pure hobbyist and exploratory (which is fun in its own right)
• ridruejo 3mo ago
  Local models are one of the main drivers for our installer / Desktop app for OpenClaw https://holaclaw.ai https://holaclaw.ai (disclaimer I am one of the founders). The smaller models are really only suitable for the most basic tasks, but if you have 32gb-64gb you can get real work done (ie complex web workflows) without third party hosted models
• holoduke 3mo ago
  Good? My Macbook m3 with 36gb locked up after it filled all memory with Gemma4. A bit useful yes. But it eats all resources. For local models to be useful we need at least 128gb of system memory and 512gb of video memory. Plus 8 times the compute of a single 5090/h200
• gregwebs 3mo ago
  All these conversations seem like they are missing talking about planning vs execution. I want the best possible frontier model to plan out my changes. I also have a 2nd agent that is a frontier model check the plan. Then at that point the implementation can be done by a lesser and possibly local model. The frontier model can still do a final code review on the implementation of the changes. Claude code supports this by setting the model to "opusplan"- it will automatically use Opus for planning and sonnet for implementation. This was completely necessary with the fable release. I was able to do this with fable and it was necessary to avoid getting quickly rate limited. In settings.json: "env": { "ANTHROPIC_DEFAULT_OPUS_MODEL": "claude-fable-5" }, Obviously have that set to "claude-opus-4-8" now.
  • noveltyaccount 3mo ago
    I do this with Codex 5.5 for planning (specs, technical design, and task list); and Qwen 3.5-35B for task by task build out. It requires more hand holding and makes more mistakes than using Codex for everything, but it helps me spread my $20 chatGPT subscription pretty far.
• Rekindle8090 3mo ago
  [dead]
• huflungdung 3mo ago
  [dead]
• pjmlp 3mo ago
  Only if blessed with enough RAM and disk space, > 64 GB RAM and 1TB storage Ah ok, not something regular joe and jane happen to have lying around at home. Additionally the whole configuration is still very much low level, bunch of CLI commands, and if the model doesn't fit for the task at hand, it starts allucinating, generating gibberish, whatever.
  • sparkling 3mo ago
    Even if i had such a machine, im not sure i would be willing to sacrifice 80% of my RAM and 50% of my disk to run a semi-okay model locally.
• bthornbury 3mo ago
  the qwopus 27b model is good for grunt work style tasks, even across multiple files. Piping a bunch of things through, small factoring changes, stuff that just takes time to type out. I wouldn't rely on it for large stuff like codex though. I haven't tried out deepseek/kimi, if we could run those locally it would be great.
• andix 3mo ago
  Because I've seen too many people spending a lot of money on expensive hardware, without really using it in the end: Most of those models are also available via Openrouter and many other platforms. Dirt cheap, and much faster than on consumer GPUs. Perfect to try and compare the different options.
• jszymborski 3mo ago
  I run local models and they work fine for me, but specifically for use in coding harnesses, I'm having a hard time. Tools tend to end up in the same loop, trying to `ls` the same folder or `grep` the same file, over and over and eating up the whole context. Super hard to get it to do anything but that. Any tips?
• ta-run 3mo ago
  Not related, but, I can't seem to get my copilot-cli (office is an MS shop) use qwen3.5:27b on ollama for some odd reason. After the recent changes to usage, I've spent an annoyingly long number of hours trying to get this to work.
• b3ing 3mo ago
  They are ok for simple stuff, coding is weak, chat is alright, writing is ok. But I had many of them write stories for ideas and they kept using the same names regardless of what the story was about. I can’t complain, it’s free. Can’t wait till they get even better, but for local image generation they are good, slow but just create a bunch in the background while you do other things otherwise it’s like 14.4k modems
• frollogaston 3mo ago
  "Good" refers to the speed and not the quality. There's so much hype about Macs being great for LLMs, but nobody seems to be seriously using them for that because the open models are unfortunately so far behind.
• jlengrand 3mo ago
  Just wanna say it's always fun and nostalgic to see authors pass by here who I was reading back when I started my career. I was reading Vicki's blogs way back, even remember learning some email parsing in python from her over 10 years ago. TY!
• osigurdson 3mo ago
  Running AI on timesharing mainframes does seem like an odd final state for the world.
• RishiByte 3mo ago
  [flagged]
• deleted 3mo ago
  [deleted]
• aplomb1026 3mo ago
  [flagged]
• MrKoby07 3mo ago
  I think a lot of people just don't have specs like that, making it still painful.
• eugmai86 3mo ago
  [flagged]
• robertkarl 3mo ago
  You can trade off latency / accuracy / cost for any ML task. And with the local models.... the cost is free. Having a local Qwen check another Qwen's work increases the accuracy quite a bit at the cost of more latency. You can't have your cake and eat it too. In benchmarking local models, I'm having success increasing even a 9B qwen's score on terminal-bench adjacent problems, just by asking it to plan and handing the plan back to qwen with a fresh context. Try it with Qwen3.5, unsloth Q4+, and a thinking budget of around 1024 tokens.
• blobbers 3mo ago
  Have you tried optimizing for MLX? It seems like a waste to have neural cores and not use them. I've often wondered why the hype around apple neural core when 99% of software doesn't use them.
  • genxy 3mo ago
    Yeah, first think I looked for on the post was MLX and it wasn't there. https://github.com/ml-explore/mlx-lm https://github.com/ml-explore/mlx-lm Having used half the systems that Vicki mentioned, mlx was the best balance between power and ease of use. Just a pip install away.
• atulmy 3mo ago
  Exact reason I'm building csuite.so, do check it out and let me know if you need early access!
• nikagrawal121 3mo ago
  I tried for my legal AI application that I'm building and it was able to do majority of the tasks. I used gemma4:26B
• ptx 3mo ago
  > Security: I run every Pi session in a Docker container and give it permissions only to bash so that it can’t run Python code or do web browsing How does that work? The script in the post references the file "docker-compose.sandbox.yml", but I don't anything about what that file does. The post that this one links to, that it's based on, says that Pi doesn't do proper sandboxing. Presumably bash can still execute other binaries, otherwise it would be fairly useless. What stops it from executing Python? Or opening a network connection and downloading Python?
• Lapsa 3mo ago
  [dead]
• jauntywundrkind 3mo ago
  i'd love to get to a point where big models can launch subagents that are fast and local. there's a lot of focus on token rate, but just as much, the way cloud providers have other latencies & processing styles not optimized for latency (running large batches all at once), and i think local might have some real wins. Gemma 4 seems already on the right track. lfm2.5-8b-a1b (https://www.liquid.ai/blog/lfm2-5-8b-a1b https://www.liquid.ai/blog/lfm2-5-8b-a1b) and DiffusionGemma seem to both be very high token rate. but getting that latency down, so that a series of tool calls can happen faster, would be a real win. I think especially with good prompting that becomes much more possible. One caveat, I have absolutely no patience for a lot of subagent systems, like opencode, where the subagent is walled off and incommunicatable. My subagents really should be their own session, that i can deal with as I please, with some MessageChannel like offerings/tools available to them. Ideally with modes where messages auto-flow in and out, and modes where I can be a gate-monitor. https://developer.mozilla.org/en-US/docs/Web/API/MessageChannel https://developer.mozilla.org/en-US/docs/Web/API/MessageChan... Not really super related but MCP has been working on Events for a while. That ability to respond fast would be great. https://github.com/modelcontextprotocol/experimental-ext-triggers-events/pull/1 https://github.com/modelcontextprotocol/experimental-ext-tri... Asking local to be fast feels like an obvious folly, but given how much better small models have got, and seeing these models tune themselves for speed: I want to hope!
• minton 3mo ago
  I’m glad people are looking into this because I do think it’s the future. However, why would you not take advantage of the heavily subsidized frontier models while you can. It’s obvious that they’re gonna have to raise prices at which point it might make sense to consider local models, but not today.
  • fendy3002 3mo ago
    Curiosity or anticipation I think. I have tried it in the name of those 2 factors, because when the frontier model price increase happens and we don't know anything about local models, we're screwed
• deleted 3mo ago
  [deleted]
• deleted 3mo ago
  [deleted]
• 0xbadcafebee 3mo ago
  Local models have been good for a while. But this being the HN echo chamber, people here think that local models can only be used for coding, and are expecting Opus 4.8 on their iPhone. Turns out AI can be used for things other than just coding. Even tiny models (<4B parameters) can do tons of useful things on local devices. Search, index, summarization, troubleshooting, crafting documents/formatting, image analysis, transcription, object identification, robot navigation, text-to-speech, speech-to-text, browser/window control, MCP/tool calls, and much more. Larger models just do more complex reasoning. But if you want them to be really good, you need a beefy Mac. They have the best combination of memory bandwidth and RAM to allow medium-sized models to run at speed. GPUs have less memory but more bandwidth, and AMD iGPUs have more memory but less bandwidth. The Mac is the best compromise on the market today. Once you do have a beefy Mac, you want to run a dense model. This gives you the best possible result with the system you have. You can go MoE for faster results, use cutting-edge inference techniques, parameter tweaks, etc. But a basic dense model (at Q6 quant) on a big-ass mac will serve 90% of your coding needs.
• jmyeet 3mo ago
  It's not "good". A more accurate description would be "sometimes useful and not far from being good". The author is using pretty small models. There have been a lot of improvements that scale in any case (eg MTP) but ultimately this is still hardware limited by 3 factors: 1. Memory bandwidth 2. VRAM size, which limits the size of a model you can use effectively. Yes you can swap but then you're taking a performance hit; 3. Raw FLOPS, including quantization. Apple here is interesting because they have a shared memory model and you can buy Macs currently with up to 128GB of RAM (previously 256/612GB on Mac Studios, both discontinued). New M5 Mac Studios are expected in Q3 but that's not guaranteed. It may take until next year Depending on the chip, Macs top out at ~900GB/s. A 5090 or 6000 Pro has 1800GB/s. A B100 is at like 3.2TB/s. A 5090 has, depending on how you count, 5-7x the FLOPS of a M5 Pro so a 5090 is still better than any current Max... except for the 32GB limit. NVidia aggressively segment the market by limiting VRAM. The RTX 6000 Pro is basically a 5090 with slightly more CUDA cores and 96GB of VRAM instead of 32GB for $10-11k instead of $3k. So let's project this into the future a little. The M6 Ultra/Max may well be 1TB+/s memory bandwidth with much higher FLOPS and thus actually be competitive for larger models. A 6090 in the current market will probably still have 32GB of VRAM if I had to guess. Maybe it goes up to 48GB. But anyway I think we're only 2-3 years away from sub-$5000 hardware that does 100-300+tok/s on models larger than 31B. And that's going to be a game changer.
• pornel 3mo ago
  [meta] I wonder why people have such wildly different bar for what is "good" agentic coding? In a way, it's absolutely amazing that we've went from "Playing 'Set a Timer' on Apple Music" intelligence to something that may pass the Turing Test, but in practical terms the small models are still far from what I'd call "good" for more than a tech demo. To me, 7B models are just a fuzzy echo of Wikipedia. Gemma models at 4 bit are too clumsy to even reliably generate JSON for tool calls or copy a line of code to apply a patch. Qwen needs so much detail and babysitting to stop it from doom looping or losing the plot, that the instructions that I need to give are usually longer than the code I end up keeping. Is there some magic prompt that I don't know? Do other people just have a lot more patience, or way lower expectations?
  • papersail 3mo ago
    I had similar doubts. I think expectations differ because the workload differs. For small scripts, glue code, or simple CRUD changes, smaller models such as Qwen3.6-27B can work wonders than they do on a larger, messier code base.
  • verdverm 3mo ago
    There is a lower bar (that gets lower over time), but ime, the config you are describing is too low still. qwen/gemma in the 27/35B range @fp8 are better than gemini-2.5, but less than gemini-3.1, you can run DS4-flash @fp8 on two DGX spark, and things keep becoming better. DiffusionGemma came out recently with 4x token gen speeds. tl;dr - the models you appear to be trying with are too small or too quant'd
  • cheschire 3mo ago
    Haves and have nots. We aren’t wealthy enough to have the hardware that would make this good. The people who have the money to buy a spare maxed out Mac mini just don’t get it. I see lots of folks with RTX 6000’s in threads like these. Or any RTX card that ends in “90”. Cloud AI is what allows the proles to participate in the broader AI conversation, but not these AI conversations.
    • monegator 3mo ago
• matrix12 3mo ago
  gemma:12b at 75% of frontier? Yeah....
  • etoxin 3mo ago
    I think 75% is about right. It calls tools pretty well and has a good knowledge base. It's absolutely not 90% there, but 75% feels right.
• Mr_Eri_Atlov 3mo ago
  I think this is a pivotal moment for LLMs. Gemma 4 and Qwen3.6 27B aren't perfect, yet they are such a step forward from the previous generation that it's both feasible to get stuff done locally with patience and very likely that future releases will subvert cloud capabilities entirely. Plus, they have definite reliability advantages over cloud models that can be wiped out by a government order or lobotomized to handle traffic surges.
• mohamedkoubaa 3mo ago
  I wonder when a cheaper consumer grade inference chip will hit the market. The general purpose GPUs have much more silicon and complex firmware than what's strictly needed for inference
• bayshark 3mo ago
  Hey everyone, made a local LLM, configured for Home Assistant called Selora AI. Specs: qwen3_17b_base.Q6_K.gguf selora-v047-answer.f16.gguf selora-v047-automation.f16.gguf selora-v047-clarification.f16.gguf selora-v047-command.f16.gguf The full base model and LoRA adapters are only 3.5GB Capabilities include configuring for smart home setup to help with answers, clarifications, commands, and creating automations in Home Assistant. The models with the LoRA adapters were made with lean scripted data made specifically for Home Assistant. A lot of work was put into this, feel free to give it a try and happy for any feedback! https://huggingface.co/selorahomes/Selora-AI https://huggingface.co/selorahomes/Selora-AI
• infogulch 3mo ago
  Anybody used a tinybox? https://tinygrad.org/#tinybox https://tinygrad.org/#tinybox The most "affordable" option is red v2 with 64GB GPU ram and costs $12,000. This is only ("only") 1.5x-3x the price of a beefy desktop (https://pcpartpicker.com/builds/ https://pcpartpicker.com/builds/), and could crush inference work even on bigger models. It could support coding tasks for a small team of developers, or run an AI agent for every person in your household...
  • pornel 3mo ago
    64GB VRAM is too little to run good coding models IMHO. May be useful if you need voice models or run some slightly-smarter-regex batch processing or RAG workflows. Perhaps you're supposed to buy 4 or 8 of these and split inference across them. If you have $12K to spend, you may be better off with DGX Spark or a Mac with 128GB VRAM. That can (barely) fit DeepSeek V4 Flash.
• lthi747 3mo ago
  Maybe it is good but it is very difficult, or at least with regular computer. For users like me with 16GB laptop it is almost impossible task.
• polotics 3mo ago
  So I've made this [me+vibe+tests]-coded Android alarm app called Promptly, and as Gemini-CLI on the Google Pro subscription is getting google-killed on June 18th, I set up two branches, one for Antigravity+Gemini3.5 and one for Pi-coding-agent with Qwen3-Coder-Next... Running the same prompt on both with the same .md memory state... Gemini3.5 is more "intelligent" but Antigravity gets it to decide to go on tangents that are quite time and token-consuming I think. Nice casino machine. Pi+Qwen3 (~80GB, llama.cpp) is like vibecoding about 1.5 years ago, when you had to babysit, structure your program to have self-contained chunks, and keep an eye on all the cross-cutting concerns to not trip it up. When it works it works fine and when it fails it's my job to ensure it fails fast. The code is about 10'000 lines of Kotlin in total so it already takes some effort to keep it simple for the AI. It's not a slopped quantity of code, i got solid feature creep :^) https://play.google.com/store/apps/details?id=com.sixteenam.promptly https://play.google.com/store/apps/details?id=com.sixteenam.... ...hat tip to the recent copycat squatter btw it's an honor!
• delis-thumbs-7e 3mo ago
  Nobody asked, but I don’t think any of us should be using SoA models to code or to do pretty much anything at all. Instead we should develop open models to work on specific tasks and learn to code, write, draw etc. using fingers made of bones and brains made of flesh. Big corporations and research facilities can run them to generate code or math or whatever, with a bunch of specialists to check the output to be correct. Then again, even that might not be worth the costs (e.g. OpenAI’s 36B$ net loss last year), when the open models are so close and the whole AI scheme is running out of scams to pull. There’s a lot of things we could use even quite small models for, which would not need an insane amount of computing power and memory, but too few of us is really researching them.
• aleksandrm 3mo ago
  Clickbait title, because running local models is still not good now.
• mrkn1 3mo ago
  [flagged]
• K0IN 3mo ago
  In a day to day base i host Qwen3.6:27b, but i *Really* want to host deepseekv4 flash, its such a "good" model for its size/speed/price. I really wonder when companies will start hosting theire model for everday tasks on prem, cause its good enough (and realative cheap), instead of paying subscriptions for all devs.
• angry_octet 3mo ago
  Programmers are used to paying nothing for tools. A basic laptop (SSD, multi core, 16GB of RAM) is hugely powerful if you are building in C/C++/Rust, even python. But all of a sudden it's no good, and we're back to using someone else's computer, hiring our tools every day. Worse, we get a different model every day, and maybe we aren't allowed to borrow the good tools some days because some mafioso are shaking down the manufacturer. Most other trades need to invest significantly in tools. If you want good tooling, you really want 64GB of GPU memory (e.g. 2x 5090) and 96GB of RAM. If I'm paying $200k for an expert engineer then $50k every other year for tooling seems pretty reasonable.
  • rsanek 3mo ago
    Who's paying the $50k? I don't see how it makes sense to pay that much for a home-grown setup when I could pay <$5k/year total for both of the two best frontier models at effectively unlimited usage.
    • fragmede 3mo ago
      > best frontier models at effectively unlimited usage. It would've been easy to spend $5k on Fable in the short week it was available. If that's the direction things are going (we can assume GPT-6 to be if similar class) $5k's not going to get you "best frontier models at effectively unlimited usage".
      • angry_octet 3mo ago
        I know some orgs that are already spending more on tokens than developer salary. That's what unlimited use leads to. Teams of agents running on expensive models, agents designing and running test suites with barely any oversight.
    • angry_octet 3mo ago
      This is for clients paying millions for deliverables, with high stakes deadlines. Expertise in the domain is in short supply. Data and model control is very important, so relying on AIaaS was already risky. You can LLM enable engineers without big AIaaS risks. A 16C Ryzen 128GB with 96GB Blackwell is ~US$16k, quite reasonable for a worker billed at $300k. In fact so reasonable it's worth having AI enabled backend for lots of things.
• Computer0 3mo ago
  I have 16GB VRAM and 96GB Ram on all my computers and I do enjoy local models. I would not use them for coding, though I have experimented with it, it is largely a waste of time on my hardware. I love local chat with different models however, when using the model in this way it is much easier to experiment with the largest models near the limit of your hardware, and I do find it useful on the airplane somewhat. I have also used local models for data classification tasks and let it run over the weekend etc and the results were acceptable.
• noveltyaccount 3mo ago
  From the recent Nvidia & Microsoft announcement about new chips for consumers: > “Our goal is to deliver unmetered intelligence to every home and every desk with Windows,” said Satya Nadella, chairman and CEO of Microsoft. “RTX Spark marks a real breakthrough towards that vision.” Makes me optimistic that those two companies are going to keep investing in quality local models.
• hottrends 3mo ago
  [flagged]
• androiddrew 3mo ago
  $2600 will buy you two AMD 9700 gpus with 32Gb ram per card running about 285 Watts per card. Less than a 5090 in both cost and power. A VLLM build patched for AITER and you can run Qwen3.6 27B FP8 at roughly 45-50TPS during real coding sessions with Opencode or PI with a full context window. I really hope more 30B dense models continue to be released, but Qwen3.6 should get you a lot of agentic mileage. ROCm stack is not for people though who aren’t willing to dig in and patch things themselves.
• skittleson 3mo ago
  i've been running qwen 3.6 35B A3B with llama.cpp on a 3090ti. i have found it better then sonnet in many ways. Speed and iterations was key. here is the gist of my current configuration: https://gist.github.com/spencerkittleson/5e44b6895a17ca45161bad3675d87069 https://gist.github.com/spencerkittleson/5e44b6895a17ca45161... I use this with tailscale so all my devices have full access to it. That machine get toasty....
• walmas 3mo ago
  Maybe the future isn't Data Centers, climate crisis, drought, and endless subscription and token fees.
• Littice 3mo ago
  [flagged]
• hank808 3mo ago
  Local models are good? Or are we saying that open source/open weights models are good? What I'm asking is, are they good because they are "local" or are they good because you can install and run them yourself, wherever you want? Same node, different node, different cluster, way out in the ether/cloud...
• zx8080 3mo ago
  > None of these are groundbreaking tasks (again, a lot of personalized Google/docs lookups) Does it really needs a GPU at 300Watts to do all that tasks?
• schmuhblaster 3mo ago
  I’ve been playing around with qwen3.6-35b-a3b and managed to boost it significantly by leveraging my own custom harness [0]. It is quite astonishing to see how far local models have progressed, and I think that if you enjoy tinkering a bit, you can save a good bit of money (if you happen to have the hardware lying around anyways). Overall it’s still hard to beat the the cost/convenience combination of a cloud based model provider though. [0] https://deepclause.substack.com/p/how-to-make-small-models-punch-way https://deepclause.substack.com/p/how-to-make-small-models-p...
  • edg5000 3mo ago
    Harness engineering is very interesting stuff. Thanks for sharing.
  • phunterlau 3mo ago
    Cool, so the determinstic harness can boost the agent pretty much!
• dakolli 3mo ago
  Imagine spending $5k to run a 32B param llm locally.. You could run much more capable open source models through Openrouter for years running 24/7 at 50tps. This will never make sense to me.
• dakolli 3mo ago
  It doesn't make sense, if your small local model is 75% as effective as a frontier model and frontier models are still what.. 50% effective maybe slightly more, with tons of downsides.. Why would I spend 5k on hardware to run these mediocre models. I don't really see the point in the frontier model either.
• linuxhansl 3mo ago
  I soooo wish that to be true. Alas, in my experience it is not... Yet. What is true is that it gets easier and faster to run local models. With QAT (quantization aware training), turboquant (or similar) K/V compression; what used to be impossible to run is now fairly easy. I can run gemma4:26b-a4b-qat on my laptop with 20-30 tokens/s with a 256k context window. That was unthinkable just 6 months ago. So the local models are "OK" for small'ish projects. But it does not at all(!) compare to the frontier models. For a large project Claude's Opus 4.6+ just work, whereas local gemma tangles itself up, makes weird mistakes, and just can't handle it (for those cases it is faster if I do it myself). If the trends continues, with 1.58bit QAT models, even better K/V compression, faster multi-token prediction et al, maybe soon it will be comparable.
• sn0n 3mo ago
  Qwen 3? Qwen 2.5 coder?? Is this an llm article written on an outdated model?? LoL
• tpurves 3mo ago
  I do think local models are huge pending market opportunity for Apple. An M5 Ultra Mac Studio (if that exists) could be decent local AI machine, though so expensive as to stay niche. But by the M6/M7 generations and a recovery in DRAM affordability, the future could be interesting moment for them to deliver a compelling local AI platform that 'just works'. But I do think that a mini-pc that is easy to configure, can be always plugged-in, always on, higher power envelope than a laptop, but not obnoxiously loud and hot, is the right form-factor
  • BenRacicot 3mo ago
    Agreed, this is what caused me to build. This thesis exactly.
• pcell 3mo ago
  [flagged]
• aidenn0 3mo ago
  Can anybody recommend sub $10k hardware that can run the models mentioned in TFA at something faster than a snails-pace?
  • kristopolous 3mo ago
    the next thing that people are going to race for is strix/gorgon halo (coming out soon). Still kind of not known. Also the R9700 rocm is 32gb, 1350, available now. It's like 1/3 the price of what 5090s go for and you can get the slimmer models for that price so you can pack more in. If I had to build right this second I'd do small form factor strix halo with a Radeon card. You can get all those parts in like 3 days, msrp, no hassles. the only thing you're paying out the nose for is the ran Good news is mobo manufacturers are adding more slots so you don't have to get robbed paying for 32 or 64gb modules
• Patchistry 3mo ago
  do you run you local models along side some of your "paid" models?
• henryoman 3mo ago
  Will there be a gemma4n
• lanycrost 3mo ago
  I'm crazy for gemma and Qwen, really hope we will be able to run LLMS everywhere like a Doom
• jnaina 3mo ago
  Running Qwen3-30B-A3B-Instruct-2507-AWQ-4bit on an Olares One with NVIDIA GeForce RTX 5090 Mobile GPU (24GB GDDR7 VRAM) and an Intel Core Ultra 9 275HX processor. Plenty fast for coding work and for sharing with my OpenClaw setup. Currently in the process of adding another external GPU (RTX 4090 with pipeline parallelism) via thunderbolt 5 to the Olares One box, for higher quantization, possibly 8-bit, larger context, better concurrency, more kv cache.
• ricardobayes 3mo ago
  They are good, and yesterday's release GLM 5.2 even benchmarks really close to Opus.
• ios-contractor 3mo ago
  I subscribe to this guy on youtube for local model stuff if anyone is interested https://www.youtube.com/@AZisk https://www.youtube.com/@AZisk. I'm not affiliated and I'm not even a paying subscriber. But I like all stuff local.
  • andwhatisthis 3mo ago
    I clicked and immediately subscribed, but then checked out his latest videos and was so put off by the stereotypical clickbait stuff (stupid faces on thumbnails, "I tried (...) and then THIS happened" etc) that I unsubscribed. I understand that it must be what one needs to do to maximize views and brown nose the recommendation algorithm but I just find it incredibly off putting
    • ios-contractor 3mo ago
      I agree. They didn't bother me that much yet but I totally feel you
• zrg 3mo ago
  tldr it is not
• AgentMasterRace 3mo ago
  If you have an extra PC and enjoy 5 tokens a second... Sure
• sieste 3mo ago
  The "middle powers" (cf Carney) should invest in local models, rather than relying on US and China allowing them to rent their AI models. It takes a single executive order to cut the rest of the world off of American AI tools. "I'm happy to pay whatever to rent frontier models from hyperscalers" makes sense if you're citizen of a superpower, but it's risky, naive, bordering on irresponsible to adopt this mindset otherwise, especially when your business or career depend on the tool.
  • k__ 3mo ago
    Training DeepSeek was magnitudes cheaper than training the SOTA models it relied upon. In theory, other countries should be able to replicate that effort and improve it.
    • sieste 3mo ago
      I'd love for European countries to embrace their priorities around people and environment and train a model on public domain data that can run efficiently on cheap hardware. Redefine benchmarks around these ideals and optimise towards them, rather than trying to enter a race that you can't possibly win and didn't want to run in the first place.
• pinstripes 3mo ago
  I very much enjoy scrolling r/homelabs ever so often, so many cool local rigs there
• jkwang 3mo ago
  [flagged]
• 0xdecrypt 3mo ago
  [flagged]
• hamburgererror 3mo ago
  Do you all use local models only for coding? What about using them as decision assistant? For instance, I work in science and sometimes I have many scattered ideas that I'd like to feed into an LLM so I can refine them and extract a meaningful research question. Are local LLM suited for that task?
  • probably_wrong 3mo ago
    After predictably failing at generating a sewing pattern, Gemini gave me yesterday this excuse: > Because AI generates pixels based on visual patterns rather than mathematical geometry, it creates the illusion of a sewing pattern without any of the functional blueprints required to actually drape and construct a real garment. If you want the illusion of a meaningful research question then sure, local models will give you that.
• mlpicker 3mo ago
  [flagged]
• asim 3mo ago
  I don't run local models, my devices are 5-6 years old and not powerful enough. It's a bit counter intuitive and different to what a lot of engineers are doing but I don't have a mac mini, I don't have a powerful laptop, a lot of my dev work has always been cloud based, on github, on a VM, I'm mostly using SSH from my laptop and now Claude Code on my phone (exe.dev is hands down the best experience I had on this front when the agent is literally on the VM). In an ideal world, yea you can run local models, but I need a powerful always on device for that, or the latest gear, and it will never be as fast as what I can use from google, anthropic, or through an API call. I really wish it was different but I have to shell out a ton of money for that, and I guess it's usecase specific right. Maybe if my phone was super powerful and could run models that would be great, but then I have this issue with cloud sync and using things anywhere else. There will be a world in which local models and self deployed models make sense, this is going to be a core experience, but I personally can't run them.
• teknologist 3mo ago
  I found a tool that makes it easy to run Salvatore Sanfilippo's (Redis creator) ds4.c on a Mac: https://github.com/notatestuser/ds4-control https://github.com/notatestuser/ds4-control His program uses quantization, but is very optimised and has builds that can fit into 96GB of memory with great results. DS4 Flash is usually my go-to for a lot of things these days, and I don't have to worry about a cloud model stopping or telling me it's concerned about my usage.
• pauljeba 3mo ago
  How do I beleive thi? you wrote this blog by hand.lol
• nullc 3mo ago
  I'm a little mystified at people taking about qwen 3.6 27b/ gemma 31b being slow in one breath and then saying they're using a 16GB gpu in the next. You do need to use sutable hardware. I get 50tok/s from Qwen 3.6 27b with Q8 & MTP (I can get more aggregate tok/s in parallel rather than using MOE, but don't have enough memory for too many full sized contexts) and 100 tok/s with 35B-A3b Q8 (no MTP as it's not that useful with MOE) on a single workstation gpu that I spent 3k on a couple years ago. These speeds are somewhat faster than what I've seen from commercial SOTA models, they're plenty fast for many applications.
• acb12 3mo ago
  What are the minimal hardware requirements to run a reasonable good local model for real SWE work these days? And have a reasonable inference speed? Seems like the requirement are pretty high and the inference speed is not ideal.
• Muaz_Ashraf 3mo ago
  for the past few days I am building things via local models and review and fix the bugs from opus. Its working okay but still local models are Just HYPE and Irritating.
• ramaseshanms 3mo ago
  Local models Inference never really took with non tech people. Everyday people who dont know the difference between autocorrect and GPTs. But thanks to recent hardware launches from Nvidia and AMD's in response to the MacMini series, it is quite evident that local AI will replace the conventional laptop market completely one day. Current laptops will be what Nokia represents to a iPhone or Android user. Huge Leaps ahead.
  • red__dragon 3mo ago
    Are you referring to Spark that has 128 GB of Unified Memory? It would be still expensive.
• restlake 3mo ago
  my most mind blowing recent development here was testing the Gemma 4 models at release for vision and image recognition vs some benchmarks I had from using Gemma 3 for the same tasks. Gemma 4 is significantly faster and massively more accurate, to a level where I fundamentally can finally turn off my wifi and run a batch of my photos through the local model and trust the results for the extensive classification that Gemma seems well suited to handle. incredible times for local LLMs
• sermakarevich 3mo ago
  I am running an experiment with local qwen3.6:36B for a week: https://news.ycombinator.com/item?id=48520757 https://news.ycombinator.com/item?id=48520757 It really is better than I would expect it to be. But it requires a special treatment. Since the model is smaller it needs a smaller and simpler tasks. I use smarter model to decompose the task into primitive subtasks, write good description, submit to worker with qwen3.6, review completion and create new task to fix if required (20% of cases). This workflow works fine.
• iyia 3mo ago
  [flagged]
• dbg31415 3mo ago
  They misspelled “Better than before. But… yeah.”
• andix 3mo ago
  > For my local setup, I’m currently [..] and LM Studio as the inference server, although it would likely be faster if I just used llama.cpp directly Is there any truth to this claim? LM Studio uses llama.cpp to run the models. I guess the overhead of LM Studio should be minimal. After all LM Studio is a really easy way to host models, are there really major drawbacks?
  • cuvinny 3mo ago
    LM Studio has a lot less tuning options when you launch it. Also it is precompiled so you don't have the latest (and sometimes buggy) releases so you may have to wait a few days to try out a MTP or a new model. LM Studio is easier though.
• ronef 3mo ago
  What's the best practice right now for setting these up? We've been primarily using Nix/Flox to set up the models pretty quickly and at least with minimized amount of commands(biased Nix/Floxer) here and found it useful
• TaniaDictee 3mo ago
  [flagged]
• shunia_huang 3mo ago
  fav-ed this post and will check again when another one poped on front page and says "local models is perfect now"
  • Natalia724 3mo ago
    "Good now" feels like the right bar. I still wouldn't use local models for everything, but for quick edits, grep-like code questions, and private notes they are already useful.
• mantlemd 3mo ago
  [flagged]
• maxchatrove 3mo ago
  [flagged]