7 ms·

I feel like the gap is closing to be able to run good enough models locally even for coding and I would assume it could make some companies a bit nervous. Am I

• fsuts 3mo ago
  Why do you think they are rushing to IPO!!
• fny 3mo ago
  The RAM requirements are still pretty painful.
  • yieldcrv 3mo ago
    equilibrium in one or two more years on the consumer/prosumer side think Apple M6 or M7 with a currently unforeseen denser memory style, 256gb RAM a couple inference or cache improvements on the algorithmic side, using less ram for context windows and doubling token speed again denser open source models, packing more experts for smaller active layers it'll still be expensive but like $8,000 - $13,000 instead of $450,000 worth of B200s
    • stingraycharles 3mo ago
      Fairly certain that model sizes and computational requirements will grow as the price for LLM compute drops.
      • 3stacks 3mo ago
        Maybe there's a conversation to be had about how much is enough... Unless something beyond my imagination happened, I would be happy enough with Opus 4.5 levels of productivity
        • stingraycharles 3mo ago
          This really sounds like “640kb should be enough”. I’m sorry, but I just can’t imagine us running smaller models than we are using right now in 5-10 years from now.
          • hajile 3mo ago
            We've already hit RAM power and size limits (about 40k electrons which is the limit before we get noise messing up the amplifier). If a model needs 2x more memory, but serves the same number of customers, the cost is going to go up per customer to cover the increased hardware and power costs. Companies are starting to implement AI limits to keep costs under control. Anthropic and OpenAI are rumored to be considering cutting inference prices trying to retain customers as LLMs commoditize and race to the bottom. It reminds me of the Chinese bike wars where bike-share companies were losing massive amounts of money, but kept running sales and lowering prices in an attempt to compete and drive out their competitors. The end of that story was a bunch of major bankruptcies and giant bike graveyards. Nvidia's hard pivot to "in the near future, everyone will run their AI at home" seems to indicate that they also see the market shifting. We've already had AI ingest everything out there. The real challenge becomes how to better optimize their algorithm to get more useful data in less space.
          • pheggs 3mo ago
            Probably not. But I also cant imagine the hardware not improving. Also there is surely a sweet spot regarding price and quality. I have strong doubts that much larger models will give much better results, which is not like disk space where every additional bit makes it one bit better
      • yieldcrv 3mo ago
        have you seen the open source LLM space? people fulfill all niches and there are active communities at every range of RAM and all are looking for the most capable in their respective range a lot of innovation occurring
• CamouflagedKiwi 3mo ago
  The hardware requirements to run this locally are still very high. Seems far enough off mainstream for those companies not to be too worried yet.
• cogman10 3mo ago
  I don't think so. I could easily see a company deciding to host and run these models for their own development. If you have a dev team of about 10 people, a one time $50k investment in an LLM server has to be pretty tempting. Unlimited tokens, decent performance, upgrade options, and potential product integrations. For companies wanting LLMs in their products in general, I have to think going the local llm route is even more tempting. Somewhat dumb models are more than good enough for a lot of the things people are integrating LLMs into their products.
  • eventualcomp 3mo ago
    Where is $50k coming from again?
    • stingraycharles 3mo ago
      That’s less than the monthly salary of 10 software engineers, and assuming they pay API prices, probably earns itself back in about a year. Having said that, I don’t think it’s all that tempting for companies at all, considering this whole market is developing rapidly and it’s nearly impossible to predict where we’ll be at in a year or two.
      • cogman10 3mo ago
        The hardware requirements aren't evolving and the local models have only been improving. It's not like you'd lose capabilities, if anything this solution just gets better with time.
        • chatmasta 3mo ago
          If the newer models require more/better hardware then you’ll lose capabilities. I think you’re better off renting GPU instances and running all the software on those. It’ll be cheaper than Anthropic and OpenRouter but slightly more expensive than electricity and depreciation of hardware.
• UncleOxidant 3mo ago
  If we didn't have a RAM/GPU shortage right now they would be more nervous than they are. But as it is very few people are going to be able to afford a rig that can run this model effectively. That's probably not going to change for several more years yet. I think if the Z.ai folks decide to come out with a flash version of GLM-5.2 specialized for coding that came in about about 80B params, then the US frontier labs would probably be more worried. Overall, the Chinese AI companies have been showing the way to do the same amount with less (sometimes much less) and as that trend continues it's going to make the frontier labs worried - but even the Chinese AI companies are going to want to protect their moat by not releasing capable models that are significantly smaller than their current flagship models. AliBaba Qwen seems to be there now - it's gotten mighty quiet from them lately - their latest 395B model is just too large for most folks to run at home and they don't seem to be making any noises about releasing smaller ones this time around.
  • gpm 3mo ago
    The ram/gpu shortage won't last forever though. Moreover we can be pretty confident that long-term the prices will obey wrights law and come down in cost significantly (from the pre-shortage prices) as we learn to produce them more efficiently. LLM companies are valued as if they're going to have some enduring monopoly that they can extract money from... GLM-5.2 and similar models make that valuation very very questionable.
    • UncleOxidant 3mo ago
      > The ram/gpu shortage won't last forever though. No disagreement there, but it could easily last another 3 to 5 years which is a long time in tech terms.
      • DougN7 3mo ago
        Long enough for them to IPO and all the execs to retire. I doubt they care beyond the IPO.
        • 3mo ago
• notatoad 3mo ago
  locally on what hardware? something like the new dgx spark, ryzen halo, or mac studio will cost you ~ $4k plus whatever you pay for power. at the rate AI is currently progressing, i think you'd be optimistic to consider that as having a 2 year depreciation. for $4k, you can get 20 months of claude max 200. i'd take claude over the hardware. anthropic will have something to worry about when you can run a local model on your macbook that can code. but i think we're quite a ways off from that.
  • chatmasta 3mo ago
    Just a hunch, but I think the most cost effective “local” deployment method right now is renting GPU clusters by the hour and running all the inference software on them yourself. This will be cheaper than capital expenditure on hardware that will depreciate and become last-gen, and cheaper than OpenRouter pay per token.
  • tomr75 3mo ago
    people who can't afford Claude max 200 are using qwen 3.6 27b for local coding assistance already
  • fc417fc802 3mo ago
    > at the rate AI is currently progressing, i think you'd be optimistic to consider that as having a 2 year depreciation. How so? Model capability at a fixed hardware level has been consistently (and rapidly) increasing. You might or might not be able to run state of the art 2 (or 4 or whatever) years from now but you can reasonably expect the hardware to last upwards of a decade with model performance consistently improving over that time frame. You can get a tolerable (at least by some metrics) experience using 10 year old hardware today.
  • c7b 3mo ago
    You can get a 128GB Strix Halo for under $3k. Used to be under $2k. Even if you believe it'll be completely obsolete for AI in two years, it'll still be good for many other things. Games for at least several more years, a great home server and/or desktop almost indefinitely. Plus, we might actually reach good enough levels for some AI use cases, if we're not already there. And never underestimate the potential for enshittification. Your local rig will only deliver better performance over time as more and more tweaks come out. With cloud services expect the opposite to happen as subsidies run out. It's entirely possible that they will intersect on a bang per buck basis within two years.
• stymaar 3mo ago
  Honestly, Qwen3.6 is already what you need for the large majority of tasks. (I only ask Opus every 5 to 10 requests, when my local Qwen fails or when I encounter a situation that is too world-knowledge specific to be worth asking, but that way you can live easily with Claude's cheapest plan without ever facing usage limit).
• simplyluke 3mo ago
  You don't even need to run them locally for them to be a threat. Plenty of companies are looking at paying third party companies to host these models and they come in at fractions of the price of the frontier labs.
  • pheggs 3mo ago
    thats true. also, I watched the glm prices and it didnt take long before the prices dropped even lower for some providers. its like another layer of competition between hosters
• scosman 3mo ago
  It's not economic to run them locally. It's amazing for privacy, and fun hobby. But you're either looking at super slow CPU builds with $10k in RAM, $90k worth of GPUs, or a really quantized model that doesn't compare in quality. I might build one for fun, but it's not going to change the economics alone. Still exciting it's possible.
  • oceanplexian 3mo ago
    It depends what you’re using it for. Real time interactive Claude code session? No, it’s kind of impractical. But if you already have agent loops dialed in (For example I have one that uses a browser testing framework), it wouldn’t really affect me at all if it crunched away at 7 tokens per second all night long.
  • leansensei 3mo ago
    Not really, you can do great things without them. I've been summarizing hundreds of documents. I've added MCP servers to my internal business tools (Elixir apps) and can chat with the Nous Hermes agent over Telegram about pending orders, inventory level, historical product prices, etc., without having to click/dick around with a web UI. Sure, it cannot replace SOTA models for agentic coding, except for small, well-scoped refactorings. But even a model like ministral-3:8b or qwen3.5:9b is a boon for so many smaller use cases!