7 ms·
Using an open model feels surprisingly good
- irishcoffee 2mo agoIt surprises me that this concept took as long as it did to gain traction in… hacker news. 15 years ago folks here were compiling kernels and gentoo distros. Lately it’s been “you should just pay the man, it’s cheaper than running these things yourself”
- brcmthrowaway 2mo agoRsync the dropbox, dude
- irishcoffee 2mo agoStill working on the fields medal, I’ll give that a shot next.
- loire280 2mo agoRolling your own Linux is nearly free and could be done on any computer you had lying around. Dropping >$5k on a computer to run a local model (badly) doesn't really scratch my "hacker" itch. The author of this article works for an AI infrastructure company and ran Kimi K3 on their infrastructure - this post reads like marketing.
- irishcoffee 2mo agoYou can run a small model on almost any hardware, for free, and just play around. It doesn’t need to be some “frontier” (whatever the hell that means) model by any means. Grab some new tech and slap it on whatever hardware you have laying around, see what you think. Making it about the hardware costs is not the play, they’re not an actual barrier.
- anuramat 2mo ago> you can run a small model but why?
- irishcoffee 2mo ago… to hack. For fun. To play around. Why do anything?
- anuramat 2mo agowhat's there to hack that you couldn't do with $3 in openrouter credits?
- irishcoffee 2mo agoThere is value in learning how to launch a model and develop an environment with which to interact with it. This understanding is basically a pillar of a “hacker” ethos. But if you just wanna outsource understanding for 3 bucks in tokens, that’s cool too.
- anuramat 1mo agodo you think typing `make` into a terminal is some sort of an insane intellectual achievement?
- hbn 2mo agoI don't want to use a crappy model that can't do any of the things I want it to. I'm not coding with Claude cause I enjoy the inherent novelty of LLMs. I'm doing it cause it's enabling me to quickly solve problems without getting stuck on the hitches that have deterred me from bothering with dozens of side projects my entire life.
- kelnos 2mo agoSure, that's fine. But some people like to play around and hack just for the sake of hacking. If you don't like that, that's cool, no issue with that. But some people do. (FWIW, I have the same attitude/preference as you do. Until I can run SOTA models on my own hardware without taking out a second mortgage, I'll pay our AI overlords for the privilege.)
- c7b 2mo agoThe inflation-adjusted cost for getting one of the early personal computers to do some BASIC programming was probably a lot higher than $5k today, and you could still go cheaper. Local AI is still (expensive) hobby territory. A capable local AI machine runs for about as much as a good mountainbike.
- eru 2mo agoIt's fairly easy (and cheap!) to run your own email server, but most people don't bother and just use gmail. And rightly so!
- alpineman 2mo agoTrue, I see so many posts along the lines of 'I am a quasi programming god with no time on my hands (apart from time wasted here) so I will gladly pay $10,000 a month to Anthropic if it makes me 1% more effective than if I were to use GLM'
- khimaros 2mo agothis seems to be an advertisement for Modal. is it really your own infrastructure if the hardware is leased?
- jdw64 2mo agoActually, if you break tasks down into small enough units, there's not much difference between open models and closed models. The only reason to use agents is the hope that they'll work with natural language input—but that's where the difficulty lies. For example, if you modify things at the level of small functions, open models seem to perform just as wel
- paradox460 2mo agoYup. Been using big expensive models like opus, gpt, or glm for planning, then switch to something like gpt-oss-120B on cerebras and watch it fly. Total cost will be less than $5 for even very large tasks
- jdw64 2mo agoPeople tend to think alike. I work in much the same way.First, I write the draft myself, then let AI handle the revisions, and finally run it through an open model in small units before I check the results.
- verdverm 2mo agoI think this is where they say "loop engineering" / orchestration comes in, though I would also include context/harness engineering in the toolbox
- m_ke 2mo agoGLM 5.2 feels better than Opus and K3 is as good as Fable. Now I can't wait for someone to distill K3 into a Qwen 3.6 27b or Poolside S 2.1 sized models for a proper fast local Composer 2.5 replacement.
- arjie 2mo agoTo be honest, I am surprised by how good DeepSeek V4 Flash is. I use Claude Code and Codex on Claude 5 Opus and GPT-5.6 Sol most of the time, but when I use DS V4 Flash I don't feel like it's really that bad. And with oh-my-pi and just plain pi it's pretty good. To be honest the frontier models are much better at tool calling so in an assistant flow they're better but I did the dumb thing and optimized the harness for the model instead, rewriting the tools so they match what it guesses at, and DS V4 Flash does just fine. The TTFT and tok/s are much higher on the small model so that makes it competitive for a bunch of things. It feels like what old Sonnet used to by the end of last year which is honestly damned good.
- pimeys 2mo agoI wrote my own OpenClaw one weekend and I am running it as my assistant through Matrix with DeepSeek v4 Flash (and Qwen). It probably costs me about 2 dollars a month and is even more useful than ChatGPT would be due to me having full control on what tools it has access to. I can do things like take a photo of a doctor's note among add the appointment to my calendar, send a PDF to my archive tagged, OCR'd etc, search info from internet, look data from Google Maps. I can even integrate this to home assistant and talk to my agent with my open source Alexa-like system.
- mayank 2mo agoOddly enough, I did the exact same thing this past weekend, for a couple hundred usd in Fable overage. It was truly remarkably easy to build and package it exactly to my whims, in my case as a single docker container with a process reaper that runs llama, my Go code, tts, chat harness, browser in xvfb, and even a mailer daemon. With Gemma, it even runs on a RPi 5. What’s absolutely wild to me is that over a couple hours, I could probably have it import parts of Home Assistant for my devices directly, and do other wacky stuff in what is essentially software for one.
- pimeys 2mo agoFunny. I use Fable a lot at work, but this one I paid from my own pocket and coded it with GLM 5.2. Paid 20 euros in total.
- loufe 2mo agoThis is only a thinly veiled ad. It's fine, I was curious about this exact setup, anyways. What would be useful is a cost metric. I'm curious how much I'd be willing to spend as a premium to not have those companies piping my conversations directly to the NSA. Maybe only some conversations? Claude and OpenAI are heavily subsidized, by all accounts, so Kimi K3 on a private endpoint might end up costing more or less - that's what I want to know.
- mayank 2mo agoI was curious about the cost angle too, i.e. how much "free" coding agent I can get for what cost. Here's the research by Fable if you're interested: https://claude.ai/public/artifacts/2c9a5001-0b7e-4944-beb1-9888434fd989 https://claude.ai/public/artifacts/2c9a5001-0b7e-4944-beb1-9...
- aprilnya 2mo agoWith OpenRouter this is pretty easy to see - each model page has various providers listed along with an icon marking providers with zero data retention, and the price. You can also set up an account-wide "guardrail" to prevent routing to non-ZDR providers.
- lukeschlather 2mo agoOpenAI and Anthropic also claim they don't retain data. I'm not sure what it would take for me to trust any of these inference companies, and why I should trust them more than the big ones.
- theshrike79 2mo agoI trust both OpenAI And Anthropic, they have literal billions to lose if someone finds out they've been training with in-house corporate data. Just the GDPR/CCPA fines alone WILL bankrupt both companies.
- nujabe 2mo agoThis is such a poor quality post, reads more like a diary entry than a substantive technical post.
- PontifexCipher 2mo agohttps://en.wikipedia.org/wiki/Blog https://en.wikipedia.org/wiki/Blog
- sroerick 2mo agolol
- nujabe 2mo ago?? Edit: I see, had typo last word of sentence was meant to be “blog”
- PontifexCipher 2mo agoNo, my point is that it's a post on a personal blog not a peer reviewed paper.
- deleted 2mo ago[deleted]
- kelnos 2mo agoAs in, the post is a blog, which is often literally someone's diary. Also, if you don't like a submission, flag it and move on. Commenting that you don't like it doesn't add anything to the discussion.
- Gigachad 2mo agoI will take 5000 "diary entry" blog posts over LLM generated walls of text that say nothing. People need to learn to just post the prompt rather than the LLM output which just fluffs the prompt.
- sudo_cowsay 2mo agoI love OpenCode and the blankness of it too. Clean, light, and manual. It's a good change of pace from what we are used to on the internet (very cool but slow). Also, can you hmu with that modal plan XD
- roywiggins 2mo agoPi Agent is even leaner feeling, it's worth a try.
- sudo_cowsay 2mo agoWill do! Heard a lot of great things about it from my friends.
- Shorel 2mo agoMy own agent is cleaner and lighter. Written in D, now only the local models are the bottleneck.
- alpineman 2mo agoFeel the same about HN :)
- deleted 2mo ago[deleted]
- wps 2mo agoThe reason Claude code is so popular is because it’s really good at taking super vague human prose “Claude build me a million dollar SaaS”-type prompts and spitting out thousands of lines of code which cover tons of surface-level edge cases, build in tons of functionality, etc The smaller/open models are less good at that. But that’s not how software development is done. You don’t prompt a whole app and be done with it. If you’re using it as an aid to traditional software dev, iterating on small, targeted functions, GLM works as good if not better than Claude. Anthropic expects low quality prompts. If you rubber duck GLM, you get absolutely pristine output in most cases.
- lordnacho 2mo agoAre local models more snappy? I'm at a stage where I can work with the output of LLMs, and the next win is really just getting things written out quickly.
- suprjami 2mo agoDepends on your setup. If you drop $10k on an RTX Pro 6000 then yeah Qwen 35B MoE will absolutely fly. If you have a pair of 3090s and run Qwen 27B, or an old Threadripper with heaps of system RAM and Deepseek or MiniMax or Kimi, no it won't be as fast as Claude. Most local LLM nerds are not running locally for superior speed, we're doing it for sovereignty and/or privacy, or maybe just because it's fun which accidentally became useful this year.
- Gigachad 2mo agoPersonally I've found Deepseek v4 Flash to be as useful to me as Opus. But I don't do these silly one shot tech demos. I have the technical understanding to ask for exactly the change I want with the right terminology. I loaded up some credit on openrouter and it took me ages to hit $1 in spend.
- wps 2mo agoI’m the exact same way. $15 on openrouter lasted me so long it would’ve got me fired at FAANG. Despite this, my number of commits is dramatically higher. Small, beautifully scoped changes is just good software development, and good for the wallet as well. I think the issue is that no one is content with incremental progress. We all know one shots are mostly possible, so the age of the personal project is kind of over. There’s no motive to invest dozens of hours getting a working prototype when Claude can give you something right now. So you can’t make small changes until you have that codebase in place already. You’re forced to make sweeping changes if you use AI from the beginning. And it’s not like it matters, there’s no personal attachment to any one part of the code, it’s not even seen!
- spicyusername 2mo agoSomething I've been thinking about a lot is that this new technology's primary UI is natural language. Human's are REALLY primed for natural language. If it sounds good it must be good. The code this new model writes is incredible! It told me so!
- adithyassekhar 2mo agoIt did say it was an unusually clean codebase.
- wxw 2mo ago> The best way I can describe it is like opening vim after spinning a bunch of time in a big fancy editor. Harness-aside, I get this feeling sometimes when I swap from a big frontier model to something more nimble like Composer. I can get into a better thinking and q&a loop with fast models, similar to how I can flow through a file more easily with vim.
- shepherdjerred 2mo agoI’ve enjoyed using Kimi K3. The biggest issue for me so far is finding a good harness. Claude Code is excellent for tasks that require backgrounding processes/agents, or fanning out. Codex and OpenCode I’ve found to be less capable.
- danny_codes 2mo agoI suspect FOSS will converge/create a best-in-class harness within the next 6 months or so. As will most (all?) software, the FOSS stuff somehow always ends up being way better than the closed source stuff.
- seaal 2mo agoIs there any resource or benchmark actually comparing how different harnesses perform with a given model? Really just feels like endless FOMO with how fast the iteration cycle is for harness and model development.
- shepherdjerred 2mo agoI found this: https://artificialanalysis.ai/agents/coding-agents#harness-comparison https://artificialanalysis.ai/agents/coding-agents#harness-c... I don't think it's very good though. As an example I can use one CC instance to delegate to several to achieve complicated/open-ended goals. That just isn't possible with other harnesses, and it's definitely not benchmarked.
- shepherdjerred 2mo agoI am not sure -- Anthropic probably does benefit from the vertical integration here. They can train their models to work better with CC, which isn't really an option for third party harnesses. With that said I think Codex/Kimi code are all behind CC as well, so maybe it's a question of effort and not the model.
- bjertoref 2mo agoIn the fullness of time, LLMs will eventually become commodities and their price will be a race to the bottom. As the competitive advantage of one LLM over another shrinks - particularly as open source models expand - the value proposition will turn to the neuro/symbolic harnesses that use those models, structures that encompass agents, agent orchestration, tooling, and component plugins. Even those from frontier AI organizations will fall to open source frameworks. The last thing standing will be those who command the data centers on which these things run, and even then, that will become a commodity, with the number of players shrinking to just a few survivors. They too will find pressure: as computational power rises, more and more such systems will be run locally. - Grady Booch
- deleted 2mo ago[deleted]
- ljlolel 2mo agoyou can use claude code or codex with open models using AllRouter https://github.com/Lore-Hex/AllRouter https://github.com/Lore-Hex/AllRouter
- _ink_ 2mo agoFor claude it is enough to set a few env variables. It lets you easily map each model class (opus, sonnet, haiku) to an open model with Openrouter: https://openrouter.ai/docs/cookbook/coding-agents/claude-code-integration https://openrouter.ai/docs/cookbook/coding-agents/claude-cod...
- ljlolel 2mo agoyes but AllRouter does automatic failover and can burst to cloud as well or use local when needed
- luciana1u 2mo ago[flagged]
- mbanerjeepalmer 2mo agoThis is an ad. I know the author has a long history here and I'm sure the post is a genuine reflection. But how is posting "I used my own product and it felt really good" anything other than self promotion?
- dan_gee 2mo ago[dead]
- armchairhacker 2mo agoIt would be acceptable iff the author prefixed it with Show HN. My suggestion to mods if the post is worth keeping up is to do that.
- witx 2mo agoSo all the blog posts, every 10s on HN, shilling for Anthropic are not ads but this is?
- sd9 2mo agoThe author works for a company that provides Kimi. It’s different. It doesn’t mean the post doesn’t accurately reflect the author’s views - it probably does. And the conflict of interest is well disclosed in the brief post. In fact, it’s quite natural for somebody who likes open models to work somewhere associated with them. So personally I don’t find any impropriety going on here - but yes, it’s an ad.
- 2mo ago
- madhu_ghalame 2mo ago[dead]
- wayknow 2mo ago[flagged]
- Alien1Being 2mo agoLooks like everyone is hoping that Chinese AI wins over AI from Trump's land.
- capestart 2mo ago[dead]
- deleted 2mo ago[deleted]
- KunYuan 2mo agoWhat struck me wasn’t so much the model itself, but the feeling of no longer being tied to one company’s product.
- mcapodici 2mo agoHi msaltz, I like your other introspection posts alot.
- msaltz 2mo agoThanks! I’m most proud of those. Glad you liked them :)
- prplxd_nihilist 2mo agothis blog is a mini nothing burger.
- nottorp 2mo agoThis being an ad aside, does the model do the job or does it only make you feel good?
- stasomatic 2mo agoIs nobody worried that by using hosted models, Chinese or otherwise, we are sending our trade secrets who knows where? I've tried using local models, but they run slow on MPB M5 (base) 16GB, and I am not upgrading my laptop any time soon.
- luciana1u 2mo ago[flagged]