5 ms·
Any recommendations on good open ones? What are you using primarily?
by nate8bit 5mo ago
Any recommendations on good open ones? What are you using primarily?
- blahblaher 5mo agoqwen3.5/3.6 (30B) works well,locally, with opencode
- zozbot234 5mo agoMind you, a 30B model (3B active) is not going to be comparable to Opus. There are open models that are near-SOTA but they are ~750B-1T total params. That's going to require substantial infrastructure if you want to use them agentically, scaled up even further if you expect quick real-time response for at least some fraction of that work. (Your only hope of getting reasonable utilization out of local hardware in single-user or few-users scenarios is to always have something useful cranking in the background during downtime.)
- pitched 5mo agoFor a business with ten or more engineers/people-using-ai, it might still make sense to set this up. For an individual though, I can’t imagine you’d make it through to positive ROI before the hardware ages out.
- zozbot234 5mo agoIt's hard to tell for sure because the local inference engines/frameworks we have today are not really that capable. We have barely started exploring the implications of SSD offload, saving KV-caches to storage for reuse, setting up distributed inference in multi-GPU setups or over the network, making use of specialty hardware such as NPUs etc. All of these can reuse fairly ordinary, run-of-the-mill hardware.
- DeathArrow 5mo agoSince you need at least a few of H100 class hardware, I guess you need at least few tens of coders to justify the costs.
- pitched 5mo agoI see the 512GB Mac Studios aren’t for sale anymore but that was a much cheaper path
- wuschel 5mo agoWhat near SOTA open models are you referring to?
- cyberax 5mo agoI'm backing up a big dataset onto tapes, so I wanted to automate it. I have an idle 64Gb VRAM setup in my basement, so I decided to experiment and tasked it with writing an LTFS implementation. LTFS is an open standard for filesystems for tapes, and there's an implementation in C that can be used as the baseline. So far, Qwen 3.6 created a functionally equivalent Golang implementation that works against the flat file backend within the last 2 days. I'm extremely impressed.
- Gareth321 5mo agoIt is surprisingly competent. It's not Opus 4.6 but it works well for well structured tasks.
- cpursley 5mo agoHow are you running it with opencode, any tips/pointers on the setup?
- pitched 5mo agoI want to bump this more than just a +1 by recommending everyone try out OpenCode. It can still run on a Codex subscription so you aren’t in fully unfamiliar territory but unlocks a lot of options.
- jherdman 5mo agoIs this sort of setup tenable on a consumer MBP or similar?
- pitched 5mo agoFor a 30B model, you want at least 20GB of VRAM and a 24GB MBP can’t quite allocate that much of it to VRAM. So you’d want at least a 32GB MBP.
- zozbot234 5mo agoIt's a MoE model so I'd assume a cheaper MBP would simply result in some experts staying on CPU? And those would still have a sizeable fraction of the unified memory bandwidth available.
- pitched 5mo agoI haven’t tried this myself yet but you would still need enough non-vram ram available to the cpu to offload to cpu, right? This is a fully novice question, I have not ever tried it.
- tredre3 5mo agoYou're correct. If you don't have enough RAM for the model, it can still run but most of it will run on the CPU and be continuously reloaded from the SSD (through mmap). A medium MoE like 35B can still achieve usable speeds in that setup, mind you, depending on what you're doing.
- _blk 5mo agoIs there any model that practically compares to Sonnet 4.6 in code and vision and runs on home-grade (12G-24G) cards?
- macwhisperer 5mo agoim currently running a custom Gemma4 26b MoE model on my 24gb m2... super fast and it beat deepseek, chatgpt, and gemini in 3 different puzzles/code challenges I tested it on. the issue now is the low context... I can only do 2048 tokens with my vram... the gap is slowly closing on the frontier models
- equasar 5mo agoThe thing I dislike about OpenCode is the lack of capabilities of their editor, also, resource intensive, for some reason on a VM it chuckles each 30 mins, that I need to discard all sessions, commits, etc. I don't know if it is bun related, but in task manager, is the thing that is almost at the top always on CPU usage, turns out for me, bun is not production ready at all. Wish Zed editor had something like BigPickle which is free to use without limits.
- Jarred 5mo ago> turns out for me, bun is not production ready What issue did you run into?
- cmrdporcupine 5mo agoGLM 5.1 via an infra provider. Running a competent coding capable model yourself isn't viable unless your standards are quite low.
- myaccountonhn 5mo agoWhat infra providers are there?
- elbear 5mo agoThere's DeepInfra. There's also OpenRouter where you can find several providers.
- culi 5mo agoLMArena actually has a nice Pareto distribution of ELO vs price for this model elo $/M --------------------------------------- glm-5.1 1538 2.60 glm-4.7 1440 1.41 minimax-m2.7 1422 0.97 minimax-m2.1-preview 1392 0.78 minimax-m2.5 1386 0.77 deepseek-v3.2-thinking 1369 0.38 mimo-v2-flash (non-thinking) 1337 0.24 https://arena.ai/leaderboard/code?viewBy=plot&license=open-source https://arena.ai/leaderboard/code?viewBy=plot&license=open-s...
- logicprog 5mo agoLMArena isn't very useful as a benchmark, however I can vouch for the fact that GLM 5.1 is astonishingly good. Several people I know who have a $100/mo Claude Code subscription are considering cancelling it and going all in on GLM, because it's finally gotten (for them) comparable to Opus 4.5/6. I don't use Opus myself, but I can definitely say that the jump from the (imvho) previous best open weight model Kimi K2.5 to this is otherworldly — and K2.5 was already a huge jump itself!
- DeathArrow 5mo agoI am using GLM 5.1 and MiniMax 2.7.