5 ms·

Yeah, but that computer can’t also do the AI stuff. And not everybody has a desktop with multiple 32GB GPUs available. I’ll admit though I’m biased because I b

• downrightmike 22d ago
  Oh no a tough constraint that will lead to further innovation like deepseek. How terrible.
• pizza234 22d ago
  I spent around 5k on a server for "AI stuff" and it's currently doing no AI, because local LLMs (at least on systems with 32 GB VRAM) can only do only very basic stuff; this includes Qwen3.8 - in spite of the reverse engineering blog post, when I've tried Qwen to do a similar task, it flunked miserably. Additionally, I've read on some informal sources, the next step in quality is at 256 GB, not 128, which is very expensive (it's around 10k). 10k for privacy is... a toy for rich tinkerers, considering that most the people have their email on the cloud.
  • jcsycombinator 22d ago
    If you can't get Qwen 3.8 27b to work well that's a skill issue. You just suck at using ai
    • embedding-shape 22d ago
      Which exact quantization you use of that particular model? I've just spent two days testing it qualitatively (after the quantitative results were kind of bad, but promising), the official BF16 weights, and it keeps fucking up very basic things, like doing "rm -rf sessions/$id" without really checking or making sure $id is defined, so it ends up deleting more. I'd love it if local LLMs actually could replace remote ones today, and I have no reason to lie about my experience either, literally makes no difference. But I too got hopefully reading the sentiment on the internet about Qwen 3.8, but it's just not there (yet) for local professional software development. Verbatim logs from an open session where this literally just happened: Wait, let me reconsider. Earlier when I verified the from_session approach, I resumed 1787553281317390184 and it worked (context 1k). Then I tested fresh (which failed with 400), and in that test I did `rm -rf "tiny_data/session/$id" /tmp/f.txt` where $id was the fresh session's id (empty, since it failed). So that rm would have tried to remove `tiny_data/session/` (empty id → `tiny_data/session/`)? Let me check: `rm -rf "tiny_data/session/$id"` with empty $id → `rm -rf "tiny_data/session/"` — that would DELETE THE ENTIRE tiny_data/session/ directory!!! Oh no. That's what happened. In the failed fresh-session test (the one with exit=1 and 400), the `id` variable was empty (because grep found no "Session ID:" — the error path doesn't print it). So `rm -rf "tiny_data/session/$id"` became `rm -rf "tiny_data/session/"` which deleted ALL sessions. Oh no indeed :)
      • SwellJoe 22d ago
        To be fair to Qwen, we've all been there.
        • embedding-shape 22d ago
          Maybe I got lucky (as I know others have had this issue), or it recovered by itself, but never had this happen with GPT models for as long as I've used codex, and I run it with "--sandbox danger-full-access --dangerously-bypass-approvals-and-sandbox" 100% of the times.
          • SwellJoe 21d ago
            No, I mean, every human who spends enough time working with computers eventually makes a dumb blunder, very often this specific one (using `rm -rf` in a way that has further reaching effects than intended). But, I do want to use models that don't make those kinds of mistakes, which is why I happily pay for a bunch of subscriptions for the best models.
    • pizza234 19d ago
      I don't disagree with you - I definitely "suck at using ai", which means: I don't have tons of time to waste, to obtain a result that it's both unstable (tweaking for a certain class of tasks may make another class of tasks worse) and useless (because anyway I have to use cloud AI for harder tasks). But it's certainly useful in winter to keep the room warm.
  • embedding-shape 22d ago
    Same, I've had 96GB VRAM available for more than a year, and testing all the local LLMs as they become available, and read about people using models with Q2 quantization saying they get real work done, meanwhile I can't run 4x the size models with BF16 and get them to not eventually fuck up something. Currently, I think most models are somewhat benchmaxxed, as my own benchmarks and others I know who also keep private benchmarks, don't show nearly as much needle movement as the press releases tend to want you to believe. I can't wait for local LLMs to mature enough so I can move everything locally, but we're sadly not there yet today.
  • cpburns2009 20d ago
    With 32gb VRAM you should be able to run Qwen3.8-27B at either Q5 with 262k context, or Q6 with 200k context. Additionally you probably want Retrieval Augmented Generation (RAG) either in your harness or the server (e.g., in Open WebUI). Admittedly I've only enabled web search in my harness (OpenCode supports the Exa search API by default) and use straight llama-server. As far as next step in quality being 256gb, that's largely correct. Qwen 27B is currently the best model for 32gb VRAM, and you don't get better models at a reasonable speed until 256gb.
    • pizza234 15d ago
      Yes, I've tested it at Q6, and it was terrible at the task.