9 ms·

I have! I care about data privacy and LLMs being free. I'm using the Pi coding harness but containerized and sandboxed, to make sure it's running completely off

I have! I care about data privacy and LLMs being free. I'm using the Pi coding harness but containerized and sandboxed, to make sure it's running completely offline. On my Mac Studio with 128GB RAM (or MacBook with 36GB RAM) I'm using Qwen3.6 35b, with only 3b active parameters so that it runs really fast. I've done a complete redesign for my website's homepage and blog with Django + Wagtail. The latter is interesting, because Wagtail is a bit less well-known, so the agent, without giving it internet access, doesn't always know how to develop for Wagtail. I've used Qwen3.5 122b for when things get more complex. At 10b active parameters, it's significantly slower though. I've noticed a few things compared to large models like Claude. For starters, you really need to know what you're asking, and be precise; it doesn't do much thinking for you. Any assumptions left open, and it'll take the easiest route to reach the goal (e.g. CSS in HTML), often not the best in terms of architecture. It gets into loops quite often, and surprisingly often gets the edit tool call wrong, after which it will spend lots of thinking tokens and re-read files instead of retrying (despite the system prompt suggesting so). Comparing agentic Qwen3.6 35b to Claude Opus is like a junior with knowledge across the board, that you really need to guide, versus a senior that thinks with you on architecture. If Opus gives a 15x speedup, local and fully offline Qwen gives a 5x speedup. Which, given that it's completely free, is still mind-boggling to me :)

p0w3n3d 3mo ago
which coding agent are you using?
GardenLetter27 3mo ago
Could the harness not check for a failed tool call and pass it to a small model for correction without clogging up the main context?
- Greenpants 3mo ago
  I'm actually quite sure that directly retrying the tool call would often fix the edit-call already. But these models have been trained to "think" for a while for any problem solving, so they'll presume the problem of the edit is more fundamental and spend unnecessary tokens filling up the context. I'll experiment more with the effectiveness of AGENTS.md rules for local Pi agents. I feel like smaller (local) LLMs just lack in attentiveness to elements in the context window, like precise instructions, compared to e.g. Claude models.
- lambda 3mo ago
  The thing is, to do a proper fix it would really need all of the context (maybe the tool call that failed was for an edit to a file that was last touched way at the beginning of the context), so you'd need to either keep that smaller model running doing prompt processing all the time, or have a very long wait while it does prompt processing on your whole session. And then also, sometimes the tool call errors are because of something like a file was changed out from under it; the larger model is probably going to do a better job of figuring that out and fixing it up. Finally, in Pi, you can always just use the /tree command to skip back to before a series of failed tool calls, with a summary if you want to let the model know what happened. The Pi /tree command is pretty powerful in managing your context
  - everforward 3mo ago
    An illustrative example I've seen a lot is creating Jira tickets in projects with custom fields marked as mandatory. It tries to create the ticket without the field and the tool call fails. The LLM needs access to the full context so that it can generate text to put in the "Why couldn't this meeting be an email?" field.
0xbadcafebee 3mo ago
The harness and the LLM parameters are pretty essential to getting better results and reducing loops. Tweak the parameters and you can mostly eliminate loops without negatively affecting performance (it's a bit complex but ask a SOTA AI to guide you and it's not hard). The harness should also react more intelligently to failures; it can do things like return additional context or hints as it tracks error rates and avg duration of calls. Pi can be easily extended, and it's suggested by the author you modify it to perform better for your use case.
lambda 3mo ago
This is very similar to my setup. Pi in a container (I do let it have network access, just no access to creds or anything, only the one directory that I'm working on at the time and my ~/.pi directory), talking to llama.cpp in another container. I'm on a Strix Halo 128 GiB unified memory laptop. I've never used the frontier models in earnest, I don't believe in using proprietary tools for my programming, so I can't really compare. And I'm still a AI skeptic, so I'm doing more testing and kicking the tires than I am actually using it. That means I spend a lot of time trying to break various models, probe them for strengths and weaknesses, etc. But I find that when I do try to use it for real for agentic coding, Qwen 3.6 35B-A3B is definitely the one I reach for the most often. For other chat tasks and translation, I'll frequently use Gemma 4 31B. For audio, I'll use Gemma 4 12B. I keep a bunch of other models around to try out every once in a while (Qwen 3.5 122B-A10B, Qwen 3.6 27B, Nemotron 3 Super 122B-A12B, Step 3.7 Flash and Minimax M2.7 both at somewhat more aggressive quants, and GPT-OSS 120B if I want super fast but not terribly smart), but so far Qwen 3.6 35B-A3B is really the sweet spot for coding on a setup like this.
- mahadevank 3mo ago
  Thanks a lot for your comment. I was using Qwen3 but asn't aware ofo the A3B Mixture-of-experts model. Works much better, thanks
- chakspak 3mo ago
  Hopefully this isn't off-topic, but your setup sounds just like mine, Strix Halo and (I'm assuming) llama.cpp on ROCm, and I'm finding that the Qwen hybrid models don't handle prompt caching and instead re-process the context in full on every turn. I'm wondering if you were able to solve this and how?
  - lambda 3mo ago
    I use Vulkan mostly instead of ROCm. Vulkan is actually a bit faster, paradoxically. I do switch out and try them both out, and it's not a huge difference, but I've been mostly saying on Vulkan. The re-processing context every turn problem is definitely something I've hit. Some of the causes have been solved upstream in llama.cpp; make sure you're up to date. But another cause of the issue that has a big effect is that older Qwen models didn't support preserving thinking. This means that each time you have a long sequence of tool calls with interleaved thinkging, as soon as you had your next turn in the chat, it would have to re-process all of that as it would drop all of the reasoning. Qwen 3.6, however, now supports preserving thinking. This can use a bit more context, becasue you're not dropping the thinking every turn, but it re-uses the cache better, not causing you to have to reprocess a whole turn at a time each time. In my models.ini, I have this for the Qwen3.6 models: chat-template-kwargs = {"preserve_thinking": true} There are still occasional issues I hit where it will have to re-process, but getting up to date and enabling preserve_thinking has helped a ton.
nobody_r_knows 3mo ago
[dead]
jmuguy 3mo ago
Given your knowledge on this - do you think we'll see an open source model with Opus levels of capability? IMO if/when this happens - I would 100% stop using Anthropic.
- zozbot234 3mo ago
  People can't seem to agree on what "Opus class" even means (the latest Opus is apparently pretty weak) but DeepSeek Pro, Kimi and GLM all are quite capable.
  - computerex 3mo ago
    Nothing compares to Opus when it comes to "taste" in web design in my experience. Nothing compares to opus in very difficult HPC/model inference development. I worked on this with opus: https://github.com/computerex/dlgo https://github.com/computerex/dlgo OpenAI was offering 2x usage at one point and I still used opus just because it's so much more effective.
    - lambda 3mo ago
      Which Opus? Anthropic has been releasing models named Opus since 2024 with Claude 3 Opus. Opus has gotten vastly more capable since then. Local model far surpass Opus 3. They even surpass Opus 4 on most benchmarks. Sure, if you compare to the latest Opus 4.8 or even 4.6, they're not there yet. But there's a huge difference in performance between 4 and 4.8.
      - jkells 3mo ago
        Can't speak for anyone else but there was a step change in frontier models last November. Opus 4.5 and GPT 5.2 I think. When I colloquially say Opus level I really mean Opus 4.5 or later
        lambda 3mo ago
        Right. Local models haven't quite hit that level yet. The biggest open models, which you need tens of thousands of dollars of hardware to run at reasonable speed, have pretty much hit that level of capability, but most models you can reasonably run at home aren't quite there yet. But given the gap, if local models keep improving, you'd expect to maybe see that level by this November.
adyavanapalli 3mo ago
For the edit tool, you should consider implementing a hash-based approach where each line of code is hashed and referenced by it when doing replacements. You can read up on the approach here: https://blog.can.ac/2026/02/12/the-harness-problem/ https://blog.can.ac/2026/02/12/the-harness-problem/ I didn't do much benchmarking, but anecdotally, I found it to be making less edit errors. YMMV
- pieterk 3mo ago
  Yup, I used this for a while and IME it may get you a few percentages more of useful context initially, so quality feels a bit higher, but things start breaking down in funnier ways when you do run out of that quality for any reason later, so definitely caveat emptor.
hparadiz 3mo ago
I am right there with you. Mind-boggling. It's a indistinguishable from magic technology!! I tried running some basic tasks through Qwen with Opencode on a 10 year old dual Xeon server for shits and giggles. I gave it a simple task like "use ffprobe first but convert this webm to mp4" and it was able to complete the task with zero network calls outside my network. On 10 year old hardware. It took about 3 minutes to complete the task. Now you may be saying 3 minutes? pfft. But I dare you to do it yourself. You're gonna be googling the CLI switches for at least 10 minutes and setting up your command. I had it actually optimize all the switches on the fly for me based on an initial ffprobe to see what is optimal.
- bluerooibos 3mo ago
  > 10 year old dual Xeon server...On 10 year old hardware. Hold on, what are the specs of your rig? How much RAM? I've been considering getting an old refurbished 2018 Mac Mini with 64Gb of DDR4 RAM but everything I've read suggests this will be way slower than my 16Gb M1 Pro Macbook.
  - hparadiz 3mo ago
    I inherited a box with dual Xeons and 256 GB of DDR4. I then ran several tests and benchmarks of the hardware with several models. I've been meaning to write a blog post but well whatever here's the md. https://gist.github.com/hparadiz/f3596d00a62d8ebb2dadcc46ee5822c7 https://gist.github.com/hparadiz/f3596d00a62d8ebb2dadcc46ee5... Qwen3.5 9B performed best. You can absolutely still use this to do some basic stuff like tell opencode to convert a video file from one format to another. But frankly you're better off getting two AMD GPUs. Say a dual 7900XT would get way better performance.
  - linzhangrun 3mo ago
    No need to touch the Macintosh from the X86 era
- bandrami 3mo ago
  > You're gonna be googling the CLI switches for at least 10 minutes So there's this really amazing program called "man"
electronsoup 3mo ago
> It gets into loops quite often, and surprisingly often gets the edit tool call wrong I find that running better quantization, like Q8 tend to prevent this even though its a bit slower to run, it saves overall time with less churn Using 3.6-27b is even slower again than 3.6-35b, but I find the accuracy really pays off
- girvo 3mo ago
  Right. Tokens/s decode isn't the most important thing to me: wall clock time for task completion is. And tracking all of that, on my GB10-based Asus box, Step 3.7 Flash at IQ4_XS beats Qwen 3.6 27B despite the latter having MTP, on all of my actual coding task evaluations in real codebases. Qwen seems better at one-shotting things based on vague prompts to an acceptable degree, but thats literally not what I use these things for! One thing if people do play with it, is it seems very very sensitive to quantisation of the K part of the KV cache. F16 K and Q8 V got rid of a lot of the loops that it was otherwise hitting. There's also a regression in llama.cpp wrt. Step Flash, where quantisation is getting worse KLD and Perplexity than it otherwise was previously, for the exact same quants. Very odd, but it's being looked into at least!
  - ttoinou 3mo ago
    I tried Step 3.7 Flash on my mac 128GB and it seemed very dumb. antirez ds4 flash is much better !
    - girvo 3mo ago
      It isn’t though, I’ve run both through a bunch of coding evals. You nearly certainly didn’t have the right sampling parameters or quantised the KV cache? Ds4 is impressive for what it is, but it loops and over thinks even more, burning massive wall clock time to not even get great outcomes. It’s also limited to a slow speed on my Spark
      - ttoinou 3mo ago
        I tried a bunch of stuff with step 3.5 and step 3.7 maybe not as much as you. Could you tell me what parameters and launched you’re using ? Antirez ds4 flash q2-q4 works almost out of the box for me
ltononro 3mo ago
What kind of coding do you do? Do you keep track of frontier models to vibe check the differences and re-evaluate constantly or are you ok with having a nerfed model forever? (not being judmental, just really wanto to know your framework here)
- Greenpants 3mo ago
  Some of the work I do, I do for an (EU) organisation that doesn't have clear rules or guidelines on the use of AI yet. Though I have seen colleague-developers blatantly putting source code into external Claude-like models, I stay true to my principles and don't. I know for certain that everything that I run through my local, offline Pi Container Sandbox cannot leave the machine, and thus can't result in a data breach. I do this for the peace of mind. I do (unscientifically) experiment whenever a new capable local LLM (<=130b) releases with a license that permits commercial use. As for knowing my models require more work than Opus, I don't mind still having to puzzle on getting the architecture right. In any case, it forces me to stay in the loop of what's being built, which is a good thing.
  - kordlessagain 3mo ago
    I'm adding Pi to Nemesis8 right now because I saw your comment, so thank you! https://github.com/DeepBlueDynamics/nemesis8 https://github.com/DeepBlueDynamics/nemesis8
  - psychoslave 3mo ago
    Could you give more details on how to make such a set up? I'm not familiar with Pi, and not sure which kind of container you are referring to. Something mainstream like docker, or more classic like a BSD jail? I started to experiment with locale LLMs, through ollama and Lemonade. Enough to throw simple prompts with code excerpts and get small scope code refactors. Though I still struggled to make them work with external tools, like my IDE, so they can be leveraged on to an agentic level with access to a full repository. That's mainly for work, as they push for using LLMs, though with the new copilote license they provide it doesn't take me even a week to burn the whole token credit. The tool can be useful, but in my experience without heavy guard rails and loops over tests. I suspect late models to also burn many token into rabbit hole of nonsense hypothesis, instead of doing straight forward correct implemention as you would expect from any entity with such a huge cumulated resources eaten and experimental playground to leverage on. Maybe incentives don't help model provider to minimize sold token, maybe it's just so hard to tame the beast all these bright minds with virtually infinite resources are not good enough. Anyway, sorry for digression, but I would be extremely interested with a step by step tutorial to make a local LLM work in agentic level, including which kind of hardware is required to make it work properly.
amelius 3mo ago
Sounds super cool, don't get me wrong, but I suppose for most people the bar is higher than HTML/CSS.
- q3k 3mo ago
  I love to warm up a whole rack of servers just so that some shitass buggy TUI can generate a line of bash that comments out my test runner. We truly live in the dumbest timeline.
- nozzlegear 3mo ago
  I use local LLMs on my Mac Studio to write and pass unit test suites in F#, among other boring project chores I don't want to do myself.
motbus3 3mo ago
Try deepseek V4 flash
- agnelnieves 3mo ago
  there goes the rest of my night
yieldcrv 3mo ago
> It gets into loops quite often matches my experience and a deal breaker also the context window sizes are too low. I can't operate in 65,000 windows any more because even just reading the code's file structure overruns it and gets me nowhere. Definitely its own art form. 200k context windows and above for me now I saw a paper last night that should help this a lot though
- kennywinker 3mo ago
  Qwen3.6-35b handles 256k context fine if you’ve got room for it. I’m running it with 128k context with just 16gb vram.
- Greenpants 3mo ago
  I get that it's a deal breaker to some; it definitely requires patience. In Pi, /new is my best friend and most-used command for sure. For simple tasks (I decompose complex ones anyway since I don't trust small local LLMs to do this for me), the model doesn't need much context, given that I'm proficient in my codebase myself: "I'd like Feature X. Look into files 1, 2 and 3 to make your edits."
nyxtom 3mo ago
Have you found that being much more spec driven helps guide it better?
dotancohen 3mo ago
> you really need to know what you're asking, and be precise Any chance that you could share some recent prompts to give other HNers a head start on his to approach Qwen? If you are uncomfortable posting them here, my Gmail username is the same as my HN username. Thank you.
- Greenpants 3mo ago
  I'm glad you're asking. I already started writing a blog post on how to best make use of local models. I'll share it as soon as I have a complete enough list. If anyone else reading this would like to chime in with their tips & tricks, let us know! For the time being, off the top of my head, I'd say: - Prompt Engineering tips & tricks apply here (like being complete in the relevant context you provide in your question, and the specific task(s) the agent should do like reasoning, modifying one file, or trying to fix a complex task all at once (not recommended)). - If you already know which files the agent should look into, mention them to save time and potentially context. - In my personal workflow, I write down lots of atomic TODOs needed to solve a problem. As I write it down, I'll notice assumptions I'm making, or the fact that the TODO could still be decomposed further into (atomic) subtasks. - It's best to get a feeling yourself for how Qwen handles your repository. I noticed if I don't specify an architecture for development, it'll make quick & dirty fixes. If I don't tell it to remove debug statements, it won't. This is what was meant with "be precise" – Claude Opus might think for you and act in your best interest. Smaller Qwen models will just do what you ask them to, and no more. They have design knowledge, but you have to explicitly ask them to "activate" that part of their knowledge.
  - dotancohen 3mo ago
    Thank you, that was extraordinarily helpful. I look forward to that blog post!
  - tsss 3mo ago
    But if you have to write everything down in such detail, isn't it faster to just do the task yourself?
  - thefossguy69 3mo ago
spullara 3mo ago
This is the only setup that I think is reasonable to use locally right now. I had an agent set it up for me from this guys recipe: https://ikyle.me/blog/2026/how-to-setup-a-local-coding-agent-on-macos https://ikyle.me/blog/2026/how-to-setup-a-local-coding-agent... One thing I did change was the context length to 256k rather than 64k.
pieterk 3mo ago
Yup, it's fantastically useful. Maybe even more useful than Opus when I have all the constraints to an issue. There is less "knowledge" in the model (I get by with 48GB of RAM allocated to an 8b quant), so it has fewer things to hallucinate about. I've been getting to know its limits pretty well over the last few weeks and would say it's an excellent code search/replacement/generation* engine. It's got the "in-context script generation" flow down as well, so it will easily help automate tasks that you describe with text and perhaps example commands, or tools, or skills* that you provide. *Think of it + Pi as an NLP abstraction layer over grep, or a shell, rather than a jack of all trades + world knowledge all-in-one.
timmit 3mo ago
I got a 48GB Ram MacBook, somehow I cannot even run a 20b model, I was suprised that you get 35b model locally.
- klardotsh 3mo ago
  4-5 bit quants would probably fit pretty well on your rig. Check HuggingFace for Qwen3.6-35B-A3B-MTP-GGUF [1]. They've also got a cool UI thing these days to help indicate which quants of a model will run on your hardware. Full octane isn't gonna fit on much of anything south of a 128GB machine once adding KV cache. [1]: https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF
ojr 3mo ago
I can use Gemini 3 Flash with the harness I built for around 8 years and still not exceed the cost of a Mac Studio with 128GB, the price for privacy is very high. Agentic flows that get stuck can be worked around but I prefer developer velocity.
- disqard 3mo ago
  Under-rated take, thanks for stating this! Not everyone can plough $$$$ into hardware right now (more power to those who can), so choosing to rent is an A-Ok strategy.
  - tpm 3mo ago
    It's ok if you can send your code and data to the provider. Some of us can't.
    - _zoltan_ 3mo ago
      We're discussing home use. You can. You just don't want to. Huge difference.
      - monooso 3mo ago
        > We're discussing home use. You may be, but the topic of discussion is whether anyone is using a local model as their main coding tool.
        _zoltan_ 3mo ago
        for corporate use it's a mistake not to use a frontier model.
      - tpm 3mo ago
        Well plenty of people work from home. For corporate use, if the corporation would break the law sending anything to the open internet or to the US, then you can't use any model that's not hosted in house. And there are many such cases.
geophile 3mo ago
My experience is almost identical. I have found that I need to be very careful with planning, breaking things down into small isolated steps (I can have qwen do this); and also (me) writing a very clear design. Relying on qwen to fill in a lot of those precise details results in those about-to-write loops. Yeah, that edit inability is weird. I’ve updated AGENTS.md to limit editing (as opposed to rewriting) and that helps a little.
westoque 3mo ago
> Comparing agentic Qwen3.6 35b to Claude Opus is like a junior with knowledge across the board, that you really need to guide, versus a senior that thinks with you on architecture. that's why i use the frontier models because its a senior co-worker vs a junior. if you use the junior for the sake of privacy i think you're missing out on the best insights for a specific task.
- physix 3mo ago
  The dilemma I am facing is cost. Consumer-grade subscriptions of the frontier models give you superb capabilities per dollar, them being heavily subsidized. But if you're working in an enterprise setting, that won't work. You need to upgrade, and that gets significantly more expensive. Furthermore, basing the SDLC on leveraging the bargain subscriptions risks falling apart in the future, both from a cost perspective as well as the question of availability (e.g. Mythos). So from a strategic perspective, going local on the LLM and still achieving great results with the right approach is very relevant.
  - willisrocks 3mo ago
    Or you can get the best of both worlds--use frontier models to build a spec/plan, and use cheap models (open source or not) for implementation. Your max or team plan can go a lot further this way without giving up much for quality. Play with something like Superpowers to make this really approachable.
- bxk76 3mo ago
  Best insights can be over rated due to bandwith limitation of the brain. Even if Einstein is sitting next to you the whole day and helping out Theory of Bounded Rationality applies.
gwerbin 3mo ago
I've noticed the same about the edit tool, in both Gemma and Qwen. Maybe I'm not running them with the right sampler settings, but I'm happy to hear I'm not the only one. Lots of mismatched whitespace and stuff, the model ends up doing hex dumps and maybe 5 or 6 attempts at editing a 5-line function into a 250-line Python file. All of these models also seem to get stuck in long thinking loops, sometimes tripling the tokens of a frontier closed model which is really painful when inference is already on the slow side (on my Macbook).
rjblackman 3mo ago
it might be worth trying oh-my-pi in your case as it claims to improve the edit calls by using a unique patching format.
awllau 3mo ago
Based on your explanation, it doesn't sound feasible for me, a complete non-engineer, to switch to fully offline? I do a lot of back and forth discussion with LLMs as someone who reads and writes 0 code.
- Greenpants 3mo ago
  I'm afraid I'd have to agree. That is, unless you have 512GB+ RAM sitting on a shelf and run the much larger SOTA-comparable local models.
krainboltgreene 3mo ago
> is like a junior with knowledge across the board, that you really need to guide, versus a senior that thinks with you on architecture I don't want to be rude, but your linkedin has a sumtotal (generous) of like 8 months of programming as a profession (job title is AI Engineer). The rest is at best programming adjacent. How would you know what either of these situations are really like?
- SoftTalker 3mo ago
  I haven't logged in to LinkedIn or looked at it since a former employer demanded that everyone create a profile. So mine is now about 20 years out of date.
  - krainboltgreene 3mo ago
    His is very up to date. Not everyone is you.
animanoir 3mo ago
[dead]
nicman23 3mo ago
about the edit tool it is almost always trailing white spaces. if you give it a skill with a sed 's/( )*$//g' or something like that it speeds up things
kristopolous 3mo ago
I've got a tool that sits in between the harness and inference engine called petsitter. It is a middleman validator to avoid just these kinds of issues. You can stack the fixes as needed (they're called tricks in the petsitter parlance) It's what I use. Fixes the problem https://github.com/day50-dev/petsitter https://github.com/day50-dev/petsitter
robertlagrant 3mo ago
How are you sandboxing your Pi coding harness? Directly only mounting certain folders, using capabilities to kill the network and not giving it all your shell env vars, that sort of thing? Or do you use a tool?
- throw10920 3mo ago
  And, is the sandboxing for security (avoid RCE on the host) or merely guardrails for the models? I've wanted the latter quite a bit for Pi, because weaker models like Deepseek V4 have extreme issues with obeying prompts (e.g. I'll instruct it to find a bug but not fix it, and it'll "helpfully" try to fix it anyway), so having a "read-only mode" actually backed by the OS would be very useful.
  - SeriousM 3mo ago
    Haha, yes! Last time I asked it for options how to tackle a task and only do the research without touching any code. With xhigh rasoning, it echoed the options that many times until it was convinced that option A is the better choice and started implementing it.
- fidelramos 3mo ago
  I'm using firejail to sandbox Opencode, for security and to keep the agents from personal data. I documented it in my blog [1]. [1] https://blog.fidelramos.net/software/how-i-sandbox-ai-agents#how-i-sandbox-ai-agents https://blog.fidelramos.net/software/how-i-sandbox-ai-agents...
  - robertlagrant 3mo ago
    Thank you - great blog post.
stared 3mo ago
Why I do like Qwen 3.6 35B A3B, I have found that the difference improvement of Qwen 3.6 27B is massive. Sure, it is 3x slower (https://github.com/stared/benching-local-llms-on-apple-silicon https://github.com/stared/benching-local-llms-on-apple-silic...), but for the total development time it felt that still 27B is faster to get the goal. Is it that in your case is it different?
underdeserver 3mo ago
Nit - it is not completely free. You are paying for the extra power draw.
MoonWalk 3mo ago
This is good info, thanks. I want to do something similar, but know very little about how to set the components of LLMs up. I've read a bit on what the various components are. What I don't see in your comment is what you're using to run your model locally. Ollama?
calenti 3mo ago
What IDE do you use? How do you integrate it? I was using Continue but it exited its funding round to the Titler octopus and the Chat function in VSCode is choking on the Ollama responses.
- Greenpants 3mo ago
  That's precisely why my agent use is IDE-agnostic: I run Pi in any terminal. Often use it with the terminal inside VSCodium, though sometimes in a terminal outside an IDE if I don't expect to edit any files myself (e.g. for small one-shot projects).
vizually 3mo ago
@greenpants, "Pi coding harness but containerized and sandboxed" care to address some specifics and/or reference implementation for this. may be a GITHub URL?

p0w3n3d 3mo ago

which coding agent are you using?

GardenLetter27 3mo ago

Could the harness not check for a failed tool call and pass it to a small model for correction without clogging up the main context?

Greenpants 3mo ago
I'm actually quite sure that directly retrying the tool call would often fix the edit-call already. But these models have been trained to "think" for a while for any problem solving, so they'll presume the problem of the edit is more fundamental and spend unnecessary tokens filling up the context. I'll experiment more with the effectiveness of AGENTS.md rules for local Pi agents. I feel like smaller (local) LLMs just lack in attentiveness to elements in the context window, like precise instructions, compared to e.g. Claude models.
lambda 3mo ago
The thing is, to do a proper fix it would really need all of the context (maybe the tool call that failed was for an edit to a file that was last touched way at the beginning of the context), so you'd need to either keep that smaller model running doing prompt processing all the time, or have a very long wait while it does prompt processing on your whole session. And then also, sometimes the tool call errors are because of something like a file was changed out from under it; the larger model is probably going to do a better job of figuring that out and fixing it up. Finally, in Pi, you can always just use the /tree command to skip back to before a series of failed tool calls, with a summary if you want to let the model know what happened. The Pi /tree command is pretty powerful in managing your context
- everforward 3mo ago
  An illustrative example I've seen a lot is creating Jira tickets in projects with custom fields marked as mandatory. It tries to create the ticket without the field and the tool call fails. The LLM needs access to the full context so that it can generate text to put in the "Why couldn't this meeting be an email?" field.

0xbadcafebee 3mo ago

The harness and the LLM parameters are pretty essential to getting better results and reducing loops. Tweak the parameters and you can mostly eliminate loops without negatively affecting performance (it's a bit complex but ask a SOTA AI to guide you and it's not hard). The harness should also react more intelligently to failures; it can do things like return additional context or hints as it tracks error rates and avg duration of calls. Pi can be easily extended, and it's suggested by the author you modify it to perform better for your use case.

lambda 3mo ago

This is very similar to my setup. Pi in a container (I do let it have network access, just no access to creds or anything, only the one directory that I'm working on at the time and my ~/.pi directory), talking to llama.cpp in another container. I'm on a Strix Halo 128 GiB unified memory laptop. I've never used the frontier models in earnest, I don't believe in using proprietary tools for my programming, so I can't really compare. And I'm still a AI skeptic, so I'm doing more testing and kicking the tires than I am actually using it. That means I spend a lot of time trying to break various models, probe them for strengths and weaknesses, etc. But I find that when I do try to use it for real for agentic coding, Qwen 3.6 35B-A3B is definitely the one I reach for the most often. For other chat tasks and translation, I'll frequently use Gemma 4 31B. For audio, I'll use Gemma 4 12B. I keep a bunch of other models around to try out every once in a while (Qwen 3.5 122B-A10B, Qwen 3.6 27B, Nemotron 3 Super 122B-A12B, Step 3.7 Flash and Minimax M2.7 both at somewhat more aggressive quants, and GPT-OSS 120B if I want super fast but not terribly smart), but so far Qwen 3.6 35B-A3B is really the sweet spot for coding on a setup like this.

mahadevank 3mo ago
Thanks a lot for your comment. I was using Qwen3 but asn't aware ofo the A3B Mixture-of-experts model. Works much better, thanks
chakspak 3mo ago
Hopefully this isn't off-topic, but your setup sounds just like mine, Strix Halo and (I'm assuming) llama.cpp on ROCm, and I'm finding that the Qwen hybrid models don't handle prompt caching and instead re-process the context in full on every turn. I'm wondering if you were able to solve this and how?
- lambda 3mo ago
  I use Vulkan mostly instead of ROCm. Vulkan is actually a bit faster, paradoxically. I do switch out and try them both out, and it's not a huge difference, but I've been mostly saying on Vulkan. The re-processing context every turn problem is definitely something I've hit. Some of the causes have been solved upstream in llama.cpp; make sure you're up to date. But another cause of the issue that has a big effect is that older Qwen models didn't support preserving thinking. This means that each time you have a long sequence of tool calls with interleaved thinkging, as soon as you had your next turn in the chat, it would have to re-process all of that as it would drop all of the reasoning. Qwen 3.6, however, now supports preserving thinking. This can use a bit more context, becasue you're not dropping the thinking every turn, but it re-uses the cache better, not causing you to have to reprocess a whole turn at a time each time. In my models.ini, I have this for the Qwen3.6 models: chat-template-kwargs = {"preserve_thinking": true} There are still occasional issues I hit where it will have to re-process, but getting up to date and enabling preserve_thinking has helped a ton.

nobody_r_knows 3mo ago

[dead]

jmuguy 3mo ago

Given your knowledge on this - do you think we'll see an open source model with Opus levels of capability? IMO if/when this happens - I would 100% stop using Anthropic.

zozbot234 3mo ago
People can't seem to agree on what "Opus class" even means (the latest Opus is apparently pretty weak) but DeepSeek Pro, Kimi and GLM all are quite capable.
- computerex 3mo ago
  Nothing compares to Opus when it comes to "taste" in web design in my experience. Nothing compares to opus in very difficult HPC/model inference development. I worked on this with opus: https://github.com/computerex/dlgo https://github.com/computerex/dlgo OpenAI was offering 2x usage at one point and I still used opus just because it's so much more effective.
  - lambda 3mo ago
    Which Opus? Anthropic has been releasing models named Opus since 2024 with Claude 3 Opus. Opus has gotten vastly more capable since then. Local model far surpass Opus 3. They even surpass Opus 4 on most benchmarks. Sure, if you compare to the latest Opus 4.8 or even 4.6, they're not there yet. But there's a huge difference in performance between 4 and 4.8.
    - jkells 3mo ago
      Can't speak for anyone else but there was a step change in frontier models last November. Opus 4.5 and GPT 5.2 I think. When I colloquially say Opus level I really mean Opus 4.5 or later
      - lambda 3mo ago
        Right. Local models haven't quite hit that level yet. The biggest open models, which you need tens of thousands of dollars of hardware to run at reasonable speed, have pretty much hit that level of capability, but most models you can reasonably run at home aren't quite there yet. But given the gap, if local models keep improving, you'd expect to maybe see that level by this November.

adyavanapalli 3mo ago

For the edit tool, you should consider implementing a hash-based approach where each line of code is hashed and referenced by it when doing replacements. You can read up on the approach here: https://blog.can.ac/2026/02/12/the-harness-problem/ https://blog.can.ac/2026/02/12/the-harness-problem/ I didn't do much benchmarking, but anecdotally, I found it to be making less edit errors. YMMV

pieterk 3mo ago
Yup, I used this for a while and IME it may get you a few percentages more of useful context initially, so quality feels a bit higher, but things start breaking down in funnier ways when you do run out of that quality for any reason later, so definitely caveat emptor.

hparadiz 3mo ago

I am right there with you. Mind-boggling. It's a indistinguishable from magic technology!! I tried running some basic tasks through Qwen with Opencode on a 10 year old dual Xeon server for shits and giggles. I gave it a simple task like "use ffprobe first but convert this webm to mp4" and it was able to complete the task with zero network calls outside my network. On 10 year old hardware. It took about 3 minutes to complete the task. Now you may be saying 3 minutes? pfft. But I dare you to do it yourself. You're gonna be googling the CLI switches for at least 10 minutes and setting up your command. I had it actually optimize all the switches on the fly for me based on an initial ffprobe to see what is optimal.

bluerooibos 3mo ago
> 10 year old dual Xeon server...On 10 year old hardware. Hold on, what are the specs of your rig? How much RAM? I've been considering getting an old refurbished 2018 Mac Mini with 64Gb of DDR4 RAM but everything I've read suggests this will be way slower than my 16Gb M1 Pro Macbook.
- hparadiz 3mo ago
  I inherited a box with dual Xeons and 256 GB of DDR4. I then ran several tests and benchmarks of the hardware with several models. I've been meaning to write a blog post but well whatever here's the md. https://gist.github.com/hparadiz/f3596d00a62d8ebb2dadcc46ee5822c7 https://gist.github.com/hparadiz/f3596d00a62d8ebb2dadcc46ee5... Qwen3.5 9B performed best. You can absolutely still use this to do some basic stuff like tell opencode to convert a video file from one format to another. But frankly you're better off getting two AMD GPUs. Say a dual 7900XT would get way better performance.
- linzhangrun 3mo ago
  No need to touch the Macintosh from the X86 era
bandrami 3mo ago
> You're gonna be googling the CLI switches for at least 10 minutes So there's this really amazing program called "man"

electronsoup 3mo ago

> It gets into loops quite often, and surprisingly often gets the edit tool call wrong I find that running better quantization, like Q8 tend to prevent this even though its a bit slower to run, it saves overall time with less churn Using 3.6-27b is even slower again than 3.6-35b, but I find the accuracy really pays off

girvo 3mo ago
Right. Tokens/s decode isn't the most important thing to me: wall clock time for task completion is. And tracking all of that, on my GB10-based Asus box, Step 3.7 Flash at IQ4_XS beats Qwen 3.6 27B despite the latter having MTP, on all of my actual coding task evaluations in real codebases. Qwen seems better at one-shotting things based on vague prompts to an acceptable degree, but thats literally not what I use these things for! One thing if people do play with it, is it seems very very sensitive to quantisation of the K part of the KV cache. F16 K and Q8 V got rid of a lot of the loops that it was otherwise hitting. There's also a regression in llama.cpp wrt. Step Flash, where quantisation is getting worse KLD and Perplexity than it otherwise was previously, for the exact same quants. Very odd, but it's being looked into at least!
- ttoinou 3mo ago
  I tried Step 3.7 Flash on my mac 128GB and it seemed very dumb. antirez ds4 flash is much better !
  - girvo 3mo ago
    It isn’t though, I’ve run both through a bunch of coding evals. You nearly certainly didn’t have the right sampling parameters or quantised the KV cache? Ds4 is impressive for what it is, but it loops and over thinks even more, burning massive wall clock time to not even get great outcomes. It’s also limited to a slow speed on my Spark
    - ttoinou 3mo ago
      I tried a bunch of stuff with step 3.5 and step 3.7 maybe not as much as you. Could you tell me what parameters and launched you’re using ? Antirez ds4 flash q2-q4 works almost out of the box for me

ltononro 3mo ago

What kind of coding do you do? Do you keep track of frontier models to vibe check the differences and re-evaluate constantly or are you ok with having a nerfed model forever? (not being judmental, just really wanto to know your framework here)

Greenpants 3mo ago
Some of the work I do, I do for an (EU) organisation that doesn't have clear rules or guidelines on the use of AI yet. Though I have seen colleague-developers blatantly putting source code into external Claude-like models, I stay true to my principles and don't. I know for certain that everything that I run through my local, offline Pi Container Sandbox cannot leave the machine, and thus can't result in a data breach. I do this for the peace of mind. I do (unscientifically) experiment whenever a new capable local LLM (<=130b) releases with a license that permits commercial use. As for knowing my models require more work than Opus, I don't mind still having to puzzle on getting the architecture right. In any case, it forces me to stay in the loop of what's being built, which is a good thing.
- kordlessagain 3mo ago
  I'm adding Pi to Nemesis8 right now because I saw your comment, so thank you! https://github.com/DeepBlueDynamics/nemesis8 https://github.com/DeepBlueDynamics/nemesis8
- psychoslave 3mo ago
  Could you give more details on how to make such a set up? I'm not familiar with Pi, and not sure which kind of container you are referring to. Something mainstream like docker, or more classic like a BSD jail? I started to experiment with locale LLMs, through ollama and Lemonade. Enough to throw simple prompts with code excerpts and get small scope code refactors. Though I still struggled to make them work with external tools, like my IDE, so they can be leveraged on to an agentic level with access to a full repository. That's mainly for work, as they push for using LLMs, though with the new copilote license they provide it doesn't take me even a week to burn the whole token credit. The tool can be useful, but in my experience without heavy guard rails and loops over tests. I suspect late models to also burn many token into rabbit hole of nonsense hypothesis, instead of doing straight forward correct implemention as you would expect from any entity with such a huge cumulated resources eaten and experimental playground to leverage on. Maybe incentives don't help model provider to minimize sold token, maybe it's just so hard to tame the beast all these bright minds with virtually infinite resources are not good enough. Anyway, sorry for digression, but I would be extremely interested with a step by step tutorial to make a local LLM work in agentic level, including which kind of hardware is required to make it work properly.

amelius 3mo ago

Sounds super cool, don't get me wrong, but I suppose for most people the bar is higher than HTML/CSS.

q3k 3mo ago
I love to warm up a whole rack of servers just so that some shitass buggy TUI can generate a line of bash that comments out my test runner. We truly live in the dumbest timeline.
nozzlegear 3mo ago
I use local LLMs on my Mac Studio to write and pass unit test suites in F#, among other boring project chores I don't want to do myself.

motbus3 3mo ago

Try deepseek V4 flash

agnelnieves 3mo ago
there goes the rest of my night

yieldcrv 3mo ago

> It gets into loops quite often matches my experience and a deal breaker also the context window sizes are too low. I can't operate in 65,000 windows any more because even just reading the code's file structure overruns it and gets me nowhere. Definitely its own art form. 200k context windows and above for me now I saw a paper last night that should help this a lot though

kennywinker 3mo ago
Qwen3.6-35b handles 256k context fine if you’ve got room for it. I’m running it with 128k context with just 16gb vram.
Greenpants 3mo ago
I get that it's a deal breaker to some; it definitely requires patience. In Pi, /new is my best friend and most-used command for sure. For simple tasks (I decompose complex ones anyway since I don't trust small local LLMs to do this for me), the model doesn't need much context, given that I'm proficient in my codebase myself: "I'd like Feature X. Look into files 1, 2 and 3 to make your edits."

nyxtom 3mo ago

Have you found that being much more spec driven helps guide it better?

dotancohen 3mo ago

> you really need to know what you're asking, and be precise Any chance that you could share some recent prompts to give other HNers a head start on his to approach Qwen? If you are uncomfortable posting them here, my Gmail username is the same as my HN username. Thank you.

Greenpants 3mo ago
I'm glad you're asking. I already started writing a blog post on how to best make use of local models. I'll share it as soon as I have a complete enough list. If anyone else reading this would like to chime in with their tips & tricks, let us know! For the time being, off the top of my head, I'd say: - Prompt Engineering tips & tricks apply here (like being complete in the relevant context you provide in your question, and the specific task(s) the agent should do like reasoning, modifying one file, or trying to fix a complex task all at once (not recommended)). - If you already know which files the agent should look into, mention them to save time and potentially context. - In my personal workflow, I write down lots of atomic TODOs needed to solve a problem. As I write it down, I'll notice assumptions I'm making, or the fact that the TODO could still be decomposed further into (atomic) subtasks. - It's best to get a feeling yourself for how Qwen handles your repository. I noticed if I don't specify an architecture for development, it'll make quick & dirty fixes. If I don't tell it to remove debug statements, it won't. This is what was meant with "be precise" – Claude Opus might think for you and act in your best interest. Smaller Qwen models will just do what you ask them to, and no more. They have design knowledge, but you have to explicitly ask them to "activate" that part of their knowledge.
- dotancohen 3mo ago
  Thank you, that was extraordinarily helpful. I look forward to that blog post!
- tsss 3mo ago
  But if you have to write everything down in such detail, isn't it faster to just do the task yourself?
- thefossguy69 3mo ago

spullara 3mo ago

This is the only setup that I think is reasonable to use locally right now. I had an agent set it up for me from this guys recipe: https://ikyle.me/blog/2026/how-to-setup-a-local-coding-agent-on-macos https://ikyle.me/blog/2026/how-to-setup-a-local-coding-agent... One thing I did change was the context length to 256k rather than 64k.

pieterk 3mo ago

Yup, it's fantastically useful. Maybe even more useful than Opus when I have all the constraints to an issue. There is less "knowledge" in the model (I get by with 48GB of RAM allocated to an 8b quant), so it has fewer things to hallucinate about. I've been getting to know its limits pretty well over the last few weeks and would say it's an excellent code search/replacement/generation* engine. It's got the "in-context script generation" flow down as well, so it will easily help automate tasks that you describe with text and perhaps example commands, or tools, or skills* that you provide. *Think of it + Pi as an NLP abstraction layer over grep, or a shell, rather than a jack of all trades + world knowledge all-in-one.

timmit 3mo ago

I got a 48GB Ram MacBook, somehow I cannot even run a 20b model, I was suprised that you get 35b model locally.

klardotsh 3mo ago
4-5 bit quants would probably fit pretty well on your rig. Check HuggingFace for Qwen3.6-35B-A3B-MTP-GGUF [1]. They've also got a cool UI thing these days to help indicate which quants of a model will run on your hardware. Full octane isn't gonna fit on much of anything south of a 128GB machine once adding KV cache. [1]: https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF

ojr 3mo ago

I can use Gemini 3 Flash with the harness I built for around 8 years and still not exceed the cost of a Mac Studio with 128GB, the price for privacy is very high. Agentic flows that get stuck can be worked around but I prefer developer velocity.

disqard 3mo ago
Under-rated take, thanks for stating this! Not everyone can plough $$$$ into hardware right now (more power to those who can), so choosing to rent is an A-Ok strategy.
- tpm 3mo ago
  It's ok if you can send your code and data to the provider. Some of us can't.
  - _zoltan_ 3mo ago
    We're discussing home use. You can. You just don't want to. Huge difference.
    - monooso 3mo ago
      > We're discussing home use. You may be, but the topic of discussion is whether anyone is using a local model as their main coding tool.
      - _zoltan_ 3mo ago
        for corporate use it's a mistake not to use a frontier model.
    - tpm 3mo ago
      Well plenty of people work from home. For corporate use, if the corporation would break the law sending anything to the open internet or to the US, then you can't use any model that's not hosted in house. And there are many such cases.

geophile 3mo ago

My experience is almost identical. I have found that I need to be very careful with planning, breaking things down into small isolated steps (I can have qwen do this); and also (me) writing a very clear design. Relying on qwen to fill in a lot of those precise details results in those about-to-write loops. Yeah, that edit inability is weird. I’ve updated AGENTS.md to limit editing (as opposed to rewriting) and that helps a little.

westoque 3mo ago

> Comparing agentic Qwen3.6 35b to Claude Opus is like a junior with knowledge across the board, that you really need to guide, versus a senior that thinks with you on architecture. that's why i use the frontier models because its a senior co-worker vs a junior. if you use the junior for the sake of privacy i think you're missing out on the best insights for a specific task.

physix 3mo ago
The dilemma I am facing is cost. Consumer-grade subscriptions of the frontier models give you superb capabilities per dollar, them being heavily subsidized. But if you're working in an enterprise setting, that won't work. You need to upgrade, and that gets significantly more expensive. Furthermore, basing the SDLC on leveraging the bargain subscriptions risks falling apart in the future, both from a cost perspective as well as the question of availability (e.g. Mythos). So from a strategic perspective, going local on the LLM and still achieving great results with the right approach is very relevant.
- willisrocks 3mo ago
  Or you can get the best of both worlds--use frontier models to build a spec/plan, and use cheap models (open source or not) for implementation. Your max or team plan can go a lot further this way without giving up much for quality. Play with something like Superpowers to make this really approachable.
bxk76 3mo ago
Best insights can be over rated due to bandwith limitation of the brain. Even if Einstein is sitting next to you the whole day and helping out Theory of Bounded Rationality applies.

gwerbin 3mo ago

I've noticed the same about the edit tool, in both Gemma and Qwen. Maybe I'm not running them with the right sampler settings, but I'm happy to hear I'm not the only one. Lots of mismatched whitespace and stuff, the model ends up doing hex dumps and maybe 5 or 6 attempts at editing a 5-line function into a 250-line Python file. All of these models also seem to get stuck in long thinking loops, sometimes tripling the tokens of a frontier closed model which is really painful when inference is already on the slow side (on my Macbook).

rjblackman 3mo ago

it might be worth trying oh-my-pi in your case as it claims to improve the edit calls by using a unique patching format.

awllau 3mo ago

Based on your explanation, it doesn't sound feasible for me, a complete non-engineer, to switch to fully offline? I do a lot of back and forth discussion with LLMs as someone who reads and writes 0 code.

Greenpants 3mo ago
I'm afraid I'd have to agree. That is, unless you have 512GB+ RAM sitting on a shelf and run the much larger SOTA-comparable local models.

krainboltgreene 3mo ago

> is like a junior with knowledge across the board, that you really need to guide, versus a senior that thinks with you on architecture I don't want to be rude, but your linkedin has a sumtotal (generous) of like 8 months of programming as a profession (job title is AI Engineer). The rest is at best programming adjacent. How would you know what either of these situations are really like?

SoftTalker 3mo ago
I haven't logged in to LinkedIn or looked at it since a former employer demanded that everyone create a profile. So mine is now about 20 years out of date.
- krainboltgreene 3mo ago
  His is very up to date. Not everyone is you.

animanoir 3mo ago

[dead]

nicman23 3mo ago

about the edit tool it is almost always trailing white spaces. if you give it a skill with a sed 's/( )*$//g' or something like that it speeds up things

kristopolous 3mo ago

I've got a tool that sits in between the harness and inference engine called petsitter. It is a middleman validator to avoid just these kinds of issues. You can stack the fixes as needed (they're called tricks in the petsitter parlance) It's what I use. Fixes the problem https://github.com/day50-dev/petsitter https://github.com/day50-dev/petsitter

robertlagrant 3mo ago

How are you sandboxing your Pi coding harness? Directly only mounting certain folders, using capabilities to kill the network and not giving it all your shell env vars, that sort of thing? Or do you use a tool?

throw10920 3mo ago
And, is the sandboxing for security (avoid RCE on the host) or merely guardrails for the models? I've wanted the latter quite a bit for Pi, because weaker models like Deepseek V4 have extreme issues with obeying prompts (e.g. I'll instruct it to find a bug but not fix it, and it'll "helpfully" try to fix it anyway), so having a "read-only mode" actually backed by the OS would be very useful.
- SeriousM 3mo ago
  Haha, yes! Last time I asked it for options how to tackle a task and only do the research without touching any code. With xhigh rasoning, it echoed the options that many times until it was convinced that option A is the better choice and started implementing it.
fidelramos 3mo ago
I'm using firejail to sandbox Opencode, for security and to keep the agents from personal data. I documented it in my blog [1]. [1] https://blog.fidelramos.net/software/how-i-sandbox-ai-agents#how-i-sandbox-ai-agents https://blog.fidelramos.net/software/how-i-sandbox-ai-agents...
- robertlagrant 3mo ago
  Thank you - great blog post.

stared 3mo ago

Why I do like Qwen 3.6 35B A3B, I have found that the difference improvement of Qwen 3.6 27B is massive. Sure, it is 3x slower (https://github.com/stared/benching-local-llms-on-apple-silicon https://github.com/stared/benching-local-llms-on-apple-silic...), but for the total development time it felt that still 27B is faster to get the goal. Is it that in your case is it different?

underdeserver 3mo ago

Nit - it is not completely free. You are paying for the extra power draw.

MoonWalk 3mo ago

This is good info, thanks. I want to do something similar, but know very little about how to set the components of LLMs up. I've read a bit on what the various components are. What I don't see in your comment is what you're using to run your model locally. Ollama?

calenti 3mo ago

What IDE do you use? How do you integrate it? I was using Continue but it exited its funding round to the Titler octopus and the Chat function in VSCode is choking on the Ollama responses.

Greenpants 3mo ago
That's precisely why my agent use is IDE-agnostic: I run Pi in any terminal. Often use it with the terminal inside VSCodium, though sometimes in a terminal outside an IDE if I don't expect to edit any files myself (e.g. for small one-shot projects).

vizually 3mo ago

@greenpants, "Pi coding harness but containerized and sandboxed" care to address some specifics and/or reference implementation for this. may be a GITHub URL?