9 ms·
It’s hard to elaborate just how wild this model might be if it performs as claimed. The claims are this can perform close to Sonnet 4.5 for assisted coding (SW
by skhameneh 8mo ago
It’s hard to elaborate just how wild this model might be if it performs as claimed. The claims are this can perform close to Sonnet 4.5 for assisted coding (SWE bench) while using only 3B active parameters. This is obscenely small for the claimed performance.
- cirrusfan 8mo agoIf it sounds too good to be true…
- theshrike79 8mo agoShould be possible with optimised models, just drop all "generic" stuff and focus on coding performance. There's no reason for a coding model to contain all of ao3 and wikipedia =)
- noveltyaccount 8mo agoI think I like coding models that know a lot about the world. They can disambiguate my requirements and build better products.
- regularfry 8mo agoI generally prefer a coding model that can google for the docs, but separate models for /plan and /build is also a thing.
- noveltyaccount 8mo ago> separate models for /plan and /build I had not considered that, seems like a great solution for local models that may be more resource-constrained.
- regularfry 8mo agoYou can configure aider that way. You get three, in fact: an architect model, a code editor model, and a quick model for things like commit messages. Although I'm not sure if it's got doc searching capabilities.
- MarsIronPI 8mo agoBut... but... I need my coding model to be able to write fanfiction in the comments...
- moffkalast 8mo agoThat's what Meta thought initially too, training codellama and chat llama separately, and then they realized they're idiots and that adding the other half of data vastly improves both models. As long as it's quality data, more of it doesn't do harm. Besides, programming is far from just knowing how to autocomplete syntax, you need a model that's proficient in the fields that the automation is placed in, otherwise they'll be no help in actually automating it.
- theshrike79 8mo agoBut as far as I know, that was way before tool calling was a thing. I'm more bullish about small and medium sized models + efficient tool calling than I'm about LLMs too large to be run at home without $20k of hardware. The model doesn't need to have the full knowledge of everything built into it when it has the toolset to fetch, cache and read any information available.
- jstummbillig 8mo agoThere is: It works (even if we can't explain why right now). If we knew how to create a SOTA coding model by just putting coding stuff in there, that is how we would build SOTA coding models.
- wongarsu 8mo agoNow I wonder how strong the correlation between coding performance and ao3 knowledge is in human programmers. Maybe we are on to something here /s
- FuckButtons 8mo agoThere have been advances recently (last year) in scaling deep rl by a significant amount, their announcement is in line with a timeline of running enough experiments to figure out how to leverage that in post training. Importantly, this isn’t just throwing more data at the problem in an unstructured way, afaik companies are getting as many got histories as they can and doing something along the lines of, get an llm to checkpoint pull requests, features etc and convert those into plausible input prompts, then run deep rl with something which passes the acceptance criteria / tests as the reward signal.
- Der_Einzige 8mo agoIt literally always is. HN Thought DeepSeek and every version of Kimi would finally dethrone the bigger models from Anthropic, OpenAI, and Google. They're literally always wrong and average knowledge of LLMs here is shockingly low.
- cmrdporcupine 8mo agoNobody has been saying they'd be dethroned. We're saying they're often "good enough" for many use cases, and that they're doing a good job of stopping the Big Guys from creating a giant expensive moat around their businesses. Chinese labs are acting as a disruption against Altman etcs attempt to create big tech monopolies, and that's why some of us cheer for them.
- Der_Einzige 8mo ago"Nobody says X" is as presumptuous and wrong (both metaphorically and literally) as "LLMs can't do X". It is one of the worst thought terminating cliches. Thousands have been saying this, you aren't paying attention.
- cmrdporcupine 8mo agoAs thought terminating as "HN Thought [insert strawman here]" C'mon.
- Aurornis 8mo agoI experimented with the Q2 and Q4 quants. First impression is that it's amazing we can run this locally, but it's definitely not at Sonnet 4.5 level at all. Even for my usual toy coding problems it would get simple things wrong and require some poking to get to it. A few times it got stuck in thinking loops and I had to cancel prompts. This was using the recommended settings from the unsloth repository. It's always possible that there are some bugs in early implementations that need to be fixed later, but so far I don't see any reason to believe this is actually a Sonnet 4.5 level model.
- Kostic 8mo agoI would not go below q8 if comparing to sonnet.
- anon373839 7mo agoYeah. Q2 in any model is just severely damaged, unfortunately. Wish it weren’t so.
- cubefox 8mo ago> I experimented with the Q2 and Q4 quants. Of course you get degraded performance with this.
- Aurornis 8mo agoObviously. That's why I led with that statement. Those are the quant thresholds where people with mid-high end hardware can run this locally at reasonable speed, though. In my experience Q2 is flakey, but Q4 isn't dramatically worse.
- deleted 8mo ago[deleted]
- cubefox 7mo ago> Obviously. That's why I led with that statement. Then why did you write this? > It's always possible that there are some bugs in early implementations that need to be fixed later, but so far I don't see any reason to believe this is actually a Sonnet 4.5 level model.
- cmrdporcupine 8mo agoIt feels more like Haiku level than Sonnet 4.5 from my playing with it.