5 ms·
I've been saying for a while that given a proper harness, small local models can perform incredibly well. When you have a system that can try everything, it wil
by Escapade5160 4mo ago
I've been saying for a while that given a proper harness, small local models can perform incredibly well. When you have a system that can try everything, it will eventually get it right as long as you can prevent it from getting it wrong in the meantime.
- koolba 4mo agoA thousand monkeys on a thousand typewriters…
- deleted 4mo ago[deleted]
- zambelli 4mo agoThat is the whole challenge, actually! A new metric I'm going to dogfood into forge is ETTWS - estimated time to working solution. A simple retry loop around your whole workflow could, in some cases, be all you need. But it could mean many blind attempts to get through a workflow successfully. And hopefully there isn't a payment step partway through! The fewer hard errors nix the whole workflow, the lower your ETTWS.
- beacon294 4mo agoHave you read the MAKER/MDAP paper? 1 million sequential tasks.
- zambelli 4mo agoNo, I haven't - hadn't heard of it. I'll try to squeeze in a quick read in the coming weeks!
- killing_time 4mo agoIs it strange that I immediately interpreted ETTWS to be Estimated Time To William Shakespeare?
- DiogenesKynikos 4mo agoThis is a thousand unusually smart monkeys who speak every major human language fluently and are proficient in every major programming language, but sometimes still make bizarre mistakes and need to be put back on track.
- jplusequalt 4mo agoThis is fun for you?
- bratbag 4mo agoI found it fun to read.
- zambelli 4mo agoLol, I love that framing. Yeah, the small models have impressed me a lot during this work. The reasoning can be quite good, and definitely sufficient for a lot of cases. Just gotta nudge em back on track Every now and then and they'll figure it out.
- cornholio 4mo agoIf I understood correctly, the model will get it right because it knows when it isn't right.
- zambelli 4mo agoEssentially, yes that's right! There's some subtlety in how to let it know it was wrong (returning things as tool errors because it trained on that), but that's the gist of it - sort of a self-correcting architecture.
- tomjakubowski 4mo agohttps://en.wikipedia.org/wiki/Apophatic_theology https://en.wikipedia.org/wiki/Apophatic_theology
- jon_richards 4mo agoI was expecting this https://knowyourmeme.com/memes/the-missile-knows-where-it-is https://knowyourmeme.com/memes/the-missile-knows-where-it-is
- forlorn_mammoth 4mo agothe missile knows where it is because it knows where it isn't
- deleted 4mo ago[deleted]
- Aurornis 4mo agoThe problem is that you get similar quality as if you gave a junior unlimited time to work on a problem and told them to keep trying different things until the goal is reached. Even the SOTA models have this problem when the work is complicated enough. The problem is amplified more with the small models.
- Zetaphor 4mo agoThere's a lot of valuable things that can be done in that range, especially when token costs aren't a concern. Not every problem requires SOTA
- Aurornis 4mo ago> especially when token costs aren't a concern. Not every problem requires SOTA If token costs aren’t a concern I’m using SOTA for everything. Even SOTA gets it wrong and hallucinates, but at a lower rate. I don’t want to waste my time.
- coip 4mo agoOne important facet of this is it’s not far from “giving unlimited juniors unlimited time…” Where the limits are set by hardware for agentic execution (compute/network/storage) && inference speed
- andai 4mo agoPrior art: https://ghuntley.com/ralph/ https://ghuntley.com/ralph/