jarodrh

43 ms·

1.
by jarodrh 2mo ago
True, but human review isn't visible in the logs, hmm, however I suppose retry patterns can, but frugon doesn't currently track this, so we can't quantify this yet. Frugon analyzes your existing logs offline on cost and quali
2.
by jarodrh 2mo ago
> As well. Someday costed opcodes and sharded redundancy for all of this; Hmm, noted. > If an algorithm selects according to rank, there's a cost/score/fitness/survival/error function. - Yes, what I'm say
3.
by jarodrh 2mo ago
Down the rabbit hole we go. > Is this a convex optimization or non-convex optimization problem? - Neither, frugon sidesteps that issue by having a small finite candidate selection that it evaluates completely. No search needed because th
4.
by jarodrh 2mo ago
Very interesting thoughts. This is a whole discussion on its own. The ones that I mulled over that I feel have legs in my line of thinking are: > minimize cost to converge upon sufficiently low error problem - This is the one. Frugon doe
5.
by jarodrh 2mo ago
Yes they do, it's quite the popular topic atm. RouteLLM (OSS - frugon's savings bands are based on their research), OpenRouter's auto mode; I actually commented on a /show post not long ago: Wayfinder Router (neat proje
6.
by jarodrh 2mo ago
Thanks. Yes, local models are gaining a lot of traction. The measure/judge step uses LiteLLM, so it does sample local models. I just tested "--candidates ollama/llama3.2:1b", and that works - ignoring the lack of rich UX
7.
by jarodrh 2mo ago
Interesting project. Most of what I read appears to deal well with a linear agent session. How does it handle concurrent agent/subagent spawning? Do they fall under the same sessionId (where they're potentially fighting over the w
8.
by jarodrh 2mo ago
Thanks. I'm not sure if you're asking "how does frugon compare provider models" or "can it compare across provider models" or even "how does frugon compare to other tools"... On the first: It does thi
9.
by jarodrh 2mo ago
You got cut off at the end, but I hear your point. I completely agree that cost-per-successful-outcome is the goal, and "cheapest" is never the entire solution, but only routing to the cheapest isn't what frugon does. If it d
10.
by jarodrh 2mo ago
Thanks. Yes, that's exactly right, and aptly named so in the code. A few things to note: the prompt is actually deliberately tie-biased; the tie only then breaks on clear material differences: factual error, missing information the pro
11.
by jarodrh 2mo ago
That's an interesting thought, and one I'll take note of, but that would be a different tool. However, if you look hard enough I'd say we're tackling the same issue. I'm just choosing to look at the problem from a c
12.
by jarodrh 2mo ago
Thanks! I love a good meaningful name Frugal + on :P
13.
by jarodrh 2mo ago
We've all been there where we choose the easy least efficient route just cuz (laziness...can afford it), but we've mostly done this consciously. The example in the article however does take this to another level and understandably
14.
by jarodrh 2mo ago
Hmm, I honestly think a harness should be more about workflow, control, ease of use, memory optimization...other things I can't think of right now, the model/subscription being the least of them. Most of my experience has come fro
15.
Show HN: Frugon – Find which LLM calls a cheaper model could handle (local, MIT)(github.com)
67 points by jarodrh 2mo ago|24 comments
16.
by jarodrh 2mo ago
The article's own example makes the point I'm about to make: > To give an example, just doing Typescript type fixes with this model across 50 files cost me $54 this afternoon. That's all because it ran through the most exp
17.
by jarodrh 2mo ago
reactordev pretty much got it right. In the config, you say "this role uses this model". You decide up front which model handles which kind of task, instead of one model doing everything. All config driven. You're not choosin
18.
by jarodrh 3mo ago
Orchestration works very well for me, but not in the way most people seem to be pushing for, with middlemen scoring and routing every request. For coding, the routing is mostly solved at the config level. The harness lets you pin models per
19.
by jarodrh 3mo ago
Interesting. Read through the comments and the cache issues and context loss issues were already mentioned and they are all good valid points. I'll go from the positive side. I think the part that resonates most with me is the determin