zambelli

5 ms·

1.
by zambelli 4mo ago
v0.7.1 is now out - Forge can now sit behind Claude Code! Proxy mode can talk to supported backends and handles format translation. Anthropic > forge > OpenAI; or Anthropic > forge > Anthropic. Will get this ported over into vLL
2.
by zambelli 4mo ago
Thanks for the thoughtful comment! Let me try to unpack some of what's there and what's missing. Forge is at its core a mechanical reliability layer, whereas a lot of memory/skill management would be more of an orchestration
3.
by zambelli 4mo ago
You're very welcome! I've seen some good PRs come through and merges are starting. I need 24-48 hours to get the conference demo and travel sorted then work can continue at a faster pace! No intention of dropping forge, plenty mor
4.
by zambelli 4mo ago
Thanks to everyone for the great discussion! v0.7.0 is out now. It was in flight when this landed - changes tool error channel based on dogfooding observations with some larger models, eval re-run (numbers shift but within CI), and most imp
5.
by zambelli 4mo ago
This is a really neat writeup, and the empirical data for coding agents is super useful. Will take a closer read and see if there's anything I easily lift into my harness!
6.
by zambelli 4mo ago
Merged! Thanks for that catch. I'll try to sequence the in-flight work ASAP to get the vllm branch merged in as a whole.
7.
by zambelli 4mo ago
Oh that's a good find, I'll book ark this for a GitHub issue. Glad to hear it's working!
8.
by zambelli 4mo ago
Yeah I got it working as a quick test run to confirm a model issue vs backend issue on a consumer app. It worked on my dual-5070 Ti rig, but I didn't have time to formalize all the way and merge it in. Thanks for linking it!
9.
by zambelli 4mo ago
Nice symmetry with tool call failures being sent to LLM that made the call without bugging the user. The artifact-generating entity gets the error back, effectively. 100% correct, and stackable. Could have topic refusal in LLM training itse
10.
by zambelli 4mo ago
Oh, interesting - thanks for the link. I really haven't explored this but it should slot in fairly easily I think? Gotta dig into it more.
11.
by zambelli 4mo ago
Very cool! I'll try to get an issue open on lmstudio support and add it to the backlog.
12.
by zambelli 4mo ago
Interesting, catching the problem upstream, effectively. How did you enforce the grammar?
13.
by zambelli 4mo ago
Retry nudges do generate an extra LLM call, and those average extra calls time impacts are captured in the eval data. But that's the difference between the call failing and succeeding (eventually). On successful calls the presence of
14.
by zambelli 4mo ago
Oh, awesome! I'll take a look.
15.
by zambelli 4mo ago
Thank you! I've been trying to catch those replies and redirect people, but hopefully your comment be upvoted for others. Very embarrassing to put up the post with the wrong link lol.
16.
by zambelli 4mo ago
Ohhhh, that's much more interesting. I haven't looked into that at all, but now I'm curious. I'd need to think way more about how to layer that into forge, but the principle could likely be applied somewhere. I get it no
17.
by zambelli 4mo ago
This is not an agentic coding harness. It's a generic tool-calling guardrail stack. I have built a coding harness built on Forge since, but that's not what this is.
18.
by zambelli 4mo ago
Nice ;). I'll take a closer read of it, that's on me - I am definitely seeing more people looking in this direction as agents start to ramp in production at the enterprise level, which I suspect is highlighting some of these failu
19.
by zambelli 4mo ago
I know :( - I posted the wrong link and now it's there forever. Dashboard is in here: https://github.com/antoinezambelli/forge/tree/main/docs/resu...
20.
by zambelli 4mo ago
Not stupid at all! Some of the older models did do this (like 3.5-era ish I think), and the harness would parse the results. The newer way frontier has setup is structured tool calls. `tool_use` or `tool_calls`. The response is then recei
21.
by zambelli 4mo ago
Yeah I would think so! A lot of current tooling is layered mostly at the workflow level. Auth for the agent, or memory management for the agent (like some smart skills stuff), but Forge sits below that. In most cases I've looked at, it
22.
by zambelli 4mo ago
Thanks! No this was my own time, just evenings and weekends - life-permitting.
23.
by zambelli 4mo ago
Love this question! A few points: - First, there's totally a "risk" there. I built both the harnesses and the eval suite and that's hardly a double-blind study. There's no world where some bias doesn't leak thr
24.
by zambelli 4mo ago
Good luck! Frontier models are called frontier for a reason. I've seen Forge get local models close to frontier on these evals, even beat it in some cases, but frontier still has an edge overall - no denying it. The key I think is to l
25.
by zambelli 4mo ago
Definitely! A lot of tasks are within reach of small models, much more than people would think. Big models still shine in vague contexts or for breadth, or for very long running tasks, but yeah. The small ones just need help on longer multi
26.
by zambelli 4mo ago
Thanks! Did you try it with lmstudio? I actually never tried it with that. Only published ollama, llamfile, llama.cpp native/prompt - and unofficially tested vLLM, but never lmstudio.
27.
by zambelli 4mo ago
Yes I've now used it "in the wild" for a handful of use-cases. I still run into the backend thing even when declaring params though, which is odd to me. But there might be params not typically passed in with the model that ba
28.
by zambelli 4mo ago
Interesting - so you're thinking give the model two parallel shots at the tool call and take the winner if there is one, or fallback to retry if not? That would certainly work in theory, but I'm not as familiar with parallel calls
29.
by zambelli 4mo ago
Definitely, there's several failure modes and Forge doesn't address all of them. This is just one tool in the toolbox to getting things stable enough for production use at reduced costs. Forge sits one level lower - in my mind - t
30.
by zambelli 4mo ago
Hi! Latency is definitely a factor in any system, and the dashboard and paper do report elapsed time - but at the workflow level. On a per-call basis, the wrappers are pure python ifs and such, measured in ms easily, and frankly negligible

More ›