7 ms·
Their published benchmarks seem to indicate that it's pretty good at coding and multimodal, but VERY good at successful tool calls. What kind of use case would
by EgregiousCube 2mo ago
Their published benchmarks seem to indicate that it's pretty good at coding and multimodal, but VERY good at successful tool calls.
What kind of use case would be best for that shape?
- alansaber 2mo agoThis sounds... kind of useless? Really good JSON or similar constrained decoder performance is interesting, but normal decoder > tool validator loop with good error message > tool retry is almost always able to get a tool to work second try, and input is cached so it's not expensive.
- aldanor 2mo agoThings are not always that simple, eg https://lucumr.pocoo.org/2026/7/4/better-models-worse-tools/ https://lucumr.pocoo.org/2026/7/4/better-models-worse-tools/
- winstonp 2mo agoThe avg coding session has hundreds or thousands of tool calls. Even a 5% failure rate noticeably notches up token use and cost. See Gemini.
- alansaber 2mo agoYes, but each tool call has a different failure %. The tool calls that make up the majority of volume like grep are going to have nowhere near a 5% failure. A custom user-defined skill having a 5% failure rate is probably fine.
- xnorswap 2mo agoDebugging and diagnosis is very tool call heavy, whether that's grepping / transforming logs, calling out to profilers/tracers, or even just writing up incident reports. Bug diagnostics is about being okay at coding but better at tooling. Given a good diagnostic report, it can be handed to opus for the fix. Opus is okay at writing reports, but it still regularly gets table widths wrong in typst documents, leaving the last column full of text but only a handful of characters wide.
- paytonjjones 2mo agoI wonder if we'll start to see that pattern with every new release. Tool use likely changes rapidly, so the newest, rather than most intelligent, model may always have an edge.
- ai_fry_ur_brain 2mo agoWhat you mean.. The tools are all just invoking bash and terminal/cli cmds and http requests. Paradigms that have existed and stayed mostly unchanged for decades.
- paytonjjones 2mo agoThese do make up a huge % of tool calls, but I don't think these make up a huge % of tool call failures. I see models fail on tool calls that involve API requests to a specific API, internal or cloned Makefile calls, npm run commands, etc.
- ai_fry_ur_brain 2mo agoGemini 3.5 flash is better than fable at tool calling. Tool calling is probably one of the easier things to do post training for.
- ralusek 2mo agoI don't use Gemini cli because they're so bad at agentic work/tool calling. I use their chatbot all the time, though.