5 ms·
This likely comes down to how it accessed the bus schedules (i.e. web search tool) and not intelligence. You need to add the actual bus schedule to context som
by deno 2mo ago
This likely comes down to how it accessed the bus schedules (i.e. web search tool) and not intelligence.
You need to add the actual bus schedule to context somehow (research agent, custom tool or just dump in prompt) and even the simpler modern models will be able to do the planning.
- solenoid0937 2mo agoTool usage competency is part of overall intelligence. If the model can't get the information it needs, it must clarify that in the response.
- deno 2mo agoThis isn't tool usage competency, it's tool quality and/or luck. Regular web search is not good for grounding if you want accurate results. You can ask the model to make a tool for getting bus schedules and then use it only then you are comparing apples to apples in this case.
- solenoid0937 2mo agoIf the model can't get the information it needs to accurately answer the question, it must surface that risk to you instead of guessing. This is part of model intelligence and tool use competency. Fable and to a lesser extent Opus is very good at this
- deno 2mo agoThis would be hallucination rate and neither of those models excels in this area and in fact Grok does, or at least prior version.
- nijave 2mo agoIt's both. Some models, given web search and web fetch may only run a search and assume the summary text is correct and blindly return it. Others will validate by running a web fetch and checking the whole page contents. Even better, the model will run multiple searches and fetches to cross check the information. The best models will attempt to verify whether a source is authoritative or not and try to only return authoritative results. I have an example here: https://gist.github.com/nijave/2873b8b10d8c732e46264237b075507a https://gist.github.com/nijave/2873b8b10d8c732e46264237b0755... Tldr; all the Claude models had identical tools and some used them efficiently and verified data while others did a crap job and hallucinated responses. Additionally, Exa MCP tools generally worked better even on older/smaller model (Llama)
- deno 2mo agoUnless I'm missing something I think to a large degree you're just comparing system prompts. If I add "Research the question extensively" to your prompt at the end I get the correct answer from Haiku and Sonnet Med on first try and I've reproduced the original prompt not returning the answer. Unfortunately every other run now gets your gist in results.