4 ms·
It's live on openrouter now. In my personal benchmark it's bad. So far the benchmark has been a really good indicator of instruction following and agentic beha
by pcwelder 7mo ago
It's live on openrouter now.
In my personal benchmark it's bad. So far the benchmark has been a really good indicator of instruction following and agentic behaviour in general.
To those who are curious, the benchmark is just the ability of model to follow a custom tool calling format. I ask it to using coding tasks using chat.md [1] + mcps. And so far it's just not able to follow it at all.
[1] https://github.com/rusiaaman/chat.md https://github.com/rusiaaman/chat.md
- manofmanysmiles 7mo agoI love the idea of chat.md. I'm developing a personal text editor with vim keybindings and paused work because I couldn't think of a good interface that felt right. This could be it. I think I'll update my editor to do something like this but with intelligent "collapsing" of extra text to reduce visual noise.
- pcwelder 7mo agoCool! Please share your work if possible! I couldn't decide on folding and reducing noise so I'm stuck on that front. I believe there is some elegant solution that I'm missing, hope to see your take.
- nolist_policy 7mo agoCould also be the provider that is bad. Happens way too often on OpenRouter.
- pcwelder 7mo agoI had added z-ai in allow list explicitly and verified that it's the one being used.
- sergiotapia 7mo agoBe careful with openrouter. They routinely host quantized versions of models via their listed providers and the models just suck because of that. Use the original providers only.
- nullbyte 7mo agoI specifically do not use the CN/SG based original provider simply because I don't want my personal data traveling across the pacific. I try to only stay on US providers. Openrouter shows you what the quantization of each provider is, so you can choose a domestic one that's FP8 if you want
- sschueller 7mo agoFunny, living in Europe, I prefer using EU and Chinese hosts because as I don't want my data going to the US. The trust in US firms and state is completely gone.
- lostmsu 7mo agoTangent note: this sounds like the same mistake as EU's reliance on Russia.
- lossolo 7mo agoNot really. China doesn't share a border with us, doesn't claim any EU territory, and didn't historically rule our lands the way the USSR did. In the context of spheres of influence and security interests, its strategic goals aren't directly at odds with the EU's core interests.
- lostmsu 7mo agoEU is not a singular country, and Germany or France don't border Russia either. Considering China is ok to supply Russia, I don't see how your second point has any standing either.
- lossolo 7mo ago> EU is not a singular country, and Germany or France don't border Russia either. But soon they could, that's the problem. > Considering China is ok to supply Russia, I don't see how your second point has any standing either. Supply? China supplies Ukraine too. Ukraine's drone sector runs heavily on Chinese supply chains. And if China really wanted to supply Russia, the war would likely be over by now, Russia would have taken all of Ukraine.
- data-ottawa 7mo agoCustom tool calling formats are iffy in my experience. The models are all reinforcement learned to follow specific ones, so it’s always a battle and feels to me like using the tool wrong. Have you had good results with the other frontier models?
- thegeomaster 7mo agoNot the parent commenter, but in my testing, all recent Claudes (4.5 onward) and the Gemini 3 series have been pretty much flawless in custom tool call formats.
- data-ottawa 7mo agoThanks. I’ve tested local models from Qwen, GLM, and Devstral families.
- pcwelder 7mo agoAll anthropic models. Gemini 2.5 pro and above. Gemini 3 flash is very good too. GPT models can follow tool format correctly but don't keep on going. Grok-4+ are decent but with issues in longer chats. Kimi 2.5 has issues with it reverting to its RL tool format.