Thanks for the insight here. One thing to note about the first graph: nobody is doing as well as the first part on tool calling, and it's not close. This migh

Thanks for the insight here. One thing to note about the first graph: nobody is doing as well as the first part on tool calling, and it's not close. This might be the fault of the other providers, but it's probably just something slightly different that the first party does with the model inference program than anybody else, and that's not sure to weights it's due to vLLM twiddling (or whatever) and probably becuase the first part actually uses their own customized inference program rather than the standard methods that all the third party providers use. This isn't nefarious, it's just the challenge of these sorts of stochastic systems. Having been in science for decades now, and seen benchmarking across many different fields, these results are completely expected for me. LLM serving is not mechanical, it's hard to get right and has lots of unknown footguns. Even something as extreme as scrambling a matrix will still likely get results that are nearly as good as normal, and if there's a bug deep in vLLM or the tensors metadata that results in that, then it's going to be pretty hard to find unless you're an active researcher with knowledge of the particular model you're running inference on. I kind of doubt that's happening here, but maybe! In the scientific literature, when benchmarking methods, everybody's own method performs best in their own hands. Some attribute it to researchers gaming benchmarking for publication purposes, but I think it's just what we see here: the people who made a method are just the best at using it because they know all the quirks and use it best. Programmers are not used to thinking with that nuance, and jump to conclusions about lying about quantizations, etc., but this is really just an unavoidable part of AI/ML methods: when things aren't perfect they're still pretty good and it's going to take the model creator to truly debug it. At least until the open weights ecosystem gets a lot better at ensuring reproducibility, and model cards are nowhere detailed enough for that to happen yet.