5 ms·

Hey! Thanks for the feedback! I work on many of these things at Databricks, so figured I'd chime in on this. Firstly, while routing is important, simple things

Hey! Thanks for the feedback! I work on many of these things at Databricks, so figured I'd chime in on this. Firstly, while routing is important, simple things like observability into the token costs of various features, which can drive optimization of better default parameters, are low hanging fruit everyone should do. 1. As for the cache busting, we're very aware of this. Had said this in a thread above too, so copying it here. We're going to do a followup blog detailing our routing approach, but we're designing it to be cache aware. The router takes in the task description and infers what models and harnesses are available and makes a recommendation up-front. So essentially the routing decision is made when the harness + model is kicked off and it's only changed halfway through if there's a major delta in complexity from the initial judgment. Therefore, most of the time the cache is maintained just as it would be before. This does have disadvantages too, but the cache is the dominating cost reduction force. 2. We do love Pi (we published this too: https://www.databricks.com/blog/benchmarking-coding-agents-databricks-multi-million-line-codebase https://www.databricks.com/blog/benchmarking-coding-agents-d...), but we didn't mention it here because we're still in the process of making it available to devs internally. There's a lot that goes into this more than just cost (e.g. feature gaps vs. other common harnesses). 3. As for existing utilization dashboards, it's good that each tool has one, but we found that as we gave developers more freedom to use more tools, it was impossible to have a single source of truth. Without that, it's hard to really understand all-in AI spend and optimize it.

Terretta 1mo ago
> Firstly, while routing is important, simple things like observability STRONG agree. Most devs, if they can see their costs, try. Even just putting it in a starship prompt goes a long way! Observability supports effective, responsible, and accountable use, "cost controls" (as typically enterprise defined) supports bean counters. Ensuring Observability is recognized as a valid Control™ should be on every engineering executive's TODO list. > the cache is the dominating cost reduction force TY for the rich reply. As you're very aware of the cache issue, it might have been nice to have the word "cache" more prominent on your "Key Techniques" image (currently Lower Cost Models, Smart Routing, Spend Controls, Context Optimization; cache tuning buried at the end) or higher in the body sections. It was mostly the image that prompted (ha) me to call out the marketing driven angle.

Terretta 1mo ago

> Firstly, while routing is important, simple things like observability STRONG agree. Most devs, if they can see their costs, try. Even just putting it in a starship prompt goes a long way! Observability supports effective, responsible, and accountable use, "cost controls" (as typically enterprise defined) supports bean counters. Ensuring Observability is recognized as a valid Control™ should be on every engineering executive's TODO list. > the cache is the dominating cost reduction force TY for the rich reply. As you're very aware of the cache issue, it might have been nice to have the word "cache" more prominent on your "Key Techniques" image (currently Lower Cost Models, Smart Routing, Spend Controls, Context Optimization; cache tuning buried at the end) or higher in the body sections. It was mostly the image that prompted (ha) me to call out the marketing driven angle.