I ran some tests last quarter with a similar multi-model routing setup focused on code completion tasks. Using an open-source Claude alternative combined with C

I ran some tests last quarter with a similar multi-model routing setup focused on code completion tasks. Using an open-source Claude alternative combined with Claude-instruct for fallback, I saw about a 45% cost reduction compared to consistently hitting the Claude Code API alone, with latency penalties under 300ms for the fallback requests. The key was tuning the confidence threshold so that around 70% of calls got handled by the cheaper local models without sacrificing accuracy much. One interesting data point was that for smaller completions (~50 tokens or less), open-source models like OpenHands outperformed cloud-hosted Claude alternatives in terms of 95th percentile latency—likely because no network involved. But for bigger, multi-file refactoring queries, fallback to frontier models was unavoidable to hit the 90+% pass rate. The dynamic routing balanced those two extremes to keep API spend down while still scaling. That said, the caching layer remains a bottleneck. Changes in the input context or tool states quickly cold the cache, forcing more queries to the expensive model. For workflows with lots of sequential tool calls (e.g., automated code lint + formatting + test generation), the raw switch cost was around 15-20% overhead compared to fixed-model baselines until cache warm-up happened. This matches the experiences others have reported.