10 ms·
provider fallbacks are also expensive since you lost the prompt cache. openrouter have a 'session' concept to help with this, but we ended up going direct to pr
by probabletrain 6d ago
provider fallbacks are also expensive since you lost the prompt cache. openrouter have a 'session' concept to help with this, but we ended up going direct to providers for higher rate limits and writing our own fallback logic.