6 ms·
What about request caching? If you swap to a cheaper model mid execution it might cost more that to make multiple requests to the already cached provider?
by k9294 3mo ago
What about request caching? If you swap to a cheaper model mid execution it might cost more that to make multiple requests to the already cached provider?
- adchurch 3mo agoYep 100%, mentioned this in another thread (https://news.ycombinator.com/item?id=48689448 https://news.ycombinator.com/item?id=48689448) but tl;dr we build the router to be cache aware