6 ms·
There are none, this isn't a problem specific to OR as much as it is a problem with serving LLMs in general. If you use any other meta-provider that routes you
by bakugo 5d ago
There are none, this isn't a problem specific to OR as much as it is a problem with serving LLMs in general.
If you use any other meta-provider that routes your requests to third party providers, you'll likely face the same issues. If you try using any of those providers directly, you'll likely face some of the same issues as well, except you won't have the option of quickly swapping to a different one and taking your credits with you.
Extreme variance in quality and feature support per provider is probably the biggest obstacle holding back adoption of open weights models.