6 ms·
What is the basic logic by which you route?
by BrunkenClaas 2mo ago
What is the basic logic by which you route?
- Aperswal 2mo agoThe basic logic for default routing or flex routing?
- BrunkenClaas 2mo agoThe flex routing you offer.
- Aperswal 2mo agoIf you give an SLA of lets say 10 seconds to find cheaper inference. I look around different providers that offer flex/batch and try to fulfill in that time. If I can't then I escalate to a default request. This means you definitely get a response in 10 seconds (could still be a 5xx or 429 from default), but in those 10 seconds I might find you cheaper inference and drop costs by 50%, then charge 20% of that as FlexInference's cut, effectively giving you a 40% discount. If I don't find the cheaper inference and have to escalate, then I don't charge at all, so it is free.
- zach7 2mo ago[dead]