5 ms·
True. But there are other meanings of "free". I.e. nobody can say "from now on you no longer have access to model X because you're an asshole"
by cicko 3mo ago
True. But there are other meanings of "free". I.e. nobody can say "from now on you no longer have access to model X because you're an asshole"
- trollbridge 3mo agoSome obvious examples of why you'd want to spend the capital on this would be, for example, making some kind of autonomous system which needs to be periodically be offline, or you need complete confidentiality of what you're using the model for, etc. To be cost effective with inference providers, you have to find some way to be using it 24/7.
- Der_Einzige 3mo agoThe ecosystem for inference is centralized around a few core projects, i.e. vLLM, sglang, and llamacpp. If they decided to collude, they could absolutely say "from now on you no longer have access to model X because you're an asshole" The commercial inference offering are also downstream of one of those 3 projects (or trt-LLM if they're nvidia). It would impact Ollama, and fireworks, together, and everyone else. Don't tempt fate.