6 ms·
When people talk about the commodification of inferencing, they imagine a future where everyone has access to frontier models and can run them at the same cost,
by empath75 24d ago
When people talk about the commodification of inferencing, they imagine a future where everyone has access to frontier models and can run them at the same cost, and what will actually happen is closer to the commodification of _oil_, where only a few companies have the scale to produce it at a competitive price, and advances like this are _why_.
Once models are more or less interchangeable, the price of LLMs will drop to essentially the price of energy required to run them, and the big labs will be able to run them cheaper than anyone else.
- simianwords 24d agoI don't believe models will be commodified because each model is unique with strengths and weaknesses. Its not like Steel which is more or less the same no matter where you purchase it from. If what you said were true, you would hardly see people complaining about the quality of Opus 5 or good writing from Sol. But people do.
- airspresso 24d agoThis depends heavily on what the use-case is. Yes, if it's a coder making software and having to read LLM output then writing style matters. If the LLM is used in an automated data processing pipeline with a capped level of complexity, entirely different aspects matter and LLMs become more interchangeable.
- lelanthran 24d ago> I don't believe models will be commodified because each model is unique with strengths and weaknesses. They are all converging.
- zurfer 24d agoThe same level of intelligence gets roughly 10x cheaper per year. So you might both be correct where a large part are commodity tasks but frontier is hard and valuable and not commodities.
- skhameneh 24d ago> Its not like Steel which is more or less the same no matter where you purchase it from. I’m not an expert in metallurgy by any means, but this seems really off. There are many recipes for steel and varied processes that also impact the final product.
- imtringued 24d agoSteel has different varieties like carbon steel, alloy steel, stainless steel, and tool steel. Each of those categories then has different grades of quality. Tokens are a lot more like steel than oil, especially since a lot of tokens are used as structural material in the form of code.
- impossiblefork 24d agoI don't agree. At the moment companies like NVIDIA take several times what it costs to make a chip. I think the fair split for the technology contribution is more like 50-50, maybe even 30-70 in favour of the manufacturer. With competition we will actually have the fair split, whatever that is, and thus much lower prices. At the moment, to have a big AI firm, or really AI firm at all, you need to be blessed by NVIDIA, in the form of receiving circular financing for your compute. They know that their prices aren't fair, or competitive. Commoditization of inference is the end of that. The end of the mega-premium on inference hardware, and it's good not only for people who like running their LLMs, but it's the first step towards commoditization of training.
- kubb 24d ago10-90 is the fair split.
- impossiblefork 24d agoOne could hope so, or it would be best for me if it were. I don't think I can count on that though. I don't think I can hope for anything better than 30-70 in my lifetime, and I think 10-90 almost requires you to buy the design firm.