6 ms·
To me, the amazing thing is that we now have local models that rival the reasoning of high end models from about a year ago. I hope this trend continues.
by RachelF 1mo ago
To me, the amazing thing is that we now have local models that rival the reasoning of high end models from about a year ago.
I hope this trend continues.
- refactor_master 1mo agoUnlike cloud infra in general which offers things like automatic backups, regional redundancy, and effectively unlimited scalability, it seems like the value proposition of cloud LLM gets ever shakier. * Many businesses don't need frontier level intelligence anyway. * It's completely stateless. If your local LLM machine catches fire? Nothing was lost. Buy another.
- SV_BubbleTime 1mo ago[dead]
- NhanH 1mo agoThe whole cloud story lies on two aspects: - Hyperscaling “we are going to serve billions of people in our applications”, which is becoming increasing unlikely as regional tech companies become more dominant than than the global one (this one is as much about geopolitics as technology) - Operations is hard, in which case non-frontier models should be increasingly capable. Devops for small-ish deployment is one of the few cases where it is hard to clam you need deep expertise and AI can’t do it. Previously, the claim is that you need people specialized in ops, which is expensive. Now… My prediction is that not just cloud LLM, but cloud business general will have to change. Not yet in the next 5 years, but probably 8-20 years-ish
- redrove 1mo agoI don’t disagree but saying “cloud will change in the next 8-20y-ish” is a bit of a non-argument, you’re not really stating any thesis to speak of; Change is a given over that time frame.
- NhanH 1mo agoAh it was not explicit enough. They will die, for some definitions of death -- I don't think they disappear, but they should be a niche, rather than the dominant doctrine.
- redrove 1mo agoI’m not so sure I agree, given the overwhelming concentration of capital and regulatory capture they have, I just don’t see them going anywhere; becoming more niche rather than even more of a standard is “going away” to a certain extent as far as I can see.
- ipdashc 1mo agoI mean, I might be missing something, but isn't part of the idea with cloud infra that you can scale down as well? Large orgs with significant demand might go out and buy local LLM hardware, but most businesses probably don't want to bother dropping $2k on a box with a beefy GPU and would rather just pay the lowest subscription tier so their employees can occasionally make queries. Plus, you know, the whole economies of scale thing. Local LLM has a lot of privacy and independence benefits, but I'm not really seeing the world where it becomes more energy- or cost-efficient to buy your own hardware (and use it 1% of the time) versus sharing a giant machine, or even the same machine, in a datacenter (where it has a much higher utilization factor).
- mattmaroon 1mo agoAll white collar work will be done by AI soon, there won’t be scaling down, just scaling up. I do food trucks and festivals and I’ve got AI doing so much of my non-meatspace work now that I’m buying $10-$20 a day in tokens. At that rate, hardware starts to look cheap, and I have less of a use case for it than most white collar workers.
- ipdashc 1mo agoGenuine question, what are you spending that on? The $20/month ChatGPT/Codex subscription has largely been enough for me as an IT worker.
- mattmaroon 1mo agoThere. I have the business plan with two seats and I use them both and blow through it pretty fast. I think it’s because much of what I have it do involves using a browser. For instance I have it pull various permits from cities and there’s no API for that.
- qeternity 1mo agoComputer use will blow through tokens because it's doing image capture for everything. You may have better and more reproducible results using browser controls that aren't image based, or writing tools that completely sidestep browser use.
- fweimer 1mo agoCentralized inference can easily increase batch size, leading to huge efficiency gains in the usual scenario where most users have just one or very few session. Using local resources efficiently requires some way to increase the batch size. I'm not sure if we are there yet.
- janalsncm 1mo agoI think the point is that if people are able to run inference on their laptops batch size efficiency won’t matter. And before that, businesses will be able to get decent results with dedicated inference hardware.
- qeternity 1mo agoJevons paradox: large purpose-fit data centers increase efficiency such that you can use AI in more places, and use more tokens for those tasks. The future is not a single chat bot session of bs=1. The future is many agents performing many tasks in parallel for a single user. Large GPU clusters will always have the edge in efficiency.
- pessimizer 1mo ago> The future is many agents performing many tasks in parallel for a single user. Large GPU clusters will always have the edge in efficiency. Agentic AI has pluses and minuses for cloud efficiency. The plus is that usage could be very bursty, but the minus is that agents will more fully utilize a local system. The main disadvantage of local AI is that you would be paying a large amount for a system mostly doing nothing. If it's constantly working on different projects and integrating that data, you get use out of every penny that you spent. Every GPU you added would instantly make the thing smarter. What's more, your local AI could offload an agent to the cloud if it needed to. It could do this rationally, based on your personal desire for privacy.
- janalsncm 1mo agoJevon’s paradox suggests that total datacenter resource consumption will increase. It doesn’t say that people will choose datacenters over their own personal hardware when the latter is sufficient.
- manmal 1mo agoYeah. Local agent sessions are not backed up in the cloud. And uptime is better with local models.