5 ms·
One of the big things to think about is whether local LLMs will be things companies want to deploy. If you think of for e.g. some proprietary piece of software
by physicsguy 27d ago
One of the big things to think about is whether local LLMs will be things companies want to deploy.
If you think of for e.g. some proprietary piece of software that wants to embed an LLM they've fine tuned or trained, they will want to make back some of their research cost right. So they are not going to want to put this on-device even if the hardware is there, unless there's some way of locking it down. I suspect we'll need on-hardware validation/verification and a way of preventing extraction of weights for this move to happen for many use cases.
- esseph 27d ago> One of the big things to think about is whether local LLMs will be things companies want to deploy. Non-tech enterprise was already doing this years ago. Regulatory reasons, privacy reasons, security, etc. They want on-prem and total ownership of the data. Sometimes air-gapped.
- roryirvine 27d agoHonestly, I've seen a lot less of this than I'd have expected. Lots of discussion, plenty of experiments or toy implementations at the level of individual teams (often driven by one or two enthusiastic individuals), a few proofs of concept for internal services at a more strategic level, but pretty much nothing beyond that. This is in the UK where there's currently a big focus on data sovereignty in general, and I'm genuinely surprised by how little that's spilled over into demand for inference sovereignty (so far). I still expect demand to grow substantially, but I've been saying that for the past couple of years and am beginning to wonder if there'll need to be some sort of trigger event before it happens (eg. the datacentre bubble bursting, or some sort of major scandal).