8 ms·
You are not wrong, but the practical reality of local hardware is to be batch-constrained in comparison with a multi-user inference cloud. You will never be abl
by ttul 2mo ago
You are not wrong, but the practical reality of local hardware is to be batch-constrained in comparison with a multi-user inference cloud. You will never be able to compete cost effectively in your home lab with a cloud that has >100M end users streaming millions of inference requests per second across a gigantic fleet of machines.
Can I run a few inferences in parallel on my Mac Mini? Yes. But put 1,000 Mac Minis in a datacenter serving 1,000 copies of myself? That's going to be more efficient.
- anon373839 2mo agoOh certainly, a local machine can't parallelize like the cloud can. Does anyone think that, though? What it can do is provide good-enough inference to support common business needs, without creating unnecessary data risks.
- dickersnoodle 2mo agoWhy on earth would you need to do that for most things? Not everyone wants to be the next Coreweave. It sounds like you're killing a fly with a cannon to me.