6 ms·
> Edit: I'm thinking of a headless Mac mini, if you meant running it on the same machine you're using of course you'll need more memory, but LLMs are best serve
by qeternity 16d ago
> Edit: I'm thinking of a headless Mac mini, if you meant running it on the same machine you're using of course you'll need more memory, but LLMs are best served from a headless server so that's what I'd recommend.
What? LLMs are best served from a massive PD disaggregated cluster of B300s connected via NVLink.
If you're running LLMs on a Mac Mini, it's because you want to run local, not because it's the best setup.
- redox99 15d ago>massive PD disaggregated cluster of B300s connected via NVLink. So a headless server. Macs were mentioned because that's what the post is about. It could be a PC (I use a 2x3090 PC). The point is that it's a better experience to have a box dedicated to the LLM than running it in your system. Obviously in your home, so local.
- qeternity 14d agoHas nothing to do with being headless. That's just a natural outcome. If someone wanted to use an 8x B300 as their daily driver...go ahead. It would still be the best way to serve a given model.