17 ms·
Apple M series chips deserve a mention as another option, especially since you get a whole Mac laptop or desktop workstation too. They have unified memory and
by api 3mo ago
Apple M series chips deserve a mention as another option, especially since you get a whole Mac laptop or desktop workstation too.
They have unified memory and respectable inference performance, and for some variations can be cheaper than video cards, especially if you get an older-gen high-end M series with a lot of RAM used or refurbished.
I've read that Apple has plans once the RAM bottleneck passes to offer more RAM in all their models, and that future M series GPUs and NPUs will be even better for local inference, so in the future I expect Apple to be a serious offering for local inference and AI research workstations.
And what about AMD and Intel Arc GPUs? They don't get as much love but I've heard they can be compelling for certain shapes of a local LLM configuration.
At this point though, I think we may be in a "renters market" for LLM compute. If you want privacy it might be better to rent GPU time in raw form or use spot pricing at various providers. It probably only makes sense to build if you have extreme privacy/security needs or just want to do it cause it's cool.
- mwcampbell 3mo ago> once the RAM bottleneck passes Do we have evidence that this will actually happen? Maybe the belief that it won't pass is what requires evidence, but I think there's a widespread feeling right now that things are just getting permanently worse and this is one example.
- justincormack 3mo agoMicron have sold RAM for the next 4 years at current prices, so there are buyers expecting this to stay the same.
- api 3mo agoThat means buyers have basically purchased options. If the price falls, they're underwater a little, but if the price spikes it protects them. People do that all the time, and sometimes it doesn't pay off.
- api 3mo agoIt'll probably take a few years. There's many fabs under construction. One thing holding back capacity expansion is that a lot of people are concerned this is a bubble. They're worried it'll pop and leave them with orphaned assets if they over-invest in production. Of course maybe they're right and that will happen. If the data center construction boom ends, RAM prices will fall.
- maxxxml 3mo agoMLX is super underrated right now, tons of performance unlocked as of recent. Love to see it!