6 ms·
> That being said, such a split design may help with improved AI inferencing. Unified memory is the only reason Macs are so coveted right now for local AI. A
by vimy 2y ago
> That being said, such a split design may help with improved AI inferencing.
Unified memory is the only reason Macs are so coveted right now for local AI.
A single 192 gb ram Mac costs less than the equivalent in standalone GPUs.
- Tostino 2y agoI'm sorry, I think you mispronounced market segmentation... This is Apple we're talking about.
- novok 2y agoThe execution speed for LLM inference gets so slow once you reach models that even fill out a 64GB machine. I was tempted, too, but then realized it was unusable past 48GB-ish and stacking used 3090s was the best price / perf / vram ratio. What are the good use cases for very large memory amounts?
- behnamoh 2y agoI ran all my research papers (about LLMs) on a Mac Studio.
- reitzensteinm 2y agoMixture of Expert models, where all parameters must be in memory but only a subset are accessed per token, are a sweet spot for Macs. DeepSeek v3 for instance has 671B params, but should have the memory bandwidth of a 37B dense model with a batch size of one.
- rpastuszak 2y agoAny model you'd recommend for a Mac with 48 GB RAM?
- reitzensteinm 2y ago48GB is maybe just enough to squeeze a quantized 70B model in, like Llama 3.3, but you'll need to raise the GPU memory allocation limit [1] and it might not be super fast. You could also try Qwen 2.5 32b, which you should just work with ollama or LM Studio with no config changes. I've got a 32gb M1 Max and a 24gb 4090, and I barely ever run models on my Mac, as the memory bandwidth and compute for prefill is much better on the 4090. But I'm essentially locked out of Llama 3 70B class models, which I only use via API. [1] See: https://www.reddit.com/r/LocalLLaMA/comments/186phti/m1m2m3_increase_vram_allocation_with_sudo_sysctl/ https://www.reddit.com/r/LocalLLaMA/comments/186phti/m1m2m3_...
- Terretta 2y agoUnusable in what way? Llama 3.3 70b q8 with 100k+ context runs as well as GPT 3.5 did a couple years back, except local on Mac and smarter.
- lumost 2y agoLarge context has different perf than large model. Op was likely thinking of running 400b models and finding the compute wasn’t enough to make the memory useful.
- imtringued 2y agoIf you don't have tensor parallelism even 8 x A100 is going to struggle with that kind of model.
- novok 2y agoLook on youtube running various models and even a 70b model does a slow 3tok/s
- shikon7 2y agoIt's slower than using just GPU RAM, but it's still faster than using a regular PC that has a much more limited bandwith between the main memory and the GPU. It's some sort of middle ground on how fast and how cheap you can do inference with LLMs that don't fit into a consumer GPU's RAM.
- smcleod 2y agoIt's definitely usable past 48GB, I have a 96GB M2 Max and regularly run models that use around 70GB that are very usable. I also have a home server with 2x3090 and 2xA4000 (80GB vRAM) - yes it's a lot faster, but it's a pain in the ass to build, it takes up a lot of space, uses 10x the power, and honestly - cost about the same as my MacBook Pro.
- gazchop 2y agoI don’t think the principal market actually cares about that.
- mrtksn 2y agoWhen LLMs and diffusion models began spreading freely, it was very funny to have support for some beefy high end GPUs and MacBook Air M1. I hope Apple sticks with the architecture. Even if its not very practical, its great to have it as possible.
- halJordan 2y agoThis isn't splitting the unified memory, it's splitting the soc into a sip. The ram is still on the same interposer/substrate. The actual mm distance won't be regressing to say the distance of a dimm socket to the cpu socket. The software will still see a single memory pool I know im disagreeing with the article
- UncleOxidant 2y agoYeah, this is the edge Macs have right now in the AI space. It's why people are looking forward to the Strix Halo from AMD as it will also have a unified memory architecture and will probably cost a good bit less than a Mac.