15 ms·

Right, performance is more than just the aggregated memory speed of the hardware you have. I'm fairly sure, at least last time I looked, maybe Apple launched so

Right, performance is more than just the aggregated memory speed of the hardware you have. I'm fairly sure, at least last time I looked, maybe Apple launched something new in the last 2-3 months that has completely changed the picture?

swiftcoder 6d ago
> performance is more than just the aggregated memory speed of the hardware you have In other fields, sure, but for big LLMs it's a very significant part of the performance picture. That stack of Macs also is going to be able to natively run models 4-5x larger than the dual Blackwells can hold in memory - any model over about 128GB of weights isn't going to fit on the GPUs, and is going to be heavily performance constrained by moving data across the PCIE bus. > maybe Apple launched something new in the last 2-3 months that has completely changed the picture Indeed. The M5 Ultra (currently up for pre-order) has 50% higher memory bandwidth than its predecessor, and a claimed 4x improvement in prompt prefill.
- embedding-shape 6d ago
  > any model over about 128GB of weights isn't going to fit on the GPUs Not sure if you misunderstand what GPUs we're talking about, one RTX Pro 6000 has 96GB of VRAM. > Indeed. The M5 Ultra (currently up for pre-order) has 50% higher memory bandwidth than its predecessor, and a claimed 4x improvement in prompt prefill. Exciting! Eagerly awaiting the benchmarks and comparisons then. Lets hope "4x improvement" had a good enough baseline so 4x actually ends up useful in practice compared to the current hardware they offer. > 50% higher memory bandwidth Seems this lands on ~1.2 TB/s if what Apple claims is correct. For reference, RTX Pro 6000 does 1.8 TB/s, so seems Apple is indeed getting closer incrementally.
  - swiftcoder 5d ago
    > Not sure if you misunderstand what GPUs we're talking about, one RTX Pro 6000 has 96GB of VRAM. Right, but the top option here is a pair of RTX Pro 6000s, hence 192 GB of VRAM in total. Should be enough for a 128GB model plus context, cache, etc.
    - embedding-shape 5d ago
      Yeah, that's why the "any model over about 128GB of weights isn't going to fit on the GPUs" part doesn't make sense, you'll easily be able to run weights in that weight class on two of them. Or did I misunderstand what "on the GPUs" you meant?
      - swiftcoder 5d ago
        Right, easily run 128-160 GB models, yes. Not easily run models much larger than that. Anything that doesn't fit in the 192 GB (including context, cache, etc) is going to have to be sparse/MoE, and the rather anaemic PCIE bandwidth is going to hurt. By comparison, the 5x Mac cluster should be able to run a dense ~800GB-1TB model without a drastic slowdown.

swiftcoder 6d ago

> performance is more than just the aggregated memory speed of the hardware you have In other fields, sure, but for big LLMs it's a very significant part of the performance picture. That stack of Macs also is going to be able to natively run models 4-5x larger than the dual Blackwells can hold in memory - any model over about 128GB of weights isn't going to fit on the GPUs, and is going to be heavily performance constrained by moving data across the PCIE bus. > maybe Apple launched something new in the last 2-3 months that has completely changed the picture Indeed. The M5 Ultra (currently up for pre-order) has 50% higher memory bandwidth than its predecessor, and a claimed 4x improvement in prompt prefill.

embedding-shape 6d ago
> any model over about 128GB of weights isn't going to fit on the GPUs Not sure if you misunderstand what GPUs we're talking about, one RTX Pro 6000 has 96GB of VRAM. > Indeed. The M5 Ultra (currently up for pre-order) has 50% higher memory bandwidth than its predecessor, and a claimed 4x improvement in prompt prefill. Exciting! Eagerly awaiting the benchmarks and comparisons then. Lets hope "4x improvement" had a good enough baseline so 4x actually ends up useful in practice compared to the current hardware they offer. > 50% higher memory bandwidth Seems this lands on ~1.2 TB/s if what Apple claims is correct. For reference, RTX Pro 6000 does 1.8 TB/s, so seems Apple is indeed getting closer incrementally.
- swiftcoder 5d ago
  > Not sure if you misunderstand what GPUs we're talking about, one RTX Pro 6000 has 96GB of VRAM. Right, but the top option here is a pair of RTX Pro 6000s, hence 192 GB of VRAM in total. Should be enough for a 128GB model plus context, cache, etc.
  - embedding-shape 5d ago
    Yeah, that's why the "any model over about 128GB of weights isn't going to fit on the GPUs" part doesn't make sense, you'll easily be able to run weights in that weight class on two of them. Or did I misunderstand what "on the GPUs" you meant?
    - swiftcoder 5d ago
      Right, easily run 128-160 GB models, yes. Not easily run models much larger than that. Anything that doesn't fit in the 192 GB (including context, cache, etc) is going to have to be sparse/MoE, and the rather anaemic PCIE bandwidth is going to hurt. By comparison, the 5x Mac cluster should be able to run a dense ~800GB-1TB model without a drastic slowdown.