5 ms·

You need memory on the GPU, not in the system itself (unless you have unified memory such as the M-architecture). So we're talking about cards like the H200 tha

You need memory on the GPU, not in the system itself (unless you have unified memory such as the M-architecture). So we're talking about cards like the H200 that have 141GB of memory and cost between 25 to 40k.

Borealid 1y ago
Did you casually glance at how the hardware in the Framework Desktop (Strix Halo) works before commenting?
- michaelanckaert 1y ago
  I didn't glace at it, I read it :-) The architecture is a 'unified memory bus', so yes the GPU has access to that memory. My comment was a bit unfortunate as it implied I didn't agree with yours, sorry for that. I simply want to clarify that there's a difference between 'GPU memory' and 'system memory'. The Frame.work desktop is a nice deal. I wouldn't buy the Ryzen AI+ myself, from what I read it maxes out at about 60 tokens / sec which is low for my use cases.
- ramon156 1y ago
  These don't run 200B models at all, results show it can run 13B at best. 70B is ~3 tk / s according to someone on Reddit.
  - Borealid 1y ago
    I don't know where you've got those numbers, but they're wrong. https://www.reddit.com/r/LocalLLaMA/comments/1n79udw/inference_on_new_framework_desktop/nc644nz/ https://www.reddit.com/r/LocalLLaMA/comments/1n79udw/inferen... seems comparable to the Framework Desktop and reputable - they didn't just quote a number, they showed benchmark output. I get far more than 3 t/s for a 70B model on normal non-unified RAM, so that's completely unfeasible performance for a unified memory architecture like Halo.
    - mhast 1y ago
      It depends on the model. It's typically ok for MoE models but if you try to run something non-MoE the speed will plummet. In that same thread there are people getting 50 tok/s on MoE models and 5 on non MoE. (https://www.reddit.com/r/LocalLLaMA/comments/1n79udw/comment/nc9oyxe/ https://www.reddit.com/r/LocalLLaMA/comments/1n79udw/comment...) And while it has unified memory the memory is quite slow. 250GB/s compared to 500+ for M4 Max or 1800 GB/s for a 5090. So it's fast for a CPU, but pretty slow for a GPU. (That said, there are not a lot of cheap options for running large models locally. They all have significant compromises.)

Borealid 1y ago

Did you casually glance at how the hardware in the Framework Desktop (Strix Halo) works before commenting?

michaelanckaert 1y ago
I didn't glace at it, I read it :-) The architecture is a 'unified memory bus', so yes the GPU has access to that memory. My comment was a bit unfortunate as it implied I didn't agree with yours, sorry for that. I simply want to clarify that there's a difference between 'GPU memory' and 'system memory'. The Frame.work desktop is a nice deal. I wouldn't buy the Ryzen AI+ myself, from what I read it maxes out at about 60 tokens / sec which is low for my use cases.
ramon156 1y ago
These don't run 200B models at all, results show it can run 13B at best. 70B is ~3 tk / s according to someone on Reddit.
- Borealid 1y ago
  I don't know where you've got those numbers, but they're wrong. https://www.reddit.com/r/LocalLLaMA/comments/1n79udw/inference_on_new_framework_desktop/nc644nz/ https://www.reddit.com/r/LocalLLaMA/comments/1n79udw/inferen... seems comparable to the Framework Desktop and reputable - they didn't just quote a number, they showed benchmark output. I get far more than 3 t/s for a 70B model on normal non-unified RAM, so that's completely unfeasible performance for a unified memory architecture like Halo.
  - mhast 1y ago
    It depends on the model. It's typically ok for MoE models but if you try to run something non-MoE the speed will plummet. In that same thread there are people getting 50 tok/s on MoE models and 5 on non MoE. (https://www.reddit.com/r/LocalLLaMA/comments/1n79udw/comment/nc9oyxe/ https://www.reddit.com/r/LocalLLaMA/comments/1n79udw/comment...) And while it has unified memory the memory is quite slow. 250GB/s compared to 500+ for M4 Max or 1800 GB/s for a 5090. So it's fast for a CPU, but pretty slow for a GPU. (That said, there are not a lot of cheap options for running large models locally. They all have significant compromises.)