6 ms·
Ran this on a 64 GB M4 Max MacBook. I figured having Gemma available with a small footprint would be a nice setup. No more unloading models when I need more RAM
by pwython 2mo ago
Ran this on a 64 GB M4 Max MacBook. I figured having Gemma available with a small footprint would be a nice setup. No more unloading models when I need more RAM for work? Hell yea.
Got 48 tok/s decode at 1.9 GB RSS (2.4 GB peak), faster than the 24 GB M5 Pro mentioned in the benchmarks. The ~2.0 GB/s SSD number quoted for M4 is the base chip. This M4 Max does ~7 GB/s.
Page cache seems to be why it beats the M5 Pro. With 64 GB the whole 12 GB packed_experts set stays resident, and iostat shows only ~1.6 GB per run actually reaching disk, against the ~79 GB that 98 fully cold tokens would need.
I then tested with DaVinci Resolve open and under load (playback): 42.6 tok/s. Also held 38 GB of incompressible memory to squeeze the page cache: 41.8. At 48 GB it ranged 32 to 41.5. Degrades gradually rather than a cliff. It's a beautiful thing.
- fouc 2mo agoM4 Max is typically better than M5 Pro for inference IIRC.
- harrouet 2mo agoIt depends on what you are looking at. Time to 1st token is faster on the M5 because of HW accelerators helping the prompt interpretation (and it is CPU-bound). Token generation after that is GPU-bound and will profit from the higher bandwidth of the M4 Max.
- sznio 2mo agoat that much ram you can just load it outright without tricks. it will be much faster even if it ends up swapping.
- gitpusher42 2mo agoThank you very much for sharing! Great results and useful info!
- deleted 2mo ago[deleted]
- pitchlatte 2mo agoplayback in Resolve would probably just use hardware decoding and barely hit your CPU or GPU. RAM usage would also not be much.
- anon373839 2mo agoNice! Can you mention what kind of prefill numbers you’re seeing?