6 ms·
> like 180W-250W TDP > running GLM 5.2 would be cool at like ~100 tokens per second for a single session Your power consumption estimates are off for this gene
by acchow 2mo ago
> like 180W-250W TDP
> running GLM 5.2 would be cool at like ~100 tokens per second for a single session
Your power consumption estimates are off for this generation of GPUs. A 27B dense model gets 50-80 tps on an RTX 6000 using 600 watts.
- ThunderSizzle 2mo agoAn AMD R9700 gets 20-50 TPS at ~300 watts on 27B. 100 TPS for the 35B MOE model. And there might be some more optimizations to that as AMD software support gets better with ROCm's latest versions.
- syntaxing 2mo agoWhich quants? I get these speed (only 20-30 TPS) at Q4_K_M with MTP for 27B on my framework desktop. I draw sub 130W for the whole machine.
- acchow 2mo agoThat's about right. Half the TPS with half the power. GLM-5.2 will be much more demanding tho
- mhjkl 2mo agoIf you’re on AMD, use llamacpp with the vulkan backend. For the 35B I get around 115 tps without mtp and 150-170tps with it. Also the pcie slot you use affects it
- ThunderSizzle 2mo agoI have issues with the vulkan backend. It seems to cause over all GUI sluggishness, but maybe that was actually Firefox. I'll give it another go.