9 ms·
I'm running it on my Intel Xeon W5 with 256GB of DDR5 and Nvidia 72GB VRAM. Paid $7-8k for this system. Probably cost twice as much now. Using UD-IQ4_NL quant
by SlavikCA 5mo ago
I'm running it on my Intel Xeon W5 with 256GB of DDR5 and Nvidia 72GB VRAM. Paid $7-8k for this system. Probably cost twice as much now.
Using UD-IQ4_NL quants.
Getting 13 t/s. Using it with thinking disabled.
- GrayShade 5mo agoI get 20 t/s on the UD-Q6_K_XL quant, Radeon 6800 XT.