5 ms·

I'm running it on an M2 Max with 96GB, and have plenty of room to spare. And it's fast. Faster than I can get responses from ChatGPT.

• coder543 3y ago
  How many tokens/s? Which quantization? If you could test Q4KM and Q3KM, it would be interesting to hear how the M2 Max does!
  • teilo 3y ago
    No quantization (8_0). The full 48GB model. As for token count, I haven't tested it on more than 200 or so.
    • pilotneko 3y ago
      Isn’t 8_0 8-bit quantization?
      • teilo 3y ago
        Sorry. That was a major brain fart. Yes. 8-bit quantization, and using 49G of RAM.