8 ms·

They are int4 quantized

• Kranar 3y ago
  Does int4 mean 4 bits per integer, or 4 bytes/32-bits. If it means that weights for an LLM can be 4 bits well that's just mind boggling.
  • sillysaurusx 3y ago
    Four bits per parameter. (A parameter is what you call an integer here.) I was skeptical of it for some time, but it seems to work because individual parameters don’t encode much information. The knowledge is embedded thanks to having a massive number of low bit parameters.
  • justahuman74 3y ago
    4 bits