7 ms·
How bad is the token-per-second drop-off on the Mac once you scale up the context window? Does it hit a wall at 32K or 64k tokens?
by yahavthehackern 3mo ago
How bad is the token-per-second drop-off on the Mac once you scale up the context window? Does it hit a wall at 32K or 64k tokens?