11 ms·
I've run it with a large context window on 256GB ram + 4090. It wasn't super fast, but it was manageable and it completed the tasks I gave it well.
by kybernetikos 1mo ago
I've run it with a large context window on 256GB ram + 4090. It wasn't super fast, but it was manageable and it completed the tasks I gave it well.
- xiconfjs 1mo agoHow many tps and size of ctx?