5 ms·
You should make a whole post about this! Like how a single MI300X outperforms groq at bs=1. 300 tokens/s with bs=1 for a llama-2 70B on a single card is no jok
by Lindon4290 2y ago
You should make a whole post about this! Like how a single MI300X outperforms groq at bs=1.
300 tokens/s with bs=1 for a llama-2 70B on a single card is no joke.
- latchkey 2y agoThis is why I sponsored doing the chipsandcheese tests on my hardware. That instigated Elio to up the game even further. All open source by the way.
- fancyfredbot 2y agoThank you for sponsoring this. There's so little buzz about this hardware despite the fact it's clearly amazing for AI use cases. I don't understand why not. Maybe this is why Nvidia is the most valuable company in the world - nobody can be bothered to try a competitor.