Lindon4290

6 ms·

1.
by Lindon4290 2y ago
Yes, a single sequence with 256 prompt tokens and 256 output tokens. This is a batch size = 1. No one is saying anything about 256 batches. The first step in understanding this is to notice that the model (llama2) generates 1 output token a
2.
by Lindon4290 2y ago
Yeah, I've asked questions along those lines as well. Something sketchy is going on. See here - https://news.ycombinator.com/item?id=40833109
3.
by Lindon4290 2y ago
Now, I don't have any MI300X, so I can't make any definite claims here. I am hoping someone else can replicate the results shown here or at the least educate me on how this is possible. Good part is the docker container and associ
4.
by Lindon4290 2y ago
You should make a whole post about this! Like how a single MI300X outperforms groq at bs=1. 300 tokens/s with bs=1 for a llama-2 70B on a single card is no joke.
5.
by Lindon4290 2y ago
Yeah, the rumours(?) are a groq system required to produce 300+ t/s on a Llama-2 70B (bs=1) requires 576 chips (9 racks) [1] So, that's like $10M+ for serving bs=1 Llama-2 70B vs whatever a single MI300X costs? [1] https:/&#
6.
by Lindon4290 2y ago
Going by the results from the article/video, a single MI300X is even outperforming a Groq system [1] The video shows that the optimized run with Llama-2 70B gives 314 tokens/s for a bs=1 with 256 prompt + 256 generation. The Groq
7.
by Lindon4290 2y ago
Right. On that track, I want to confirm something. Maybe I am doing my math wrong or don't understand how transformers work. There is a video about the bs=1 case, i.e. a single prompt with input 256 tokens and output 256 tokens with a
8.
by Lindon4290 3y ago
Looks their performance is better than llama.cpp - https://news.ycombinator.com/item?id=37018989 - and scales to batches of prompts.