6 ms·
imo they dont have batching because they pack sequences before passing through the model. so a single sequence in a batch on OpenAI might have requests from mul
by advaith08 3y ago
imo they dont have batching because they pack sequences before passing through the model. so a single sequence in a batch on OpenAI might have requests from multiple customers in it
- deleted 3y ago[deleted]
- sidnb13 3y agoAh that would make sense. Similar to vLLM which does dynamic packing.