8 ms·
Batched often means a higher latency option (minutes/hours instead of seconds), which providers can schedule more efficiently on their GPUs.
by sophiebits 2y ago
Batched often means a higher latency option (minutes/hours instead of seconds), which providers can schedule more efficiently on their GPUs.