22 ms·
I believe they batch requests so the same weights in vram are shared across many requests https://www.baseten.co/blog/continuous-vs-dynamic-batching-for-ai-infe
by mnahkies 2mo ago
I believe they batch requests so the same weights in vram are shared across many requests https://www.baseten.co/blog/continuous-vs-dynamic-batching-for-ai-inference/ https://www.baseten.co/blog/continuous-vs-dynamic-batching-f...