5 ms·
Correct, it's breaks the single prompt, single completion assumption baked into the frameworks. Conceptually it's still prompt/completion but for low latency re
by red2awn 9mo ago
Correct, it's breaks the single prompt, single completion assumption baked into the frameworks. Conceptually it's still prompt/completion but for low latency response you have to do streaming KV cache prefill with a websocket server.
- whimsicalism 9mo agoI imagine you have to start decoding many speculative completions in parallel to have true low latency.