6 ms·
yes it does continous batching along with paged attention and prefix caching. i am also goint to be adding some more inference techniques
by ubermenchh 9mo ago
yes it does continous batching along with paged attention and prefix caching.
i am also goint to be adding some more inference techniques