6 ms·

does this do continuous batching or just static? couldn't tell from the code

• ubermenchh 9mo ago
  yes it does continous batching along with paged attention and prefix caching. i am also goint to be adding some more inference techniques