ubermenchh

5 ms·

1. 1.▲
   Anki-CLI(github.com)

   2 points by ubermenchh 7mo ago|0 comments
2. 2.▲
   by ubermenchh 9mo ago

   yes it does continous batching along with paged attention and prefix caching. i am also goint to be adding some more inference techniques
3. 3.▲
   by ubermenchh 9mo ago

   Haha, i just wanted my repo to be out here. If someone finds it interesting they can always just check the repo. And you're close, its about getting faster responses from the model by manipulating the request queues and memory.
4. 4.▲
   Show HN: Mini-vLLM in ~500 lines of Python(github.com)

   5 points by ubermenchh 9mo ago|4 comments