5 ms·
vLLM is originally marketed as paged attention, but in hindsight, separating the web server and GPU process, continuous batching, kv caching / chunking, and a h
by gdiamos 1mo ago
vLLM is originally marketed as paged attention, but in hindsight, separating the web server and GPU process, continuous batching, kv caching / chunking, and a huge model library including low precision mattered more.
I wonder how much it would cost to vibe code the whole thing from scatch?
I wonder how much better models need to get before such a thing wouldn't look like code vomit?
- mmastrac 1mo agoI've been working on a fresh, AI assisted port of DiffusionGemma from scratch and it takes a significant amount of time to deslop. I've spend a nonzero amount of time on refactoring and comment-vomit cleanup. https://github.com/mmastrac/diffgemma https://github.com/mmastrac/diffgemma
- brainless 1mo agoAny plans to support smaller models? I have a M4 Mac Mini with 16GB unified memory and an RTX 3060 (Laptop) with 6GB VRAM. My own product experiments all revolve around small models and harness around them. Happy to contribute.
- mmastrac 1mo agoI've been pondering a smaller quantization and experts swapping for this! Happy to take on PRs if you want to experiment as well.
- boredatoms 1mo agoSomewhat related, this vibe translation of vllm https://old.reddit.com/r/LocalLLaMA/comments/1vh9lx4/i_ported_vllms_serving_stack_to_c20_66_mib_binary/ https://old.reddit.com/r/LocalLLaMA/comments/1vh9lx4/i_porte...