venkat_2811

5 ms·

1.
by venkat_2811 2mo ago
while i'm a fan of wafer's work, 1024 input token length is not a great benchmark anymore, that number is useful only at 1-4x single node a100 / h100 at various concurrency levels
2.
ns scale ultra-low-latency fabric over shm and MMAP for IPC(crates.io)
2 points by venkat_2811 4mo ago|0 comments
3.
by venkat_2811 6mo ago
ai agents failing silently or just lying is a big problem Steadwing and openalerts save a lot of headache for sure ! congrats on the launch !
4.
Show HN: Nvidia's CUDA libraries are generic and not optimized for LLM inference(github.com)
1 points by venkat_2811 8mo ago|1 comments
5.
by venkat_2811 8mo ago
With so much improvements in LLM Inference Kernels, Inter-GPU comms are becoming the bottleneck. Introducing my project YALI - Yet Another Low-Latency Implementation. A custom CUDA kernel library that provides ultra low-latency primitives f
6.
by venkat_2811 8mo ago
Wisdom from CPU land translate well to GPUs. Static Scheduling, Pre-fetching, 3-Stage Double-Buffering, Pre-allocation & memory ordering in custom CUDA kernel helps outperform NVIDIA NCCL. Experimental integration in vllm.rs shows ~20%
7.
I Beat Nvidia NCCL by 2.4x(venkat-systems.bearblog.dev)
2 points by venkat_2811 8mo ago|2 comments
8.
by venkat_2811 8mo ago
100% OSS, MIT License. YALI - Yet Another Low-Latency Implementation. Achieves 80-85% Speed-of-Light SW efficiency by using ultra low-latency primitives for p2p all_reduce_sum comms collective. Very important operation in multi-gpu llm trai