ani17

6 ms·

1.
by ani17 5mo ago
Author here. I wanted to understand what vLLM and llama.cpp are actually doing under the hood, but the codebases are massive. So I wrote a stripped down version from scratch to see the core ideas without the production complexity. Code: ht
2.
Why do output tokens cost 5x more than input tokens?(anirudhsathiya.com)
3 points by ani17 5mo ago|2 comments
3.
by ani17 5mo ago
The blog walks through why your first token is always the slowest, why output tokens cost 5x more, and how stuff like speculative decoding and chunked prefill actually work, from the perspective of a systems engineer!
4.
by ani17 5mo ago
Author here. A bit more context: By day I'm a systems engineer building AI networking infrastructure. So I kept ending up in conversations where I'm not exactly able to wrap my head on the latest inference magic trick. Like when s
5.
LLM inference engine from scratch in C++ – why output tokens cost 5x(anirudhsathiya.com)
9 points by ani17 5mo ago|3 comments
6.
by ani17 10mo ago
Definitely an alternative solution. For the purpose of this script, I wouldn't prefer that though.
7.
Ask HN: How cam I auto-switch shared Google Meet tab?
1 points by ani17 10mo ago|2 comments
8.
by ani17 10mo ago
It's insane if the data is accurate. Only time will tell
9.
How Much OpenAI Spends on Inference and Its Revenue Share with Microsoft(wheresyoured.at)
68 points by ani17 10mo ago|25 comments
10.
by ani17 1y ago
You forgot "Middle Out" by Pied Piper!
11.
by ani17 1y ago
thanks for sharing!