adiraja

7 ms·

1.
Using Nsight Compute to profile kernels in vLLM without creating repro scripts(blog.ncompass.tech)
1 points by adiraja 8mo ago|0 comments
2.
Runtime injection of temporary debug code(loom.com)
1 points by adiraja 8mo ago|1 comments
3.
by adiraja 8mo ago
We're building a developer tool (currently a VSCode / Cursor extension) that helps performance optimization engineers. One of the things we hated when analyzing performance and writing kernels ourselves was that things like timing
4.
A VSCode extension for GPU/AI trace profiling, viewing and performance analysis(marketplace.visualstudio.com)
3 points by adiraja 10mo ago|2 comments
5.
by adiraja 2y ago
Thank you! Absolutely, I'll send over a DM and we can take it from there!
6.
by adiraja 2y ago
Thanks for your comments. Absolutely, as we were mentioning in one of the other threads, we are really keen on building towards having a reproducible dashboard of efficiency and other metrics. Also regarding the no rate limits, we agree thi
7.
by adiraja 2y ago
Awesome, we really appreciate the suggestions! We'll look into getting these up and running shortly!
8.
by adiraja 2y ago
It now makes sense that when we tested the domain ncompass.com it took us to a Microsoft home page, which is why we're ncompass.tech :)
9.
by adiraja 2y ago
Hey, great that you mentioned this. We actually had BAAI/bge-m3 on our list of models to put up in the near future to see if people had use for it over an API. It's great to hear that this is something you're looking for. If
10.
by adiraja 2y ago
Per user throughput might be lower at the moment yes. We're working on GPU kernel level optimizations now to fix that. But across all users on our system, the throughput is better because doing more prefills or a large number of groupe
11.
by adiraja 2y ago
We focused mainly on the scheduling side of things. So we essentially prioritize prefills over decodes. In order to do this correctly, we had to monitor KV cache usage and whenever it's close to running out of memory, we schedule more
12.
Show HN: NCompass Technologies – yet another AI Inference API, but hear us out(ncompass.tech)
37 points by adiraja 2y ago|34 comments