nirw4nna

4 ms·

1. 1.▲
   by nirw4nna 11mo ago

   I'm currently chipping away at DSC, a tensor library I wrote from scratch to play with large language models. Last week I re-wrote flash attention from scratch in CUDA and was able to get good perf. [1]: https://github.com&#
2. 2.▲
   Why I Ditched Malloc for AI Inference(gilli.dev)

   4 points by nirw4nna 1y ago|0 comments
3. 3.▲
   by nirw4nna 1y ago

   I'm currently working on DSC, a tensor library I wrote from scratch in C++ with a PyTorch-like API. Right now it works on both CPU and GPU (both AMD and NVIDIA) and is capable of running LLMs like Qwen, I'm currently implementing
4. 4.▲
   by nirw4nna 1y ago

   Because I happen to know C++ and I just wanted to build something rather than learn a new language. Zig looks very interesting though, there are already other projects in this space that use it with great success (see: https://gi
5. 5.▲
   by nirw4nna 1y ago

   You just need a foundation of C/C++. If you already have that then just start programming, it's way better than reading books/guides/blogs (at least until you're stuck!). Also, you can read the source code of other
6. 6.▲
   by nirw4nna 1y ago

   Yes! This was actually one of my initial goals! I actually like to work in a C-style-C++ let's say where I turn off C++ features I don't need and just use the one I actually need like templates, objects ecc... I find this style to
7. 7.▲
   by nirw4nna 1y ago

   Thanks for pointing this out! I'll definitely have to investigate other approaches. nanobind looks interesting but I don't need to expose complex C++ objects, I just need the 'fastest' way of calling into a C API. I gues
8. 8.▲
   by nirw4nna 1y ago

   I developed this on an HP Omen 15 with an i7-8750H, a GTX 1050TI and 32GB or RAM with Linux Mint as my OS.
9. 9.▲
   by nirw4nna 1y ago

   Right now I can load tensors directly from a safetensors file or from a NumPy array so I don't really have in mind to add my own custom format but I do plan to support GGUF files.
10. 10.▲
    by nirw4nna 1y ago

    You are absolutely correct! I started working on a sort of compiler a while back but decided to get the basics down first. The templates and switch(s) are not really the issue but rather going back and forth between C & Python. This is
11. 11.▲
    by nirw4nna 1y ago

    Thanks! To be honest, it started purely as a learning project. I was really inspired when llama.cpp first came out and tried to build something similar in pure C++ ( https://github.com/nirw4nna/YAMI ), mostly for fun and
12. 12.▲
    by nirw4nna 1y ago

    Yes, when I designed the API I wanted to keep a clear distinction between Python and C. At some point I had two APIs: 1 in Python and the other in high-level C++ and they both shared the same low-level C API. I find this design quite clean
13. 13.▲
    Show HN: I built a tensor library from scratch in C++/CUDA(github.com)

    119 points by nirw4nna 1y ago|28 comments