wowitsbase

8 ms·

1. 1.▲
   by wowitsbase 8d ago

   Somewhat, but most of the optimizations are cuda based. I would guess it would be a little faster, like 20-40%, but not 200% like on Nvidia.
2. 2.▲
   I promise this Ollama replacement will give you the fastest t/s you've ever had(github.com)

   2 points by wowitsbase 8d ago|3 comments
3. 3.▲
   by wowitsbase 8d ago

   A small developer team and I had worked on this for a while for personal reasons, so I have decided to port it to cpp and release it to the general public. The stats are in the github. I promise if you have a gpu this will improve your spee
4. 4.▲
   Show HN: Ollama stand in replacement, 2-4x faster, not just basic optimizations(github.com)

   3 points by wowitsbase 9d ago|0 comments
5. 5.▲
   by wowitsbase 10d ago

   Speculative decoding was on in all of my tests, the gain came from moving the round's inputs off host memory.
6. 6.▲
   by wowitsbase 10d ago

   agreed, if not for this project I've been making I would at least be using base llama.cpp Personally I think it comes down to simplicity, but there's no reason for it's performance drops compared to llama.cpp while it's
7. 7.▲
   Ollama replacement 2-4x faster for no extra compute cost(github.com)

   4 points by wowitsbase 10d ago|4 comments