i_am_rocoe

7 ms·

1. 1.▲
   by i_am_rocoe 2mo ago

   I'm working on a variation of MTP that recovers PP TPS (back to the same as with MTP disabled), keeping most of MTP's benefits to TG TPS. https://news.ycombinator.com/item?id=48700782 Will propose a patch back to
2. 2.▲
   by i_am_rocoe 3mo ago

   Thank you!
3. 3.▲
   I patched llama.cpp to gain 20% prompt processing TPS. Help me make a PR

   6 points by i_am_rocoe 3mo ago|2 comments
4. 4.▲
   by i_am_rocoe 3mo ago

   If you want to run the PoC locally to replicate: Clone the masked-nextn-skip-catchup branch: https://codeberg.org/rocoe/llama.cpp/src/branch/masked-nextn... Run llama-server with at least --n-gpu-layers
5. 5.▲
   As; HN: I was curious why MTP affects PP TPS in llama.cpp. My PoC recovers it?

   4 points by i_am_rocoe 3mo ago|1 comments