5 ms·
Based on the integration examples, I don't think they are simply repackaging llama.cpp Rather it looks like they are reimplementing their own quantization sche
by polishgladiator 3y ago
Based on the integration examples, I don't think they are simply repackaging llama.cpp
Rather it looks like they are reimplementing their own quantization scheme, in such a way that it is a little easier to integrate for basic python users, at the cost of performance (compared to llama.cpp and others).
Given that the bar for integrating something with higher perf like llama.cpp isn't very high (and that's the way the world is heading -- ask any 15 year old interested in this stuff), I can't see anything of value here.
- Lindon4290 3y agoLooks their performance is better than llama.cpp - https://news.ycombinator.com/item?id=37018989 https://news.ycombinator.com/item?id=37018989 - and scales to batches of prompts.
- polishgladiator 3y agoActually no -- that post shows they are not performing measurements and comparisons correctly. These are not serious people.