polishgladiator

9 ms·

1.
by polishgladiator 3y ago
Something doesn't smell right. Such sloppy errors with measurement and comparison (from people who are supposedly experts?), and cageyness about answering technical questions, reminds me of the era of crypto currency scams..
2.
by polishgladiator 3y ago
Actually no -- that post shows they are not performing measurements and comparisons correctly. These are not serious people.
3.
by polishgladiator 3y ago
OK, so this is a case of bad measurement and comparison. If you bothered to look at the llama.cpp output, you would see this line: llama_model_load_internal: offloaded 32/35 layers to GPU The "-ngl 32" means that only 32 ou
4.
by polishgladiator 3y ago
> [...] llama.cpp is a fantastic framework to run models locally for the single-user case (batch=1) > [...] I don't think it would be particularly fair to compare and show that MKML is better at a given perplexity, compression r
5.
by polishgladiator 3y ago
> If anyone has specific technical questions I'd be happy to answer as best I can. What is the context size for these measurements? Is it the full 4k for llama-2? And just to be clear, when you say memory footprint, this is the enti
6.
by polishgladiator 3y ago
Based on the integration examples, I don't think they are simply repackaging llama.cpp Rather it looks like they are reimplementing their own quantization scheme, in such a way that it is a little easier to integrate for basic python u
7.
by polishgladiator 3y ago
I've been doing some hacking with Llama2 on an AMD 7900 XTX this weekend, using llama.cpp and q5_k_s quantization. Compared to MK600 on an RTX 4090 in their data, I am measuring higher throughput and lower perplexity (again, note that
8.
by polishgladiator 3y ago
In the realm of text editors, Vim has been my unwavering companion for 25 years. It has become an integral part of my daily existence, an inseparable bond that defies the imagination of life without its embrace. Farewell Bram!