6 ms·

We're on the cusp of Kimi K3 becoming usable on sub-10k hardware. https://github.com/gavamedia/deltafin https://github.com/gavamedia/deltafin

edot 2mo ago
14 seconds per token? Not tokens per second. Seconds per token? That’s nowhere near the cusp!
- dotancohen 2mo ago
  It's need it to be an order of magnitude cheaper, but for some queries ("What to discuss at tomorrow's meeting") I can wait 12+ hours.
  - throw10920 2mo ago
    I'd rather just use my actual human brain to compute the answer at that point. I don't see the value at throughput that is this low.
    - dotancohen 2mo ago
      I'd rather not be thinking about tomorrow's meeting all night! Just trust "the robot has got this"... and scramble in the morning when the robot has failed. But at least I got a good night's sleep!
- antonvs 2mo ago
  At that rate, it would take me a mere 15 days to generate the number of tokens I typically use in a day. "Yes, I'm using AI to speed up development. I'll submit that PR in two weeks time!"
  - adrian_b 2mo ago
    That speed is significantly lower that what can be attained with streaming from SSDs. Not only in big desktops, but even in most recent mini-PCs, it is possible to read concurrently from one PCIe 5.0 SSDs and one PCIe 4.0 SSD, at a total sustained reading throughput of around 20 GB/s. With an optimized inference implementation, it should be possible to overlap completely the computations with streaming weights from SSDs. This should improve the inference speed to around at least 1 token per second, on a cheap computer, under $2000 even at the current super-inflated prices. There are enough tasks where this would be useful. Obviously one should use for most tasks a fast small LLM and use the big one only when this actually saves time.
ekidd 2mo ago
For values of "usable" that include "14.6 seconds/token". It's a cool accomplishment! And newer hardware would speed it up some. But I think I'd want something a bit faster before declaring it usable in practice.

edot 2mo ago

14 seconds per token? Not tokens per second. Seconds per token? That’s nowhere near the cusp!

dotancohen 2mo ago
It's need it to be an order of magnitude cheaper, but for some queries ("What to discuss at tomorrow's meeting") I can wait 12+ hours.
- throw10920 2mo ago
  I'd rather just use my actual human brain to compute the answer at that point. I don't see the value at throughput that is this low.
  - dotancohen 2mo ago
    I'd rather not be thinking about tomorrow's meeting all night! Just trust "the robot has got this"... and scramble in the morning when the robot has failed. But at least I got a good night's sleep!
antonvs 2mo ago
At that rate, it would take me a mere 15 days to generate the number of tokens I typically use in a day. "Yes, I'm using AI to speed up development. I'll submit that PR in two weeks time!"
- adrian_b 2mo ago
  That speed is significantly lower that what can be attained with streaming from SSDs. Not only in big desktops, but even in most recent mini-PCs, it is possible to read concurrently from one PCIe 5.0 SSDs and one PCIe 4.0 SSD, at a total sustained reading throughput of around 20 GB/s. With an optimized inference implementation, it should be possible to overlap completely the computations with streaming weights from SSDs. This should improve the inference speed to around at least 1 token per second, on a cheap computer, under $2000 even at the current super-inflated prices. There are enough tasks where this would be useful. Obviously one should use for most tasks a fast small LLM and use the big one only when this actually saves time.

ekidd 2mo ago

For values of "usable" that include "14.6 seconds/token". It's a cool accomplishment! And newer hardware would speed it up some. But I think I'd want something a bit faster before declaring it usable in practice.