5 ms·

I can't imagine this setup will get more than 1 token per second. I would love to see Deepseek running on premise with a decent TPS.

• thomquaid 2y ago
  It says 4.25 TPS in the first para.
  • ricardobeat 2y ago
    Honest mistake. Some people think HN is just a series of short tweets and haven’t realized they are links yet!
    • thomquaid 2y ago
      4.25 is enough tps for a lot of use cases.
    • 4ndrewl 2y ago
      It's the modern way. Why read when you can just imagine facts straight out of your own brain.
      • plagiarist 2y ago
        I agree but also found your comment funny in the context of LLMs. People love getting facts straight out of their models.
  • weatherlight 2y ago
    That's still pretty slow, considering there's that "thinking" phase.
    • thomquaid 2y ago
      True, but 4.25 is the number we all want to know.
• october8140 2y ago
  You can get 1t/s on a raspberry pi. https://youtu.be/o1sN1lB76EA?si=i8ecEBjLdV0zewFQ https://youtu.be/o1sN1lB76EA?si=i8ecEBjLdV0zewFQ
  • klohto 2y ago
    this has nothing to do with the full 671B and the ollama models are distilled qwen2.5
    • thomquaid 2y ago
      I appreciate both of these comments, thank you both.