6 ms·
> on a decent speed But you said 7-9 tokens/second, that's not a decent speed. I'm not an expert by all means but in my local experiments, less than 12 to 16 t
by reidrac 2mo ago
> on a decent speed
But you said 7-9 tokens/second, that's not a decent speed. I'm not an expert by all means but in my local experiments, less than 12 to 16 tps is too slow.
- aaulia 2mo agoI tried Qwen MoE a while back. Using my 8GB RX470, somehow got 10+ token/sec, lot's of trial and error with llama.cpp config, and it's still slow to be used for my usecase. Even at 12 to 16 IMO it's slow. For chat, maybe it's enough, but for any other tasks, it's not viable IMO
- dwa3592 2mo agoi am working on making it faster but to me 7-9 tokens/sec feels very good. it was 0 tokens/sec a year ago.
- jckahn 2mo agoIgnore the haters. What you've done is incredible!
- Aurornis 2mo agoIf you’re interested in these projects you should check out the project this was based on: https://github.com/JustVugg/colibri https://github.com/JustVugg/colibri It says so right in the readme. They’re not hiding anything.
- bigyabai 2mo agoYep, and we don't even know how long they spent on prefill. A typical 50-100k token session could take 10-20 minutes to prefill on a Mac. Pending any new hardware or radically different LLM architectures, we're going to be waiting a lot longer than 2027 for 200B models on local hardware. SOC platforms like Apple Silicon are hamstrung by their obsession over raster performance, the hardware lacks the fundamental GPGPU hardware to be a replacement for real-world inference.
- LeifCarrotson 2mo agoI think that any workflow that requires the user to stare at the tokens being generated live is using it wrong. Delegate, don't stare! https://mikeveerman.github.io/tokenspeed/?rate=10&mode=text https://mikeveerman.github.io/tokenspeed/?rate=10&mode=text You think of an idea that you want to have the LLM process, queue it up, and go back to what you were doing. Once you've finished reading the next article on HN about a 5 tps Xeon, your task will be complete. It's kind of like using a 3D printer: It doesn't matter if a print takes 10 hours, because when you come back in the morning it will be done. Yes, with top-tier GPU farms you can hit hundreds of tokens per second. But if the old Xeon in the closet can get useful work done at 5 tokens per second, there are lots of people and lots of use cases where a free, unlimited 5 TPS stream is worth more than paying a dollars per day to get access to a 500 TPS source.
- parineum 2mo ago> there are lots of people and lots of use cases where a free, unlimited 5 TPS stream is worth more than paying a dollars per day to get access to a 500 TPS source. That Xeon is going to be devouring power.
- bitpush 2mo agois there a good tool to manage these workloads? batch process a bunch, handle failures, retry things etc?
- adastra22 2mo agoWe aren’t there yet. Not for frontier development work at least.
- ctoth 2mo ago[dead]
- allknowingfrog 2mo agoWe clearly have different goals. I want an LLM to review my code, not the other way around.
- 2mo ago
- aziis98 2mo agoFor me, at least for agentic use, you need at least ~40tps. Less might be good only for tasks you could run in the background (like at night maybe). Instead of coding agents like tool for local models I would like to see more "docker ps" like tools where you can queue up tasks that get processes incrementally (maybe when the pc i idle) and are specialized for doing retries and caching as much work as possible to work decently with slow local models.
- ClikeX 2mo agoThe slower models seem fine for home lab usecases such as processing document transcriptions and tagging them, for example. I don’t need that to be live, it can just churn overnight.
- kevincox 2mo agoI think 10 t/s output is usable for many coding workflows if the input speed is much much higher (~100 t/s is a rough minimum line). The low output speed can really hurt for heavy reasoning output but it can still be used to get some jobs done as long as you don't expect highly interactive use.
- Wowfunhappy 2mo agoHow fast does a human write code?
- sigmoid10 2mo agoIt's fast if you want to automate things that run independently or overnight. It's slow if you want to iterate code together with it.
- Wowfunhappy 2mo agoIf you were iterating on code with a human partner, how fast would they write? Like, obviously faster is better and the entire point of computers is to do things faster, but I find it kind of surprising how many people consider 9 tokens per second—heck, even much less than that—to be unusably slow. It's still automating a class of task that virtually no one before 2022 was able to automate, and it's faster than basically any human can write code. I guess the better question is, why would you use a 9 token per second system when you could use a much faster cloud model. Obviously, if you want speed, that's the way to go. But a lot of people seem to find the idea of sending their requests to a third party server untenable. If 9 tokens per second is the best you can do—I don't know, that seems usable and useful to me.
- mh- 2mo agoThis isn't just measuring the speed of tokens that are written to a file (code). Your human partner is thinking a lot faster than 5 tok/s.
- sigmoid10 2mo agoI guess you haven't used Codex/Copilot much? All good LLMs now use reasoning. Especially for code it is almost a requirement to produce correct results. But that means even having them implement a comparatively simple function or algorithm can easily produce 3000 output tokens. Waiting 5+ minutes every time you iterate because you want a tiny tweak in the code is not a practical strategy for live development.
- 2mo ago