5 ms·
For anyone wondering “how slow is this?” IIUC, Kimi K3 on RTX 6000 Ada (48GB) takes 292 s/token https://github.com/lyogavin/airllm/releases/tag/v3.1.0 https:/
by imenani 2mo ago
For anyone wondering “how slow is this?”
IIUC, Kimi K3 on RTX 6000 Ada (48GB) takes 292 s/token
https://github.com/lyogavin/airllm/releases/tag/v3.1.0 https://github.com/lyogavin/airllm/releases/tag/v3.1.0
- jackb4040 2mo agoAhaha thank you, I naively assumed the unlabeled graph in the readme was tps, not spt!
- throwawayffffas 2mo agoRunning 292 tps on K3 would make your gpu a money printer.
- meneton 1mo agospt not tps
- deleted 2mo ago[deleted]
- bensyverson 2mo agoI do think these “run a bigger model than will fit in VRAM” projects are necessary steps, but are they functionally useful or helpful to anyone currently? For example, is anyone out there running a big Qwen for coding on a 16-32GB machine with these techniques?
- pizza234 2mo ago> but are they functionally useful or helpful to anyone currently? Yes and no, depend on your expectations. Some/many like to run local LLMs just for the sake of it, so anything will do. MoE are useful on PC systems, at the condition of having high enough memory bandwidth (and large amounts of RAM) - that is, Threadripper/Pro. The advantage of MoE is that only a subset of the model's experts is used for each token, so not all weights need to be present in VRAM at once. The remaining weights can reside in system RAM, although moving and accessing them still carries a substantial performance cost (and that's why high memory bandwidth is needed).
- ddevnyc 2mo agoDoes MoE help with multimodality? Can it in general enable reasoning in imagery (technical drawings, diagrams, schematics) rather than text-based?
- dannyw 2mo agoMoE has nothing to do with multimodality. MoE is a concept proposed in 1991, before the deep learning era (which is before what I call the transformers era). You can think of it like sharing. Contrary to popular belief; 'experts' in MoE LLMs do not specialize. There's no expert trained to be good at maths, or python, or writing, or whatever. It's an inference optimization. As for reasoning in non-text modalities, you might find this paper interesting :) https://huggingface.co/papers/2502.05171 https://huggingface.co/papers/2502.05171
- dummydummy1234 1mo agoWait I thought the router ends up specializing the experts? Like there is no explicit goal aside from each 'expert' getting roughly equal weight? And it happens that when you train the router you do end up passing certain classes of problem to each expert - just as a training result nothing as clean as a python expert. But math vs creative writing will tend to rely on different experts over the majority of the inference? I do not know what I am talking about, this is my limited understanding...
- juliendorra 1mo agoYou are right that at training the main goal is balancing between the sections to avoid certain paths becoming the only path. In the end the inference will be routed token by token to a mixture of say 3 or 4 sections of the model. The combination can change at each turn. It’s really a statistical optimization. As for many things in neural networks, the original intuition coming from anthropomorphism once implemented becomes something very non human!
- ahepp 1mo ago
- nickpsecurity 2mo agoIf I could justify wear and tear and electricity, I was willing to do something like this for batch processing. The batches would be a bunch of prompts whose outputs I'd look at the next day. Maybe common operations, like QA or refactoring, on whatever software I wrote. If so, I could use a larger model than I have real-time hardware for. The largest, well-trained models can often get the output mostly right in one try. I also would be using AI's as a supplement to, not replacement for, my own brain. So, issues with the outputs wouldn't be a problem because I'm just keeping what's helpful. If I still need to re-generate it all, it might still save money over time by avoiding cloud costs. Also, hardware that's already paid for is a sunk cost that doesn't inflate over time. Glitches in loading or destroying VM's might blow up into a big bill.
- parineum 1mo agoat 5 minutes per token, you could look at the results next week
- bensyverson 1mo agoone week later: "It says 'You're absolutely right! Let me look at the seams so I'm checking, not guessing—' and I guess that's when my SSD melted."
- hex4def6 1mo agoTurns out the SSD was load-bearing.
- nickpsecurity 1mo agoThat's way slower than I thought it would be. I struggle to imagine a use case. If you had a deadly condition, and no diagnosis worked, and a specific model had the answer... past that I wouldn't use it.
- deleted 1mo ago[deleted]
- zapataband1 1mo ago[flagged]
- dsl 1mo ago> but are they functionally useful or helpful to anyone currently? I've been building a SaaS that deals with data that can't be distributed to third parties. Some of the useful AI stuff I can add is not time sensitive and can run overnight. Things like this allow me to use higher quality models without selling my house for GPUs.
- xnorswap 2mo agoI wonder what this measures in J/token.
- throwawayffffas 2mo agoAssuming 30% gpu power utilization because of all the loading and unloading 29.2 kJ per token
- donquichotte 1mo agoDamn 15 AK47 bullets per token
- a96 1mo agoAmericans will use anything but ISO standard units.
- logicallee 2mo agothat's 0.003 tokens/second. To get an hour's work done that's normally 30 tokens/second (108k output tokens in an hour) will take 416 days at this rate. And if you're using 100 watts, during that time you will spend $124.61 in electricity, as well as not being able to use your device for something else, plus the noise and heat from your device. For $124, on Moonshot's official Kimi K3 API rates ($0.30 per 1M cached input, $3 per 1M fresh input, $15 per 1M fresh output), you can purchase 42 million fresh-input tokens, or 8.3 million generated output tokens, in whatever mix you want. So what you get is 80x more expensive and you wait 416 days to get it.
- mring33621 2mo agoHow many is that in tokens per Scaramucci?
- fhars 1mo agoIt is a bit more than four tokens per millifortnight.
- exe34 1mo agoHah I was looking for it and couldn't work out how many years/token. 292s is pretty good.
- aaroninsf 1mo agoWow, you could do a lot at 292 tokens a sec—oh. I have all praise for those taking this on and in my idiom would call it *the lord's work." The image I reliably summon to mind is that compilation video showing the progress of Boston Dynamics bots. The curve between technically functional, to comically slow, to too slow for "real" work, on to, OMFG, may prove a (rough) curve. It's work like this that moves things forward.
- root-parent 1mo agoIt matches my coding speed...its ok.
- killingtime74 1mo agoit takes you almost 5 mins to write at most 1 word?
- weberer 1mo agoWhen averaged over 24 hours, probably.
- root-parent 1mo agoIf you look at examples of complex code and based on a public NASA dataset: https://raw.githubusercontent.com/timm/ourmine/refs/heads/master/our/arffs/effest/nasa93.arff https://raw.githubusercontent.com/timm/ourmine/refs/heads/ma... that has 93 NASA software projects, you are looking at less than 9 LOC per engineer/day 9 LOC per engineer/day...yes its about 5 to 6 min per token. Do you do better?
- killingtime74 1mo agoI write more but I'm no NASA engineer
- xg15 1mo agoAt that point, how does this compare with simply running the model on the CPU?
- sixothree 1mo agoNot an answer to your question, but maybe this has more info? I think with these optimizations and quants, it compares negatively. But can these optimizations be applied to models you want? Another question. https://news.ycombinator.com/item?id=48353348 https://news.ycombinator.com/item?id=48353348
- p91paul 1mo agoif you take a model that requires 200GB of VRAM and you run it on the CPU, it requires 200GB of RAM instead. Still unfeasible on consumer hardware. With this approach you can easily do it on 12GB or less of either RAM of VRAM, at several seconds per token instead of tokens per seconds. Very unusable, but certainly interesting!
- xg15 1mo agoIf token speed doesn't count, you can have 200GB "RAM" in swap space. But I agree, what the OP does is a lot more efficient than this.
- malshe 1mo agoI hope I'm not the only one who misread it as 292 tokens/s and got excited momentarily
- awesomeusername 1mo agoMost definitely not. Weeps...
- senectus1 1mo agohaha its something i look out for now. having seen a few recent stories like this.
- 361994752 1mo agoI thought I met some huge breakthrough on computer science until I saw your comment
- xlayn 1mo agoI have see a lot of "run <gigantic llm model name here> using cpu" or using 1gb of ram... and the short answer is sure, you can run it in 1mb of ram, or in a 286, it will just take a couple of years to produce the first token. the canary in the coal that this has not change is nvidia share price.
- k1e 1mo agoAre there benchmarks for smaller models?