9 ms·
Speculative Decoding in vLLM on AMD GPUs
- intothemild 10d agoWhilst this is an excellent post from vLLM, one of the truly baffling things from either their team or AMDs team, is how much the workstation grade AMD r9700 has been ignored. Stock vLLM runs so slowly on these cards compared with vLLM forks like Radiance. Going from say 20-30t/s gen, to 150-200t/s Most of AMD/vLLM work seems to be around their data centre cards, or the AMD AI Halo/Ryzen and ignores the R9700 AI Pro. Really wish this would change.
- minraws 10d agoI don't see a reason why it should AMD doesn't care about lower end prosumers atm. They might in the future but future is in the future ofc Edit: to be clear I think it's ridiculous they don't but from a company's stand point it doesn't make much sense
- websap 10d agoYeah, as a business AMD should first care about getting their DC grade hardware optimized for inference workloads. It's unfortunate that most of HN discussion has devolved to me-ish.
- mrhenio 10d ago[flagged]
- _factor 10d agoThen they should stop selling hardware they don’t plan to support. Me-ish when you spend $1,500 on a piece of hardware is completely acceptable.
- brookst 10d agoNever buy hardware based on expectations of future features, especially if there’s no promise from the vendor.
- dist-epoch 10d agoGeorge Hotz in June 2023: > I have had direct contact with members of the AMD RTG team and I was disgusted to find that AMD doesn't even provide them with hardware to work on. The developer I was working with had to buy the GPU he was writing drivers for.
- da-x 10d agoI think this has changed since then, their policies toward open source improved (e.g ROCm).
- dist-epoch 10d agoThe market says the problem is still there. An NVIDIA consumer GPU sells for 50+% or more than an equivalent AMD GPU. Because people are buying NVIDIA GPUs to run local models instead of AMD ones. I did the same thing, I paid 50% more to get an 5070 Ti instead of the equivalent AMD. This is probably good for gamers, AMD GPUs are not price inflating to the same degree as NVIDIA, because they are bad at LLMs. > That was the reason for comparing them in the first place: based on performance, they are direct competitors, or at least they are meant to be. However, as things stand today, there is a massive price divide between the two, with the RTX Ti GPU now commanding a premium of more than 50%. https://www.techspot.com/review/3168-geforce-rtx-5070-vs-radeon-rx-9070-xt/ https://www.techspot.com/review/3168-geforce-rtx-5070-vs-rad...
- androiddrew 10d agoThankfully, there are still people willing to jump on the R9700 bandwagon and get a vLLM fork working. If you have an RDNA4 card check out https://hub.docker.com/r/stilldeadcode/vllm-radiance https://hub.docker.com/r/stilldeadcode/vllm-radiance
- intothemild 10d agoDeadcode is currently working on INT4 right now on his R4D kernel. The MXFP4 fork is excellent too. Its my daily driver right now. https://codeberg.org/ggz14/radiance-vllm-mxfp4 https://codeberg.org/ggz14/radiance-vllm-mxfp4 Also has PARO quant support there too (early stage) Also speedups in both repos for 4x R9700s
- karmakaze 10d agoThanks! Didn't expect to see this here. Exactly what I needed to run Qwen3.8-27B-Quark-AWQ-MXFP4-native.gguf as well as other experiments on one or 2x R9700's (I hope).
- nicce 10d agoThat MXFP4 is an excellent project. But I have difficulties on reading that README. Is it intentionally generated like that with LLMs?
- intothemild 10d agoFeed the setup and run scripts to your LLM.
- karmakaze 10d agoThe way it does tensor splitting without all-reduce cost over PCIe bus wasn't something I thought was possible. What kind of performance are you getting with 4x R9700s--what do you do with all the VRAM (batching, concurrent requests, etc)?
- intothemild 10d ago
- roenxi 10d ago> ...one of the truly baffling things from either their team or AMDs team, is how much the workstation grade AMD r9700 has been ignored. It makes a huge amount of sense after considering AMD's approach to graphics cards from around 2010 to 2025. They just didn't see graphics cards as viable compute platform and many who made the mistake of believing that good specs would translate into in-practice performance got badly burned. I'd have been involved in the AI boom but for an expensive AMD graphics card, I'm not going to forget that for a while. George Hotz was interesting as a public example, but I think his story probably repeated a few times outside the public eye. People tried to make AMD work and ended up the worse for it. People who had an interest in using AMD cards to get things done are probably by and large waiting for a new generation of hopefuls to prove this time is different. The mutterings out of AMD are promising, but that isn't persuasive enough given the scale of the failures.
- hgoel 10d agoGCN was such a promising compute architecture, AMD even pioneered stuff like async compute and compute shader heavy rendering pipelines, only to never seriously go beyond that on consumer gear. I agree with your assessment that the story of supporting the competition, only to get burned, has repeated many times with AMD outside the public eye. It's why I don't put much stock in claims that things work great as long as specific flags are used.
- lrvick 10d agoI have 4x r9700s as my coding daily drivers running qwen 3.8 27b at 80tps each. Zero complaints. Especially at $1200 each.
- SomeHacker44 10d agoPlease share what operating system and model runtime you use? I have two and don't get close to that with AMD's own Lemonade. Thanks!
- lrvick 10d agoLemonade is one of the worst performing options. Run any modern Linux distro and ask your current LLM to setup llama.cpp with dflash2 for you as an unprivileged container running from a systemd user unit. Obviously only on a system you do not trust at all.
- Roark66 10d agoI'd rather buy two used rtx3090 than a single r9700 AI pro. More VRAM (some wasted due to it being non continuous), more RAM bandwidth, more aggregate compute. Only if AMD made a card like this with 48G+ I'd consider it. Also these 20-30t/s jumping to 150-200... Watch out for the massaged numbers coming from vendors. I believe Intel has claimed something like 1400tok/s (generation! Not prefill) of Qwen3.6-moe on Arc b70. I was actually very interested in this so I checked the details. Turns out it was 200 simultaneous users running the same 1024 token prompt :D so all the experts got maximum parallelism. How often are you going to run 200 parallel sessions with a tiny context and same prompt running at 7tok/s. Based on how much my rtx3090 is getting on a single user (150tok/s) I'm estimating b70 to probably get less than that. Sadly nvidia is king now. Also, most of us already have nvidia cards and no inference software supports mixing let's say nvidia, Intel and amd cards in inference of one model.
- nicce 10d ago2x RTX 3090 is not enough for proper use. ideal is 64GB+ so that you get proper cache and 256k context with good enough models. (e.g. Qwen 3.8 27B with MXFP4). And that is tight already. You would need 3x RTX 3090 - but then the PCIe bandwidth comes an issue if you really want tensor parallelism with three cards. 3x PCIe5 x16 is not cheap with direct CPU access. So surprisingly, 2x r9700 starts be a nice deal. > Also these 20-30t/s jumping to 150-200... Watch out for the massaged numbers coming from vendors. Well, luckily these are not vendor numbers. Prefill also scales almost linearly with the amount of GPUs.
- formerly_proven 10d agoYou don't need PCIe 5.0 x16 since RTX 30 are not PCIe 5.0 to begin with.
- nicce 10d agoWell, that makes them just even slower then
- 10d ago
- 3abiton 9d agoHonestly a big part of this is AMD's lackluster strategy to GPU software. To say it's lacking is understatement. At least for non-data center gpus.
- hn45e7pbij 10d ago[dead]
- flufluflufluffy 10d agoThis is way outside my expertise so might be a dumb question, but how does the target model verify candidate tokens? Naively, I would assume it must perform its normal auto regressive decoding to know what the “correct” token is in order to have something to compare the candidate token with. But obviously that would defeat the purpose of speculative decoding so there must be some other way. Also, what is the difference between “target model” and “target-model,” if any? I feel like half the instances of that phrase included the hyphen and half didn’t.
- lucrbvi 10d agoThe target model is the original LLM that is large and expensive. It can verify candidate tokens in a single forward pass. It means you give all the context + candidate tokens that passes in parallel in the backbone, then you pass the language head (a matmul transformation to produce the token distribution) on all the candidate tokens and you can keep or drop tokens based on how many "quality" you want.
- deleted 10d ago[deleted]
- daemonologist 10d agoYou're correct that it needs to run the full model to "verify" a token, but LLM inference benefits from batching - it's much faster to run twice in parallel than sequentially. So the draft model runs ~2-5 tokens ahead, and the full model then runs ~3-6 batches in parallel using those tokens, and can skip ahead by however many results match. If you have some other source of parallel data (lots of users, many separate tasks) then speculative decoding might not provide any benefit.
- porridgeraisin 10d ago> it must perform it's normal autoregressive decoding to know what is the correct token in order to have something to compare with Correct except for the word "autoregressive". When you have to verify a sequence of tokens (which were autoregressively generated by the cheap model), you can do each token in parallel. This amortizes the cost of loading the weights from vram to the processors (the primary cost in LLM serving) across those tokens. Cost here is wall clock time, as well as power. The autoregressive decoding that generates this batch of tokens is delegated to the cheaper model where the cost of loading the weights is lower and so not amortizing it is fine. Verification means, how close is each token in this sequence to the one I would have output. You keep the longest prefix that is close enough for your liking.
- jeanmichelselli 10d agoIn my opinion, LLMs are one of the most fascinating result coming from machine learning in recent years. Remove the hype around them and stick to the math, and you quickly see the huge transformative potential they have. It's great to see a lot of research being done in that direction, I wish it would mainly come from academia though..
- suprjami 10d agoStrange comment. "If you remove the hype about how transformative they are, they really are transformative". No. If you remove the hype about how transformative they are, you're left with what they actually are: a sometimes mildly useful tinker toy.
- deleted 10d ago[deleted]
- myuzio 10d agoVery true, I see it for example when my co-workers apply DRY without actually thinking about the complexity it might cause. It objectively makes code smaller, but often comes with complexity cost.
- foota 10d agoYou got an article off by one error, I think you meant to post on https://news.ycombinator.com/item?id=49558685 https://news.ycombinator.com/item?id=49558685 :)
- ThiraSoft 10d ago[dead]
- quietraster 10d agonice to see amd getting first-class vllm support. how do the acceptance rates compare to nvidia on the same models?
- InvectusXIV 9d ago[flagged]
- InvectusXIV 9d ago[flagged]