4 ms·
I'd rather buy two used rtx3090 than a single r9700 AI pro. More VRAM (some wasted due to it being non continuous), more RAM bandwidth, more aggregate compute.
by Roark66 9d ago
I'd rather buy two used rtx3090 than a single r9700 AI pro. More VRAM (some wasted due to it being non continuous), more RAM bandwidth, more aggregate compute.
Only if AMD made a card like this with 48G+ I'd consider it.
Also these 20-30t/s jumping to 150-200... Watch out for the massaged numbers coming from vendors.
I believe Intel has claimed something like 1400tok/s (generation! Not prefill) of Qwen3.6-moe on Arc b70.
I was actually very interested in this so I checked the details. Turns out it was 200 simultaneous users running the same 1024 token prompt :D so all the experts got maximum parallelism.
How often are you going to run 200 parallel sessions with a tiny context and same prompt running at 7tok/s.
Based on how much my rtx3090 is getting on a single user (150tok/s) I'm estimating b70 to probably get less than that.
Sadly nvidia is king now.
Also, most of us already have nvidia cards and no inference software supports mixing let's say nvidia, Intel and amd cards in inference of one model.
- nicce 9d ago2x RTX 3090 is not enough for proper use. ideal is 64GB+ so that you get proper cache and 256k context with good enough models. (e.g. Qwen 3.8 27B with MXFP4). And that is tight already. You would need 3x RTX 3090 - but then the PCIe bandwidth comes an issue if you really want tensor parallelism with three cards. 3x PCIe5 x16 is not cheap with direct CPU access. So surprisingly, 2x r9700 starts be a nice deal. > Also these 20-30t/s jumping to 150-200... Watch out for the massaged numbers coming from vendors. Well, luckily these are not vendor numbers. Prefill also scales almost linearly with the amount of GPUs.
- formerly_proven 9d agoYou don't need PCIe 5.0 x16 since RTX 30 are not PCIe 5.0 to begin with.
- nicce 9d agoWell, that makes them just even slower then
- snovv_crash 9d agoThe whole point of dual 3090 over other cards is the nvlink support. At that point the pcie doesn't really matter.
- nicce 9d agoOops
- latentsea 9d ago> 2x RTX 3090 is not enough for proper use I don't see how. Even on 32GB you can run Q6_K_XL quant with MTP at 200k context k=q8_0, v=q5_1. So 48GB VRAM is good enough to run Q8 at long context. Also with tings like ninfer and it's various forks I'm seeing people get very good performance out of Qwen3.8 models on all sorts of NVIDIA cards.