6 ms·
25L Portable NV-linked Dual 3090 LLM Rig
- tensorlibb 1y agoI'm a huge fan of OpenRouter and their interface for solid LLM's but I recently jumped into fine tuning / modifying my own vision models for FPV drone detection (just for fun) and my daily workstation and it's 2080 just wasn't good enough. Even in 2025 it's cool how solid a setup dual 3090's still are. nvlink is an absolute must but it's incredibly powerful. I'm able to run the latest Mistral thinking models and relatively powerful yolo based VLM's like the ones RoboFlow is based on. Curious if anyone else is still using 3090's or has feedback for scaling up to 4-6 3090s. Thanks everyone ;)
- deleted 1y ago[deleted]
- CraigJPerry 1y agoif it's just for detection would audio not be cheaper to process? I'm imagining a cluster of directional microphones, and then i don't know if it's better to perform some sort of band pass filtering first since it's so computationally cheap or whether it's better to just feed everything into the model directly. No idea. I guess my first thought was just sounds from a drone likely is detectable reliably at a greater distance than visual, they're so small and a 180 degree by 180 degree hemisphere of pixels is a lot to process. Fun problem either wayway.
- fxtentacle 1y agoThe 3090 are a sweet spot for training. It’s the first generation with seriously fast VRAM. And it’s the last generation before Nvidia blocked NVlink. If you need to copy parameters between GPUs during training, the 3090 can be up to 70% faster than 4090 or 5090. Because the latter two are limited by PCI express bandwidth.
- jacquesm 1y agoTo be fair though, the 4090 and 5090 are much easier capable of saturating PCI express than the 3090 is, even at 4 lanes per card the 3090 rarely manages to saturate the links, it still handsomely pays off to split down to 4 lanes and add more cards. I used: https://c-payne.com/ https://c-payne.com/ Very high quality and manageable prices.
- ericdotlee 1y agoI've purchase 16 of these - cpayne is great! Hope he finds a US distributor to help with tariffs a bit!
- jacquesm 1y agoWhat blew me away is the quality and price point of what obviously can't be a very high volume product. This guy makes amazing stuff.
- jacquesm 1y agoI've built a rig with 14 of them. NVLink is not 'an absolute must', it can be useful depending on the model and the application software you use and whether you're training or inferring. The most important figure is the power consumed per token generated. You can optimize for that and get to a reasonably efficient system, or you can maximize token generation speed and end up with two times the power consumption for very little gain. You also will likely need to have a way to get rid of excess heat and all those fans get loud. I stuck the system in my garage, that made the noise much more manageable.
- breakds 1y agoI am curious about the setup of 14 GPUs - what kind of platform (motherboard) do you use to support so many PCIe lanes? And do you even have a chassis? Is it rack-mounted? Thanks!
- jacquesm 1y agoI used a large supermicro server chassis, a dual Xeon motherboard with 7 8 lane PCI Express slots, all the ram it would take (bought second hand), splitters, four massive powersupplies. I extended the server chassis with aluminum angle riveted onto the base. It could be rack mounted but I'd hate to be the person lifting it in. The 3090s were a mix, 10 of the same type (small, and with blower style fans on them) and 4 much larger ones that were kind of hard to accommodate (much wider and longer). I've linked to the splitter board manufacturer in another comment in this thread. That's the 'hard to get' component but once you have those and good cables to go with them the remaining setup problems are mostly power and heat management.
- breakds 1y agoThanks that is very inspiring. I thought there are no blower type consumer GPUs, but apparently they exist!
- jacquesm 1y agoI got them second hand off some bitcoin mining guy. https://www.tomshardware.com/news/asus-blower-rtx3090 https://www.tomshardware.com/news/asus-blower-rtx3090 Is the model that I have.
- vladgur 1y agoI am exploring options just for fun. a used 3090 is around $900 on ebay. a used rtx 6000 ADA is around $5k 4 3090s are slower at inference and worse at training than 1 rtx 6000. 4x3090 would consume 1400W at load. Rtx 6000 would consume 300W at load. If you god forbid live in California and your power averages 45 cents per kwh, 4x3090 would be $1500+ more per year to operate than a single RTX 6000[0] [0] Back of the napkin/ChatGPT calculation of running the GPU at load for 8 hours per day. Note: I own a pc with a 3090, but if i had to build an AI training workstation, i would seriously consider cost to operate and resale value(per component).
- logicallee 1y ago>I am exploring options just for fun. Since you're exploring options just for fun, out of curiosity, would you rent it out whenever you're not using it yourself, so it's not just sitting idle? (Could be noisy and loud). You'd be able to use your computer for other work at the same time and stop whenever you wanted to use it yourself.
- vladgur 1y agoIt depends. At my electricity cost, 1 hour of 3090 or 1 hour of Rtx 6000 would cost the same 0.45 Just checked vast.ai. I will be losing money with 3090 at my electricity cost and making a tiny bit with rtx 6000. Like with boats it’s probably better to rent GPUs then buy them
- justinclift 1y agoWould a solar panel setup be an option for fixing that? :)
- logicallee 1y ago(you should also be compensated for the noise and inconvenience from it, not only electricity.) It sounds like you might rent it out if the rental price were higher.
- supermatt 1y ago
- AJRF 1y agoYou really don't need NVLink, you won't saturate the PCIe lanes on a modern motherboard with dual 3090s. Tim Dettmers amazing GPU blog post posits NVLink doesn't start to become useful until you are at 128+ GPUs https://timdettmers.com/2023/01/30/which-gpu-for-deep-learning/#What_is_NVLink_and_is_it_useful https://timdettmers.com/2023/01/30/which-gpu-for-deep-learni...
- XCSme 1y agoI bought a 2nd 3090 2 years ago for like 800eur, still a good price even today I think. It's in my main workstation, and my idea was to always have Ollama running locally. The problem is that once I have a (large-ish) model running, all my VRAM is almost full and GPU struggles to do things like playing back a YouTube video. Lately I haven't used local AI much, also because I stopped using any coding AIs (as they wasted more time than they saved), I stopped doing local image generations (the AI image generation hype is going down), and for quick questions I just ask ChatGPT, mostly because I also often use web search and other tools, which are quicker on their platform.
- lifeinthevoid 1y agoI run my desktop environment on the iGPU and the AI stuff on the dGPUs.
- XCSme 1y agoThat's a real good point! Unfortuatenly, my CPU (5900x) doesn't have an iGPU. The last 5 years iGPU got a bit out of trend. Now maybe they actually make a lot of sense, as there is a clear use-case which involves having dedicated GPU always in-use which is not gaming (and gaming is different, cause you don't often multi-task while gaming). I do expect to see a surge in iGPU popularity, or maybe a software improvement to allow having a model always available without constantly hogging the VRAM.
- XCSme 1y agoPS: I thought Ollama had a way to use RAM instead of VRAM (?) to keep the model active when not in use, but in my experience that didn't solve the problem.
- username12349 1y agototal cost?
- bigiain 1y agoIt says $3090 (maybe easy to miss since it also talks about RTX 3090s?)
- jszymborski 1y agoIt's written quite large on the page, just over 3K
- jszymborski 1y agoI just don't get why the RTX 4090 is still so expensive on the used market. New Rtx 5090s are almost as expensive!
- tayo42 1y agoAre these just for ai now? Or are games pushing video cards that much?
- Our_Benefactors 1y ago4090 is a great gaming card, the spiritual successor to the 1080. It will be viable for years and years.
- renewiltord 1y agoThey're dropping. I'm trying to offload 8x 4090s and I'll average $1500 I think.
- robotswantdata 1y ago“Easy” to mod to 48gb
- suladead 1y agoI built pretty much this exact rig myself, but now it's gathering dust, any other uses for this rather than localLLMS
- DaSHacka 1y agovidya
- winkelmann 1y ago3D rendering and fluid simulation stuff could be interesting.
- Tepix 1y agoSell it? There are people who want a rig like this.
- asimovDev 1y agoPlay DnD by yourself with Llama as a DM
- robotswantdata 1y agoHeating
- ProllyInfamous 1y agoI use an older machine/GPU for wintertime heating, mining Monero (xmrig). Should one get lucky and guess the next valid block, that pays the entire month's electricity — since an electric space heater would already be consuming the exact same amount of kWH as this GPU, there is no "negative cost" to operate. This machine/GPU used to be my main workhorse, and still has ollama3.2 available — but even with HBM, 8GB of VRAM isn't really relevant in LLM-land.
- thiago_fm 1y agoPlaying games, it has a good graphics card
- dotnet00 1y ago
- AJRF 1y agoThose GPUs are so close to each other, doesn’t the heat cause instability?
- deevus 1y agoI’m really interested in this space from an AI sovereignty pov. Is it feasible for SMB/SME to use a box like in the article to get offline analysis of their data? It doesn’t have the worry of sending it off to the cloud. I wanted to speak with businesses in my local area but no one took me up on it.
- ang_cire 1y agoYes, this is absolutely doable, and many companies are rolling their own ML models (I work with a MedTech company that does, in fact). LLMs are a little more involved, and you'd probably want something beefier than this (maybe a Framework Desktop cluster, if you're not wanting to get into rackmount stuff), but it's definitely feasible for companies to have their own offline LLMs and ML models.
- deleted 1y ago[deleted]
- Deepen5 1y agois it that easy to get started?
- AJRF 1y ago> WARNING - these components don't fit if you try to copy this build. The bottom GPU is resting on the Arctic p12 slim fans at the bottom of the case and pushing up on the GPU. I built a dual 3090 rig, and this point was why I spent a long time looking for a case where the GPU's could fit side by side with a little gap for airflow I eventually went with a SilverStone GD11 HTPC which is a PC case for building a media centre, but it's huge inside, has a front fan that takes up 75% of width of the case and also allows the GPUs to stand up right so they don't sag and pull on their thin metal supports. Highly recommend for a dual GPU build! If you can get dual 5090s instead of 3090s (good luck!) you'd even be able to get "good" airflow in this case.
- Tepix 1y agoOK, here's my quick critique of the article (having built a similar AM4-based system in 2023 for 2300€): 1) [I thought] The page is blocking cut & paste. Super annoying! 2) The exact mainboard is not specified exactly. There are 4 different boards called "ASUS ROG Strix X670E Gaming" and some of them only have one PCIe x16 slot. None of them can do PCIe x8 when using two GPUs. 3) The shopping link for the mainboard leads to the "ASUS ROG Strix X670E-E Gaming" model. This model can use the 2nd PCIe 5.0 port at only x4 speeds. The RTX 3090 can only do PCIe 4.0 of course so it will run at PCIe 4.0 x4. If you choose a desktop mainboard for having two GPUs, make sure it can run at PCIe x8 speeds when using both GPU slots! Having NVLink between the GPUs is not a replacement for having a fast connection between the CPU+RAM and the GPU and its VRAM. 4) Despite having a last-modified date of September 22nd, he is using his rig mostly with rather outdated or small LLMs and his benchmarks do not mention their quantization, which makes them useless. Also they seem not to be benchmarks at all, but "estimates". Perhaps the headline should be changed to reflect this?
- rtaylorgarlock 1y ago[flagged]
- prmoustache 1y agoI have js enabled and I can copy text on this page.
- Tepix 1y agoIn general I can too, but try copying items from the "key specifications". Or perhaps I just had the impression because you can't mark text because I can't tell which text is marked and which isn't when marking text under "Key Specifications". Mea culpa.
- prmoustache 1y agoyeah the selection is dark grey over black so it is not super visible but you can copy text.
- whoami730 1y agoAnybody else getting 403 Forbidden error?
- cl3misch 1y ago> The workplace of the coworker I built this for is truly offline, with no potential for LAN or wifi, so to download new models and update the system periodically I need to go pick it up from him and take it home. I'm surprised that a "truly offline" workplace allows servers to be taken home and being connected to the internet.
- bedstefar 1y agoI worked in the Arctic for the better part of a decade. There's Starlink now, but I've been TRULY OFFLINE for weeks (with plenty diesel generated power) as recently as 2018. Technically we could use Iridium at like $10 per MB, but my full Wikipedia mirror (+ Debian/Ubuntu packages, PyPI etc) did come in handy more than once. I know some Antarctic research stations (like McMurdo for example) still have connectivity restrictions depending on time-of-day, and I wouldn't be surprised if they also had mirrors of these sort of things, and/or dual-3090 rigs for llama.cpp in the off hours.
- arkj 1y agoThe link is down with 403 error.
- lifeinthevoid 1y agoI built a similar system, meanwhile I've sold one of the RTX 3090's. Local inference is fun and feels liberating, but it's also slow, and once I was used to the immense power of the giant hosted models, the fun quickly disappeared. I've kept a single GPU to still be able to play a bit with light local models, but not anymore for serious use.
- nenenejej 1y agoGraphics cards are so expensive (list price) they are cheap (no depreciation liquid market)
- Our_Benefactors 1y agoDid you really claim GPUs have zero depreciation? That’s obviously false.
- deleted 1y ago[deleted]
- imiric 1y agoI have a similar setup as the author with 2x 3090s. The issue is not that it's slow. 20-30 tk/s is perfectly acceptable to me. The issue is that the quality of the models that I'm able to self-host pales in comparison to that of SOTA hosted models. They hallucinate more, don't follow prompts as well, and simply generate overall worse quality content. These are issues that plague all "AI" models, but they are particularly evident on open weights ones. Maybe this is less noticeable on behemoth 100B+ parameter models, but to run those I would need to invest much more into this hobby than I'm willing to do. I still run inference locally for simple one-off tasks. But for anything more sophisticated, hosted models are unfortunately required.
- mycall 1y ago> 20-30 tk/s or ~2.2M tk/day. This is how we should be thinking about it imho.
- kapone 1y agoI'm failing to see the point of this article? I mean, people have been building dual GPU workstations for a long long time. What's so special about this one?
- loudmax 1y agoThere was an interesting post to r/LocalLLaMA yesterday from someone running inference mostly on CPU: https://carteakey.dev/optimizing%20gpt-oss-120b-local%20inference/ https://carteakey.dev/optimizing%20gpt-oss-120b-local%20infe... One of the observations is how much difference memory speed and bandwidth makes, even for CPU inference. Obviously a CPU isn't going to match a GPU for inference speed, but it's an affordable way to run much larger models than you can fit in 24GB or even 48GB of VRAM. If you do run inference on a CPU, you might benefit from some of the same memory optimizations made by gamers: favoring low-latency overclocked RAM.
- mistercheph 1y agoOutside of prompt processing, the only reason GPU's are better than CPU's for inference is memory bandwidth, the performance of apple M* devices at inference is a consequence of this, not of their UMA.
- JKCalhoun 1y agoI love how the prices for various Llama builds are all over the map on this site. Oh look, here's one for $43K: https://www.llamabuilds.ai/build/a16zs-personal-ai-workstation-with-four-nvidia-rtx-6000-pro-blackwell-max-q-gpus https://www.llamabuilds.ai/build/a16zs-personal-ai-workstati...
- gbolcer 1y agoI was going to say you need an extension cable. My first dual 3090 build I had three issues. First was the pcie extension wouldn't support gen4, so I had to change to gen3 in the bios. Second issue was that depending on which slot, you couldn't get x16/x16 and it would drop to x16/x8 unless you had it configured right. Third, I finally gave up and just had the card resting first inside the case and then outside which if fan kicks up, it'll jiggle around, so I had to make some makeshift holder to keep the card sitting there.
- logicziller 1y agoI get a 403 error.
- tomhow 1y agoURL updated from https://www.llamabuilds.ai/build/portable-25l-nvlinked-dual-3090-llm-rig https://www.llamabuilds.ai/build/portable-25l-nvlinked-dual-..., which points to this.