12 ms·
Serving AI from the Basement – 192GB of VRAM Setup
- choilive 2y agoI have a similar setup in my basement! Although its multiple nodes, with a total of 16x3090s. Also needed to install a 30A 240V circuit as well.
- lvl155 2y agoThat last part is often overlooked. This is also why sometimes it’s just not worth going local especially if you don’t need all that compute power beyond a few days.
- buildbot 2y ago100% agree, anything beyond 4x gpu’s is getting into the very annoying to power territory and makes the cloud very attractive. I already can trip a 15A circuit on 115v power with just 3x4090s and a SPR-X cpu. It also costs a lot to power. In the summer, 2x more than you expect, because unless it’s outside, you need cool 1000+ watts of extra heat with your AC. All that together and runpod starts to look very tempting!
- choilive 2y agoGetting that circuit installed was pretty cheap likely because its in an unfinished and unconditioned basement. The basement stays comfortable even during the summer. The heat does seem to work its way into the rest of the house but the additional cooling load is only about 20% more than usual. It lowers the heating cost about the same amount during the winter so it works itself out.
- buildbot 2y agoYeah location of your place, climate, and placement of the server in the house will affect this a lot. I'm on the top story of a building, even in the winter I rarely need to turn on my heat, just getting by on the waste heat of the rest of the building. My assortment of machines will easily keep the living room at 25C+ with a window open unless it's below 10C out! If I could keep the servers in the cool parking garage, I'd save a lot of money... Getting a circuit put is also much more difficult in a shared building... Runpod has 3090s for .43 per hour! .22 spot. If your power costs .3$ per kWh, and you need to spend _another_ .3$ per kWh in cooling, say if you live in apartment in the Bay Area and it's summer, that's ~48 days to equal the cost of 30 days on runpod. So you are still saving some money, though much less than you might think and possibly spending more than spot instances!
- choilive 2y agoYeah I worked it out and I am saving ~75% vs running my inference workloads on RunPod. $650/mo in electricity vs $2,500/mo to do the same thing on RunPod. Been in near continuous operation over 9 months, so the system has basically paid for itself with the savings.
- XMasterrrr 2y agoHey guys, this is something I have been intending to share here for a while. This setup took me some time to plan and put together, and then some more time to explore the software part of things and the possibilities that came with it. Part of the main reason I built this was data privacy, I do not want to hand over my private data to any company to further train their closed weight models; and given the recent drop in output quality on different platforms (ChatGPT, Claude, etc), I don't regret spending the money on this setup. I was also able to do a lot of cool things using this server by leveraging tensor parallelism and batch inference, generating synthetic data, and experimenting with finetuning models using my private data. I am currently building a model from scratch, mainly as a learning project, but I am also finding some cool things while doing so and if I can get around ironing out the kinks, I might release it and write a tutorial from my notes. So I finally had the time this weekend to get my blog up and running, and I am planning on following up this blog post with a series of posts on my learnings and findings. I am also open to topics and ideas to experiment with on this server and write about, so feel free to shoot your shot if you have ideas you want to experiment with and don't have the hardware, I am more than willing to do that on your behalf and sharing the findings Please let me know if you have any questions, my PMs are open, and you can also reach me on any of the socials I have posted on my website.
- nrp 2y agoHow are you finding 2b/3b quantized llama 405B? Is it behaving better than 8b or 16b llama 70B?
- nullindividual 2y agoDo you run this 24/7? What is your cost of electricity per kilowatt hour and what is the cost of this setup per month?
- michaelt 2y agoI have a much smaller setup than the author - a quarter the GPUs and RAM - and I was surprised to find it draws 300W at idle
- freeqaz 2y agoHow much do the NVLinks help in this case? Do you have a rough estimate of how much this cost? I'm curious since I just built my own 2x 3090 rig and I wondered about going EPYC for the potential to have more cards (stuck with AM5 for cheapness though). All in all I spent about $3500 for everything. I'm guessing this is closer to $12-15k? CPU is around $800 on eBay.
- lvl155 2y agoMy reason for going Epyc was for Pcie lanes and cheaper enterprise SSDs via U.3/2. With AM5, you tap out the lanes with dual GPUs. Threadripper is preferable but Epyc is about 1/2 of the price or even better if you go last gen.
- Eisenstein 2y agoWhy do you need such high cross card bandwidth for inference? Are you hosting for a lot of users at once?
- oceanplexian 2y agoThe Epyc boards make things way easier (I have 4 epyc boards of various generations) because they have loads of x16 slots and you’re not screwing around with bifurcation and sketchy PCI splitters. Another oft-forgotten item that consumes lanes is 25 or 40Gb NICs which you might fine you want if you’re pushing big model files around to other machines or storage.
- darknoon 2y agoI tried this w/ AM5, but realized that despite there theoretically being enough lanes for dual x16 PCI-e 4.0 GPUs, I couldn't find any motherboards that are actually configured this way, since dual-GPU is dead in consumer for gaming.
- Tepix 2y agoI built this in early 2023 out of used parts and ended up with a cost of 2300€ for AM4/128GB/2x3090 @ PCIe 4.0x8 +nvLink
- rvnx 2y agoYou could just buy a Mac Studio for 6500 USD, have 192 GB of unified RAM and have way less power consumption.
- flemhans 2y agoAre people running llama 3.1 405B on them?
- rspoerri 2y agoI'm running 70B models (usually in q4 .. q5_k_m, but possible to q6) on my 96Gbyte Macbook Pro with M2-Max (12 cpu cores, 38 gpu cores). This also leaves me with plenty of ram for other purposes. I'm currently using reflection:70b_q4 which does a very good job in my opinion. It generates with 5.5 tokens/s for the response, which is just about my reading speed. edit: I usually dont run larger models (q6) because of the speed. I'd guess a 405B model would just be awfully slow.
- throwthrowuknow 2y agoNot going to work for training from scratch which is what the author is doing.
- rspoerri 2y ago192GByte of RAM are not enough to train 405B models. Reflection 70B requires 140GByte of RAM in fp16, 405 would need ~810Gbyte of RAM.
- throwthrowuknow 2y agoPretty sure he said he’s inferencing llama3 405 and training his own custom model from scratch. He didn’t say how big his custom model will be.
- angoragoats 2y agoYou could for sure, but the nVidia setup described in this article would be many times faster at inference. So it’s a tradeoff between power consumption and performance. Also, modern GPUs are surprisingly good at throttling their power usage when not actively in use, just like CPUs. So while you need 3kW+ worth of PSU for an 8x3090 setup, it’s not going to be using anywhere near 3kW of power on average, unless you’re literally using the LLM 24x7.
- 3eb7988a1663 2y agoWhat is the power draw under load/idle? Does it noticeably increase the room temperature? Given the surroundings (aka the huge pile of boxes behind the setup), curious if you could get away with just a couple of box fans instead of the array of case fans. Are you intending to use the capacity all for yourself or rent it out to others?
- throwpoaster 2y agoDid you write this with the LLM running on the rig?
- emptiestplace 2y agoDoes this post actually seem LLM generated to you?
- throwpoaster 2y agoIt reads like an LLM draft with a human edit, yes.
- cranberryturkey 2y agothis is why we need an actual AI blockchain, so we can donate GPU and earn rewards for the p2p api calls using the distributed model.
- walterbell 2y ago> donate GPU .. earn rewards Is a blockchain needed to sell unused GPU capacity?
- bschmidt1 2y agoThat's actually interesting. While crypto GPU mining is "purposeless" or arbitrary, would be way cooler if to GPU mine meant to chunk through computing tasks in a free/open queue (blockchain). Eventually there could be some tipping point where networks are fast enough and there are enough hosting participants it could be like a worldwide/free computing platform - not just for AI for anything.
- yunohn 2y agoThis idea has been brought up tons of times by grifters aiming to pivot from Crypto to AI. The reason that GPUs are used for blockchains is to compute large numbers or proofs - which are truly useless but still verifiable so they can be distributed and rewarded. The free GPU compute idea misses this crucial point, so the blockchain part is (still) useless unless your aim is to waste GPU compute instead. IRL all you need is a simple platform to pay and schedule jobs on other’s GPUs.
- SmellTheGlove 2y agoI thought I was balling with my dual 3090 with nvlink. I haven’t quite yet figured out what to do with 48GB VRAM yet. I hope this guy posts updates.
- lxe 2y agoRun 70B LLM models of course
- thelastparadise 2y agoOr train a cute little baby llama.
- bogwog 2y agoAwesome! I've always wondered what something like this would look like for a home lab. I'm excited to see your benchmarks :)
- InsomniacL 2y agoWhen you moved in to your house, did you think you would finish a PC build with 192GB of VRAM before you would finish the plaster boarding?
- killingtime74 2y agoMaybe they removed it for better ventilation
- walterbell 2y agoAn adjacent project for 8 GPUs could convert used 4K monitors into a borderless mini-wall of pixels, for local video composition with rendered and/or AI-generated backgrounds, https://theasc.com/articles/the-mandalorian https://theasc.com/articles/the-mandalorian > the heir to rear projection — a dynamic, real-time, photo-real background played back on a massive LED video wall and ceiling, which not only provided the pixel-accurate representation of exotic background content, but was also rendered with correct camera positional data.. “We take objects that the art department have created and we employ photogrammetry on each item to get them into the game engine”
- renewiltord 2y agoI have a similar one with 4090s. Very cool. Yours is nicer than mine where I've let the 4090s rattle around a bit. I haven't had enough time to find a way to split inference which is what I'm most interested in. Yours is also much better with the 1600 W supply. I have a hodge podge.
- modeless 2y agoI wonder how the cost compares to a Tinybox. $25k for 6x 4090 or $15k for 6x 7900XTX. Of course that's the full package with power supplies, CPU, storage, cooling, assembly, shipping, etc. And a tested, known good hardware/software configuration which is crucial with this kind of thing.
- itomato 2y agoWith a rental option coming, it’s hard for me to imagine a more profitable way to use a node like that.
- Tepix 2y agoIf you merely want CUDA and lots of VRAM there‘s no reason to pick expensive 4090s over used 3090s
- halJordan 2y agoWell there is and it's called performance. You dont have to push your version of what an appropriate price/performance ratio is
- Tepix 2y agoWell, if you run out of VRAM, you drop off a performance cliff. That's a whole order of magnitude slower than just using a slower GPU but fitting everything into VRAM.
- angoragoats 2y agoYou can build a setup like in the OP for somewhere around $10k, depending on several factors, the most important of which are the price you source your GPUs at ($700 per 3090 is a reasonable going rate) and what CPU you choose (high core count, high frequency Epyc CPUs will cost more).
- maaaaattttt 2y agoLooking forward to reading this series. As a side note I’d love to find a chart/data on the cost performance ratio of open source models. And possibly then a $/ELO value (where $ is the cost to build and operate the machine and ELO kind of a proxy value for the average performance of the model)
- wkat4242 2y ago> And who knows, maybe someone will look back on my work and be like “haha, remember when we thought 192GB of VRAM was a lot?” I wonder if this will happen. It's already really hard to buy big HDDs for my NAS because nobody buys external drives anymore. So the pricing has gone up a lot for the prosumer. I expect something similar to happen to AI. The big cloud parties are all big leaders on LLMs and their goal is to keep us beholden to their cloud service. Cheap home hardware work serious capability is not something they're interested in. They want to keep it out of our reach so we can pay them rent and they can mine our data.
- thelastparadise 2y ago> It's already really hard to buy big HDDs for my NAS IME 20tb drives are easy to find. I don't think the clouds have access to bigger drives or anything. Similarly, we can buy 8x A100s, they're just fundamentally expensive whether you're a business or not. There doesn't seem to be any "wall" up like there used to be with proprietary hardware.
- wkat4242 2y agoThey are easy to find but extremely expensive. I used to pay below 200€ for a 14TB Seagate 8 years ago. That's now above 300. And the bigger ones are even more expensive. For me these prices are prohibitive. Just like the A100s are (though those are even more so of course). The problem is the common consumer relying on the cloud so these kind of products become niches and lose volume. Also, the cloud providers don't pay what we do for a GPU or HDD. They buy them by the ten thousands and get deep discounts. That's why the RRPs which we do pay are highly inflated.
- Dylan16807 2y agoWell if I look at Amazon I see a couple models of external 14TB for $190, and a brand new Exos 16TB for $230. Not too bad. Though personally I get much cheaper used drives and put them in RAID for a NAS. And they do have better sales.
- flixf 2y agoVery interesting! How are the 8 GPUs connected to the motherboard? Based on the article and the pictures, he doesn't appear to be using PCIe risers. I have a setup with 3 RTX 3090 GPUs and the PCIe risers are a huge source of pain and system crashes.
- Tepix 2y agoSo, how do you connect the 8th card if you have 7 PCIe 4.0 x16 slots available?
- manav 2y agoPCIe bifurcation - so splitting one of the x16 slots into two x8 or similar.
- metadat 2y agoWorth mentioning - this also cuts the available bandwidth to each card by 50%.
- angoragoats 2y agoWhile you're technically correct, assuming you're using PCIe 4.0 or higher, the performance difference between x8 and x16 is practically zero.
- metadat 2y agoIs this because the graphics cards are not using all available PCIe bandwidth? Or why?
- angoragoats 2y agoYes. In fact, even running a video card in a 4x slot (again, assuming PCIe 4.0) results in only a modest (5-20%, depending on what you’re doing) drop in speeds.
- Tepix 2y agoEven for training?
- angoragoats 2y agoI haven’t done a ton of training but everything I’ve heard and read indicates that PCIe 4.0 8x provides enough bandwidth for just about any application. You might see a negligible drop in performance, but no more than a few percent.
- LetsGetTechnicl 2y agoJust an eye watering amount of compute, electricity and money just to run LLM's... this is insane. Very cool though!
- elorant 2y agoThe motherboard has 7 PCie slots and there are 8 GPUs. So where does the spare one connect to? Is he using two GPUs in the same slot limiting the bandwidth?
- ganoushoreilly 2y agomay be using an nvme to pcie adapter, common in the crypto mining world
- buildbot 2y agoIt’s an epyc server board, it probably has actual U.2/MCIO pcie ports on the board that can be merged back into a 16x slot in the bios. I had/have several boards like that.
- system2 2y agoTypical crypto miner setup. I had two 6GPU setups with 1200W PSUs and 6 PCIE slots with PCI extender cables. Its value dropped harder than a cyber truck's after a few months. The worst thing is dust. They would accumulate so much every week I had to blow the dust off with an air compressor. Electricity cost was around $4 a day (24 x $0.20~). If online GPU renting is more expensive, maybe the initial cost could be justifiable.
- Havoc 2y ago> Typical crypto miner setup. Except not doing the sketchy x1 pcie lanes. That’s the part that makes nice LLM setups hard
- system2 2y agoCan you tell me what's sketchy about it? I have not had an issue with any one of the 12 extenders and bandwidth held well without any issues. Please explain if possible if LLM requires a different type of extender.
- Havoc 2y agoEh perhaps poor choice of words. It works fine for crypto but LLM performance is far more sensitive to bandwidth. You lose a ton of performance if you’ve got PCIe in the loop, never mind one lane pcie. That’s why nvlink is (was) a thing - trying to cut that out entirely.
- system2 2y agoGot it. I was planning to switch my miners to LLM farm. I will test and see how much of a difference it will make. Thanks.
- Havoc 2y agoVery cool. But also bit pricey unless you can actually utilize it 24/7 in some productive fashion
- anonymous247 2y ago[flagged]
- killingtime74 2y agoDid everyone just miss the fact that the post says the intention is to run Llama 3 405b but it has less than 1/4 of the VRAM required to do so? Did you just change your goals mid build? It's commonly known how much ram is required for a certain parameter size.
- nathanasmith 2y agoThe system has 512 GB of RAM so while it'll be slower at inference, he really has about 704 GB at his disposal to run the model assuming he distributes the weights across the VRAM and system RAM.
- schaefer 2y agoAmazing writeup. And what a heavy hitter of an inaugural blog entry... This might be the right time to ask: So, on the one hand, this is what it takes to pack 192gb of Nvidia flavored vram into a home server. I'm curious, is there any hope of doing any interesting work on a MacBook Pro Which currently can be max-spaced at 128 GB of unified memory (for the low, low price of $4.7k). I know there's no hope of running cuda on the macbook, and I'm clearly out of my depth here. But the possibly naive day-dream of tossing a massive LLM into a backpack is alluring...
- Eisenstein 2y agoDownload kobodlcpp and give it a try. It is a single exec and uses metal acceleration with an Apple Arm CPU.
- schaefer 2y agoThanks!
- lowbloodsugar 2y agoSometimes I think about dropping $10k to $20k on a rig like this and then I remember I can rent 8xH100s and 8xA100s with 640GB VRAM for $20/hr.
- illiac786 2y agoI dream from a future where the „home server with heat recuperation“ appliance will be common enough I can get a worker to install it physically for me - I have little electrical skills and zero plumbing skills. And I also hope that by then power consumption will have gone down.
- sireat 2y agoI was under the mistaken impression that you could not go beyond 2x3090 for reasonable inference speed. My assumption was that going beyond 2 cards incurs significant bandwidth penalty when going from NVLink between 2x3090s to PCIe for communicating between the other 3090s. What kind of T/s speeds are you getting with this type of 8x3090 setup? Presumably then even crazier 16x4090 would be an option for someone with enough PCIe slots/risers/extenders.
- deoxykev 2y agoAre you able to run 405B? 4Bit quant vram requirements are just shy of 192GB.
- tshadley 2y ago"Why PCIe Risers suck and the importance of using SAS Device Adapters, Redrivers, and Retimers for error-free PCIe connections." I'm a believer! Can't wait to hear more about this.