11 ms·
Hey guys, this is something I have been intending to share here for a while. This setup took me some time to plan and put together, and then some more time to e
by XMasterrrr 2y ago
Hey guys, this is something I have been intending to share here for a while. This setup took me some time to plan and put together, and then some more time to explore the software part of things and the possibilities that came with it.
Part of the main reason I built this was data privacy, I do not want to hand over my private data to any company to further train their closed weight models; and given the recent drop in output quality on different platforms (ChatGPT, Claude, etc), I don't regret spending the money on this setup.
I was also able to do a lot of cool things using this server by leveraging tensor parallelism and batch inference, generating synthetic data, and experimenting with finetuning models using my private data. I am currently building a model from scratch, mainly as a learning project, but I am also finding some cool things while doing so and if I can get around ironing out the kinks, I might release it and write a tutorial from my notes.
So I finally had the time this weekend to get my blog up and running, and I am planning on following up this blog post with a series of posts on my learnings and findings. I am also open to topics and ideas to experiment with on this server and write about, so feel free to shoot your shot if you have ideas you want to experiment with and don't have the hardware, I am more than willing to do that on your behalf and sharing the findings
Please let me know if you have any questions, my PMs are open, and you can also reach me on any of the socials I have posted on my website.
- nrp 2y agoHow are you finding 2b/3b quantized llama 405B? Is it behaving better than 8b or 16b llama 70B?
- nullindividual 2y agoDo you run this 24/7? What is your cost of electricity per kilowatt hour and what is the cost of this setup per month?
- michaelt 2y agoI have a much smaller setup than the author - a quarter the GPUs and RAM - and I was surprised to find it draws 300W at idle
- nullindividual 2y agoThe reason I asked is I used to run a dual X5650 server with SSDs and it was about $50/month with the cheapest (or very close to) rates in the US.
- disiplus 2y agoWe have more expensive gas then usa, but i pay like a 5 cent per kwh @220V Did not know how expensive it is in usa, especially California.
- fuzzybear3965 2y agoYep. ~$.33/kWh in Southern California (SoCal Edison) and going up all the time!
- fragmede 2y ago$0.51/KwH during peek hours in San Francisco!
- nullindividual 2y agoI pay 10 cents per KWH. Most of the electricity in my state is produced by hydropower.
- rootusrootus 2y agoUS generally has some of the cheapest electricity, about half of what Europeans pay. California has some areas that are abnormally expensive.
- nirav72 2y agoDepends on where you live in the U.S. On the west coast, you’ll definitely pay 2x the national average. I live in the south-eastern part of the U.S and electricity is quite a bit cheaper at 12 cents/kwh. But even here it varies by region in my state of Georgia. Down the road where I have friends and family , they get power from a different provider. Their cost are much higher. But they also have flexible pricing. Such as at night they pay a fraction of their day time rates. From 17 cents to 3 cents. While I have a fixed rate 24/7. Which is cheaper than their day time rates.
- trollbridge 2y agoThis is a setup that might make more sense to run at full power during winter months.
- TaylorAlexander 2y ago[flagged]
- sva_ 2y agoA single 3090 will deliver more tflops than the m2 ultra.
- angoragoats 2y agoYep. The 8x3090 setup should be at least 10 times faster than the Mac.
- wkat4242 2y agoYes though for inference memory bandwidth is more important than tflops. Not sure how Apple compares in that regard. The OP does train models too though which is more compute heavy.
- angoragoats 2y agoEach of the 3090s has something like 15% higher memory bandwidth than the whole M2 Ultra system does.
- evilduck 2y agoThe M2 Ultra doesn't require doing electrical work on your house like this 8x 3090 setup did though.
- ein0p 2y agoYes, for 20x the FLOPS at about twice the price you might need to do some work.
- bongodongobob 2y agoYeah cause it's doing like 50x less work.
- bravura 2y agoHow loud is it? Was special electrical needed?
- mattnewton 2y agoThe main thing stopping me from going beyond 2x 4090’s in my home lab is power. Anything around ~2k watts on a single circuit breaker is likely to flip it, and that’s before you get to the costs involved of drawing that much power for multiple days of a training run. How did you navigate that in a (presumably) residential setting?
- abound 2y agoNot OP, but my current home had a dedicated 50A/240V circuit because the previous owner did glass work and had a massive electric kiln. I can't imagine it was cheap to install, but I've used it for beefy, energy hungry servers in the past. Which is all to say its possible in a residential setting, just probably expensive.
- woleium 2y agoYes, or something like a residential aircon heatpump will need a 40a circuit too. Car charging usually has a 30a. Electric oven is usually 40a. There’s lots of stuff that uses that sort of power residentially
- throwthrowuknow 2y agoNot speaking from direct experience building a rig like this but the blog post mentions having 3 power supplies so the most direct solution would be to put each on their own dedicated circuit. As long as you have space in your electrical box this is straightforward to do though I would recommend having an electrician do the wiring if you aren’t experienced with that type of home electrical work.
- gizmo686 2y agoEven without space in the existing box, installing a subpanel isn't that much more of a cost.
- XMasterrrr 2y agoActually, putting each PSU on its own circuit is crazy dangerous. In the scenario of your suggestion, if one goes out, you are in for a fire. Highly recommend against that.
- pupdogg 2y agoAmazing setup. I have the capability to design, fabricate, and powder coat sheet metal. I would love to collaborate on designing and fabricating a cool enclosure for this setup. Let me know if you're interested.
- lossolo 2y agoCool, it looks similar to my crypto mining rigs (8xGPU per node) from around 7 years ago, but I used PCI-E risers and a dual power supply.
- koyote 2y agoThis is undoubtedly cool and I am a bit jealous! Maybe a bit of a stupid question, but what do you actually do with the models you run/build, a part from tinkering? I'd assume most tinkering can also be done on smaller systems? Is it in order to build a model that is actually 'useful'/competitive?
- faangguyindia 2y agoI tried self hosting LLM for commandline instant completion and guidance utility: https://github.com/zerocorebeta/Option-K https://github.com/zerocorebeta/Option-K But problem is even 7b models are too slow on my pc. Hosted models are lightening fast. I considered possibility of buying hardware but decided against it.