7 ms·
A cluster of many $8000+ gpus. You're looking at around 350GB of vram, so 30 12gb gpus - a 3090 will cost around $1800, so $54k on the gpus, probably another $1
by robbedpeter 4y ago
A cluster of many $8000+ gpus. You're looking at around 350GB of vram, so 30 12gb gpus - a 3090 will cost around $1800, so $54k on the gpus, probably another $15k in power, cooling, and infrastructure, $5k in network, and probably another $20k in other costs to bootstrap it.
Or wait 10 years, if gpu capacity scales with Moore's law, consumer hardware should be able to run a ~400GB model locally.
- adamsmith143 4y agoAlmost no one does this on prem. What would this cost on AWS?
- cardine 4y agoThis is not true. On prem is extremely common for things like this because after ~6 months you'll have paid more in cloud costs than it would have cost to purchase the GPUs. And you don't need to purchase new GPUs every 6 months. AWS would cost $50-100k/mo for something comparable.
- coolspot 4y ago> so 30 12gb gpus - a 3090 will cost around $1800 3090 has 24Gb, thus 15 GPUs X $1800 = $27,000 in GPUs
- etaioinshrdlu 4y agoCan 3090 GPUs share their memory with one another to fit such a large model? Or is the enterprise grade hardware required?
- coolspot 4y agoYes, two 3090s ($1.7k each) can be connected via NVlink with common 48Gb of memory pool. Two RTX A6000 ($4.5k each) can form 96Gb memory pool.
- coolspot 4y agoOne could use $4.5k RTX A6000 48Gb instead. They can be joined in pairs of 96Gb common memory pool with NVlink. That’s 7x$4.5=$31.5k in GPUs to get 336Gb of memory. Or 8x$4.5=$36k in GPUs to get 384Gb of memory. Add say $3k per GPU pair for surrounding computer (MB,CPU,RAM,PSU) 4x$3k=$12k. $48k total budget.