11 ms·
From the paper: "we have to use all the GPUs in DGX1 (8 V100 GPUs, each with 16GB memory) for training. We distribute the generator computation task to 4 GPUs a
by huling0 8y ago
From the paper:
"we have to use all the GPUs in DGX1 (8 V100 GPUs, each with 16GB memory) for training. We distribute the generator computation task to 4 GPUs and the discriminator computation task to the other 4 GPUs. Training takes ∼10 days for 2K resolution."
As I don't have a DGX1 here, training the 2K resolution net for 10 days on a p3.16xlarge instance (also has 8 V100 GPUs) would cost USD 5875 on AWS.
(USD24.48 per hour on-demand pricing * 24 hours/day * 10 days)
- marmaduke 8y agoTheir GitHub README says 24GB, and 12/16 GB requires cropping and performance not guaranteed. I’ve only seen P100 with 16 each, and its the big Quadros that have 24
- sp332 8y agoThe DXG-1 costs $129,000 so AWS is cheaper unless you need to do it 22 times. And you can have multiple instances running at once and get all of your results in ten days, instead of waiting ten days again for each run.
- gok 8y agoWell plus electricity. A DGX-1 takes 4 kilowatts or so that 10 day training run will take just about a megawatt hour or about $100 at retail. So the cross over point is more like 23 runs :)
- huling0 8y agoyeah, but does not look like something many hobbyists would try for fun ;-)
- p1esk 8y agoOr you can build your own 8x2080Ti rig which will have 80% of performance for 1/10th of the cost.
- twtw 8y agoDGX 1V has $7500 worth of CPU alone to feed the gpus. Throw in 8 TB of nvme ssd for training data and you're looking at something more like 1/5th the cost. V100 has ~50% higher memory bandwidth than 2080Ti, so you probably wouldn't get 80% of the performance. Also, only two 2080Ti can be connected via nvlink.
- twtw 8y agoWhich is why it's nice that the pre-trained models are available. If you want to train on your own dataset, that price does not seem unreasonable to me.