6 ms·
I'm always curious about the cost of these training runs. Some back of the envelope calculations: > Overall we reach a throughput of over 1900 tokens / second
by superpope99 3y ago
I'm always curious about the cost of these training runs. Some back of the envelope calculations:
> Overall we reach a throughput of over 1900 tokens / second / TPU-v4 chip in our training run
1 trillion / 1900 = 526315789 chip seconds ~= 150000 chip hours.
Assuming "on-demand" pricing [1] that's about $500,000 training cost.
[1] https://cloud.google.com/tpu/pricing https://cloud.google.com/tpu/pricing
- p1esk 3y agoAt these levels of spending the actual cost is heavily negotiated and is usually far below the advertised on-demand pricing. Considering I could negotiate A100 for under a dollar/hr - 8 months ago, when they were in high demand, I wouldn't be surprised if the cost was close to 100k for this training run.
- execveat 3y agoNobody in their right mind is using GCE for training. Take a look at real prices: https://vast.ai/ https://vast.ai/
- bravura 3y agoThese nodes typically have slow downstream, and thus are hard to use when training requires pulling a huge dataset.
- simonw 3y agoI got the impression that kind of thing (buying time on GPUs hosted in people's homes) isn't useful for training large models, because model training requires extremely high bandwidth connections between the GPUs such that you effectively need them in the same rack.
- p1esk 3y agoI suspect most A100s on vast.ai are actually in a datacenter, and might even be on other public clouds, such as AWS. I don't see why either vast.ai or AWS care if this was the case.
- marshray 3y agoIs there at good resource that describes the impact of bandwidth and latency between GPUs? I assume that it's completely impractical to train on distributed systems?
- superpope99 3y agoAren't they explicitly using TPUs in their training? Vast AI are only offering GPUs.
- qeternity 3y agoAnyone training this size of model is almost certainly using AWS/GCE. The GPU marketplaces are nice for people who need smaller/single GPU setups, don't have huge reliability or SLA concerns, and where data privacy risks aren't an issue.
- mrtranscendence 3y agoWell, or Azure.
- qeternity 3y agoHa yes of course. But actually has anyone been able to get instances on Azure? Thought OpenAI had them all reserved.
- jeron 3y agoalso, https://brev.dev/ https://brev.dev/
- lostmsu 3y agoOnly 19 GPUs with 30+G of VRAM in the entire North America. I might be misreading it. It might be just 12 GPUs.
- deleted 3y ago[deleted]
- jerrygenser 3y agoThey haven't trained a 1 trillion token model yet. They have only done 200bn so far
- YetAnotherNick 3y agoGoogle is generous for giving TPU for free for research, so likely it is using this. The more representative number is one from meta which required 87k A100 hours, which is close to $100-200k for 7B model training.