5 ms·

That's fair about ignoring the training cost. I did write a bit more going into that in a follow up piece here: https://notfunatparties.substack.com/p/ai-is-goo

That's fair about ignoring the training cost. I did write a bit more going into that in a follow up piece here: https://notfunatparties.substack.com/p/ai-is-good-for-the-planet https://notfunatparties.substack.com/p/ai-is-good-for-the-pl... Do you have any better sources for the power usage stats? It would be good to get a bit closer on that front. Having said that, even if the cost share is closer to 80%, that still puts it on par with a laptop for an average person.

dartos 3y ago
Well openai has about 30k A100s https://www.tomshardware.com/news/chatgpt-nvidia-30000-gpus https://www.tomshardware.com/news/chatgpt-nvidia-30000-gpus What’s the power consumption on that assuming full load at all times?
dartos 3y ago
Also, I would expect openai to be taking a loss on each individual inference request as they also have a monthly fee, dalle, and loads of VC capital. No source for that though, I just wouldn’t assume that they’re breaking even
- tomjohnneill 3y ago
  I can definitely imagine they're not covering the amortised cost of the training with the cost per individual inference request. It seems less likely to me that they're making a significant loss on each subsequent request, but again no source from me on that either. Looking a bit more into this, I found this paper: https://arxiv.org/pdf/2311.16863.pdf https://arxiv.org/pdf/2311.16863.pdf. It references a table saying that text generation uses 0.047 kWh per 1000 inferences, which is 1-2 orders of magnitude lower than my estimate. Though that is for GPT2, so possibly tracks to something roughly in the ~0.001 kWh per inference for GPT3.5.
  - dartos 3y ago
    Well doesn’t the compute time for transformers scale roughly quadratically with model size? Would it make sense for power consumption to also scale roughly quadratically?
    - tomjohnneill 3y ago
      I'm not sure. The figures I've seen suggest that GPT3 required 10x more energy to train than GPT2 (e.g. https://www.nnlabs.org/power-requirements-of-large-language-models/#:~:text=Power%20Consumption%20of%20Different%20Large%20Language%20Models&text=In%20comparison%2C%20GPT%2D3%2C,more%20energy%20than%20GPT%2D2 https://www.nnlabs.org/power-requirements-of-large-language-....), so I think a roughly 1-2 order of magnitude increase in energy usage from GPT2 to GPT3.5 makes sense.

dartos 3y ago

Well openai has about 30k A100s https://www.tomshardware.com/news/chatgpt-nvidia-30000-gpus https://www.tomshardware.com/news/chatgpt-nvidia-30000-gpus What’s the power consumption on that assuming full load at all times?

dartos 3y ago

Also, I would expect openai to be taking a loss on each individual inference request as they also have a monthly fee, dalle, and loads of VC capital. No source for that though, I just wouldn’t assume that they’re breaking even

tomjohnneill 3y ago
I can definitely imagine they're not covering the amortised cost of the training with the cost per individual inference request. It seems less likely to me that they're making a significant loss on each subsequent request, but again no source from me on that either. Looking a bit more into this, I found this paper: https://arxiv.org/pdf/2311.16863.pdf https://arxiv.org/pdf/2311.16863.pdf. It references a table saying that text generation uses 0.047 kWh per 1000 inferences, which is 1-2 orders of magnitude lower than my estimate. Though that is for GPT2, so possibly tracks to something roughly in the ~0.001 kWh per inference for GPT3.5.
- dartos 3y ago
  Well doesn’t the compute time for transformers scale roughly quadratically with model size? Would it make sense for power consumption to also scale roughly quadratically?
  - tomjohnneill 3y ago
    I'm not sure. The figures I've seen suggest that GPT3 required 10x more energy to train than GPT2 (e.g. https://www.nnlabs.org/power-requirements-of-large-language-models/#:~:text=Power%20Consumption%20of%20Different%20Large%20Language%20Models&text=In%20comparison%2C%20GPT%2D3%2C,more%20energy%20than%20GPT%2D2 https://www.nnlabs.org/power-requirements-of-large-language-....), so I think a roughly 1-2 order of magnitude increase in energy usage from GPT2 to GPT3.5 makes sense.