7 ms·

Interesting Pull quote: GPT-4 needed about 50 gigawatt-hours of energy to train. Using our scaling factor of 30x, we expect GPT-5 to need 1,500, GPT-6 to need

Interesting Pull quote: GPT-4 needed about 50 gigawatt-hours of energy to train. Using our scaling factor of 30x, we expect GPT-5 to need 1,500, GPT-6 to need 45,000, and GPT-7 to need 1.3 million.

toomuchtodo 3y ago
Assuming OpenAI is training exclusively using Microsoft compute, it should be low carbon energy. https://news.microsoft.com/europe/features/as-the-world-goes-digital-datacenters-that-make-the-cloud-work-look-to-renewable-energy-sources/ https://news.microsoft.com/europe/features/as-the-world-goes...
- fulafel 3y ago
  Do the dynamics of PPAs and RECs really translate to lowering electricity production CO2 footprint? It seems that it would work only when there is significantly more demand for that kind of electricity that would otherwise we produced, but since wind and solar are competitive on their own, and in many places there's no carbon cap-and-trade system to prevent the increased power demand leaking over to fossil energy, it's not obvious.
  - toomuchtodo 3y ago
    > Do the dynamics of PPAs and RECs really translate to lowering electricity production CO2 footprint? With new PPA models where purchases are made when generation occurs, yes. https://www.datacenterdynamics.com/en/news/google-to-expand-247-clean-energy-matching-with-flexidao/ https://www.datacenterdynamics.com/en/news/google-to-expand-... https://finance.yahoo.com/news/engie-supply-microsoft-data-centers-100000051.html https://finance.yahoo.com/news/engie-supply-microsoft-data-c... https://www.mckinsey.com/industries/electric-power-and-natural-gas/our-insights/decarbonizing-the-grid-with-24-7-clean-power-purchase-agreements https://www.mckinsey.com/industries/electric-power-and-natur...
    - fulafel 3y ago
      I'm not sure the first couple links support your claim (didn't have time for the last). in any case would be better to see analysis from some independent researchers than trade pubs.
- Nullabillity 3y ago
  Until we're in a world of 100% renewables, anything you do that uses renewables just pushes someone else to use more fossil fuel instead. Access to renewables isn't a blanket excuse to waste it. (Plus "renewable" is kind of a misnomer, there are still limited resources going into the production and maintenance of those power plants too.)
  - az09mugen 3y ago
    May people listen to your wisdom, I totally agree with you.
  - faeriechangling 3y ago
    Microsoft is hiring nuclear scientists presumably to power AI. If this becomes the case, I don't think it will be the case that Microsoft's green energy use is just pushing people to use dirty energy elsewhere. If anything they would be helping to accelerate a green transition. I don't see what's happening now as "Green" but I don't think you need 100% renewables worldwide for a datacenter to be green. You could also build a datacenter by a Dam which is in a location where people wouldn't normally live. This is not all zero sum.
- krunck 3y ago
  There is finite low carbon energy available. Would they choose to slow the rate of training the next generations of LLMs and use only greener every sources or would they go full speed regardless of the source of the energy? I think they would do the later in order to be first in the market with a next generation LLM.
  - toomuchtodo 3y ago
    Probably because they're the one making the commitments (and renewables are cheaper anyway). https://www.datacenterknowledge.com/energy/microsoft-pledges-emit-zero-carbon-2030 https://www.datacenterknowledge.com/energy/microsoft-pledges... https://www.lazard.com/media/2ozoovyg/lazards-lcoeplus-april-2023.pdf https://www.lazard.com/media/2ozoovyg/lazards-lcoeplus-april...

Interesting Pull quote: GPT-4 needed about 50 gigawatt-hours of energy to train. Using our scaling factor of 30x, we expect GPT-5 to need 1,500, GPT-6 to need 45,000, and GPT-7 to need 1.3 million.

toomuchtodo 3y ago

Assuming OpenAI is training exclusively using Microsoft compute, it should be low carbon energy. https://news.microsoft.com/europe/features/as-the-world-goes-digital-datacenters-that-make-the-cloud-work-look-to-renewable-energy-sources/ https://news.microsoft.com/europe/features/as-the-world-goes...

fulafel 3y ago
Do the dynamics of PPAs and RECs really translate to lowering electricity production CO2 footprint? It seems that it would work only when there is significantly more demand for that kind of electricity that would otherwise we produced, but since wind and solar are competitive on their own, and in many places there's no carbon cap-and-trade system to prevent the increased power demand leaking over to fossil energy, it's not obvious.
- toomuchtodo 3y ago
  > Do the dynamics of PPAs and RECs really translate to lowering electricity production CO2 footprint? With new PPA models where purchases are made when generation occurs, yes. https://www.datacenterdynamics.com/en/news/google-to-expand-247-clean-energy-matching-with-flexidao/ https://www.datacenterdynamics.com/en/news/google-to-expand-... https://finance.yahoo.com/news/engie-supply-microsoft-data-centers-100000051.html https://finance.yahoo.com/news/engie-supply-microsoft-data-c... https://www.mckinsey.com/industries/electric-power-and-natural-gas/our-insights/decarbonizing-the-grid-with-24-7-clean-power-purchase-agreements https://www.mckinsey.com/industries/electric-power-and-natur...
  - fulafel 3y ago
    I'm not sure the first couple links support your claim (didn't have time for the last). in any case would be better to see analysis from some independent researchers than trade pubs.
Nullabillity 3y ago
Until we're in a world of 100% renewables, anything you do that uses renewables just pushes someone else to use more fossil fuel instead. Access to renewables isn't a blanket excuse to waste it. (Plus "renewable" is kind of a misnomer, there are still limited resources going into the production and maintenance of those power plants too.)
- az09mugen 3y ago
  May people listen to your wisdom, I totally agree with you.
- faeriechangling 3y ago
  Microsoft is hiring nuclear scientists presumably to power AI. If this becomes the case, I don't think it will be the case that Microsoft's green energy use is just pushing people to use dirty energy elsewhere. If anything they would be helping to accelerate a green transition. I don't see what's happening now as "Green" but I don't think you need 100% renewables worldwide for a datacenter to be green. You could also build a datacenter by a Dam which is in a location where people wouldn't normally live. This is not all zero sum.
krunck 3y ago
There is finite low carbon energy available. Would they choose to slow the rate of training the next generations of LLMs and use only greener every sources or would they go full speed regardless of the source of the energy? I think they would do the later in order to be first in the market with a next generation LLM.
- toomuchtodo 3y ago
  Probably because they're the one making the commitments (and renewables are cheaper anyway). https://www.datacenterknowledge.com/energy/microsoft-pledges-emit-zero-carbon-2030 https://www.datacenterknowledge.com/energy/microsoft-pledges... https://www.lazard.com/media/2ozoovyg/lazards-lcoeplus-april-2023.pdf https://www.lazard.com/media/2ozoovyg/lazards-lcoeplus-april...