5 ms·

"Committing to buying the glass to replace the window I broke in your shop to rob the place, you're welcome." > Training a single frontier AI model will soon r

"Committing to buying the glass to replace the window I broke in your shop to rob the place, you're welcome." > Training a single frontier AI model will soon require gigawatts of power, and the US AI sector will need at least 50 gigawatts of capacity over the next several years. These things are so hideously inefficient. All of you building these things for these people should be embarrassed and ashamed.

measurablefunc 7mo ago
The numbers must go up, there is no other way.
epistasis 7mo ago
Adding new electricity demand to the grid should not be viewed as breaking windows and robbing others. When I bought an EV, I increased my electricity demand a huge amount, but it's not like I'm stealing from my neighbors. No rules were broken. We just need to make sure that I pay enough for my additional demand. > AI sector will need at least 50 gigawatts of capacity over the next several years. The error bars on this prediction are extremely large. It would represent a 5% increase in capacity in "the next several years" which is only a percent or two per year, but it could also only be 5GW over the next several years. 50GW represents about 1 year of actual grid additions. > All of you building these things for these people should be embarrassed and ashamed. I'm not building these things, and I think there should be AI critique, but this is far over the top. There's great value for all of humanity in these tools. The actual energy use of a typical user is not much more than a typical home appliance, because so many requests are batched together and processed in parallel. We should be ashamed of getting into our cars every day, that's a true harm to the environment. We should have built something better, allowed more transit. A daily commute of 30 miles is disastrous for the environment compared other any AI use that's really possible at the moment. Let's be cautious of AI but keep our critiques grounded in reality, so that we have enough powder left to fight the rest of things we need to change in society.
Dylan16807 7mo ago
> "Committing to buying the glass to replace the window I broke in your shop to rob the place, you're welcome." Buying electricity isn't inherently destructive. That's a very bad analogy. > These things are so hideously inefficient. All of you building these things for these people should be embarrassed and ashamed. I'm not arguing that they are efficient right now, but how would you measure that? What kind of output does it have to make per kWh of input to be acceptable? Keep in mind that the baseline of US power use is around 500GW and that currently AI is maybe 10.
keeda 7mo ago
> These things are so hideously inefficient. Quite the opposite, really. I did some napkin math for energy and water consumption, and compared to humans these things are very resource efficient. If LLMs improve productivity by even 5% (studies actually peg productivity gains across various professions at 15 - 30%, and these are from 2024!) the resource savings by accelerating all knowledge workers are significant. Simplistically, during 8 hours of work a human would consume 10 kWH of electricity + 27 gallons of water. Sped up by 5%, that drops by 0.5kWH and 1.35 gallons. Even assuming a higher end of resources used by LLMs, a 100 large prompts (~1 every 5 minutes) would only consume 0.25 kWH + 0.3 gallons. So we're still saving ~0.25 kWH + 1 gallon overall per day! That is, humans + LLMs are way more efficient than humans alone. As such, the more knowledge workers adopt LLMs, the more efficiently they can achieve the same work output! If we assume a conservative 10% productivity speed up, adoption across all ~100M knowledge work in the US will recoup the resource cost of a full training run in a few business days, even after accounting for the inference costs! Additional reading with more useful numbers (independent of my napkin math): https://www.nature.com/articles/s41598-024-76682-6 https://www.nature.com/articles/s41598-024-76682-6 https://cacm.acm.org/blogcacm/the-energy-footprint-of-humans-and-large-language-models/ https://cacm.acm.org/blogcacm/the-energy-footprint-of-humans...
- danielbln 7mo ago
  Do keep in mind that 1 large prompt every 5 minutes is not how e.g. coding agents are used. There it's 1 large prompt every couple of seconds.
  - keeda 7mo ago
    True, but I think in these scenarios they rely on prompt caching, which is much cheaper: https://ngrok.com/blog/prompt-caching/ https://ngrok.com/blog/prompt-caching/ I have no expertise here, but a couple years ago I had a prototype using locally deployed Llama 2 that cached the context (now deprecated https://github.com/ollama/ollama/issues/10576 https://github.com/ollama/ollama/issues/10576) from previous inference calls, and reused it for subsequent calls. The subsequent calls were much much faster. I suspect prompt caching works similarly, especially given changed code is very small compered to the rest of the codebase.
Andys 7mo ago
Every piece of progress looks like this to begin with.
esafak 7mo ago
How are you measuring efficiency? They're better than most humans, which is what I would need more of as a substitute.
- digitalPhonix 7mo ago
  A human consumes about 100 watts when not doing any physical exertion (round number, rule of thumb). So unless you can show an LLM running on 100w compute with capabilities similar to a human, they’re less efficient.
  - pmontra 7mo ago
    100 W is only the start. Let's say that I consume 100 W all along the day. I use an LLM for coding assistance in the old way of asking questions and copy pasting code. It's much faster than me at writing that code. I don't think it ever works 1 hour for me per day. It's probably 10 cumulative minutes, probably much less. Round it up to 12 minutes to make it 1/5 of a hour or round it down to 6 minutes for a 1/10. So instead of 24 it's 0.1 hours, 240 times less. Those 100 W could be 24000 W and the total power per day would be the same. Is that LLM consuming 24 kW when working for me? No idea but I hope it's less than that. Of course I could do all of my coding alone again, but I would be slower. It's like walking to the mall several times per week, several hours per time, instead of once or twice per week with a car, three cumulative hours. I trade a higher energy consumption for more time to do other things and the ability to live far away from shops.
    - digitalPhonix 7mo ago
      If you as a human are coding for 24 hours a day as the benchmark for LLM efficiency we have other problems.
      - pmontra 7mo ago
        I believe that we consume 100 W on average no matter what we do, except intense physical activities, which consume more.
dudisubekti 7mo ago
Are you against inefficiency or just LLMs? If it's the former, I assure you LLMs are nowhere near the top of the list lol You should start from beef industry.

measurablefunc 7mo ago

The numbers must go up, there is no other way.

epistasis 7mo ago

Adding new electricity demand to the grid should not be viewed as breaking windows and robbing others. When I bought an EV, I increased my electricity demand a huge amount, but it's not like I'm stealing from my neighbors. No rules were broken. We just need to make sure that I pay enough for my additional demand. > AI sector will need at least 50 gigawatts of capacity over the next several years. The error bars on this prediction are extremely large. It would represent a 5% increase in capacity in "the next several years" which is only a percent or two per year, but it could also only be 5GW over the next several years. 50GW represents about 1 year of actual grid additions. > All of you building these things for these people should be embarrassed and ashamed. I'm not building these things, and I think there should be AI critique, but this is far over the top. There's great value for all of humanity in these tools. The actual energy use of a typical user is not much more than a typical home appliance, because so many requests are batched together and processed in parallel. We should be ashamed of getting into our cars every day, that's a true harm to the environment. We should have built something better, allowed more transit. A daily commute of 30 miles is disastrous for the environment compared other any AI use that's really possible at the moment. Let's be cautious of AI but keep our critiques grounded in reality, so that we have enough powder left to fight the rest of things we need to change in society.

Dylan16807 7mo ago

> "Committing to buying the glass to replace the window I broke in your shop to rob the place, you're welcome." Buying electricity isn't inherently destructive. That's a very bad analogy. > These things are so hideously inefficient. All of you building these things for these people should be embarrassed and ashamed. I'm not arguing that they are efficient right now, but how would you measure that? What kind of output does it have to make per kWh of input to be acceptable? Keep in mind that the baseline of US power use is around 500GW and that currently AI is maybe 10.

keeda 7mo ago

> These things are so hideously inefficient. Quite the opposite, really. I did some napkin math for energy and water consumption, and compared to humans these things are very resource efficient. If LLMs improve productivity by even 5% (studies actually peg productivity gains across various professions at 15 - 30%, and these are from 2024!) the resource savings by accelerating all knowledge workers are significant. Simplistically, during 8 hours of work a human would consume 10 kWH of electricity + 27 gallons of water. Sped up by 5%, that drops by 0.5kWH and 1.35 gallons. Even assuming a higher end of resources used by LLMs, a 100 large prompts (~1 every 5 minutes) would only consume 0.25 kWH + 0.3 gallons. So we're still saving ~0.25 kWH + 1 gallon overall per day! That is, humans + LLMs are way more efficient than humans alone. As such, the more knowledge workers adopt LLMs, the more efficiently they can achieve the same work output! If we assume a conservative 10% productivity speed up, adoption across all ~100M knowledge work in the US will recoup the resource cost of a full training run in a few business days, even after accounting for the inference costs! Additional reading with more useful numbers (independent of my napkin math): https://www.nature.com/articles/s41598-024-76682-6 https://www.nature.com/articles/s41598-024-76682-6 https://cacm.acm.org/blogcacm/the-energy-footprint-of-humans-and-large-language-models/ https://cacm.acm.org/blogcacm/the-energy-footprint-of-humans...

danielbln 7mo ago
Do keep in mind that 1 large prompt every 5 minutes is not how e.g. coding agents are used. There it's 1 large prompt every couple of seconds.
- keeda 7mo ago
  True, but I think in these scenarios they rely on prompt caching, which is much cheaper: https://ngrok.com/blog/prompt-caching/ https://ngrok.com/blog/prompt-caching/ I have no expertise here, but a couple years ago I had a prototype using locally deployed Llama 2 that cached the context (now deprecated https://github.com/ollama/ollama/issues/10576 https://github.com/ollama/ollama/issues/10576) from previous inference calls, and reused it for subsequent calls. The subsequent calls were much much faster. I suspect prompt caching works similarly, especially given changed code is very small compered to the rest of the codebase.

Andys 7mo ago

Every piece of progress looks like this to begin with.

esafak 7mo ago

How are you measuring efficiency? They're better than most humans, which is what I would need more of as a substitute.

digitalPhonix 7mo ago
A human consumes about 100 watts when not doing any physical exertion (round number, rule of thumb). So unless you can show an LLM running on 100w compute with capabilities similar to a human, they’re less efficient.
- pmontra 7mo ago
  100 W is only the start. Let's say that I consume 100 W all along the day. I use an LLM for coding assistance in the old way of asking questions and copy pasting code. It's much faster than me at writing that code. I don't think it ever works 1 hour for me per day. It's probably 10 cumulative minutes, probably much less. Round it up to 12 minutes to make it 1/5 of a hour or round it down to 6 minutes for a 1/10. So instead of 24 it's 0.1 hours, 240 times less. Those 100 W could be 24000 W and the total power per day would be the same. Is that LLM consuming 24 kW when working for me? No idea but I hope it's less than that. Of course I could do all of my coding alone again, but I would be slower. It's like walking to the mall several times per week, several hours per time, instead of once or twice per week with a car, three cumulative hours. I trade a higher energy consumption for more time to do other things and the ability to live far away from shops.
  - digitalPhonix 7mo ago
    If you as a human are coding for 24 hours a day as the benchmark for LLM efficiency we have other problems.
    - pmontra 7mo ago
      I believe that we consume 100 W on average no matter what we do, except intense physical activities, which consume more.

dudisubekti 7mo ago

Are you against inefficiency or just LLMs? If it's the former, I assure you LLMs are nowhere near the top of the list lol You should start from beef industry.