5 ms·

I've seen the following quote. "The energy consumed per text prompt for Gemini Apps has been reduced by 33x over the past 12 months." My thinking is that if G

I've seen the following quote. "The energy consumed per text prompt for Gemini Apps has been reduced by 33x over the past 12 months." My thinking is that if Google can give away LLM usage (which is obviously subsidized) it can't be astronomically expensive, in the realm of what we are paying for ChatGPT. Google has their own TPUs and company culture oriented towards optimizing the energy usage/hardware costs. I tend to agree with the grandparent on this, LLMs will get cheaper for what we have now level intelligence, and will get more expensive for SOTA models.

SecretDreams 8mo ago
I do not disagree they will get cheaper, but I pointing out that none of this is being reflected in hardware pricing. You state LLMs are becoming more optimized (less expensive). I agree. This should have a knockon effect on hardware prices, but it is not. Where is the disconnect? Are hardware prices a lagging indicator? Is Nvidia still a 5 trillion dollar company if we see another 33x improvement in "energy consumed per text prompt"?
- zozbot234 8mo ago
  Jevon's paradox. As AI gets more efficient its potential scope expands further and the hardware it runs on becomes even more valuable. BTW, the absolute lowest "energy consumed per logical operation" is achieved with so-called 'neuromorphic' hardware that's dog slow in latency terms but more than compensates with extreme throughput. (A bit like an even more extreme version of current NPU/TPUs.) That's the kind of hardware we should be using for AI training once power use for that workload is measured in gigawatts. Gaming-focused GPUs are better than your average CPU, but they're absolutely not the optimum.
lelanthran 8mo ago
Google is a special case - ever since LLMs came out I've been pointing out that Google owns the entire vertical. OpenAI, Anthropic, etc are in a race to the bottom, but because they don't own the vertical they are beholden to Nvidia (for chips), they obviously have less training data, they need constant influsx of cash just to stay in that race to the bottom, etc. Google owns the entire stack - they don't need nvidia, they already have the data, they own the very important user-info via tracking, they have millions, if not billions, of emails on which to train, etc. Google needs no one, not even VCs. Their costs must be a fraction of the costs of pure-LLM companies.
- flyinglizard 8mo ago
  Gmail has 1.8b active users, each with thousands of emails in their inbox. The number of emails they can train of is probably in the trillions.
  - brokencode 8mo ago
    Email seems like not only a pretty terrible training data set, since most of it is marketing spam with dubious value, but also an invasion of privacy, since information could possibly leak about individuals via the model.
    - palmotea 8mo ago
      > Email seems like not only a pretty terrible training data set, since most of it is marketing spam with dubious value Google probably even has an advantage there: filter out everything except messages sent from valid gmail account to valid gmail account. If you do that you drop most of the spam and marketing, and have mostly human-to-human interactions. Then they have their spam filters.
      - Terr_ 8mo ago
        I'd upgrade that "probably" leak to "will absolutely" leak, albeit with some loss of fidelity. Imagine industrial espionage where someone is asking the model to roleplay a fictional email exchange between named corporate figures in a particular company.
- viraptor 8mo ago
  > OpenAI, Anthropic, etc are in a race to the bottom There's a bit of nuance hiding in the "etc". Openai and anthropic are still in a race for the top results. Minimax and GLM are in the race to the bottom while chasing good results - M2.1 is 10x cheaper than Sonnet for example, but practically fairly close in capabilities.
  - lelanthran 8mo ago
AdrianB1 8mo ago
> if Google can give away LLM usage (which is obviously subsidized) it can't be astronomically expensive There is a recent article by Linus Sebastian (LTT) talking about Youtube: it is almost impossible to support the cost to build a competitor because it is astronomically expensive (vs potential revenue)
SoftTalker 8mo ago
> Google has ... company culture oriented towards optimizing the energy usage/hardware costs. Google has a company culture of luring you in with freebies and then mining your data to sell ads.

SecretDreams 8mo ago

I do not disagree they will get cheaper, but I pointing out that none of this is being reflected in hardware pricing. You state LLMs are becoming more optimized (less expensive). I agree. This should have a knockon effect on hardware prices, but it is not. Where is the disconnect? Are hardware prices a lagging indicator? Is Nvidia still a 5 trillion dollar company if we see another 33x improvement in "energy consumed per text prompt"?

zozbot234 8mo ago
Jevon's paradox. As AI gets more efficient its potential scope expands further and the hardware it runs on becomes even more valuable. BTW, the absolute lowest "energy consumed per logical operation" is achieved with so-called 'neuromorphic' hardware that's dog slow in latency terms but more than compensates with extreme throughput. (A bit like an even more extreme version of current NPU/TPUs.) That's the kind of hardware we should be using for AI training once power use for that workload is measured in gigawatts. Gaming-focused GPUs are better than your average CPU, but they're absolutely not the optimum.

lelanthran 8mo ago

Google is a special case - ever since LLMs came out I've been pointing out that Google owns the entire vertical. OpenAI, Anthropic, etc are in a race to the bottom, but because they don't own the vertical they are beholden to Nvidia (for chips), they obviously have less training data, they need constant influsx of cash just to stay in that race to the bottom, etc. Google owns the entire stack - they don't need nvidia, they already have the data, they own the very important user-info via tracking, they have millions, if not billions, of emails on which to train, etc. Google needs no one, not even VCs. Their costs must be a fraction of the costs of pure-LLM companies.

flyinglizard 8mo ago
Gmail has 1.8b active users, each with thousands of emails in their inbox. The number of emails they can train of is probably in the trillions.
- brokencode 8mo ago
  Email seems like not only a pretty terrible training data set, since most of it is marketing spam with dubious value, but also an invasion of privacy, since information could possibly leak about individuals via the model.
  - palmotea 8mo ago
    > Email seems like not only a pretty terrible training data set, since most of it is marketing spam with dubious value Google probably even has an advantage there: filter out everything except messages sent from valid gmail account to valid gmail account. If you do that you drop most of the spam and marketing, and have mostly human-to-human interactions. Then they have their spam filters.
    - Terr_ 8mo ago
      I'd upgrade that "probably" leak to "will absolutely" leak, albeit with some loss of fidelity. Imagine industrial espionage where someone is asking the model to roleplay a fictional email exchange between named corporate figures in a particular company.
viraptor 8mo ago
> OpenAI, Anthropic, etc are in a race to the bottom There's a bit of nuance hiding in the "etc". Openai and anthropic are still in a race for the top results. Minimax and GLM are in the race to the bottom while chasing good results - M2.1 is 10x cheaper than Sonnet for example, but practically fairly close in capabilities.
- lelanthran 8mo ago

AdrianB1 8mo ago

> if Google can give away LLM usage (which is obviously subsidized) it can't be astronomically expensive There is a recent article by Linus Sebastian (LTT) talking about Youtube: it is almost impossible to support the cost to build a competitor because it is astronomically expensive (vs potential revenue)

SoftTalker 8mo ago

> Google has ... company culture oriented towards optimizing the energy usage/hardware costs. Google has a company culture of luring you in with freebies and then mining your data to sell ads.