6 ms·

Is it actually unbelievable? It's basically what every major AI lab head is saying from the start. It's the peanut gallery that keeps saying they are lying to

Is it actually unbelievable? It's basically what every major AI lab head is saying from the start. It's the peanut gallery that keeps saying they are lying to get funding.

ivape 1y ago
It's akin to us sending a rocket to space and immediately discovering a wormhole. Sure, there's a lot of science about what's out there, but to discover all this in our first few trips to orbit ...
- silveraxe93 1y ago
  Lemme start by saying this is objectively amazing. But I just really wouldn't call it a breakthrough. We had one breakthrough a couple of years ago with GPT-3, where we found that neural networks / transformers + scale does wonders. Everything else has been a smooth continuous improvement. Compare today's announcement to Genie-2[1] release less than 1 year ago. The speed is insane, but not surprising if you put in context on how fast AI is advancing. Again, nothing _new_. Just absurdly fast continuous progress. [1] - https://deepmind.google/discover/blog/genie-2-a-large-scale-foundation-world-model/ https://deepmind.google/discover/blog/genie-2-a-large-scale-...
  - ducktective 1y ago
    Wasn't the model winning gold in IMO result of a breakthrough? I doubt an stochastic parrot can solve math at IMO level...
    - Philpax 1y ago
      As far as we know, it was "just" scale on depth (model capability) and breadth (multiple agents working at the same time).
    - bakuninsbart 1y ago
      Why wouldn't it? I still have to hear one convincing argument how our brain isn't working as a function of probable next best actions. When you look at amoebas work, and animals that are somewhere between them and us in intelligence, and then us, it is a very similar kind of progression we see with current LLMs, from almost no state of the world, to a pretty solid one.
- pantalaimon 1y ago
  Joscha Bach postulates that what we call consciousness must be something rather simple, an emergent property present in all sufficiently complex biological organisms. We don't inherit any software, so cognitive function must bootstrap itself from it's underlying structure alone. https://media.ccc.de/v/38c3-self-models-of-loving-grace https://media.ccc.de/v/38c3-self-models-of-loving-grace
  - glenstein 1y ago
    >We don't inherit any software, so cognitive function must bootstrap itself from it's underlying structure alone. Hardware and software, as metaphors applied to biology, I think are better understood as a continuum than a binary, and if we don't inherit any software (is that true?), we at least inherit assembly code.
    - pantalaimon 1y ago
      > we don't inherit any software (is that true?), we at least inherit assembly code To stay with the metaphor, DNA could be rather understood as firmware that runs on the cell. What I mean with software is the 'mind' that runs on a collection of cells. Things like language, thoughts and ideas. There is also a second level of software that runs not on a single mind alone, but collection of minds, to form cliques or a societies. But this is not encoded in genes, but in memes.
      - glenstein 1y ago
        I think we have some notion of a proto-grammar or ability to linguistically conceptualize, probably at the level of some primordial conceptual units that are more fundamental than language, thoughts and ideas in the concrete forms we generally understand them to have. I think it's like Chomsky said, that we don't learn this infrastructure for understanding language any more than a bird "learns" their feathers. But I might be losing track of what you're suggesting is software in the metaphor. I think I'm broadly on board with your characterization of DNA, the mind and memes generally though.
JeremyNT 1y ago
Even as a layman and AI skeptic, to me this entirely matches my expectations, and something like this seemed like it was basically inevitable as of the first demos of video rendering responding to user input (a year ago? maybe?). Not to detract from what has been done here in any way, but it all seems entirely consistent with the types of progress we have seen. It's also no surprise to me that it's from Google, who I suspect is better situated than any of its AI competitors, even if it is sometimes slow to show progress publicly.
- westbrookt 1y ago
  https://worldmodels.github.io/ https://worldmodels.github.io/ I think this was the first mention of world models I've seen circa 2018. This is based on VAEs though.
- kranke155 1y ago
  Google seems to have had the keys to changing the world years ago and decided not to. Hard to fault them as the process towards ASI now appears to be runaway and uncontrollable.
glenstein 1y ago
>It's basically what every major AI lab head is saying from the start. I suppose it depends what you count as "the start". The idea of AI as a real research project has been around since at least the 1950s. And I'm not a programmer or computer scientist, but I'm a philosophy nerd and I know debates about what computers can or can't do started around then. One side of the debate was that it awaited new conceptual and architectural breakthroughs. I also think you can look at, say, Ted Talks on the topic, with guys like Jeff Hawkins presenting the problem as one of searching for conceptual breakthroughs, and I think similar ideas of such a search have been at the center of Douglas Hofstadter's career. I think in all those cases, they would have treated "more is different" like an absence of nuance, because there was supposed to be a puzzle to solve (and in a sense there is, and there has been, in terms of vector space and back propagation and so on, but it wasn't necessarily clear that physics could "pop out" emergently from such a foundation).
- jonas21 1y ago
  When they say "the start", I think they mean the start of the current LLM era (circa 2017). The main story of this time has been a rejection of the idea that major conceptual breakthroughs and complex architectures are needed to achieve intelligence. Instead, it's better to focus on simple, general-purpose methods that can scale to massive amounts of data and compute (i.e. the Bitter Lesson [1]). [1] http://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incompleteideas.net/IncIdeas/BitterLesson.html
  - tripzilch 1y ago
    Oof ... to call other people's decades of research into directed machine learning "a colossal waste of researcher's time" is indeed a rather toxic point of view unsurprisingly causing a bitter reaction in scientists/researchers. Even if his broader point might be valid (about the most fruitful directions in ML), calling something a "bitter lesson" while insulting a whole field of science is ... something. Also as someone involved in early RL, he should know better.
- satvikpendem 1y ago
  The start of deep neural networks, ie AlexNet

Is it actually unbelievable? It's basically what every major AI lab head is saying from the start. It's the peanut gallery that keeps saying they are lying to get funding.

ivape 1y ago

It's akin to us sending a rocket to space and immediately discovering a wormhole. Sure, there's a lot of science about what's out there, but to discover all this in our first few trips to orbit ...

silveraxe93 1y ago
Lemme start by saying this is objectively amazing. But I just really wouldn't call it a breakthrough. We had one breakthrough a couple of years ago with GPT-3, where we found that neural networks / transformers + scale does wonders. Everything else has been a smooth continuous improvement. Compare today's announcement to Genie-2[1] release less than 1 year ago. The speed is insane, but not surprising if you put in context on how fast AI is advancing. Again, nothing _new_. Just absurdly fast continuous progress. [1] - https://deepmind.google/discover/blog/genie-2-a-large-scale-foundation-world-model/ https://deepmind.google/discover/blog/genie-2-a-large-scale-...
- ducktective 1y ago
  Wasn't the model winning gold in IMO result of a breakthrough? I doubt an stochastic parrot can solve math at IMO level...
  - Philpax 1y ago
    As far as we know, it was "just" scale on depth (model capability) and breadth (multiple agents working at the same time).
  - bakuninsbart 1y ago
    Why wouldn't it? I still have to hear one convincing argument how our brain isn't working as a function of probable next best actions. When you look at amoebas work, and animals that are somewhere between them and us in intelligence, and then us, it is a very similar kind of progression we see with current LLMs, from almost no state of the world, to a pretty solid one.
pantalaimon 1y ago
Joscha Bach postulates that what we call consciousness must be something rather simple, an emergent property present in all sufficiently complex biological organisms. We don't inherit any software, so cognitive function must bootstrap itself from it's underlying structure alone. https://media.ccc.de/v/38c3-self-models-of-loving-grace https://media.ccc.de/v/38c3-self-models-of-loving-grace
- glenstein 1y ago
  >We don't inherit any software, so cognitive function must bootstrap itself from it's underlying structure alone. Hardware and software, as metaphors applied to biology, I think are better understood as a continuum than a binary, and if we don't inherit any software (is that true?), we at least inherit assembly code.
  - pantalaimon 1y ago
    > we don't inherit any software (is that true?), we at least inherit assembly code To stay with the metaphor, DNA could be rather understood as firmware that runs on the cell. What I mean with software is the 'mind' that runs on a collection of cells. Things like language, thoughts and ideas. There is also a second level of software that runs not on a single mind alone, but collection of minds, to form cliques or a societies. But this is not encoded in genes, but in memes.
    - glenstein 1y ago
      I think we have some notion of a proto-grammar or ability to linguistically conceptualize, probably at the level of some primordial conceptual units that are more fundamental than language, thoughts and ideas in the concrete forms we generally understand them to have. I think it's like Chomsky said, that we don't learn this infrastructure for understanding language any more than a bird "learns" their feathers. But I might be losing track of what you're suggesting is software in the metaphor. I think I'm broadly on board with your characterization of DNA, the mind and memes generally though.

JeremyNT 1y ago

Even as a layman and AI skeptic, to me this entirely matches my expectations, and something like this seemed like it was basically inevitable as of the first demos of video rendering responding to user input (a year ago? maybe?). Not to detract from what has been done here in any way, but it all seems entirely consistent with the types of progress we have seen. It's also no surprise to me that it's from Google, who I suspect is better situated than any of its AI competitors, even if it is sometimes slow to show progress publicly.

westbrookt 1y ago
https://worldmodels.github.io/ https://worldmodels.github.io/ I think this was the first mention of world models I've seen circa 2018. This is based on VAEs though.
kranke155 1y ago
Google seems to have had the keys to changing the world years ago and decided not to. Hard to fault them as the process towards ASI now appears to be runaway and uncontrollable.

glenstein 1y ago

>It's basically what every major AI lab head is saying from the start. I suppose it depends what you count as "the start". The idea of AI as a real research project has been around since at least the 1950s. And I'm not a programmer or computer scientist, but I'm a philosophy nerd and I know debates about what computers can or can't do started around then. One side of the debate was that it awaited new conceptual and architectural breakthroughs. I also think you can look at, say, Ted Talks on the topic, with guys like Jeff Hawkins presenting the problem as one of searching for conceptual breakthroughs, and I think similar ideas of such a search have been at the center of Douglas Hofstadter's career. I think in all those cases, they would have treated "more is different" like an absence of nuance, because there was supposed to be a puzzle to solve (and in a sense there is, and there has been, in terms of vector space and back propagation and so on, but it wasn't necessarily clear that physics could "pop out" emergently from such a foundation).

jonas21 1y ago
When they say "the start", I think they mean the start of the current LLM era (circa 2017). The main story of this time has been a rejection of the idea that major conceptual breakthroughs and complex architectures are needed to achieve intelligence. Instead, it's better to focus on simple, general-purpose methods that can scale to massive amounts of data and compute (i.e. the Bitter Lesson [1]). [1] http://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incompleteideas.net/IncIdeas/BitterLesson.html
- tripzilch 1y ago
  Oof ... to call other people's decades of research into directed machine learning "a colossal waste of researcher's time" is indeed a rather toxic point of view unsurprisingly causing a bitter reaction in scientists/researchers. Even if his broader point might be valid (about the most fruitful directions in ML), calling something a "bitter lesson" while insulting a whole field of science is ... something. Also as someone involved in early RL, he should know better.
satvikpendem 1y ago
The start of deep neural networks, ie AlexNet