22 ms·

It's insane that that this works, and that it works fast enough to render at 20 fps. It seems like they almost made a cross between a diffusion model and an RN

It's insane that that this works, and that it works fast enough to render at 20 fps. It seems like they almost made a cross between a diffusion model and an RNN, since they had to encode the previous frames and actions and feed it into the model at each step. Abstractly, it's like the model is dreaming of a game that it played a lot of, and real time inputs just change the state of the dream. It makes me wonder if humans are just next moment prediction machines, with just a little bit more memory built in.

Teever 2y ago
Also recursion and nested virtualization. We can dream about dreaming and imagine different scenarios, some completely fictional or simply possible future scenarios all while doing day to day stuff.
lokimedes 2y ago
It makes good sense for humans to have this ability. If we flip the argument, and see the next frame as a hypothesis for what is expected as the outcome of the current frame, then comparing this "hypothesis" with what is sensed makes it easier to process the differences, rather than the totality of the sensory input. As Richard Dawkins recently put it in a podcast[1], our genes are great prediction machines, as their continued survival rests on it. Being able to generate a visual prediction fits perfectly with the amount of resources we dedicate to sight. If that is the case, what does aphantasia tell us? [1] https://podcasts.apple.com/dk/podcast/into-the-impossible-with-brian-keating/id1169885840?i=1000665776221 https://podcasts.apple.com/dk/podcast/into-the-impossible-wi...
- quickestpoint 2y ago
  As Richard Dawkins theorized, would be more accurate and less LLM like :)
- jonplackett 2y ago
  What’s the aphantasia link? I’ve got aphantasia. I’m convinced though that the bit of my brain that should be making images is used for letting me ‘see’ how things are connected together very easily in my head. Also I still love games like Pictionary and can somehow draw things onto paper than I don’t really know what they look like in my head. It’s often a surprise when pen meets paper.
  - lokimedes 2y ago
    I agree, it is my own experience as well. Craig Venter In one of his books also credit this way of representing knowledge as abstractions as his strength in inventing new concepts. The link may be that we actually see differences between “frames”, rather than the frames directly. That in itself would imply that a from of sub-visual representation is being processed by our brain. For aphantasia, it could be that we work directly on this representation instead of recalling imagery through the visual system. Many people with aphantasia reports being able to visualize in their dreams, meaning that they don't lack the ability to generate visuals. So it may be that the brain has an affinity to rely on the abstract representation when "thinking", while dreaming still uses the "stable diffusion mode". I’m no where near qualified to speak of this with certainty, but it seems plausible to me.
- dbspin 2y ago
  Worth noting that aphantasia doesn't necessarily extend to dreams. Anecdotally - I have pretty severe aphantasia (I can conjure milisecond glimpses of barely tangible imagery that I can't quite perceive before it's gone - but only since learning that visualisation wasn't a linguistic metaphor). I can't really simulate object rotation. I can't really 'picture' how things will look before they're drawn / built etc. However I often have highly vivid dream imagery. I also have excellent recognition of faces and places (e.g.: can't get lost in a new city). So there clearly is a lot of preconscious visualisation and image matching going on in some aphantasia cases, even where the explicit visual screen is all but absent.
  - zimpenfish 2y ago
    Pretty much the same for me. My aphantasia is total (no images at all) but still ludicrously vivid dreams and not too bad at recognising people and places.
  - lokimedes 2y ago
    I fabulate about this in another comment below: > Many people with aphantasia reports being able to visualize in their dreams, meaning that they don't lack the ability to generate visuals. So it may be that the [aphantasia] brain has an affinity to rely on the abstract representation when "thinking", while dreaming still uses the "stable diffusion mode". (I obviously don't know what I'm talking about, just a fellow aphant)
    - dbspin 2y ago
      Obviously we're all introspecting here - but my guess is that there's some kind of cross talk in aphantasic brains between the conscious narrating semantic brain and the visual module. Such that default mode visualisation is impaired. It's specifically the loss of reflexive consciousness that allows visuals to emerge. Not sure if this is related, but I have pretty severe chronic insomnia, and I often wonder if this in part relates to the inability to drift off into imagery.
    - drowsspa 2y ago
      Yeah. In my head it's like I'm manipulating SVG paths instead of raw pixels
slashdave 2y ago
Image is 2D. Video is 3D. The mathematical extension is obvious. In this case, low resolution 2D (pixels), and the third dimension is just frame rate (discrete steps). So rather simple.
- Sharlin 2y ago
  This is not "just" video, however. It's interactive in real time. Sure, you can say that playing is simply video with some extra parameters thrown in to encode player input, but still.
  - slashdave 2y ago
    It is just video. There are no external interactions. Heck, it is far simpler than video, because the point of view and frame is fixed.
    - raincole 2y ago
      ? I highly suggest you to read the paper briefly before commenting on the topic. The whole point is that it's not just generating a video.
      - slashdave 2y ago
        I did. It is generating a video, using latent information on player actions during the process (which it also predicts). It is not interactive.
    - SeanAnderson 2y ago
      I think you're mistaken. The abstract says it's interactive, "We present GameNGen, the first game engine powered entirely by a neural model that enables real-time interaction" Further - "a diffusion model is trained to produce the next frame, conditioned on the sequence of past frames and actions." specifically "and actions" User input is being fed into this system and subsequent frames take that into account. The user is "actually" firing a gun.
stevenhuang 2y ago
> It makes me wonder if humans are just next moment prediction machines, with just a little bit more memory built in. Yup, see https://en.wikipedia.org/wiki/Predictive_coding https://en.wikipedia.org/wiki/Predictive_coding
- quickestpoint 2y ago
  Umm, that’s a theory.
  - mind-blight 2y ago
    So are gravity and friction. I don't know how well tested or accepted it is, but being just a theory doesn't tell you much about how true it is without more info
richard___ 2y ago
Did they take in the entire history as context?
nsbk 2y ago
We are. At least that's what Lisa Feldman Barrett [1] thinks. It is worth listening to this Lex Fridman podcast: Counterintuitive Ideas About How the Brain Works [2], where she explains among other ideas how constant prediction is the most efficient way of running a brain as opposed to reaction. I never get tired of listening to her, she's such a great science communicator. [1] https://en.wikipedia.org/wiki/Lisa_Feldman_Barrett https://en.wikipedia.org/wiki/Lisa_Feldman_Barrett [2] https://www.youtube.com/watch?v=NbdRIVCBqNI&t=1443s https://www.youtube.com/watch?v=NbdRIVCBqNI&t=1443s
- PunchTornado 2y ago
  Interesting talk about the brain, but the stuff she says about free will is not a very good argument. Basically it is sort of the argument that the ancient greeks made which brings the discussion into a point where you can take both directions.
mensetmanusman 2y ago
Penrose (Nobel prize in physics) stipulates that quantum effects in the brain may allow a certain amount of time travel and back propagation to accomplish this.
- wrsh07 2y ago
  You don't need back propagation to learn This is an incredibly complex hypothesis that doesn't really seem justified by the evidence
wrsh07 2y ago
Makes me wonder when an update to the world models paper comes out where they drop in diffusion models: https://worldmodels.github.io/ https://worldmodels.github.io/
dartos 2y ago
> It makes me wonder if humans are just next moment prediction machines, with just a little bit more memory built in. This, to me, seems extremely reductionist. Like you start with AI and work backwards until you frame all cognition as next something predictors. It’s just the stochastic parrot argument again.
bangaladore 2y ago
> It's insane that that this works, and that it works fast enough to render at 20 fps. It is running on an entire v5 TPU (https://cloud.google.com/blog/products/ai-machine-learning/introducing-cloud-tpu-v5p-and-ai-hypercomputer https://cloud.google.com/blog/products/ai-machine-learning/i...) It's unclear how that compares to a high-end consumer GPU like a 3090, but they seem to have similar INT8 TFLOPS. The TPU has less memory (16 vs. 24), and I'm unsure of the other specs. Something doesn't add up, in my opinion, though. SD usually takes (at minimum) seconds to produce a high-quality result on a 3090, so I can't comprehend how they are like 2 orders of magnitudes faster—indicating that the TPU vastly outperforms a GPU for this task. They seem to be producing low-res (320x240) images, but it still seems too fast.
- Philpax 2y ago
  There's been a lot of work in optimising inference speed of SD - SD Turbo, latent consistency models, Hyper-SD, etc. It is very possible to hit these frame rates now.

Teever 2y ago

Also recursion and nested virtualization. We can dream about dreaming and imagine different scenarios, some completely fictional or simply possible future scenarios all while doing day to day stuff.

lokimedes 2y ago

It makes good sense for humans to have this ability. If we flip the argument, and see the next frame as a hypothesis for what is expected as the outcome of the current frame, then comparing this "hypothesis" with what is sensed makes it easier to process the differences, rather than the totality of the sensory input. As Richard Dawkins recently put it in a podcast[1], our genes are great prediction machines, as their continued survival rests on it. Being able to generate a visual prediction fits perfectly with the amount of resources we dedicate to sight. If that is the case, what does aphantasia tell us? [1] https://podcasts.apple.com/dk/podcast/into-the-impossible-with-brian-keating/id1169885840?i=1000665776221 https://podcasts.apple.com/dk/podcast/into-the-impossible-wi...

quickestpoint 2y ago
As Richard Dawkins theorized, would be more accurate and less LLM like :)
jonplackett 2y ago
What’s the aphantasia link? I’ve got aphantasia. I’m convinced though that the bit of my brain that should be making images is used for letting me ‘see’ how things are connected together very easily in my head. Also I still love games like Pictionary and can somehow draw things onto paper than I don’t really know what they look like in my head. It’s often a surprise when pen meets paper.
- lokimedes 2y ago
  I agree, it is my own experience as well. Craig Venter In one of his books also credit this way of representing knowledge as abstractions as his strength in inventing new concepts. The link may be that we actually see differences between “frames”, rather than the frames directly. That in itself would imply that a from of sub-visual representation is being processed by our brain. For aphantasia, it could be that we work directly on this representation instead of recalling imagery through the visual system. Many people with aphantasia reports being able to visualize in their dreams, meaning that they don't lack the ability to generate visuals. So it may be that the brain has an affinity to rely on the abstract representation when "thinking", while dreaming still uses the "stable diffusion mode". I’m no where near qualified to speak of this with certainty, but it seems plausible to me.
dbspin 2y ago
Worth noting that aphantasia doesn't necessarily extend to dreams. Anecdotally - I have pretty severe aphantasia (I can conjure milisecond glimpses of barely tangible imagery that I can't quite perceive before it's gone - but only since learning that visualisation wasn't a linguistic metaphor). I can't really simulate object rotation. I can't really 'picture' how things will look before they're drawn / built etc. However I often have highly vivid dream imagery. I also have excellent recognition of faces and places (e.g.: can't get lost in a new city). So there clearly is a lot of preconscious visualisation and image matching going on in some aphantasia cases, even where the explicit visual screen is all but absent.
- zimpenfish 2y ago
  Pretty much the same for me. My aphantasia is total (no images at all) but still ludicrously vivid dreams and not too bad at recognising people and places.
- lokimedes 2y ago
  I fabulate about this in another comment below: > Many people with aphantasia reports being able to visualize in their dreams, meaning that they don't lack the ability to generate visuals. So it may be that the [aphantasia] brain has an affinity to rely on the abstract representation when "thinking", while dreaming still uses the "stable diffusion mode". (I obviously don't know what I'm talking about, just a fellow aphant)
  - dbspin 2y ago
    Obviously we're all introspecting here - but my guess is that there's some kind of cross talk in aphantasic brains between the conscious narrating semantic brain and the visual module. Such that default mode visualisation is impaired. It's specifically the loss of reflexive consciousness that allows visuals to emerge. Not sure if this is related, but I have pretty severe chronic insomnia, and I often wonder if this in part relates to the inability to drift off into imagery.
  - drowsspa 2y ago
    Yeah. In my head it's like I'm manipulating SVG paths instead of raw pixels

slashdave 2y ago

Image is 2D. Video is 3D. The mathematical extension is obvious. In this case, low resolution 2D (pixels), and the third dimension is just frame rate (discrete steps). So rather simple.

Sharlin 2y ago
This is not "just" video, however. It's interactive in real time. Sure, you can say that playing is simply video with some extra parameters thrown in to encode player input, but still.
- slashdave 2y ago
  It is just video. There are no external interactions. Heck, it is far simpler than video, because the point of view and frame is fixed.
  - raincole 2y ago
    ? I highly suggest you to read the paper briefly before commenting on the topic. The whole point is that it's not just generating a video.
    - slashdave 2y ago
      I did. It is generating a video, using latent information on player actions during the process (which it also predicts). It is not interactive.
  - SeanAnderson 2y ago
    I think you're mistaken. The abstract says it's interactive, "We present GameNGen, the first game engine powered entirely by a neural model that enables real-time interaction" Further - "a diffusion model is trained to produce the next frame, conditioned on the sequence of past frames and actions." specifically "and actions" User input is being fed into this system and subsequent frames take that into account. The user is "actually" firing a gun.

stevenhuang 2y ago

> It makes me wonder if humans are just next moment prediction machines, with just a little bit more memory built in. Yup, see https://en.wikipedia.org/wiki/Predictive_coding https://en.wikipedia.org/wiki/Predictive_coding

quickestpoint 2y ago
Umm, that’s a theory.
- mind-blight 2y ago
  So are gravity and friction. I don't know how well tested or accepted it is, but being just a theory doesn't tell you much about how true it is without more info

richard___ 2y ago

Did they take in the entire history as context?

nsbk 2y ago

We are. At least that's what Lisa Feldman Barrett [1] thinks. It is worth listening to this Lex Fridman podcast: Counterintuitive Ideas About How the Brain Works [2], where she explains among other ideas how constant prediction is the most efficient way of running a brain as opposed to reaction. I never get tired of listening to her, she's such a great science communicator. [1] https://en.wikipedia.org/wiki/Lisa_Feldman_Barrett https://en.wikipedia.org/wiki/Lisa_Feldman_Barrett [2] https://www.youtube.com/watch?v=NbdRIVCBqNI&t=1443s https://www.youtube.com/watch?v=NbdRIVCBqNI&t=1443s

PunchTornado 2y ago
Interesting talk about the brain, but the stuff she says about free will is not a very good argument. Basically it is sort of the argument that the ancient greeks made which brings the discussion into a point where you can take both directions.

mensetmanusman 2y ago

Penrose (Nobel prize in physics) stipulates that quantum effects in the brain may allow a certain amount of time travel and back propagation to accomplish this.

wrsh07 2y ago
You don't need back propagation to learn This is an incredibly complex hypothesis that doesn't really seem justified by the evidence

wrsh07 2y ago

Makes me wonder when an update to the world models paper comes out where they drop in diffusion models: https://worldmodels.github.io/ https://worldmodels.github.io/

dartos 2y ago

> It makes me wonder if humans are just next moment prediction machines, with just a little bit more memory built in. This, to me, seems extremely reductionist. Like you start with AI and work backwards until you frame all cognition as next something predictors. It’s just the stochastic parrot argument again.

bangaladore 2y ago

> It's insane that that this works, and that it works fast enough to render at 20 fps. It is running on an entire v5 TPU (https://cloud.google.com/blog/products/ai-machine-learning/introducing-cloud-tpu-v5p-and-ai-hypercomputer https://cloud.google.com/blog/products/ai-machine-learning/i...) It's unclear how that compares to a high-end consumer GPU like a 3090, but they seem to have similar INT8 TFLOPS. The TPU has less memory (16 vs. 24), and I'm unsure of the other specs. Something doesn't add up, in my opinion, though. SD usually takes (at minimum) seconds to produce a high-quality result on a 3090, so I can't comprehend how they are like 2 orders of magnitudes faster—indicating that the TPU vastly outperforms a GPU for this task. They seem to be producing low-res (320x240) images, but it still seems too fast.

Philpax 2y ago
There's been a lot of work in optimising inference speed of SD - SD Turbo, latent consistency models, Hyper-SD, etc. It is very possible to hit these frame rates now.