16 ms·
Diffusion models are real-time game engines
- arduinomancer 2y agoHow does the model “remember” the whole state of the world? Like if I kill an enemy in some room and walk all the way across the map and come back, would the body still be there?
- a_e_k 2y agoWatch closely in the videos and you'll see that enemies often respawn when offscreen and sometimes when onscreen. Destroyed barrels come back, ammo count and health fluctuates weirdly, etc. It's still impressive, but its not perfect in that regard.
- Sharlin 2y agoNot unlike in (human) dreams.
- raincole 2y agoIt doesn't. You need to put the world state in the input (the "prompt", even it doesn't look like prompt in this case). Whatever not in the prompt is lost.
- Jensson 2y agoIt doesn't even remember the state of the game you look at. Doors spawning right in front of you, particle effects turning into enemies mid flight etc, so just regular gen AI issues. Edit: Can see this in the first 10 seconds of the first video under "Full Gameplay Videos", stairs turning to corridor turning to closed door for no reason without looking away.
- csmattryder 2y agoThere's also the case in the video (0:59) where the player jumps into the poison but doesn't take damage for a few seconds then takes two doses back-to-back - they should've taken a hit of damage every ~500-1000ms(?) Guessing the model hasn't been taught enough about that, because most people don't jump into hazards.
- deleted 2y ago[deleted]
- jumploops 2y agoThis seems similar to how we use LLMs to generate code: generate, run, fix, generate. Instead of working through a game, it’s building generic UI components and using common abstractions.
- danjl 2y agoSo, diffusion models are game engines as long as you already built the game? You need the game to train the model. Chicken. Egg?
- billconan 2y agomaybe the next step is adding text guidance and generating non-existing games.
- kragen 2y agohere are some ideas: - you could build a non-real-time version of the game engine and use the neural net as a real-time approximation - you could edit videos shot in real life to have huds or whatever and train the neural net to simulate reality rather than doom. (this paper used 900 million frames which i think is about a year of video if it's 30fps, but maybe algorithmic improvements can cut the training requirements down) and a year of video isn't actually all that much—like, maybe you could recruit 500 people to play paintball while wearing gopro cameras with accelerometers and gyros on their heads and paintball guns, so that you could get a year of video in a weekend?
- w_for_wumbo 2y agoThat feels like the endgame of video game generation. You select an art style, a video and the type of game you'd like to play. The game is then generated in real-time responding to each action with respect to the existing rule engine. I imagine a game like that could get so convincing in its details and immersiveness that one could forget they're playing a game.
- vessenes 2y agoSo, this is surprising. Apparently there’s more cause, effect, and sequencing in diffusion models than what I expected, which would be roughly ‘none’. Google here uses SD 1.4, as the core of the diffusion model, which is a nice reminder that open models are useful to even giant cloud monopolies. The two main things of note I took away from the summary were: 1) they got infinite training data using agents playing doom (makes sense), and 2) they added Gaussian noise to source frames and rewarded the agent for ‘correcting’ sequential frames back, and said this was critical to get long range stable ‘rendering’ out of the model. That last is intriguing — they explain the intuition as teaching the model to do error correction / guide it to be stable. Finally, I wonder if this model would be easy to fine tune for ‘photo realistic’ / ray traced restyling — I’d be super curious to see how hard it would be to get a ‘nicer’ rendering out of this model, treating it as a doom foundation model of sorts. Anyway, a fun idea that worked! Love those.
- refibrillator 2y agoJust want to clarify a couple possible misconceptions: The diffusion model doesn’t maintain any state itself, though its weights may encode some notion of cause/effect. It just renders one frame at a time (after all it’s a text to image model, not text to video). Instead of text, the previous states and frames are provided as inputs to the model to predict the next frame. Noise is added to the previous frames before being passed into the SD model, so the RL agents were not involved with “correcting” it. De-noising objectives are widespread in ML, intuitively it forces a predictive model to leverage context, ie surrounding frames/words/etc. In this case it helps prevent auto-regressive drift due to the accumulation of small errors from the randomness inherent in generative diffusion models. Figure 4 shows such drift happening when a player is standing still.
- rvnx 2y agoThe concept is that if you train a Diffusion model by feeding all the possible frames seen in the game. The training was over almost 1 billion frames, 20 days of full-time play-time, taking a screenshot of every single inch of the map. Now you show him N frames as input, and ask it "give me frame N+1", then it gives you the frame n. N+1 back based on how it was originally seen during training. But it is not frame N+1 from a mysterious intelligence, it's simply frame N+1 given back from past database. The drift you mentioned is actually a clear (but sad) proof that the model does not work at inventing new frames, and can only spit out an answer from the past dataset. It's a bit like if you train stable diffusion on Simpsons episodes, and that it outputs the next frame of an existing episode that was in the training set, but few frames later goes wild and buggy.
- zzanz 2y agoThe quest to run doom on everything continues. Technically speaking, isn't this the greatest possible anti-Doom, the Doom with the highest possible hardware requirement? I just find it funny that on a linear scale of hardware specification, Doom now finds itself on both ends.
- Terr_ 2y ago> the Doom with the highest possible hardware requirement? Isn't that possible by setting arbitrarily high goals for ray-cast rendering?
- fngjdflmdflg 2y ago>Technically speaking, isn't this the greatest possible anti-Doom When I read this part I thought you were going to say because you're technically not running Doom at all. That is, instead of running Doom without Doom's original hardware/software environment (by porting it), you're running Doom without Doom itself.
- bugglebeetle 2y agoPierre Menard, Author of Doom.
- el_memorioso 2y agoI applaud your erudition.
- 1attice 2y agothat took a moment, thank you
- airstrike 2y agoOK, this is the single most perfect comment someone could make on this thread. Diffusion me impressed.
- jl6 2y agoKnee Deep in the Death of the Author.
- deleted 2y ago[deleted]
- ravetcofx 2y agoThere is going to be a flood of these dreamlike "games" in the next few years. This feels likes a bit of a breakthrough in the engineering of these systems.
- wkcheng 2y agoIt's insane that that this works, and that it works fast enough to render at 20 fps. It seems like they almost made a cross between a diffusion model and an RNN, since they had to encode the previous frames and actions and feed it into the model at each step. Abstractly, it's like the model is dreaming of a game that it played a lot of, and real time inputs just change the state of the dream. It makes me wonder if humans are just next moment prediction machines, with just a little bit more memory built in.
- Teever 2y agoAlso recursion and nested virtualization. We can dream about dreaming and imagine different scenarios, some completely fictional or simply possible future scenarios all while doing day to day stuff.
- lokimedes 2y agoIt makes good sense for humans to have this ability. If we flip the argument, and see the next frame as a hypothesis for what is expected as the outcome of the current frame, then comparing this "hypothesis" with what is sensed makes it easier to process the differences, rather than the totality of the sensory input. As Richard Dawkins recently put it in a podcast[1], our genes are great prediction machines, as their continued survival rests on it. Being able to generate a visual prediction fits perfectly with the amount of resources we dedicate to sight. If that is the case, what does aphantasia tell us? [1] https://podcasts.apple.com/dk/podcast/into-the-impossible-with-brian-keating/id1169885840?i=1000665776221 https://podcasts.apple.com/dk/podcast/into-the-impossible-wi...
- quickestpoint 2y agoAs Richard Dawkins theorized, would be more accurate and less LLM like :)
- jonplackett 2y agoWhat’s the aphantasia link? I’ve got aphantasia. I’m convinced though that the bit of my brain that should be making images is used for letting me ‘see’ how things are connected together very easily in my head. Also I still love games like Pictionary and can somehow draw things onto paper than I don’t really know what they look like in my head. It’s often a surprise when pen meets paper.
- masterspy7 2y agoThere's been a ton of work to generate assets for games using AI: 3d models, textures, code, etc. None of that may even be necessary with a generative game engine like this! If you could scale this up, train on all games in existence, etc. I bet some interesting things would happen
- rererereferred 2y agoBut can you grab what this Ai has learned and generate the 3d models, maps and code to turn it into an actual game that can run on a user's PC? That would be amazing.
- passion__desire 2y agoJensen Huang's vision that future games will be generated rather than rendered is coming true.
- kleiba 2y agoWhat would be the point? This model has been trained on an existing game, so turning it back into assets, maps, and code would just give you a copy of the original game you started with. I suppose you could create variations of it then... but: You don't even need to do all of that - this trained model already is the game, i.e., it's interactive, you can play the game.
- whamlastxmas 2y agoI would absolutely love if they could take this demo, add a new door that isn’t in the original, and see what it generates behind that door
- refibrillator 2y agoThere is no text conditioning provided to the SD model because they removed it, but one can imagine a near future where text prompts are enough to create a fun new game! Yes they had to use RL to learn what DOOM looks like and how it works, but this doesn’t necessarily pose a chicken vs egg problem. In the same way that LLMs can write a novel story, despite only being trained on existing text. IMO one of the biggest challenges with this approach will be open world games with essentially an infinite number of possible states. The paper mentions that they had trouble getting RL agents to completely explore every nook and corner of DOOM. Factorio or Dwarf Fortress probably won’t be simulated anytime soon…I think.
- mlsu 2y agoWith enough computation, your neural net weights would converge to some very compressed latent representation of the source code of DOOM. Maybe smaller even than the source code itself? Someone in the field could probably correct me on that. At which point, you effectively would be interpolating in latent space through the source code to actually "render" the game. You'd have an entire latent space computer, with an engine, assets, textures, a software renderer. With a sufficiently powerful computer, one could imagine what interpolating in this latent space between, say Factorio and TF2 (2 of my favorites). And tweaking this latent space to your liking by conditioning it on any number of gameplay aspects. This future comes very quickly for subsets of the pipeline, like the very end stage of rendering -- DLSS is already in production, for example. Maybe Nvidia's revenue wraps back to gaming once again, as we all become bolted into a neural metaverse. God I love that they chose DOOM.
- energy123 2y agoThe source code lacks information required to render the game. Textures for example.
- TeMPOraL 2y agoObviously assets would get encoded too, in some form. Not necessarily corresponding to the original bitmaps, if the game does some consistent post-processing, the encoded thing would more likely be (equivalent to) the post-processed state.
- throwmeaway222 2y agoYou know how when you're dreaming and you walk into a room at your house and you're suddenly naked at school? I'm convinced this is the code that gives Data (ST TNG) his dreaming capabilities.
- helloleo2024 2y ago[dead]
- dean2432 2y agoSo in the future we can play FPS games given any setting? Pog
- darrinm 2y agoSo… is it interactive? Playable? Or just generating a video of gameplay?
- vunderba 2y agoFrom the article: We present GameNGen, the first game engine powered entirely by a neural model that enables real-time interaction with a complex environment over long trajectories at high quality. The demo is actual gameplay at ~20 FPS.
- darrinm 2y agoIt confused me that their stated evaluations by humans are comparing video clips rather than evaluating game play.
- furyofantares 2y agoShort clips are the only way a human will make any errors determining which is which.
- kcaj 2y agoTake a bunch of videos of the real world and calculate the differential camera motion with optical flow or feature tracking. Call this the video’s control input. Now we can play SORA.
- piperswe 2y agoThis is honestly the most impressive ML project I've seen since... probably O.G. DALL-E? Feels like a gem in a sea of AI shit.
- bufferoverflow 2y agoThat's probably how our reality is rendered.
- rationalfaith 2y ago[dead]
- broast 2y agoMaybe one day this will be how operating systems work.
- misterflibble 2y agoDon't give them ideas lol terrifying stuff if that happens!
- sitkack 2y agoWhat most programmers don't understand, that in the very near future, the entire application will be delivered by an AI model, no source, no text, just connect to the app over RDP. The whole app will be created by example, the app developer will train the app like a dog trainer trains a dog.
- Grimblewald 2y agothat might work for some applications, especially recreational things, I think we're a while away from it doing away with all things, especially where deterministic behavior, efficiency, or reliability are important.
- sitkack 2y agoProblems for two papers down the line.
- ukuina 2y agoSo... https://websim.ai https://websim.ai except over pixels instead of in your browser?
- sitkack 2y agoYes, and that is super neat.
- Jonovono 2y agoI think it's possible AI models will generate dynamic UI for each client and stream the UI to clients (maybe eventually client devices will generate their UI on the fly) similar to Google Stadia. Maybe some offset of video that allows the remote to control it. Maybe Wasm based - just stream wasm bytecode around? The guy behind VLC is building a library for ulta low latency: https://www.kyber.video/techology https://www.kyber.video/techology. I was playing around with the idea in this: https://github.com/StreamUI/StreamUI https://github.com/StreamUI/StreamUI. Thinking is take the ideas of Elixir LiveView to the extreme.
- dysoco 2y agoAh finally we are starting to see something gaming related. I'm curious as to why we haven't seen more of neural networks applied to games even in a completely experimental fashion; we used to have a lot of little experimental indie games such as Façade (2005) and I'm surprised we don't have something similar years after the advent of LLMs. We could have mods for old games that generate voices for the characters for example. Maybe it's unfeasible from a computing perspective? There are people running local LLMs, no?
- raincole 2y ago> We could have mods for old games that generate voices for the characters for example You mean in real time? Or just in general? There are a lot of mods that use AI-generated voices. I'll say it's the norm of modding community now.
- EcommerceFlow 2y agoJensen said that this is the future of gaming a few months ago fyi.
- deleted 2y ago[deleted]
- mo_42 2y agoAn implementation of the game engine in the model itself is theoretically the most accurate solution for predicting the next frame. I'm wondering when people will apply this to other areas like the real world. Would it learn the game engine of the universe (ie physics)?
- radarsat1 2y agoThere has definitely been research for simulating physics based on observation, especially in fluid dynamics but also for rigid body motion and collision. It's important for robotics applications actually. You can bet people will be applying this technique in those contexts. I think for real world application one challenge is going to be the "action" signal which is a necessary component of the conditioning signal that makes the simulation reactive. In video games you can just record the buttons, but for real world scenarios you need difficult and intrusive sensor setups for recording force signals. (Again for robotics though maybe it's enough to record the motor commands, just that you can't easily record the "motor commands" for humans, for example)
- cubefox 2y agoA popular theory in neuroscience is that this is what the brain does: https://slatestarcodex.com/2017/09/05/book-review-surfing-uncertainty/ https://slatestarcodex.com/2017/09/05/book-review-surfing-un... It's called predictive coding. By trying to predict sensory stimuli, the brain creates a simplified model of the world, including common sense physics. Yann LeCun says that this is a major key to AGI. Another one is effective planning. But while current predictive models (autoregressive LLMs) work well on text, they don't work well on video data, because of the large outcome space. In an LLM, text prediction boils down to a probability distribution over a few thousand possible next tokens, while there are several orders of magnitude more possible "next frames" in a video. Diffusion models work better on video data, but they are not inherently predictive like causal LLMs. Apparently this new Doom model made some progress on that front though.
- ccozan 2y agoHowver, this is due how we actually digitize video. From a human point a view, looking in my room reduces the load to the _objects_ in the room and everyhing else is just noise ( like the color of the wall could be just a single item to remember, while otherwise in the digital world, it needs to remember all the pixels )
- helloplanets 2y agoSo, any given sequence of inputs is rebuilt into a corresponding image, twenty times per second. I wonder how separate the game logic and the generated graphics are in the fully trained model. Given a sufficient enough separation between these two, couldn't you basically boil the game/input logic down to an abstract game template? Meaning, you could just output a hash that corresponds to a specific combination of inputs, and then treat the resulting mapping as a representation of a specific game's inner workings. To make it less abstract, you could save some small enough snapshot of the game engine's state for all given input sequences. This could make it much less dependent to what's recorded off of the agents' screens. And you could map the objects that appear in the saved states to graphics, in a separate step. I imagine this whole system would work especially well for games that only update when player input is given: Games like Myst, Sokoban, etc.
- toppy 2y agoI think you've just encoded the title of the paper
- deleted 2y ago[deleted]
- richard___ 2y agoUhhh… demos would be more convincing with enemies and decreasing health
- Kiro 2y agoI see enemies and decreasing health on hit. But even if it lacked those, it seems like a pretty irrelevant nitpick that is completely underplaying what we're seeing here. The fact that this is even possible at all feels like science fiction.
- troupo 2y agoKey: "predicts next frame, recreates classic Doom". A game that was analyzed and documented to death. And the training included uncountable runs of Doom. A game engine lets you create a new game, not predict the next frame of an existing and copiously documented one. This is not a game engine. Creating a new good game? Good luck with that.
- nolist_policy 2y agoMakes me wonder... If you stand still in front of a door so all past observations only contain that door, will the model teleport you to another level when opening the door?
- zbendefy 2y agoI think some state is also being given (or if its not, it could be given) to the network, like 3d world position/orientation of the player, that could help the neural network anchor the player in the world.
- lukol 2y agoI believe future game engines will be state machines with deterministic algorithms that can be reproduced at any time. However, rendering said state into visual / auditory / etc. experiences will be taken over by AI models. This will also allow players to easily customize what they experience without changing the core game loop.
- jamilton 2y agoI wonder if the MineRL (https://www.ijcai.org/proceedings/2019/0339.pdf https://www.ijcai.org/proceedings/2019/0339.pdf and minerl.io) dataset would be sufficient to reproduce this work with Minecraft. Any other similar existing datasets? A really goofy way I can think of to get a bunch of data would be to get videos from youtube and try to detect keyboard sounds to determine what keys they're pressing.
- jamilton 2y agoAlthough ideally a follow up work would be something where there won’t be any potential legal trouble with releasing the complete model so people can play it. A similar approach but with a game where the exact input is obvious and unambiguous from the graphics alone so that you can use unannotated data might work. You’d just have to create a model to create the action annotations. I’m not sure what the point would be, but it sounds like it’d be interesting.
- qnleigh 2y agoCould a similar scheme be used to drastically improve the visual quality of a video game? You would train the model on gameplay rendered at low and high quality (say with and without ray tracing, and with low and high density meshing), and try to get it to convert a quick render into something photorealistic on the fly. When things like DALL-E first came out, I was expecting something like the above to make it into mainstream games within a few years. But that was either too optimistic or I'm not up to speed on this sort of thing.
- agys 2y agoIsn't that what Nvidia’s Ray Reconstruction and DLSS (frame generation and upscaler) are doing, more or less?
- qnleigh 2y agoAt a high level I guess so. I don't know enough about Ray Reconstruction (though the results are impressive), but I was thinking of something more drastic than DLSS. Diffusion models on static images can turn a cartoon into a photorealistic image. Doing something similar for a game, where a low-quality render is turned into something that would otherwise take seconds to render, seems qualitatively quite different from DLSS. In principle a model could fill in huge amounts of detail, like increasing the number of particles in a particle-based effect, adding shading/lighting effects...
- lIl-IIIl 2y agoHow does it know how many times it needs to shoot the zombie before it dies? Most enemies have enough hit points to survive the first shot. If the model is only trained on the previous frame, it doesn't know how many times the enemy was already shot at. From the video it seems like it is probability based - they may die right away or it might take way longer than it should. I love how the player's health goes down when he stands in the radioactive green water. In Doom the enemies fight with each other if they accidentally incur "friendly fire". It would be interesting to see it play out in this version.
- golol 2y agoIt gets a number of previous frames as input I think.
- meheleventyone 2y ago> I love how the player's health goes down when he stands in the radioactive green water. This is one of the bits that was weird to me, it doesn't work correctly. In the real game you take damage at a consistent rate, in the video the player doesn't and whether the player takes damage or not seems highly dependent on some factor that isn't whether or not the player is in the radioactive slime. My thought is that its learnt something else that correlates poorly.
- lupusreal 2y ago> In Doom the enemies fight with each other if they accidentally incur "friendly fire". It would be interesting to see it play out in this version. They trained this thing on bot gameplay, so I bet it does poorly when advanced strategies like deliberately inducing mob infighting are employed (the bots probably didn't do that a lot, of at all.)
- golol 2y agoWhat I understand is the folloeing: If this works so well, why didn't we have good video generation much earlier? After diffusion models were seen to work the most obvious thing to do was to generate the next frame based on previous framrs but... it took 1-2 years for good video models to appear. For example compare Sora generating minecraft video versus this method generating minecraft video. Say in both cases the player is standing on a meadow with fee inputs and watching some pigs. In the Sora video you'd expect the typical glitched to appear, like erratic, sliding movement, overlapping legs, multiplication of pigs etc. Would these glitches not appear in the GameNGen video? Why?
- Closi 2y agoBecause video is much more difficult than images (it's lots of images that have to be consistent across time, with motion following laws of physics etc), and this is much more limited in terms of scope than pure arbitrary video generation.
- golol 2y agoThis misses the point, I'm comparing two methods of generating minecraft videos.
- soulofmischief 2y agoBy simplifying the problem, we are better able to focus on researching specific aspects of generation. In this case, they synthetically created a large, highly domain-specific training set and then used this to train a diffusion model which encodes input parameters instead of text. Sora was trained on a much more diverse dataset, and so has to learn more general solutions in order to maintain consistency, which is harder. The low resolution and simple, highly repetitive textures of doom definitely help as well. In general, this is just an easier problem to approach because of the more focused constraints. It's also worth mentioning that noise was added during the process in order to make the model robust to small perturbations.
- pantalaimon 2y ago
- golol 2y agoCertain categories of youtube videos can also be viewed as some sort of game where the actions are the audio/transcript advanced a couple of seconds. Add two eggs. Fetch the ball. I'm walking in the park.
- thegabriele 2y agoWow, I bet Boston Dynamics and such are quite interested
- HellDunkel 2y agoAlthough impressive i must disagree. Diffusion models are not game engines. A game engine is a component to propell your game (along the time axis?). In that sense it is similar to the engine of the car, hence the name. It does not need a single working car nor a road to drive on do its job. The above is a dynamic, interactive replication of what happens when you put a car on a given road, requiring a million test drives with working vehicles. An engine would also work offroad.
- MasterScrat 2y agoInteresting point. In a way this is a "simulated game engine", trained from actual game engine data. But I would argue a working simulated game engine becomes a game engine of its own, as it is then able to "propell the game" as you say. The way it achieves this becomes irrelevant, in one case the content was crafted by humans, in the other case it mimics existing game content, the player really doesn't care! > An engine would also work offroad. Here you could imagine that such a "generative game engine" could also go offroad, extrapolating what would happen if you go to unseen places. I'd even say extrapolation capabilities of such a model could be better than a traditional game engine, as it can make things up as it goes, while if you accidentally cross a wall in a typical game engine the screen goes blank.
- jsheard 2y ago> Here you could imagine that such a "generative game engine" could also go offroad, extrapolating what would happen if you go to unseen places. They easily could have demonstrated this by seeding the model with images of Doom maps which weren't in the training set, but they chose not to. I'm sure they tried it and the results just weren't good, probably morphing the map into one of the ones it was trained on at the first opportunity.
- HellDunkel 2y agoThe game doom is more than a game engine, isnt it? I‘d be okay with calling the above a „simulated game“ or a „game“. My point is: let‘s not conflate the idea of a „game engine“ which is a construct of intellectual concepts put together to create a simulation of „things happening in time“ and deriving output (audio and visual). the engine is fed with input and data (levels and other assets) and then drives(EDIT) a „game“. training the model with a final game will never give you an engine. maybe a „simulated game“ or even a „game“ but certainly not an „engine“. the latter would mean the model would be capable to derive and extract the technical and intellectual concepts and apply them elsewhere.
- icoder 2y agoThis is impressive. But at the same time, it can't count. We see this every time, and I understand why it happens, but it is still intriguing. We are so close or in some ways even way beyond, and yet at the same time so extremely far away, from 'our' intelligence. (I say it can't count because there are numerous examples where the bullet count glitches, it goes right impressively often, but still, counting, being up or down, is something computers have been able to do flawlessly basically since forever) (It is the same with chess, where the LLM models are becoming really good, yet sometimes make mistakes that even my 8yo niece would not make)
- marci 2y ago'our' intelligence may not be the best thing we can make. It would be like trying to only make planes that flaps wings or trucks with legs. A bit like using a llm to do multiplication. Not the best tool. Biomimcry is great for inspiration, but shouldn't be a 1-to-1 copy, especialy in different scale and medium.
- icoder 2y agoSure, although I still think a system with less of a contrast between how well it performs 'modally' and how bad it performs incidentally, would be more practical. What I wonder is whether LLM's will inherently always have this dichotomy and we need something 'extra' (reasoning, attention or something les biomimicried), or whether this will eventually resolves itself (to an acceptable extend) when they improve even further.
- panki27 2y ago> Human raters are only slightly better than random chance at distinguishing short clips of the game from clips of the simulation. I can hardly believe this claim, anyone who has played some amount of DOOM before should notice the viewport and textures not "feeling right", or the usually static objects moving slightly.
- meheleventyone 2y agoIt's telling IMO that they only want people opinions based on our notoriously faulty memories rather than sitting comparable situations next to one another in the game and simulation then analyzing them. Several things jump out watching the example video.
- GaggiX 2y ago>rather than sitting comparable situations next to one another in the game and simulation then analyzing them. That's literally how the human rating was setup if you read the paper.
- meheleventyone 2y agoI think you misunderstand me. I don't mean a snap evaluation and deciding between two very-short competing videos which is what the participants were doing. I mean doing an actual analysis of how well the simulation matches the ground truth of the game. What I'd posit is that it's not actually a very good replication of the game but very good a replicating short clips that almost look like the game and the short time horizons are deliberately chosen because the authors know the model lacks coherence beyond that.
- GaggiX 2y ago>I mean doing an actual analysis of how well the simulation matches the ground truth of the game. Do you mean the PSNR and LPIPS metrics used in paper?
- holoduke 2y agoI saw a video a while ago where they recreated actual doom footage with a diffusion technique so it looked like a jungle or anything you liked. Cant find it anymore, but looked impressive.
- godelski 2y agoDoom system requirements: - 4 MB RAM - 12 MB disk space Stable diffusion v1 > 860M UNet and CLIP ViT-L/14 (540M) Checkpoint size: 4.27 Gb 7.7 GB (full EMA) Running on a TPU-v5e Peak compute per chip (bf16) 197 TFLOPs Peak compute per chip (Int8) 393 TFLOPs HBM2 capacity and bandwidth 16 GB, 819 GBps Interchip Interconnect BW 1600 Gbps This is quite impressive, especially considering the speed. But there's still a ton of room for improvement. It seems it didn't even memorize the game despite having the capacity to do so hundreds of times over. So we definitely have lots of room for optimization methods. Though who knows how such things would affect existing tech since the goal here is to memorize. What's also interesting about this work is it's basically saying you can rip a game if you're willing to "play" (automate) it enough times and spend a lot more on storage and compute. I'm curious what the comparison in cost and time would be if you hired an engineer to reverse engineer Doom (how much prior knowledge do they get considering pertained models and visdoom environment. Was doom source code in T5? And which vit checkpoint was used? I can't keep track of Google vit checkpoints). I would love to see the checkpoint of this model. I think people would find some really interesting stuff taking it apart. - https://www.reddit.com/r/gaming/comments/a4yi5t/original_doom_system_requirements_from_my_25_year/ https://www.reddit.com/r/gaming/comments/a4yi5t/original_doo... - https://huggingface.co/CompVis/stable-diffusion-v-1-4-original/tree/main https://huggingface.co/CompVis/stable-diffusion-v-1-4-origin... - https://cloud.google.com/tpu/docs/v5e https://cloud.google.com/tpu/docs/v5e - https://github.com/Farama-Foundation/ViZDoom https://github.com/Farama-Foundation/ViZDoom - https://zdoom.org/index https://zdoom.org/index
- snickmy 2y agoThose are valid points, but irrelevant for the context of this research. Yes, the computational cost is ridicolous compared to the original game, and yes, it lacks basic things like pre-computing, storing, etc. That said, you could assume that all that can be either done at the margin of this discovery OR over time will naturally improve OR will become less important as a blocker. The fact that you can model a sequence of frames with such contextual awareness without explictly having to encode it, is the real breakthrough here. Both from a pure gaming standpoint, but on simulation in general.
- JDEngi 2y agoThis is going to be the future of cloud gaming, isn't it? In order to deal with the latency, we just generate the next frame locally, and we'll have the true frame coming in later from the cloud, so we're never dreaming too far ahead of the actual game.
- amelius 2y agoYes, and you can use an LLM to simulate role playing games.
- deleted 2y ago[deleted]
- amunozo 2y agoThis is amazing and an interesting discovery. It is a pity that I don't find it capable of creating anything new.
- itomato 2y agoThe gibs are a dead giveaway
- aaron695 2y ago[dead]
- nuz 2y agoI wonder how overfit it is though. You could fit a lot of doom resolution jpeg frames into 4gb (the size of SD1.4)
- ciroduran 2y agoCongrats on running Doom on an Diffusion Model :D I was really entranced on how combat is rendered (the grunt doing weird stuff in very much the style that the model generates images). Now I'd like to see this implemented in a shader in a game
- deleted 2y ago[deleted]
- LtdJorge 2y agoSo is it taking inputs from a player and simulating the gameplay or is it just simulating everything (effectively, a generated video)?
- smusamashah 2y agoHas this model actually learned the 3d space of the game? Is it possible to break the camera free and roam around the map freely and view it from different angles? I noticed a few hallucinations e.g. when it picked green jacket from a corner, walking back it generated another corner. Therefore I don't think it has any clue about the 3D world of the game at all.
- kqr 2y ago> Is it possible to break the camera free and roam around the map freely and view it from different angles? I would assume only if the training data contained this type of imagery, which it did not. The training data (from what I understand) consisted only of input+video of actual gameplay, so that is what the model is trained to mimick. This is like a dog that has been trained to form English words – what's impressive is not that it does it well, but that it does it at all.
- Sohcahtoa82 2y ago> Therefore I don't think it has any clue about the 3D world of the game at all. AI models don't "know" things at all. At best, they're just very fuzzy predictors. In this case, given the last couple frames of video and a user input, it predicts the next frame. It has zero knowledge of the game world, game rules, interactions, etc. It's merely a mapping of [pixels, input] -> pixels.
- kqr 2y agoI have been kind of "meh" about the recent AI hype, but this is seriously impressive. Of course, we're clearly looking at complete nonsense generated by something that does not understand what it is doing – yet, it is astonishingly sensible nonsense given the type of information it is working from. I had no idea the state of the art was capable of this.
- acoye 2y agoNvidia CEO reckons your GPU will be replaced with AI in “5-10 years”. So this is what the sort of first working game I guess.
- acoye 2y agoI'd love to see John Carmack come back from his AGI hiatus and advance AI based rendering. This would be supper cool.
- seydor 2y agoI wonder how far it is from this to generating language reasoning about the game from the game itself, rather than learning a large corpus of language, like LLMs do. That would be a true grounded language generator
- t1c 2y agoThey got DOOM running on a diffusion engine before GTA 6
- david1542 2y ago[dead]
- lackoftactics 2y agoI think Alan's conservative countdown to AGI will need to be updated after this. https://lifearchitect.ai/agi/ https://lifearchitect.ai/agi/ This is really impressive stuff. I thought about it a couple of months ago, that probably this is the next modality worth exploring for data, but didn't imagine it would come so fast. On the other side, the amount of compute required is crazy.
- joseferben 2y agoimpressive, imagine this but photo realistic with vr goggles.
- random2024 2y ago[dead]
- gwbas1c 2y agoAm I the only one who thinks this is faked? It's not that hard to fake something like this: Just make a video of DOSBox with DOOM running inside of it, and then compress it with settings that will result in compression artifacts.
- GaggiX 2y ago>Am I the only one who thinks this is faked? Yes.
- dtagames 2y agoA diffusion model cannot be a game engine because a game engine can be used to create new games and modify the rules of existing games in real time -- even rules which are not visible on-screen. These tools are fascinating but, as with all AI hype, they need a disclaimer: The tool didn't create the game. It simply generated frames and the appearance of play mechanics from a game it sampled (which humans created).
- kqr 2y ago> even rules which are not visible on-screen. If a rule was changed but it's never visible on the screen, did it really change? > It simply generated frames and the appearance of play mechanics from a game it sampled (which humans created). Simply?! I understand it's mechanically trivial but the fact that it's compressed such a rich conditional distribution seems far from simple to me.
- darby_nine 2y ago> Simply?! I understand it's mechanically trivial but the fact that it's compressed such a rich conditional distribution seems far from simple to me. It's much simpler than actually creating a game....
- stnmtn 2y agoIf someone told you 10 years ago that they were going to create something where you could play a whole new level of Doom, without them writing a single line of game logic/rendering code, would you say that that is simpler than creating a demo by writing the game themselves?
- darby_nine 2y agoThere are two things at play here: the complexity of the underlying mechanism, and the complexity of detailed creation. This is obviously a complicated mechanism, but in another sense it's a trivial result compared to actually reproducing the game itself in its original intended state.
- alkonaut 2y agoThe job of the game engine is also to render the world given only the worlds properties (textures, geometries, physics rules, ...), and not given "training data that had to be supplied from an already written engine". I'm guessing that the "This door requires a blue key" doesn't mean that the user can run around, the engine dreams up a blue key in some other corner of the map, and the user can then return to the door and the engine now opens the door? THAT would be impressive. It's interesting to think that all that would be required for that task to go from really hard to quite doable, would be that the door requiring the blue key is blue, and the UI showing some icon indicating the user possesses the blue key. Without that, it becomes (old) hidden state.
- dabochen 2y agoSo there is no interactivity, but the generated content is not the exact view in the training data, is this the correct understanding? If so, is it more like imagination/hallucination rather than rendering?
- famouswaffles 2y agoIt's conditioned on previous frames AND player actions so it's interactive.
- wantsanagent 2y agoAnyone have reliable numbers on the file sizes here? Doom.exe from my searches was around 715k, and with all assets somewhere around 10MB. It looks like the SD 1.4 files are over 2GB, so it's likely we're looking at a 200-2000x increase in file size depending on if you think of this as an 'engine' or the full game.
- YeGoblynQueenne 2y agoMisleading Titles Are Everywhere These Days.
- jetrink 2y agoWhat if instead of a video game, this was trained on video and control inputs from people operating equipment like warehouse robots? Then an automated system could visualize the result of a proposed action or series of actions when operating the equipment itself. You would need a different model/algorithm to propose control inputs, but this would offer a way for the system to validate and refine plans as part of a problem solving feedback loop.
- Workaccount2 2y ago>Robotic Transformer 2 (RT-2) is a novel vision-language-action (VLA) model that learns from both web and robotics data, and translates this knowledge into generalised instructions for robotic control https://deepmind.google/discover/blog/rt-2-new-model-translates-vision-and-language-into-action/ https://deepmind.google/discover/blog/rt-2-new-model-transla...
- Abismith 2y ago[dead]
- harha_ 2y agoThis is so sick I don't know what to say. I never expected this, aren't the implications of this huge?
- aithrowaway1987 2y agoI am struggling to understand a single implication of this! How does this generalize to anything other than other than playing retro games in the most expensive way possible? The very intention of this project is overfitting to data in a non-generalizable way! Maybe it's just pure engineering, that good ANNs are getting cheap and fast. But this project still seems to have the fundamental weaknesses of all AI projects: - needs a huge amount of data, which a priori precludes a lot of interesting use cases - flashy-but-misleading demos which hide the actual weaknesses of the AI software (note that the player is moving very haltingly compared to a real game of DOOM, where you almost never stop moving) - AI nailing something really complicated for humans (98% effective raycasting, 98% effective Python codegen) while failing to grasp abstract concepts rigorously understood by fish (object permanence, quantity) I am genuinely struggling to see this as a meaningful step forward. It seems more like a World's Fair exhibit - a fun and impressive diversion, but probably not a vision of the future. Putting it another way: unlike AlphaGo, Deep Blue wasn't really a technological milestone so much as a sociological milestone reflecting the apex of a certain approach to AI. I think this DOOM project is in a similar vein.
- harha_ 2y agoI agree with you, when I made this comment I was simply excited but that didn't last too long. I find this technology both exciting and dystopian, the latter because the dystopic use of it is already happening all over the internet. For now, it's been used only for entertainment AFAIK, which is the kind of use I don't like either, because I prefer human created entertainment over this crap.
- KETpXDDzR 2y agoI think the correct title should be "Diffusion Models Are Fake Real-Time Game Engines". I don't think just more training will ever be sufficient to create a complete game engine. It would need to "understand" what it's doing.
- aghilmort 2y agolooking forward to &/or wondering about overlap with notion of ray tracing LLMs
- TheRealPomax 2y agoIf by "game" you mean "literal hallucination" then yes. But if we're not trying to click-bait, then no: it's not really a game when there is no permanence or determinism to be found anywhere. It might be a "game-flavoured dream simulator", but it's absolutely not a game engine.
- rrnechmech 2y ago> To mitigate auto-regressive drift during inference, we corrupt context frames by adding Gaussian noise to encoded frames during training. This allows the network to correct information sampled in previous frames, and we found it to be critical for preserving visual stability over long time periods. I get this (mostly). But would any kind soul care to elaborate on this? What is this "drift" they are trying to avoid and how does (AFAIU) adding noise help?
- gwern 2y agoPeople may recall GameGAN from May 2020: https://arxiv.org/abs/2005.12126#nvidia https://arxiv.org/abs/2005.12126#nvidia https://nv-tlabs.github.io/gameGAN/#nvidia https://nv-tlabs.github.io/gameGAN/#nvidia https://github.com/nv-tlabs/GameGAN_code https://github.com/nv-tlabs/GameGAN_code
- SeanAnderson 2y agoAfter some discussion in this thread, I found it worth pointing out that this paper is NOT describing a system which receives real-time user input and adjusts its output accordingly, but, to me, the way the abstract is worded heavily implied this was occurring. It's trained on a large set of data in which agents played DOOM and video samples are given to users for evaluation, but users are not feeding inputs into the simulation in real-time in such a way as to be "playing DOOM" at ~20FPS. There are some key phrases within the paper that hint at this such as "Key questions remain, such as ... how games would be effectively created in the first place, including how to best leverage human inputs" and "Our end goal is to have human players interact with our simulation.", but mostly it's just the omission of a section describing real-time user gameplay.
- bob1029 2y agoWere the agents playing at 20 real FPS, or did this occur like a Pixar movie offline?
- refibrillator 2y agoYou are incorrect, this is an interactive simulation that is playable by humans. > Figure 1: a human player is playing DOOM on GameNGen at 20 FPS. The abstract is ambiguously worded which has caused a lot of confusion here, but the paper is unmistakably clear about this point. Kind of disappointing to see this misinformation upvoted so highly on a forum full of tech experts.
- FrustratedMonky 2y agoYeah. If isn't doing this, then what could it be doing that is worth a paper? "real-time user input and adjusts its output accordingly"
- rvnx 2y agoThere is a hint in the paper itself: It says in a shy way that it is based on: "Ha & Schmidhuber (2018) who train a Variational Auto-Encoder (Kingma & Welling, 2014) to encode game frames into a latent vector" So it means they most likely took https://worldmodels.github.io/ https://worldmodels.github.io/ (that is actually open-source) or something similar and swapped the frame generation by Stable Diffusion that was released in 2022.
- Sohcahtoa82 2y agoIt's always fun reading the dead comments on a post like this. People love to point how how pointless this is. Some of ya'll need to learn how to make things for the fun of making things. Is this useful? No, not really. Is it interesting? Absolutely. Not everything has to be made for profit. Not everything has to be made to make the world a better place. Sometimes, people create things just for the learning experience, the challenge, or they're curious to see if something is possible. Time spent enjoying yourself is never time wasted. Some of ya'll are going to be on your death beds wishing you had allowed yourself to have more fun.
- ninetyninenine 2y agoI don’t think this is not useful. This is a stepping stone for generating entire novel games.
- Sohcahtoa82 2y ago> This is a stepping stone for generating entire novel games. I don't see how. This game "engine" is purely mapping [pixels, input] -> new pixels. It has no notion of game state (so you can kill an enemy, turn your back, then turn around again, and the enemy could be alive again), not to mention that it requires the game to already exist in order to train it. I suppose, in theory, you could train the network to include game state in the input and output, or potentially even handle game state outside the network entirely and just make it one of the inputs, but the output would be incredibly noisy and nigh unplayable. And like I said, all of it requires the game to already exist in order to train the network.
- airstrike 2y ago> (so you can kill an enemy, turn your back, then turn around again, and the enemy could be alive again) Sounds like a great game. > not to mention that it requires the game to already exist in order to train it Diffusion models create new images that did not previously exist all of the time, so I'm not sure how that follows. It's not hard to extrapolate from TFA to a model that generically creates games based on some input
- KhoomeiK 2y agoNVIDIA did something similar with GANs in 2020 [1], except users could actually play those games (unlike in this diffusion work which just plays back simulated video). Sentdex later adapted this to play GTA with a really cool demo [2]. [1] https://research.nvidia.com/labs/toronto-ai/gameGAN/ https://research.nvidia.com/labs/toronto-ai/gameGAN/ [2] https://www.youtube.com/watch?v=udPY5rQVoW0 https://www.youtube.com/watch?v=udPY5rQVoW0
- throwthrowuknow 2y agoSeveral thoughts for future work: 1. Continue training on all of the games that used the Doom engine to see if it is capable of creating new graphics, enemies, weapons, etc. I think you would need to embed more details for this perhaps information about what is present in the current level so that you could prompt it to produce a new level from some combination. 2. Could embedding information from the map view or a raytrace of the surroundings of the player position help with consistency? I suppose the model would need to predict this information as the neural simulation progressed. 3. Can this technique be applied to generating videos with consistent subjects and environments by training on a camera view of a 3D scene and embedding the camera position and the position and animation states of objects and avatars within the scene? 4. What would the result of training on a variety of game engines and games with different mechanics and inputs be? The space of possible actions is limited by the available keys on a keyboard or buttons on a controller but the labelling of the characteristics of each game may prove a challenge if you wanted to be able to prompt for specific details.
- danielmarkbruce 2y agoWhat is the point of this? It's hard to see how this is useful. Maybe it's just an exercise to show what a diffusion model can do?
- Kapura 2y agoWhat is useful about this? I am a game programmer, and I cannot imagine a world where this improves any part of the development process. It seems to me to be a way to copy a game without literally copying the assets and code; plagiarism with extra steps. What am I missing?
- jasonkstevens 2y agoAI no longer plays Doom-it is Doom.
- maxglute 2y agoRL tetris effect hallucination. Wish there was 1000s of hours of hardcore henry to train. Maybe scrape gopro war cams.
- lynx23 2y agoHehe, this sounds like the backstory of a remake of the Terminator, or "I have no mouth, but I must scream." In the aftermath of AI killing off humanity, researchers look deeply into how this could have ahppened. And after a number of dead ends, they finally realize: it was trained, in its infancy, on Doom!
- nicman23 2y agowhat i want from something like this is a mix. a model that can infinitely "zoom" into an object's texture which even if not perfect it would be fine and a model that would create 3d geometry from bump maps / normals
- mobiuscog 2y agoVideo Game streamers are next in line to be replaced by AI I guess.
- rldjbpin 2y agothis is truly a cool demo, but a very misleading title. to me it seems like a very bruteforce or greedy way to give the impression to a user that they are "playing" a game. the difference being that you already own the game to make this possible, but cannot let the user use that copy! using generative AI for game creation is at a nascent stage but there are much more elegant ways to go about the end goal. perhaps in the future with computing so far ahead that we moved beyond the current architecture, this might be worth doing instead of emulation perhaps.
- Marciakhan 2y ago[dead]
- yair99dd 2y agoYotube user hu-po streams critical in-depth streams of Ai papers. Here is his take on this (and other relevant) paper https://www.youtube.com/live/JZgqQB4Aekc https://www.youtube.com/live/JZgqQB4Aekc