7 ms·
Rendering a game in real time with AI
- theknarf 1y agoThis is like playing a boardgame while exclusively looking at the board through a snapchat filter. Who wants this?
- devinprater 1y agoCool, if the game is promptable, one may even be able to make the game accessible.
- sjsdaiuasgdia 1y agoThe "real-time" version looks awful with constantly shifting colors, inconsistently sized objects, and changing interpretations of the underlying data, resulting in what I would consider an unplayable game vs the original ASCII rendering. The "better" version renders at a whopping 4 seconds per frame (not frames per second) and still doesn't consistently represent the underlying data, with shifting interpretations of what each color / region represents.
- faeyanpiraat 1y agoYeah, but I find this fascinating regardless. This is getting into the direction of a kind of simulation where stuff is not determined by code but a kind of "real" physics.
- roxolotl 1y agoWhy does using a language/vision model feel more “real” to you than using equations which directly describe our understandings of physics?
- curl-up 1y agoNot OP, but I have long thought of this type of approach (underlying "hard coded" object tracking + fuzzy AI rendering) to be the next step, so I'll respond. The problem with using equations is that they seem to have plateaued. Hardware requirements for games today keep growing, and yet every character still has that awful "plastic skin", among all the other issues, and for a lot of people (me included) this creates heavy uncanny-valley effects that makes modern games unplayable. On the other hand, images created by image models today look fully realistic. If we assume (and I fully agree that this is a strong and optimistic assumption) that it will soon be possible to run such models in real time, and that techniques for object permanence will improve (as they keep improving at an incredible phase right now), then this might finally bring us to the next level of realism. Even if realism is not what you're aiming for, I think it's easy to imagine how this might change the game.
- jsheard 1y agoYou're comparing apples to oranges, holding up today's practical real-time rendering techniques against a hypothetical future neural method that runs many orders of magnitude faster than anything available today, and solves the issues of temporal stability, directability and overall robustness. If we grant "equation based" methods the same liberty then we should be looking ahead to real-time pathtracing research, which is much closer to anything practical than these pure ML experiments. That's not to say ML doesn't have a place in the pipeline - pathtracers can pair very well with ML-driven heuristics for things like denoising, but in that case the underlying signal is still grounded in physics and the ML part is just papering over the gaps.
- curl-up 1y agoThe question was "why does it feel more real", and I answered that - because the best AI generated images today feel more real than the best 3D renders, even when they take all the compute in the world to finish. So I can imagine that trend going forward into real-time rendering as well. I did not claim that AI-based rendering will overcome traditional methods, and have even explicitly said that this is a heavy assumption, but explained why I see it as exciting.
- jsheard 1y agoI think we'll have to agree to disagree about well done 3D renders not feeling real. Movie studios still regularly underplay how much CGI they use for marketing purposes, and get away with it, because the CGI looks so utterly real that nobody even notices it until much later when the VFX vendors are allowed to give a peek behind the curtain. e.g. Top Gun Mavericks much lauded "practical" jet shots, which were filmed on real planes, but then the pilots were isolated and composited into 100% CGI planes with the backdrops also being CGI in many cases, and huge swathes of viewers and press bought the marketing line that what they saw was all practical.
- sjsdaiuasgdia 1y agoI find it odd that you're that bothered by uncanny valley effects from game rendering but apparently not by the same in image model outputs. They get little things wrong all the time and it puts me off the image almost instantly.
- corysama 1y agoBecause our ability to simulate/render a realistic world in real time using direct equations is still very limited. We’re accustomed to these limitations and often feel “graphics are good enough”. But, we’ll always be decades behind “ILM in real time”. The AI route has a good chance of moving us from decades behind ILM to merely “years behind ILM”.
- johnnyanmac 1y ago>The AI route has a good chance of moving us from decades behind ILM to merely “years behind ILM”. Firstly: we have very accurate models. But not at real time speeds. Games only have some 30,16, or even 11 ms to render a frame. The techniques we have are faking the real physical interactions that render farms can take minutes or hours to pump out per frame. Secondly: Not at this performance rate. taking 100ms to render a frame is unaccatable in the concept of a game. Games are already so pressed for time budget; unless some hyper JIT happens we can't take all that budget querying an LLM.
- actuallyalys 1y agoYeah, as interesting as the concept is, the lack of frame to frame consistency is a real problem. It also seems like the computing requirements would be immense—the article mentions burning through $10 in seconds.
- elpocko 1y agoYou can do this at home on your own computer with a 40x0 consumer GPU at 1-2 fps. You have to choose a suitable diffusion model, there are models that provide sub-second generation of 1024x1024 images. The computing requirements and electricity costs are the same as when running a modern game.
- harph 1y agoIt seems it's because OP is generating the whole screen every frame / every move. Of course that will give inconsistent results. I wonder if this approach would work better: 1. generate the whole screen once 2. on update, create a mask for all changed elements of the underlying data 3. do an inpainting pass with this mask, with regional prompting to specify which parts have changed how 4. when moving the camera, do outpainting This might not be possible with cloud based solutions, but I can see it being possible locally.
- johnfn 1y ago> The "real-time" version looks awful, etc Dang man it's just a guy showing off a neat thing he did for fun. This reaction seems excessive.
- johnnyanmac 1y agocriticism is how we grow. And it seems we're more and more afraid for some reason to criticize when "AI" is involved.
- ozmodiar 1y agoI like the idea behind https://oasis-ai.org/ https://oasis-ai.org/ where you can actually try to take advantage of the 'dream logic' inconsistency of each frame being procedurally generated based on the last one. For example, instead of building a house, build the corner of a house, look at that, then look back up and check if it hallucinated the rest of your ephemeral house for you. Of course that uses AI as the entire gameplay loop and not just a graphics filter. It's also... not great, but an interesting concept that I could see producing a fun dream logic game in the future.
- jschomay 1y agoOP here. Thanks for the feedback. I agree that frame to frame consistency is quite bad currently. I did address that in the post, hinting at some of the techniques others have mentioned here, like in/out-painting and masking previous frames. For me, the exciting parts of this experiment was finding the opportunities and limits of realtime generation, and exploring ways of grounding generated content in a solid yet player controlled world layer.
- curtisszmania 1y ago[dead]
- g105b 1y agoI've been trying to achieve the opposite of this project: render scenes in ASCII/ANSI in the style of old BBS terminal games. I've had terrible success so far. All the AI models I've tried only understand the concept of "pixel art" and not ASCII/ANSI graphics such as what can be seen on https://www.bbsing.com/ https://www.bbsing.com/ , https://16colo.rs https://16colo.rs , or on Reddit's r/ANSIart/ . If anyone has any tips for how I could achieve this, I would love to hear your ideas.
- elpocko 1y agoDo you mean you want to use AI to generate new scenes in ANSI-art style, or do you mean you want to use AI to render pre-existing scenes as ANSI art?
- deleted 1y ago[deleted]
- ticulatedspline 1y agolike this https://asciiart.club/ https://asciiart.club/ ? you don't really need AI for that. if you have to use AI diffusion models are a poor choice for the target. might be easier to train an LLM to output actual text art.
- pjob 1y agoI suspect you'll have much better luck directly implementing the style you want rather than trying to get a (current gen) AI to reliably generate ANSI art. This video is a nice overview of implementing a variation of the style in a post-processing shader: https://www.youtube.com/watch?v=gg40RWiaHRY https://www.youtube.com/watch?v=gg40RWiaHRY
- cbm-vic-20 1y ago...where "ASCII" means an image made up of a grid of elements from a limited set of glyphs.
- tantalor 1y agoAnd those glyphs are not ASCII This is ASCII: https://commons.wikimedia.org/wiki/File:ASCII-Table-wide.svg https://commons.wikimedia.org/wiki/File:ASCII-Table-wide.svg
- mason_mpls 1y agoIf we’re talking about dwarf fortress it uses an old IBM charset, assuming this is some branch off that
- panki27 1y agoI'm pretty sure the generation could easily run locally on a low-to-mid tier graphics card. While it might take a bit longer to generate, you're still saving network and authentication latency.
- echelon 1y agoThis was built in September 2022, and it's still pretty mind-blowing today: https://madebyoll.in/posts/game_emulation_via_dnn/demo/ https://madebyoll.in/posts/game_emulation_via_dnn/demo/ https://madebyoll.in/posts/game_emulation_via_dnn/ https://madebyoll.in/posts/game_emulation_via_dnn/ Hook world state up to a server and you have multiplayer. 2025 update: https://madebyoll.in/posts/world_emulation_via_dnn/ https://madebyoll.in/posts/world_emulation_via_dnn/ https://madebyoll.in/posts/world_emulation_via_dnn/demo/ https://madebyoll.in/posts/world_emulation_via_dnn/demo/
- jschomay 1y agoThat's really cool, thanks for sharing!
- stego-tech 1y ago…but then you just have a graphics card, built to render graphics, that you could tap instead through traditional tooling that’s already widely known and which produces consistent output via local assets. While the results of the experiment here are interesting from an academic standpoint, it’s the same issue as remote game streaming: the amount of time you have to process input from the player, render visuals and sound, and transmit it back to the player precludes remote rendering for all but the most latency-insensitive games and framerates. It’s publishers and IP owners trying to solve the problem of ownership (in that they don’t want anyone to own anything, ever) rather than tackling any actually important issues (such as inefficient rendering pipelines, improving asset compression and delivery methods, improving the sandboxing of game code, etc). Trying to make AI render real-time visuals is the wrongest use of the technology.
- tliltocatl 1y ago
- steveruizok 1y agoWe did a similar thing at tldraw with Draw Fast (https://drawfast.tldraw.com/ https://drawfast.tldraw.com/) and it was very fun. Inspired a few knock offs too. We had to shut it down because it was getting popular on Russian Reddit. A related project Lens (https://lens.tldraw.com https://lens.tldraw.com) also used the same technique, but in a collaborative drawing app. At the peak, when we were streaming back video from Fal and getting <100ms of lag, the setup produced one of the most original creative experiences I’d ever had. I wish these sorts of ultra-fast image generators received more attention and research because they do open up some crazy UX.
- echelon 1y agoLCM is what Krea used to gain massive momentum and raise their first $30M. The tactile reaction to playing with this tech is that it feels utterly sci-fi. It's so freaking cool. Watching videos does not do it justice. Not enough companies or teams are trying this stuff. This is really cool tech, and I doubt we've seen the peak of what real time rendering can do. The rendering artifacts and quality make the utility for production use cases a little questionable, but it can certainly do "art therapy" and dreaming / ideation.
- dvrp 1y agohey thanks for the kind words (krea cofounder here). i agree with that! everyone’s focused on leaderboards and evals but not enough on higher order repercussions imo on a related (but shameless plug) note, did you check our real time video rendering release? > https://x.com/krea_ai/status/1961074072487620635 https://x.com/krea_ai/status/1961074072487620635
- hiatus 1y agoIs there any chance you'd open up the source for those projects so others can play with them?
- Topfi 1y agoThey've already shared it under a none-commercial use license: https://github.com/tldraw/draw-fast https://github.com/tldraw/draw-fast The TLDraw team, from what I have seen, is really open in sharing their experiments under their own license. Not FOSS strictly, but I feel their licensing approach and decision is fair considering how they fund development, know that it's a contentious topic though for some.
- pigpag 1y ago[dead]
- _1tem 1y agoThis is a pre-cursor to a dystopian future where reality will be a game generated in realtime at 60 FPS and streamed to your brain over Neuralink.
- lm28469 1y agoSome people on HN will surely cheer for it, it's the peak of efficiency, you don't even have to move anymore!
- johnnyanmac 1y agoYup, only the 2nd or 3rd most common trope of cyberpunk dystopias. We clearly just want the world to end at this point.
- mason_mpls 1y agoI think that’ll be one of the few good parts of it imho
- jebarker 1y agoIn some ways that’s a lot like how consciousness works isn’t it?
- coolKid721 1y agoI do not get the point of this at all, why not just generate game assets and run them in an engine? With this format there would be no regularity that the thing you saw before will look the same (and that is not a fixable problem). Actually figuring out and improving AI approaches for generating consistent and decent quality game assets is actually something that will be useful, this I have no idea the point of past a tech demo (and for some reason all the "ai game" people do this approach).
- abbycurtis33 1y agoThe tech will improve to far exceed the capabilities of a game engine. Real time improvisation and infinite choices, scope, etc. It makes no sense when people say AI can't do this or that. It will do it next week.
- lukan 1y ago"It makes no sense when people say AI can't do this or that. It will do it next week." So full self driving vecicles will be finally ready next week then? Great to hear, though to be honest, I remain sceptical.
- kayamon 1y agoWaymos are driving themselves around several cities right now.
- pololeono 1y agoYes, go to rome. I will be impressed when we have self driving cars in rome.
- AnotherGoodName 1y agoThey are in San Francisco today so it's not like they are doing this on easy mode.
- sealosa 1y ago[dead]
- mason_mpls 1y agoNow dwarf fortress can eat your CPU, Memory, and GPU. Exciting news.
- beacon294 1y agoI thought so too, but looks like the OP wrote their own game that just has a similar ascii view. However, no reason this can't translate to df!
- bob1029 1y agoThe power consumption of modern gaming is getting a bit out of hand. This AI stuff is taking it to the next level. Ray tracing and other forms of real time global illumination are extremely resource intensive approaches to lighting a scene. Every client machine has to figure out how to light everything every single frame. Contrast this with baked global illumination where the cost is incurred exactly once and is amortized across potentially millions of machines. We need more things like baked GI in gaming. This class of techniques makes the development iterations slower and more deliberate, but it also produces a far more efficient and refined product experience. I'd be very interested in quantifying the carbon impact of realtime vs baked lighting in gaming. It is likely a non-trivial figure at scale. Also bear in mind that baked GI is why games like the batman series still look so good in 2025, even when running on period hardware. You cannot replace your art team by consuming more electricity.
- nh23423fefe 1y agoIt doesn't seem like moralizing about resources matters for velocity or capability. > You cannot replace your art team by consuming more electricity This isn't true.
- johnnyanmac 1y ago>It doesn't seem like moralizing about resources matters for velocity or capability. Not until the Hype bubble pops. Then suddenly it'll be all that matters when you no longer have nigh limitless resources. >This isn't true. This project is interesting, but it is not making a great arguemnt for this.
- Martin_Silenus 1y agoThis kind of lazy lamer behaviour is so unlike Vulkan and hackers... Wait a minute. Where am I?
- johnnyanmac 1y agoVibe News, of course!
- yycettesi 1y ago[flagged]
- VagabundoP 1y agoHow far are we from Ender's Game?
- doawoo 1y ago[flagged]
- mediaman 1y agoIt's interesting to see software engineers realize that AI can be useful in the hands of competent engineers, but that LLMs tend to produce a mess in the hands of those with little software engineering knowledge. Then they're confused why generative AI asset creation doesn't look that good in the hands of people who have no art training. I know someone creating a game, and she is using AI in asset creation. But she's also a highly skilled environment technical artist from the industry, and so the use of AI looks totally different than all these demos from non-artists: different types of narrow AI get injected into different little parts of the asset creation pipeline, and the result is a mix of traditional tools (Substance Designer + Painter, Blender, Maya) with AI support in moodboarding, geometry creation and some parts of texture creation. The result is a 2-5x speedup, but instead of looking like slop it looks like a stylistically distinctive, cohesive world with consistent art direction. The common pattern is that people think AI will automate "other people," because they see its shortcomings in their own field. But because they don't understand the technical skill required in other fields, they assume AI will just "do it." Instead, it seems like AI can be a force multiplier for technically skilled people, but that it begins showing its weakness when asked to take over entire pipelines normally produced by technically skilled people, whether they be engineers or artists.
- doawoo 1y agoReally interesting perspective honestly, I legitimately have no issue with machine learning being used as a tool, like all technology it has benefits! It’s when it’s used as a replacement for creativity that really gets to me.
- aenvoker 1y agoHaving spent a lot of time talking to engineers working on AI tools, I find the idea that AI is intended to replace creativity comes entirely from internet comment sections and not from the engineers working on AI.
- sovietmudkipz 1y agoSci-fi readers who’ve read Ender’s Game will recognize this style of software as similar in concept to the Mind Game Ender Wiggins plays. In the book, the Mind Game renders a game world based on the subject’s mind (conscious, subconscious) in a mechanically similar way to how dreams work for us IRL. I’m excited for AI rendered games.
- gavmor 1y agoThis is awesome! I can definitely see it delivering value, especially over with procedurally generated terrain with 100s or 1000s of different terrain types and combinations—especially if user-defined properties get involved, ie what artists can't predict or prepare for, producing materials via something like https://infinite-craft.gg/ https://infinite-craft.gg/
- babush 1y agoBeautiful