7 ms·

MeshGPT: Generating triangle meshes with decoder-only transformers

• carbocation 3y ago
  On my phone so I’ve only read this promo page - could this approach be modified for surface reconstruction from a 3D point cloud?
• chongli 3y ago
  This looks really cool! Seems like it would be an incredible boon for an indie game developer to generate a large pool of assets!
  • stuckinhell 3y ago
    I think indie game development is dead with these techniques. Instead big companies will create "make your own game" games. Indie games already seems pretty derivative these days. I think this tech will kill them in mid-term as big companies use them.
    • CamperBob2 3y ago
      For values of "dead" equal to "Now people who aren't 3D artists and can't afford to hire them will be able to make games," maybe. User name checks out.
      • stuckinhell 3y ago
        AI is already taking video game illustrators’ jobs in China https://restofworld.org/2023/ai-image-china-video-game-layoffs/ https://restofworld.org/2023/ai-image-china-video-game-layof... It feels like a countdown until every creative in the videogame industry is automated.
    • owenpalmer 3y ago
      People who use "make your own game" games aren't good at making games. They might enjoy a simplified process to feel the accomplishment of seeing quick results, but I find it unlikely they'll be competing with indie developers.
      • CaptainFever 3y ago
        Yeah, and if there was going to be such a tool, people who invest more time in it would be better than those casually using it. In other words, professionals.
        • mattigames 3y ago
          Not really, "I" can make 2D pictures that look like masterpieces using stable diffusion and didn't invest more than 6 hours playing with it, the learning curve is not that high, and people already have a hard time telling apart AI art than those from real 2D masters who have a lifetime learning it, the same thing will happen with making videogames and 3D art.(Yeah nothing of this looks exiting to me, actually it looks completely bleak)
          • CaptainFever 3y ago
            I didn't mean comparing it to human-created art, I meant comparing it to other AI generated or assisted artworks. Currently the hard parts of that would probably be consistency, fidelity (e.g. multiple characters) and control, which definitely stands out when compared against the casual raw gens.
      • CamperBob2 3y ago
        Careful with that generalization. Game-changing FPS mods like Counterstrike were basically "make your own game" projects, built with the highest-level toolkits imaginable (editors for existing commercial games.)
    • chongli 3y ago
      “Make your own game” games will never replace regular games. They target totally different interests. People who play games (vast majority) just want to play an experience created by someone else. People who like “make your own game” games are creative types who just use that as a jumping off point to becoming a game designer. It’s no different than saying “these home kitchen appliances are really gonna kill off the restaurant industry.”
      • stuckinhell 3y ago
        Hmm I think it will destroy the market in a couple ways. AI creating video games would drastically increase the volume of games available in the market. This surge in supply could make it harder for indie games to stand out, especially if AI-generated games are of high quality or novelty. It could also lead to even more indie saturation( the average indie makes less than 1000 dollars). As the market expectations shift, I think most indie development dies unless you are already rich or basically have patronage from rich clients.
        • crq-yml 3y ago
          The likes of itch.io, Roblox, and the App Store already exist, each with more games than anyone can reasonably curate. The games market has been in the same place as the rest of the arts for some time now: if you want to be noticed, you have to mount a bit of a production around it, add layers of design effort, and find a marketing funnel for that particular audience. The days of just making a Pong clone passed in the 1970's. What technology has done to the arts, historically, is add either more precision or more repeatability. The relationship to production and arts as a business maps to what kinds of capital-and-labor-intensive endeavors leverage the tech. Photographs didn't end painting, they ended painting as the ideal of precisely representational art. In the classical era, just before the tech was good enough to switch, painting was a process of carefully staging a scene with actors and sketching it using a camera obscura to trace details, then transferring the result to your canvas. Afterwards, the exact scene could be generated precisely in a photo, and so a more candid, informal method became possible both through using photographs directly and using them as reference. As well, exact copies of photographs could be manufactured. What changed was that you had a repeatable way of getting a precise result, and so getting the precision or the product itself became uninteresting. But what happened next was that movies and comics were invented, and they brought us back to a place of needing production: staged scenes, large quantities of film or illustration, etc. With generative AI, you are getting a clip art tool - a highly repeatable way of getting a generic result. If you want the design to be specific, you still have to stage it with a photograph, model it as a scene, or draw it yourself using illustration techniques. And so the next step in the marketplace is simply in finding the approach to a production that will be differentiating with AI - the equivalent of movies to photography. This collapses not the indie space - because they never could afford productions to begin with - but existing modes of mobile gaming, because they were leveraging the old production framework. Nobody has need of microtransaction cosmetics if they can generate the look they want.
    • dexwiz 3y ago
      The platform layer of the "make your own game" game is always too heavy and too limited to compete with a dedicated engine in the long run. Also the monetization strategy is bad for professionals.
    • angra_mainyu 3y ago
      I couldn't disagree more. RPGMaker didn't kill RPGs, Unity/Godot/Unreal didn't kill games, Minecraft didn't kill games, and Renpy didn't kill VNs. Far more people prefer playing games than making them. We'll probably see a new boom of indie games instead. Don't forget, a large part of what makes the gaming experience unique is the narrative elements, gameplay, and aesthetics - none of which are easily replaceable. This empowers indie studios to hit a faster pace on one of the most painful areas of indie game dev: asset generation (or at least for me as a solo dev hobbyist).
      • stuckinhell 3y ago
        Sorry I guess I wasn't clear. None of those things made games automatically. The future is buying a game making game, and saying I want a zelda clone but funnier. The ai game framework handles the full game creation pipeline.
        • CaptainFever 3y ago
          The issue with that is that it probably produces generic-looking games, since the AI can't read your mind. See ChatGPT or SD for example, if you just say "write me a story about Zelda but funnier" it will do it, but it's the blandest possible story. To truly make it good requires a lot of human intention and direction (i.e. soul), typically drawn from our own human experiences and emotions.
          • stuckinhell 3y ago
            Then I'll just update my prompt to Zelda but funnier like Curb Your Enthusiasm style humor.
    • Vegenoid 3y ago
      There are more amazing, innovative and interesting indie games being created now than ever before. There's just also way more indie games that aren't those things.
• airstrike 3y ago
  This is revolutionary
• shaileshm 3y ago
  This is what a truly revolutionary idea looks like. There are so many details in the paper. Also, we know that transformers can scale. Pretty sure this idea will be used by a lot of companies to train the general 3D asset creation pipeline. This is just too great. "We first learn a vocabulary of latent quantized embeddings, using graph convolutions, which inform these embeddings of the local mesh geometry and topology. These embeddings are sequenced and decoded into triangles by a decoder, ensuring that they can effectively reconstruct the mesh." This idea is simply beautiful and so obvious in hindsight. "To define the tokens to generate, we consider a practical approach to represent a mesh M for autoregressive generation: a sequence of triangles." More from paper. Just so cool!
  • tomcam 3y ago
    Can someone explain quantized embeddings to me?
    • _hark 3y ago
      NNs are typically continuous/differentiable so you can do gradient-based learning on them. We often want to use some of the structure the NN has learned to represent data efficiently. E.g., we might take a pre-trained GPT-type model, and put a passage of text through it, and instead of getting the next-token prediction probability (which GPT was trained on), we just get a snapshot of some of the activations at some intermediate layer of the network. The idea is that these activations will encode semantically useful information about the input text. Then we might e.g. store a bunch of these activations and use them to do semantic search/lookup to find similar passages of text, or whatever. Quantized embeddings are just that, but you introduce some discrete structure into the NN, such that the representations there are not continuous. A typical way to do this these days is to learn a codebook VQ-VAE style. Basically, we take some intermediate continuous representation learned in the normal way, and replace it in the forward pass with the nearest "quantized" code from our codebook. It biases the learning since we can't differentiate through it, and we just pretend like we didn't take the quantization step, but it seems to work well. There's a lot more that can be said about why one might want to do this, the value of discrete vs continuous representations, efficiency, modularity, etc...
• sram1337 3y ago
  What is the input? Is it converting a text query like "chair" to a mesh? edit: Seems like mesh completion is the main input-output method, not just a neat feature.
  • CamperBob2 3y ago
    That's what I was wondering. From the diagram it looks like the input is other chair meshes, which makes it somewhat less interesting.
    • tayo42 3y ago
      Really the hardest thing with art is details and usually seperates good from bad. So if you can sketch what you want roughly without skill and have the details generated, that's extremely useful. And image to image with the existing diffusion models is useful and popular.
      • nullptr_deref 3y ago
        I have no idea about your background when I am commenting here. But these are my two cents. NO. Details are mostly like icing on top of the cake. Sure, good details make good art but it is not always the case. True and beautiful art requires form + shape. What you are saying is something visually appealing. So, the reason why diffusion models feel so bland is because they are good with details but do not have precise forms and shape. Nowadays they are getting better, however, it still remains an issue. Form + shape > details is something they teach in Art 101.
    • treyd 3y ago
      There's also examples of tables, lamps, couches, etc in the video.
  • deleted 3y ago
    [deleted]
  • all2 3y ago
    You prompt this LLM using 3D meshes for it to complete, in the same manner you use language to prompt language specific LLMs.
• kranke155 3y ago
  My chosen profession (3D / filmmaking) feels like being in some kind of combat trench at the moment. Both fascinating and scary
  • nextworddev 3y ago
    What do you ascertain the use case of this in your field? Does it seem high quality? (I have no context)
    • zavertnik 3y ago
      I'm not a professional in VFX, but I work in television and do a lot of VFX/3D work on the side. The quality isn't amazing, but it looks like this could be the start of a Midjourney-tier VFX/3D LLM, which would be awesome. For me, this would help bridge the gap between having to use/find premade assets and building what I want. For context, building from scratch in a 3D pipeline requires you to wear a lot of different hats (modeling, materials, lighting, framing, animating, ect). It costs a lot of time to get to not only learn these hats but also use them together. The individual complexity of those skill sets makes it difficult to experiment and play around, which is how people learn with software. The shortcut is using premade assets or addons. For instance, being able to use the Source game assets in Source Filmmaker combined with SFM using a familiar game engine makes it easy to build an intuition with the workflow. This makes Source Filmmaker accessible and its why theres so much content out there made with it. So if you have gaps in your skillset or need to save time, you'll buy/use premade assets. This comes at a cost of control, but that's always been the tradeoff between building what you want and building with what you have. Just like GPT and DALL-E built a bridge between building what you want and building with what you have, a high fidelity GPT for the 3D pipeline would make that world so much more accessible and would bring the kind of attention NLE video editing got in the post-Youtube world. If I could describe in text and/or generate an image of a scene I want and have a GPT create the objects, model them, generate textures, and place them in the scene, I could suddenly just open blender, describe a scene, and just experimenting with shooting in it, as if I was playing in a sandbox FPS game. I'm not sure if MeshGPT is the ChatGPT of the 3D pipeline, but I do think this is kind of content generation is the conduit for the DALL-E of video that so many people are terrified and/or excited for.
• trostaft 3y ago
  Seems like the bibtex on the page is broken? Or might just be an extension of mine.
• alexose 3y ago
  It sure feels like every remaining hard problem (i.e., the ones where we haven't made much progress since the 90s) is in line to be solved by transformers in some fashion. What a time to be alive.
• throwaway8318 3y ago
  [flagged]
• mclanett 3y ago
  This is very cool. You can start with an image, generate a mesh for it, render it, and then compare the render to the image. Fully automated training.
  • de6u99er 3y ago
    continous training
• j7ake 3y ago
  I love this field. Paper include a nice website, examples, and videos. So much more refreshing than the dense abstract, intro, results paper style.
• valine 3y ago
  Even if this is “only” mesh autocomplete, it is still massively useful for 3D artists. There’s a disconnect right now between how characters are sculpted and how characters are animated. You’d typically need a time consuming step to retopologize your model. Transformer based retopology that takes a rough mesh and gives you clean topology would be a big time saver. Another application: take the output of your gaussian splatter or diffusion model and run it through MeshGPT. Instant usable assets with clean topology from text.
  • toxik 3y ago
    What you have to understand is that these methods are very sensitive to what is in distribution and out of distribution. If you just plug in user data, it will likely not work.
  • mattigames 3y ago
    Lol for 3D artists, this will be used 99% by people who have have never created a mesh by hand in their lifes; to replace their need to hire a 3D artist: programmers who don't want (or can't) pay a designer, architects who never learned nothing other than CAD, fiver "jobs", et al I don't think people here realize how are we inching to automating the automation itself, and the programmers who will be able to make a living out of this will be a tiny fraction of those who can make a living out of it today.
  • bradleyishungry 3y ago
    sorry to tell you, but there’s no way anything will be generating clean topology for characters for a long long time.
    • valine 3y ago
      There’s no shortage of 3D mesh data to train on. Who to say scaling up the parameter count won’t allow for increasingly intricate topology the same way scaling language models improved reading comprehension.
• toxik 3y ago
  This was done years ago, with transformers. It was then dubbed Polygen.
  • Sharlin 3y ago
    You might want to RTFA. Polygen and other prior art are mentioned. This approach is superior.
    • toxik 3y ago
      I read the article. It has exactly the same limitations as Polygen from what I can tell.
      • dymk 3y ago
        Their comparison against PolyGen looks like it's a big improvement. What are the limitations that this has in common with PolyGen that make it still not useful?
        • toxik 3y ago
          I don’t think it’s as widely applicable as they try to make it seem. I have worked specifically with PolyGen, and the main problem is “out of distribution” data. Basically anything you want to do will likely be outside the training distribution. This surfaces as sequencing. How do you determine which triangle or vertex to place first? Why would a user do it that way? What if I want to draw a table with the legs last? Cannot be done. The model is autoregressive.
  • GaggiX 3y ago
    First, you use the word "transformers" to mean "autoregressive models", they are not synonymous, second, this model beats Polygen on every metric, it's not even close.
• mlsu 3y ago
  The next breakthrough will be the UX to create 3d scenes in front of a model like this, in VR. This would basically let you _generate_ a permanent, arbitrary 3D environment, for any environment for which we have training data. Diffusion models could be used to generate textures. Mark is right and so so early.
  • ShamelessC 3y ago
    Mark? edit: Oh, _that_ Mark? lol okay edit edit: Maybe credit Lecun or something? Mark going all in on the metaverse was definitely not because he somehow predicted deep learning would take off. Even the people who trained the earliest models weren't sure how well it would work.
• amelius 3y ago
  Is this limited to shapes that have mostly flat faces?
• catapart 3y ago
  Dang, this is getting so good! Still got a ways to go, with the weird edges, but at this point, that feels like 'iteration details' rather than an algorithmic or otherwise complex problem. It's really going to speed up my pipeline to not have to pipe all of my meshes into a procgen library with a million little mesh modifiers hooked up to drivers. Instead, I can just pop all of my meshes into a folder, train the network on them, and then start asking it for other stuff in that style, knowing that I won't have to re-topo or otherwise screw with the stuff it makes, unless I'm looking for more creative influence. Of course, until it's all the way to that point, I'm still better served by the procgen; but I'm very excited by how quickly this is coming together! Hopefully by next year's Unreal showcase, they'll be talking about their new "Asset Generator" feature.
  • truckerbill 3y ago
    Do you have a recommended procgen lib?
    • catapart 3y ago
      Oh man, sorry, I wish! I've been using cobbled together bits of python plugins that handle Blender's geometry nodes, and the geometry scripts tools in Unreal. I haven't even ported over to their new proc-gen tools, which I suspect can be pretty useful.
• circuit10 3y ago
  Can this handle more organic shapes?
• LarsDu88 3y ago
  As a machine learning engineer who dabbles with Blender and hobby gamedev, this is pretty impressive, but not quite to the point of being useful in any practical manner (as far as the limited furniture examples are concerned. A competent modeler can make these types of meshes in under 5 minutes, and you still need to seed the generation with polys. I imagine the next step will be to have the seed generation controlled by an LLM, and to start adding image models to the autoregressive parts of the architecture. Then we might see truly mobile game-ready assets!
  • th0ma5 3y ago
    This is a very underrated comment... As with any tech demo, I'd they don't show it, it can't do it. It is very very easy to imagine a generalization of these things to other purposes, which, if it could do it, would be a different presentation.
    • rawrawrawrr 3y ago
      It's research, not meant for commercialization. The main point is in the process, not necessarily the output.
      • th0ma5 3y ago
        What? If the research doesn't show it, it can't do it, is my point, or else they would've put it in their research.
  • empath-nirvana 3y ago
    > A competent modeler can make these types of meshes in under 5 minutes. I don't think this general complaint about AI workflows is that useful. Most people are not a competent <insert job here>. Most people don't know a competent <insert job here> or can't afford to hire one. Even something that takes longer than a professional do at worse quality for many things is better than _nothing_ which is the realistic alternative for most people who would use something like this.
• frozencell 3y ago
  Not reproducible with code = Not research.
• 65892 3y ago
  [flagged]
• beebeepka 3y ago
  Games and pretty much any other experience being generated by AI is obvious to anyone paying attention at this point. But how would it work. Are current ai generated images and videos using rasterisation? Will they use rasterisation, path tracing or any other traditional rendering technique, or is will it be an entirely different thing.
  • wolfgang805 3y ago
    Why would a video or an image, something generated without a mesh, be using rasterization?
    • beebeepka 3y ago
      If it's faster to generate? I don't know, that's what I am asking
• KyleLewis 3y ago
  Cant wait for the "multimodal" version that can take a written description and generate meshes
• m3kw9 3y ago
  So you train it with vector sequences that represent furnitures and it predicts the next token(triangles), so how is this different from it ChatGPT was trained with the same sequences and can output all the 3d locations and trangle size/lengths in sequence and have a 3d program piece it together?
• btbuildem 3y ago
  This is fantastic! You can broad-strokes sketch the key strokes of the shape you want, and this will generate some "best" matches around that. What I really appreciate about this is that they took the concept (transformers) and applied it in a quite different-from-usual domain. Thinking outside of the (triangulated) box!
• Stevvo 3y ago
  Fantastic, but still useless from a professional perspective. i.e. A mesh that represents a cube as 12 triangles is a better prestation of the form than previous efforts, but barely more usable. Whilst it might not be the solution I'm waiting for, I can now see it as possible. If an AI model can handle traingles, it might handle edge loops and NURBS curves.
• BrokrnAlgorithm 3y ago
  I'm not a 3D artist, but why are we still, for lack of a better word, "stuck" with having / wanting to use simple meshes? I appreciate the simplicity, but isn't this an unnecessary limitation of mesh generation? It feels like an approach that imitates the constraints of having both limited hardware and artist resources. Shouldn't AI models help us break these boundaries?
  • ipsum2 3y ago
    We're not stuck on meshes. Check out neural radiance fields as an alternative.
    • fireant 3y ago
      My understanding is that it's quite hard to make convex objects with radiance fields, right? For example the furniture in OP would be quite problematic. We can create radiance fields with photogrammetry, but IMO we need much better algorithms for transforming these into high quality triangle meshes that are usable in lower triangle budget media like games.
      • BrokrnAlgorithm 3y ago
        "Lower triangle budget media" is what I wonder if its still a valid problem. Modern game engines coupled with modern hardware can already render insane number of triangles. It feels like the problem is rather in engines not handling LOD correctly (see city skylines 2), although stuff like UE5 nanite seems to have taken the right path here. I suppose though there is a case for AI models for example doing what nanite does entirely algorithmically and research like this paper may come in handy there.
    • BrokrnAlgorithm 3y ago
      I was referring to being stuck with having to create simple / low tri polygonal meshes as opposed to using complex poly meshes such as photogrammetry would provide. The paper specifically addresses clean low poly meshes as opposed to what they call complex iso surfaces created by photogrammetry and other methods
• DeathArrow 3y ago
  So maybe in a few years we can ask AI to generate a level or entire game.
• wolfgang805 3y ago
  It would be nice to be see work and be part of a field that did work that humans could not do, instead of creating work that just replaces what humans already know how to do.
• Mizza 3y ago
  Great work. But I don't get from the demo how it knows what object to autocomplete the mesh with - if you give it four posts as an input, how does it know to autocomplete as a table and not a dog? So maybe the next step is something like CLIP, but for meshes? CLuMP?
• jhiggins777 3y ago
  Really cool, but in 3d modeling triangles are a "no no". You are taught early on to design in quads.