7 ms·
TensorKart: self-driving MarioKart with TensorFlow
- adyus 10y agoCongrats on finishing the project! As you've already linked at the bottom of your post, it's possible that OpenAI could've solved most of your I/O issues. One thing I'd suggest is exploring a reward function, instead of using only pre-recorded training data. That is, give the AI a goal to complete (in this case, finish the race) and let it learn by itself!
- Drdrdrq 10y agoI would love to learn how to do that - any suggestions? EDIT: to clarify: what should I google for?
- paulbaumgart 10y agoReinforcement Learning. Here's a good intro: http://www0.cs.ucl.ac.uk/staff/d.silver/web/Teaching.html http://www0.cs.ucl.ac.uk/staff/d.silver/web/Teaching.html
- adyus 10y agoHere's what I could find in a couple minutes: https://github.com/openai/universe-starter-agent https://github.com/openai/universe-starter-agent OpenAI's example universe agent. Remember that while their goal is an agent that works in any and all environments (read: games), you could certainly optimize yours just for MarioKart.
- Drdrdrq 10y agoThanks, looks promising! Can't wait to try it! :)
- bitL 10y agoIt's basically a project these days at Udacity's Self-driving car nanodegree under "Behavioral Cloning" ;-)
- CM30 10y agoPretty interesting I must say. Have to admit though, I kind of expected the self driving AI to be trying to win Grand Prix or Versus races instead of doing well in Time Trials. But hey, I can see how that would be utterly painful to try and set up, especially given how times you get hit by items or rammed off the track in more recent games.
- bisby 10y agoStep 1 is to make the AI find an ideal path through the course. Step 2 is to make AI figure out how to return to the ideal path through the course when other people are stealing your items or shelling you. step 3 is to make the AI figure out how to counter attack to slow down the opponents. Step 4 is OH GOD WE TAUGHT THE AI HOW TO ATTACK RUN FOR YOUR LIVES.
- bryondowd 10y agoStep 2.5 would be to make the AI figure out how to evade or minimize the effect of or ability to initiate opponents' offensive moves. That would be the most interesting bit to me. Would be neat to see an AI intentionally stay in 2nd place with an item at the ready until the home stretch, to avoid being blue-shelled.
- bisby 10y agoIntentionally stay in second, unless it has reason to believe that it can stay in 1st place, even after getting blue-shelled. But yes. point being, self driving is a feat of it's own, competing with opponents is a whole different ballgame with it's own set of challenges.
- gm-conspiracy 10y agoI appreciate the write-up. Thank you!
- tjfontaine 10y agoNext, have it upload its race results to kartlytics
- TomAnthony 10y agoYeah - that was a very cool project!
- glaberficken 10y agoomg! thanks for the refference to kartlytics. I had no idea this existed!
- TomAnthony 10y agoIt would be very interesting to see how well this does with more training data, especially with multiple players.
- jostmey 10y agoQuote: "Driving a new (untrained) section of the Royal Raceway:" So the author did a proper test of the model by scoring it on an unseen track to make sure it generalizes! This is very awesome!
- jupiter90000 10y agoIt is pretty cool! I'd love to see how it did on the other parts of that track as well, if that was tested
- ska 10y agoHow did we get from "bare minimum sensible testing" to "This is very awesome!"? Are things that bad on average?
- jordigh 10y agoI was ready to be impressed about seeing an AI that could consistently beat the game's own AI, with blue turtle shells and all. Oh well, still pretty impressive to be able to drive on the easiest course without opponents.
- JonnieCache 10y agoIn contrast, here's what is effectively an oracle machine playing mario kart: https://www.youtube.com/watch?v=ZBNgbJ5hXtQ https://www.youtube.com/watch?v=ZBNgbJ5hXtQ (Amazingly detailed) info: http://tasvideos.org/5243S.html http://tasvideos.org/5243S.html
- rl3 10y agoI like how it just glitches itself to an almost instant win on half of the maps.
- dclowd9901 10y agoThese are Tool-Assisted Speedruns. That means that it's a human player using things like slow motion, mem dumps and other mechanisms to play perfect games. It's more an example of human's abilities when augmented with computers than AI discovering those glitches itself.
- hkmurakami 10y agoThe most amazing run I've seen so far was a RTA (realtime time attack) of mega man 2. A human player is manually collision glitching and writing over memory with a sequence of inputs. And the RTA time in 2016 is now faster than the initial TAS records.
- sgrove 10y agoDo you have a link? Sounds like it'd be a very interesting watch. Edit: I found one that has an example (I think) around 7:38 http://www.nicovideo.jp.am/watch/sm13963118 http://www.nicovideo.jp.am/watch/sm13963118 - the collision detection pushes megaman into the wall and jumps between different sections. Very interesting indeed!
- hkmurakami 10y agoSorry I think the one I was thinking of was Megaman 2. I did find a link for you. Starts at 2:36. http://www.nicovideo.jp/watch/sm23825129 http://www.nicovideo.jp/watch/sm23825129 Also you might be interested in the Final Fantasy 6 memory overwrite bug that was discovered in 2016 as well. It uses the Window Color menu settings as the data reference. Btw regarding the Megaman 2 RTA, there's an even more ridiculous collision bug being used around 11:30 http://www.nicovideo.jp/watch/sm28321223 http://www.nicovideo.jp/watch/sm28321223
- rl3 10y agoThe inevitable follow-up article that delves into training offensive banana peel usage should be interesting.
- ramzyo 10y agoThis is really cool, and any reason to bring this game back into my life is warmly welcomed
- cjmcqueen 10y agoBest part, "With this in mind I played more MarioKart to record new training data. I remember thinking to myself while trying to drive perfectly, “is this how parents feel when they’re driving with their children who are almost 16?”"
- nartam11 10y agoHow are the original computer opponents able to play MarioKart?
- taway_1212 10y ago1. The AI in games has access to internal representations of game state and does not have to recognize it from pixels on screen. This is a massive difference. 2. The logic is usually a bunch of (human-authored) scripts consisting of if-else spaghetti.
- bryondowd 10y agoAlso, the AI opponents don't have to play by the same rules. They go by fun > fairness to keep things interesting. That's why you normally can't keep a huge lead on AI opponents, because they "rubberband" back up to you faster than they should be able to. Wouldn't surprise me if they don't even 'drive' in any sense while off-screen, just increment some abstract position relative to the track length. But I don't know this for a fact.
- ionforce 10y agoHow do you know this stuff? I find it insanely interesting. Tell me more!
- bryondowd 10y agoMostly experience, but you should get plenty of results by just Googling "mario kart rubber banding". Looks like the top result mentions a patent on an algorithm for it, but I'm blocked at work. It's a common enough term, though, there's even a page for the trope here: http://tvtropes.org/pmwiki/pmwiki.php/Main/RubberBandAI http://tvtropes.org/pmwiki/pmwiki.php/Main/RubberBandAI
- dclowd9901 10y ago"Wouldn't surprise me if they don't even 'drive' in any sense while off-screen, just increment some abstract position relative to the track length. But I don't know this for a fact." This seems unlikely, especially given how item pickup zones operate. Since an item box disappears for a short period of time after someone drives over it, it's imperative that the position of the CPU player who drove over it, and the one that comes after that (and inherently gets no item) is represented accurately. Even off-screen, AI continues to collect and utilize items. Then again, maybe this is just done by cheating simply with an RNG.
- cr0sh 10y agoThis is pretty cool; as someone who is currently working on the second project (traffic sign recognition) for the Udacity "Self-Driving Car Engineer" nanodegree, using TensorFlow - it is interesting to me how it seems like the "standard" MNIST CNN can be adapted to so many other use cases. For the project I am currently working on, I'm using a slightly modified form of LeNet - which isn't too different from the TF MNIST tutorial; after all, recognizing traffic signs isn't much different than recognizing hand-written numbers... ...but "driving" a course? That seems radically different to my less-than-expert-at-TensorFlow understanding, but that is only due to my ignorance. I'm glad that these examples and demos are being investigated and made public for others - especially people learning like myself - to look at and learn from.
- halflings 10y agoFrom the post: > Later, I switched to use Nvidia’s Autopilot... So I guess he didn't use the MNIST CNN model.
- cr0sh 10y agoHowever, if you look at the code: https://github.com/SullyChen/Autopilot-TensorFlow/blob/master/model.py https://github.com/SullyChen/Autopilot-TensorFlow/blob/maste... You can see that it follows much the same pattern as LeNet CNN for MNIST - a few (ok, more than a few!) convolutional layers followed by a few fully connected layers. Maybe you could call it a "follow on" or perhaps an ANN pattern?: Conv -> Conv -> Reshape/Flatten -> FC -> FC -> FC (disregarding activation and such) ...which is really the lesson of the LeNet MNIST CNN - at least, that's my takeaway.
- halflings 10y agoYou're right, that does look similar... I expected this to be based on some type of RNN!
- melvinmt 10y ago> the Udacity "Self-Driving Car Engineer" nanodegree That looks like a great course by the way, thanks for sharing.
- bduerst 10y agoPersonally I think the most impressive thing here isn't that you created a self-driving MarioKart, but that you trained TensorFlow based on input screenshots of your desktop. I feel like that could be a good next step - a ubiquitous neural net model that, after mapping inputs, will learn to play any video game that's on your screen.
- jboggan 10y agoEspecially since the hard work of increasing the screen resolution has already been done. Also, bravo on including the stupid little bugs that gave you trouble. It always sustains me working on a hard project to know that a self-driving video game was blocked by a missing newline in a C HTTP request. It makes me step back and laugh at the ridiculous complexity of what we take for granted in our day to day work.
- Hydraulix989 10y agoThere's work being done to allow reinforcement nets to do transfer learning.
- dylanbfox 10y agogreat write up! this is awesome
- eli_gottlieb 10y agoPersonally, I'm just a little impressed that you can train an active agent to play a game using old-fashioned supervised learning on screen states and controller states rather than relying on "action-oriented" learning techniques like reinforcement learning, online learning, or even a recurrent model. It really shows how simple many control tasks actually are!
- tobilarscheid 10y agoThis is exactly what I wondered about. So what exactly is the function you are training for? Is it basically like "if the screen (showing the track) looks like this, apply these controls"?
- eli_gottlieb 10y agoIn TFA it says that he was training a supervised learner to predict the control state from the screen state. So yes, "if the screen looks like this, apply these controls", and that can play Mario Kart 64.
- PeterisP 10y agoAn more accurate description of the function would be "given this picture of a screen, what is the most likely key my author was pressing in this situation" - no goals, no values, no optimization, but simply learning to imitate the actions performed by a human. Coincidentally, one of the neural network components in AlphaGo did pretty much the same, i.e. attempted to guess what human player would usually play in this situation purely based on the image and nothing else.
- sakabaro 10y agoCheck out also MarI/O, very impressive: https://www.youtube.com/watch?v=qv6UVOQ0F44 https://www.youtube.com/watch?v=qv6UVOQ0F44
- xigency 10y agoI'm interested in knowing why the Python and C components communicate with HTTP, beyond reading about the bugfix. Wouldn't it be easier to use sockets or files or some other mechanism to integrate the two languages? Just something to think about as a developer. I would imagine that on a local machine, using HTTP as the protocol might add latency.
- haikuginger 10y agoThis was my initial reaction as well; it seems like a raw socket or even embedding a Python interpreter would be better ways to go.
- tomrod 10y agoI love this! I'm working (albeit very slowly, as a beginner) on a similar project with Geometry Dash and Python. You're a great inspiration!
- holografix 10y agoThis is very cool and I think if Kevin spends a bit of time learning reinforcement learning it could be amazing. It seems like a lot of people doing reinforcement learning on video games get bogged down on training on raw pixels only... it would take a tremendous amount of data to make the driver recognise when and where to use certain power ups, however if you encoded this as a variable, wow it could be really cool. I believe this is fundamentally how we humans learn with so few examples. Other humans "encode features for our brain to track" by telling us how it should be done and what information to prioritise.
- jondiggsit 10y agoNo power slide? Failure.