8 ms·
Lane Following Autopilot with Keras and Tensorflow
- robinson-wall 10y agoThis is nice work, but anyone wanting to try it for themselves should be warned that you shouldn't unpickle data received from an untrusted source. https://blog.nelhage.com/2011/03/exploiting-pickle/ https://blog.nelhage.com/2011/03/exploiting-pickle/
- str33t_punk 10y agoglad to see this as the first comment
- kordless 10y agoAnti-pickler.
- wroscoe 10y agoI copied this method of loading datasets from Keras. https://github.com/fchollet/keras/blob/master/keras/datasets/imdb.py#L43 https://github.com/fchollet/keras/blob/master/keras/datasets.... What's a better alternative.
- Drdrdrq 10y agoAnother serialization format which doesn't create objects, like JSON, XML, CSV,...
- deleted 10y ago[deleted]
- Cyph0n 10y agoNicely done! But I'm assuming that this is more of an exercise rather than a real-world application of ML? I say this because the task of keeping a car between two lines is trivially done using control algorithms. Of course, the CV part -- "seeing" the lines -- requires some form of ML to work in the real world.
- webaba 10y agoObviously, "Lane Following Autopilot using my brain and controls theory" would not make it to the top of HN. Welcome to the new era where Tensorflow replaces Lyapunov and ML spares you the need of understanding hard problems... until you need guarantees and safety... but but it's ok let's add more data.
- deleted 10y ago[deleted]
- gugagore 10y agoI agree with you. If you can leverage control theory from the 1950s to solve your problem, what's the point? However, I will state that using e.g. Lyapunov functions to prove the stability of the system requires a model of the system. And even if you need a guarantee for your system, that guarantee is only as good as the fidelity of your model. For an inexpensive RC car, with slippage and saturation, without torque control or inertial sensing, you're going to have a hard time doing something that sounds as principled as what you suggest.
- deleted 10y ago[deleted]
- aub3bhat 10y agoYou seem to be forgetting the entire vision pipeline that automatically extracts "lanes" and that information gets incorporated in an end to end manner requiring only true steering angles and nothing else. Its easy to comment but its not as straightforward or trivial as one might assume.
- gugagore 10y agoAre you disagreeing with my comment? Or stating that I should have included additional points in my comment? In any case I think I understand your comment, that in addition to the control problem, there's a perception problem.
- agumonkey 10y agoI remember seeing keras in commaai source. Who else uses it ?
- wroscoe 10y agoTensorflow anounced last week that it will incorporate Keras as its default higher level abstraction. It's a pleasure to use.
- ipunchghosts 10y agoX is in the range 0, 255. They don't show code converting it to a much saner range for the network they've chosen. Is the full source somewhere?
- cr0sh 10y agoHave you already looked at the pickled data? Because it looks like the model is outputting a single label value out of 256 labels; depending on the training data (steering angle) and how it is represented in the data (signed float or integer?), each one of those 256 learned should (?) be similar - I think. Again, I'm not an expert. Or - maybe it is outputting a number 0-255, and then taking that number and converting it (and maybe other operations) into values suitable for the servo on the car (perhaps centered around 0 - so -128 to 127 or something like that - then scaled for servo PPM width or whatever values needed)... All guesses, of course.
- wroscoe 10y agoThe input values are image arrays 120x160 pixels with 3 channels for red,green,blue. The values range from 0-255 and are not normalized before they are fed into the convolution layer. I found this did not make a difference. The output of the model is a single real number between -90(left) and 90(right). I believe a better approach would be to bin the outputs and use a classifier. This way you'd know when the model was getting confused (ie, approaching a perpendicular line.
- cr0sh 10y agoAlso the full repo is mentioned (I think the article is just a general highlighting of the full repo): https://github.com/wroscoe/donkey https://github.com/wroscoe/donkey
- zxcvvcxz 10y agoGreat summary, I always think it's best when machine learning projects have visuals and videos to showcase what is actually being learned. This simple project is a good example of supervised learning from what I can tell - the network will learn to steer "as good as" the human that provides the training data. For a different (and more complex) flavor of algorithm, check out reinforcement learning, where the "agent" (computer system) can actually learn to outperform humans. Stanford's autonomous helicopters always come to mind - http://heli.stanford.edu/ http://heli.stanford.edu/
- ramshanker 10y agoBefore even opening the link, I was thinking, which jurisdiction would it be legal to program "personal" autopilot. Awesome tutorial.
- nilkn 10y agoWas the track changed at all during the training? I'm wondering if there's some subtle overfitting here where the car learned to drive along only this specific track. It mentions this but I'm not sure what concrete actions were taken to avoid overfitting: > The biggest problem I ran into was over fitting the model so that it would not work in evenlly slightly different scenarios. Regardless, a very cool project.
- wroscoe 10y agoThe method to avoid overfitting was to use the model with the lowest validation loss, not training loss. I was able to change the track around my house with reasonable success. I think would need many more example turns in the training data to become robust.
- grandalf 10y agoConsider the fairly massive changes to the competitive landscape ushered in by the combined factors of self-driving and electric vehicles: - For liability reasons, most of the algorithmic IP will likely be open sourced. Either because it's required by regulators or because it's the most efficient way for car makers to socialize risk of an algorithmic failure. - Electric vehicles have many fewer moving parts, which means that the remaining parts are likely to be converged upon by the industry and used widely. This breaks a lot of platform-dependency issues and allows for the commoditization of parts like motors. As these become standardized and commoditized, and easily comparable on the basis of size, torque, and efficiency, there will be virtually no benefit to carmakers to manufacture their own. The same applies to aluminum monocoque frames, charging circuitry, etc. Tesla currently differentiates its models based on how many motors and what size batteries, but beyond that it's mostly just cabin shape, along with new innovations like the hepa filter cabin air cleansing which will likely be a standard part of all future models. - Battery tech works the same way as motors, with little competitive advantage to be gained by automakers, especially since most of the IP in this area is already spoken for. Compare the number of patentable parts in a model T vs a 1998 Taurus vs a 2017 internal combustion vehicle vs a Telsa. Tesla is one innovator, and GM has already likely patented many inventions relating to EV technology back in the original Chevy Volt era. All this is why Tesla acquired SolarCity and is attempting to make an infrastructure play rather than a technology play. Only due to Musk's rare ability to self-finance big risks is this even possible, since infrastructure moonshots featuring $30K+ hardware units are hard to fund.
- mulmen 10y agoHow do you see car makers differentiating their products in a world where all the parts including the frame are commoditized and the software is open source? Also, GM built an electric car back in the 90s called the EV-1. I wonder how much innovation was in that car vs the Volt.
- Drdrdrq 10y agoInterior design. Cars will evolve into mobile living spaces so the quality of interior becomes more significant to exterior.
- nojvek 10y agoWhat I would love to see is an end to end neural network soln. On one end camera input comes through, on the other outputs for speed and steering angle. But rather than a black box, it's explainable what the different layers are doing. If neural nets are turing machines then we should be able to compile some parts of the net from code. Then the net is a library of layers. Some Layers trained with back prop, some compiled from code.
- wroscoe 10y agoI'm working on adding the throttle. This is difficult because you need to drive the correct speed and stopping or running off course can mess up the training data. This project was inspired by Otavio's carputer which does predict throttle, steering angle, and odometer.
- argonaut 10y agoAlmost all neural nets are not Turing complete. Only very specific RNNs are; most RNNs aren't, including pretty much any RNN model used in the real world right now (https://uclmr.github.io/nampi/talk_slides/grefenstette-nampi.pdf https://uclmr.github.io/nampi/talk_slides/grefenstette-nampi...). Also, this is a useless fact, because so many other random things are Turing complete.
- deepnotderp 10y agoThe end to end approach of regressing steering wheel angle already exists, check nvidia's paper.
- sja 10y agoI might be missing it, but I don't see instructions for installing TensorFlow/Keras on the Raspberry Pi in the Donkey repo or in this blog post (needed to actually run the trained model, it looks like). For TensorFlow, there are pre-built binaries and instructions to build from source here: https://github.com/samjabrahams/tensorflow-on-raspberry-pi https://github.com/samjabrahams/tensorflow-on-raspberry-pi Note: I am the owner of this repo
- billyzs 10y agoNot to put down the OP's work (I think it's a great project), but I'm just wondering what advantages might an ML approach have over "traditional" CV algorithms. In a really well controlled environment lanes will be easy to detect, and computing the difference between the current heading and lane direction should be doable; maybe if we're talking about complex outdoor environments and poor sensors then ML would have an advantage? Or if we're teaching the robot what the concept of a lane is? I think back to the days when I basically implemented lane following with an array of photo resistors, an Arduino, a shitty robot made from Vex parts and some c code. The problem is much simpler than the one presented in this article, but then the computational resource used was order of magnitudes less. At what point then, do you decide that "OK I think the complexity and nature of the problem warrants the use of ML" or "Hmmm I think neural network is an overkill here"?
- wroscoe 10y agoTraditional CV approaches are much easier to debug as well. I chose the ML approach with the assumption that it would be easier to build a robust autopilot that would work in many lighting conditions. Actually my short term goal is to get the car to drive around my block on the sidewalk (no lines). From my experience CV approaches have many parameters that need to be tuned specifically for each environment. While ML approaches also have parameters that need tuning they stay constant between environments.
- billyzs 10y agoI see, that makes sense. It'd be indeed worth it if we can apply a model trained on controlled environment to a more challenging one with little to no modification. Good luck with the project and keep us updated!
- platz 10y agoMy first thought was something that used several PID mechanisms.
- option_greek 10y agoBecause ML approaches can adapt to different environments like a forest trail. While this can probably be achieved with OpenCV, this just feels natural: https://www.youtube.com/watch?v=umRdt3zGgpU https://www.youtube.com/watch?v=umRdt3zGgpU
- argonaut 10y agoTwo major errors: 1) This doesn't seem to be controlling overfitting on the right validation set. 2) There isn't a test set at all (separate from validation). Using Keras' "validation_split" parameter will just randomly select a validation set. This is not the right thing to do when your data is image sequences, because you will get essentially identical data in training and validation. Because of this, the numbers/plot here might as well be training accuracy numbers.
- mhanus 10y agoKeras uses the end of the data set as validation, and only randomizes it if the "shuffle" argument is set to True [1]. [1]: https://keras.io/getting-started/faq/#how-is-the-validation-split-computed https://keras.io/getting-started/faq/#how-is-the-validation-...
- feelix 10y agoApologies if I'm being stupid, but I can't find the details on how to physically connect the hardware together anywhere. Is this still on the todo list? I'm interested in applying this tutorial and making an autonomous RC car.
- wroscoe 10y agoI updated this post with some of the great feedback from the comments. Also I just ported the algo used by the last DIYRobocar race winner, CompoundEye. Here's that post: https://wroscoe.github.io/compound-eye-autopilot.html#compound-eye-autopilot https://wroscoe.github.io/compound-eye-autopilot.html#compou... Thanks!