5 ms·
AI model on a $3 chip (ESP32)
We finished a design that allows a plug-and-play solution with TensorFlow Lite and a web server with the UI on ESP32. Here is a video of the process: https://youtu.be/aEZX3JMzwTo https://youtu.be/aEZX3JMzwTo
I wanted to share the design in case anyone is interested as the camera is aimed at developers who want to play with AI models on the embedded side.
- anigbrowl 4y agoHow much do you plan to offer it for?
- amelius 4y agoHow would you characterize the performance of this chip, and especially the AI part?
- h317 4y agoIt does surprisingly well on object detection for its value. The tested use cases so far are: animal detection, sophisticated version of a motion sensor, plant recognition, people counter. Most likely Edge Impulse can work even a bit better.
- deleted 4y ago[deleted]
- inspirerhetoric 4y agoWhat camera chip are you using?
- moffkalast 4y agoNice job, though a smaller PCB would've been more practical? The ESP32-CAM is definitely a bit tricky to work with software-wise. I'm not sure how much sense it actually makes to run anything onboard unless you want to take a snapshot every few seconds and then spending a while classifying it. I've eventually just resorted to streaming at SVGA and decent framerates and then processing the output on another machine.
- mr_toad 4y agoIt makes sense if you can’t connect to another device, for example a sensor in a remote location.
- moffkalast 4y agoBut if you take the internet out of IoT it's just things.
- ottobonn 4y agoI signed up for updates! I have been looking for something exactly like this for home automation, to detect occupancy, pets, etc. I am very excited to pair it with HomeAssistant!
- h317 4y agoThanks mate! We will send the update within a few weeks with the production plan or buy options.
- drdaeman 4y agoHmm... does it really work as advertised? Can it really run some basic object detection in real-time on-device, and at what resolution/framerate combination(s)? Sorry for being skeptical, just that it sounds too incredible to believe. I've toyed around with the same chips you're using, OV2640 with ESP32-S3 and at any decent resolution it simply seemed to lack the processing power to run even simple motion detection, let alone anything fancier. Surely it was kind of fine at low-res and it kinda works spotting an elephant in a room, but it was completely incapable for detecting small fast-moving targets (roaches) under less than perfect lighting conditions (bathroom ceiling lights, decent but not overly bright). Best it could do was serving a 1024p@5fps MJPEG stream over a network to a more powerful machine for further processing.
- daveloyall 4y agoI'm skeptical, too. But apparently OP isn't the only one using (or trying to use) "TensorFlow-Lite for Microcontrollers" on an ESP32-S3. See this (little bit of) chatter: https://github.com/tensorflow/tflite-micro/search?q=esp32&type=issues https://github.com/tensorflow/tflite-micro/search?q=esp32&ty...
- drdaeman 4y agoI haven't tried but it must be doable at low-res low-framerate conditions, where CPUs still have plenty of time left between the frames and frames aren't big (so maybe it can even fully decode those JPEGs, not just extract the DC coefficients for a quick-and-dirty hacks). It's just the advertising page that sounds kinda unbelievable: low power, night vision, image analysis on the device, perfect for wildlife monitoring, can detect pests on crops (implying high resolution unless we're talking about deer and rabbits lol), etc etc.
- ge96 4y agoHere's a video of a guy squeezing an ML model into a Teensy granted the Teensy is pretty powerful https://youtu.be/6raRftH9yxM?t=436 https://youtu.be/6raRftH9yxM?t=436 edit: I will say processing IMU data is probably (a lot) easier than frame by frame of some video
- 4y ago
- graiz 4y agoThe ESP32 is a pretty standard sensor chip and the camera version is also widely available. A basic version of OpenCV was ported ~3 years ago, so it's cool but not sure what's new here.
- txtai 4y agoVery interesting! Looking forward to seeing more embedded AI devices coming out. More NLP based, but here is an article on an effort to build Transformers micromodels to run on embedded devices. The model in this example is under 1MB. Goal would be to ultimately convert this from ONNX to TFLite. https://neuml.hashnode.dev/train-a-language-model-from-scratch https://neuml.hashnode.dev/train-a-language-model-from-scrat...
- xnx 4y agoSemi-related: $80 board from Google: https://blog.tensorflow.org/2023/02/tensorflow-lite-micro-with-ml-acceleration.html https://blog.tensorflow.org/2023/02/tensorflow-lite-micro-wi...
- m00x 4y agoIf you can get them... I've tried to get one for over a year. You're probably better off getting a Jetson Nano.
- mitjam 4y agoAlso there is https://github.com/jomjol/AI-on-the-edge-device https://github.com/jomjol/AI-on-the-edge-device which digitalizes analoge water, gas, power and other meters. It uses the ESP32 camera module and local inference for character recognition. I found this very useful during the gas price hikes to identify effective saving measures. The inference speed is anything but realtime but for this use case fast enough. I took measurements every 5 minutes, the project states: Values smaller than 2 minutes do not make sense, as this is the time for one detection.
- grepLeigh 4y agoVery cool! I love that y'all integrate with Edge Impulse as well. Will you be at the TinyML Summit, March 27-29? https://www.tinyml.org/event/summit-2023 https://www.tinyml.org/event/summit-2023
- h317 4y agoThere is a quite a big chance we will be there!
- iamflimflam1 4y agoThe ESP32 is very capable - I made a voice controlled robot using TFLite a couple of years ago: https://youtu.be/cp2qRrhaZRA https://youtu.be/cp2qRrhaZRA
- rozzie 4y agoNot unrelated, this is quite an amazing device - even before you consider its $9.95 price at Sparkfun. https://usefulsensors.com/person-sensor/ https://usefulsensors.com/person-sensor/ https://usefulsensors.com/about/ https://usefulsensors.com/about/
- dang 4y agoI realize hardware project are, er, harder, and I buy that this project is real, but at the same time: signup pages and fundraisers aren't supposed to be Show HNs. Please see https://news.ycombinator.com/showhn.html https://news.ycombinator.com/showhn.html. So I've taken Show HN out of the title now.
- m00x 4y agoIf you want to use it for video, I'd wait until the ESP32-P4 releases. You could do some rudimentary AI on the ESP32-S3, but you can't send video at a decent rate. The P4 will have H.264 encoding, MIPI connectors, and a 400mhz dual-core. MJPEG isn't a great way to stream video, and the AI has to be super small to fit into SRAM. You could use flash over SPI, but that's not the greatest. You can get some decent results, but not for anything I thought was interesting for actual use.