8 ms·
Running a 28.9M parameter LLM on an $8 microcontroller
- codelion 2mo ago[dead]
- deleted 2mo ago[deleted]
- cr125rider 2mo ago9.7 tokens/sec actually seems like a lot! That’s fun!
- althea_tx 2mo agoThis is a really cool project. Thanks for sharing!
- rao-v 2mo agoThis is a really neat use of the per-layer embedding trick. It's also worth noting that there viable TTS models that are ~20-30M param, so it might mean you can have a ESP32 with no network access read stuff out to you in near real time!
- Lerc 2mo agoOne of the things I have been wanting to try for a while now is something like this with a layer per MCU. I have some crazy ideas with RP2350's talking to each other with dedicated lines fed by PIO going through a combination of interpolators and dual multiply instructions. PSRAM, Flash, and even SD cards may not have the best bandwidth individually, but they can reach quite impressive rates when you have a shitton of them running all at the same time. The large scale dedicated hardware systems will still have the edge for performance per watt, but the low entry level and slow incline does make these things quite appealing.
- NuclearPM 2mo agoRun the numbers before you waste time here. I doubt this will work.
- Lerc 2mo agoPIO to PIO between two rp2350s should be able to transfer as many bits per clock as you can spare pins for. They have a single cycle double multiply per core, and the interpolators give you a heap of ability The PIO can be awkward, but you can run a bunch of them at once. Going from MCU to MCU you don't even need to involve the CPU cores, PIO to PIO Comms via pins You are obviously not going to get big TOPS from it because a Trillion is a ridiculous amount anyway. But never underestimate the power of controlling the whole pipeline. Ultimately none of the other things I'm doing with MCUs are practical, why would this to be any different.
- JK-Swizzle 2mo agoRunning some quick numbers shows you should be able to get >1Gbps. But I seriously doubt you could get those speeds in reality. You would need to get them perfectly in sync, which would likely take a dedicated board and some great knowledge of the oscillator. As someone who has done a reasonable amount with PIO, I do not think this is possible. However, that should not stop you. If you get it to work, please ping me.
- hgoel 2mo agoWouldn't a layer per MCU be heavily bandwidth constrained?
- monocasa 2mo agoDepending on where you slice the model up, it can be not a whole lot of data. For instance each transformer block outputs a single vector in an embedding space. I can see that being cheaper to bitbang with PIO than to actually compute. There's certainly some latency stack up, but throughput should be remarkably good.
- chrishynes 2mo agoWhy can't this scale to run much larger models on CPU backed by flash with good access patterns?
- AussieWog93 2mo agoSomeone did this exact thing recently, but running GLM-5.2 with something like 16GB of DRAM, a standard desktop CPU and nVME SSD. I think they got something like 10 _seconds per token_ (not tokens per second). EDIT: it was 25GB of ram and up to 20 seconds per token! https://github.com/JustVugg/colibri https://github.com/JustVugg/colibri
- 3eb7988a1663 2mo agoThat's incredible. Sure, not practical for most applications, but if you really want a local top tier model, you can run it on anything as long as you are patient. As someone with a healthy amount of RAM, but just a 16GB GPU, I am wondering what kind of work I could queue up for overnight runs. I thought the best models were fully out of reach, but the 128GB CPU only test had a 1.8 tokens/second. While not speedy, you could probably do something with that given extensive coffee breaks. This speed simulator[0] demos what it looks like. [0] https://shir-man.com/tokens-per-second/?speed=1.8 https://shir-man.com/tokens-per-second/?speed=1.8
- qingcharles 2mo agoI used to run Povray as a kid by setting up a scene in the evening and then letting it render all night and admiring the output in the morning. I could see where you could set up some coding task and let it churn all night.
- kamranjon 2mo agoSo while SSD streaming is interesting I'm not sure it's exactly the same thing as the per-layer embedding that is being utilized in tandem with streaming here. To utilize per-layer embedding, it would have had to be trained that way, which GLM 5.2 was not.
- 2mo ago
- rvba 2mo agoWont this wear out the flash memory quickly? I wonder hiw many read cycles can it survive
- logifail 2mo agohttps://docs.espressif.com/projects/esp-idf/en/stable/esp32/api-reference/storage/wear-levelling.html https://docs.espressif.com/projects/esp-idf/en/stable/esp32/... It's write and/or erase cycles that cause wear.
- vitally3643 2mo agoReads are effectively infinite. I'm not aware of any upper limit shorter than "end of the universe" timescales. Flash wears out from writing and the answer is in the tens to hundreds of thousands of writes per cell.
- fennecbutt 1mo agohttps://en.wikipedia.org/wiki/Electromigration https://en.wikipedia.org/wiki/Electromigration See also ds/3ds flash used in game carts. They need to be powered on every so often to avoid bitrot. Their memory controllers use a form of ecc that causes weak bits to be rewritten. But if it's left too long then ecc cannot correct. This is over 15-30 year timescale fwiw. But all chips will die in like 50-100 years or so because physics gets em with electromigration.
- smy20011 2mo agoSuper cool, thank you for sharing!
- spacedoutman 2mo ago>esp32-s3 This microcontroller is a beast, currently using it to do dev work on a pi4. Having two usb ports with one otg lets you do some neat things that would cost $100+ otherwise
- raphlinus 2mo agoIf you want to do this at the $1 price point, you can on RP2350, albeit with some limitations. In particular, it maxes out at full speed (12Mbps). The trick is to use the on-chip USB peripheral for one, and connect the other to GPIO pins backed by PIO. This works today with tinyusb and pico-pio-usb, but I'm also playing with a Rust port which I'm hoping will have higher performance.
- Rohansi 2mo ago$8 ish gets you an ESP32-S3 board with PSRAM, flash, and two USB-C ports. The PSRAM and flash are specifically used for this LLM project. I can't find anything like that with the RP2350 for $1.
- raphlinus 2mo agoEntirely fair, $1 is just the chip, not the board. No question the ESP32-S3 is incredibly good value.
- titzer 2mo agoIt's crazy what $5 can buy you in a microcontroller these days. Have a look at these Milk-V boards: https://milkv.io https://milkv.io The duo has up to 256MB of memory, and a 1TOPS@INT8 TPU. They run Linux and are $5. I bought 5!
- skippyfish 2mo agoYou can buy sub-$0.50 microcontrollers. But even at $5, I don't know why you'd want to run models on them, it's an environment constrained to the point of being useless for this task. And I hope it stays that way, I don't want MCU shortages...
- titzer 2mo agoI want to run music models on the Milk-V and have them jam in realtime with me.
- NooneAtAll3 2mo agoisn't floor price 0.10$ for last couple years?
- skippyfish 2mo agoWell, that's sub-$0.50. But yeah, CH32V003 is in that ballpark, and some of the cheapest Microchip and Infineon products are around $0.20. It's almost never worth it to buy the cheapest chip unless you're making a million of something, but there are very good ones around $1-$2, and $5 is the upscale stuff.
- londons_explore 2mo agoThere are 2 cent microcontrollers. Used in the cheapest devices you wouldn't even imagine to contain any software - eg. To control a touch sensitive on-off switch.
- NooneAtAll3 2mo agowhere can I learn about those?
- ReactiveJelly 2mo agoFinally, an LLM that can run on my i5
- NooneAtAll3 2mo agoWhile running LLM on tiny device is awesome, I'm more impressed by whatever training has produced the weights
- DoctorOetker 2mo agoIt's quite sad people collectively behave as if leaderboards have served their time. In the small parameter regime there is no room for benchmaxxing, so instead of leaderboards becoming useless, their utility was merely reduced to establishing ever smaller models with similar performance on the benchmarks, forcing compression or redundancy to be recognized and eliminated at the modeling level.
- kamranjon 2mo agoPretty incredible performance for the footprint - really interested to see what could be done on slightly more powerful SBCs like some that have been mentioned in this thread.
- helsinkiandrew 2mo agoVoice-to-Text and Text-to-Voice models are approaching that size. I wonder how close we are to getting small devices that can chat with us. Imagine a world where your toothbrush could give you tips about dental hygiene - or advertise toothpaste. What a time to be alive!
- EtienneK 2mo agoAnd this is why I am extremely bullish on Apple.
- mschild 2mo ago> tips about dental hygiene They already can. Philips' higher end models have Bluetooth connections and provide feedback through the app. > advertise toothpaste God please no. I don't want to have to look for an adblocker for a toothbrush AI.
- left-struck 2mo agoMy goto for what I hate about modern tech is toothbrushes having Bluetooth and needing apps. Not that I hate all modern tech but if it needs an app I probably will.
- mschild 2mo agoI'm with you there. My fridge has a wifi connection with an app. The only function it can do is notify me if the door is open. Thats it. Absolute waste of resources. I openend the enclosure and removed the antenna of course.
- walrus01 2mo agoPLEASE DRINK A VERIFICATION CAN
- mcgin 2mo agoHow lovely j
- caruasdo 2mo agoHow accurate is this quantized model?
- anonymous344 2mo agoi don't care about microcontroller, what is the decent option to have local llm in my raspi4 that does not take 30 seconds to answer?
- Rohansi 2mo agoRun a smaller LLM. It won't be as smart but the one shown here isn't either.
- daflkfdslkfds 2mo ago[dead]
- fintuner 2mo ago[flagged]