7 ms·
Embedded Rust or C firmware? Lessons from an industrial microcontroller use case
- Rochus 5mo ago[dead]
- dgacmu 5mo agoAuthors are from STMicro, polytechnic Turin, Freie universitat Berlin, and Inria. Examined writing firmware for an IOT sensor platform. From the abstract: > Two teams concurrently developing the same functionality (one in C, one in Rust) are analyzed over a period of several months. A comparative analysis of their approaches, results, and iterative efforts is provided. The analysis and measurements on hardware indicate no strong reason to prefer C over Rust for microcontroller firmware on the basis of memory footprint or execution speed. Furthermore, Ariel OS is shown to provide an efficient and portable system runtime in Rust whose footprint is smaller than that of the state-of-the-art bare-metal C stack traditionally used in this context. It is concluded that Rust is a sound choice today for firmware development in this domain.
- noosphr 5mo ago[flagged]
- staticassertion 5mo ago> Rust is evolving far too fast to be used in code which needs to run for years to decades down the line. That statement deserves support.
- pezo1919 5mo ago[flagged]
- fluffybucktsnek 5mo agoWhy, given the existence of editions?
- whytevuhuni 5mo agoVery surprised to hear that, since editions are exactly the kind of mechanism Rust is using to make sure software will keep working unchanged for decades. The Rust compiler can build a 2024 edition application which depends on a 2015 edition library, which in turn depends on a 2018 edition library. Every crate can upgrade at their own pace, or even never at all.
- fluffybucktsnek 5mo agoOh, I'm aware about the benefits of editions. For some reason, I've misread OP's comment.
- staticassertion 5mo agoIt's a very strong, plainly stated assertion that contradicts stated goals of the project so it seems like it deserves supporting evidence. Editions exist to avoid breaking code over time.
- fluffybucktsnek 5mo agoAh, sorry. I got confused about your comment.
- jagged-chisel 5mo agoThe code won't magically stop running because the Rust community continued evolving the language. The old toolchains will be available if there's a compatibility change. Where's the problem exactly?
- ShyCodeGardener 5mo agoProbably just depends on what you are doing. Library support could move forward and new features / security updates for libraries that are not part of core Rust could possibly be an issue if they don't work on older versions. Might not matter for a lot of embedded, but if you are doing something like exposing functionality via a webserver or something that would be network-connected, then security updates in third-party libraries may be important. For example, it would be really easy for me to run old code that's pinned to something like Python 3.7, but if libraries have updated to Python 3.x without backwards compatibility, then I'm stuck using the out of date versions or just backporting myself.
- lawn 5mo agoAre you implying Rust will stop working or something?
- estebank 5mo agoCan you point at any piece of code from 5 years ago that doesn't work today?
- oytis 5mo agoI only tried Rust for small hobby projects, but I did experience weird code rot when you just leave the code there and after a while it does not compile. Might have something to do with how Cargo manages dependencies
- Aurornis 5mo ago> Might have something to do with how Cargo manages dependencies Build against the lockfile to use the same versions. Unless they were pulled from upstream, they won’t suddenly stop building against the same compiler version. Rustup makes it easy to switch compiler versions to get back to the same one you used, too.
- estebank 5mo agoEven if a crate is yanked, if you have the version in a lock file it will still download and build. (This was done precisely after seeing the left-pad incident.)
- tredre3 5mo agoI'm sure you have ways to entirely purge a crate. And the situation will arise that you need to do so. In which case all the old code will, indeed, break. Vendoring is the only solution to this but it's really discouraged in rust-land and there is no first-party support for it. You can kind of manually vendor your deps with cargo, and there are third party tools. But compare that to go-land where `go mod vendor` gets you 95-100% of the way there.
- kibwen 5mo ago> I'm sure you have ways to entirely purge a crate. No, the lesson from left-pad that every centralized package manager learned was that you cannot allow users to remove uploaded packages at their leisure. All outright code removal can only be done manually by the admins themselves, and it's unlikely to happen outside of some legal compulsion. > Vendoring is the only solution to this but it's really discouraged in rust-land and there is no first-party support for it. This is completely incorrect. Cargo ships with `cargo vendor` out of the box, it's neither discouraged nor unsupported by first-party tools: https://doc.rust-lang.org/cargo/commands/cargo-vendor.html https://doc.rust-lang.org/cargo/commands/cargo-vendor.html
- whiatp 5mo agoI'm curious what the concern is with the rust editions mechanics in place. Each crate gets to define the language edition it is compiled with. Even if dependencies up convert to later editions they can still be linked against by crates that are an older edition. As for the broader crate ecosystem, if crates you depend on drop support for APIs you depend on, that could cause you to get stuck on older unsupported releases. Though that is no different of a problem than any other language.
- xandrius 5mo agoWait, are you implying that code written in Rust somehow "rusts"?
- estebank 5mo agoCode in all languages bitrots. Even if your dependencies are "done", the language is unchanging, the toolchain mature, a vendor can introduce a new platform and all of a sudden your code won't compile anymore, because IBM introduced a new RISC server platform, or macOS changed the definition of time_t, or Windows blocked direct win32.DLL access (I know, a stretch), that your older libraries didn't know about.
- mananaysiempre 5mo agoStretch or not, MDAC can no longer be installed on Windows. (The Microsoft Data Access Components are a rollup of database interface libraries from when they seemed to do around one a year, to the point that Spolsky remarked on it[1].) This means a significant corpus of old but still 32-bit line-of-business apps no longer runs, like anything written in VB6 or VBA that needs to access a database. [1] https://www.joelonsoftware.com/2002/01/06/fire-and-motion/ https://www.joelonsoftware.com/2002/01/06/fire-and-motion/
- IshKebab 5mo agoRust is much more stable than C in my experience. Try compiling a decade old C code and see how that goes...
- irishcoffee 5mo agoHow do you figure? I do that weekly.
- JoshTriplett 5mo agoI've often found that trying to compile decade-old C code with a current toolchain and current libraries will have issues. It isn't always clear what versions the code is expecting (no equivalent to a lockfile), newer C compilers or standards can break old code, and newer libraries especially can break old code. It might still build if you could recreate exactly what it expects, but it becomes decreasingly possible to do that if you weren't compiling it a decade ago and archived off exactly what worked then.
- Aurornis 5mo ago> Rust is evolving far too fast to be used in code which needs to run for years to decades down the line. Code doesn’t stop running on existing hardware when the language changes in a future compiler. You can still use the same old toolchain. I’ve done a lot of embedded development in a past life. Keeping old tool chains around for each old platform was standard. I would much rather go through the easy process of switching to an older Rust tool chain to build something than all of the games we played to keep entire VMs archived with a snapshot of a vendor tool chain that worked to build something.
- whiatp 5mo agoI remember a coworker having to fight with an old platform's build not working because our user/group IDs were bigger than 2^16. I can't remember which utility was causing the problem, I'd have to guess tar. This is when we learned to play the archive a VM game.
- butvacuum 5mo agoI can't imagine theres much overlap between "we will need to update this firmware for the next decade." and "Let's bet the farm on the documentation being perfect, and all the downloads still available."
- cyberpunk 5mo agoI know a defence company that has a bunch of vaxes stored in low oxygen environments because they legally have to be able to provide software updates to firmware they’ve written for the next 20 or so years and it was written on a vax. They had some great stories trying to get something or other running again where they had to fly one of the original designers over to hand solder a board back into action. How we do that today is a bit of an interesting problem I don’t think they’ve convincingly solved; basically maintaining nightly builds forever — a couple 1U’s of kubernetes in deep storage ain’t gonna do it, you’re not gonna be able to solder a xeon back to life.. I know I’d rather be trying get a load of c99 rebuilt for some mips or other after 20 years that some random version of rust.
- kitd 5mo agoIsn't that (part of) the purpose of using something like Ariel OS, to isolate you from changes?
- api 5mo agoI think that was true maybe 5-10 years ago. We have Rust code in a living code base that is more than 5 years old and it's required maybe one touch in the last 5 years to fix some issues due to stricter rules. It was simple enough it could have been automated.
- vmg12 5mo ago> Rust is evolving far too fast I'm curious why I've seen this sentiment repeated in so many places, I learned Rust once 5 years ago and I haven't had to learn any new idioms and there have been no backwards incompatible changes to it that required migrating any of my code.
- fluidcruft 5mo agoI think people don't like the JavaScript treadmill. People want to think about using tools and getting proficient with them rather than relearning tools. I'm not saying rust is like that, but I do feel that way about python and JavaScript. Those are dynamic languages but it is what all this editions stuff evokes. It's an if it were stable, it wouldn't be changing sort of thing.
- nine_k 5mo ago> using tools and getting proficient with them rather than relearning tools This attitude works in carpentry, but not in software. You need to get proficient, but your tools will keep evolving, like everything else in the software world.
- estimator7292 5mo agoThat's exactly the point. This is not normal even in software. You can, in fact, learn C exactly once. Or any number of other languages. The entire argument being made here is that the world you're suggesting is a problem. Software developers should not have to continually relearn their tools and it is abnormal to suggest they should.
- dwattttt 5mo agoI've seen C written by people who learned it "exactly once", in let's say the 2000s. They're the same people who insist that all the safety & linting introduce since was pointless. I'll take C written by people who've learned and improved since then any & every day of the week.
- rowanG077 5mo agoRust is evolving too fast? Rust is evolving EXTREMELY slowly. I dare say even slower than C++ or Python.
- drzaiusx11 5mo agoRust uses "Editions" (e.g., 2015, 2018, 2021, 2024) to introduce breaking changes without splitting the ecosystem. Every edition remains supported by newer compiler versions _indefinitely_. The only churn is on projects targeting "nightlies" but there's no reason you can't target a stable one for projects that need that stability.
- jonathanstrange 5mo agoThat invariably leads to bitrot and low maintainability. It's one among many reasons why I don't use Rust.
- monocasa 5mo agoI don't see how that leads to bitrot, or low maintainability.
- orochimaaru 5mo agoYou have the same issue with C, no? C is upgrading versions, compilers have changed, hardware evolves and somethings in the past aren't supported as well anymore.
- megous 5mo agoNot really. Not in the scale of my C dev life, which has been 20 years so far.
- fluffybucktsnek 5mo agoNeither have we with Rust.
- orochimaaru 5mo agoI did C and C++ till about 2010 or so. The major platform which I did dev for (hp ux) is no longer supported.
- elcritch 5mo ago
- iknowstuff 5mo agoRust is stable. If you don’t plan on changing crates, newer compilers will compile your 5yo code.
- Lvl999Noob 5mo agoAren't there companies that still use C89 for their production systems? I don't know any in particular but I have read comments here on HN implying that. Just do the same for Rust. Stick to the one major version you started with instead of trying to update the toolchain regularly.
- pjmlp 5mo agoK&R C, C89, C90, C11, C17, C23, C2y. Not counting compiler specific extensions from GCC, clang, Microsoft, Intel, NVidia, AMD, IBM, Oracle, Apple, Green Hills, TI, Microbit, Mikroe, and many more C compiler vendors that could have been used to compile a specific project during the last 50 years.
- blub 5mo agoOne of the authors commented below that the “teams” were actually persons and the Rust person was an intern. This is even less serious than the typical pattern of grabbing random students for experiments and then drawing conclusions about the general population.
- torginus 5mo agoNot sure about your life experiences, but every new, from-scratch project I have undertaken has looked like 1-2 or at most 3-4 people on good terms who really pulled their weight, with the rest being basically not dead weight, but the management overhead they caused ate up most of the productivity they brought to the table.
- blub 4mo agoEven if that were the case, working in a team is very different from working solo.
- kaspar030 5mo agoOne of the author's here, if there are any questions!
- Galanwe 5mo agoIsn't there a nasty selection/volunteer bias at play with the developers?
- kaspar030 5mo agoYou mean with the "two teams" that were tasked to develop the C / Rust versions? Yeah of course. Then again - they were one person teams, where the C "team" had years of experience in stm32 / embedded C / stm32 cube development and churned out that handwritten state machine in just days. The Rust "team" was a pre-masters intern with only minimal embedded Rust experience. They ran into all the pitfalls with (async) embedded Rust, but corrected towards the end.
- jacquesm 5mo agoThat does not seem like even close to a fair comparison and makes me wonder how valid the conclusion is. Effectively this is two times n=1, if you use 'teams' when you actually mean 'individuals' then that's not really proper reporting. I do applaud you for having the same work done twice but it would have been far more meaningful to have two actual teams of seasoned developers do this sort of thing side-by-side. The biggest item on the checklist would be the number of undiscovered UB or UB related bugs in the C codebase and to compare that with the Rust codebase on 'defect escape rate' or some other meaningful metric.
- pitched 5mo agoI think there’s another hidden issue of testing how new devs use the language vs. those seasoned devs. I expect someone with a few months of experience would prefer Rust (fewer footguns) but someone with more experience would prefer C (the sharper knife). The flavour of the thing changes as we age.
- cmrdporcupine 5mo agoI'm a big fan of Rust on embedded (and think embassy in particular is awesome, haven't tried this Ariel OS.) I would say however that there's still toolchain issues here. There all kinds of MCUs that simply don't/won't have a viable compiler toolchain that would support Rust. e.g. I recently came from a job where they built their own camera board around an older platform because it offered a compelling bundle of features (USB peripheral support and MIPI interface mainly). We were stuck with C/C++ as the toolchain there, as there was no reasonable way to make this work with Rust as it was a much older ARM ISA
- serhack_ 5mo agooff topic question: why is there no source attached to this paper?
- kaspar030 5mo ago"The open source code will be published on https://github.com/stm32-hotspot/ https://github.com/stm32-hotspot/ for the final version of the paper." -> paper is not final. And IIUC ST will be releasing the code at some point.
- serhack_ 5mo agonope, I was meaning the TeX source for the paper https://info.arxiv.org/help/faq/whytex.html https://info.arxiv.org/help/faq/whytex.html
- fjfaase 5mo agoI find this a bit disappointing. Why not publish it with the preprint. Now we have no way to establish the quality of the two solutions or whether it is even possible to improve one of the solutions. I wonder why the C variant could not implement a JSON parser without malloc and free, while the RUST variant could.
- deleted 5mo ago[deleted]
- bArray 5mo ago> It is concluded that Rust is a sound choice today for firmware development in this domain. This conclusion was reached with a single experiment. > Two teams concurrently developing the same functionality — one in C, one in Rust — are analyzed over a period of several months. > Furthermore, Ariel OS is shown to provide an efficient and portable system runtime in Rust whose footprint is smaller than that of the state-of-the-art bare-metal C stack traditionally used in this context. > The authors thank Davide Aliprandi and Davide Sergi of the STAIoTCraft team, and the wider Ariel OS team. So one team had Ariel OS developer support, and it's unclear what support the other team had. Seems fair. In Figure 12, they simply stop optimizing the code once desired rate is reached. Just at the end of the project the Rust firmware gets over a third performance boost, most likely from their OS developers. Additionally, there is a claim that "Ariel OS is shown to provide an efficient and portable system runtime" - but there are no real tests for portability are conducted. Worst still: > Where C-based projects require a separate project setup and manual code copying per target, Rust on Ariel OS consolidates everything within a single project [..] This claim is just not true. This sounds like somebody that is not as familiar with C.
- ambicapter 5mo ago> This conclusion was reached with a single experiment. No shit. This is the conclusion reached at the conclusion of this experiment. This part of your comment can be removed with no loss of clarity, I think.
- bArray 5mo agoI think you miss my point. I don't think that this conclusion can be reached with the (singular) experiments performed because there is a lack of data to draw it. If I ran an experiment where I gave a cancer patient bread, and then they recovered from cancer, I couldn't then say: "It is concluded that <bread> is a sound choice today for <cancer treatment> in this domain.". You would rightfully jump up and down and demand further experiments to increase the confidence of the result before drawing the conclusion. It could have been concluded instead that there is a case for further experiments to be conducted, or that Rust could be approaching a maturity where it could be considered for some firmware projects. But as it stands, the conclusion is far too strong given the experiments performed.
- eggy 5mo agoWe passed on Rust for Ada/SPARK2014 to write to bare metal on Cortex-M processor for real-time, high-integrity, and verifiable mission-critical software. Rust is making strides to be a future competitor, but it's new to the formal verification tooling and lacks any real world legacy in our domain. Ada's latest spec. is 2022. Other than AdaCore's verified Rust compiler, Rust still does not have a stable language specification like C/C++, Lisp, or Ada, SPARK 2014. I have no doubt that it will start rising to tick all the boxes that Ada/SPARK do right now with their decades of legacy in high-intetrity, mission-critical applications. The mandate to use memory-safe software put into effect this past Jan 1 2026 puts some wind in Rust's sails, but it's more than memory-safety in this domain. Plus, I do not enjoy Rust, but Cargo is nice. We're looking at Lean for further assistance in verifying our work. I think there was and is lot of Rust evangelism that will also carry it forward and boost even more Rust popularity,
- Filligree 5mo agoPresumably, if you use formal verification then that includes memory safety anyway? Would seem strange if it does not.
- irishcoffee 5mo agoFormal verification requires a spec and a very large, very expensive amount of tooling to be developed. My understand is that both these things are in work, and that neither of these things exist yet.
- eggy 5mo agoYes, and AdaCore's tooling is formally verified and produces reports already familiar to aerospace, railway, and auto auditors for verifying certifications making it attractive to this industry segment of high-integrity apps. Memory safety is taken care of mainly through the features Ada/SPARK2014 offer in creating safe, high-integrity programs, correct.
- irishcoffee 5mo ago
- fjfaase 5mo agoReally strange the the C JSON parser has to use malloc where the RUST version does not. As if it is not possible to write a JSON parser in C that does use malloc. I presume that the syntax of the commands that the device will accept is known, and than there is no reason why you have to build a DOM of the JSON before you can process it. Apparently, the RUST version can do it. I really begin to question the abilities of two teams if the one team failed to implement a JSON parser solution without using memory allocations.
- megous 5mo agoYeah, you can comfortably work with JSON in C directly on top of the string buffer containing it. Your representation for any JSON entity will just be const char pointer. It's possible to implement JSON path on top of this, and all kinds of niceties, and it's not slow. Megatools is an example of such a code https://xff.cz/megatools/ https://xff.cz/megatools/ / https://xff.cz/git/megatools/tree/lib/sjson.c https://xff.cz/git/megatools/tree/lib/sjson.c
- kaspar030 5mo agoPart of the C protocol implementation is generated, and that generator chose the JSON parser. As it worked and there was plenty of memory left on the MCU, it was kept. We're mentioning this in the paper: "The heap is entirely attributable to Parson's dynamic allocation of JSON tree nodes; as memory usage minimization was not a key goal, we kept Parson (the JSON parser used by the PNPL code generator by default), noting that there are less memory heavy options that do not require a heap at all."
- dundarious 5mo agoWasn't memory one of the key indicators looked at? > The analysis and measurements on hardware indicate no strong reason to prefer C over Rust for microcontroller firmware on the basis of memory footprint or execution speed. I admit I have not carefully read the paper, and am collating info from comments here, so I may be fully mistaken. The word "strong" also allows for much interpretation, that I'm not a priori critical of, but am skeptical of.
- the__alchemist 5mo agoGood article! I will give you my 2c, as someone in this space mostly for hobbies, but with one active work project: Rust is fantastic for embedded. There are no hard obstacles. The reason to do it IMO is not memory safety, but because holistically the language and tools are (to me) nicer. Enums, namespacing, no headers, `cargo run --release` "just works". (I have found, at least in OSS, compiling C embedded project is a mess. Linker errors, you need to have a certain OS with certain dependencies, there are many scripts etc). Good error messages, easy to structure your programs in a reliable way etc. Overall, I just find it to be a better designed language. I have found the most fundamental tooling for rust on Espressif's RiscV, and Cortex-M ARM for various STM-32 variants to be great. The cortex-m crate, defmt, and probe-rs, and the PAC project is fantastic. On the down side, I have have to build my own tooling. I wrote and maintain my own HAL for STM32, and have had to write my own libraries for every piece of hardware. This comes with the territory of a new language, and suspect this will gradually improve this time - especially with vendor support. Because the fundamental libraries are around, this is just reading datasheets and making rust functions/structs etc that do the MMIO as described in the datasheets. Can be tedious (Especially if building a complete library instead of implementing what you need for a given project), but is not an obstacle. My most complicated rust embedded firmware was a FPV-style UAS. I did it without an RTOS, using interrupt-based control flow.
- acstapleton 5mo agoAs a professional in the space, this echoes my experiences. I’m far more productive in Rust than C, despite having many more years programming embedded software in C. Cargo and the crate ecosystem are a dream compared to the lack of any easy-to-use build tooling and the difficulty of integrating third party libraries in C. Furthermore, the code I’ve produced in Rust is generally as fast (or faster) than the code I’ve written in C for the same task, and it’s easier and faster to write. And I also maintain an open source HAL for an STM32 family! Previously those have just been in house HALs in C because there were no such community efforts.
- the__alchemist 5mo agoVery cool! Which one? I have been using mostly G4 and H7 on my personal and work projects, but the HAL (`stm32-hal2` is the crate name) works for most of the one s in a certain time band; i.e. not any that were obsolte when I started it, and spotty or no support on some of the newer ones like U series and H5. And weaker / non-vetted support on MCU variants I haven't used, or haven't used a MCU/feature combo on. I think in the future if I do an embedded rust project on a new MCU where there isn't an existing HAL, or one that is more work to repair than start over, I would just implement the subset needed for a project's reqs. Easier to keep track of scope that way. Currently the challenge is "X periph on Y variant of Z STM-32 family doesn't work under A condition" or "Doesn't work after this PAC update changed the syntax".
- davemp 5mo agoI'll start of by saying I really hate C (also love it), and welcome improvements; but I have a few criticisms: - Sensor agent is such a rancid name for a remote sensor that I feel a need to public say so. Please don't use marketing names for things that already have more descriptive names. - Rust uses a full RTOS and C uses the mediocre ST HAL (vendor specific). Immediately apples to oranges. Also I've never heard of the C JSON library and it looks sketchy at a glance so that will also hurt the comparison. - Streaming slow sensor data with a 160MHz 786KB/2MB MCU is not a good test in the slightest. You could probably use something like micro python here and be done. No one is reaching for bare metal C here. Also no one serious about performance is using JSON serdes. If you're using bare metal C, you're likely trying to push the limits of your hardware or doing something so simple that you won't be tempted to reach for terrible third party libraries. - Does the Rust code base use the 'unsafe' keyword anywhere, including the RTOS? If so, it's not memory safe without additional formal verification. Overall I'd say this paper has approximately zero value wrt its stated goal of comparison.
- ezekiel68 5mo agoLots of handwaving about fear of rust changing too often and even MORE downvoting of this as if there is no concern whatever. Neither of these extremes are valid. For me it comes down to the old standby: just vendor the .cargo build chain into your repo and be done with it. There you go. Lock in the year edition, the version, the features, the quirks, and the bugs. Just like game devs did with game engines or OpenGL or Direct X versions.
- cozzyd 5mo agoDo linker scripts look the same in the rust tool chain? (E.g. for implementing a bootloader?)
- QuiEgo 5mo agoYes. It uses the same linker as a C/llvm toolchain, by the time you link the object files from Rust are in the same format as the object files from C would be. You use the exact same linker script format too.
- torginus 5mo agoHhaving worked on quite complex embedded projects, my 2 cents is that dynamic allocation should be avoided as much as possible (and its been possible to avoid it 100% of the time for me). Memory layouts are often pre planned, with hand-written linker files, which are sometimes even necessary, as there are quirks like DMA only being able to access certain addresses etc. In embedded, engineers often want hard real time guarantees, very high (essentially unfailing) reliablity, at the lowest possible price points. Dynamic memory allocation is often no good - embedded allocators either waste RAM, have large code sizes, have worse runtimes than their desktop-grade cousins - something like jemalloc generates 10x as much code as the rest of your app, and assumes your heap is at least megabytes in size. On the other hand, embedded-grade allocators often use algorithms that are prone to fragmentation, unpredictable runtimes, and tend to be less tested in general, while still wasting RAM and Flash. Having an allocator that has a bad runtime behavior can basically ruin your hard runtime guarantees, and if you prealloc all you memory you will never run out - on the other hand, even the paper states they measured not calculated the max memory usage, meaning we have no knowledge of what the actual max is.
- TheMagicHorsey 5mo agoMy biggest gripe with Rust, which certainly reflects my own shortcomings, is that when I go back and revisit simple Rust programs I wrote more than a year ago, it takes me a long time to understand what I was even doing in a particular part of the program. This is my weakness ... I'm not great with Rust and I don't use it enough to get better. But it is what it is. In contrast, when I go back and read Go or C code I wrote years ago, I have no trouble at all quickly figuring out what I was doing in the small programs I write. The way these issues manifest themselves, as it recently did, was I went back to add a simple addition to a CLI tool I wrote for myself a year ago, and I was having trouble doing it because I couldn't quickly understand what I had been doing a year ago ... so I just had an AI agent do it for me. This was the kind of change that if it was a Go program, I would've done manually myself in about 5 or 10 mins.