5 ms·

Subsecond: A runtime hotpatching engine for Rust hot-reloading

220 points by varbhat 1y ago

mmastrac 1y ago
I'll have to give this a shot for some of the Rust server work I'm doing. progscrape.com uses a lot of tricks to boot quickly specifically because of the edit-compile-run cycle being slow (mostly deferred loading of indexes, etc). My current day job @ Gel has me working on a Rust socket frontend for some pretty complex code and that could also be pretty interesting. It seems to require that you choose a good "cutover" point in your codebase, but TBH that's probably not too hard to pick. The HTTP service handler in a webserver, the socket handlers in non-web serving code, etc. It does appear to have a limitation where it will only allow the main crate to be hotpatched. That's less than ideal, but I suppose the convenience might justify some code structure changes to allow that.
jkelleyrtp 1y ago
Creator here - haven't had a chance to write up a blog post yet! Stay tuned. The gist of it is that we intercept the Rust linking phase and then drive `rustc` manually. There's some diffing logic that compares assembly between compiles and then a linking phase where we patch symbols against the running process. Works across macOS, Windows, Linux, iOS, Android, and WASM. On my m4 I can get 130ms compile-patch times, quite wicked stuff. We handle the hard parts that the traditional dylib-reloading doesn't including TLS, statics, constructors, etc. I've been posting demos of it to our twitter page (yes twitter, sorry...) - With bevy: https://x.com/dioxuslabs/status/1924762773734511035 https://x.com/dioxuslabs/status/1924762773734511035 - On iOS: https://x.com/dioxuslabs/status/1920184030173278608 https://x.com/dioxuslabs/status/1920184030173278608 - Frontend + backend (axum): https://x.com/dioxuslabs/status/1913353712552251860 https://x.com/dioxuslabs/status/1913353712552251860 - Ratatui (tui apps): https://x.com/dioxuslabs/status/1899539430173786505 https://x.com/dioxuslabs/status/1899539430173786505 Our unfinished release notes are here: https://github.com/DioxusLabs/dioxus/releases/tag/v0.7.0-alpha.0 https://github.com/DioxusLabs/dioxus/releases/tag/v0.7.0-alp... More details to come!
- 1oooqooq 1y ago
  can't access the xitter posts... is the axum part using the whole of dioxus or bare axum + code reloading?
  - jkelleyrtp 1y ago
    There's a custom `axum::serve` equivalent we built that wraps the router construction in a hot-patchable function. When the patches are loaded, we reset the TCP connections. It's a little specific to how dioxus uses axum today, but we plan to release an axum-only integration in the future.
    - 1oooqooq 1y ago
      awesome. thanks. i will definitely follow the project and hopefully participate. lack of hot reload for the FE folks is the biggest blocker we get from other options.
- deleted 1y ago
  [deleted]
- sureglymop 1y ago
  The axum example looks amazingly useful! Very cool project and idea.
- bjackman 1y ago
  [How] do you track when it's safe to delete the old version of a patched piece of code? Edit: or, I guess since this doesn't seem to be something intended for use in prod, maybe that's not necessary. You can just bloat the runtime process more or less indefinitely. I was curious because IIUC Linux kernel livepatches handle this via something related to RCU, which I guess is not possible in this context.
- deleted 1y ago
  [deleted]
weinzierl 1y ago
Very nice. For a long time I wondered who would use hotpatching but working with large Java applications made me appreciate the possibility even if it is not 100% reliable (as it is in Java). From the docs Subsecond looks almost perfect. The only downside I found is that (if I understood correctly) you have to modify the function call in the source code of every function you want to hotpatch. It is a bit mitigated in that the change does not cost anything in release builds, but it still is a big thing. Do I want sprinkle my code with call for every function I might potentially have to patch in a debugging session?
- jkelleyrtp 1y ago
  Creator here - you only need one `subsecond::call` to hook into the runtime and it doesn't even need to be in your code - it can be inside a dependency. Currently Dioxus and Bevy have subsecond integration so they get automatic hot-patching without any end-user setup. We hope to release some general purpose adapters for axum, ratatui, egui, etc.
  - weinzierl 1y ago
    Very nice! Thanks.
  - vlovich123 1y ago
    Could there be general purpose adapters for something like tokio more broadly so that if I have an app that’s not based on a framework I can leverage this?
  - juancampa 1y ago
    Amazing! Looking forward to the egui adapter
- toast0 1y ago
  > For a long time I wondered who would use hotpatching As someone who used a lot of hotpatching and now can't... This isn't aimed at production, but ... Hotpatching is essential to update code without losing program state. A lot of people work in http request/response stuff and program state lasts the duration of the request/response; you don't usually need hotpatching for that unless your responses are very long --- there's lots of ways to swap in new code where requests after some time hit the new code and requests that started in old code finish and that's usually what you want. If you've got something with long requests and you might want to change things during the request, hot patching is needed. If you've got something with long running connections or some other elaborate session, hot patching eliminates a lot of disconnect/reconnect session movement and lets you get everything running on the new version with a lot less hassle; as long as you accept the hassles of hot patching.
prideout 1y ago
Neat but I would prefer simply using a dylib for the part of my code that I want to be reloadable.
- whytevuhuni 1y ago
  I've used this for a toy game I was once working on [1], and it works pretty well for a while, but the problem is that sometimes the OS decides that your dlclose will be a no-op. I haven't ever found a way to force the OS to unload it, sometimes it just keeps it there. [1] https://github.com/andreivasiliu/demimud/tree/master/netcore https://github.com/andreivasiliu/demimud/tree/master/netcore
modeless 1y ago
Interesting, but the documentation makes it sound like you have to preemptively wrap all the code you think you might want to change in a special wrapper "call" function. If true that makes this a lot less appealing than existing solutions for other languages that can modify any function without special annotations.
- jkelleyrtp 1y ago
  You basically need to wrap your program's `tick()` function. Otherwise you might be in the middle of malloc, hot-patch, and your struct's layout and alignment changes, and your program crashes due to undefined behavior. The goal is that frameworks just bake `subsecond::current` into their `tick()` function and end-users get hot-patching for free.
  - deleted 1y ago
    [deleted]
  - jesse__ 1y ago
    How would you preempt the running program during malloc? Isn't there a well-defined reload point? Major red flags going up if your program can just change at any random point.. Also, didn't the article say explicitly that struct layout changes aren't supported??
    - anderskaseorg 1y ago
      There is a well-defined reload point—it’s the `subsecond::call` wrapper around `tick()`. But the hypothetical design that you seem to have in mind where this doesn’t exist would not have a well-defined reload point, so it would need to be able to preempt your program anywhere. Layout changes are supported for structs that don’t persist across the well-defined reload point.
- jesse__ 1y ago
  Strong agree here. The 'purity' BS of not modifying the running programs address space appears to come at the cost of significant programmer pain-in-the-ass. Having to hand-hold the library to maintain the indirection table is a hard no for me. Metaprogramming that maintenance burden seems like it should be relatively straight-forward, if you've written a linker already.
written-beyond 1y ago
I really want rust dylibs to be a reality. A plugin system where a library can implement a specific versioned number of a trait and we can dynamically load in that implementation to get changed behaviour. Right now implementing anything like that requires a lot of unsafe which I'm not comfortable with.
- ameliaquining 1y ago
  Do stabby and abi_stable not work for your use case?
cyberax 1y ago
Can we have the same for Go, pretty please?
- deleted 1y ago
  [deleted]
- mdaniel 1y ago
  > The gist of it is that we intercept the Rust linking phase and then drive `rustc` manually. There's some diffing logic that compares assembly between compiles and then a linking phase where we patch symbols against the running process makes it sound like it's actually not super rust specific, just a ton of squinting at assembly dumps. I do wonder if there are some "reproducible builds" traps hiding in this, but maybe it just needs to be selective about what it patches, and that turns out to be the really hard problem I didn't follow the tick() problem cited elsewhere, but my mental model is that, sure, one should not patch currently executing code without expecting "well that didn't do what I wanted" type shenanigans
cchance 1y ago
Wonder if the other web frameworks like leptos will adopt subsecond or adopt their own
majke 1y ago
About structs: https://docs.rs/subsecond/0.7.0-alpha.1/subsecond/index.html#struct-layout-and-alignment https://docs.rs/subsecond/0.7.0-alpha.1/subsecond/index.html... "In practice, frameworks that implement subsecond patching properly will throw out the old state and thus you should never witness a segfault due to misalignment or size changes. Frameworks are encouraged to aggressively dispose of old state that might cause size and alignment changes." I don't think "throwing out the old state" is a sensible recommendation. "re-instancing" is called "update/upgrade/downgede of internal state" in OTP: https://www.erlang.org/docs/24/man/gen_server#Module:code_change-3 https://www.erlang.org/docs/24/man/gen_server#Module:code_ch... Perhaps I'm missing something, maybe subsecond is a good tool for toy apps or for a developer workflow. But for anything serious, I'd think that managing layout of structs is a primary concern.
- jkelleyrtp 1y ago
  It’s a developer tool to speed up rust iteration, not a production tool like erlang. That being said, bevy is using bevy-reflect to implement proper struct hot-reloading between hot-patches.
- conradludgate 1y ago
  > Subsecond is only enabled when debug_assertions are enabled so you can safely ship your application with Subsecond enabled without worrying about the performance overhead. debug_assertions is usually only enabled in the development phase (release build default to debug_assertions=false), so yes this is not intended for production workloads
eigenspace 1y ago
I would recommend looking into how julia handles code reloading with our builtin infrastructure and the Revise.jl package. Basically, every method to a function and as of v1.12 (currently in beta), every binding and struct definition has a "world-age" associated with it. Basically, julia is dynamically typed, but inside a function it acts like a statically type language within a fixed world-age. So that means that based on the input types to a function, the compiler is able to know that the method table is not allowed to change, const-global, and types also can't change. Between world-ages however, anything goes. You can redefine methods, redefine structs, etc. What's especailly nice is that old data doesn't become invalid, it's just living in an old world, and if you get a Foo struct you can know what world it comes from. We have an invokelatest and invoke_in_world functions for advancing the world-age inside functions, and users creating something like an event loop just wrap their event loop iterations in an `invokelatest` and suddenly everything hot reloads automatically as you change code in your editor.
- ameliaquining 1y ago
  This sounds cool from a language-design perspective, but impossible to use a design like this in Rust. Unless I've misunderstood how it works?
schonfinkel 1y ago
This is great, I wonder if it was inspired by Erlang/OTP hot code reloading.

220 points by varbhat 1y ago

I'll have to give this a shot for some of the Rust server work I'm doing. progscrape.com uses a lot of tricks to boot quickly specifically because of the edit-compile-run cycle being slow (mostly deferred loading of indexes, etc). My current day job @ Gel has me working on a Rust socket frontend for some pretty complex code and that could also be pretty interesting. It seems to require that you choose a good "cutover" point in your codebase, but TBH that's probably not too hard to pick. The HTTP service handler in a webserver, the socket handlers in non-web serving code, etc. It does appear to have a limitation where it will only allow the main crate to be hotpatched. That's less than ideal, but I suppose the convenience might justify some code structure changes to allow that.

jkelleyrtp 1y ago

Creator here - haven't had a chance to write up a blog post yet! Stay tuned. The gist of it is that we intercept the Rust linking phase and then drive `rustc` manually. There's some diffing logic that compares assembly between compiles and then a linking phase where we patch symbols against the running process. Works across macOS, Windows, Linux, iOS, Android, and WASM. On my m4 I can get 130ms compile-patch times, quite wicked stuff. We handle the hard parts that the traditional dylib-reloading doesn't including TLS, statics, constructors, etc. I've been posting demos of it to our twitter page (yes twitter, sorry...) - With bevy: https://x.com/dioxuslabs/status/1924762773734511035 https://x.com/dioxuslabs/status/1924762773734511035 - On iOS: https://x.com/dioxuslabs/status/1920184030173278608 https://x.com/dioxuslabs/status/1920184030173278608 - Frontend + backend (axum): https://x.com/dioxuslabs/status/1913353712552251860 https://x.com/dioxuslabs/status/1913353712552251860 - Ratatui (tui apps): https://x.com/dioxuslabs/status/1899539430173786505 https://x.com/dioxuslabs/status/1899539430173786505 Our unfinished release notes are here: https://github.com/DioxusLabs/dioxus/releases/tag/v0.7.0-alpha.0 https://github.com/DioxusLabs/dioxus/releases/tag/v0.7.0-alp... More details to come!

1oooqooq 1y ago
can't access the xitter posts... is the axum part using the whole of dioxus or bare axum + code reloading?
- jkelleyrtp 1y ago
  There's a custom `axum::serve` equivalent we built that wraps the router construction in a hot-patchable function. When the patches are loaded, we reset the TCP connections. It's a little specific to how dioxus uses axum today, but we plan to release an axum-only integration in the future.
  - 1oooqooq 1y ago
    awesome. thanks. i will definitely follow the project and hopefully participate. lack of hot reload for the FE folks is the biggest blocker we get from other options.
deleted 1y ago
[deleted]
sureglymop 1y ago
The axum example looks amazingly useful! Very cool project and idea.
bjackman 1y ago
[How] do you track when it's safe to delete the old version of a patched piece of code? Edit: or, I guess since this doesn't seem to be something intended for use in prod, maybe that's not necessary. You can just bloat the runtime process more or less indefinitely. I was curious because IIUC Linux kernel livepatches handle this via something related to RCU, which I guess is not possible in this context.
deleted 1y ago
[deleted]

weinzierl 1y ago

Very nice. For a long time I wondered who would use hotpatching but working with large Java applications made me appreciate the possibility even if it is not 100% reliable (as it is in Java). From the docs Subsecond looks almost perfect. The only downside I found is that (if I understood correctly) you have to modify the function call in the source code of every function you want to hotpatch. It is a bit mitigated in that the change does not cost anything in release builds, but it still is a big thing. Do I want sprinkle my code with call for every function I might potentially have to patch in a debugging session?

jkelleyrtp 1y ago
Creator here - you only need one `subsecond::call` to hook into the runtime and it doesn't even need to be in your code - it can be inside a dependency. Currently Dioxus and Bevy have subsecond integration so they get automatic hot-patching without any end-user setup. We hope to release some general purpose adapters for axum, ratatui, egui, etc.
- weinzierl 1y ago
  Very nice! Thanks.
- vlovich123 1y ago
  Could there be general purpose adapters for something like tokio more broadly so that if I have an app that’s not based on a framework I can leverage this?
- juancampa 1y ago
  Amazing! Looking forward to the egui adapter
toast0 1y ago
> For a long time I wondered who would use hotpatching As someone who used a lot of hotpatching and now can't... This isn't aimed at production, but ... Hotpatching is essential to update code without losing program state. A lot of people work in http request/response stuff and program state lasts the duration of the request/response; you don't usually need hotpatching for that unless your responses are very long --- there's lots of ways to swap in new code where requests after some time hit the new code and requests that started in old code finish and that's usually what you want. If you've got something with long requests and you might want to change things during the request, hot patching is needed. If you've got something with long running connections or some other elaborate session, hot patching eliminates a lot of disconnect/reconnect session movement and lets you get everything running on the new version with a lot less hassle; as long as you accept the hassles of hot patching.

prideout 1y ago

Neat but I would prefer simply using a dylib for the part of my code that I want to be reloadable.

whytevuhuni 1y ago
I've used this for a toy game I was once working on [1], and it works pretty well for a while, but the problem is that sometimes the OS decides that your dlclose will be a no-op. I haven't ever found a way to force the OS to unload it, sometimes it just keeps it there. [1] https://github.com/andreivasiliu/demimud/tree/master/netcore https://github.com/andreivasiliu/demimud/tree/master/netcore

modeless 1y ago

Interesting, but the documentation makes it sound like you have to preemptively wrap all the code you think you might want to change in a special wrapper "call" function. If true that makes this a lot less appealing than existing solutions for other languages that can modify any function without special annotations.

jkelleyrtp 1y ago
You basically need to wrap your program's `tick()` function. Otherwise you might be in the middle of malloc, hot-patch, and your struct's layout and alignment changes, and your program crashes due to undefined behavior. The goal is that frameworks just bake `subsecond::current` into their `tick()` function and end-users get hot-patching for free.
- deleted 1y ago
  [deleted]
- jesse__ 1y ago
  How would you preempt the running program during malloc? Isn't there a well-defined reload point? Major red flags going up if your program can just change at any random point.. Also, didn't the article say explicitly that struct layout changes aren't supported??
  - anderskaseorg 1y ago
    There is a well-defined reload point—it’s the `subsecond::call` wrapper around `tick()`. But the hypothetical design that you seem to have in mind where this doesn’t exist would not have a well-defined reload point, so it would need to be able to preempt your program anywhere. Layout changes are supported for structs that don’t persist across the well-defined reload point.
jesse__ 1y ago
Strong agree here. The 'purity' BS of not modifying the running programs address space appears to come at the cost of significant programmer pain-in-the-ass. Having to hand-hold the library to maintain the indirection table is a hard no for me. Metaprogramming that maintenance burden seems like it should be relatively straight-forward, if you've written a linker already.

written-beyond 1y ago

I really want rust dylibs to be a reality. A plugin system where a library can implement a specific versioned number of a trait and we can dynamically load in that implementation to get changed behaviour. Right now implementing anything like that requires a lot of unsafe which I'm not comfortable with.

ameliaquining 1y ago
Do stabby and abi_stable not work for your use case?

cyberax 1y ago

Can we have the same for Go, pretty please?

deleted 1y ago
[deleted]
mdaniel 1y ago
> The gist of it is that we intercept the Rust linking phase and then drive `rustc` manually. There's some diffing logic that compares assembly between compiles and then a linking phase where we patch symbols against the running process makes it sound like it's actually not super rust specific, just a ton of squinting at assembly dumps. I do wonder if there are some "reproducible builds" traps hiding in this, but maybe it just needs to be selective about what it patches, and that turns out to be the really hard problem I didn't follow the tick() problem cited elsewhere, but my mental model is that, sure, one should not patch currently executing code without expecting "well that didn't do what I wanted" type shenanigans

cchance 1y ago

Wonder if the other web frameworks like leptos will adopt subsecond or adopt their own

majke 1y ago

About structs: https://docs.rs/subsecond/0.7.0-alpha.1/subsecond/index.html#struct-layout-and-alignment https://docs.rs/subsecond/0.7.0-alpha.1/subsecond/index.html... "In practice, frameworks that implement subsecond patching properly will throw out the old state and thus you should never witness a segfault due to misalignment or size changes. Frameworks are encouraged to aggressively dispose of old state that might cause size and alignment changes." I don't think "throwing out the old state" is a sensible recommendation. "re-instancing" is called "update/upgrade/downgede of internal state" in OTP: https://www.erlang.org/docs/24/man/gen_server#Module:code_change-3 https://www.erlang.org/docs/24/man/gen_server#Module:code_ch... Perhaps I'm missing something, maybe subsecond is a good tool for toy apps or for a developer workflow. But for anything serious, I'd think that managing layout of structs is a primary concern.

jkelleyrtp 1y ago
It’s a developer tool to speed up rust iteration, not a production tool like erlang. That being said, bevy is using bevy-reflect to implement proper struct hot-reloading between hot-patches.
conradludgate 1y ago
> Subsecond is only enabled when debug_assertions are enabled so you can safely ship your application with Subsecond enabled without worrying about the performance overhead. debug_assertions is usually only enabled in the development phase (release build default to debug_assertions=false), so yes this is not intended for production workloads

eigenspace 1y ago

I would recommend looking into how julia handles code reloading with our builtin infrastructure and the Revise.jl package. Basically, every method to a function and as of v1.12 (currently in beta), every binding and struct definition has a "world-age" associated with it. Basically, julia is dynamically typed, but inside a function it acts like a statically type language within a fixed world-age. So that means that based on the input types to a function, the compiler is able to know that the method table is not allowed to change, const-global, and types also can't change. Between world-ages however, anything goes. You can redefine methods, redefine structs, etc. What's especailly nice is that old data doesn't become invalid, it's just living in an old world, and if you get a Foo struct you can know what world it comes from. We have an invokelatest and invoke_in_world functions for advancing the world-age inside functions, and users creating something like an event loop just wrap their event loop iterations in an `invokelatest` and suddenly everything hot reloads automatically as you change code in your editor.

ameliaquining 1y ago
This sounds cool from a language-design perspective, but impossible to use a design like this in Rust. Unless I've misunderstood how it works?

schonfinkel 1y ago

This is great, I wonder if it was inspired by Erlang/OTP hot code reloading.