8 ms·
gVisor is open-source, and `cuda-checkpoint` is freely available. gVisor's `runsc checkpoint` subcommand supports a `--save-restore-exec-argv` which lets you s
by eperot 3mo ago
gVisor is open-source, and `cuda-checkpoint` is freely available.
gVisor's `runsc checkpoint` subcommand supports a `--save-restore-exec-argv` which lets you specify a program to execute before gVisor starts taking the process snapshot.
You can fill in the blanks from there.
- za_mike157 3mo agoUs and the team from Modal have been upstreaming things to the GVisor repo (https://github.com/google/gvisor/pulls https://github.com/google/gvisor/pulls) in order to make it compatible with cuda-checkpoint and other parts of our system. While we are both contributing fixes and performance improvements we are unfortunately leaving some secret sauce on the side but hopefully it should get most folks to a successful implementation as is
- ranjanayush 3mo agoRecently gVisor added cuda checkpoint automation support, please consider using that: https://github.com/google/gvisor/commit/b3c184ce6b808648ddcaed4e59744200015d8c0b https://github.com/google/gvisor/commit/b3c184ce6b808648ddca...
- za_mike157 3mo agoThanks for this! We did see it and it was committed long after we already implemented this functionality. However, there are a lot of edge cases that this commit doesn't deal with that users are going to have to do