6 ms·

Writing systemd units that stop gracefully before shutdown

228 points by dghubble 4y ago

buscoquadnary 4y ago
akeck 4y ago
I love this. Lots of details I didn't know.
kzrdude 4y ago
Halt is apparently not the same as poweroff.
- chasil 4y ago
  A "halt -fp" just unmounts file systems and immediately shuts down. I find that CentOS systems that I've used for a while seem to hang on shutdowns; halt -fp is a way to get them down quickly. It is important to terminate any sensitive processes beforehand.
  - SoftTalker 4y ago
    For systems that hang or take intolerably long to shutdown, I typically do: systemctl --force [poweroff|reboot] From the man page, this means that "shutdown of all running services is skipped, however all processes are killed and all file systems are unmounted or mounted read-only, immediately followed by the powering off."
- andrewaylett 4y ago
  Kids of today, etc. AT power supplies didn't have any mechanism for the system to tell the power supply it wasn't needed any more. So when you shut down the computer, it would wind up at a screen with a message approximating "it is now safe to switch off your computer", at which point the system would halt. ATX power supplies added the ability for the OS to trigger an actual power off. But that's a different end-state to halting, and if you halt the system then it stays on. You may wonder why anyone would want to halt when power off is an option, and to be honest I'm not entirely sure -- possibly because you have a hardware watchdog which will trigger a reboot of a halted machine but not of a powered off machine?
CGamesPlay 4y ago
A much more challenging task is writing a systemd unit that starts gracefully before shutdown. I wanted to write a unit that could issue an API call to delete the instance rather than doing a normal power off. Putting it at a reasonable place in the sequence took a lot of trial and error! The trick is actually to have the unit be started at some point in normal boot up (e.g. “armed”) and then do the actual task when the unit is stopped. Here’s the unit I ended up with: https://github.com/CGamesPlay/infra/blob/master/private-server/ansible/self-destruct.service https://github.com/CGamesPlay/infra/blob/master/private-serv...
- dghubble 4y ago
  That's what this post builds to solving at the end and in the next post - having a unit deletes the instance from a cluster before shutdown
- hinkley 4y ago
  I tend to push the metaphor of software being meant to be read and only incidentally to be run by a computer as far as I can. We are telling a story to future us or our successors. Stories have rules and it's jarring when you violate them. There's an idea in software that's a bit like the corollary of Chekhov's gun. Chekhov's gun is about not presaging jarring story elements that will never come to pass. But it's nearly as jarring to leave important story arcs as complete surprises until the end. Producing the gun moments before the curtain goes down would be quite a WTF. We didn't know that was a possibility. That's a niche that some people occupy, but it is a niche. Introducing things early fights with Locality of Reference, but when we're talking about things of deep, dramatic importance (like an attempted murder, or a reaper process) it's important to introduce that "character" early in the story so that people know that it exists. Failing to do so is a form of deus ex machina and we only appreciate that in very small doses. So framed that way, I don't see a problem with having to start a killswitch while you're spinning everything up. It's there, people can see it, and know to ask questions about it.
  - slivanes 4y ago
SoftTalker 4y ago
I fundamentally disagree with the idea that software should require or even expect a graceful shutdown. You can never stop the user from yanking the power cord out of the socket, which is what they will do if you force a bunch of housekeeping to happen before shutdown. You have to deal with crash/power failure recovery anyway. So do your housekeeping on startup. Shutdown should be a quick and simple termination.
- nerdponx 4y ago
  Hope for the best, plan for the worst, right? Here's a contrived analogy: modern airplanes are designed to stay in the air even if an engine burns out, but we would still rather fly with both engines at full power whenever possible.
- empthought 4y ago
  This is a weird take; most systems in data centers don’t have people walking from rack to rack yanking power cords, and most consumer systems don’t even have a power cord to yank.
  - bravetraveler 4y ago
    While I agree it's a bit of a weird take, for example -- there may be performance tradeoffs made in any given workload to make the disk consistent, inconsistently The 'most' there is doing some effort It is actually quite a common practice for those being audited for disaster recovery to do exactly that -- yank cables. More realistically, flip some switches We do it once a year, set aside a region and time... then test our processes It serves a few purposes, most importantly -- are our services fault tolerant, and can we bring them back? I think it's reasonable to trap the signals and make a best effort basis, knowing that PID 1 (or the environment) will eventually have to SIGKILL you -- ready or not Just because we can't save all of the state doesn't mean we shouldn't try
    - empthought 4y ago
      Right, there are failure modes that have to be tested and accounted for, and one of them is the state being inconsistent after a shutdown. The previous poster seemed to advocate for not thinking of this as a failure mode at all but rather normal operation, which I just don’t see as true.
exikyut 4y ago
Meta: I can't reach this website! Chrome is giving me an instant NXDOMAIN error. Dig shows that $ dig psdn.io @1.1.1.1 ... ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 19283 ... ;; QUESTION SECTION: ;psdn.io. IN A so then I prefix "www." like is in the URL... $ dig www.psdn.io @1.1.1.1 ... ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 64024 ... ;; QUESTION SECTION: ;www.psdn.io. IN A ;; ANSWER SECTION: www.psdn.io. 300 IN CNAME poseidon-www.pages.dev. Okay, fine: $ dig poseidon-www.pages.dev @1.1.1.1 ... ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 55471 ... ;; QUESTION SECTION: ;poseidon-www.pages.dev. IN A ...wat?? (Where there's no ANSWER section, none was returned, just an AUTHORITY section) This is reproducible for me with 1.1.1.1, 8.8.8.8 and 9.9.9.9.
- dghubble 4y ago
  Hmm, sorry you're not seeing it. Its just a CNAME to Cloudflare Pages, nothing fancy dig www.psdn.io @1.1.1.1 ; <<>> DiG 9.16.33-RH <<>> www.psdn.io @1.1.1.1 ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 37362 ;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1 ;; QUESTION SECTION: ;www.psdn.io. IN A ;; ANSWER SECTION: www.psdn.io. 126 IN CNAME poseidon-www.pages.dev. poseidon-www.pages.dev. 126 IN A 172.66.45.44 poseidon-www.pages.dev. 126 IN A 172.66.46.212
SrslyJosh 4y ago
It shouldn't be this hard to stop a service gracefully. This is far, far more complicated than SysV init, where you just need to drop a script into /etc/init.d symlink it from the appropriate rc directories. (For shutdown/reboot, you'd create symlinks in rc5.d and rc6.d named KNNwhatever, where NN is an integer that specifies the order the script will be run in. The "K" stands for "kill".) Edit: Note that my example runlevels are for Solaris, other UNIX/Linux OSes will vary.
- LukeShu 4y ago
  > It shouldn't be this hard to stop a service gracefully. It's not. > you just need to drop a script into /etc/init.d symlink it from the appropriate rc directories. You just need to drop a unit file into /etc/systemd/system/ and symlink it from the appropriate /etc/systemd/system/${target}.wants/ directories. Don't tell me that "shutdown.target.wants" and "reboot.target.wants" are harder than "rc0.d" and "rc6.d". A lot of the article is about ordering of dependencies (don't stop a dependency until after the dependent has stopped). Don't tell me that adding `Before=` and `After=` lines in the unit file is harder than having to remember all of the dependencies and manually figure out the correct "NN" for it all to work correctly. A lot of the article is about either having your daemon handle SIGTERM, or coming up with the appropriate `ExecStop=` command. The same command you'd be writing in your rc script (the "handle SIGTERM" stuff being for if your rc script simply says `kill $PID`). That is: The complex parts of the article are things that were complex with sysvinit too.
  - ivan23178 4y ago
    There's also middle ground. OpenRC doesn't need those weird numbers while still being simple sysvinit/bash scripts-based system (it has a simple dependency system too). I'm not completely sure about `shutdown` level scripts having access to network though (too lazy to check), but it's still worth mentioning here.
    - LukeShu 4y ago
      OpenRC's great! But it doesn't detract from the point that systemd isn't nearly as complicated as some people act like it is. PS: Who's still using OpenRC with heavy ol' sysvinit instead of the lighter openrc-init?
rfmoz 4y ago
The macOS init manager, LaunchD, doesn't offer an easy way to execute an script at shutdown. By definition, it sends a SIGTERM signal to all of the daemons that it started. But as the script isn't started before, and doesn't keep a running PID, you don't have a clean way to do it. I don't understand why they only implemented the SIGTERM call without any alternative.
MarkusWandel 4y ago
Quoting from the article: TimeoutStopSec=0 That's cost me more than one hard power button powerdon (on a desktop machine with the system partition on SSD - unnerving). One of the innumerable things that systemd stops on shutdown gets stuck - permanently - and the machine goes into a state out of which, to my knowledge, is only a powerdown or reset. I ended up searching for the above and replacing them with a reasonable timeout (several minutes).
- dghubble 4y ago
  That's fair, that example would be better with a modest timeout (e.g. if podman had a bug that caused it to hang), without taking away from the main points. Updated. Someone asked about the opposite on Twitter: https://twitter.com/DannoHung/status/1585350836074446869 https://twitter.com/DannoHung/status/1585350836074446869

228 points by dghubble 4y ago

buscoquadnary 4y ago

akeck 4y ago

I love this. Lots of details I didn't know.

kzrdude 4y ago

Halt is apparently not the same as poweroff.

chasil 4y ago
A "halt -fp" just unmounts file systems and immediately shuts down. I find that CentOS systems that I've used for a while seem to hang on shutdowns; halt -fp is a way to get them down quickly. It is important to terminate any sensitive processes beforehand.
- SoftTalker 4y ago
  For systems that hang or take intolerably long to shutdown, I typically do: systemctl --force [poweroff|reboot] From the man page, this means that "shutdown of all running services is skipped, however all processes are killed and all file systems are unmounted or mounted read-only, immediately followed by the powering off."
andrewaylett 4y ago
Kids of today, etc. AT power supplies didn't have any mechanism for the system to tell the power supply it wasn't needed any more. So when you shut down the computer, it would wind up at a screen with a message approximating "it is now safe to switch off your computer", at which point the system would halt. ATX power supplies added the ability for the OS to trigger an actual power off. But that's a different end-state to halting, and if you halt the system then it stays on. You may wonder why anyone would want to halt when power off is an option, and to be honest I'm not entirely sure -- possibly because you have a hardware watchdog which will trigger a reboot of a halted machine but not of a powered off machine?

CGamesPlay 4y ago

A much more challenging task is writing a systemd unit that starts gracefully before shutdown. I wanted to write a unit that could issue an API call to delete the instance rather than doing a normal power off. Putting it at a reasonable place in the sequence took a lot of trial and error! The trick is actually to have the unit be started at some point in normal boot up (e.g. “armed”) and then do the actual task when the unit is stopped. Here’s the unit I ended up with: https://github.com/CGamesPlay/infra/blob/master/private-server/ansible/self-destruct.service https://github.com/CGamesPlay/infra/blob/master/private-serv...

dghubble 4y ago
That's what this post builds to solving at the end and in the next post - having a unit deletes the instance from a cluster before shutdown
hinkley 4y ago
I tend to push the metaphor of software being meant to be read and only incidentally to be run by a computer as far as I can. We are telling a story to future us or our successors. Stories have rules and it's jarring when you violate them. There's an idea in software that's a bit like the corollary of Chekhov's gun. Chekhov's gun is about not presaging jarring story elements that will never come to pass. But it's nearly as jarring to leave important story arcs as complete surprises until the end. Producing the gun moments before the curtain goes down would be quite a WTF. We didn't know that was a possibility. That's a niche that some people occupy, but it is a niche. Introducing things early fights with Locality of Reference, but when we're talking about things of deep, dramatic importance (like an attempted murder, or a reaper process) it's important to introduce that "character" early in the story so that people know that it exists. Failing to do so is a form of deus ex machina and we only appreciate that in very small doses. So framed that way, I don't see a problem with having to start a killswitch while you're spinning everything up. It's there, people can see it, and know to ask questions about it.
- slivanes 4y ago

SoftTalker 4y ago

I fundamentally disagree with the idea that software should require or even expect a graceful shutdown. You can never stop the user from yanking the power cord out of the socket, which is what they will do if you force a bunch of housekeeping to happen before shutdown. You have to deal with crash/power failure recovery anyway. So do your housekeeping on startup. Shutdown should be a quick and simple termination.

nerdponx 4y ago
Hope for the best, plan for the worst, right? Here's a contrived analogy: modern airplanes are designed to stay in the air even if an engine burns out, but we would still rather fly with both engines at full power whenever possible.
empthought 4y ago
This is a weird take; most systems in data centers don’t have people walking from rack to rack yanking power cords, and most consumer systems don’t even have a power cord to yank.
- bravetraveler 4y ago
  While I agree it's a bit of a weird take, for example -- there may be performance tradeoffs made in any given workload to make the disk consistent, inconsistently The 'most' there is doing some effort It is actually quite a common practice for those being audited for disaster recovery to do exactly that -- yank cables. More realistically, flip some switches We do it once a year, set aside a region and time... then test our processes It serves a few purposes, most importantly -- are our services fault tolerant, and can we bring them back? I think it's reasonable to trap the signals and make a best effort basis, knowing that PID 1 (or the environment) will eventually have to SIGKILL you -- ready or not Just because we can't save all of the state doesn't mean we shouldn't try
  - empthought 4y ago
    Right, there are failure modes that have to be tested and accounted for, and one of them is the state being inconsistent after a shutdown. The previous poster seemed to advocate for not thinking of this as a failure mode at all but rather normal operation, which I just don’t see as true.

exikyut 4y ago

Meta: I can't reach this website! Chrome is giving me an instant NXDOMAIN error. Dig shows that $ dig psdn.io @1.1.1.1 ... ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 19283 ... ;; QUESTION SECTION: ;psdn.io. IN A so then I prefix "www." like is in the URL... $ dig www.psdn.io @1.1.1.1 ... ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 64024 ... ;; QUESTION SECTION: ;www.psdn.io. IN A ;; ANSWER SECTION: www.psdn.io. 300 IN CNAME poseidon-www.pages.dev. Okay, fine: $ dig poseidon-www.pages.dev @1.1.1.1 ... ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 55471 ... ;; QUESTION SECTION: ;poseidon-www.pages.dev. IN A ...wat?? (Where there's no ANSWER section, none was returned, just an AUTHORITY section) This is reproducible for me with 1.1.1.1, 8.8.8.8 and 9.9.9.9.

dghubble 4y ago
Hmm, sorry you're not seeing it. Its just a CNAME to Cloudflare Pages, nothing fancy dig www.psdn.io @1.1.1.1 ; <<>> DiG 9.16.33-RH <<>> www.psdn.io @1.1.1.1 ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 37362 ;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1 ;; QUESTION SECTION: ;www.psdn.io. IN A ;; ANSWER SECTION: www.psdn.io. 126 IN CNAME poseidon-www.pages.dev. poseidon-www.pages.dev. 126 IN A 172.66.45.44 poseidon-www.pages.dev. 126 IN A 172.66.46.212

SrslyJosh 4y ago

It shouldn't be this hard to stop a service gracefully. This is far, far more complicated than SysV init, where you just need to drop a script into /etc/init.d symlink it from the appropriate rc directories. (For shutdown/reboot, you'd create symlinks in rc5.d and rc6.d named KNNwhatever, where NN is an integer that specifies the order the script will be run in. The "K" stands for "kill".) Edit: Note that my example runlevels are for Solaris, other UNIX/Linux OSes will vary.

LukeShu 4y ago
> It shouldn't be this hard to stop a service gracefully. It's not. > you just need to drop a script into /etc/init.d symlink it from the appropriate rc directories. You just need to drop a unit file into /etc/systemd/system/ and symlink it from the appropriate /etc/systemd/system/${target}.wants/ directories. Don't tell me that "shutdown.target.wants" and "reboot.target.wants" are harder than "rc0.d" and "rc6.d". A lot of the article is about ordering of dependencies (don't stop a dependency until after the dependent has stopped). Don't tell me that adding `Before=` and `After=` lines in the unit file is harder than having to remember all of the dependencies and manually figure out the correct "NN" for it all to work correctly. A lot of the article is about either having your daemon handle SIGTERM, or coming up with the appropriate `ExecStop=` command. The same command you'd be writing in your rc script (the "handle SIGTERM" stuff being for if your rc script simply says `kill $PID`). That is: The complex parts of the article are things that were complex with sysvinit too.
- ivan23178 4y ago
  There's also middle ground. OpenRC doesn't need those weird numbers while still being simple sysvinit/bash scripts-based system (it has a simple dependency system too). I'm not completely sure about `shutdown` level scripts having access to network though (too lazy to check), but it's still worth mentioning here.
  - LukeShu 4y ago
    OpenRC's great! But it doesn't detract from the point that systemd isn't nearly as complicated as some people act like it is. PS: Who's still using OpenRC with heavy ol' sysvinit instead of the lighter openrc-init?

rfmoz 4y ago

The macOS init manager, LaunchD, doesn't offer an easy way to execute an script at shutdown. By definition, it sends a SIGTERM signal to all of the daemons that it started. But as the script isn't started before, and doesn't keep a running PID, you don't have a clean way to do it. I don't understand why they only implemented the SIGTERM call without any alternative.

MarkusWandel 4y ago

Quoting from the article: TimeoutStopSec=0 That's cost me more than one hard power button powerdon (on a desktop machine with the system partition on SSD - unnerving). One of the innumerable things that systemd stops on shutdown gets stuck - permanently - and the machine goes into a state out of which, to my knowledge, is only a powerdown or reset. I ended up searching for the above and replacing them with a reasonable timeout (several minutes).

dghubble 4y ago
That's fair, that example would be better with a modest timeout (e.g. if podman had a bug that caused it to hang), without taking away from the main points. Updated. Someone asked about the opposite on Twitter: https://twitter.com/DannoHung/status/1585350836074446869 https://twitter.com/DannoHung/status/1585350836074446869