7 ms·

Linux fu: getting started with systemd

92 points by drpixie 2y ago

drpixie 2y ago
> Like many long-time Unix users, I’m not a big fan of systemd. Then again, I’m also waiting for the whole “windows, icon, mouse, pointer” fad to die down. I like a writer with perspective :)
- askvictor 2y ago
  I'm not against systemd, it's just that the muscle memory takes a _long_ time to change, particularly for things that you don't do frequently.
  - kingspact 2y ago
    Nobody knows what the end game of systemd even is. Started out as an init replacement, now it's doing all kinds of stuff. I had to bail on Debian because systemd failed hard when it took over user directories.
    - tristan957 2y ago
      systemd is a suite of tools. systemd(1) is a service manager. You have a misunderstanding of systemd.
lakomen 2y ago
Oh boy. I'm currently battling with Kubernetes, and I mean it. Compared to k8s systemd is simple and easy. I can wholeheartedly say I hate k8s and its guts. Everything is so overly complicated. A bazillion configurations for every single little detail. And she's a touchy little princess. Hard to find help, expensive courses and a not so great documentation site, that kind of explains the components, but then again really doesn't in detail, also no complete configuration reference. And no matter which way to setup you choose, something is always wrong. It's a tool to drive you to expensive public cloud offerings. For the small price of only ~$2600 per month you can have your 5 node k8s cluster on GCP, cheap cheap. Burn money money burn money money burn. Managing and maintaining k8s is a full-time job. In comparison systemd is well documented and you don't really need to ask people for help. You can easily use the shell, you don't have to battle with wrong nginx configurations that were autogenerated, because you wrote them and you know what you're doing. Fleet was cool, but Redhat bought CoreOS and killed fleet, can't have a simple effective system, it has to be complex and enterprise so you can sell services and tutelage. Fucking IT people.
- tonoto 2y ago
  I honestly fail to see in what aspect Kubernetes is poorly documented? It is complex yes, but just about any aspect I've come by is documented. I think that one reason that the documentation at kubernetes.io is kept in a rather short format may be to avoid it to become overwhelming.
- haolez 2y ago
  Fleet's code is still around if the pain of Kubernetes outweighs the benefits. The problem, in this case, is that systemd is not exactly minimalist, and Fleet built on top of it. I've used it in the past and it felt complex as well, especially when debugging problems.
- okasaki 2y ago
  I wrote my own shitty deployment/monitoring GUI that uses just systemd and ssh and a single yaml file: https://github.com/dvolk/sc https://github.com/dvolk/sc
- jeroenhd 2y ago
  Kubernetes is great for what it's designed to do. Many, if not most, companies don't need the things Kubernetes is designed for, though. It's interesting tech and I can see why people are drawn to it, but I feel like some people pick it more because they want to use Kubernetes rather than it solving a real problem a company or organisation is facing.
  - ChuckMcM 2y ago
    This. As Borg it is a thing to behold but when people think they know 'scale' but don't really, well then it seems very over engineered.
  - juujian 2y ago
    To this day, I cannot tell what Kubernetes is designed to do. I hear about it constantly from this website, and based on the conversations you would think it was designed to do anything and everything, and all at the same time.
psacawa 2y ago
A relevant bit of self-promotion maybe: I created a small language server for systemd unit files which may help those having to integrate services with it. https://github.com/psacawa/systemd-language-server https://github.com/psacawa/systemd-language-server
- 1oooqooq 2y ago
  the service syntax is the least of your concerns. the windows service manager like UI of systemctl exploratory mode is one thing to tackle. the lack of visibility on dependencies that are not obviously direct. or even relevant (like ssh depending on key generation at first boot, which depends on time, which depends on time sync, which depends on network target... which causes ssh to not get started, without any logs, if you don't have network... despite it not being the first boot... :mindblown) i won't bother with whole list because 1. there's no alternative 2. "I'm just another systemd denier" like i wouldn't have to be using it from early on to accumulate so much grievances
  - alexeldeib 2y ago
    have you seen systemd-analyze?
    - 1oooqooq 2y ago
      yes and it also fails to catch that. and when reported years ago it was dismissed as "debian is doing weird shit. nobody sane depends things like so"
    - usr1106 2y ago
      systemd-analyze dot is nice in theory. In practice the sheer amount of dependencies makes the results difficult to use or understand. That it's a mixture of before and requires (probably also wants, don't rememeber) does not make it easier. Edit: Newer version have options to separate ordering from requirements. I don't think that existed on the system where I last used it.
  - jamespo 2y ago
akira2501 2y ago
> Like it or not, systemd is here and probably here to stay for the foreseeable future. Which is why I love the extreme compatibility and openness of Linux. systemd is free to stay and I'm free to just never use it. This fact only seems to bother one of the groups.
- PlutoIsAPlanet 2y ago
  > This fact only seems to bother one of the groups. Because people think other people who they're not paying should not be allowed to rely on systemd for things and support other service managers.
- msla 2y ago
  Yep, the anti-systemd people certainly got angry.
constantcrying 2y ago
>Here’s a fairly simple service unit: >You can probably puzzle most of that out. That's hilarious. No, you can not. Why is there a "WantedBy" for multi-user.target when it is started "After" network.target and auditd.service? If you understand systemd the answer is obvious, but if you don't this should confuse you. >It can replace inetd, syslog, and many other traditional services. On Ubuntu it replaced the fstab. >This is a benefit or a drawback, depending on your point of view. I don't know from which perspective it is good to have one program extend itself into random unrelated areas and absorb their functionality into itself. Certainly no university course or anyone I ever worked with had that perspective. Usually people talked about defining and limiting scope and having a clear vision of what your software should do. I think the "Unix Philosophy" debate is largely silly, mostly because it misses the point. Regardless of what some people in the 80s thought about UNIX system programming, it is bad software engineering to not have a defined scope for your software and let it sprawl endlessly. Whether software should do one thing only is neither here nor there, but it certainly shouldn't do a couple dozen unrelated things while replacing perfectly functional existing system software.
- 1oooqooq 2y ago
  systemd is nothing but red hat market capture. you cannot sell Linux certification if you must teach people to code, as was the case with other inits. nobody write code to get a mcse. there's a reason they were called mouse clicking solutions expert. redhat wants that too. faster and cheaper certification. and yes it's a stupid plan. but everything noteworthy on Linux was contributed by companies. either when nobody was looking what a programmer was doing, or when some old code get donated, or thanks to stupid plans like this. so we took it. it's not worse than before, and hopefully mr systemd will get bored counting money at Microsoft now and let the project evolve to something sane.
  - PlutoIsAPlanet 2y ago
    So only people who know how to code should be allowed to setup and configure system services? What an awful way to gate keep computing.
shmerl 2y ago
I never got complaints about systemd. A few times I had to deal with writing custom unit files, it wasn't hard to figure out.
- remram 2y ago
  That's a low bar. I never had any issue doing this with any other tool (supervisord, inetd scripts, docker-compose, Kubernetes... running a command isn't hard)
brokenmachine 2y ago
>Editor’s note: And this logging functionality was exactly what was abused in last week’s insane liblzma / ssh backdoor. How was the logging functionality of systemd abused by the xz backdoor?
- zekrioca 2y ago
  I guess systemd uses liblzma to automatically compress logs after a configurable timeframe (e.g., daily).
  - brokenmachine 2y ago
    But wasn't it a backdoor, ie the attacker has to connect via sshd with a specific pubkey to make it run bad commands? As I understand it, just the mere fact liblzma runs on a schedule wouldn't cause it to do anything nefarious.
    - 1oooqooq 2y ago
      so you don't understand it... read on https://www.openwall.com/lists/oss-security/2024/03/29/4 https://www.openwall.com/lists/oss-security/2024/03/29/4 look for the part: "These functions get resolved during startup"
      - brokenmachine 2y ago
        Aah, so sshd gets backdoored because systemd calls xz.
        1oooqooq 2y ago
        no. anything in the dependency tree could call that lib. systemd is the first. systemd also provides libs to write things the exploit will use. but the important part you missed again is that the exploit is executed on symbols loading. it just choose to do just of the work when checking for a key. but after simply starting the code it's all lost already. nothing needs to call any code on libzma, just linking against it is enough to run the exploit.
Dowwie 2y ago
I've become so much more capable with systemd thanks to ChatGPT4.

92 points by drpixie 2y ago

> Like many long-time Unix users, I’m not a big fan of systemd. Then again, I’m also waiting for the whole “windows, icon, mouse, pointer” fad to die down. I like a writer with perspective :)

askvictor 2y ago
I'm not against systemd, it's just that the muscle memory takes a _long_ time to change, particularly for things that you don't do frequently.
- kingspact 2y ago
  Nobody knows what the end game of systemd even is. Started out as an init replacement, now it's doing all kinds of stuff. I had to bail on Debian because systemd failed hard when it took over user directories.
  - tristan957 2y ago
    systemd is a suite of tools. systemd(1) is a service manager. You have a misunderstanding of systemd.

lakomen 2y ago

Oh boy. I'm currently battling with Kubernetes, and I mean it. Compared to k8s systemd is simple and easy. I can wholeheartedly say I hate k8s and its guts. Everything is so overly complicated. A bazillion configurations for every single little detail. And she's a touchy little princess. Hard to find help, expensive courses and a not so great documentation site, that kind of explains the components, but then again really doesn't in detail, also no complete configuration reference. And no matter which way to setup you choose, something is always wrong. It's a tool to drive you to expensive public cloud offerings. For the small price of only ~$2600 per month you can have your 5 node k8s cluster on GCP, cheap cheap. Burn money money burn money money burn. Managing and maintaining k8s is a full-time job. In comparison systemd is well documented and you don't really need to ask people for help. You can easily use the shell, you don't have to battle with wrong nginx configurations that were autogenerated, because you wrote them and you know what you're doing. Fleet was cool, but Redhat bought CoreOS and killed fleet, can't have a simple effective system, it has to be complex and enterprise so you can sell services and tutelage. Fucking IT people.

tonoto 2y ago
I honestly fail to see in what aspect Kubernetes is poorly documented? It is complex yes, but just about any aspect I've come by is documented. I think that one reason that the documentation at kubernetes.io is kept in a rather short format may be to avoid it to become overwhelming.
haolez 2y ago
Fleet's code is still around if the pain of Kubernetes outweighs the benefits. The problem, in this case, is that systemd is not exactly minimalist, and Fleet built on top of it. I've used it in the past and it felt complex as well, especially when debugging problems.
okasaki 2y ago
I wrote my own shitty deployment/monitoring GUI that uses just systemd and ssh and a single yaml file: https://github.com/dvolk/sc https://github.com/dvolk/sc
jeroenhd 2y ago
Kubernetes is great for what it's designed to do. Many, if not most, companies don't need the things Kubernetes is designed for, though. It's interesting tech and I can see why people are drawn to it, but I feel like some people pick it more because they want to use Kubernetes rather than it solving a real problem a company or organisation is facing.
- ChuckMcM 2y ago
  This. As Borg it is a thing to behold but when people think they know 'scale' but don't really, well then it seems very over engineered.
- juujian 2y ago
  To this day, I cannot tell what Kubernetes is designed to do. I hear about it constantly from this website, and based on the conversations you would think it was designed to do anything and everything, and all at the same time.

psacawa 2y ago

A relevant bit of self-promotion maybe: I created a small language server for systemd unit files which may help those having to integrate services with it. https://github.com/psacawa/systemd-language-server https://github.com/psacawa/systemd-language-server

1oooqooq 2y ago
the service syntax is the least of your concerns. the windows service manager like UI of systemctl exploratory mode is one thing to tackle. the lack of visibility on dependencies that are not obviously direct. or even relevant (like ssh depending on key generation at first boot, which depends on time, which depends on time sync, which depends on network target... which causes ssh to not get started, without any logs, if you don't have network... despite it not being the first boot... :mindblown) i won't bother with whole list because 1. there's no alternative 2. "I'm just another systemd denier" like i wouldn't have to be using it from early on to accumulate so much grievances
- alexeldeib 2y ago
  have you seen systemd-analyze?
  - 1oooqooq 2y ago
    yes and it also fails to catch that. and when reported years ago it was dismissed as "debian is doing weird shit. nobody sane depends things like so"
  - usr1106 2y ago
    systemd-analyze dot is nice in theory. In practice the sheer amount of dependencies makes the results difficult to use or understand. That it's a mixture of before and requires (probably also wants, don't rememeber) does not make it easier. Edit: Newer version have options to separate ordering from requirements. I don't think that existed on the system where I last used it.
- jamespo 2y ago

akira2501 2y ago

> Like it or not, systemd is here and probably here to stay for the foreseeable future. Which is why I love the extreme compatibility and openness of Linux. systemd is free to stay and I'm free to just never use it. This fact only seems to bother one of the groups.

PlutoIsAPlanet 2y ago
> This fact only seems to bother one of the groups. Because people think other people who they're not paying should not be allowed to rely on systemd for things and support other service managers.
msla 2y ago
Yep, the anti-systemd people certainly got angry.

constantcrying 2y ago

>Here’s a fairly simple service unit: >You can probably puzzle most of that out. That's hilarious. No, you can not. Why is there a "WantedBy" for multi-user.target when it is started "After" network.target and auditd.service? If you understand systemd the answer is obvious, but if you don't this should confuse you. >It can replace inetd, syslog, and many other traditional services. On Ubuntu it replaced the fstab. >This is a benefit or a drawback, depending on your point of view. I don't know from which perspective it is good to have one program extend itself into random unrelated areas and absorb their functionality into itself. Certainly no university course or anyone I ever worked with had that perspective. Usually people talked about defining and limiting scope and having a clear vision of what your software should do. I think the "Unix Philosophy" debate is largely silly, mostly because it misses the point. Regardless of what some people in the 80s thought about UNIX system programming, it is bad software engineering to not have a defined scope for your software and let it sprawl endlessly. Whether software should do one thing only is neither here nor there, but it certainly shouldn't do a couple dozen unrelated things while replacing perfectly functional existing system software.

1oooqooq 2y ago
systemd is nothing but red hat market capture. you cannot sell Linux certification if you must teach people to code, as was the case with other inits. nobody write code to get a mcse. there's a reason they were called mouse clicking solutions expert. redhat wants that too. faster and cheaper certification. and yes it's a stupid plan. but everything noteworthy on Linux was contributed by companies. either when nobody was looking what a programmer was doing, or when some old code get donated, or thanks to stupid plans like this. so we took it. it's not worse than before, and hopefully mr systemd will get bored counting money at Microsoft now and let the project evolve to something sane.
- PlutoIsAPlanet 2y ago
  So only people who know how to code should be allowed to setup and configure system services? What an awful way to gate keep computing.

shmerl 2y ago

I never got complaints about systemd. A few times I had to deal with writing custom unit files, it wasn't hard to figure out.

remram 2y ago
That's a low bar. I never had any issue doing this with any other tool (supervisord, inetd scripts, docker-compose, Kubernetes... running a command isn't hard)

brokenmachine 2y ago

>Editor’s note: And this logging functionality was exactly what was abused in last week’s insane liblzma / ssh backdoor. How was the logging functionality of systemd abused by the xz backdoor?

zekrioca 2y ago
I guess systemd uses liblzma to automatically compress logs after a configurable timeframe (e.g., daily).
- brokenmachine 2y ago
  But wasn't it a backdoor, ie the attacker has to connect via sshd with a specific pubkey to make it run bad commands? As I understand it, just the mere fact liblzma runs on a schedule wouldn't cause it to do anything nefarious.
  - 1oooqooq 2y ago
    so you don't understand it... read on https://www.openwall.com/lists/oss-security/2024/03/29/4 https://www.openwall.com/lists/oss-security/2024/03/29/4 look for the part: "These functions get resolved during startup"
    - brokenmachine 2y ago
      Aah, so sshd gets backdoored because systemd calls xz.
      - 1oooqooq 2y ago
        no. anything in the dependency tree could call that lib. systemd is the first. systemd also provides libs to write things the exploit will use. but the important part you missed again is that the exploit is executed on symbols loading. it just choose to do just of the work when checking for a key. but after simply starting the code it's all lost already. nothing needs to call any code on libzma, just linking against it is enough to run the exploit.

Dowwie 2y ago

I've become so much more capable with systemd thanks to ChatGPT4.