6 ms·
What practical problems do you run into with systemd?
All the compliants I see tend to be philisophical criticism of systemd being "not unixy" or "monolithic".
What practical problems do you run into with systemd?
All the compliants I see tend to be philisophical criticism of systemd being "not unixy" or "monolithic".
But there's a reason it's being adopted: it does it's job well. It's a pleasure being able to manage timers, socket activations, sandboxing, and resource slices, all of which suck to configure on script based init systems.
People complain in website comment sections how "bloated" systemd is, while typing into reddit webpage that loads megabytes of JS crap.
Meanwhile a default systemd build with libraries is about 1.8MB. That's peanuts.
Systemd is leaps and bounds in front of other init systems, with robust tooling and documentation, and despite misconceptions it actually quite modular, with almost all features gated with options. It gives a consistent interface for linux across distributions, and provides a familar predictible tools for administators.
Ohh... I have sooooo many issues with systemd. The core systemd is fine, and the ideas behind it are sound.
But it lacks any consistency. It's not a cohesive project with a vision, it's a collection of tools without any overarching idea. This is reflected in its documentation, it's an OK reference manual, but go on and try to build a full picture of system startup.
To give you concrete examples:
1. Systemd has mount units, that you would expect to behave like regular units but for mounts. Except that they don't. You can specify the service retry/restart policy for regular units, including start/stop timeouts, but not for mounts.
2. Except that you can, but only if you use the /etc/fstab compat.
3. Except that you can not, if systemd thinks that your mounts are "local". How does it determine if mounts are local? By checking its mount device.
4. Systemd has separate behaviors for network and local filesystems.
5. One fun example of above, there's a unit that fires up after each system update. It inserts itself _before_ the network startup. Except that in my case, the /dev/sda is actually an iSCSI device and so it's remote. So systemd deadlocks, but only after a system update. FUN!!!
6. How does systemd recognize network filesystems? Why, it has a pre-configured list of them: https://github.com/systemd/systemd/blob/4c6afaab193fcdcb1f5adfe29cd9e0d27e67c55c/src/basic/mountpoint-util.c#L235 https://github.com/systemd/systemd/blob/4c6afaab193fcdcb1f5a... Yes, you read it correctly. A low-level mount code has special case for sshfs, that it detects by string-matching.
7. But you can override it, right? Nope. This list is complete and authoritative. Nobody would ever need fuse.s3fs . And if you do, see figure 1.
I can go on for a looooong time.
I wrote up some issues with service reliability here https://github.com/andrewbaxter/puteron/?tab=readme-ov-file#origin-story https://github.com/andrewbaxter/puteron/?tab=readme-ov-file#...
Design-wise, I think having users modify service on/off state *and* systemd itself modify those states is a terrible design, which leads to stuff turning back on when you turn it off, or things turning off despite you wanting them on, etc. (also mentioned higher up)
FWIW after making puteron I found dinit https://github.com/davmac314/dinit https://github.com/davmac314/dinit which has a very similar design, so presumably they hit similar issues.
> But there's a reason it's being adopted: it does it's job well
My problem with systemd is that it's taking over more and more and locking in. It is encouraging developers to have a hard dependency on it, and making it harder to have an alternative.
My problem is not philosophical with "it's a monolith, it's not unixy". My problem is "it's on the way to lock me in".
We like to complain about lock-in across the board. I don't see why it would be different with systemd.
This is one example of a more general complaint about systemd and related projects: they force policy, rather than simply providing mechanisms.
I recently did a deep dive on my laptop because I was curious about an oddity - the /sys file to change my screen backlight (aside, why /sys and not /dev anyway?) was writable only by root - yet any desktop shell running as my user had no problem reacting to brightness hotkeys. I wondered, how did this privilege escalation work? Where was the policy, and what property of my user account granted it the right to do this?
It turns out the answer is that the desktop shells are firing off a dbus request to org.freedesktop.login1, which is caught by systemd-logind - or elogind in my case, since I do not care for systemd. A login manager seemed an odd place for screen brightness privilege escalation, but hey if it works whatever - it seemed like logind functioned as a sort of miscellaneous grab bag of vaguely console-related stuff. Generally speaking, it consults polkit rules to determine whether a user is allowed to do a thing.
Not screen brightness, though. No polkit rules. Nothing in pkaction. logind was unilaterally consenting to change the brightness on my behalf. And on what grounds? It wasn't documented anywhere so I had to check the source code, where I found a slew of hardcoded criteria that mostly revolve around physical presence at the machine. Want to change screen brightness over ssh? Oh but why would you ever want to do that? Hope you have root access, you weirdo.
I removed elogind. A few odds and ends broke. But nobody tells me what to do with my machine.
OK, think it through...
How do we determine that a specific instance of a filesystem mount is "remote", or even requires a "network"? Consider that the network endpoint might be localhost, a netlink/unix/other socket, or, say, an IP address of the virtual host (practically guaranteed to be there and not truly "remote").
systemd has .mount units which are way more configurable than /etc/fstab lines, so they'd let you, as the administrator, describe the network dependency for that specific instance.
But what if all we have is the filesystem type (e.g. if someone used mount or /etc/fstab)?
Linux doesn't tell us that the filesystem type is a network filesystem. Linux doesn't tell us that the specific mount request for that filesystem type will depend on the "network". Linux doesn't tell us that the specific mount request for that filesystem type will require true network connectivity beyond the machine itself.
So, before/without investing in a long-winded and potentially controversial improvement to Linux, we're stuck with heuristics. And systemd's chosen heuristic is pretty reasonable - match against a list of filesystem types that probably require network connectivity.
If you think that's stupid, how would you solve it?
> How do we determine that a specific instance of a filesystem mount is "remote", or even requires a "network"?
The '_netdev' option works a treat on sane systems. From mount(8):
_netdev
The filesystem resides on a device that requires network access
(used to prevent the system from attempting to mount these
filesystems until the network has been enabled on the system).
It should work on SystemD and is documented to in systemd.mount
Mount units referring to local and network file systems are distinguished by their file system type specification. In some cases this is not sufficient (for example network block device based mounts, such as iSCSI), in which case _netdev may be added to the mount option string of the unit, which forces systemd to consider the mount unit a network mount.
but -surprise surprise- it doesn't reliably work as documented because SystemD is full of accidental complexity.
Sure, and systemd would translate that directly into a dependency on network startup, which is precisely equivalent to the approach I mentioned that depends on operator knowledge. It's configuration, not "just works" inference.
> Sure, and systemd would translate that directly into a dependency on network startup...
You'd think so, but the Github Issue linked by GP shows that the machinery is unreliable:
In practice, adding `_netdev` does not always force systemd to [consider the mount unit a network mount], in some instances even showing *both* local and remote ordering. ... This can ultimately result in dependency cycles during shutdown which should not have been there - and were not there - when the units were first loaded.
> ...not "just works" inference.
Given that SystemD can't reliably handle explicit use of _netdev, I'd say it has no hope of reliably doing any sort of "just works" inference.
> How do we determine that a specific instance of a filesystem mount is "remote", or even requires a "network"?
Like systemd authors do! Hard-code the list of them in the kernel, including support for fuse and sshfs. Everything else is pure blasphemy and should be avoided.
Me? I'd have an explicit setting in the mount unit file, with defaults inferred from the device type. I would also make sure to not just randomly add landmines, like systemd-update-done.service. It has an unusual dependency requirements, it runs before the network filesystems but after the local filesystems.
I bet you didn't know about it? It's a service that runs _once_ after a system update. So the effect is that your system _sometimes_ fails to boot.
> systemd has .mount units which are way more configurable than /etc/fstab lines
It's literally the inverse. As in, /etc/fstab has _more_ options than native mount units. No, I'm not joking.
Look at this man page: https://www.freedesktop.org/software/systemd/man/latest/systemd.mount.html https://www.freedesktop.org/software/systemd/man/latest/syst... The options with "x-systemd." prefix are available for fstab.
Look for the string: "Note that this option can only be used in /etc/fstab, and will be ignored when part of the Options= setting in a unit file."
Sounds like your admin, distro, or the systemd team could pay some attention to systemd-update-done.service
The "can only be used in /etc/fstab" systemd settings are essentially workarounds to do those things via fstab (and workaround fstab related issues) rather than depend on other systemd facilities (c.f. systemd-gpt-auto-generator). From a "what can you do in /etc/fstab without knowing systemd is working behind the scenes" point of view, then yes, systemd units are vastly more configurable.
This service is the standard part of systemd. And my distro is a bog-standard Fedora, with only iSCSI as a complication.
Are you surprised that such a service exists? I certainly was. And doubly so because it has unusual dependency requirements that can easily lead to deadlocks. And yes, this is known, there are open issues, and they are ignored.
> From a "what can you do in /etc/fstab without knowing systemd is working behind the scenes" point of view, then yes, systemd units are vastly more configurable.
No, they are not. In my case, I had to use fstab to be able to specify a retry policy for mount units (SMB shares) because it's intentionally not exposed.
And yes, there's a bug: https://github.com/systemd/systemd/issues/4468 https://github.com/systemd/systemd/issues/4468 with the expected GTFO resolution: https://github.com/systemd/systemd/issues/4468#issuecomment-1180918535 https://github.com/systemd/systemd/issues/4468#issuecomment-...
So there's literally functionality that has been requested by people and it's available only through fstab.
I love systemd, but you've hit on one of my biggest complaints. The mounting promises a cohesive system and instead gives you a completely broken mess, with mounts being split across .mount unit files, fstab, and worst of all, .service unit files. It's a totally incoherent mess, and that's only _after_ you figure out why nothing is working right, and build a complex mental model of every single feature that does or doesn't work in which scenario. Knowledge you only gain after screaming and tearing your hair out for a weekend. Your reward? A totally incoherent, inconsistend mess.
I hate mounts in systemd.
> Systemd is a standardization that is appealing to developers. They want to adopt it because it makes their life easier. It is just nice to know that all the tools you need for a system are there and work together. Pluggability is hard to maintain and is only done if there is no standardization.
That's the official story, but like most official stories, it doesn't really hold up to scrutiny.
I built an entire system from scratch with over 1,500 packages installed. Everything under the sun. Works just fine with sysvinit. Completely seamless.
If KDE/Gnome can't figure out how to fit in with the overall system design the same way thousands of other packages somehow manage to do, then their services are no longer required. Good riddance to their bloated asses. I prefer to invest my CPU cycles in better software.
Init scripts for services and such are properly handled by the distro maintainer (packager), not the developer, although it's always nice to see examples he's provided to help guide the development of my preferred version.
I am honestly happy for you that you made your system the way you want it. That is a good thing and please keep doing what you are doing.
This is not relevant to the average user. The average PC user doesn't use Linux and the average Linux user uses an off the shelve distro. For these distros it is very attractive to have a bunch of core services ready that work together because they are released as one. It can be done but why the hassle? What is the upside for the maintainer apart from maybe the moral high ground?
Software projects can also benefit from standardization. They can concentrate on writing functionality instead of maintaining abstraction layers. And I believe the more mainstream distros choose the SystemD stack the more it becomes the default or at least the initial implementation for their software.
We also have to keep in mind that this kind of standardization is nothing new. Pretty much every distro depends on the GNU coreutils. Maybe not on the binaries themselves but at least on their API. That is not very different from SystemD. We have a POSIX standard.
Final word regarding sysvinit: I worked with sysvinit, upstart and systemd and having an opinionated config format for services is so much better, in my opinion. Not having to read a whole shell script to know how a service works is such an improvement and the easy overrides for units (for example distro packaged ones) is amazing.
Note: In my post I counted distro maintainers as developers.
You lost me when you started talking about the average user. I don't care about that guy or his desires. At all.
I miss the days when computing was about the above average guy--not the simpleton who needs his hand held, so everything has to be dumbed down to the lowest level to suit him.
Heard it all before, and I'm not interested in anything systemd has to "offer." Especially all the bugs and security issues.
This distro isn't for you. That's OK. systemd, and wayland, etc that some are so excited about isn't for me or a number of others, and it will never be. We are going our separate way. Just look at all the comments below. Lots of upvotes too.
It's not a lock-in as much as making a much better product.
For example, I never liked the idea of having my programs to manually daemonize, manage logs, set up permissions and all that boring, but security-critical stuff. And with systemd, I don't have to! Program reads from stdin/stdout, maybe gets socket from socket activation, and systemd does the rest.
Is it lock-in? Only because other system suck. Like, seriously, what stopped xinetd from having rudimentary volatile "on/off" control, so I could disable misbehaving service? Or why is start-stop-daemon so _stupid_, discarding all startup error messages? And don't get me started on all the different init file dialects for each system.
Maybe if the sysvinit programmers actually cared about providing nicer services to app developers, we would never end up with systemd.
The problem with the word "sysvinit" here is it's sort of a red herring. BSD init is better, in my opinion. I don't like managing all those symlinks. Plus, sysvinit is an old 90s application and its code does have some cruft built up over the years that could be removed and simplified. I'm devising a new init for my system that's much simpler than sysvinit and much closer to BSD.
"BSD init", "much simpler"... So does this mean you still expect applications to manage their own logs, daemonization and security setup themselves?
If yes, that's yet another init system not made for application writers.
Manage their own logs, daemonization, and security? The humanity! How will they ever manage all of that?
Come on man. It's been done for decades.
It doesn't take a giant bloated infrastructure to manage most people's needs, which are quite basic in most cases.
.. and that opinion is a great explanation of why systemd won.
Turns out, a lot of people are not happy with "Come on man. It's been done for decades." attitude, and they wanted something new and much better. And so when something new came up, they jumped on it with both feet.
It's instructive to read Debian CTTE discussion on init systems (btw I think it's best tech drama of 2013, highly recommend) - a lot of people dismissed the sysvinit early on because it had no features (example [0]), which means the choices were either upstart and systemd. And between two of those, systemd is a clear win.
Read the thread and look at how many highly technical people with no relation to Fedora or Poettering is ready to choose _anything else_ just to get away from "it's been done for decades".
[0] https://lists.debian.org/debian-ctte/2013/12/msg00234.html https://lists.debian.org/debian-ctte/2013/12/msg00234.html
> .. and that opinion is a great explanation of why systemd won.
Completely wrong and ignorant.
> Turns out, a lot of people are not happy with "Come on man. It's been done for decades." attitude, and they wanted something new and much better.
And then they got systemd. LOL
Like Dr. Phil said, "How's that workin out for ya?" LOL
> And so when something new came up, they jumped on it with both feet.
You just did what your type always does: whatever you're told.
> It's instructive to read Debian CTTE discussion on init systems
No, it really isn't. lol
> the choices were either upstart and systemd. And between two of those, systemd is a clear win.
Well, it's too bad none of the other good options were considered, isn't it? When your only "options" are a giant douche or a turd sandwich, the outcome can't possibly be good.
See U.S. presidential elections for one of the best examples of this dynamic. Two complete fucking losers are presented every time, and 40% of the population are mesmerized by the spectacle and think there can be no other possible options at all. That's you.
The fact is, many of you noobs don't even know how to write a shell script, yet somehow feel qualified to comment on this subject, as if your opinion is worth anything at all.
How many daemons have you personally written? Hmm? Do you even know how to write any C at all? Daemonizing a process isn't rocket science. It's a double fork. So simple even you could do it, I bet.
The problem is you're too technically ignorant to understand that none of your "technical" arguments hold any water at all. It's just you repeating the bullshit you were told, as usual.
Every Big Lie being told relies on Useful Idiots like you to help support it.
Logging is not difficult. Double forking is not difficult. If you find any of that to be a challenge, you're not qualified to write a daemon. If you can't successfully set up and run something like runit or any of the many other good sysvinit alternatives, you're not qualified to administer a Linux system. Period.
You make all these appeals to authority ("highly technical people" saying this or that) like that means something. You forget that you're speaking to the guy who built his own operating system. I don't need any guidance from "highly technical people" on what init system to pick. Apparently you're the type who does.
That's what your entire argument basically boils down to--one giant appeal to authority.
If everyone else was jumping off a bridge, would you do it too? Of course you would, without a moment's hesitation. Because you're a God damned lemming.
Now get off my lawn.