9 ms·
The order of files in /etc/ssh/sshd_config.d/ matters
- wruza 1y agoThat's why I erase sshd_config and put what I really meant there. You may say "but isn't it better to patch it properly?". It is not. Yet another vps hoster –> yet another /etc/ssh directory template that may have all sorts of access issues in it. Better to replace it and make it do exactly what you have planned.
- timewizard 1y agoI've never liked the directory.d/* infrastructure. In so many cases, even with a properly configured sshd_config, the resulting configuration file is not so large that it benefits from being split up. You have to deal with ordering issues, symlink management in some cases, and unless the "namespace" of sorting number prefixes is strictly defined, it's never something that's convenient or durable to "patch" new files into. The proliferation of 99_* files shows the anti-utility this actually provides. I much prefer configuration files with a basic "include" or "include directory" configuration item. Then I can scope and scale the configuration in ways that are useful to me and not some fragile distribution oriented mechanism. Aside from that with xz I don't think I want my configurations "patchable" in this way.
- sneak 1y agoThe conf.d isn’t because the config file is large. It’s because it’s easier to disable or enable something with an “echo blah > conf.d/10-addin.conf” or an “rm conf.d/50-blah.conf” than it is to do sed -i or grep blah || echo blah >>
- claudex 1y agoAlso, it allows different packages to handle the configuration and add their specific parameters.
- deleted 1y ago[deleted]
- shawnz 1y agoExactly: if your templating logic accidentally produces a syntax error, now you can't log in to SSH. There's much less chance of that scenario with include directories. This applies for infrastructure as code scenarios, changes made by third party packages, updates of ssh, manual one-off changes, etc.
- wruza 1y agoIf any logic produces a syntax error anywhere in the sshd_config include chain, ssh is broken now. And you will have templating logic in automatic configuration one way or another, at least for different dns/ips. I don't grep this argument at all. It feels like everyone's comparing to that "regular [bad] detergent" in this thread. A templating system will be as good and as error-prone to change and as modular etc as you make it, just like any program. It applies only to local patchers (like e.g. certbot nginx) and manual changes, but that's exactly out of scope of templating and configuration automation. So it can't be better, cause these two things are in XOR relationships. Edit to clarify: I don't disagree with foo.d approach in general. I just don't get the arguments that in automation setting it plays any positive role, when in fact you may step on a landmine by only writing your foo.d/00-my. Your DC might have put some crap into foo.d/{00,99}-cloud, so you have to erase and re-create the whole foo.d anyway. Or at least lock yourself into a specific cloud.
- shawnz 1y agoIt's still possible to break the config with a syntax error, but there are less kinds of syntax errors that are possible if you aren't writing into the middle of an existing block of syntax. For example, there's no chance that you unintentionally close an existing open block due to incorrect nesting of options or anything like that. Plus, if you are writing into the middle of an existing file, there's a chance you could corrupt other parts of the file besides the part you intended to write. For example, if you have an auto-generated section that you intend to update occasionally, you will need to make sure you only delete and recreate the auto-generated parts and don't touch any hand-written parts, which could involve complicated logic with sentinel comments, etc. Then you need to make sure that users who edit the file in future don't break your logic. In addition it's harder to test your automation code when you're writing into an existing file because there's more edge cases to deal with regarding the surrounding context, etc.
- loodish 1y agoThe .d directories make management via tools such as ansible much much easier. You don't have weird file patching going on with the potential to mess things up in super creative ways if someone has applied a hand edit. With .d directories you have a file, you drop in that file, you manage that file, if that file changes then you change it back.
- ecef9-8c0f-4374 1y agoI love that you can use validate: sshd -T -f %s To check if changes would break things.
- hnlmorg 1y agoConfig directories are there to solve change management problems like idempotency. If you have one big file then different tools, or even the same tool but different points of that tools life cycle, can result in old config not correctly removed, new config applied multiple times, or even a corrupt file entirely. This isnt an issue if you’re running a personal system which you hand edit those config files. But when you have fleets of servers, it becomes a big problem very quickly. With config directories, you then only need to track the lifecycle of files themselves rather than the content of those files. Which solves all of the above problems.
- wruza 1y agoI never managed a fleet. I mean I occasionally manage up to 30 instances, does that count? Either way, my notion about doing it properly is to have a set of scripts (ansible/terraform?) that rebuild the configuration from templates and rewrite, restart everything. Afaiu, there's no "let's turn that off by rm-ing and later turn it on again by cat<<EOF-ing", cause there's no state database that could track it, unless you rely on [ -e $path ], which feels not too straightforward for e.g. state monitoring. (I do the same basically, but without ansible. Instead I write a builder script and then paste its output into the root shell. Poor man's ansible, but I'm fine.) So as I understand it, these dirs are only really useful for manual management, not for fleets where you just re-apply your "provisioning", or what's the proper term, onto your instances, without temporary modifications. When you have a fleet, any state that is not in the "sources" becomes a big problem very quickly. And if you have "sources", there's no problem of turning section output on and off. For example when I need a change, I rebuild my scripts and re-paste them into corresponding instances. This ensures that if I lose any server to crash, hw failure, etc, all I have is to rent another one and right-click a script into its terminal. So I believe that gp has a point, or at least that I don't get the rationale that replies to gp suggest itt. Feels not that important for automatic management.
- hnlmorg 1y agoI’ve found the template system starts to have shortcomings if you need differences between different classes of systems within your fleet (eg VDIs for data scientists vs nodes in a ML pipeline pool) Yeah a good templating language will allow you to apply conditionals but honestly, it’s just generally easier to have different files inside a config directory than have to manage all these different use cases in a single monolithic template or multiple different templates in which you need to remember to keep shared components in sync. At the end of the day, we aren’t talking about solving problems that were impossible before but rather solving problems that were just annoying. It’s more a quality of life improvement than something that couldn’t be otherwise solved with enough time and effort.
- badgersnake 1y agoTotally, don’t use .d for ssh. The configuration is not that complicated. If it is, you’re doing it wrong.
- rlpb 1y agoThe .d directories are important on Debian and Ubuntu where packaging needs to provide different snippets based on the set of installed packages, the VM environment, other configuration inputs like through cloud-init and so forth, and update them during upgrades, but also (as per policy) preserve user customisations on anything in /etc. Since pretty much every file has different syntax, this is virtually impossible to do any other way.
- vbezhenar 1y agoThat's one of my issues with most Linux distros. 1. They add huge configuration files where 99% are commented out. 2. Sometimes they invent whole new systems of configuration management. For example debian with apache httpd does that. I don't need all of that. I just need simple 5-line configuration file. My wish: ship absolutely minimal (yet secure) configuration. Do not comment out anything. Ask user to read manuals instead. Ship your configuration management systems as a separate packages for those who need it. Keep it simple by default.
- eadmund 1y agoYeah, first-wins is definitely surprising. Off the top of my head, it feels like one would have to go out of one’s way to write a parser that does that (by storing an extra bit of state for each configuration item, and then checking it before setting the configuration item and toggling the state, rather than just applying the configuration item each time it is encountered). Is there a good reason for this design? I can’t think of one, again off the top of my head, but of course I could be missing something.
- o11c 1y agoIt probably makes a bit more sense when you think about the fact that SSH frequently does a "try to match this host against a list of configured host-patterns" operation. In that case, "first match" is the obvious thing to do.
- deleted 1y ago[deleted]
- ajross 1y agoIt's actually the simplest scheme. Reparse from the top whenever you need to query a setting. When you see one, exit. No need to even bother to store an intermediate representation. No idea if this matches the actual ssh implementation, but that's the way many historical parsers worked. The idea of cooking your text file on disk (into precious RAM!) is fairly modern.
- Dylan16807 1y agoLoading up your parsing code and reopening the file every time a setting is queried sounds to me like it would increase the average memory use of most programs.
- naniwaduni 1y agoYou don't care about average memory use, you care about peak memory use.
- deleted 1y ago[deleted]
- N2yhWNXQN3k9 1y agotitle could use a clean up
- pdimitar 1y agoYikes. What a nasty surprise. These tools really should be phased out in their current form.
- mrunkel 1y agoRTFM No tool can protect you from your own assumptions about how said tool works.
- shermantanktop 1y agoIn other words, prepare for maximum surprise? As a defensive posture in a hostile or random environment, that makes sense. But as a design approach, most designs go for the “principle of least surprise.” And that’s how I read the original comment: a well designed system wouldn’t do this. Joke is on them, though, because nobody designed this.
- ablob 1y agoSurprise is also based on convention. What could be surprising to you might be just a stroll in the park for others. In Japan people would be surprised to see others wearing shoes in a house while it's perfectly normal for people of other countries. Reading the manual is something to prevent surprises and only takes one sentence to explain. I'd go for that any day of the week! > ... a well designed system wouldn’t do this. ... A well designed system would be able to explain their decisions and document that somewhere. Perhaps in the manual.
- shermantanktop 1y agoA well-documented system would explain all decisions. A well-designed system would enable the user to learn a small set of principles and apply them in adjacent areas successfully without reference to documementation. As you say, I am used to checking the docs in Linux. It’s the convention that no convention shall be assumed. Is that good design?
- deleted 1y ago[deleted]
- casey2 1y ago[flagged]
- 77pt77 1y agoIn what world is ssh obsolete? What made it obsolete?
- squiggleblaz 1y agoI'm trying to brainstorm an answer. My best guess is that SSH is obsoleted by disposable instances. You can spin up a new instance for every version of your configuration, transition to it, and dispose of the original (or set it aside or whatever). That way, you could probably have a reasonably complete tech career and only ever use ssh as an implementation detail of git.
- bdangubic 1y agolol
- egberts1 1y agoPurpose of first-define is the rule: In placing configuration files higher than user-defined configuration but Only with SSH client, can want user to have control from their config files: Remove from config files Place a couple under Match/MatchGroup using deny/accept. SSHD (server/non-client) still support admin-defined by having system-wide settings done firstly. For those who have multi-file SSHD configurations, breakdown of the many config file locations and scopes here as it covers default user, system-wide, specific user: https://egbert.net/blog/articles/ssh-openssh-options-ways.html https://egbert.net/blog/articles/ssh-openssh-options-ways.ht... Also I broken out each and every SSHD and SSH options along with their ordering by execution by using file name and numbering as well as its various state machine, dispatch, CLI equivalence, network context, and function nesting, all in: https://github.com/egberts/easy-admin/tree/main/490-net-ssh https://github.com/egberts/easy-admin/tree/main/490-net-ssh https://github.com/egberts/easy-admin/blob/main/490-net-ssh/sshd_config.d/410-protocol-auth-methods.conf https://github.com/egberts/easy-admin/blob/main/490-net-ssh/... Disclaimer: I do regular code reviews of OpenSSH and my employer authorizes me to release them (per se contract and NDA) Also this showed how to properly mix and match authentication types using OR and AND logic(s) in https://serverfault.com/a/996992 https://serverfault.com/a/996992 It is my dump mess so wade 'em and enjoy.
- egberts1 1y agoFor those that are exploring software-based public certificate and OpenSSH, Ive broken down the settings for most PKI handlers. https://egbert.net/blog/articles/openssh-file-authorized_keys.html https://egbert.net/blog/articles/openssh-file-authorized_key...
- memco 1y agoThanks for sharing this! I think I may now have what I need to set up a system with multi-user shared keys that only work for a given set of users.
- egberts 1y agoI do enjoy dual-PK-certificate authentication in my homelab: one by equipment, and one by user/group. Only misgiving is that the key management issues have worsen only for the key administrator(s). But it is a viable and sustainable AA model because there is the most important security component: instant denial of a user and/or a equupment.
- therein 1y agoThe only time I hear or see anything about cloudinit, it is always a problem. Nobody ever said "we don't need worry about that, cloudinit takes care of it". What good does cloudinit do really?
- nightfly 1y agoProvides a moderately-configured starting point for new cloud VM deployments without requiring custom images
- naniwaduni 1y ago> Nobody ever said "we don't need worry about that, cloudinit takes care of it". Well, why would it come up? You don't need to worry about things you don't need to worry about.
- aflukasz 1y agoIn this particular case cloudinit presence in the story is incidental, delivery mechanism of said config file could have been different. It's useful for initializing state that could not have been initialized before booting in the target environment. Canonical example, I guess, being ssh server and client keys management, but the list of modules it implements is long.
- drpixie 1y agoThere is a nice sshd option (-T) that tells you what it's really doing. Just run sudo sshd -T | grep password
- mmsc 1y agoExcept that doesn't tell you what it's doing, that tells you what it _might_ do, if you (re)start the server. sshd -T reads the configuration file and prints information. It doesn't print what the server's currently-running configuration is: https://joshua.hu/sshd-backdoor-and-configuration-parsing https://joshua.hu/sshd-backdoor-and-configuration-parsing
- eliaspro 1y agoThat's why I only use socket-activated per-connection instances of sshd. Every configuration change immediately applies to every new connection - no need to restart the service!
- aflukasz 1y agoYes. Run this as a validation step during base os image creation, if such image is intended to start system with sshd. That way you can verify that distro you use did not pull the carpet from under your feet by changing something with base sshd config that you implicitly rely on.
- sneak 1y agoSame if you use ~/.ssh/conf.d/. I have been bitten by this before. :(
- 0xbadcafebee 1y agoThis is perhaps old sysadmin knowledge, but different tools have very different heuristics about how they parse configuration, and you have to check every time and not assume. Among the consequences to not checking are gaping security holes.
- INTPenis 1y agoYou mean there are consequences to making assumptions? ;) (also old sysadmin)
- czernobog 1y agoThis is interesting, usually it's the latter because the config is ran line by line Also, if it's not too much trouble, would someone help me understand why such files are required to start with numbers? In this case it's 10-no-password.conf. I have noticed similar structure for apt and many more packages
- glitchcrab 1y agoA lot of software which reads drop-in files will load them in numerical (or indeed alphabetical) order. Obviously this is important if the order your config files are loaded in matters, but otherwise it's just become a convention so people do it even if the load order doesn't actually matter.
- 47282847 1y agoTypically, config files are merged into one by loading them like ./conf.d/*, the order being determined alphabetically from their file names. You do not need to use numbers but they help to see that order.
- eapriv 1y agoOf course the order matters, that’s why the file names have numbers in them.
- 1oooqooq 1y agoyeah, this is confdir 101... but i guess learning is better late than never type of thing. also what confuses people more on this is that openssh is properly designed, so configs are first seen wins. exactly so that file 0_ wins from 99_... but most people see badly designed software where 99_ overrides 0_. openssh way is exactly so it works best with confir or ssh/options where it matches by hosts and you can place the more important stuff first without fear defaults will override it.
- deleted 1y ago[deleted]
- lubutu 1y agoI initially read "the order of files in /etc/ssh/sshd_config.d/" to mean the order of files in the underlying directory inode, i.e. as returned by `ls -f` — and thought, "oh god"... But the lexicographical order, that's not too surprising.
- extraduder_ire 1y agoI think their surprise comes from earlier config wins conflicts, rather than the other way around. That's not reflected in the title.
- immibis 1y agoI don't like these systems where configuration is built from a million separate files. They're unpleasant to work with. The best reason to do it this way seems to be that files are the unit of package management. Perhaps we need a smarter package manager. My nginx.conf life got better when I deleted sites-available and sites-enabled and defined my sites inline in nginx.conf. The only thing worse is when the configuration is actually a program that generates the configuration, like in ALSA. And the only thing worse than ALSA style is Xorg style, with a default configuration generated by C code and you can only apply changes to it without seeing it. Xorg also has this weird indirection thing where the options are like Option "Foo" "Bar" instead of Foo "Bar", but that's a nitpick in comparison.
- johnisgood 1y agoMost of my configuration files are in one file, but there are cases where it makes sense, such as /etc/modules-load.d, for one.
- samlinnfer 1y agoI've made a big stink about this last time: https://news.ycombinator.com/item?id=42133181 https://news.ycombinator.com/item?id=42133181 They've updated the documentation on /etc/ssh/sshd_config https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/2088207 https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/20...
- barotalomey 1y agoHence the tradition of numeric file naming in *.d directories.
- jmclnx 1y agoCurious, since when is directory "/etc/ssh/sshd_config.d/" a thing ? I checked on my OpenBSD (7.6) System and Slackware (15.0) and that directory does not exist. I checked the man page for sshd and there is no mention of that dir. Is this a new thing/patch Linux people came up with ?
- deleted 1y ago[deleted]
- dfc 1y agoIt might not be an OpenBSD thing. It may be a Debian/ubuntu-ism.
- SoftTalker 1y agoOpenBSD is the author of OpenSSH and yes they support the Include directive in sshd_config but they do not use it in a default install.
- jmclnx 1y agoJust read the manual, nice to see they are not patching ssh for this.
- nubinetwork 1y agoGentoo started doing this last year and I absolutely hate it.
- noufalibrahim 1y agoThis is weird. I've been hitting funny problems while trying to get ssh to authenticate (using passwords) via. a keycloak instance. I've been trying to do using PAM script but have been pretty unsuccessful till now. Apparently, they don't play nice together.
- NikkiA 1y agothis is true of all 'config.d' schemes, and why most such schemes suggest/use number-name.ext style filenames to deal with sorting.
- deleted 1y ago[deleted]
- teknopaul 1y agoIt's a pet gripe with cli apps that do first wins as well. You should be able to set alias foo='foo -p 80' And still write foo -p 81 espeak suffers this affliction