6 ms·
Yes! SSH certificates are awesome, both for host- and client-verification. Avoiding Trust on First Use is potentially a big benefit, but the workflow improveme
by rhco 3y ago
Yes! SSH certificates are awesome, both for host- and client-verification.
Avoiding Trust on First Use is potentially a big benefit, but the workflow improvements for developers, and especially non-technical people, is a huge win too.
At work, we switched to Step CA [1] about 2 years ago. The workflow for our developers looks like:
1. `ssh client-hosts-01`
2. Browser window opens prompting for AzureAD login
3. SSH connection is accepted
It really is that simple, and is extremely secure. During those 3 steps, we've verified the host key (and not just TOFU'd it!), verified the user identity, and verified that the user should have access to this server.
In the background, we're using `@cert-authority` for host cert verification. A list of "allowed principals" is embedded in the users' cert, which are checked against the hosts' authorized_principals [2] file, so we have total control over who can access which hosts (we're doing this through Azure security groups, so it's all managed at our Azure portal). The generated user cert lasts for 24 hours, so we have some protection against stolen laptops. And finally, the keys are stored in `ssh-agent`, so they work seamlessly with any app that supports `ssh-agent` (either the new Windows named pipe style, or "pageant" style via winssh-pageant [3]) - for us, that means VSCode, DBeaver, and GitLab all work nicely.
My personal wishlist addition for GitHub: Support for `@cert-authority` as an alternative to SSH/GPG keys. That would effectively allow us to delegate access control to our own CA, independent of GitHub.
[1] https://smallstep.com/docs/step-ca https://smallstep.com/docs/step-ca
[2] https://man.openbsd.org/sshd_config#AuthorizedPrincipalsFile https://man.openbsd.org/sshd_config#AuthorizedPrincipalsFile
[3] https://github.com/ndbeals/winssh-pageant https://github.com/ndbeals/winssh-pageant
- JohnFen 3y agoRequiring the use of a browser, though, limits the usefulness a bit.
- xyst 3y agoI like this solution, thanks for sharing. Just need to swap it with my own OIDC compliant federated authentication server.
- v7n 3y agoThat is also the flow for Tailscale SSH https://tailscale.com/kb/1193/tailscale-ssh/ https://tailscale.com/kb/1193/tailscale-ssh/
- deleted 3y ago[deleted]
- deathanatos 3y ago> we've verified the host key (and not just TOFU'd it!), How. Specifically, what I cannot determine from their docs is how the VM obtains a host key/cert signed by the CA. How does the CA know the VM is who the VM says it is? (I.e., the bootstrap problem.) (I assume that you also need your clients to trust the CA … and that has its own issues, but those are mostly human-space ones, to me. In theory, you can hand a dev a laptop pre-initialized with it.)
- mjg59 3y agoI can't say in the general sense, but with GCP you can retrieve the EKpub of a VM's TPM via the control plane, and then use https://github.com/google/go-attestation https://github.com/google/go-attestation to verify that an application key is associated with that TPM and hence that VM
- rhco 3y agoStepCA supports quite a few authentication methods, including an "admin provisioner" (basically a passphrase that can be pasted into the CLI tools' stdin). Because each of our servers are bespoke, we can use the admin provisioner when the server is first being set up (and actually, Ansible handles this part). I don't have experience with it, but StepCA also has Kubernetes support, and I imagine the control plane could authenticate the pod when a cert needs to be issued or renewed.
- tjoff 3y agoIf you are in the terminal and don't have access to a browser?
- hnlmorg 3y agoNot the OP but if anyone doesn’t have access to a browser in my org then I can safely say they’re not accessing from a company laptop and thus should be denied access.
- hamburglar 3y agoNot sure why you'd get downvoted for this comment. This is likely very applicable for many orgs that have operator workstation standards -- they're some kind of window/osx/linux box with a defined/enforced endpoint protection measures, and they all have a browser. Any device I can imagine ssh'ing from that doesn't have a browser is definitely out of policy.
- 0xedd 3y agobecause both of you narrow visioned the scenario to what you do daily. it is a common use case to ssh from a jump server, use ssh based CLI tools and debugging. the issue stems from windows users who are coupled to GUIs. the behavior pattern increases IT and DevOps costs unnecessarily. an alternative example: our org solves the issue with TOTP, required every 8 hours for any operation; from ssh/git CLI based actions (prompted at the terminal) to SSO integrations. decoupling security from unrelated programs. simple and elegant.
- hnlmorg 3y agoThere are better ways to access remote servers than using a jump box. If you’ve gone to the lengths to tie SSH auth into a web based SSO then you should have at least set up you’re other infra to manage network access already (since that’s a far bigger security concern). Plus, as others have said, you can jump through SSH sessions with on the client ssh command (ie without having to manually invoke ssh on the jump box).
- sam_lowry_ 3y agoNow try to automate that.
- Deathmax 3y agoGitHub does have support for SSH CAs, but it's an Enterprise feature: https://docs.github.com/en/enterprise-cloud@latest/organizations/managing-git-access-to-your-organizations-repositories/about-ssh-certificate-authorities https://docs.github.com/en/enterprise-cloud@latest/organizat...
- rhco 3y agoThat's very interesting, thank you for linking!
- dmagyari 3y agoVault also supports both client and server ssh certificates [1]. I use terraform and vault to sign server certificates at creation time. [1] https://developer.hashicorp.com/vault/docs/secrets/ssh/signed-ssh-certificates https://developer.hashicorp.com/vault/docs/secrets/ssh/signe...
- deleted 3y ago[deleted]
- simiones 3y agoAt work, we switched to Step CA [1] about 2 years ago. The workflow for our developers looks like: 1. `ssh client-hosts-01` 2. Browser window opens prompting for AzureAD login 3. SSH connection is accepted How is that simple, compared to `ssh -i .ssh/my-cert.rsa someone@destination --> connection is accepted, here's your prompt` ?
- Aeolun 3y agoThis is simple because it doesn’t require you to take any specific actions to make new/different hosts accessible. If you deactivate someone in AD, poof, all their access is magically gone, instead of having to remove their public key from every server.
- shrx 3y agoWhat if you're ssh-ing from a headless client, like a raspberry pi or a VPS?
- WastingMyTime89 3y agoThen it doesn’t work but their developers are ssh-ing from their work laptops so it doesn’t matter. Something doesn’t have to be a solution for all use cases to be a good solution.
- lucideer 3y agoThe former is discoverable: it doesn't require developers having ANY knowledge of command switches (no matter how basic) nor following a set of out-of-band instructions; the "how to" is included within the workflow.
- misnome 3y agoOne thing I've never understood about SSH certificates for client identification - it looks like it causes the requirement that _at some point_ ssh private keys and the certificate private key need to both be in the same place? And if this is the case, then doesn't that imply that you need to have a service where users upload their private key? Which would mean you have one single point of attack/DOS/failure that needs to be kept utterly secure at all costs?
- karatinversion 3y agoI thought the way it worked was that the certificate signed with the certificate private key only contains the public key, and the ssh server, after checking the certificate is valid, validates that the client has the private key corresponding to the public key in the certificate.
- sokoloff 3y agoYou give your public key (typically into ~/.ssh/authorized_keys) and then prove you have access to the matching private key as the essential part of the challenge. You always keep the private key.
- calvinmorrison 3y agoAlso - key forwarding. Private key is on your local, you can forward it through ssh so you can hop around from your next destination
- counttheforks 3y agoHow do you get the browser to open? Does it work on all operating systems and ssh clients, such as Android's JuiceSSH?
- philo23 3y agoThis is a bit off topic, but does anyone know how the mechanism that triggers the web page prompt from an ssh connection actually works? Is it some kind of alternate ssh authentication method (like password/publickey) or something entirely out-of-band coming directly from the VPN app intercepting the connection? Ever since I saw it in action with Tailscale I've always wondered how it actually works, and I guess if anyone would know they'd be on HN
- fisian 3y agoSmallstep uses ProxyCommand [0]. Not sure how Tailscale does it. 0: https://smallstep.com/docs/ssh/how-it-works https://smallstep.com/docs/ssh/how-it-works
- v7n 3y agoOOB: ".. during the SSH protocol’s authentication phase, the Tailscale SSH server already knows who the remote party is and takes over, not requiring the SSH client to provide further proof (using the SSH authentication type none)." https://tailscale.com/kb/1193/tailscale-ssh/#how-does-it-work https://tailscale.com/kb/1193/tailscale-ssh/#how-does-it-wor...