11 ms·
TLS certificates for internal services done right
- ramblurr 2mo ago> TLS certificates for internal services* done right * "internal services" = on a single server that is publicly routable
- nijave 2mo ago[dead]
- mrl5 2mo agoI've documented how to securely set up TLS certificates for internal services without creating TLS issues for http clients downstream. All thanks to split-horizon DNS, WAF and ACME protocol. All for free!
- pizzalife 2mo agoI don't agree that tunneling everything through some external facing proxy is "TLS certificates for internal services done right".
- nijave 2mo agoArguably it's 1/2. You can put public certs on proxy then give proxy private CA to backend services. Then you don't need public certs for all the private stuff nor need to trust the private CA on all your devices.
- wrxd 2mo agoI use the acme dns-1 challenge on my public domain. That gives you certificates you can use as you see fit, without needing to expose anything else to the public internet. I also use Tailscale so I configure my DNS to use my Tailscale IP addresses. If you don’t want to expose them on a public DNS server you can add them only to an internal DNS server.
- adontz 2mo agoMoreover you can delegate domain to improve security. https://www.eff.org/deeplinks/2018/02/technical-deep-dive-securing-automation-acme-dns-challenge-validation https://www.eff.org/deeplinks/2018/02/technical-deep-dive-se...
- throw0101d 2mo ago> I use the acme dns-1 challenge on my public domain. See also perhaps DNS aliasing in case you are not able to dynamically update your 'primary' domain, but can update a secondary or sub-domain: * https://github.com/acmesh-official/acme.sh/wiki/DNS-alias-mode https://github.com/acmesh-official/acme.sh/wiki/DNS-alias-mo... So if "example.com" is control by Corporate IT, and they don't want 'random' folks fiddling with it, then you can create a "dnsauth.example.com" and point the dns-1 challenge record from "…foo.example.com" to "foo.dnsauth.example.com" (or a completely different domain, like "…example.net"). There are DNS servers written strictly focused on this use case: * https://github.com/acme-dns/acme-dns https://github.com/acme-dns/acme-dns Also code that handles a bunch of DNS provider APIs so you don't have to roll your own for ACME client hooks: * https://github.com/dns-lexicon/dns-lexicon https://github.com/dns-lexicon/dns-lexicon
- spydum 2mo agothis is all fine and good, if you are okay broadcasting your internal hostnames. I suppose it's a trade off some might make.
- CartwheelLinux 2mo agoThere's one way around that which is requesting a wildcard cert, but then that has its own rammifications
- gh02t 2mo agoThere's really two ways, the other is to manage your own CA. But it seems like every browser/piece of software/etc out there is hell bent on making that as difficult as possible. It'd also be nice if it was easier to scope a certificate authority to a specific domain, but support for that is pretty patchy which is functionally the same as no support at all. And that's not to mention software that ignores the system certificate store. Or how tedious and nonstandardized it can be to get a trusted certificate store in a Docker container in cases where you have services that need to trust each other. Or how annoying it is to install your own trusted CA on devices (though, step-cli does help a lot at least on normal computers... phones however...). On and on and on, the barriers to what should be the obvious solution are extremely high.
- brewmarche 2mo agoIf scoping were actually viable, public CAs could also sign your private intermediate CA with a name constraint and it would be trusted everywhere.
- Hamuko 2mo agoI use a registered domain with DNS validation and then CNAMEs that I resolve locally. Basically: 1. Register a domain ("server.com") and put it on some public DNS that can do DNS validation with acme.sh. 2. Use DNS validation to get a certificate on your domain from Let's Encrypt. You can just grab a wildcard one ("*.server.com"). 3. CNAME all of your services on a public DNS to an internal address ("email.server.com" → "server.internal", "plex.server.com" → "server.internal"). 4. Resolve your internal address on a local DNS server with an A record ("server.internal" → 192.168.0.123). This can often just be done on your router. Since you use DNS validation, you just API keys for your public DNS service that acme.sh can use. No need to have any VPN network interfaces for getting your certificate. Your wildcard certificate also doesn't leak any details about your services.
- iambvk 2mo agoI like this approach. Thank you. Would it work if a user's device that is already connected to the VPN, but has custom DNS override to say 8.8.8.8 ? How can I allow my users to be able to use 8.8.8.8 DNS override and still work seamlessly?
- Hamuko 2mo agoI think you'd at least have to have the fallback DNS set to your local DNS resolver in order to get the server.internal resolved to a A record.
- cobertos 2mo agoWhy not just map the domain to an internal IP and call it a day? Then the only way it can be accessed is through a VPN. Then use a wildcard so none of leaks into cert transparency logs
- throw0101d 2mo ago> Then use a wildcard so none of leaks into cert transparency logs You now also have to build infrastructure to distribute the wildcard from (presumably) central place where you generate it to all the different places where it is desired. And hope the wildcard's private key does not leak from one of myriad of places it now lives.
- cobertos 2mo agoI have a few Traefik instances that request wildcards independently of each other. Each with the same config, per server. Leaking is an issue but we're talking about internal services too.
- AlexanderYamanu 2mo agowell, I configure my services te request their own wildcard certs from a caching proxy acme to letsencrypt. Easy peasy.
- patrakov 2mo agoMy preferred procedure is to use DNS-01 validation and have no publicly accessible "A" or "AAAA" record for internal services. Or even a more extreme example: https://crt.sh/?id=27555237869 https://crt.sh/?id=27555237869 (sorry for any possible crt.sh downtime) - the domain name in question never existed in public or private DNS by itself. It is used only for a WPA3-Enterprise network, as the CN that WiFi clients expect to be present in the RADIUS server certificate, but never resolve. In the public DNS, only the "_acme-challenge" TXT record exists.
- otabdeveloper4 2mo agoSounds bonkers. Why not make an overlay LAN and host your own DNS server in 10.0.0.0/8?
- patrakov 2mo agoI do have a DNS server in my LAN, with some records served to internal clients only. But the _acme-challenge record needs to be public for the DNS-01 validation to succeed. The point was that you can obtain a certificate for a domain name without creating any records other than the _acme-challenge TXT record. I.e., that the domain might be completely empty all the time except for this record.
- EvanAnderson 2mo agoThe relative proximity of the words "done right" and "split-horizon DNS" makes my insides hurt a little bit. Use DNS validation to allow these internal services to pull ACME certs. There's so much less headache, long-term. Split-horizon DNS (and the tedious make-work it can create when you start needing to mirror public-accessibly records in the private DNS) has always been something to aspire to move away from in my experience.
- bombcar 2mo agoJust LE a wild card cert and slap it everywhere.
- EvanAnderson 2mo agoI get antsy about a private key being in lots of places. You've also got to worry about renewal so you might as well just have "consumer" of the wildcard just provision its own non-wildcard certificate.
- sandermvanvliet 2mo agoI run a reverse proxy that handles the TLS termination for that reason. Services themselves are constrained to the server, bound to listen on 127.0.0.1 only. Key is only available to the reverse proxy.
- stock_toaster 2mo agoOnce dns-persist-01 becomes available/usable[1], it should make dns validation even easier. [1]: https://letsencrypt.org/2026/02/18/dns-persist-01 https://letsencrypt.org/2026/02/18/dns-persist-01
- isomorphic 2mo agoThank you, I was unaware of that. It looks like it's already support in the acme.sh client, but there is a Let's Encrypt discussion saying it's still pending at LE: https://community.letsencrypt.org/t/dns-persist-01-deployment-status-and-timeline/246468/9 https://community.letsencrypt.org/t/dns-persist-01-deploymen... I wonder if the interim version has been rolled out to some CAs.
- thomashabets2 2mo agoPersonally, I hate split horizon DNS. I prefer the "BeyondCorp" model. I MUCH prefer putting an mTLS cert in my trusted devices over relying on VPNs in same devices. I've yet to see a "clever" DNS setup not cause annoyances. Specifically grafana is nice to be able to see on the phone, and split horizon DNS and corp VPN is a hassle, to say the least, on phones. I bet you can do it with HA-Proxy, but I use https://github.com/ThomasHabets/sni-router https://github.com/ThomasHabets/sni-router
- cactacea 2mo agoZero trust or bust. I actually cringed when I read "done right" applied to split-horizon DNS.
- raquuk 2mo agoI am looking forward to finally using DNS-PERSIST-01 for validation. No more dynamic DNS updates, DNS credentials or forwarding necessary.
- Thom2000 2mo agoSadly most tools still doesn't support it: https://github.com/cert-manager/cert-manager/issues/8373#issuecomment-4344858213 https://github.com/cert-manager/cert-manager/issues/8373#iss... And then the issue is protecting the private key of the issuer and monitoring certificates (it's a good idea to do that anyway).
- zeeZ 2mo agoOne of the draft authors is already working on a cert-manager implementation. I'm optimistic it'll make it into cert-manager once the spec is ready and CAs actually start to implement it.
- boscillator 2mo agoThe real answer here is that configuring HTTPS clients to trust a self-signed cert (or signed by an internal CA) shouldn't be as difficult as it is. I find it extremely annoying that every programming language has it's own idea of where certificates should live instead of just checking the os trust store.
- spacebanana7 2mo agoIt's hard to get right when OSs, programming languages, browsers and sometimes other applications have their own opinions about trust stores. I understand why our IT department want corporate devices to use internal CA certs on paper but it just breaks stuff in the real world.
- afarah1 2mo agoEven if all applications look at the OS trust store, in my experience there's always a gap distributing the CA to every consumer, leading to time spent on debugging from time to time... Maybe that's not the case in perfectly homogeneous or sufficiently small environments where every team uses the same infra / stack.
- WhyNotHugo 2mo agoYeah, adding CAs to the store sucks. On Linux it needs to be in /etc/ssl/certs (this varies slighly per distribution), which is only writable by root. A single user can't trivially trust a CA, and a great deal of applications/libraries don't support overriding the store's path.
- thayne 2mo agoIt's worse than that. OpenSSL uses /etc/ssl/certs (or sometimes a different directory, depending on the distro) by default. But Java uses its own store that uses an incompatible java-specific format, and Chromium derivatives and Firefox have their own trust stores. So to add a CA to a system it is often necessary to add it in multiple places.
- skywhopper 2mo ago
- xorcist 2mo agoThis is crazy. If you have a home network with a few internal services, or some sort of network where you don't control the endpoints, just use DNS validation. That's why it exists. But on hosts you control, you should absolutely provision them with an identity and join the local CA. You're going to need it for a multitude of other reasons.
- maqnius 2mo agoCan you elaborate why one shouldn't use DNS validation for hosts you control in general?
- xorcist 2mo agoThat's not it. If you have an internal network, where every host is provisioned by you, you already control identity. In that case there's no need to validate anything as names, dns records, certificates and anything else should already be in place.
- llama052 2mo agoThere's certainly something to be said for ease of use and not having to ensure you push trusted certs to every device that touches your internal network. Unless you enjoy that sort of thing.
- raffraffraff 2mo agoHmm. I don't really care enough about leaking home network host names because they are all super generic names like 'router', 'laptop', 'tv', 'nas'. So I use my public zone on cloudflare. I just use internal ip addresses (eg: nas.example.com = 10.1.2.3) on the public zone and DNS01 challenge for let's encrypt. Anyone can resolve the ip for any of my hosts, but obviously you'd need to be on the wireguard vpn to hit them. This means that I can always use public DNS servers like 1.1.1.1, 8.8.8.8, nextDNS etc This is not "done right" by any stretch but it's extremely low effort to set up and has never once failed me, unlike countless complex meshy things.
- luckman212 2mo agoFair, but what about names that are specific enough to give an attacker a clue to a potential attack surface, like "authelia.example.com" - now they know you've likely got an Authelia setup, and can start digging for exploitable CVEs etc. I'm in the process of removing all my individual certs and replacing with a wildcard cert served by Traefik. Is that a bad idea?
- bbkane 2mo agoCan they dig for exploitable CVEs if they're not on the Wireguard network? It is a clue to your infrastructure, but I personally think the simplicity is worth it.
- nijave 2mo agoMy IaC is on public GitHub. They could do a network scan to find software then fingerprint to find version anyway. Removing attack surface is better than trying to hide it.
- icedchai 2mo agoDo the names resolve to publicly routeable IPs? If not, I wouldn't worry about it.
- akerl_ 2mo agoHow many people out there have attackers doing individualized research to identify services on their home LAN so they can chain a network attack with CVEs in their self-hosted service?
- AtNightWeCode 2mo agoDon't do this. Public certs are for public services.
- cpach 2mo agoWhy does it matter?
- aliasxneo 2mo agoI wonder if the author realizes that getting public certificates results in them being recorded in CT logs.
- Thom2000 2mo agoYep. Especially with non-wildcard certs this leaks all service names (privacy concern).
- jmbwell 2mo agoI don’t know much in this space, but I find myself wishing there was a dead simple self hosted CA solution and also that trust on first use (à la ssh) was A Thing for self-managed root certs in client implementations. TOFU is such an elegant, good-enough solution for these use cases. Fixed deployment is always still an option, but in this day and age it feels so much like we are unnecessarily still dealing with solved problems
- nijave 2mo agoOn k8s, there's cert-manager but also you need k8s... Most browsers support trust on first use for leaf certs
- jmbwell 2mo agoI guess I mean treat it as a clear first class feature. Right now most browsers treat it as an arcane error. I’m thinking more “This is the first time you’re connecting to this site. Do you trust it?” And later if something changes, then they can do the whole DOING SOMETHING NASTY! thing, which is effectively the experience today
- nijave 2mo agoYeah, that's fair. I think they're optimized for non technical users without a decent escape hatch.
- notTooFarGone 2mo agoFirefox now started that you can't even go on the page on some occasions. Using a browser in an air gapped environment is so much more pain than it should be.
- zufallsheld 2mo agoWell, if your cert-manager distributes its own CA, you'd still need the clients to trust the CA, even in k8s.
- 2mo ago
- sandeepkd 2mo agoI was under impression that I understand networking and DNS resolution. It was really hard to follow, the OP did worked hard, just not sure what exact problem was being solved with the proposed solution that isnt already been solved.
- gmuslera 2mo ago"Right" without use case can be wrong. And by use case I include scale. For a small team, few machines, some in-place infrastructure may worth it. Smaller than that may be overkill, bigger than that may not be enough, or end being cumbersome, insecure or not work for everyone.
- jabart 2mo agoA Github Action running acme.sh that pushes certs to S3 solves the split dns issue for hosts, which can cause all sorts of weirdness after a while. You can then grab a cert on a schedule and even make them wildcard if you want. Then you will get NXDOMAIN if you are not on the VPN so ideally no public traffic.
- samgranieri 2mo agoSplit horizon DNS is not something I'm willing to do. I'll just rock out with .internal or .home.arpa, have step-ca and bind communicate to each other, either in step-issuer in kube or maybe even rfc2136 if i feel like a bit of the dns-01 strategy is in order. I slap the internal ca root certs everywhere, and keep my home infra out of the crt.sh logs. I get it, I could just do *.mydomain.com and slap that wildcard cert everywhere, but it's still in the public logs..
- Eduard 2mo agoto help with passive reconnaissance, here are tuxnet.dev's SSL certificates and associated subdomains: https://www.certkit.io/tools/ct-logs/?query=tuxnet.dev https://www.certkit.io/tools/ct-logs/?query=tuxnet.dev
- waynesonfire 2mo ago¯\_(ツ)_/¯ Not quite done right.
- 28304283409234 2mo agoWhen using letsencrypt for internal services it becomes very hard to distinguish badplayer.com LE certificate from the good players. You only ensure encryption. Not identity. Do not use letsencrypt internally for things that matter.
- xurukefi 2mo agoOr... - Don't use split DNS. Don't use any special internal or dev domain. Leave it to your infrastructure to route/NAT those public IPs to your internal network. - Don't use the HTTP-01 challenge. Use DNS-01. - Don't run your own internal CA. Use Let's Encrypt. If you care about name leakage (CT Logs), use wildcard certs. Use a central reverse proxy/load balancer for termination.
- preisschild 2mo ago> Use a central reverse proxy/load balancer for termination. If you do that anyways, you could also use something like oidc authN/authZ on the reverse proxy level and just expose it to the internet. You dont even need to self host the oidc idp, you can use Google/Github or even something like ATProto
- jeroenhd 2mo agoUsing wildcard certs and/or a central reverse proxy defeats the purpose of internal TLS. Split-horizon DNS for a publicly usable domain is almost always a bad idea, but running your own ACME server is pretty easy (maybe 10 lines of Caddy config) and using an internal domain (an actual one, not a randomly picked TLD you don't think exists yet) solves the problem pretty easily. You'll want a safe backup for your root certificate private key, of course, but that's pretty much all you need to really worry about.
- Plasmoid 2mo agoLet's encrypt is great but if you operate any kind of scale you can quickly hit their rate limit. AWS only recently started allowing you to pull the certificate out of ACM.
- Walf 2mo ago>Don't use split DNS So what's your solution when you have a wholly private service that will never have a public v4 address, nor a publicly routable v6? How do clients get the address for a nice domain name without the addresses in public DNS? I use acme.sh with DNS validation, and use common domains that have both public and private services on subdomains. I use split horizon so private.domain.example resolves only on LAN and VPN, and public.domain.example resolves everywhere, but the address changes depending on the network one is connected to.
- 0010010111 2mo agoFeasible in local (7) btrfs.
- sigio 2mo agoMy setup is having a wildcard DNS record and a wildcard certificate for my 'home' domain. It has a fixed IP from my ISP, so you always end up on haproxy, which then forwards to individual ports/ip's in the internal network. I can do filtering based on source-ip from there, so traffic from myself/internal will be allowed, and outside traffic (not from some allowlisted ip's) will get blocked. ACME validation is done via DNS, so nothing needs to be accessable for that to issue certificates. Internally I will usually also use the public IP for services, so no need for a split-dns.
- guptadagger 2mo agoim not understanding why you would need a waf at all
- blueflame7 2mo ago[dead]
- denkmoon 2mo agoBogus nonsense. Sign your internal certs with an internal CA, make your clients trust the CA. Use it properly.
- NicolaiS 2mo ago> "make your clients trust the CA" There's no single trust store: the OS has one, Firefox, Java (cacerts), Python (certifi), Node, Go containers, all your Docker images, ... Failure? People just toggle TLS verification off. Easy to do in a one-man shop, but almost impossible in any big company. Try grepping for verify=False / -k / InsecureSkipVerify in any fortune 500 and you'll find plenty
- denkmoon 2mo agoYes, it feels like I echo the same sentiment at work every few weeks. No, we can’t tick the PR with verify=false. Yes, you have to configure the trust store. Yes, it’ll take maybe an hour to work this out. Go do so please. The only reason is laziness.
- woranl 2mo agoI use client certificate authentication (or mTLS) to access internal services. It requires the client device to present a digital certificate during the TLS handshake, so I don’t need to white list any IP addresses.
- mnahkies 2mo agoPersonally I think wildcard certs are more secure in many instances - if you're just running one load balancer/reverse proxy then all the private keys are going to the same place anyway. You get the upside of not leaking your subdomains to transparency logs, and having fewer certs to renew. It also forces you to use DNS challenges, which means you don't require the publicly route able address. Personally what I do is: - Predominantly use wildcards - Run a tiny coredns instance with my A records for a internal subdomain of my normal domain - Configure tailscale to use the coredns resolver - Run two haproxy instances, one for internal services, one for public facing. The public facing one can't route directly to the internal services. When even the obscure DNS provider I'm using is supported for DNS challenges, I really don't see much upside to using HTTP challenges anymore
- waynesonfire 2mo ago> - Run two haproxy instances, This made me consider my haproxy architecture. I use haproxy acl rules and the use_backend directive to enforce security policy and routing. It denies http requests based on source ip address per service (host header) I think a good middle-ground is to use a single haproxy but two frontends; one binds to the public interface and one for private. haproxy is now no longer responsible for routing, the router is.
- mnahkies 2mo agoYeah you could run a single instance with multiple frontends - but the way I'm currently running things the public haproxy instance doesn't have network connectivity to the private services, meaning there is no way I can misconfigure things and expose an internal service to the internet.
- master_crab 2mo agoSplit brain =/= done right. This introduces all sorts of aberrant behavior. Most notably that clients will get different responses depending on their query and route that you now have to test for. It also is a “silent” deviation for audiences that may not be aware of. I’m not sure what problem that is trying to be solved here? That the OP wanted internal users seeing a different site than external users? Or using a different route in? In both those cases the correct user behavior is to use a different DNS record.
- nntlol 2mo ago[dead]
- lee_ars 2mo agoOP, why mess with acme.sh's cron schedule with your own cron job that calls its cron job? You might instead consider running acme.sh again with the "--install-cert" flag, coupled with its "--reloadcmd" flag, in order to automatically have it do the cert+key copying and nginx reloading for you when it renews: acme.sh --install-cert -d grafana.tuxnet.dev --key-file /etc/ssl/private/grafana.tuxnet.dev.key --fullchain-file /etc/ssl/certs/grafana.tuxnet.dev.crt --reloadcmd "systemctl reload nginx" Then you can ditch your custom cron and let acme handle everything on its own, as intended.
- maxgashkov 2mo agoI've settled on using .internal and Knot as a authoritative NS, step CA + ACME to issue short-lived certs, and a Split DNS resolver from Tailscale as the only external dependency (mostly as a convenience for when I'm on the road). I do have a luxury of all the homelab VMs being rebuildable via IaC, so I've just injected CA trust at that step. The biggest PITA so far were 3rd party docker images, each with its own way to inject custom CA. iOS devices were surprisingly easy to handle.
- ivlad 2mo agoEnterprise PKI is not hard and has many uses besides issuing certificates to web servers. Any company of 1000+ users or endpoints should just set one up. I did it multiple times, most recently using YubiHSM as root key store for offline enterprise root CA.