6 ms·
An open source, self-hosted implementation of the Tailscale control server
- mountainriver 1y agoLove headscale, we just took it to production and it’s been great
- syntaxing 1y agoAs in you rolled out an internal service for the whole company?!
- sshine 1y agoI’d love to see a write-up on that. Especially in the unlikely event that you used Nix for the deployment.
- benley 1y agoI've done exactly that: headscale in production at work, a few hundred client devices, infrastructure mostly powered by nix. What would you want to hear about it?
- squiggleblaz 1y ago* Does it work well? * Do you recommend it? * Do your users care? * Is it difficult? Do you have to maintain it or is it basically set it and forget it? * What was memorable about setting it up? * Why did you go for Headscale vs Tailscale or Netbird or some other solution?
- benley 1y agoI posted a reply to another subthread with some of this: https://news.ycombinator.com/item?id=43647368 https://news.ycombinator.com/item?id=43647368 > * Does it work well? Very well! There are some limitations (see link above), but what's implemented is reliable. > * Do you recommend it? Yes, provided your requirements fit headscale's capabilities. If you need things like device trust attestation (e.g. Kandji MDM or Crowdstrike Falcon integration), SCIM provisioning, or various other enterprise features you may find it inadequate. If you can afford to pay for Tailscale, you should just use Tailscale because it's really good. > * Do your users care? They like it way better than our previous OpenVPN setup, that's for sure. I don't think they care about Headscale vs commercial Tailscale - the backend implementation is largely invisible to them. > * Is it difficult? Do you have to maintain it or is it basically set it and forget it? Not hard at all to set up, and it requires little maintenance attention. I have barely had to touch the control plane (other than version upgrades) since setting it up a year ago. > * What was memorable about setting it up? We had to do some custom coding to have automatic user offboarding when employees leave the company, and to emulate app connectors / dynamic routing (this is now OSS! https://github.com/singlestore-labs/tailscale-manager https://github.com/singlestore-labs/tailscale-manager). And I've been contributing to the headscale codebase to smooth out some quirks that affected our SSO integration. The headscale authors have been pretty flexible in welcoming outside contributors. > * Why did you go for Headscale vs Tailscale or Netbird or some other solution? vs Tailscale: It was way easier to build this myself than to get funding to use the commercial solution. I'm not good navigating corporate politics, but I am pretty good at building infrastructure and writing code. vs Netbird: Mostly because I already liked Tailscale from using it at home, I like its implementation, and I like the way Tailscale (the company) have behaved. The handful of folks I know who work there are people I deeply respect.
- sshine 1y ago> headscale in production at work - How much effort do you put into key management compared to plain WireGuard? - How automated is the onboarding process; do you generate and hand over keys? - How do you cope without the commercial Tailscale dashboard? - Do you run some kind of dashboard or metrics system? - How long did it take to set up? - Were there any gotchas?
- avtar 1y ago> How do you cope without the commercial Tailscale dashboard? There are a couple open source dashboard options but right now only this one comes to mind: https://github.com/tale/headplane https://github.com/tale/headplane
- benley 1y agothere are a bunch of them: https://headscale.net/stable/ref/integration/web-ui/?h=web https://headscale.net/stable/ref/integration/web-ui/?h=web The one I've deployed is https://github.com/gurucomputing/headscale-ui https://github.com/gurucomputing/headscale-ui, which is basic but does what I need.
- benley 1y ago> - How much effort do you put into key management compared to plain WireGuard? Less effort than plain wireguard; the only key management I do is for non-human clients > - How automated is the onboarding process; do you generate and hand over keys? Fully automated. Auth is done via OIDC to my company's SSO provider, so users can enroll their own machines without IT involvement. > - How do you cope without the commercial Tailscale dashboard? I don't really miss it. The headscale CLI tool is pretty good, and I use one of the headscale web UI projects (three are several: https://headscale.net/stable/ref/integration/web-ui/?h=web https://headscale.net/stable/ref/integration/web-ui/?h=web) for quick access to a few features (https://github.com/gurucomputing/headscale-ui https://github.com/gurucomputing/headscale-ui) > - Do you run some kind of dashboard or metrics system? Yes, I scrape headscale's Prometheus metrics endpoint and have put together a simple Grafana dashboard. The metrics it emits are somewhat limited, but enough to keep an eye on its health. > - How long did it take to set up? I had a prototype up and running on Kubernetes with OIDC integration and a web UI in about 1 day of hacking. Going into full production took a few months, but the majority of that time was about planning the migration of all the existing users from OpenVPN. Come to think of it, maybe I should share my terraform modules for deploying it. > - Were there any gotchas? A few, yeah: - Setting up mobile clients is a bit fiddly, because they hide the "connect to a non-default control plane URL" under a debug menu. The mac and windows apps are similar - it's too easy for users to accidentally try to connect to tailscale.com instead of your headscale instance. If you have the ability to deploy MDM profiles (mac) or windows registry tweaks this is easy to fix, and the headscale server will even generate the configs for you. - The headscale control plane doesn't support any kind of HA or replication. This doesn't disqualify it since tailscale can handle brief control plane outages without breaking the network, but it's likely to be a concern for serious enterprise users. It's possible to use an external Postgres database, so you can at least replicate data that way, but only one headscale server replica can be active at a time because they don't share runtime state. - The tailscale API is not fully implemented, so you can't use things like the tailscale Kubernetes operator. - Some features are missing: tailscale funnel, tailscale serve, app connectors, `autogroup:self` ACLs, SCIM provisioning, SSO group membership sync, and I forget what else. These may or may not be important to you. For app connectors, I wrote an app to emulate the core functionality: https://github.com/singlestore-labs/tailscale-manager https://github.com/singlestore-labs/tailscale-manager (it's in Haskell, but deployers don't need to care about that) It's possible to implement group sync with some custom scripting - a python app to scrape your LDAP (or whatever) and generate tailscale ACLs isn't hard to write. But you do have to write it. `autogroup:self` might be a big deal - you would need this if you want to stop users from seeing or connecting directly to each other's devices. I think there is an implementation of this coming in the next release of headscale. Summary: headscale is great if you have relatively simple needs and can't afford to pay for Tailscale. You will probably outgrow it if you're running a serious business and need to comply with fancy audit requirements.
- cassianoleal 1y agoAs opposed to what? This seems pretty normal. We considered it as well but there was a feature missing that meant we couldn’t use it for one of our main requirements. Had that not been the case, we’d have rolled it out.
- mrklol 1y agoMind sharing which feature?
- cassianoleal 1y agoHonestly I'm hazy on the details but we're running a fairly complex environment in GCP with PSC everywhere, connections to on-prem and other external environments, and something wouldn't quite work due to all that. Sorry I can't provide any more details but I really don't remember the specifics. We were in touch with Tailscale engineers and they offered some workarounds that we had already worked out but that wouldn't help us achieve what we were after.
- mountainriver 1y agoAll our infra
- linsomniac 1y agoI've been running headscale for 2.5 years and it's been pretty good. We use our gmail domain for logging in, which gives a big benefit that users can self-serve their devices. Unlike with OpenVPN in the past where ops had to hand off the certs and configs. Really the only downside has been when they accidentally connect to the tailscale login server instead of our own and then can't figure out why they can't reach any services. We use user groups to set up what services users can access. We are still running the old headscale, because we have some integrations that will need to be ported to the new control plane. According to "headscale node list | wc" we have ~250 nodes, most of them are servers. One thing I really don't love about tailscale some of the magic it does with the routing tables and adding firewall rules, but it has mostly not been an issue. Tailscale has worked really quite well.
- telotortium 1y agoShould add the project name, Headscale, to the title Headscale has been on HN many times.
- 3abiton 1y agoThis looks interesting! What's the added value over wireguard + openwrt setup?
- watusername 1y agoYour devices will connect to each other peer-to-peer (even behind complex NATs) with no manual configuration, subject to ACLs you centrally manage. It just works. People sometimes dismiss Tailscale as "just" a WireGuard orchestrator, but it's actually much more than that - From a product perspective, WireGuard is just an implementation detail.
- compootr 1y agoit's wireguard that doesn't make me hate myself :)
- usagisushi 1y agoIt's a mesh VPN, so peers communicate directly without additional delay. I opted for Netbird myself because Headscale's UI felt too basic for me back then. Has that improved over the years probably?
- udev4096 1y agoHow is netbird? Is it more stable than tailscale/headscale? How is your performance while streaming a video?
- usagisushi 1y agoThey are both based on WireGuard (kernel-space and user-space `wireguard-go`), so I guess there's no significant difference in performance for typical usage. In terms of stability, Netbird has been pretty good for me. I've been using Netbird as the backhaul network for my laptop, phone and inter-site k3s cluster for several years without major issues. One major downside of Netbird is that its Android client can be quite a battery drainer [1]. (It keeps your fingers warm during winter, though!) As for Tailscale, it offers some neat features like Funnel, which is missing in Netbird, but in my case, covered by DNS and k8s Ingress. [1]: https://github.com/netbirdio/netbird/pull/3379 https://github.com/netbirdio/netbird/pull/3379
- pluto_modadic 1y agowonder if some of the bugs with self-managing it have been worked out :)
- snvzz 1y agoHeadscale has been serving me well for half a year now. It is great, to the point I have no idea how I lived without a tailscale network before. It is packaged in openbsd, and that package is the server I am using.
- pilif 1y agoKeep in mind that for many use cases (mobile access, GUI on macOS), this relies on the official Tailscale clients keeping the ability to set the control server. The moment the inevitable enshitification will start at Tailscale, this feature will go away. I’m saying this as a currently super happy Tailscale customer who was burned multiple times in the past by other companies being sold or running out of VC money
- coconut08 1y agoarent most of the the tailscale clients open source aside from the gui portion of the non open source os's?
- pilif 1y agoYes they are, unless you're using a mainstream OS and/or want to use a GUI, which is probably the most common use case.
- __float 1y agoWhile the GUI is somewhat helpful, at the end of the day it's not the key piece, and it could easily be rebuilt.
- notpushkin 1y agoI think the whole Windows client is closed. On macOS though you can use it from the command line just fine (apart from a couple quirks due to a completely different VPN implementation [1]). [1]: they have three: https://tailscale.com/kb/1065/macos-variants https://tailscale.com/kb/1065/macos-variants
- squiggleblaz 1y agoFrom https://github.com/tailscale/tailscale https://github.com/tailscale/tailscale "This repository contains the majority of Tailscale's open source code. Notably, it includes the tailscaled daemon and the tailscale CLI tool. The tailscaled daemon runs on Linux, Windows, macOS, and to varying degrees on FreeBSD and OpenBSD. The Tailscale iOS and Android apps use this repo's code, but this repo doesn't contain the mobile GUI code." and "The macOS, iOS, and Windows clients use the code in this repository but additionally include small GUI wrappers. The GUI wrappers on non-open source platforms are themselves not open source." Moreover, there's https://github.com/tailscale/tailscale-chocolatey https://github.com/tailscale/tailscale-chocolatey to aid the build process. I haven't built it or run it. On the other hand, while I suppose the Windows app is probably reasonably straightforward to replicate, I guess it would be much harder to produce an iOS or Android app because of the vagaries of mobile programming.
- SuperShibe 1y agoEvery few months I come back to this repo to check if they finally got Tailnet lock running or if someone security audited them in the meanwhile. Unfortunately neither of these things seem to make any progress and thus, I’ve grown uncertain in how much I can trust this as a core part of my infrastructure. The entire premise of Tailscale SaaS builds on creating tunnels around your firewalls, then enabling the user to police what is allowed to be routed through these tunnels in a intuitive and unified way. Headscale seems to have nailed down the part of bypassing the firewall and doing fancy NAT-traversal, but can they also fulfill the second part by providing enough of their own security to make up for anything they just bypassed, or will they descend to just being a tool for exposing anything to the internet to fuck around with your local network admin? To me, not giving your Tailscale implementation any way for the user to understand or veto what the control server is instructing the clients to do while also not auditing your servers code at all sure seems daring…
- gpi 1y agoOne of the maintainers work for tailscale now.
- wutwutwat 1y agomaintainer's employment != security audit
- gpi 1y agoMy thinking is their time is divided now and could lead to less efforts spent on headscale.
- palotasb 1y agoNot compared to the previous state where he worked for an unrelated company and only had his free time to contribute to Headscale.
- kradalby 1y agoPerson with split time here, I definitely have more time to spend on it now, I have half a work week vs sometime in the evenings or weekends if I had excess energy after having my other job.
- voxadam 1y agoDoes it run on Plan 9?
- bradfitz 1y agoHa! But seriously, it's written in Go, so probably.
- udev4096 1y agoHow does headscale hold up when you're streaming video over jellyfin/plex?
- scottyeager 1y agoDo you mean when using it as a relay because p2p connectivity isn't possible? The preferred operating mode of Tailscale networks is for the bulk of traffic to go p2p, using various tricks for NAT and firewall traversal.
- cassianoleal 1y agoI’ve used it extensively to stream video across continents. No issues as long as you can get a P2P connection going. If it needs to go through a DERP server, then it may suffer but in my experience that’s pretty rare.
- watusername 1y ago> If it needs to go through a DERP server, then it may suffer but in my experience that’s pretty rare. It's semi-frequent in my case, and it's painful every time it does that since Tailscale's official DERP servers are very slow (they seem to have some aggressive QoS). It would be nice if Tailscale supported using regular TURN servers so I could just use one of the hosted solutions.
- cassianoleal 1y agoYou can self-host DERP if you're up for it.
- LilBytes 1y agoYep and most of us are already using Subnet routers it's not technically much harder. Finding a cloud or VPS provider with free or cheap bandwidth (egress and ingress) is likely the biggest issue.
- Happily2020 1y agoIf you're interested in self-hosting your orchestration server, you can look into Netbird. It's a very similar tool, but has the server open sourced as well. So you have a self-hosted control server with a nice GUI and all the features the paid version does. https://netbird.io/knowledge-hub/tailscale-vs-netbird https://netbird.io/knowledge-hub/tailscale-vs-netbird
- yamrzou 1y agoDoes it do the fancy NAT-traversal Tailscale does?
- mrbluecoat 1y agoYes: https://www.netbird.io/knowledge-hub/netbird-network-routes https://www.netbird.io/knowledge-hub/netbird-network-routes
- davidcollantes 1y agoCompared to Headscale, Netbird has so many moving pieces! It looks robust, and powerful, and featureful... yet, self-hosting Headscale is super simple, and less demanding.
- mynameisvlad 1y agoI've been slowly moving everything over from Tailscale to Netbird and aside from some shenanigans with Tailscale taking over the entire CGNAT route, it works wonderfully! Tailscale is still running for now, but I'm getting closer and closer to decommissioning it and switching entirely to Netbird.
- unixfox 1y agoNo IPv6 though. Which is real deal breaker: https://github.com/netbirdio/netbird/issues/46 https://github.com/netbirdio/netbird/issues/46
- nchmy 1y agoI'd love to use netbird, but it doesn't yet have capabilities to be embedded in a go binary, like tsnet for tailscale allows. Here's a gh issue for it. https://github.com/netbirdio/netbird/issues/1103 https://github.com/netbirdio/netbird/issues/1103
- infogulch 1y agoI think it would be neat if headscale allowed peering / federating between instances. (Maybe after the ACL rework.) One of the main problems is address collisions. So here's my proposal: commit to ipv6-only overlay network in the unique local address (ULA) range, then split up the remaining 121 bits into 20 low bits for device addresses (~1M) and 101 high bits that are the hash of the server's public key. Federate by adding the public key of the other instance and use policy and ACLs to manage comms between nodes. I think it's a nice idea, but the maintainer kradalby said it's out of scope when I brought it up in 2023: https://github.com/juanfont/headscale/issues/1370 https://github.com/juanfont/headscale/issues/1370
- aborsy 1y agoHow much is the risk of my devices being compromised if Tailscale coordination server is compromised, and tailnet lock is enabled?
- 1vuio0pswjnm7 1y ago"To me, not giving your Tailscale implementation any way for the user to understand or veto what the control server is instructing the clients to do while also not auditing your servers code at all sure seems daring..." This statement sugggests that publishing the Headscale control server source code is not enough to allow the user to "understand or veto what the control server is instructing the clients to do". If using the Headscale control server, the user can "understand or veto" anything "the control server is instructing the clients to do". This may be accomplished by reading, editing and compiling the source code. If using the Tailscale control server, the user can only "understand or veto what the control server is instruction the clients to do" to the extent that the Tailscale company permits. The user is prohibited from editing or compiling the source code. Not all users want the option to read, edit and compile third party software that they use. Some users may be comfortable relying on the ongoing assurances of companies funded by Silicon Valley VC. For those users that want the option of 100% open source projects, not dependent on venture capital, Headscale can be useful. The author of Headscale calls the Tailscale coordination server "essentially a shared dropbox for public keys".