5 ms·

How is it with high bandwidth application? E.g would it be okay to put my media server behind it? Currently tunneling it through a VPS so cloudflare doesn't get

How is it with high bandwidth application? E.g would it be okay to put my media server behind it? Currently tunneling it through a VPS so cloudflare doesn't get mad.

xena 3y ago
Tailscalar here: there is a bandwidth limit, it's a funnel, not a hose. We don't announce what the bandwidth limit is, but please keep in mind that it does exist. I would suggest setting up your media server inside your tailnet for the best experiences, but it depends on who you are sharing it with and why.
- pciexpgpu 3y ago
  Hola, how would the bandwidth limit work within the tailnet if I am accessing it from outside my home network? Wouldn't it incur some bandwidth on Tailscale's end? I wonder if the DERPy-stuff helps remove most of the bandwidth concerns - thinking out loud...
  - _joel 3y ago
    Only the setting up of the session, it's effectively P2P then. Routing traffic back out onto the general internet for people without tailscale in your private net will be b/w limited, as mentioned.
- jonpurdy 3y ago
  I might be missing something; isn’t a Tailnet a bunch of user devices with wireguard tunnels connecting to each other directly? Where does the limit happen? (And thanks for your work!) Edit after 1 minute: of course, limit on Tailscale Funnel itself. (Too deep into thinking about Tailscale and forgot about the actual topic of the post. )
  - dijit 3y ago
    fundamentally, something has to be punching NAT somehow, so they're probably taking the traffic on their own servers and relaying it to your machine via the tailnet.
  - ripdog 3y ago
    Tailscale internally tries to punch NATs to make connections, but if it fails, it will use relay servers. These connections are throttled, and there'd probably be usage limits at some point. If you allow tailscale through by using ipv6 or port forwards, tunnels will run at line speed with no limits.
5e92cb50239222b 3y ago
Since tailscaled uses the tun/tap driver and thus copies all traffic to userspace (and back), it is extremely inefficient. On my Haswell i5 (plus multiple servers with comparable hardware) the process consumes 40% of CPU time at just 4 MiB/s, and close to 100% at 10-11 MiB/s (with recent sendmmsg/recvmmsg patches¹). This is about ~2-3x worse than similar applications written in highly optimized C, so don't expect any miracles from further optimizations unless they switch to kernel Wireguard (which doesn't seem likely in the nearby future). They claim it's very difficult if not impossible, but this sounds like an issue with their architecture — a similar application from their competitors² has had kernel WireGuard support from the start (no relation, I don't even use it and cannot recommend for or against it). 1: https://tailscale.com/blog/throughput-improvements https://tailscale.com/blog/throughput-improvements 2: https://github.com/netbirdio/netbird https://github.com/netbirdio/netbird
- xena 3y ago
  Tailscalar here, for what it's worth, I run my plex server on Tailscale (i5 10600) and I haven't noticed any observable lag due to the TUN/TAP driver. Even with 4k bluray rips at several tens of megabits per second of video quality. I also regularly get near the limit of gigabit ethernet when transferring big files like machine learning models (the 1280 byte MTU plus WireGuard overhead adds up over time and can make the application observed rate be less than what the NIC is actually doing). Kernel WireGuard for Tailscale is hard because of DERP (HTTPS/TCP fallback relay, all connections start over DERP so that they can Just Work if hole punching fails), but I'm sure it could happen with the right combination of eBPF and Rust in the kernel. It'd be a bit easier if there was a high level abstraction for using the kernel TLS stack to do outgoing TLS connections.
  - klabb3 3y ago
    Isn’t it also a UDP issue in general or at least the way packet switching works in Golang on major OSs? I did a bandwidth benchmark over local network over tailscale vs vanilla (in the 100MB/s ballpark) and tailscale was 10-20% slower and used tons of CPU. As a baseline I tried pushing blank UDP packets with Golang (on Darwin and Linux) at saturated capacity and it ALSO used similar excess CPU, causing dropped packets. My take at the time was that it was primarily the syscall overhead per packet (vs per arbitrarily sized buffer in TCP), and a lack of efficient OS APIs in Golang. Is there truth to this analysis?

How is it with high bandwidth application? E.g would it be okay to put my media server behind it? Currently tunneling it through a VPS so cloudflare doesn't get mad.

xena 3y ago

Tailscalar here: there is a bandwidth limit, it's a funnel, not a hose. We don't announce what the bandwidth limit is, but please keep in mind that it does exist. I would suggest setting up your media server inside your tailnet for the best experiences, but it depends on who you are sharing it with and why.

pciexpgpu 3y ago
Hola, how would the bandwidth limit work within the tailnet if I am accessing it from outside my home network? Wouldn't it incur some bandwidth on Tailscale's end? I wonder if the DERPy-stuff helps remove most of the bandwidth concerns - thinking out loud...
- _joel 3y ago
  Only the setting up of the session, it's effectively P2P then. Routing traffic back out onto the general internet for people without tailscale in your private net will be b/w limited, as mentioned.
jonpurdy 3y ago
I might be missing something; isn’t a Tailnet a bunch of user devices with wireguard tunnels connecting to each other directly? Where does the limit happen? (And thanks for your work!) Edit after 1 minute: of course, limit on Tailscale Funnel itself. (Too deep into thinking about Tailscale and forgot about the actual topic of the post. )
- dijit 3y ago
  fundamentally, something has to be punching NAT somehow, so they're probably taking the traffic on their own servers and relaying it to your machine via the tailnet.
- ripdog 3y ago
  Tailscale internally tries to punch NATs to make connections, but if it fails, it will use relay servers. These connections are throttled, and there'd probably be usage limits at some point. If you allow tailscale through by using ipv6 or port forwards, tunnels will run at line speed with no limits.

5e92cb50239222b 3y ago

Since tailscaled uses the tun/tap driver and thus copies all traffic to userspace (and back), it is extremely inefficient. On my Haswell i5 (plus multiple servers with comparable hardware) the process consumes 40% of CPU time at just 4 MiB/s, and close to 100% at 10-11 MiB/s (with recent sendmmsg/recvmmsg patches¹). This is about ~2-3x worse than similar applications written in highly optimized C, so don't expect any miracles from further optimizations unless they switch to kernel Wireguard (which doesn't seem likely in the nearby future). They claim it's very difficult if not impossible, but this sounds like an issue with their architecture — a similar application from their competitors² has had kernel WireGuard support from the start (no relation, I don't even use it and cannot recommend for or against it). 1: https://tailscale.com/blog/throughput-improvements https://tailscale.com/blog/throughput-improvements 2: https://github.com/netbirdio/netbird https://github.com/netbirdio/netbird

xena 3y ago
Tailscalar here, for what it's worth, I run my plex server on Tailscale (i5 10600) and I haven't noticed any observable lag due to the TUN/TAP driver. Even with 4k bluray rips at several tens of megabits per second of video quality. I also regularly get near the limit of gigabit ethernet when transferring big files like machine learning models (the 1280 byte MTU plus WireGuard overhead adds up over time and can make the application observed rate be less than what the NIC is actually doing). Kernel WireGuard for Tailscale is hard because of DERP (HTTPS/TCP fallback relay, all connections start over DERP so that they can Just Work if hole punching fails), but I'm sure it could happen with the right combination of eBPF and Rust in the kernel. It'd be a bit easier if there was a high level abstraction for using the kernel TLS stack to do outgoing TLS connections.
- klabb3 3y ago
  Isn’t it also a UDP issue in general or at least the way packet switching works in Golang on major OSs? I did a bandwidth benchmark over local network over tailscale vs vanilla (in the 100MB/s ballpark) and tailscale was 10-20% slower and used tons of CPU. As a baseline I tried pushing blank UDP packets with Golang (on Darwin and Linux) at saturated capacity and it ALSO used similar excess CPU, causing dropped packets. My take at the time was that it was primarily the syscall overhead per packet (vs per arbitrarily sized buffer in TCP), and a lack of efficient OS APIs in Golang. Is there truth to this analysis?