8 ms·
Vxlan over WireGuard (On OpenBSD)
- blakesterz 4y agoI love how this starts off "What is a VXLAN..." that was going to be my first question! So often posts like this seem to assume everyone knows what the topics are at the start. I know what WireGuard is, and I've at least heard of VXLAN before, but I couldn't remember what it was.
- yokem55 4y agoThe feature I would be interested in is if this can do link state toggling on the on the vxlan interface if the wg handshake timer goes stale. If that works, then it becomes practical to do things like run ospf routing over the vxlan interface.
- gorkish 4y agoWhat exactly would you be intending to accomplish? OSPF already has state timers, and furthermore runs just fine on a wireguard interface without having to introduce a vxlan tunnel.
- deleted 4y ago[deleted]
- floatinglotus 4y agoThe difficulty isn’t setting it up, it’s in determining how much traffic you can put through it before it breaks. How does it handle segmenting jumbo frames? Etc.
- candiddevmike 4y agoJust do static routes or BGP over Wireguard. Simpler, scalable, less error prone.
- vxxzy 4y agoDoes this get you Layer 2 connectivity?
- candiddevmike 4y agoNo, because most applications don't need layer 2 adjacency.
- vxxzy 4y agoYou may consider VoIP phones. When a phone boots up on a network segment the DHCP process comes into play. This process is a Layer 2 process. The DHCP packet could contain a boot server field (this is typical) so the VoIP phone can grab configuration. One may want the boot server info "isolated" from other network segments. Utilizing VLANs is one way to do this. Additionally, it is typical that QoS is applied at Layer 2 (something necessary for real-time protocols like VoIP).
- candiddevmike 4y agoI hope you're not trying to run VoIP phones via vxlan over wireguard because you don't want to setup a local DHCP server
- chungy 4y agoMaybe he'll write a blog post about it :)
- vxxzy 4y agoOver VXLAN yeah... You won't believe how many small-medium companies are moving from on-prem to small managed private clouds. VXLAN allows them to maintain their existing network configuration. "You hope" ha! I suppose by this you mean that the wireguard + vxlan may not be a "mature" combination for something as mission-critical as voice traffic? VXLAN implementations by mature companies (Netgate, Fortinet, etc...) those work. Maybe this wireguard+vxlan would work too.. who knows!? Edit: I see you changed your comment to consider remote DHCP which causes my comment above to be irrelevant. Oh well. The fact still holds that VXLAN is meant to handle Layer 2 so saying BGP is an alternative is like putting a square peg in a round hole.
- icedchai 4y agoI'm doing OSPF over Wireguard (running BIRD on Debian, not OpenBSD though.) It works pretty well.
- iso1631 4y agoBut doesn't provide layer 2 between networks. Think of devices that are hardcoded to communicate with broadcast or multicast with TTL of 1, you either need some active reflector, and cope with any perculiarities of the device, or you simply extend a single vlan between two routers (using vxlan or another solution) I sometimes need to extend a system like this from one site to another. One is a calrec system (an audio mixer, I think it's the control traffic that needs to be sent), and I don't have enough access or time to see if I could build some kind of transparent proxy -- it won't work with multicast routing. I do however have enough time to create a layer2 network between two nics. I tend to use mikrotiks for that, create an eoip tunnel (GRE with proprietary addons to cope with fragmentation) between the two endpoints and pop the interface in a bridge with a physical port, and move on.
- johnklos 4y ago"Here's how I did this thing." "You don't need to do that thing." How do you know?
- sophacles 4y agoI've been in or very close to networking for a long time. Here's a certainty: there's lots of ways to do anything. Here's a corollary to that: someone will be sure to tell you how bad any choice you make is.
- tristor 4y agoThere's a weird font-rendering bug on this site that causes the text in the code blocks to be unreadable unless you highlight it with your mouse. If you enable Javascript, it seems to fix it. Not sure if the author is reading this thread, but it's something you may find worth investigating fixing.
- Arnavion 4y agoWorks for me with JS disabled, unless the author fixed it in the two minutes since you posted your comment. Code blocks have: style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4" ... in the HTML itself.
- tristor 4y agoThe bug still appears present for me: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/109.0 Code blocks without JS: https://i.imgur.com/Tcq34IK.png https://i.imgur.com/Tcq34IK.png Code blocks without JS under highlight: https://i.imgur.com/Lbvk2LS.png https://i.imgur.com/Lbvk2LS.png Code blocks with JS enabled: https://i.imgur.com/kDCW4Q0.png https://i.imgur.com/kDCW4Q0.png
- deleted 4y ago[deleted]
- crest 4y agoIP over Ethernet over VXLAN over UDP over IP over WireGuard over UDP over IP over Ethernet… sigh OpenBSD does support both routing domains and multiple routing tables and includes multiple routing daemons in the base system. I would recommend to the author to stop hacking at the keyboard, grab whatever not to structured visualisation tool works for them (e.g. a whiteboard, a block of paper, a random drawing app, Visio) and (re-)phrase the problem. Are you solving a problem or showing of how many acronyms you can expand without looking them up? This n layer encapsulation can work and can even be required to reproduce some (problematic) organisational structure, but it's far from elegant. Given the chance I would vastly prefer to just use multiple routing domains for the WireGuard tunnel interfaces and the underlay. It would result in far less complexity to manage as well as less overhead. Why do so many people insist on tunneling Ethernet over IP? What's keeping operators from using IP routing (and just one layer of encapsulation) instead? Is IP routing so scary or everyone that indispensable applications that only work over Ethernet?
- deleted 4y ago[deleted]
- lstodd 4y agoBecause stuff that requires this circus of encapsulation is usually so brain-dead that it can't be gotten to work in any other less horrible way. (also I think you lost one 'over UDP')
- crest 4y agoYou're right. I forgot that WireGuard sits on top of UDP.
- aroulin 4y agoUsing vxlan you can also connect L3 networks, not just L2 networks. i.e., virtualize an L3 network
- jacob019 4y agoWireguard virtualizes L3 out of the box.
- generalizations 4y agoThis can cause massive packet fragmentation. I'd be most interested in the performance degradation due to the L2 encapsulation. Are there any benchmarks available for this kind of project?
- jacob019 4y agoThat's what I was thinking, unless you have jumbo frames you're going to have a hard time stuffing ethernet frames into IP payloads. Does Vxlan mitigate this somehow?
- Denatonium 4y agoI've tunneled VXLAN over Wireguard on Linux. In my setup, my WAN's MTU was 1500 bytes, and my Wireguard tunnel's MTU was 1550, with the VXLAN's MTU being 1500. Surprisingly, traffic and iperf3 tests going over the VXLAN had much better throughput than traffic going directly over the Wireguard connection. IIRC, over the VXLAN, I was pulling ~800Mbps over the VXLAN/WG setup with iperf3. Where this would fall apart is if there are firewalls in between that silently drop UDP fragments. In a case like that, it may be necessary to do VXLAN/Wireguard/Wireguard to conceal the fragmented packets with MTUs of 1500/1550/1440 respectively, assuming IPv4 and WAN MTU of 1500. I bet this would come with a significant performance hit though.
- noobface 4y agoThis is fun, but applications requiring L2 adjacency do it to limit latency/distance. Creating a L2 domain between here and the moon, what are you gonna use it for? Certainly not anything other than fun.
- TechBro8615 4y agoCan you use this to get Apple bonjour / mDNS working over a remote network (connected via VPN)? Or similarly, could you use it for a cloud seedbox to cast to a chromecast on your local network (via the VPN obviously)?
- gorkish 4y agoFor mDNS, you should run avahi to relay between subnets. For chromecast, that is SSDP/DLNA which is a multicast, so it is a matter of establishing mulitcast routing between sites. Bridging L2 is not the optimal solution for either of your scenarios.
- skullone 4y agoThere's a number of specific scenarios this could be useful, like, some SANs can only replicate to L2 adjacent units. Say you wanted a replica off-site, and your gear is older/proprietary, you used to have to buy enterprise network gear to encap L2 and ship VLANs to remote sites. I wouldn't be dismissive of using VXLAN over wireguard to accomplish that.
- lillecarl 4y agoYou just explained how to increase technical debt in as few steps as possible. The solution is getting a wavelength or dark fiber to the off-site, or throwing out the piece of junk SAN that only works on L2, it's too old by now.
- skullone 4y agoI didn't say "this solution is easier", simply said someone might find a use case for it. Ya'll are dismissive of something neat with a number of use-cases (I only tossed out just one use case that popped into my head, based on actual experience on a million dollar SAN, that is still supported (and sold!) to this day).
- miller_joe 4y agoThis is great! One small suggestion: try it with the newer veb bridge device. It should be a bit faster.
- jmclnx 4y agoI could not get into the site, so from archive: https://web.archive.org/web/20230214134248/https://rob-turner.net/post/vx-lan/ https://web.archive.org/web/20230214134248/https://rob-turne... A little bit over my head, but an interesting read.
- anton5mith2 4y agoI took this to extremes last year: I used it to run MAAS from Australia to sweden (which requires layer 2). Granted I used tailscale to make the WireGuard part even easier, but it was a lot of fun. https://medium.com/@antongslismith/bare-metal-cloud-provisioning-from-gcp-de4b65747de https://medium.com/@antongslismith/bare-metal-cloud-provisio...
- lillecarl 4y agoMAAS doesn't require L2, it requires DHCP and DNS to be configured correctly.
- frupert52 4y agoWow, I can’t believe the HN audience is so accepting of stretched layer 2 as a solution. It’s almost as though we’ve been invaded by middle management. Stretched layer 2 is almost always a mistake.