8 ms·

Defend against vampires with 10 gbps network encryption

270 points by alxjsn 2y ago

andrewla 2y ago
I had a friend who worked in federal law enforcement who once described a vampire device that they used. It would clamp around a power cable and inject a UPS in the mix so that an electronic device could be removed without turning it off. Seemed like a useful little trick.
- 0cf8612b2e1e 2y ago
  If nothing else, would let you move a Frogger machine. More seriously, I have wondered if you can detect these kinds of external interference. Auto lock the machine if power/network/wifi/Bluetooth/USB conditions change. Nabbing an unlocked laptop was how they got the Silk Road guy (though they probably already had sufficient evidence elsewhere). https://arstechnica.com/tech-policy/2015/05/sunk-how-ross-ulbricht-ended-up-in-prison-for-life/ https://arstechnica.com/tech-policy/2015/05/sunk-how-ross-ul...
  - MertsA 2y ago
    Wifi would probably be the easiest. Either hide a dummy AP in the house or use a combination of multiple neighbors APs. If you don't see any beacon frames from the dummy SSID for a 30 second period then lock/shred the computer.
    - justsomehnguy 2y ago
      Wifi 5/6 sometimes rake up to a couple of minutes to get online (DFS and whatever) so 30 seconds is like smoking near an open can of gasoline: mostly fine but when it's not...
  - amluto 2y ago
    Maybe attack the problem from a different angle: use an accelerometer. Or spend a little bit more money to add a gyro and make a real, if very low accuracy, IMU.
    - adgjlsfhk1 2y ago
      Seems like an mems accelerometer would be all you need. Rotation isn't really a threat...
      - amluto 2y ago
        Rotation itself isn’t a threat, but if you want to directly estimate displacement to distinguish between earthquakes and someone stealing the machine, without relying on heuristics, actual inertial measurement would do the trick. And inertial measurement involves tracking the direction of acceleration, which involves tracking rotation.
    - 0cf8612b2e1e 2y ago
      That is a great suggestion. I think Android just implemented a “snatch detection” system for phones. Although, I like the idea of not requiring additional hardware. I guess when I start running a drug empire I will have to pony up for the extra dongle.
    - TeeMassive 2y ago
      That's a great idea. Authorizing any kind of physical change should be a default security measures.
    - TimeBearingDown 2y ago
      BusKill was created for this, USB with a magnetic attachment to a keyring that can be configured to take action on disconnect.
    - mcpherrinm 2y ago
      Some HSMs I've used (payshields) have tamper sensors that can detect motion for this reason. > The ADXL362 accelerometer in the PayShield 10K acts as a "Motion Sensor" detecting tilt movements. An alarm triggers an alert if the HSM is moved (for example, slid out of the rack)
  - jeroenhd 2y ago
    One trick you could use is to abuse the fact that law enforcement often plugs in a mouse wiggler on an unlocked desktop and kill your server the moment you see a new HID device (make sure to run some kind of desktop on your server so they think they can keep the session open, best to do it in a VM). You could also monitor the ethernet link. They can move your server but they can't move the entire network, set up an encrypted tunnel between two distant physical servers and self destruct the moment that tunnel gets disrupted. Some computers come with gyros/accelerometers built in. My old HP laptop had some kind of head crash prevention that used that hardware. I know this, because Gnome thought it was a tablet style sensor and turned my screen upside down if I didn't disable the sensor. Maybe getting a HP server can already get you a whole bunch of movement sensors. You could probably figure out if the server is being moved by measuring capacitance of the case, measuring accelerometers, maybe add a GPS dongle. Or you could add an LTE connector and measure any signals you may receive that you shouldn't from inside a server room. You can probably measure _something_ in the server room, though, so to make sure your LTE dongle doesn't get interrupted, also measure whatever reliable signal you can find to detect Faraday cages. Lastly, you could put a video camera in the case on all sides and measure changes. Detecting law enforcement badges probably isn't that hard with opencv if you're dedicated enough. You have to hide your security measures and never tell anyone, though, or they'll just leave the server as-is and use the classic rubber hose exploit to make you give up the key material.
    - tbrownaw 2y ago
      > Or you could add an LTE connector and measure any signals you may receive that you shouldn't from inside a server room. Incoming Bluetooth Low Energy announcements should have a receive power level associated with them. Stick a beacon (like say a standard ble temperature/humidity sensor) somewhere, and you should be able to tell if the distance to it changes.
    - Onavo 2y ago
      [flagged]
    - gosub100 2y ago
      https://github.com/defuse/swatd https://github.com/defuse/swatd
  - dullcrisp 2y ago
    If they’re seizing your laptop and your laptop will only work inside your house, wouldn’t they just seize your whole house?
    - lobsterthief 2y ago
      They absolutely would
    - 0cf8612b2e1e 2y ago
      It is a secret one way lock. Disturbing the machine and it locks/encrypts/sheds data. Bringing the machine back to the safe zone would not decrypt the data.
  - akira2501 2y ago
    > detect these kinds of external interference. Easily. Bolt the machine to the floor in such a way where the case has to be opened and a trip sensor activated to actually move the machine. You can switch my power source without noticing? Who cares. The attack is taking the machine where it is not supposed to be. That's a problem we've been solving since forever.
- the_real_cher 2y ago
  So it would emulate a UPS? So they could just remove the existing UPS? what is inject a UPS?
  - amiga-workbench 2y ago
    Its a parasitic tap that connects to the mains power cable going into the device. It then phase locks an inverter with said mains power, allowing the mains power cable to be unplugged and the whole lot transported elsewhere on battery power.
    - amelius 2y ago
      How do you reliably get to the copper without shorting it?
      - aaronmdjones 2y ago
        Careful application of a box cutter for the outer sheath followed by something resembling a scotchlok connector for line and neutral. Edit: If the machine is plugged into a power bar / power strip / whatever you want to call it, this is much easier still: Plug the vampire UPS into the power bar as well, wait for it to sync up to the grid, and disconnect the bar from the outlet. The UPS continues to feed power into the bar and thus keeps the machine powered.
        pests 2y ago
        Power strips make this easier of course, but every outlet usually has two plugs and most* of the time they are wired together. You just need to plug into the other plug. * In case they are split for whatever reason (switched plug, different circuit) whatever, just take off the faceplate, pull out the outlet, and now you have direct access to the screw terminals and copper wiring on the outlet. You could wire into the plug using the second set of terminals or via the other connection method (one being the screw terminals, the other being the "insert into the hole" depending on which is used) and take the whole outlet with you.
- cruffle_duffle 2y ago
  Isn’t that kinda what they used for Ross Ulbright’s computer? I know it was a laptop but they probably didn’t want to take chances given if that thing shut down the entire thing would be encrypted?
  - andrewla 2y ago
    I thought they had an attractive agent distract him for a moment while another agent grabbed his still-unlocked-and-open laptop to prevent him from locking it or closing it up. At least I think that was the cloak-and-dagger story I heard.
    - gosub100 2y ago
      two agents posing as a couple feigned a raucous quarrel that distracted him, while a third agent sitting across the table yanked the laptop at the precise moment he was distracted
      - thebruce87m 2y ago
        And they had to improvise as the cafe he was originally going to visit was too busy and he went to a library instead. Super interesting story!
- immibis 2y ago
  Someone successfully did this for copper gigabit ethernet and presented at one of the security conferences - but with a few milliseconds interruption in signal.
- sschueller 2y ago
  That is why you put in special outlets that communicate with the PC over the power line encrypted. You would need to drill holes in the concrete wall to get to the power lines in the wall in order to take the outlet along and hope that there isn't an additional device in the breaker panel.
- deleted 2y ago
  [deleted]
dathery 2y ago
Really cool article, I enjoy reading through all the details behind the decision making. Just spit-balling a little, but I wonder if Wireguard is the best tool here given that the author is only using it for a single point-to-point link and they control the devices on both ends. That CPU supports AES-NI and probably does it a lot faster than Wireguard's ChaCha20 (hard to get numbers for their server CPU, but the tiny little x86 mini PC I use as my router does AES XTS at 43Gbps according to `cryptsetup benchmark`). You might see better performance by tunneling the vxlan connection using a different technology which can use AES-NI? Then again, Wireguard is definitely still a good tool for stuff like this, and maybe the performance penalty isn't a big deal here.
- wmf 2y ago
  Because Wireguard is cool and AES is uncool.
  - tptacek 2y ago
    I guess it depends on whether you're more concerned about transport security or cipher cycles/byte.
    - dathery 2y ago
      Is there reason to think AES used appropriately would be any less secure here? Not trying to be argumentative, genuinely curious. My understanding is that AES has some design warts that make it not ideal (basically, it's easy to both implement and use in ways that leak information if you're not careful) but that it's still essentially perfect symmetric encryption if you're using it as recommended. Is that wrong? FWIW, the reason I brought up performance was because the OP spends a large chunk of the post talking about it, so I assume it's an important requirement for them.
      - wmf 2y ago
        AES is probably fine as a cipher but the VPN protocols that aren't Wireguard tend to have various footguns available. In theory someone could create NoisyESP but I'm not aware of it.
exabrial 2y ago
Why MACSEC isn't the default is pretty crazy! given that is is extremely stateless (encrypting at the frame level) and counters should be pretty reliable (only go up, since there's two parties) you could take advantages of some AES and GCM modes that would pretty quickly spot injection, replay, and other attacks. But getting back to the main topic of the paper: why not just S2S IPSec the link?
- justsomehnguy 2y ago
  > Why MACSEC isn't the default TFA explains it pretty well. Also every encryption is adding the load and latency, so defaulting to it when it wasn't asked for isn't the best way > why not just S2S IPSec the link? Because IPSec is still PITA and also sucks bad performance wise against WG.
  - nullc 2y ago
    I don't recall the specifics of macsec but it's possible to build a link encryptor that adds essentially zero latency. (like... no more latency than the gate delay of a single xor gate... plus some once-an-hour packet-length delay of some rekeying traffic).
nsteel 2y ago
Did Cisco really invent MACSec?! I thought it was cooked up by the IEEE and supported in hardware from many vendors. I imagine they all have their own bugs though, it's quite a complicated spec. I know some switch/router vendors also now offer hardware-accelerated end-to-end encryption, similar to IPsec, Nokia call their's anysec but I'm sure the other players have their own. The benefit of those is you'd get full bandwidth (e.g. Tbps).
- wmf 2y ago
  Usually one vendor prototypes a feature then they take it to IEEE/IETF for standardization. Probably half of all network protocols were invented by Cisco.
theideaofcoffee 2y ago
This is a great writeup! Especially for those that may want to DIY it, the how and the why and all of that, and not have to shell out for carrier-quality Layer 1 encryption devices. Nice to see that even off-the-shelf components can do it with relative ease at those rates. Also nice to see sane sysctl tunes as well. Anything to make an adversary's day a bit harder. I low key love the explanation of old 10B5 taps, something that so well and truly dead, but the legacy carries on into everything new today. This is actually a well-trodden area of datacenter interconnect (DCI) devices that do line-rate encryption (to crazy levels like 400G+) to protect those links that may have easily accessible fibers strung along poles, for instance, to prevent just the vampirism described in the post. Packetlight, Ciena, Infinera and others.
aaronmdjones 2y ago
# setup a 8020 MTU on wg0 interface to account for the 80 bytes wireguard headers overhead # 20-byte IPv4 header or 40 byte IPv6 header, 8-byte UDP header 4-byte type, 4-byte key index, 8-byte nonce, 16-byte authentication tag) /sbin/ip li set dev wg0 mtu 8020 Shouldn't that be 8920? To go with the 9000 byte MTU on the outer interface above it.
- bb88 2y ago
  That would probably add only a smidge more performance.
  - aaronmdjones 2y ago
    Sure, but the comment block says that they arrived at 8020 by subtracting 80 from 9000, which is just wrong.
icehawk 2y ago
I did something like this to stretch L2 as I was moving into a new home. Worked great after I realized t-mobile does not like passing IP fragments. Got to use it again to set up a remote telescope for the eclipse.
westurner 2y ago
A notebook with pandas would have had a df.plot(). 9.71 Gbps with wg on a 10GBps link with sysctl tunings, custom MTUs,. I had heard of token ring, but not 10BASE5: https://en.wikipedia.org/wiki/10BASE5 https://en.wikipedia.org/wiki/10BASE5
dietr1ch 2y ago
Missing attack: Cause a disruption that obviously breaks the connection while further away you get time to tap it properly. "Oh, no, a truck run into the pole carrying the copper/fiber, it must be an accident and no intervention is going on undetected because of the outage." What we really need is promiscuous connectivity , but fully untrusted connections. It's maddening why it's hard to communicate 2 wireless devices while they are literally sharing the same radio spectrum and multiple radios could be used to talk to each other.
nullc 2y ago
Tapping is even easier if you have access to the cable end in a patch panel. I have a computer setup with a one-way gige connection for reviewing potentially malicious content in an air-gapped manner. The transmit side transceiver needs to see an incoming signal, so I just use one of these to feed its own output back into it: https://www.amazon.com/dp/B0B8ZHBK26 https://www.amazon.com/dp/B0B8ZHBK26
deleted 2y ago
[deleted]
ggernov 2y ago
I prefer to just run fiber inside of a copper gas line.
- sulandor 2y ago
  because it's easily available and unconspicuous? asking because copper is neither cheap nor particularly hard to penetrate.
c0l0 2y ago
As I had posted a few weeks ago (https://news.ycombinator.com/item?id=41085314 https://news.ycombinator.com/item?id=41085314), I recently implemented a very similar thing myself. My solution ended up using tc's mirred[0] action for implementing a fully L2-transparent frame relay. I wonder if their setup achieves the same degree of transparency, because afaiui, that's just not possible involving a 802.1Q-compliant (Linux) bridge. I spent close to a week optimizing my setup looking at kernel flame graphs and perf results, reading adapter-specific tuning guides and driver source, and can say that the only really meaningful performance optimizations (in both the Broadwell- and Zen3/Vermeer-based implementations I tried) were disabling mitigations in the kernel (esp. on Zen3, that boosted performance by more than 20%), and getting CPU frequency scaling/idle states sorted out correctly (which yielded much higher wins on the older Broadwell uarch, because power state transition appears to happen much quicker on Zen3). As for the solution presented in the (on the whole really great; I love it!) article, I have my doubts about the effectiveness of the cargo-culted "sysctl tuning" mentioned - TCP, for example, is simply not involved at all in the described setup, so "tuning" its buffer allocations cannot have any effect on the workload. Kudos to the writers for solving their problem in a creative, cost-effective and maintainable way! :) [0]: https://www.man7.org/linux/man-pages/man8/tc-mirred.8.html https://www.man7.org/linux/man-pages/man8/tc-mirred.8.html
- sulandor 2y ago
  > CPU frequency scaling/idle states sorted out correctly please elaborate
  - c0l0 2y ago
    To "fix" performance (i.e., increase throughput by close to 35%) one has to mess with the "energy performance bias" on the (Broadwell) platform, e. g. using x86_energy_perf_policy[0] or cpupower[1]. Otherwise, the CPUs/platform firmware will select to operate in a very dissatisfactory compromise between high-ish power consumption (~90W per socket), but substantially less performance than with having all idle states disabled (= CPU in POLL at all times, resulting in ~135W per core) completely. One can tweak things to reach a sweet spot in the middle, where you can achieve ~99% of the peak performance at very sensible idle power draw (i.e., ~25W when the link isn't loaded). With Zen3, this hardly mattered at all. I also got to witness that using IPv4 for the wireguard "overlay" network yielded about 30% better performance than when using IPv6 with ULA prefixes. [0]: https://man.archlinux.org/man/x86_energy_perf_policy.8 https://man.archlinux.org/man/x86_energy_perf_policy.8 [1]: https://linux.die.net/man/1/cpupower https://linux.die.net/man/1/cpupower
qhwudbebd 2y ago
I'm interested in their last step, in which they set vm.dirty_ratio = 40 vm.dirty_background_ratio = 10 vm.swappiness=10 governor=performance energy_perf_bias=performance min_perf_pct=100 to raise a surprisingly low performance ceiling. I can't believe they were under any memory pressure, so the first three presumably made no difference, but it's also quite surprising to me that the default ondemand cpu governor was responsible for such a dramatic performance hit. Not throttling up quickly enough leading to higher latency maybe? Very interesting anyway.
- justsomehnguy 2y ago
  > Not throttling up quickly enough leading to higher latency maybe? Quite amusingly it's the load isn't big enough for the default governor to rise the clock up to max. If they posted the cpu load it would be pretty obvious.
mrbluecoat 2y ago
> Eth/IP/UDP/WG/IP/UDP/VXLAN/Eth/802.1q/IP/Payload I'd love to see your IT manager's face when you propose it :D
- wmf 2y ago
  This is not uncommon. At my last job we had double VXLAN.

270 points by alxjsn 2y ago

I had a friend who worked in federal law enforcement who once described a vampire device that they used. It would clamp around a power cable and inject a UPS in the mix so that an electronic device could be removed without turning it off. Seemed like a useful little trick.

0cf8612b2e1e 2y ago
If nothing else, would let you move a Frogger machine. More seriously, I have wondered if you can detect these kinds of external interference. Auto lock the machine if power/network/wifi/Bluetooth/USB conditions change. Nabbing an unlocked laptop was how they got the Silk Road guy (though they probably already had sufficient evidence elsewhere). https://arstechnica.com/tech-policy/2015/05/sunk-how-ross-ulbricht-ended-up-in-prison-for-life/ https://arstechnica.com/tech-policy/2015/05/sunk-how-ross-ul...
- MertsA 2y ago
  Wifi would probably be the easiest. Either hide a dummy AP in the house or use a combination of multiple neighbors APs. If you don't see any beacon frames from the dummy SSID for a 30 second period then lock/shred the computer.
  - justsomehnguy 2y ago
    Wifi 5/6 sometimes rake up to a couple of minutes to get online (DFS and whatever) so 30 seconds is like smoking near an open can of gasoline: mostly fine but when it's not...
- amluto 2y ago
  Maybe attack the problem from a different angle: use an accelerometer. Or spend a little bit more money to add a gyro and make a real, if very low accuracy, IMU.
  - adgjlsfhk1 2y ago
    Seems like an mems accelerometer would be all you need. Rotation isn't really a threat...
    - amluto 2y ago
      Rotation itself isn’t a threat, but if you want to directly estimate displacement to distinguish between earthquakes and someone stealing the machine, without relying on heuristics, actual inertial measurement would do the trick. And inertial measurement involves tracking the direction of acceleration, which involves tracking rotation.
  - 0cf8612b2e1e 2y ago
    That is a great suggestion. I think Android just implemented a “snatch detection” system for phones. Although, I like the idea of not requiring additional hardware. I guess when I start running a drug empire I will have to pony up for the extra dongle.
  - TeeMassive 2y ago
    That's a great idea. Authorizing any kind of physical change should be a default security measures.
  - TimeBearingDown 2y ago
    BusKill was created for this, USB with a magnetic attachment to a keyring that can be configured to take action on disconnect.
  - mcpherrinm 2y ago
    Some HSMs I've used (payshields) have tamper sensors that can detect motion for this reason. > The ADXL362 accelerometer in the PayShield 10K acts as a "Motion Sensor" detecting tilt movements. An alarm triggers an alert if the HSM is moved (for example, slid out of the rack)
- jeroenhd 2y ago
  One trick you could use is to abuse the fact that law enforcement often plugs in a mouse wiggler on an unlocked desktop and kill your server the moment you see a new HID device (make sure to run some kind of desktop on your server so they think they can keep the session open, best to do it in a VM). You could also monitor the ethernet link. They can move your server but they can't move the entire network, set up an encrypted tunnel between two distant physical servers and self destruct the moment that tunnel gets disrupted. Some computers come with gyros/accelerometers built in. My old HP laptop had some kind of head crash prevention that used that hardware. I know this, because Gnome thought it was a tablet style sensor and turned my screen upside down if I didn't disable the sensor. Maybe getting a HP server can already get you a whole bunch of movement sensors. You could probably figure out if the server is being moved by measuring capacitance of the case, measuring accelerometers, maybe add a GPS dongle. Or you could add an LTE connector and measure any signals you may receive that you shouldn't from inside a server room. You can probably measure _something_ in the server room, though, so to make sure your LTE dongle doesn't get interrupted, also measure whatever reliable signal you can find to detect Faraday cages. Lastly, you could put a video camera in the case on all sides and measure changes. Detecting law enforcement badges probably isn't that hard with opencv if you're dedicated enough. You have to hide your security measures and never tell anyone, though, or they'll just leave the server as-is and use the classic rubber hose exploit to make you give up the key material.
  - tbrownaw 2y ago
    > Or you could add an LTE connector and measure any signals you may receive that you shouldn't from inside a server room. Incoming Bluetooth Low Energy announcements should have a receive power level associated with them. Stick a beacon (like say a standard ble temperature/humidity sensor) somewhere, and you should be able to tell if the distance to it changes.
  - Onavo 2y ago
    [flagged]
  - gosub100 2y ago
    https://github.com/defuse/swatd https://github.com/defuse/swatd
- dullcrisp 2y ago
  If they’re seizing your laptop and your laptop will only work inside your house, wouldn’t they just seize your whole house?
  - lobsterthief 2y ago
    They absolutely would
  - 0cf8612b2e1e 2y ago
    It is a secret one way lock. Disturbing the machine and it locks/encrypts/sheds data. Bringing the machine back to the safe zone would not decrypt the data.
- akira2501 2y ago
  > detect these kinds of external interference. Easily. Bolt the machine to the floor in such a way where the case has to be opened and a trip sensor activated to actually move the machine. You can switch my power source without noticing? Who cares. The attack is taking the machine where it is not supposed to be. That's a problem we've been solving since forever.
the_real_cher 2y ago
So it would emulate a UPS? So they could just remove the existing UPS? what is inject a UPS?
- amiga-workbench 2y ago
  Its a parasitic tap that connects to the mains power cable going into the device. It then phase locks an inverter with said mains power, allowing the mains power cable to be unplugged and the whole lot transported elsewhere on battery power.
  - amelius 2y ago
    How do you reliably get to the copper without shorting it?
    - aaronmdjones 2y ago
      Careful application of a box cutter for the outer sheath followed by something resembling a scotchlok connector for line and neutral. Edit: If the machine is plugged into a power bar / power strip / whatever you want to call it, this is much easier still: Plug the vampire UPS into the power bar as well, wait for it to sync up to the grid, and disconnect the bar from the outlet. The UPS continues to feed power into the bar and thus keeps the machine powered.
      - pests 2y ago
        Power strips make this easier of course, but every outlet usually has two plugs and most* of the time they are wired together. You just need to plug into the other plug. * In case they are split for whatever reason (switched plug, different circuit) whatever, just take off the faceplate, pull out the outlet, and now you have direct access to the screw terminals and copper wiring on the outlet. You could wire into the plug using the second set of terminals or via the other connection method (one being the screw terminals, the other being the "insert into the hole" depending on which is used) and take the whole outlet with you.
cruffle_duffle 2y ago
Isn’t that kinda what they used for Ross Ulbright’s computer? I know it was a laptop but they probably didn’t want to take chances given if that thing shut down the entire thing would be encrypted?
- andrewla 2y ago
  I thought they had an attractive agent distract him for a moment while another agent grabbed his still-unlocked-and-open laptop to prevent him from locking it or closing it up. At least I think that was the cloak-and-dagger story I heard.
  - gosub100 2y ago
    two agents posing as a couple feigned a raucous quarrel that distracted him, while a third agent sitting across the table yanked the laptop at the precise moment he was distracted
    - thebruce87m 2y ago
      And they had to improvise as the cafe he was originally going to visit was too busy and he went to a library instead. Super interesting story!
immibis 2y ago
Someone successfully did this for copper gigabit ethernet and presented at one of the security conferences - but with a few milliseconds interruption in signal.
sschueller 2y ago
That is why you put in special outlets that communicate with the PC over the power line encrypted. You would need to drill holes in the concrete wall to get to the power lines in the wall in order to take the outlet along and hope that there isn't an additional device in the breaker panel.
deleted 2y ago
[deleted]

dathery 2y ago

Really cool article, I enjoy reading through all the details behind the decision making. Just spit-balling a little, but I wonder if Wireguard is the best tool here given that the author is only using it for a single point-to-point link and they control the devices on both ends. That CPU supports AES-NI and probably does it a lot faster than Wireguard's ChaCha20 (hard to get numbers for their server CPU, but the tiny little x86 mini PC I use as my router does AES XTS at 43Gbps according to `cryptsetup benchmark`). You might see better performance by tunneling the vxlan connection using a different technology which can use AES-NI? Then again, Wireguard is definitely still a good tool for stuff like this, and maybe the performance penalty isn't a big deal here.

wmf 2y ago
Because Wireguard is cool and AES is uncool.
- tptacek 2y ago
  I guess it depends on whether you're more concerned about transport security or cipher cycles/byte.
  - dathery 2y ago
    Is there reason to think AES used appropriately would be any less secure here? Not trying to be argumentative, genuinely curious. My understanding is that AES has some design warts that make it not ideal (basically, it's easy to both implement and use in ways that leak information if you're not careful) but that it's still essentially perfect symmetric encryption if you're using it as recommended. Is that wrong? FWIW, the reason I brought up performance was because the OP spends a large chunk of the post talking about it, so I assume it's an important requirement for them.
    - wmf 2y ago
      AES is probably fine as a cipher but the VPN protocols that aren't Wireguard tend to have various footguns available. In theory someone could create NoisyESP but I'm not aware of it.

exabrial 2y ago

Why MACSEC isn't the default is pretty crazy! given that is is extremely stateless (encrypting at the frame level) and counters should be pretty reliable (only go up, since there's two parties) you could take advantages of some AES and GCM modes that would pretty quickly spot injection, replay, and other attacks. But getting back to the main topic of the paper: why not just S2S IPSec the link?

justsomehnguy 2y ago
> Why MACSEC isn't the default TFA explains it pretty well. Also every encryption is adding the load and latency, so defaulting to it when it wasn't asked for isn't the best way > why not just S2S IPSec the link? Because IPSec is still PITA and also sucks bad performance wise against WG.
- nullc 2y ago
  I don't recall the specifics of macsec but it's possible to build a link encryptor that adds essentially zero latency. (like... no more latency than the gate delay of a single xor gate... plus some once-an-hour packet-length delay of some rekeying traffic).

nsteel 2y ago

Did Cisco really invent MACSec?! I thought it was cooked up by the IEEE and supported in hardware from many vendors. I imagine they all have their own bugs though, it's quite a complicated spec. I know some switch/router vendors also now offer hardware-accelerated end-to-end encryption, similar to IPsec, Nokia call their's anysec but I'm sure the other players have their own. The benefit of those is you'd get full bandwidth (e.g. Tbps).

wmf 2y ago
Usually one vendor prototypes a feature then they take it to IEEE/IETF for standardization. Probably half of all network protocols were invented by Cisco.

theideaofcoffee 2y ago

This is a great writeup! Especially for those that may want to DIY it, the how and the why and all of that, and not have to shell out for carrier-quality Layer 1 encryption devices. Nice to see that even off-the-shelf components can do it with relative ease at those rates. Also nice to see sane sysctl tunes as well. Anything to make an adversary's day a bit harder. I low key love the explanation of old 10B5 taps, something that so well and truly dead, but the legacy carries on into everything new today. This is actually a well-trodden area of datacenter interconnect (DCI) devices that do line-rate encryption (to crazy levels like 400G+) to protect those links that may have easily accessible fibers strung along poles, for instance, to prevent just the vampirism described in the post. Packetlight, Ciena, Infinera and others.

aaronmdjones 2y ago

# setup a 8020 MTU on wg0 interface to account for the 80 bytes wireguard headers overhead # 20-byte IPv4 header or 40 byte IPv6 header, 8-byte UDP header 4-byte type, 4-byte key index, 8-byte nonce, 16-byte authentication tag) /sbin/ip li set dev wg0 mtu 8020 Shouldn't that be 8920? To go with the 9000 byte MTU on the outer interface above it.

bb88 2y ago
That would probably add only a smidge more performance.
- aaronmdjones 2y ago
  Sure, but the comment block says that they arrived at 8020 by subtracting 80 from 9000, which is just wrong.

icehawk 2y ago

I did something like this to stretch L2 as I was moving into a new home. Worked great after I realized t-mobile does not like passing IP fragments. Got to use it again to set up a remote telescope for the eclipse.

westurner 2y ago

A notebook with pandas would have had a df.plot(). 9.71 Gbps with wg on a 10GBps link with sysctl tunings, custom MTUs,. I had heard of token ring, but not 10BASE5: https://en.wikipedia.org/wiki/10BASE5 https://en.wikipedia.org/wiki/10BASE5

dietr1ch 2y ago

Missing attack: Cause a disruption that obviously breaks the connection while further away you get time to tap it properly. "Oh, no, a truck run into the pole carrying the copper/fiber, it must be an accident and no intervention is going on undetected because of the outage." What we really need is promiscuous connectivity , but fully untrusted connections. It's maddening why it's hard to communicate 2 wireless devices while they are literally sharing the same radio spectrum and multiple radios could be used to talk to each other.

nullc 2y ago

Tapping is even easier if you have access to the cable end in a patch panel. I have a computer setup with a one-way gige connection for reviewing potentially malicious content in an air-gapped manner. The transmit side transceiver needs to see an incoming signal, so I just use one of these to feed its own output back into it: https://www.amazon.com/dp/B0B8ZHBK26 https://www.amazon.com/dp/B0B8ZHBK26

deleted 2y ago

[deleted]

ggernov 2y ago

I prefer to just run fiber inside of a copper gas line.

sulandor 2y ago
because it's easily available and unconspicuous? asking because copper is neither cheap nor particularly hard to penetrate.

c0l0 2y ago

As I had posted a few weeks ago (https://news.ycombinator.com/item?id=41085314 https://news.ycombinator.com/item?id=41085314), I recently implemented a very similar thing myself. My solution ended up using tc's mirred[0] action for implementing a fully L2-transparent frame relay. I wonder if their setup achieves the same degree of transparency, because afaiui, that's just not possible involving a 802.1Q-compliant (Linux) bridge. I spent close to a week optimizing my setup looking at kernel flame graphs and perf results, reading adapter-specific tuning guides and driver source, and can say that the only really meaningful performance optimizations (in both the Broadwell- and Zen3/Vermeer-based implementations I tried) were disabling mitigations in the kernel (esp. on Zen3, that boosted performance by more than 20%), and getting CPU frequency scaling/idle states sorted out correctly (which yielded much higher wins on the older Broadwell uarch, because power state transition appears to happen much quicker on Zen3). As for the solution presented in the (on the whole really great; I love it!) article, I have my doubts about the effectiveness of the cargo-culted "sysctl tuning" mentioned - TCP, for example, is simply not involved at all in the described setup, so "tuning" its buffer allocations cannot have any effect on the workload. Kudos to the writers for solving their problem in a creative, cost-effective and maintainable way! :) [0]: https://www.man7.org/linux/man-pages/man8/tc-mirred.8.html https://www.man7.org/linux/man-pages/man8/tc-mirred.8.html

sulandor 2y ago
> CPU frequency scaling/idle states sorted out correctly please elaborate
- c0l0 2y ago
  To "fix" performance (i.e., increase throughput by close to 35%) one has to mess with the "energy performance bias" on the (Broadwell) platform, e. g. using x86_energy_perf_policy[0] or cpupower[1]. Otherwise, the CPUs/platform firmware will select to operate in a very dissatisfactory compromise between high-ish power consumption (~90W per socket), but substantially less performance than with having all idle states disabled (= CPU in POLL at all times, resulting in ~135W per core) completely. One can tweak things to reach a sweet spot in the middle, where you can achieve ~99% of the peak performance at very sensible idle power draw (i.e., ~25W when the link isn't loaded). With Zen3, this hardly mattered at all. I also got to witness that using IPv4 for the wireguard "overlay" network yielded about 30% better performance than when using IPv6 with ULA prefixes. [0]: https://man.archlinux.org/man/x86_energy_perf_policy.8 https://man.archlinux.org/man/x86_energy_perf_policy.8 [1]: https://linux.die.net/man/1/cpupower https://linux.die.net/man/1/cpupower

qhwudbebd 2y ago

I'm interested in their last step, in which they set vm.dirty_ratio = 40 vm.dirty_background_ratio = 10 vm.swappiness=10 governor=performance energy_perf_bias=performance min_perf_pct=100 to raise a surprisingly low performance ceiling. I can't believe they were under any memory pressure, so the first three presumably made no difference, but it's also quite surprising to me that the default ondemand cpu governor was responsible for such a dramatic performance hit. Not throttling up quickly enough leading to higher latency maybe? Very interesting anyway.

justsomehnguy 2y ago
> Not throttling up quickly enough leading to higher latency maybe? Quite amusingly it's the load isn't big enough for the default governor to rise the clock up to max. If they posted the cpu load it would be pretty obvious.

mrbluecoat 2y ago

> Eth/IP/UDP/WG/IP/UDP/VXLAN/Eth/802.1q/IP/Payload I'd love to see your IT manager's face when you propose it :D

wmf 2y ago
This is not uncommon. At my last job we had double VXLAN.