6 ms·
Hetzner subsea fibre cut outage
- deleted 4y ago[deleted]
- sbf501 4y agoIs it one big horkin' cable that needs to be spliced by divers? Or is it hundreds of tiny little fibers?
- jaywalk 4y agoIt's one big cable with eight individual fibers. But they aren't fused by divers, the cable will be brought to the surface and fused onboard the repair ship.
- sizzzzlerz 4y agoWhen something like this happens, does the traffic get distributed on other operational fibers that have sufficient unused bandwidth or do the operators fire up some dark fibers to carry the load? Or maybe it’s both?
- yuliyp 4y agoIt depends. Generally the larger carriers and consumers of fibers have a network that is able to route around outages (fiber cuts like this are basically routine) just the same as on land. If they don't have enough, then they'll need to either reduce demand or rent some capacity from someone else.
- solarkraft 4y ago> then they'll need to either reduce demand Worst case everything just gets a bit slower (right?), depending on how much traffic went through that cable.
- deleted 4y ago[deleted]
- hericium 4y ago> Recovery of the service is in progress (Last update: 2022-05-24 05:49 UTC+0) > There is currently an ongoing fibercut at our subsea cable. Our cable provider is working on fixing this. There is currently no ETA. (Last update: 2022-05-23 14:09 UTC+0)
- tinco 4y agoBitten by a shark, or snagged on a russian submarine? Whichever the case, I just ran an ansible deployment over our cluster that's spread over their Frankfurt and Helsinki datacenters, and I didn't notice any difference. I guess that's to be expected, their line between those datacenters is just an optimization, the internet is still functioning as it always is and maybe Hetzner gets a slightly larger egress bill from their ISP as the inter-dc traffic gets rerouted the general internet.
- HL33tibCe7 4y ago> Bitten by a shark, or snagged on a russian submarine? In all likelihood much less interesting, probably snagged in a fishing net or hit by an anchor.
- HatchedLake721 4y agoCan’t rule out Russian military dolphins these days. https://www.theguardian.com/world/2022/apr/27/russia-black-sea-military-dolphins-crimea https://www.theguardian.com/world/2022/apr/27/russia-black-s...
- dtx1 4y ago> Global Pandemic? Check > Transnational Giga Companies controlling the world? Check > Military Dolphins? Check > Smuggling Gigabytes of Data via Implants? Not yet. I rate the current situation as a strong 3/4 Johnny Mnemonics!
- Mo3 4y agoMilitary dolphins...
- fghorow 4y ago"With frickin laser beams!"
- konart 4y ago>Russian >these days https://en.wikipedia.org/wiki/Military_marine_mammal https://en.wikipedia.org/wiki/Military_marine_mammal
- jacooper 4y agoAha! That's why my status server has been acting weird saying things were offline when they weren't.
- snsr 4y agoBest guess this is the cable based on Hetzner investment and landings: https://www.submarinecablemap.com/submarine-cable/c-lion1 https://www.submarinecablemap.com/submarine-cable/c-lion1
- iruoy 4y agoThat seems to be correct. https://www.hetzner.com/presse-berichte/2016/01/156818 https://www.hetzner.com/presse-berichte/2016/01/156818 > Hetzner Online GmbH, the first major Central European provider of data center services to construct a new data center park in Finland, is also a user and co-investor in the Cinia CLion1 cable connection.
- travisporter 4y agoCan they somehow tell exactly where the cable was cut?
- polishdude20 4y agoYeah there's special tech for that! Has something to do with sending some frequency over the line and measuring how long it takes to "reflect" off of the breakage and back to the shore.
- dsr_ 4y agoThe term is time domain reflectometry, TDR. It used to be really expensive, and now it's filtered down into medium-cost optical adapters.
- bostik 4y agoIn the non-optical domain, that used to be a feature in the most expensive cable measuring gears (to the tune of €25k about 20 years ago). I had the pleasure of using one of the devices that had that when I wired my old house. Borrowed the kit from ... an institution that allowed me to get their gear for a weekend. (Yes, I knew people in there.) Having the device tell me that a given strand in a CAT5e cable was faulty at <this many meters> from my location made it surprisingly easy to detect where I'd messed up.
- colanderman 4y agoMany (low-cost) Mikrotik routers have this feature built in now: https://wiki.mikrotik.com/wiki/Manual:Interface/Ethernet#Detect_Cable_Problems https://wiki.mikrotik.com/wiki/Manual:Interface/Ethernet#Det...
- ahartmetz 4y agoI first saw the feature on a motherboard integrathed Ethernet chip over 10, possibly over 15 years ago.
- bombcar 4y agoI was today years old when I learned Hetzner has its own undersea cables.
- daneel_w 4y agoIt's not their cable.
- ArchOversight 4y agoThey are a co-investor... so its partially their cable.
- daneel_w 4y agoAnd as I'm an EU citizen paying top taxes, plenty of which went into funding the project, it's partially my cable as well.
- devoutsalsa 4y agoFix your damn cable @daniel_w!!!
- Trias11 4y agoLol, To be specific - you invested in EU taxes, while others invested in cable.
- OJFord 4y agoBy that logic (that it's not (partially) Hetzner's I mean, not the sarcasm) it's nobody's cable then?
- bombcar 4y agoIf it’s everyone’s cable, it’s no one’s! Syndroooooooome out
- pchristensen 4y agoIf any part of this interests you, you must read "Mother Earth, Motherboard" by Neal Stephenson. It's epic longform from the early Wired. https://www.wired.com/1996/12/ffglass/ https://www.wired.com/1996/12/ffglass/
- carlsborg 4y agoThen you will also like this: https://www.submarinecablemap.com https://www.submarinecablemap.com
- lizardactivist 4y agoThe use of that cable is just to reduce latency. There are many other paths between their datacenters, so there won't be any real breakage.
- lordnacho 4y agoHappens quite often to subsea cables. I was renting a number of them and every now and again you'd get a break. You can follow the NOC's update emails: Break detected Ship has been commissioned Ship is at harbor Ship has sailed out to the cut Fusing the ends Service restored
- evan_ 4y agoWhat's the typical timeline for that cycle? Hours, days, weeks? Just curious. I'm sure it costs an astronomical amount of money.
- lordnacho 4y agoDays, but I had one time where the ships were booked up and it took longer.
- dabeeeenster 4y agoHow the hell do you fuse the ends of a massive cable like this at the bottom of the sea?!
- lordnacho 4y agoYou find the ends, pick them up, and there's a kit that can fuse the ends to your new piece. I'm not sure about the details, but I think there are repeaters on the wire that need to be added as well.
- dataflow 4y agoA repeater on the ocean floor? Do they run power cables to it alongside the data all the way from the coast?
- detaro 4y agoYes, commonly there are high-voltage cables in there. Nowadays, for shorter runs (that are still long enough to need repeaters) there are also purely optical repeaters where instead separate fiber strands deliver light from a pump laser.
- tiffanyh 4y agoDumb question: I thought TCP/IP was designed to route around cut fibre cuts.
- pvg 4y agoThat's pretty much what happens but no amount of routing can make up for the loss of capacity when you physically take out a part of the network. You end up with less total available bandwidth and more latency.
- caymanjim 4y agoIt is, and a cut like this rarely causes a complete outage, but it depends on whether and what kind of backup connectivity your ISP has, and the connectivity of your destination. Bigger entities tend to have more backups. Smaller entities tend to have fewer backups, and often the backups are much lower bandwidth/quality, so it's typical for service to degrade.
- Arnavion 4y agoAlso to be clear, this is handled by BGP, not by TCP/IP. BGP is what ASs use to identify who connects to what so that they can route IP appropriately.
- AtNightWeCode 4y agoDon't know about Hetzner but cloud traffic these days are not routed over the public Internet in that way.
- daneel_w 4y agoThis isn't Hetzner's own cable, and it's not just "any" cable - it's an Alcatel-Lucent + Cinia + EU project, with 150 terabits worth of bandwidth. It's a high profile target, but it's just as likely that it's just an anchor accident.
- ArchOversight 4y agoAs Hetzner is a co-investor in the cable, it is partially their cable.
- R0b0t1 4y agoSo do they own a solid chunk of it somewhere, or like a really long pie-slice sliver down the whole thing?
- k__ 4y agoSince it's a cable, I would imagine the latter.
- Denvercoder9 4y agoNeither, it's shared ownership. It's similar to if you buy a house with your partner: neither of you owns specific rooms, but you own the whole thing together. There's likely agreements about which fibers (there's 8 fiber pairs in the cable) and/or how much bandwidth everyone can use, though.
- foepys 4y agoWhy only 8? Why don't they put like 100 in there?
- cure 4y agoPresumably because the cable would be (much) heavier and thicker. There is some movement towards having more pairs in newer cables, e.g. the Grace Hopper will have 16 pairs, cf. https://cloud.google.com/blog/products/infrastructure/announcing-googles-grace-hopper-subsea-cable-system https://cloud.google.com/blog/products/infrastructure/announ...
- caymanjim 4y agoWhen I lived in Cayman, the main fiber link (Maya-1[0]) was down for about a week (anchor drag, I think). Most of the island Caribbean has limited backup connectivity. The whole island only has one fiber line (around the whole western/southern Gulf of Mexico) and a crappy backup copper line to Jamaica. For a while, it was like being on a modem again. Forget about streaming anything. This outage Hetzner is reporting isn't a big deal due to robust backups, but these things can cripple a small island outpost. 0: https://www.submarinecablemap.com/submarine-cable/maya-1 https://www.submarinecablemap.com/submarine-cable/maya-1
- user3939382 4y ago> For a while, it was like being on a modem again Internet in Congo if you're not on the UN satellite
- davidhariri 4y agoIf, like me, you didn't know how these get cut I'll save you the trouble. The most likely cause is improper boat anchoring. Never shark bites.
- gmcquistin 4y agoSharks do bite these cables! https://amp.theguardian.com/technology/2014/aug/14/google-undersea-fibre-optic-cables-shark-attacks https://amp.theguardian.com/technology/2014/aug/14/google-un...