5 ms·

It's actually interesting why this idea is so totally wrong. So Telecom Malaysia messed up a config, and Global Crossing accepted their updates automatically.

It's actually interesting why this idea is so totally wrong. So Telecom Malaysia messed up a config, and Global Crossing accepted their updates automatically. Global Crossing didn't have to accept the bad update. They generally trust updates from other organisations that are generally trustworthy. They apply checks and restrictions proportionate to the risks involved. These mistakes happen rarely. If they were to happen more often, major operators would apply more checks and restrictions. If they were to stop happening, operators would apply less checks and restrictions, because they have a cost in manpower, complexity, and loss of flexibility. That's how the internet works. You could almost say that's what the internet is--the idea of being actively managed by people who know what they're doing and are not bound by exhaustive predefined policies is defining of how the internet came about and how it came to be dominant. If you want a network guaranteed to be resistant to this kind of f---up, build one. The internet is that network which does not work that way, which is flexible, expandable, mostly "good enough" but not ever designed for absolute reliability.

ryanlol 11y ago
This isn't about how the internet works. This is someone messing up and breaking the internet. If I'm driving a car without paying attention to the road and kill a bunch of kids, that's my fault. If Telecom Malaysia pushes new configs without testing them and breaks the internet, that's their fault. Obviously these two things aren't even remotely comparable in seriousness, but it's clear that in both cases the people messing up should be held responsible.
- Robin_Message 11y ago
  Excuse me Mr Lol, but since it happened, it would appear that it in fact is how the internet works. If the postal service misdelivers some mail are there criminal charges? I would suggest resisting the urge to hit every problem and mistake in the world with the law hammer; it is rarely productive, and generally isn't how the world works.
  - ryanlol 11y ago
    I never said that it isn't how the internet works? Your comparison to postal service misdelivering some mail is both irrelevant and utterly ridiculous. A relevant comparison would be someone disrupting mail delivery on a global scale. Which would in fact be a crime in quite a few countries. For example in the US: https://www.law.cornell.edu/uscode/text/18/1701 https://www.law.cornell.edu/uscode/text/18/1701 https://www.law.cornell.edu/uscode/text/18/1706 https://www.law.cornell.edu/uscode/text/18/1706 https://www.law.cornell.edu/uscode/text/18/1702 https://www.law.cornell.edu/uscode/text/18/1702 https://www.law.cornell.edu/uscode/text/18/1700 https://www.law.cornell.edu/uscode/text/18/1700 E: (Yeah, downvote me because you disagree. Instead of explaining why I'm wrong)
    - mfoy_ 11y ago
      You are wrong because this IS about how the Internet works and less about how one person/company screwed up. Apparently all it takes is one ISP misconfiguring something to break large swathes of the Internet. I believe the consensus on HN is that no one entity should have an Internet kill switch. If someone managed to disrupt mail delivery on a global scale, people would be less concerned with THAT it happened than that it COULD HAVE happened in the first place. Why would global mail delivery be so not-fault-tolerant that one mistake brought it to a grinding halt for hours? Same deal here.
      - ryanlol 11y ago
        I don't think anyone is saying that the fact that this CAN happen isn't a huge problem, it most definitely is. Thing is, everybody knows (and has always known) this can happen. Everybody also knows how to avoid it. Well-intentioned people tend to try to avoid breaking the internet. That really doesn't make the negligence of Telecom Malaysia any more defensible. I really don't see what's the point of defending Telecom Malaysia, a plenty of people manage to operate their equipment in a manner that doesn't break the internet. Just because this was a mistake doesn't mean they should not be held responsible (and no, I'm not saying someone should go to prison.)
tomjen3 11y ago
>It's actually interesting why this idea is so totally wrong. No it is not. You accept their claims of "mistakes" I see no evidence of that - how, exactly, do you leak a full table by accident? and this is too big a security hole to leave to hackers. Leak a bunch of table, shut down an entire country.
- ryanlol 11y ago
  Even if was a mistake, that doesn't suddenly make it OK. People make mistakes, yeah. But this isn't a simple mistake, in fact this incident consists of multiple mistakes. 1) Someone wrote an incorrect config 2) They did not test it 3) They pushed it to production systems without testing it 4) They did not monitor their systems after pushing new configs 5) They took ages to fix the problem after it was detected. That definitely isn't a single mistake.
  - laumars 11y ago
    And how would you propose they test it? It's a little difficult to test this kind of config without emulating the entire internet - which is quite clearly beyond the scope of all bar a very small number of organisations.
- spydum 11y ago
  full table leaks happen ALL the time. The reasons you dont notice or hear about it is: - the providers which do this by accident are too small (multiple asn hops away from a major transit provider) to become the best choice for most people - the small guy who does leak the full table to a major transit provider, is adequately filtered by the major transit provider by default - the small guy who leaks to the medium transit provider might take an outage, but may not leak to his upstreams due to outbound filtering or the upstreams filtering you would be surprised, BGP is an old protocol, has had very little serious security improvements. It currently works more or less based upon the goodwill and discipline of network engineers around the world, because if they screw it up, they usually end up offline and out of a job.
MichaelGG 11y ago
If anything, the outrage should be directed at Global Crossing/L3. They shouldn't be allowing some little ISP to screw up their route tables like that. Because this means that the Malaysian government could secretly announce just a few blocks, route them back out so they still work, and easily do MITM. OTOH, so much in telecom is hung together on the assumption of basically good actors everywhere.

ryanlol 11y ago

This isn't about how the internet works. This is someone messing up and breaking the internet. If I'm driving a car without paying attention to the road and kill a bunch of kids, that's my fault. If Telecom Malaysia pushes new configs without testing them and breaks the internet, that's their fault. Obviously these two things aren't even remotely comparable in seriousness, but it's clear that in both cases the people messing up should be held responsible.

Robin_Message 11y ago
Excuse me Mr Lol, but since it happened, it would appear that it in fact is how the internet works. If the postal service misdelivers some mail are there criminal charges? I would suggest resisting the urge to hit every problem and mistake in the world with the law hammer; it is rarely productive, and generally isn't how the world works.
- ryanlol 11y ago
  I never said that it isn't how the internet works? Your comparison to postal service misdelivering some mail is both irrelevant and utterly ridiculous. A relevant comparison would be someone disrupting mail delivery on a global scale. Which would in fact be a crime in quite a few countries. For example in the US: https://www.law.cornell.edu/uscode/text/18/1701 https://www.law.cornell.edu/uscode/text/18/1701 https://www.law.cornell.edu/uscode/text/18/1706 https://www.law.cornell.edu/uscode/text/18/1706 https://www.law.cornell.edu/uscode/text/18/1702 https://www.law.cornell.edu/uscode/text/18/1702 https://www.law.cornell.edu/uscode/text/18/1700 https://www.law.cornell.edu/uscode/text/18/1700 E: (Yeah, downvote me because you disagree. Instead of explaining why I'm wrong)
  - mfoy_ 11y ago
    You are wrong because this IS about how the Internet works and less about how one person/company screwed up. Apparently all it takes is one ISP misconfiguring something to break large swathes of the Internet. I believe the consensus on HN is that no one entity should have an Internet kill switch. If someone managed to disrupt mail delivery on a global scale, people would be less concerned with THAT it happened than that it COULD HAVE happened in the first place. Why would global mail delivery be so not-fault-tolerant that one mistake brought it to a grinding halt for hours? Same deal here.
    - ryanlol 11y ago
      I don't think anyone is saying that the fact that this CAN happen isn't a huge problem, it most definitely is. Thing is, everybody knows (and has always known) this can happen. Everybody also knows how to avoid it. Well-intentioned people tend to try to avoid breaking the internet. That really doesn't make the negligence of Telecom Malaysia any more defensible. I really don't see what's the point of defending Telecom Malaysia, a plenty of people manage to operate their equipment in a manner that doesn't break the internet. Just because this was a mistake doesn't mean they should not be held responsible (and no, I'm not saying someone should go to prison.)

tomjen3 11y ago

>It's actually interesting why this idea is so totally wrong. No it is not. You accept their claims of "mistakes" I see no evidence of that - how, exactly, do you leak a full table by accident? and this is too big a security hole to leave to hackers. Leak a bunch of table, shut down an entire country.

ryanlol 11y ago
Even if was a mistake, that doesn't suddenly make it OK. People make mistakes, yeah. But this isn't a simple mistake, in fact this incident consists of multiple mistakes. 1) Someone wrote an incorrect config 2) They did not test it 3) They pushed it to production systems without testing it 4) They did not monitor their systems after pushing new configs 5) They took ages to fix the problem after it was detected. That definitely isn't a single mistake.
- laumars 11y ago
  And how would you propose they test it? It's a little difficult to test this kind of config without emulating the entire internet - which is quite clearly beyond the scope of all bar a very small number of organisations.
spydum 11y ago
full table leaks happen ALL the time. The reasons you dont notice or hear about it is: - the providers which do this by accident are too small (multiple asn hops away from a major transit provider) to become the best choice for most people - the small guy who does leak the full table to a major transit provider, is adequately filtered by the major transit provider by default - the small guy who leaks to the medium transit provider might take an outage, but may not leak to his upstreams due to outbound filtering or the upstreams filtering you would be surprised, BGP is an old protocol, has had very little serious security improvements. It currently works more or less based upon the goodwill and discipline of network engineers around the world, because if they screw it up, they usually end up offline and out of a job.

MichaelGG 11y ago

If anything, the outrage should be directed at Global Crossing/L3. They shouldn't be allowing some little ISP to screw up their route tables like that. Because this means that the Malaysian government could secretly announce just a few blocks, route them back out so they still work, and easily do MITM. OTOH, so much in telecom is hung together on the assumption of basically good actors everywhere.