10 ms·

Observations of an Internet Middleman

424 points by staticsafe 12y ago

wdewind 12y ago
Here's the part that doesn't make sense to me, and hopefully someone can explain it: Netflix pays Level 3 and Cogent to connect them to Comcast's network. Comcast claims that only the Level 3 connection is saturated, and that Netflix is sending all their bandwidth over Level 3 because it's cheaper for them. Doesn't Level 3 buy a contract from Comcast that says "we get to send this much data per month"? If Netflix (or Level 3) tries to push more data through that, which causes congestion, it seems like a contractual fact that Comcast is either holding up their end of the bargain or not. If that's true, and assuming that Comcast is transferring the contractually agreed upon part, isn't this actually Netflix and Level 3's fault? Isn't it reasonable to assume that Netflix would need to either use another entry point into Comcast's network, or build one? If Netflix/Level 3 congest one entry point, so movies stream slowly for Comcast users, it doesn't seem like Comcast is being unneutral, in the sense of packet inspection and routing based on content/source. The problem seems upstream from the last mile networks, and it's very unclear to me whose fault it is, and if this even has anything to do with net neutrality at all (it doesn't seem to). Please correct me where I'm wrong, I'm sure I am!
- exelius 12y ago
  I think Level3's issue is that Comcast is unwilling to increase the capacity of their links without payment from Level3. If Level3 engages in settlement-free peering with Comcast and their network traffic graph looks anything like the 100gbit link in the article, I don't blame them: it's a 20:1 disparity in traffic in vs traffic out. What's happening here is that Level3 and other transit providers are starting to see their industry be squeezed by the big ISPs. If you have 5 ISPs that serve 90% of the customers in the US, and 10 service providers that generate 90% of the bandwidth, why do transit providers even exist? I think they see their market share decreasing significantly as more of their customers follow in Netflix's footsteps, so they're trying to pile on and start a grass-roots outrage like Netflix did. Problem there is that most people haven't heard of Level3, so it makes it more difficult to get people behind them. And you are completely right about the whole Netflix issue being Netflix and Level3/Cogent's problem. Level3 and Cogent have business models based on selling transit acquired through settlement-free peering. Settlement-free peering assumes that bandwidth usage is roughly symmetric: when it's not symmetric, the side sending more bandwidth has to pay. Level3's position here seems a bit hypocritical: if they had a customer that was routinely sending data at a 20:1 ratio, they would charge that customer for sending more bandwidth. But they're expecting the big ISPs not to do the same to them?
  - dllthomas 12y ago
    "Settlement-free peering assumes that bandwidth usage is roughly symmetric: when it's not symmetric, the side sending more bandwidth has to pay." I've never understood why this is - especially in a case like this, where the sending side is sending because (the customers of) the receiving side explicitly asked for it.
    - exelius 12y ago
      Because the cost of a packet is borne by the receiver while the sender is the one monetizing it. The assumption is that if you're sending traffic to a customer, you are being paid for that activity through subscription fees, ads, donations, etc. That packet costs the receiver some minuscule amount to process. Even though it was requested by the ultimate receiver, the ultimate sender is only sending the packet because they make money from doing so. The middlemen only pass on the packet because their costs are covered and they make a small amount of profit. This setup also ensures that if the packets aren't being monetized in some way (i.e. if it's a DDoS) that there's a financial incentive to stop them.
      - wmf 12y ago
        Because the cost of a packet is borne by the receiver... That's true for hot-potato routing, but CDNs generally use cold-potato routing. As for monetization, both Netflix and Comcast are getting paid by their respective customers.
        exelius 12y ago
        It's no less true for cold-potato routing; the source is just closer to the endpoint. And CDNs doing cold-potato routing happily pay for their transit because the service THEY make money off of is providing CDN services to their customers, who theoretically make money off the content they pay the CDNs to distribute. What it comes down to is Comcast has no incentive to ensure its routes to various Internet transit providers are GOOD if they're not monetizing them. Why should Comcast be treated differently than any other transit provider and not allowed to monetize its transit services? Netflix wasn't paying Comcast, so Comcast had no responsibility to help Netflix make its own service better when there were other ways Netflix could have alleviated the bandwidth situation (e.g. buy transit through another company).
- preinheimer 12y ago
  I think your issue is here: "Doesn't Level 3 buy a contract from Comcast that says "we get to send this much data per month"? If Netflix (or Level 3) tries to push more data through that, which causes congestion, it seems like a contractual fact that Comcast is either holding up their end of the bargain or not." Level 3 doesn't buy the connection from Comcast. Comcast buys a connection from Level 3. Comcast needs to do this because they're selling a connection to the "Internet" to their customers, not a connection to the servers Comcast owns, and to other Comcast customers. Comcast's paying customers are requesting data from the Internet, data that Comcast doesn't have on its network. To get there those bits travel over Level 3's (or Cogent, or another major provider) network. The issue at hand is not one of Level 3 (or Netflix) trying to PUSH data onto Comcast's network. It is one of Comcast's customers trying to PULL data from Level 3, over a connection that Comcast refuses to upgrade.
  - exelius 12y ago
    This would all be fine and good if Level3 didn't make money from selling bandwidth. Comcast doesn't have to buy a connection from Level3; Level3 will give Comcast a link for free because they know they can turn around and charge their own customers for access to it. Comcast is really just cutting out the middleman here; and that scares Level3.
    - mixologic 12y ago
      But they cant cut out the middleman. Comcast doesnt have a direct connection to netflix or anywhere else on the internet. I think Level 3 should straight up shut off their peering agreement with comcast and watch how all the last mile customers blame comcast for the issue, until such time as comcast stops playing games.
      - exelius 12y ago
        But the Internet doesn't work that way: if Level3 shut off their peering with Comcast, Level3's customers would just find another transit provider and route around L3 because those companies are still in business to make money. They don't care about squabbles between Comcast and L3; they just want their monetized traffic to get delivered. The last-mile customers probably wouldn't even notice; but Level3's customers would likely be forced to pay more for transit from another company. But they would still pay.
- cschroed 12y ago
  Comcast promises a certain amount of download capability. When customer requests for content that originate from Comcast's network (that would not normally exceed that download capability) end up saturating a link to which Comcast is party to, then it is Comcast's obligation to attempt to upgrade that link. Comcast cannot control the routes via which the content is returned, they must react to their customers usage patterns to improve the performance of their network. So if tomorrow, everyone wants to download content from some other peer then Comcast will need to upgrade other peer connections. The exception to this is if Comcast wants to charge rent to the internet to have access to Comcast customers. Then it is in Comcast's corporate interest to let peer connections saturate and then charge upstream content providers rent to remove the congestion. A pretty ballsy move for a company (who's assets are stretched out over thousands of miles and in plain sight) to attempt to extort the entire planet. ;)
  - mixologic 12y ago
    Not very ballsy if they've bought enough politicians and lobbyists.
    - josephschmoe 12y ago
      He means it's ballsy because any one of us could just start cutting their physical lines.
  - jessaustin 12y ago
    ...who's assets are stretched out over thousands of miles and in plain sight... I like the cut of your jib.
- johnjac 12y ago
  My understanding is this. No Money exchanges hands between Level3 and Comcast, but there is an agreement for Comcast to provide a port for Level3 to connect to. Level3 brings the traffic to Comcast door Step. Level3 pays for the port on their side. Level3 is happy to add more ports, BUT Comcast Refuses. Not because more ports are that expensive but those in charge at Comcast are from the TV side of the house where blackouts are common negotiating tactics in order to be paid. And Level3 refuse to be held over a barrel for payment when Comcast has a local monopoly power.
  - deleted 12y ago
    [deleted]
- cortesoft 12y ago
  I think a big problem with your reasoning centers on word choice; you phrased it as Level 3 having a contract that says "we get to send this much data per month", and then Level 3 attempting to send more data than that agreement. However, you must remember that every single packet that Level 3 sends to the Comcast network is a packet that was requested BY A COMCAST CUSTOMER. Level 3 isn't just deciding to send a bunch of traffic over Comcasts network, they are sending the data requested by Comcast customers to that network. This is an important distinction, and is usually part of all of these 'settlement-free' peering arrangements. In agreeing to peer with Level 3, I am sure Comcast has an agreement that they will not send any traffic to Comcast's network that is not actually destined for a Comcast customer. Now, if Level 3 were sending traffic to Comcast and expecting Comcast to route, for free, that traffic to another 3rd part network, that would NOT be kosher, and Comcast could fairly ask for either money or for Level 3 to stop sending that traffic to them. Level 3 is ONLY sending the traffic to Comcast that Comcast is requesting; it is THEIR customers who are creating this demand for Level 3's customer's content. Comcast told their customers that for $X per month, they would get Y amount of internet bandwidth, which will in almost all cases be traffic originating from a non-Comcast network; that is just the nature of the internet. Even though Level 3 is willing to send the data that Comcast's customers want to the Comcast network for free (settlement free), Comcast is instead refusing to accept all the traffic that their own customers are requesting and demanding additional payment to provide the service they already sold and are paid for.
  - peterwwillis 12y ago
    "Level 3 is ONLY sending the traffic to Comcast that Comcast is requesting; it is THEIR customers who are creating this demand for Level 3's customer's content." This still misses the mark. Netflix creates a service on Level3 that Comcast customers want. So Netflix/Level3 are responsible for making a product that people want to use, and Comcast customers are responsible for wanting to use that product and increasing bandwidth use. Neither Level3 nor Comcast want to pay more money just because Netflix and Comcast customers want to use more tubes, so both Level3 and Comcast wash their hands of any responsibility for the networks they both run. In a very real sense, both Level3 and Comcast are equally responsible for the increase in traffic [in this case]. Yet neither seem to accept their share in the cost. If it's because it's hard for Level3 to prove, they should put up some money to shore up the connection, and then write a public letter to Comcast customers telling them how Comcast is shorting them and to get angry. (Sadly, many customers have no alternative provider, so their only option would be to quit the internet, and nobody's going to do that)
- zero_intp 12y ago
  MSO revenue is threatened by alternate entertainment content, and is terrified of becoming a dumb pipe with no add-on fees. The issue is that Comcast wants to both connect to the internet for free & get paid by both the viewer and the content provider as they are in yesterday's cable model. Comcast is connected to the internet through 'settlement-free' peering. Comcast can command this because they have grown large enough to trade access to their customer base for access to the transport network's data. Comcast would like to sell direct access to their network to the content providers, but would like to see more money for the service that the content providers are willing to pay. Comcast has allowed (through inaction of normal peering upgrades) the interconnections to Level-3/Cogent to become saturated. The primary data on those interconnections is data that competes with the normal content that Comcast is used to delivering to their customers. Cogent and Level-3 offer less expensive bandwidth than their competitors. This is because of their lack of incumbent telephony models, and being dedicated fiber bit pushers. Comcast is claiming that it is too expensive for them to deliver the service levels to their customers that their customers have contracted. Comcast is demanding that the companies providing the data (not the transit network delivering the data) must pay to have their traffic delivered the last mile. Simultaneously to crying that their internal networks are saturated and require upgrade, the MSOs are offering products that allow their customers to use the local provider's network (for locally sourced content) without incurring billing penalties or usage caps. (veteran employee of a transit network involved)
_stephan 12y ago
Does anyone have an idea what the European ISP is that refuses to provide enough peering capacity?
- revelation 12y ago
  They said its not from the UK and judging from their connection points, I guess its Deutsche Telekom.
  - josephlord 12y ago
    Don't know they described them as a monopoly position and cable is pretty strong in Germany. I think Spain or Italy might be better guesses. Telefonica in Spain might be a reasonable guess. Ono the cable company isn't huge. I think Telecom Italian is dominant in Italy and there is virtually no cable at all so that would be my guess.
    - josephlord 12y ago
      Too late to fix typo: Telecom Italia (not Italian).
  - frik 12y ago
    Yes, it seems it's the "Telekom" from Germany: * http://www.golem.de/news/level-3-sechs-grosse-internetprovider-bremsen-peering-absichtlich-aus-1405-106318.html http://www.golem.de/news/level-3-sechs-grosse-internetprovid... * http://www.welt.de/wirtschaft/article117963991/Bei-Europas-Telekom-Riesen-ist-die-EU-unpatriotisch.html http://www.welt.de/wirtschaft/article117963991/Bei-Europas-T...
- seszett 12y ago
  I would have guessed it was French ISP Proxad/Free, which is notorious for having generally shitty peering to a large part of Internet (like Youtube, Imgur or Github, among others). But though they are a large ISP they are not dominant, and their peering with Level3 is actually quite good... which is why I tunnel most of my connection to my server (the route to which happens to be through Level3) so I can get usable internet. It really sucks to be able to download an iso from some server in the US at 2 MB/s, but seeing small Imgur gifs load frame by frame, or Github repos cloning at 20 kB/s.
deleted 12y ago
[deleted]
josephlord 12y ago
Summary of situation Comcast cripples customers internet services to extort money from companies connecting Comcast to the internet for free.
- LandoCalrissian 12y ago
  It's completely crazy and just blatant extortion on Comcast's part. The current system is pretty straight forward. Level 3: Server(Who Level 3 Charges) -> Level 3 -> Comcast -> End User Comcast: End User(Who Comcast Charges) -> Comcast -> Level 3 -> Server Now Comcast wants to charge everyone in the chain that isn't them. They are attempting to do this by degrading (or at the very least not upgrading) their services for their customer. The techno-libertarians that try to defend this nonsense are going to sink the whole ship and make every pay to receive even less.
  - erichocean 12y ago
    Ahem. "Techno-libertarians" aren't the ones who gave Comcast its monopolies on the last mile. Please leave them out of this.
    - nullc 12y ago
      No, physics and economics did. When you're talking about building infrastructure that will take 20 years to pay itself off _absent_ competition, only the most profitable of the profitable locations can afford extensive physical plant competition. In communications networks monopolies are natural, especially in last mile networks, and will exist absent any government action. That in some cases some governments have mildly strengthened these monopolies in exchange for commitments to cover otherwise unprofitable areas, its not really that important this subject.
    - wpietri 12y ago
      They have been a strong voice for less regulation, especially less regulation of technology. We reined in AT&T through regulation, forcing them to share their last-mile plant with others. We could (and should) have done the same with monopoly cable providers, but the anti-regulation fever in the US prevented that.
deleted 12y ago
[deleted]
johngalt 12y ago
I enjoy seeing those MRTG/RRDtool graphs everywhere. I find it so surreal that $X Billions in infrastructure, that is forwarding $Y Billion of internet traffic. All monitored by a single tool created by one guy from Switzerland. Since this is HN: yes I realize that there has been substantial work subsequently, and that cacti/munin/nagios etc.. are more commonly used.
- zero_intp 12y ago
  It's crazy to think that Rancid, MRTG, & RRD are all fundamentally underpinning of all of these major carriers.
  - NDizzle 12y ago
    It's not so crazy when you consider a huge portion of peering agreements took place over IRC.
    - voltagex_ 12y ago
      Got any articles on this?
  - rhizome 12y ago
    What's crazy about it?
    - zero_intp 12y ago
      quote from the author, "rancid is named rancid because it stinks." it covers 80% of what you want it to do, but is indispensable because it is so lightweight and functional. more developed tools exist but networks are not interested (by in large) in integrating/developing new ones because they fool themselves into thinking these are good enough.
- dfc 12y ago
  I am surprised that the hobby project by the funny sounding guy from Finland did not mentally prepare you for this.
cs702 12y ago
In short: if you're in the US, you're paying your ISP for a certain amount of bandwidth, but your ISP is not giving it to you, because its connections to middlemen like Level 3 are maxed out. Level 3 proposes to split the cost of expanding those connections (as is common), but your ISP refuses unless it gets additional payment from Level 3, Netflix, or someone else. Meanwhile, you don't get the bandwidth you've purchased from your ISP. That doesn't seem right to me.
- andruby 12y ago
  For once, I am glad to be living in a densely populated European country. We can easily switch ISP's if one decides to pull these kinds of dirty tricks.
  - DrJokepu 12y ago
    Well, except that the local telco (e.g. BT in the UK) almost certainly has monopoly over the last mile of your connection, regardless of who your ISP is (they rent the last mile from e.g. BT Openreach or maybe TalkTalk) and it's not uncommon for them to refuse to fix congestions at their exchanges (or even admit their presence) for extended periods. So, you know, it's not like it's all utopistic here or something.
    - estel 12y ago
      Don't Virgin use their own last mile for fibre?
      - geocar 12y ago
        Virgin offer a "cable" service which is available where they have cable television (not a wide distribution area), and that this is what they refer to as their fibre service. Virgin also resell the same ADSL that everyone else does (complete with BT line rental).
      - __alexs 12y ago
        If by their own you mean one of numerous other cable companies that they purchased then yes. There used to be a plethora of small cable companies in the UK. Now there is only Virgin. The widespread availability of high quality ADSL does seem to do a good job of keeping them honest though. I've been a pretty satisfied customer of them for many years now.
MaysonL 12y ago
Level 3's filing with the FCC about this: http://www.level3.com/~/media/Assets/legal/l3_openinternet_march2014.pdf http://www.level3.com/~/media/Assets/legal/l3_openinternet_m...
jeremycole 12y ago
If Comcast doesn't like the amount of bandwidth their customers use watching Netflix, they always have the option to notify their customers and null-route or otherwise stop Netflix from working at all. However while it "works" but sucks they appear to be doing their jobs while their customers blame Netflix for sucking while largely being unaware that it's actually Comcast that sucks.
sergiosgc 12y ago
If this is a game of chicken, Level 3 and Cogent should play it to the end game. They are powerful enough to get Comcast and the other ISPs to concede. Just null route any interconnect that consistently drops packets and fails to get negotiated for an upgrade. The catch? It must be done by both L3 and Cogent. Comcast is playing the monopoly card against what is an oligopoly of backbone providers. Comcast isn't strategically as well positioned as their actions would indicate. It is one thing for Comcast to have degraded Netflix service. It is a different thing to present their customers with access to only half of the Internet .
- DontGiveTwoFlux 12y ago
  Except that it would probably violate their contracts to just stop services at an interconnect. It's not like Level 3 is losing customers or money from the congested interconnects. Their incentive to play hardball just isn't there.
  - zero_intp 12y ago
    Level3 and Cogent both bill the majority (by traffic volume) of their network consumers by utilization. This means that these congested pipes are actually costing potential revenue. Obviously Cogent would be willing to cut off their own nose to spite their face, but Level3 plays a much more sane game. Hardball by coordinating efforts of the backbone providers is somewhat thwarted by the coordinated efforts of the MSO's, some with their own backbones (VZ, AT&T). Cutting off Comcast would also invalidate a number of (government) customer contracts that require service 'to the full Internet'.
  - sergiosgc 12y ago
    If it violates their contract, doesn't a soft fail violate it also? Or will violate, whenever the rate of dropped packets goes above a given threshold. I view the hard fail as just speeding up the interconnect failure, with the positive effect of prompting action from the last mile ISP. Whenever their performance drops towards part of the internet, their value proposition gets lower. You can view the value chain as (residential customer) <-> Comcast <-> Level 3 <-> (hosting customer) Both Comcast and Level 3 have customers, in opposite ends of the currently centralized internet. The status quo is that each provider charges to the customers on their end. Comcast (and each of the other non-net-neutral ISPs) are eyeing the revenue stream on the other end of the pipeline. This revenue stream has historically not been owned by the last-mile ISP. Level 3 and equivalents, by cutting off non-performant interconnects will hurt their own customers, of course. However, they are just speeding up the same effect being applied softly over time, by Comcast and equivalents. A softly failing interconnect moves revenue over to last mile ISPs, thus permanently lowering Level 3's value proposition. By failing the interconnect sharply, instead of softly, they actually improve the steady state of the system, in general (good side effect), and in particular (the real objective).
chrisBob 12y ago
If you are experiencing a slow connection and packet loss, what is the best tool for identifying the issue? There must be a good way to find out where the problem is.
- wpietri 12y ago
  I use mtr. But reading it takes experience, so for a month or so just fire it up everywhere you go, especially when things are working well.
  - chrisBob 12y ago
    Thanks. MTR looks interesting.

424 points by staticsafe 12y ago

Here's the part that doesn't make sense to me, and hopefully someone can explain it: Netflix pays Level 3 and Cogent to connect them to Comcast's network. Comcast claims that only the Level 3 connection is saturated, and that Netflix is sending all their bandwidth over Level 3 because it's cheaper for them. Doesn't Level 3 buy a contract from Comcast that says "we get to send this much data per month"? If Netflix (or Level 3) tries to push more data through that, which causes congestion, it seems like a contractual fact that Comcast is either holding up their end of the bargain or not. If that's true, and assuming that Comcast is transferring the contractually agreed upon part, isn't this actually Netflix and Level 3's fault? Isn't it reasonable to assume that Netflix would need to either use another entry point into Comcast's network, or build one? If Netflix/Level 3 congest one entry point, so movies stream slowly for Comcast users, it doesn't seem like Comcast is being unneutral, in the sense of packet inspection and routing based on content/source. The problem seems upstream from the last mile networks, and it's very unclear to me whose fault it is, and if this even has anything to do with net neutrality at all (it doesn't seem to). Please correct me where I'm wrong, I'm sure I am!

exelius 12y ago
I think Level3's issue is that Comcast is unwilling to increase the capacity of their links without payment from Level3. If Level3 engages in settlement-free peering with Comcast and their network traffic graph looks anything like the 100gbit link in the article, I don't blame them: it's a 20:1 disparity in traffic in vs traffic out. What's happening here is that Level3 and other transit providers are starting to see their industry be squeezed by the big ISPs. If you have 5 ISPs that serve 90% of the customers in the US, and 10 service providers that generate 90% of the bandwidth, why do transit providers even exist? I think they see their market share decreasing significantly as more of their customers follow in Netflix's footsteps, so they're trying to pile on and start a grass-roots outrage like Netflix did. Problem there is that most people haven't heard of Level3, so it makes it more difficult to get people behind them. And you are completely right about the whole Netflix issue being Netflix and Level3/Cogent's problem. Level3 and Cogent have business models based on selling transit acquired through settlement-free peering. Settlement-free peering assumes that bandwidth usage is roughly symmetric: when it's not symmetric, the side sending more bandwidth has to pay. Level3's position here seems a bit hypocritical: if they had a customer that was routinely sending data at a 20:1 ratio, they would charge that customer for sending more bandwidth. But they're expecting the big ISPs not to do the same to them?
- dllthomas 12y ago
  "Settlement-free peering assumes that bandwidth usage is roughly symmetric: when it's not symmetric, the side sending more bandwidth has to pay." I've never understood why this is - especially in a case like this, where the sending side is sending because (the customers of) the receiving side explicitly asked for it.
  - exelius 12y ago
    Because the cost of a packet is borne by the receiver while the sender is the one monetizing it. The assumption is that if you're sending traffic to a customer, you are being paid for that activity through subscription fees, ads, donations, etc. That packet costs the receiver some minuscule amount to process. Even though it was requested by the ultimate receiver, the ultimate sender is only sending the packet because they make money from doing so. The middlemen only pass on the packet because their costs are covered and they make a small amount of profit. This setup also ensures that if the packets aren't being monetized in some way (i.e. if it's a DDoS) that there's a financial incentive to stop them.
    - wmf 12y ago
      Because the cost of a packet is borne by the receiver... That's true for hot-potato routing, but CDNs generally use cold-potato routing. As for monetization, both Netflix and Comcast are getting paid by their respective customers.
      - exelius 12y ago
        It's no less true for cold-potato routing; the source is just closer to the endpoint. And CDNs doing cold-potato routing happily pay for their transit because the service THEY make money off of is providing CDN services to their customers, who theoretically make money off the content they pay the CDNs to distribute. What it comes down to is Comcast has no incentive to ensure its routes to various Internet transit providers are GOOD if they're not monetizing them. Why should Comcast be treated differently than any other transit provider and not allowed to monetize its transit services? Netflix wasn't paying Comcast, so Comcast had no responsibility to help Netflix make its own service better when there were other ways Netflix could have alleviated the bandwidth situation (e.g. buy transit through another company).
preinheimer 12y ago
I think your issue is here: "Doesn't Level 3 buy a contract from Comcast that says "we get to send this much data per month"? If Netflix (or Level 3) tries to push more data through that, which causes congestion, it seems like a contractual fact that Comcast is either holding up their end of the bargain or not." Level 3 doesn't buy the connection from Comcast. Comcast buys a connection from Level 3. Comcast needs to do this because they're selling a connection to the "Internet" to their customers, not a connection to the servers Comcast owns, and to other Comcast customers. Comcast's paying customers are requesting data from the Internet, data that Comcast doesn't have on its network. To get there those bits travel over Level 3's (or Cogent, or another major provider) network. The issue at hand is not one of Level 3 (or Netflix) trying to PUSH data onto Comcast's network. It is one of Comcast's customers trying to PULL data from Level 3, over a connection that Comcast refuses to upgrade.
- exelius 12y ago
  This would all be fine and good if Level3 didn't make money from selling bandwidth. Comcast doesn't have to buy a connection from Level3; Level3 will give Comcast a link for free because they know they can turn around and charge their own customers for access to it. Comcast is really just cutting out the middleman here; and that scares Level3.
  - mixologic 12y ago
    But they cant cut out the middleman. Comcast doesnt have a direct connection to netflix or anywhere else on the internet. I think Level 3 should straight up shut off their peering agreement with comcast and watch how all the last mile customers blame comcast for the issue, until such time as comcast stops playing games.
    - exelius 12y ago
      But the Internet doesn't work that way: if Level3 shut off their peering with Comcast, Level3's customers would just find another transit provider and route around L3 because those companies are still in business to make money. They don't care about squabbles between Comcast and L3; they just want their monetized traffic to get delivered. The last-mile customers probably wouldn't even notice; but Level3's customers would likely be forced to pay more for transit from another company. But they would still pay.
cschroed 12y ago
Comcast promises a certain amount of download capability. When customer requests for content that originate from Comcast's network (that would not normally exceed that download capability) end up saturating a link to which Comcast is party to, then it is Comcast's obligation to attempt to upgrade that link. Comcast cannot control the routes via which the content is returned, they must react to their customers usage patterns to improve the performance of their network. So if tomorrow, everyone wants to download content from some other peer then Comcast will need to upgrade other peer connections. The exception to this is if Comcast wants to charge rent to the internet to have access to Comcast customers. Then it is in Comcast's corporate interest to let peer connections saturate and then charge upstream content providers rent to remove the congestion. A pretty ballsy move for a company (who's assets are stretched out over thousands of miles and in plain sight) to attempt to extort the entire planet. ;)
- mixologic 12y ago
  Not very ballsy if they've bought enough politicians and lobbyists.
  - josephschmoe 12y ago
    He means it's ballsy because any one of us could just start cutting their physical lines.
- jessaustin 12y ago
  ...who's assets are stretched out over thousands of miles and in plain sight... I like the cut of your jib.
johnjac 12y ago
My understanding is this. No Money exchanges hands between Level3 and Comcast, but there is an agreement for Comcast to provide a port for Level3 to connect to. Level3 brings the traffic to Comcast door Step. Level3 pays for the port on their side. Level3 is happy to add more ports, BUT Comcast Refuses. Not because more ports are that expensive but those in charge at Comcast are from the TV side of the house where blackouts are common negotiating tactics in order to be paid. And Level3 refuse to be held over a barrel for payment when Comcast has a local monopoly power.
- deleted 12y ago
  [deleted]
cortesoft 12y ago
I think a big problem with your reasoning centers on word choice; you phrased it as Level 3 having a contract that says "we get to send this much data per month", and then Level 3 attempting to send more data than that agreement. However, you must remember that every single packet that Level 3 sends to the Comcast network is a packet that was requested BY A COMCAST CUSTOMER. Level 3 isn't just deciding to send a bunch of traffic over Comcasts network, they are sending the data requested by Comcast customers to that network. This is an important distinction, and is usually part of all of these 'settlement-free' peering arrangements. In agreeing to peer with Level 3, I am sure Comcast has an agreement that they will not send any traffic to Comcast's network that is not actually destined for a Comcast customer. Now, if Level 3 were sending traffic to Comcast and expecting Comcast to route, for free, that traffic to another 3rd part network, that would NOT be kosher, and Comcast could fairly ask for either money or for Level 3 to stop sending that traffic to them. Level 3 is ONLY sending the traffic to Comcast that Comcast is requesting; it is THEIR customers who are creating this demand for Level 3's customer's content. Comcast told their customers that for $X per month, they would get Y amount of internet bandwidth, which will in almost all cases be traffic originating from a non-Comcast network; that is just the nature of the internet. Even though Level 3 is willing to send the data that Comcast's customers want to the Comcast network for free (settlement free), Comcast is instead refusing to accept all the traffic that their own customers are requesting and demanding additional payment to provide the service they already sold and are paid for.
- peterwwillis 12y ago
  "Level 3 is ONLY sending the traffic to Comcast that Comcast is requesting; it is THEIR customers who are creating this demand for Level 3's customer's content." This still misses the mark. Netflix creates a service on Level3 that Comcast customers want. So Netflix/Level3 are responsible for making a product that people want to use, and Comcast customers are responsible for wanting to use that product and increasing bandwidth use. Neither Level3 nor Comcast want to pay more money just because Netflix and Comcast customers want to use more tubes, so both Level3 and Comcast wash their hands of any responsibility for the networks they both run. In a very real sense, both Level3 and Comcast are equally responsible for the increase in traffic [in this case]. Yet neither seem to accept their share in the cost. If it's because it's hard for Level3 to prove, they should put up some money to shore up the connection, and then write a public letter to Comcast customers telling them how Comcast is shorting them and to get angry. (Sadly, many customers have no alternative provider, so their only option would be to quit the internet, and nobody's going to do that)
zero_intp 12y ago
MSO revenue is threatened by alternate entertainment content, and is terrified of becoming a dumb pipe with no add-on fees. The issue is that Comcast wants to both connect to the internet for free & get paid by both the viewer and the content provider as they are in yesterday's cable model. Comcast is connected to the internet through 'settlement-free' peering. Comcast can command this because they have grown large enough to trade access to their customer base for access to the transport network's data. Comcast would like to sell direct access to their network to the content providers, but would like to see more money for the service that the content providers are willing to pay. Comcast has allowed (through inaction of normal peering upgrades) the interconnections to Level-3/Cogent to become saturated. The primary data on those interconnections is data that competes with the normal content that Comcast is used to delivering to their customers. Cogent and Level-3 offer less expensive bandwidth than their competitors. This is because of their lack of incumbent telephony models, and being dedicated fiber bit pushers. Comcast is claiming that it is too expensive for them to deliver the service levels to their customers that their customers have contracted. Comcast is demanding that the companies providing the data (not the transit network delivering the data) must pay to have their traffic delivered the last mile. Simultaneously to crying that their internal networks are saturated and require upgrade, the MSOs are offering products that allow their customers to use the local provider's network (for locally sourced content) without incurring billing penalties or usage caps. (veteran employee of a transit network involved)

_stephan 12y ago

Does anyone have an idea what the European ISP is that refuses to provide enough peering capacity?

revelation 12y ago
They said its not from the UK and judging from their connection points, I guess its Deutsche Telekom.
- josephlord 12y ago
  Don't know they described them as a monopoly position and cable is pretty strong in Germany. I think Spain or Italy might be better guesses. Telefonica in Spain might be a reasonable guess. Ono the cable company isn't huge. I think Telecom Italian is dominant in Italy and there is virtually no cable at all so that would be my guess.
  - josephlord 12y ago
    Too late to fix typo: Telecom Italia (not Italian).
- frik 12y ago
  Yes, it seems it's the "Telekom" from Germany: * http://www.golem.de/news/level-3-sechs-grosse-internetprovider-bremsen-peering-absichtlich-aus-1405-106318.html http://www.golem.de/news/level-3-sechs-grosse-internetprovid... * http://www.welt.de/wirtschaft/article117963991/Bei-Europas-Telekom-Riesen-ist-die-EU-unpatriotisch.html http://www.welt.de/wirtschaft/article117963991/Bei-Europas-T...
seszett 12y ago
I would have guessed it was French ISP Proxad/Free, which is notorious for having generally shitty peering to a large part of Internet (like Youtube, Imgur or Github, among others). But though they are a large ISP they are not dominant, and their peering with Level3 is actually quite good... which is why I tunnel most of my connection to my server (the route to which happens to be through Level3) so I can get usable internet. It really sucks to be able to download an iso from some server in the US at 2 MB/s, but seeing small Imgur gifs load frame by frame, or Github repos cloning at 20 kB/s.

deleted 12y ago

[deleted]

josephlord 12y ago

Summary of situation Comcast cripples customers internet services to extort money from companies connecting Comcast to the internet for free.

LandoCalrissian 12y ago
It's completely crazy and just blatant extortion on Comcast's part. The current system is pretty straight forward. Level 3: Server(Who Level 3 Charges) -> Level 3 -> Comcast -> End User Comcast: End User(Who Comcast Charges) -> Comcast -> Level 3 -> Server Now Comcast wants to charge everyone in the chain that isn't them. They are attempting to do this by degrading (or at the very least not upgrading) their services for their customer. The techno-libertarians that try to defend this nonsense are going to sink the whole ship and make every pay to receive even less.
- erichocean 12y ago
  Ahem. "Techno-libertarians" aren't the ones who gave Comcast its monopolies on the last mile. Please leave them out of this.
  - nullc 12y ago
    No, physics and economics did. When you're talking about building infrastructure that will take 20 years to pay itself off _absent_ competition, only the most profitable of the profitable locations can afford extensive physical plant competition. In communications networks monopolies are natural, especially in last mile networks, and will exist absent any government action. That in some cases some governments have mildly strengthened these monopolies in exchange for commitments to cover otherwise unprofitable areas, its not really that important this subject.
  - wpietri 12y ago
    They have been a strong voice for less regulation, especially less regulation of technology. We reined in AT&T through regulation, forcing them to share their last-mile plant with others. We could (and should) have done the same with monopoly cable providers, but the anti-regulation fever in the US prevented that.

deleted 12y ago

[deleted]

johngalt 12y ago

I enjoy seeing those MRTG/RRDtool graphs everywhere. I find it so surreal that $X Billions in infrastructure, that is forwarding $Y Billion of internet traffic. All monitored by a single tool created by one guy from Switzerland. Since this is HN: yes I realize that there has been substantial work subsequently, and that cacti/munin/nagios etc.. are more commonly used.

zero_intp 12y ago
It's crazy to think that Rancid, MRTG, & RRD are all fundamentally underpinning of all of these major carriers.
- NDizzle 12y ago
  It's not so crazy when you consider a huge portion of peering agreements took place over IRC.
  - voltagex_ 12y ago
    Got any articles on this?
- rhizome 12y ago
  What's crazy about it?
  - zero_intp 12y ago
    quote from the author, "rancid is named rancid because it stinks." it covers 80% of what you want it to do, but is indispensable because it is so lightweight and functional. more developed tools exist but networks are not interested (by in large) in integrating/developing new ones because they fool themselves into thinking these are good enough.
dfc 12y ago
I am surprised that the hobby project by the funny sounding guy from Finland did not mentally prepare you for this.

cs702 12y ago

In short: if you're in the US, you're paying your ISP for a certain amount of bandwidth, but your ISP is not giving it to you, because its connections to middlemen like Level 3 are maxed out. Level 3 proposes to split the cost of expanding those connections (as is common), but your ISP refuses unless it gets additional payment from Level 3, Netflix, or someone else. Meanwhile, you don't get the bandwidth you've purchased from your ISP. That doesn't seem right to me.

andruby 12y ago
For once, I am glad to be living in a densely populated European country. We can easily switch ISP's if one decides to pull these kinds of dirty tricks.
- DrJokepu 12y ago
  Well, except that the local telco (e.g. BT in the UK) almost certainly has monopoly over the last mile of your connection, regardless of who your ISP is (they rent the last mile from e.g. BT Openreach or maybe TalkTalk) and it's not uncommon for them to refuse to fix congestions at their exchanges (or even admit their presence) for extended periods. So, you know, it's not like it's all utopistic here or something.
  - estel 12y ago
    Don't Virgin use their own last mile for fibre?
    - geocar 12y ago
      Virgin offer a "cable" service which is available where they have cable television (not a wide distribution area), and that this is what they refer to as their fibre service. Virgin also resell the same ADSL that everyone else does (complete with BT line rental).
    - __alexs 12y ago
      If by their own you mean one of numerous other cable companies that they purchased then yes. There used to be a plethora of small cable companies in the UK. Now there is only Virgin. The widespread availability of high quality ADSL does seem to do a good job of keeping them honest though. I've been a pretty satisfied customer of them for many years now.

MaysonL 12y ago

Level 3's filing with the FCC about this: http://www.level3.com/~/media/Assets/legal/l3_openinternet_march2014.pdf http://www.level3.com/~/media/Assets/legal/l3_openinternet_m...

jeremycole 12y ago

If Comcast doesn't like the amount of bandwidth their customers use watching Netflix, they always have the option to notify their customers and null-route or otherwise stop Netflix from working at all. However while it "works" but sucks they appear to be doing their jobs while their customers blame Netflix for sucking while largely being unaware that it's actually Comcast that sucks.

sergiosgc 12y ago

If this is a game of chicken, Level 3 and Cogent should play it to the end game. They are powerful enough to get Comcast and the other ISPs to concede. Just null route any interconnect that consistently drops packets and fails to get negotiated for an upgrade. The catch? It must be done by both L3 and Cogent. Comcast is playing the monopoly card against what is an oligopoly of backbone providers. Comcast isn't strategically as well positioned as their actions would indicate. It is one thing for Comcast to have degraded Netflix service. It is a different thing to present their customers with access to only half of the Internet .

DontGiveTwoFlux 12y ago
Except that it would probably violate their contracts to just stop services at an interconnect. It's not like Level 3 is losing customers or money from the congested interconnects. Their incentive to play hardball just isn't there.
- zero_intp 12y ago
  Level3 and Cogent both bill the majority (by traffic volume) of their network consumers by utilization. This means that these congested pipes are actually costing potential revenue. Obviously Cogent would be willing to cut off their own nose to spite their face, but Level3 plays a much more sane game. Hardball by coordinating efforts of the backbone providers is somewhat thwarted by the coordinated efforts of the MSO's, some with their own backbones (VZ, AT&T). Cutting off Comcast would also invalidate a number of (government) customer contracts that require service 'to the full Internet'.
- sergiosgc 12y ago
  If it violates their contract, doesn't a soft fail violate it also? Or will violate, whenever the rate of dropped packets goes above a given threshold. I view the hard fail as just speeding up the interconnect failure, with the positive effect of prompting action from the last mile ISP. Whenever their performance drops towards part of the internet, their value proposition gets lower. You can view the value chain as (residential customer) <-> Comcast <-> Level 3 <-> (hosting customer) Both Comcast and Level 3 have customers, in opposite ends of the currently centralized internet. The status quo is that each provider charges to the customers on their end. Comcast (and each of the other non-net-neutral ISPs) are eyeing the revenue stream on the other end of the pipeline. This revenue stream has historically not been owned by the last-mile ISP. Level 3 and equivalents, by cutting off non-performant interconnects will hurt their own customers, of course. However, they are just speeding up the same effect being applied softly over time, by Comcast and equivalents. A softly failing interconnect moves revenue over to last mile ISPs, thus permanently lowering Level 3's value proposition. By failing the interconnect sharply, instead of softly, they actually improve the steady state of the system, in general (good side effect), and in particular (the real objective).

chrisBob 12y ago

If you are experiencing a slow connection and packet loss, what is the best tool for identifying the issue? There must be a good way to find out where the problem is.

wpietri 12y ago
I use mtr. But reading it takes experience, so for a month or so just fire it up everywhere you go, especially when things are working well.
- chrisBob 12y ago
  Thanks. MTR looks interesting.