5 ms·

I think the era of traffic shaping as you describe it is no longer. Most connections are encrypted these days, a good thing. But that also means, me as an ISP

by fjni 3y ago

I think the era of traffic shaping as you describe it is no longer. Most connections are encrypted these days, a good thing. But that also means, me as an ISP in the middle, I only see the layer 4 packet. So I know where a packet is coming from and where it's going (and ttl and some other not really helpful data for this purpose.) I don't even know the port. So I'm left with doing (dumb) traffic shaping by destination and target. If I had to guess, I'd say that they incorrectly thought that some specific IP address (range) serves predominantly one type of data. So they throttle by the only data point they have, destination ip, and the collateral damage is everything else hosted on that ip address.

kaszanka 3y ago
Wait, why wouldn't they know the port? I'm not aware of any cases where the TCP header itself is encrypted.
- fjni 3y ago
  You are correct. The port is part of the tcp header and my statement above is incorrect!
- kevincox 3y ago
  99% of consumer traffic is on 443 these days (citation needed)
  - robocat 3y ago
    HTTP3 uses UDP and doesn’t have to use the same port, although I presume it does. “A browser first connects the server with HTTP/2 To discover the service. The server responses with an Alt-Svc header, including the port for HTTP/3, such as Alt-Svc: h3-29=":443"”
    - Avamander 3y ago
      This changes with HTTPS DNS records, browsers can make a H2/H3 connection straight if they get the hints.
- deleted 3y ago
  [deleted]
kevin_nisbet 3y ago
So I had this conversation with one of the traffic shaping vendors many years ago. At the time when encryption still wasn't that common, they didn't sound too worried about the shift to all encrypted connections. Their product was already starting to use a behavioral analysis to shape the traffic most of their customers were interested in. So to detect bittorrent, they'd build a profile about how many bit torrent clients operate, the packet and connection creation patterns used, and then slap a throttle on. Looking at some independent analysis, these products might only detect 50% of the bittorent traffic, and have a false positive rate, especially for bittorent users also doing something else. And the ISPs don't care, they get what they need if they clamp 50% of the traffic. So I'm not disputing that everything encrypted is a good thing, just pointing out that because it's encrypted doesn't necessarily mean the shaping equipment can't figure out enough to throttle bit torrent. > If I had to guess, I'd say that they incorrectly thought that some specific IP address (range) serves predominantly one type of data. So they throttle by the only data point they have, destination ip, and the collateral damage is everything else hosted on that ip address. This is plausible. As I recall, the way some of the equipment worked was it would sniff out DNS requests, and then mark the IP address as this destination. So if someone set's a rule for example.com, it might accidentally apply to alice.com using the same IP address. My knowledge on the industry is out of date though.
- fjni 3y ago
  This is great insight. Thank you for that. I imagine lots of people are or have spent lots of money and time trying to figure out the type of data or connection from patterns as you say. A more nuanced and correct statement would have been to say that it's much harder to do than it used to be, when you could just look at the mime-type or similar to figure out what to throttle.
  - Meleagris 3y ago
    If you're interested in learning more, deep packet inspection, and specifically "encrypted traffic classification" are fairly mature in industry. Many traffic shapers are using products like Enea's Qosmos ixEngine or home-grown equivalents which can identify thousands of applications and protocols. Most providers that use traffic shaping don't care about content, and the encrypted traffic classification is enough to make traffic policy decisions.

by fjni 3y ago

kaszanka 3y ago

Wait, why wouldn't they know the port? I'm not aware of any cases where the TCP header itself is encrypted.

fjni 3y ago
You are correct. The port is part of the tcp header and my statement above is incorrect!
kevincox 3y ago
99% of consumer traffic is on 443 these days (citation needed)
- robocat 3y ago
  HTTP3 uses UDP and doesn’t have to use the same port, although I presume it does. “A browser first connects the server with HTTP/2 To discover the service. The server responses with an Alt-Svc header, including the port for HTTP/3, such as Alt-Svc: h3-29=":443"”
  - Avamander 3y ago
    This changes with HTTPS DNS records, browsers can make a H2/H3 connection straight if they get the hints.
deleted 3y ago
[deleted]

kevin_nisbet 3y ago

So I had this conversation with one of the traffic shaping vendors many years ago. At the time when encryption still wasn't that common, they didn't sound too worried about the shift to all encrypted connections. Their product was already starting to use a behavioral analysis to shape the traffic most of their customers were interested in. So to detect bittorrent, they'd build a profile about how many bit torrent clients operate, the packet and connection creation patterns used, and then slap a throttle on. Looking at some independent analysis, these products might only detect 50% of the bittorent traffic, and have a false positive rate, especially for bittorent users also doing something else. And the ISPs don't care, they get what they need if they clamp 50% of the traffic. So I'm not disputing that everything encrypted is a good thing, just pointing out that because it's encrypted doesn't necessarily mean the shaping equipment can't figure out enough to throttle bit torrent. > If I had to guess, I'd say that they incorrectly thought that some specific IP address (range) serves predominantly one type of data. So they throttle by the only data point they have, destination ip, and the collateral damage is everything else hosted on that ip address. This is plausible. As I recall, the way some of the equipment worked was it would sniff out DNS requests, and then mark the IP address as this destination. So if someone set's a rule for example.com, it might accidentally apply to alice.com using the same IP address. My knowledge on the industry is out of date though.

fjni 3y ago
This is great insight. Thank you for that. I imagine lots of people are or have spent lots of money and time trying to figure out the type of data or connection from patterns as you say. A more nuanced and correct statement would have been to say that it's much harder to do than it used to be, when you could just look at the mime-type or similar to figure out what to throttle.
- Meleagris 3y ago
  If you're interested in learning more, deep packet inspection, and specifically "encrypted traffic classification" are fairly mature in industry. Many traffic shapers are using products like Enea's Qosmos ixEngine or home-grown equivalents which can identify thousands of applications and protocols. Most providers that use traffic shaping don't care about content, and the encrypted traffic classification is enough to make traffic policy decisions.