6 ms·

NAT is pretty computationally intensive, this is why e.g. ISPs & mobile carriers are pushing IPv6 over CGNAT.

Spivak 3y ago
This is missing the point mostly, my own sites have supported ipv6 for a going on a decade because it was fun to get it working. But that's a very different thing than supporting only IPv6.
- p1mrx 3y ago
  It's best for an ISP to deploy IPv6 and CGNATv4 in parallel, so the NAT only needs to handle traffic for services that don't support IPv6 (e.g. news.ycombinator.com)
NoZebra120vClip 3y ago
For example, rather than simply routing IP packets and then forgetting them, you need to statefully inspect every TCP segment and every supposedly connectionless UDP conversation, you need to maintain state for every live conversation, and you need to mitigate DOS with all those resources. At that point, you might as well be running a Layer 7 Firewall or an Intrusion Protection System.
- tptacek 3y ago
  UDP is connectionless precisely so you can build novel stateful protocols on it. There’s no promise in UDP that you’ll be able to statelessly monitor it.
  - debugnik 3y ago
    Which is why game networking libraries put a lot of emphasis on NAT traversal, forcing NATs to recognise the "connection". And why game console manufacturers tell users to just forward all incoming traffic unmanaged by the NAT to the console.
    - deleted 3y ago
      [deleted]
  - colmmacc 3y ago
    UDP is actually more expensive to NAT than TCP is. The reason is UDP fragmentation, which is my vote for the worst, and least forgivable, design error of TCP/IP. Instead of putting the fragmentation in L4 (like QUIC now does) and including a UDP header on every fragmented packet in a datagram, UDP only includes the header on the first packet. With fragmentation happening; firewalls, NATs, and end-hosts have to buffer and coalesce IP packets based on IP IDs, before the destination can be identified. It's a real nuisance. A lot of CGNAT "stateless" implementations can't handle this and you get very hard to debug issues when there are fragmentation and MTU mismatches.
    - dgemm 3y ago
      Even well-behaved unfragmented UDP should be more expensive to NAT because it doesn't have an end-of-stream "FIN" marker, meaning stateful middleboxes need to retain state for longer because they can only time out.
      - richardwhiuk 3y ago
        Timeouts on UDP are usually much shorter than TCP, so it's not as bad as it sounds.
    - pclmulqdq 3y ago
      This is probably more accurately called IP fragmentation (since that is the layer where the fragmentation happens), and a lot of companies make it optional to support in networking gear. I'm surprised that you are using it or seeing it, because it is essentially obsolete today. It has a legitimate purpose in old-timey systems which have bespoke MTUs on each link, but now the usual thing is to use 1500 bytes for WAN traffic, which is the generic Ethernet MTU, and reserve larger sizes for intra-datacenter communications.
      - colmmacc 3y ago
        There's a number of UDP protocols that have large enough payloads to fragment. DNSSEC and EDNS0 in particular made it much more common, though the EDNS0 flag day in 2020 partially undid some of the damage by getting folks to ratchet down their EDNS0 buffer sizes. 1500 is absolutely not a pervasively usable WAN MTU, you're going to need pMTUd if you're sending 1500 byte packets broadly. Plenty of WAN links won't tolerate it. If you don't want to deal with fragmentation at all ... 500 is the minimum guaranteed MTU, but in practice it's exceptionally rare to see anything below about 1200 require fragmentation. But you can always only control what you send, not what others are sending you.
- Bluecobra 3y ago
  > At that point, you might as well be running a Layer 7 Firewall or an Intrusion Protection System. If you go down this path consider using Transit Gateway so you can route multiple VPC traffic to a central security VPC in a region. I’ve done this a Palo Alto VM and it seems to work well.
xxpor 3y ago
It's not really computationally expensive, it's memory expensive. You need per connection state.
- blibble 3y ago
  it already has stateful firewall so that's: source ip, dest ip, protocol, source port, dest port, connection state (say 16 bytes total) doing NAT too is what, 3 more bytes per connection (8 bits for an offset into an IP table and 16 bits for the translated port)
  - p1mrx 3y ago
    Generally an ISP does not have a stateful firewall prior to deploying CGNAT.
  - dijit 3y ago
    NAT and Stateful firewalling are commonly bundled together (especially on home systems) but I would not go so far as to say “NAT has a stateful firewall”- I hear such takes all the time and its really frustrating; usually in threads regarding IPv6, incidentally it is usually programmers who think they understand everything about networks because they know how tcp operates.
    - blibble 3y ago
      > but I would not go so far as to say “NAT has a stateful firewall”- > I hear such takes all the time and its really frustrating maybe you'd be less frustrated if you understood what people were saying, because I didn't say that AWS already do 1:1 NAT and there's additionally a stateful firewall, which necessitates connection state tracking adding the extra few bytes to do port translation shouldn't vastly increase the memory required > incidentally it is usually programmers who think they understand everything about networks because they know how tcp operates. from someone who has written a commercial packet filter: in terms of complexity, TCP blows the preceding layers of the stack out of the water
    - lxgr 3y ago
amluto 3y ago
AWS NAT gateway is $0.045 per hour plus $0.045 per GB. The hourly fee seems mostly okay - for largish users, one or two per region is fine. $0.045 per GB is nuts. That’s $20.25/hour or $14580/mo for 1 Gbps. One can buy a cheap gadget using very little power that can NAT 1 Gbps at line rate for maybe $200 (being generous). One can buy a perfectly nice low power server that can NAT 10Gbps line rate for $1k with some compute to spare. One can operate one of these systems, complete with a rack and far more power than needed, plus the Internet connection, for a lot less money than $14580/mo. (Never mind that your $14580 doesn’t actually cover the egress fee on AWS.) A company with a couple full time employees could easily operate quite a few of these out of any normal datacenter, charge AWS-like fees, and make a killing, without breaking a sweat. But they wouldn’t get many clients because most datacenter customers already have a NAT-capable router and don’t need this service to begin with. In other words, the OpEx associated with a service like this, including the sysadmin time, is simply not in the ballpark of what AWS charges.
- ttt3ts 3y ago
  Bit confused. Couldn't you just run a Linux VM to do your NAT and only pay normal egress?
  - deadmutex 3y ago
    > just run a Linux VM + Run extra for failover, HA etc + manage security + Monitor performance + ...
    - ttt3ts 3y ago
      You would have to run that in your own data center which is what original poster was comparing to.
      - otterley 3y ago
        You also have to do it in AWS if you don't want to use the NAT Gateway service and still desire reliability over and above the MTBF for an EC2 instance or AZ, or ever want to do anything requiring a reboot.
meragrin_ 3y ago
> ISPs & mobile carriers are pushing IPv6 over CGNAT LOL. Not Metronet. They are doubling down on CGNAT. They've acquired ISPs with IPv6 and killed it in favor of CGNAT.

This is missing the point mostly, my own sites have supported ipv6 for a going on a decade because it was fun to get it working. But that's a very different thing than supporting only IPv6.

p1mrx 3y ago
It's best for an ISP to deploy IPv6 and CGNATv4 in parallel, so the NAT only needs to handle traffic for services that don't support IPv6 (e.g. news.ycombinator.com)

NoZebra120vClip 3y ago

For example, rather than simply routing IP packets and then forgetting them, you need to statefully inspect every TCP segment and every supposedly connectionless UDP conversation, you need to maintain state for every live conversation, and you need to mitigate DOS with all those resources. At that point, you might as well be running a Layer 7 Firewall or an Intrusion Protection System.

tptacek 3y ago
UDP is connectionless precisely so you can build novel stateful protocols on it. There’s no promise in UDP that you’ll be able to statelessly monitor it.
- debugnik 3y ago
  Which is why game networking libraries put a lot of emphasis on NAT traversal, forcing NATs to recognise the "connection". And why game console manufacturers tell users to just forward all incoming traffic unmanaged by the NAT to the console.
  - deleted 3y ago
    [deleted]
- colmmacc 3y ago
  UDP is actually more expensive to NAT than TCP is. The reason is UDP fragmentation, which is my vote for the worst, and least forgivable, design error of TCP/IP. Instead of putting the fragmentation in L4 (like QUIC now does) and including a UDP header on every fragmented packet in a datagram, UDP only includes the header on the first packet. With fragmentation happening; firewalls, NATs, and end-hosts have to buffer and coalesce IP packets based on IP IDs, before the destination can be identified. It's a real nuisance. A lot of CGNAT "stateless" implementations can't handle this and you get very hard to debug issues when there are fragmentation and MTU mismatches.
  - dgemm 3y ago
    Even well-behaved unfragmented UDP should be more expensive to NAT because it doesn't have an end-of-stream "FIN" marker, meaning stateful middleboxes need to retain state for longer because they can only time out.
    - richardwhiuk 3y ago
      Timeouts on UDP are usually much shorter than TCP, so it's not as bad as it sounds.
  - pclmulqdq 3y ago
    This is probably more accurately called IP fragmentation (since that is the layer where the fragmentation happens), and a lot of companies make it optional to support in networking gear. I'm surprised that you are using it or seeing it, because it is essentially obsolete today. It has a legitimate purpose in old-timey systems which have bespoke MTUs on each link, but now the usual thing is to use 1500 bytes for WAN traffic, which is the generic Ethernet MTU, and reserve larger sizes for intra-datacenter communications.
    - colmmacc 3y ago
      There's a number of UDP protocols that have large enough payloads to fragment. DNSSEC and EDNS0 in particular made it much more common, though the EDNS0 flag day in 2020 partially undid some of the damage by getting folks to ratchet down their EDNS0 buffer sizes. 1500 is absolutely not a pervasively usable WAN MTU, you're going to need pMTUd if you're sending 1500 byte packets broadly. Plenty of WAN links won't tolerate it. If you don't want to deal with fragmentation at all ... 500 is the minimum guaranteed MTU, but in practice it's exceptionally rare to see anything below about 1200 require fragmentation. But you can always only control what you send, not what others are sending you.
Bluecobra 3y ago
> At that point, you might as well be running a Layer 7 Firewall or an Intrusion Protection System. If you go down this path consider using Transit Gateway so you can route multiple VPC traffic to a central security VPC in a region. I’ve done this a Palo Alto VM and it seems to work well.

xxpor 3y ago

It's not really computationally expensive, it's memory expensive. You need per connection state.

blibble 3y ago
it already has stateful firewall so that's: source ip, dest ip, protocol, source port, dest port, connection state (say 16 bytes total) doing NAT too is what, 3 more bytes per connection (8 bits for an offset into an IP table and 16 bits for the translated port)
- p1mrx 3y ago
  Generally an ISP does not have a stateful firewall prior to deploying CGNAT.
- dijit 3y ago
  NAT and Stateful firewalling are commonly bundled together (especially on home systems) but I would not go so far as to say “NAT has a stateful firewall”- I hear such takes all the time and its really frustrating; usually in threads regarding IPv6, incidentally it is usually programmers who think they understand everything about networks because they know how tcp operates.
  - blibble 3y ago
    > but I would not go so far as to say “NAT has a stateful firewall”- > I hear such takes all the time and its really frustrating maybe you'd be less frustrated if you understood what people were saying, because I didn't say that AWS already do 1:1 NAT and there's additionally a stateful firewall, which necessitates connection state tracking adding the extra few bytes to do port translation shouldn't vastly increase the memory required > incidentally it is usually programmers who think they understand everything about networks because they know how tcp operates. from someone who has written a commercial packet filter: in terms of complexity, TCP blows the preceding layers of the stack out of the water
  - lxgr 3y ago

amluto 3y ago

AWS NAT gateway is $0.045 per hour plus $0.045 per GB. The hourly fee seems mostly okay - for largish users, one or two per region is fine. $0.045 per GB is nuts. That’s $20.25/hour or $14580/mo for 1 Gbps. One can buy a cheap gadget using very little power that can NAT 1 Gbps at line rate for maybe $200 (being generous). One can buy a perfectly nice low power server that can NAT 10Gbps line rate for $1k with some compute to spare. One can operate one of these systems, complete with a rack and far more power than needed, plus the Internet connection, for a lot less money than $14580/mo. (Never mind that your $14580 doesn’t actually cover the egress fee on AWS.) A company with a couple full time employees could easily operate quite a few of these out of any normal datacenter, charge AWS-like fees, and make a killing, without breaking a sweat. But they wouldn’t get many clients because most datacenter customers already have a NAT-capable router and don’t need this service to begin with. In other words, the OpEx associated with a service like this, including the sysadmin time, is simply not in the ballpark of what AWS charges.

ttt3ts 3y ago
Bit confused. Couldn't you just run a Linux VM to do your NAT and only pay normal egress?
- deadmutex 3y ago
  > just run a Linux VM + Run extra for failover, HA etc + manage security + Monitor performance + ...
  - ttt3ts 3y ago
    You would have to run that in your own data center which is what original poster was comparing to.
    - otterley 3y ago
      You also have to do it in AWS if you don't want to use the NAT Gateway service and still desire reliability over and above the MTBF for an EC2 instance or AZ, or ever want to do anything requiring a reboot.

meragrin_ 3y ago

> ISPs & mobile carriers are pushing IPv6 over CGNAT LOL. Not Metronet. They are doubling down on CGNAT. They've acquired ISPs with IPv6 and killed it in favor of CGNAT.