10 ms·

Mitmproxy 11: Full HTTP/3 Support

388 points by mhils 2y ago

38 2y ago
uh: https://github.com/mitmproxy/mitmproxy/issues/4170 https://github.com/mitmproxy/mitmproxy/issues/4170
- halJordan 2y ago
  Oh no, the summer intern didn't close the issue
- cryptonector 2y ago
  https://github.com/mitmproxy/mitmproxy/issues/6199 https://github.com/mitmproxy/mitmproxy/issues/6199
Onavo 2y ago
Do http/2 and http/3 offer any benefits if they are only supported by the reverse proxy but not the underlying web server? Most mainstream frameworks for JS/Python/Ruby don't support the newer http standards. Won't the web server be a bottleneck for the reverse proxied connection?
- masspro 2y ago
  Probably not, but mitmproxy is not a reverse proxy for any production purpose. It’s for running on your local machine and doing testing of either low-level protocol or web security stuff.
  - codetrotter 2y ago
    > mitmproxy is not a reverse proxy for any production purpose At a startup I was working on a few years ago, I set up mitmproxy in dev and eventually if memory serves right I also sometimes enabled it in prod to debug things. That being said, we did not have a lot of users. We had in fact very very few users at the time.
    - hedora 2y ago
      I’ve been patiently waiting for someone to write a howto that uses mitmproxy to transparently obtain acme certificates for any web servers that are behind it. I’d totally pay a cloud provider to just do this and forward requests to my port 80 or 443 with self signed certificates. Https+acme is already open to this attack vector, so why inconvenience myself by pretending it is not?
      - dandandan 2y ago
        ngrok can do that https://ngrok.com/docs/network-edge/domains-and-tcp-addresses/#branded-domains https://ngrok.com/docs/network-edge/domains-and-tcp-addresse... https://ngrok.com/docs/network-edge/tls-certificates/#automated https://ngrok.com/docs/network-edge/tls-certificates/#automa...
      - codetrotter 2y ago
        In our setup, TLS was already being terminated by Nginx or Caddy (I don’t remember which, but it was one of those two) sitting in front of another web server on the same host. So inserting mitmproxy into the setup was just a case of putting it between the Nginx or Caddy that did TLS termination, and the web server that served the backend API. So to mitmproxy it was all plain HTTP traffic passing through it, locally on the same machine. I bound the mitmweb web UI to the VPN interface so that us devs could connect to the dev server with VPN and then have access to the mitmweb web UI to inspect requests and responses.
- connicpu 2y ago
  Depends. If they're running on the same box, the reverse proxy will be able to initiate tcp connections to the web server much more cheaply. Even if they're just in the same datacenter, the lower round trip latency will reduce the time for establishing TCP connections. Plus, the proxy might be load balancing across multiple instances of the backend.
  - apitman 2y ago
    Also browsers limit the number of HTTP/1.1 requests you can have in flight to a specific domain
    - ahoka 2y ago
      The limit is much higher for proxies, though.
      - connicpu 2y ago
        With a reverse proxy the browser doesn't know it's talking to one
- Narhem 2y ago
  http/3 seems to be an excellent opportunity to optimize HTMX or any of the libraries which leverage HTML fragments like JSX. The obvious advantage of http/3 is for gaming. The servers which run the frameworks have to http/3. In most cases the advantages should be transparent to the developers.
  - deznu 2y ago
    I’m curious what about HTTP/3 is particularly advantageous with HTMX?
    - Narhem 2y ago
      A common use case of HTMX is sending fragments when scrolling. Since http/3 uses udp to send the fragments, duplicate packet information doesn’t have to be sent. Kind of funny the newer protocol effectively works in the opposite direction of GraphQl.
      - deleted 2y ago
        [deleted]
      - greenavocado 2y ago
        Network congestion management is gonna be wild in the coming decade with the proliferation of udp based protocols
- markasoftware 2y ago
  the transport between reverse proxy <-> backend is not always http, eg python w/ uwsgi and php w/ fastcgi. And even when it is HTTP, as other commenters said, the reverse proxy is able to handshake connections to the backend much more quickly than an actual remote client would, so it's still advantageous to use http/2 streams for the slower part of the connection.
  - account42 2y ago
    > the transport between reverse proxy <-> backend is not always http, eg python w/ uwsgi and php w/ fastcgi. That's just called a web server and not a reverse proxy then. Both are just evolutions of CGI.
- AgentME 2y ago
  Yes, because http/2 or http/3 will improve the reliability of the connection between the client and the reverse proxy. The connection between the reverse proxy and the underlying web server is usually much faster and more reliable, so that part would benefit much less from being upgraded to http/2 or http/3.
- mhils 2y ago
  One of the main promises of HTTP/3 is better performance under worse network conditions (e.g. no head-of-line blocking as in HTTP/2, connection migration, 0-RTT). For all of that HTTP/3 between client and proxy is really great. HTTP/3 between proxy and server is not required for that.
- lemagedurage 2y ago
  Yes. Besides other performance benefits, HTTP/3 saves a full roundtrip for connection by combining TCP and TLS handshakes.
- nitely 2y ago
  Something not mentioned: web-browsers limit the number of connections per domain to 6. With +http/2 they will use a single connection for multiple concurrent requests.
- jeltz 2y ago
  Yes, for http/3 since it handles network issues better. Http/2 is of more doubtful value since it can choke really bad on packet loss.
bluejekyll 2y ago
Thanks for the shoutout to Hickory. It’s always fun to see what people build with it. Nice work!
- mhils 2y ago
  Thank you for your work on Hickory! It's super exciting to see how PyO3's Python <-> Rust interop enables us to use a production-grade DNS library with Hickory and also a really solid user-space networking stack with smoltcp. These things wouldn't be available in Python otherwise.
envoked 2y ago
It’s great to see that Mitmproxy is still being developed - it indirectly made my career. Back in 2011, I was using it to learn API development by intercepting mobile app requests when I discovered that Airbnb’s API was susceptible to Rails mass assignment (https://github.com/rails/rails/issues/5228 https://github.com/rails/rails/issues/5228). I then used it to modify some benign attributes, reached out to the company, and it landed me an interview. Rest is history.
- danmur 2y ago
  It's absolutely insane how many core devs argued against change there
- RamRodification 2y ago
  > Rest is history ;)
- JeremyNT 2y ago
  To this day it remains incredibly useful to me, and weirdly obscure to people who I would've thought should know better. Sometimes it's easier to use mitmproxy with an existing implementation than to read the documentation!
deleted 2y ago
[deleted]
febusravenga 2y ago
Only slightly related ... > Chrome does not trust user-added Certificate Authorities for QUIC. Interesting. In linked issue chrome team says: > We explicitly disallow non-publicly-trusted certificates in QUIC to prevent the deployment of QUIC interception software/hardware, as that would harm the evolvability of the QUIC protocol long-term. Use-cases that rely on non-publicly-trusted certificates can use TLS+TCP instead of QUIC. I don't follow evolution of those protocols, but i am not sure how disallowing custom certificates has anything with "evolvability" of protocol ... Anyone knows are those _reasons_?
- sbinder 2y ago
  Perhaps they're referring to this famous objection of financial institutions to TLS 1.3, motivated by them not wanting to update their MitM software needed for compliance: https://mailarchive.ietf.org/arch/msg/tls/CzjJB1g0uFypY8UDdr6P9SCQBqA/ https://mailarchive.ietf.org/arch/msg/tls/CzjJB1g0uFypY8UDdr...
  - eptcyka 2y ago
    TLS1.3 breaks MITM boxes because a client can establish a session key outside of the network with the middle box and continue using it afterwards in the middlebox’s network.
- globular-toast 2y ago
  It can seem confusing but it all makes sense when you realise Chrome is designed to work for Google, not for you. I remember people switching their Grandmas to Chrome 15 years ago when they could've chosen Firefox. Many of us knew this would happen, but convenience and branding is everything, sadly.
  - le-mark 2y ago
    > Chrome is designed to work for Google, not for you. Maybe more accurately “chrome is designed to work for you in so far as that also works for google”. I share the long standing dismay that so many willingly surrendered their data and attention stream to an ad company.
nilslindemann 2y ago
I wonder, can I use it like Privoxy/Proxomitron/Yarip? E.g. can I strip out script tags from specific sites, which I request with my browser (Ungoogled Chromium), using Mitmproxy as a Proxy? And how will this affect performance?
- jeroenhd 2y ago
  In theory: yes. In practice: mitmproxy is written in Python so there will be a delay because of the language not being all that fast. When you're visiting web pages with hundreds of small delays, you'll notice. That said, for many people who care about this stuff, this could be an option. There's nothing preventing you from doing this technically speaking. There's a small risk of triggering subresource integrity checks when rewriting Javascript files, but you can probably rewrite the hashes to fix that problem if it comes up in practice.
rnhmjoj 2y ago
Unfortunately there is still the issue[1] of fingerprinting. Until it can spoof the TLS handshake of a typical browser, you get these "Just a quick check..." or "Sorry, it looks like you're a bot" pages on about 80% of the web. [1]: https://github.com/mitmproxy/mitmproxy/issues/4575 https://github.com/mitmproxy/mitmproxy/issues/4575
- account42 2y ago
  > Until it can spoof the TLS handshake of a typical browser, you get these "Just a quick check..." or "Sorry, it looks like you're a bot" pages on about 80% of the web. Evidently Firefox is not a typical browser anymore.
systems 2y ago
is mitmproxy an alternative to fiddler?
- RockRobotRock 2y ago
  Pretty much. What I like about mitmproxy is how easy it is to write a Python plugin to intercept/modify requests and responses. https://docs.mitmproxy.org/stable/addons-overview/ https://docs.mitmproxy.org/stable/addons-overview/

388 points by mhils 2y ago

38 2y ago

uh: https://github.com/mitmproxy/mitmproxy/issues/4170 https://github.com/mitmproxy/mitmproxy/issues/4170

halJordan 2y ago
Oh no, the summer intern didn't close the issue
cryptonector 2y ago
https://github.com/mitmproxy/mitmproxy/issues/6199 https://github.com/mitmproxy/mitmproxy/issues/6199

Onavo 2y ago

Do http/2 and http/3 offer any benefits if they are only supported by the reverse proxy but not the underlying web server? Most mainstream frameworks for JS/Python/Ruby don't support the newer http standards. Won't the web server be a bottleneck for the reverse proxied connection?

masspro 2y ago
Probably not, but mitmproxy is not a reverse proxy for any production purpose. It’s for running on your local machine and doing testing of either low-level protocol or web security stuff.
- codetrotter 2y ago
  > mitmproxy is not a reverse proxy for any production purpose At a startup I was working on a few years ago, I set up mitmproxy in dev and eventually if memory serves right I also sometimes enabled it in prod to debug things. That being said, we did not have a lot of users. We had in fact very very few users at the time.
  - hedora 2y ago
    I’ve been patiently waiting for someone to write a howto that uses mitmproxy to transparently obtain acme certificates for any web servers that are behind it. I’d totally pay a cloud provider to just do this and forward requests to my port 80 or 443 with self signed certificates. Https+acme is already open to this attack vector, so why inconvenience myself by pretending it is not?
    - dandandan 2y ago
      ngrok can do that https://ngrok.com/docs/network-edge/domains-and-tcp-addresses/#branded-domains https://ngrok.com/docs/network-edge/domains-and-tcp-addresse... https://ngrok.com/docs/network-edge/tls-certificates/#automated https://ngrok.com/docs/network-edge/tls-certificates/#automa...
    - codetrotter 2y ago
      In our setup, TLS was already being terminated by Nginx or Caddy (I don’t remember which, but it was one of those two) sitting in front of another web server on the same host. So inserting mitmproxy into the setup was just a case of putting it between the Nginx or Caddy that did TLS termination, and the web server that served the backend API. So to mitmproxy it was all plain HTTP traffic passing through it, locally on the same machine. I bound the mitmweb web UI to the VPN interface so that us devs could connect to the dev server with VPN and then have access to the mitmweb web UI to inspect requests and responses.
connicpu 2y ago
Depends. If they're running on the same box, the reverse proxy will be able to initiate tcp connections to the web server much more cheaply. Even if they're just in the same datacenter, the lower round trip latency will reduce the time for establishing TCP connections. Plus, the proxy might be load balancing across multiple instances of the backend.
- apitman 2y ago
  Also browsers limit the number of HTTP/1.1 requests you can have in flight to a specific domain
  - ahoka 2y ago
    The limit is much higher for proxies, though.
    - connicpu 2y ago
      With a reverse proxy the browser doesn't know it's talking to one
Narhem 2y ago
http/3 seems to be an excellent opportunity to optimize HTMX or any of the libraries which leverage HTML fragments like JSX. The obvious advantage of http/3 is for gaming. The servers which run the frameworks have to http/3. In most cases the advantages should be transparent to the developers.
- deznu 2y ago
  I’m curious what about HTTP/3 is particularly advantageous with HTMX?
  - Narhem 2y ago
    A common use case of HTMX is sending fragments when scrolling. Since http/3 uses udp to send the fragments, duplicate packet information doesn’t have to be sent. Kind of funny the newer protocol effectively works in the opposite direction of GraphQl.
    - deleted 2y ago
      [deleted]
    - greenavocado 2y ago
      Network congestion management is gonna be wild in the coming decade with the proliferation of udp based protocols
markasoftware 2y ago
the transport between reverse proxy <-> backend is not always http, eg python w/ uwsgi and php w/ fastcgi. And even when it is HTTP, as other commenters said, the reverse proxy is able to handshake connections to the backend much more quickly than an actual remote client would, so it's still advantageous to use http/2 streams for the slower part of the connection.
- account42 2y ago
  > the transport between reverse proxy <-> backend is not always http, eg python w/ uwsgi and php w/ fastcgi. That's just called a web server and not a reverse proxy then. Both are just evolutions of CGI.
AgentME 2y ago
Yes, because http/2 or http/3 will improve the reliability of the connection between the client and the reverse proxy. The connection between the reverse proxy and the underlying web server is usually much faster and more reliable, so that part would benefit much less from being upgraded to http/2 or http/3.
mhils 2y ago
One of the main promises of HTTP/3 is better performance under worse network conditions (e.g. no head-of-line blocking as in HTTP/2, connection migration, 0-RTT). For all of that HTTP/3 between client and proxy is really great. HTTP/3 between proxy and server is not required for that.
lemagedurage 2y ago
Yes. Besides other performance benefits, HTTP/3 saves a full roundtrip for connection by combining TCP and TLS handshakes.
nitely 2y ago
Something not mentioned: web-browsers limit the number of connections per domain to 6. With +http/2 they will use a single connection for multiple concurrent requests.
jeltz 2y ago
Yes, for http/3 since it handles network issues better. Http/2 is of more doubtful value since it can choke really bad on packet loss.

bluejekyll 2y ago

Thanks for the shoutout to Hickory. It’s always fun to see what people build with it. Nice work!

mhils 2y ago
Thank you for your work on Hickory! It's super exciting to see how PyO3's Python <-> Rust interop enables us to use a production-grade DNS library with Hickory and also a really solid user-space networking stack with smoltcp. These things wouldn't be available in Python otherwise.

envoked 2y ago

It’s great to see that Mitmproxy is still being developed - it indirectly made my career. Back in 2011, I was using it to learn API development by intercepting mobile app requests when I discovered that Airbnb’s API was susceptible to Rails mass assignment (https://github.com/rails/rails/issues/5228 https://github.com/rails/rails/issues/5228). I then used it to modify some benign attributes, reached out to the company, and it landed me an interview. Rest is history.

danmur 2y ago
It's absolutely insane how many core devs argued against change there
RamRodification 2y ago
> Rest is history ;)
JeremyNT 2y ago
To this day it remains incredibly useful to me, and weirdly obscure to people who I would've thought should know better. Sometimes it's easier to use mitmproxy with an existing implementation than to read the documentation!

deleted 2y ago

[deleted]

febusravenga 2y ago

Only slightly related ... > Chrome does not trust user-added Certificate Authorities for QUIC. Interesting. In linked issue chrome team says: > We explicitly disallow non-publicly-trusted certificates in QUIC to prevent the deployment of QUIC interception software/hardware, as that would harm the evolvability of the QUIC protocol long-term. Use-cases that rely on non-publicly-trusted certificates can use TLS+TCP instead of QUIC. I don't follow evolution of those protocols, but i am not sure how disallowing custom certificates has anything with "evolvability" of protocol ... Anyone knows are those _reasons_?

sbinder 2y ago
Perhaps they're referring to this famous objection of financial institutions to TLS 1.3, motivated by them not wanting to update their MitM software needed for compliance: https://mailarchive.ietf.org/arch/msg/tls/CzjJB1g0uFypY8UDdr6P9SCQBqA/ https://mailarchive.ietf.org/arch/msg/tls/CzjJB1g0uFypY8UDdr...
- eptcyka 2y ago
  TLS1.3 breaks MITM boxes because a client can establish a session key outside of the network with the middle box and continue using it afterwards in the middlebox’s network.
globular-toast 2y ago
It can seem confusing but it all makes sense when you realise Chrome is designed to work for Google, not for you. I remember people switching their Grandmas to Chrome 15 years ago when they could've chosen Firefox. Many of us knew this would happen, but convenience and branding is everything, sadly.
- le-mark 2y ago
  > Chrome is designed to work for Google, not for you. Maybe more accurately “chrome is designed to work for you in so far as that also works for google”. I share the long standing dismay that so many willingly surrendered their data and attention stream to an ad company.

nilslindemann 2y ago

I wonder, can I use it like Privoxy/Proxomitron/Yarip? E.g. can I strip out script tags from specific sites, which I request with my browser (Ungoogled Chromium), using Mitmproxy as a Proxy? And how will this affect performance?

jeroenhd 2y ago
In theory: yes. In practice: mitmproxy is written in Python so there will be a delay because of the language not being all that fast. When you're visiting web pages with hundreds of small delays, you'll notice. That said, for many people who care about this stuff, this could be an option. There's nothing preventing you from doing this technically speaking. There's a small risk of triggering subresource integrity checks when rewriting Javascript files, but you can probably rewrite the hashes to fix that problem if it comes up in practice.

rnhmjoj 2y ago

Unfortunately there is still the issue[1] of fingerprinting. Until it can spoof the TLS handshake of a typical browser, you get these "Just a quick check..." or "Sorry, it looks like you're a bot" pages on about 80% of the web. [1]: https://github.com/mitmproxy/mitmproxy/issues/4575 https://github.com/mitmproxy/mitmproxy/issues/4575

account42 2y ago
> Until it can spoof the TLS handshake of a typical browser, you get these "Just a quick check..." or "Sorry, it looks like you're a bot" pages on about 80% of the web. Evidently Firefox is not a typical browser anymore.

systems 2y ago

is mitmproxy an alternative to fiddler?

RockRobotRock 2y ago
Pretty much. What I like about mitmproxy is how easy it is to write a Python plugin to intercept/modify requests and responses. https://docs.mitmproxy.org/stable/addons-overview/ https://docs.mitmproxy.org/stable/addons-overview/