6 ms·

Good writeup. > It’s worth noting that DoH (DNS-over-HTTPS) traffic remained relatively stable as most DoH users use the domain cloudflare-dns.com, configured

Good writeup. > It’s worth noting that DoH (DNS-over-HTTPS) traffic remained relatively stable as most DoH users use the domain cloudflare-dns.com, configured manually or through their browser, to access the public DNS resolver, rather than by IP address. Interesting, I was affected by this yesterday. My router (supposedly) had Cloudflare DoH enabled but nothing would resolve. Changing the DNS server to 8.8.8.8 fixed the issues.

bauruine 1y ago
How does DoH work? Somehow you need to know the IP of cloudflare-dns.com first. Maybe your router uses 1.1.1.1 for this.
- ta1243 1y ago
  And even if you have already resolved it the TTL is only 5 minutes
- deleted 1y ago
  [deleted]
- nelox 1y ago
  [flagged]
  - k1t 1y ago
    Smells like AI and completely fails to answer the question. How is the IP address of the DoH server obtained?
    - MayeulC 1y ago
      Firefox accepts a bootstrap IP, or uses the system resolver: > network.trr.bootstrapAddress > (default: none) by setting this field to the IP address of the host name used in "network.trr.uri", you can bypass using the system native resolver for it. Use this to get the IPs of the cloudflare server: https://dns.google/query?name=mozilla.cloudflare-dns.com https://dns.google/query?name=mozilla.cloudflare-dns.com > Starting with Firefox 74 setting the bootstrap address is no longer required in mode 3. Firefox will attempt to use regular DNS in order to get the IP address of the trusted resolver. However, if DNS resolution of the resolver domain fails, setting the bootstrap address is again necessary. Source: https://wiki.mozilla.org/Trusted_Recursive_Resolver https://wiki.mozilla.org/Trusted_Recursive_Resolver
- stavros 1y ago
  Are we meant to use a domain? I've always just used the IP.
  - landgenoot 1y ago
    You need a domain in order to get the s in https to work
    - federiconafria 1y ago
      What about a reverse DNS lookup?
    - yread 1y ago
      what about certificate for IP address?
      - landgenoot 1y ago
        What about a route that gets hijacked? There is no HSTS for IP addresses.
        sathackr 1y ago
        Presumably the route hijacker wouldn't have a valid private key for the certificate so they wouldn't pass validation
    - bigiain 1y ago
      That's not correct. LetEncrypt are trialling ip address https/TLS certificates right now: https://letsencrypt.org/2025/07/01/issuing-our-first-ip-address-certificate/ https://letsencrypt.org/2025/07/01/issuing-our-first-ip-addr... They say: "In principle, there’s no reason that a certificate couldn’t be issued for an IP address rather than a domain name, and in fact the technical and policy standards for certificates have always allowed this, with a handful of certificate authorities offering this service on a small scale."
- stingraycharles 1y ago
  Yeah I don’t understand this part either, maybe it’s supposed to be bootstrapped using your ISP’s DNS server?
  - tom1337 1y ago
    Pretty much that. You set up a bootstrap DNS server (could be your ISPs or any other server) which then resolves the IP of the DoH server which then can be used for all future requests.
- maxloh 1y ago
  Yeah, your operating system will first need to resolve cloudflare-dns.com. This initial resolution will likely occur unencrypted via the network's default DNS. Only then will your system query the resolved address for its DoH requests. Note that this introduces one query overhead per DNS request if the previous cache has expired. For this reason, I've been using https://1.1.1.1/dns-query https://1.1.1.1/dns-query instead. In theory, this should eliminate that overhead. Your operating system can validate the IP address of the DNS response by using the Subject Alternative Name (SAN) field within the CA certificate presented by the DoH server: https://g.co/gemini/share/40af4514cb6e https://g.co/gemini/share/40af4514cb6e
Hamuko 1y ago
My (Unifi) router is set to automatic DoH, and I think that means it's using Cloudflare and Google. Didn't notice any disruptions so either the Cloudflare DoH kept working or it used the Google one while it was down.
- zahrc 1y ago
  Check Jallmann’s response https://news.ycombinator.com/item?id=44578490#44578917 https://news.ycombinator.com/item?id=44578490#44578917 TLDR; DoH was working
  - Thorrez 1y ago
    AFAICS, Jallmann just left 1 comment and it was top-level. I'm not sure what you mean by "Jallmann’s response".
sneak 1y ago
I disagree. The actual root cause here is shrouded in jargon that even experienced admins such as myself have to struggle to parse. It’s corporate newspeak. “legacy” isn’t a clear term, it’s used to abstract and obfuscate. > Legacy components do not leverage a gradual, staged deployment methodology. Cloudflare will deprecate these systems which enables modern progressive and health mediated deployment processes to provide earlier indication in a staged manner and rollback accordingly. I know what this means, but there’s absolutely no reason for it to be written in this inscrutable corporatese.
- willejs 1y ago
  If you carry on reading, its quite obvious they misconfigured a service and routed production traffic to that instead of the correct service, and the system used to do that was built in 2018 and is considered legacy (probably because you can easily deploy bad configs). Given that, I wouldn't say the summary is "inscrutable corporatese" whatever that is.
  - bigiain 1y ago
    I agree it's not "inscrutable corporatese" It's carefully written so my boss's boss thinks he understands it, and that we cannot possibly have that problem because we obviously don't have any "legacy components" because we are "modern and progressive". It is, in my opinion, closer to "intentionally misleading corporatese".
    - noduerme 1y ago
      Joe Shmo committed the wrong config file to production. Innocent mistake. Sally caught it in 30 seconds. We were back up inside 2 minutes. Sent Joe to the margarita shop to recover his shattered nerves. Kid deserves a raise. Etc.
      - sathackr 1y ago
        Yea the "timeline" indicating impact start/end is entirely false when you look at the traffic graph shared later in the post. Or they have a different definition of impact than I do
noduerme 1y ago
Funny. I was configuring a new domain today, and for about 20 minutes I could only reach it through Firefox on one laptop. Google's DNS tools showed it active. SSH to an Amazon server that could resolve it. My local network had no idea of it. Flush cache and all. Turns out I had that one FF browser set up to use Cloudflare's DoH.
sathackr 1y ago
Good writeup except the entirely false timeline shared at the beginning of the post
- bartvk 1y ago
  You need to clarify such a statement, in my opinion.

bauruine 1y ago

How does DoH work? Somehow you need to know the IP of cloudflare-dns.com first. Maybe your router uses 1.1.1.1 for this.

ta1243 1y ago
And even if you have already resolved it the TTL is only 5 minutes
deleted 1y ago
[deleted]
nelox 1y ago
[flagged]
- k1t 1y ago
  Smells like AI and completely fails to answer the question. How is the IP address of the DoH server obtained?
  - MayeulC 1y ago
    Firefox accepts a bootstrap IP, or uses the system resolver: > network.trr.bootstrapAddress > (default: none) by setting this field to the IP address of the host name used in "network.trr.uri", you can bypass using the system native resolver for it. Use this to get the IPs of the cloudflare server: https://dns.google/query?name=mozilla.cloudflare-dns.com https://dns.google/query?name=mozilla.cloudflare-dns.com > Starting with Firefox 74 setting the bootstrap address is no longer required in mode 3. Firefox will attempt to use regular DNS in order to get the IP address of the trusted resolver. However, if DNS resolution of the resolver domain fails, setting the bootstrap address is again necessary. Source: https://wiki.mozilla.org/Trusted_Recursive_Resolver https://wiki.mozilla.org/Trusted_Recursive_Resolver
stavros 1y ago
Are we meant to use a domain? I've always just used the IP.
- landgenoot 1y ago
  You need a domain in order to get the s in https to work
  - federiconafria 1y ago
    What about a reverse DNS lookup?
  - yread 1y ago
    what about certificate for IP address?
    - landgenoot 1y ago
      What about a route that gets hijacked? There is no HSTS for IP addresses.
      - sathackr 1y ago
        Presumably the route hijacker wouldn't have a valid private key for the certificate so they wouldn't pass validation
  - bigiain 1y ago
    That's not correct. LetEncrypt are trialling ip address https/TLS certificates right now: https://letsencrypt.org/2025/07/01/issuing-our-first-ip-address-certificate/ https://letsencrypt.org/2025/07/01/issuing-our-first-ip-addr... They say: "In principle, there’s no reason that a certificate couldn’t be issued for an IP address rather than a domain name, and in fact the technical and policy standards for certificates have always allowed this, with a handful of certificate authorities offering this service on a small scale."
stingraycharles 1y ago
Yeah I don’t understand this part either, maybe it’s supposed to be bootstrapped using your ISP’s DNS server?
- tom1337 1y ago
  Pretty much that. You set up a bootstrap DNS server (could be your ISPs or any other server) which then resolves the IP of the DoH server which then can be used for all future requests.
maxloh 1y ago
Yeah, your operating system will first need to resolve cloudflare-dns.com. This initial resolution will likely occur unencrypted via the network's default DNS. Only then will your system query the resolved address for its DoH requests. Note that this introduces one query overhead per DNS request if the previous cache has expired. For this reason, I've been using https://1.1.1.1/dns-query https://1.1.1.1/dns-query instead. In theory, this should eliminate that overhead. Your operating system can validate the IP address of the DNS response by using the Subject Alternative Name (SAN) field within the CA certificate presented by the DoH server: https://g.co/gemini/share/40af4514cb6e https://g.co/gemini/share/40af4514cb6e

Hamuko 1y ago

My (Unifi) router is set to automatic DoH, and I think that means it's using Cloudflare and Google. Didn't notice any disruptions so either the Cloudflare DoH kept working or it used the Google one while it was down.

zahrc 1y ago
Check Jallmann’s response https://news.ycombinator.com/item?id=44578490#44578917 https://news.ycombinator.com/item?id=44578490#44578917 TLDR; DoH was working
- Thorrez 1y ago
  AFAICS, Jallmann just left 1 comment and it was top-level. I'm not sure what you mean by "Jallmann’s response".

sneak 1y ago

I disagree. The actual root cause here is shrouded in jargon that even experienced admins such as myself have to struggle to parse. It’s corporate newspeak. “legacy” isn’t a clear term, it’s used to abstract and obfuscate. > Legacy components do not leverage a gradual, staged deployment methodology. Cloudflare will deprecate these systems which enables modern progressive and health mediated deployment processes to provide earlier indication in a staged manner and rollback accordingly. I know what this means, but there’s absolutely no reason for it to be written in this inscrutable corporatese.

willejs 1y ago
If you carry on reading, its quite obvious they misconfigured a service and routed production traffic to that instead of the correct service, and the system used to do that was built in 2018 and is considered legacy (probably because you can easily deploy bad configs). Given that, I wouldn't say the summary is "inscrutable corporatese" whatever that is.
- bigiain 1y ago
  I agree it's not "inscrutable corporatese" It's carefully written so my boss's boss thinks he understands it, and that we cannot possibly have that problem because we obviously don't have any "legacy components" because we are "modern and progressive". It is, in my opinion, closer to "intentionally misleading corporatese".
  - noduerme 1y ago
    Joe Shmo committed the wrong config file to production. Innocent mistake. Sally caught it in 30 seconds. We were back up inside 2 minutes. Sent Joe to the margarita shop to recover his shattered nerves. Kid deserves a raise. Etc.
    - sathackr 1y ago
      Yea the "timeline" indicating impact start/end is entirely false when you look at the traffic graph shared later in the post. Or they have a different definition of impact than I do

noduerme 1y ago

Funny. I was configuring a new domain today, and for about 20 minutes I could only reach it through Firefox on one laptop. Google's DNS tools showed it active. SSH to an Amazon server that could resolve it. My local network had no idea of it. Flush cache and all. Turns out I had that one FF browser set up to use Cloudflare's DoH.

sathackr 1y ago

Good writeup except the entirely false timeline shared at the beginning of the post

bartvk 1y ago
You need to clarify such a statement, in my opinion.