10 ms·
Looks like a DNSSEC issue, not a nameserver outage. Validating resolvers SERVFAIL on every .de name with EDE: RRSIG with malformed signature found for a0d5d1
by krystofbe 5mo ago
Looks like a DNSSEC issue, not a nameserver outage. Validating resolvers SERVFAIL on every .de name with EDE:
RRSIG with malformed signature found for
a0d5d1p51kijsevll74k523htmq406bk.de/nsec3 (keytag=33834)
dig +cd amazon.de @8.8.8.8 works, dig amazon.de @a.nic.de works. Zone data is intact, DENIC just published an RRSIG over an NSEC3 record that doesn't validate against ZSK 33834. Every validating resolver therefore refuses to answer.
Intermittency fits anycast: some [a-n].nic.de instances still serve the previous (good) signatures, so retries occasionally land on a healthy auth. Per DENIC's FAQ the .de ZSK rotates every 5 weeks via pre-publish, so this smells like a botched rollover.
- qazwsxedchac 5mo agoSo a single configuration mistake in a single place wiped out external reachability of a major economy. It happened in the evening local time and should be fixable, modulo cache TTLs, by morning. This will limit the blast radius somewhat. Still, at this level, brittle infrastructure is a political risk. The internet's famous "routing around damage" isn't quite working here. Should make for an interesting post mortem.
- walrus01 5mo agoIt looks like a failed key replacement during a scheduled maintenance event. Normally this sort of thing is thoroughly tested and has multiple eyes on for detailed review and planning before changes get committed, but obviously something got missed.
- account42 4mo agoWould be interesting to know how something could get missed. You'd think the system was set up so that new keys could not be published without being verified working in a staging system.
- wildylion 4mo ago.ru [had this](https://habr.com/ru/news/790214/ https://habr.com/ru/news/790214/) in 2024. On the other hand, the russian government in itself had been doing a stellar job at breaking the internet. (I hope I'll live to see them all sentenced to life without parole)
- the8472 5mo agofail-closed protocols have introduced some brittleness. A HTTP 1.0 server from 1999 probably still can service visitors today. A HTTPS/TLS 1.0 server from the same year wouldn't.
- fc417fc802 5mo agoYou're not wrong but objecting to fail-closed in a security sensitive context is entirely missing the point.
- the8472 4mo agoGiving up on robustness is capitulation on a social-technical interface. "users can't be trusted, we can't have any fallbacks, ever" is a problem that needs to be solved.
- zelon88 5mo agoI think I see the point you're making here and I agree. There is designing something to be fail-closed because it needs to be secure in a physical sense (actually secure, physically protected), and then there's designing something fail-closed because it needs to be secure from an intellectual sense (gatekept, intellectually protected). While most of the internet is "open source" by nature, the complexity has been increased to the point where significant financial and technical investment must be made to even just participate. We've let the gatekeepers raise the gates so high that nobody can reach them. AI will let the gatekeepers keep raising the gates, but then even they won't be able to reach the top. Then what? I think the point you're trying to make, put another way is in the context of "availability" and "accessibility" we've compromised a lot of both availability and accessibility in the name of security since the dawn of the internet. How much of that security actually benefits the internet, and how much of that security hinders it? How much of it exists as a gatekeeping measure by those who can afford to write the rules?
- sam_lowry_ 5mo agoThis is why I still run my blog on HTTP/1.1 only.
- pocksuppet 5mo agoDNS is a centralization risk, yes. Somehow we've decided this is fine. DNSSEC isn't the only issue - your TLD's nameservers could also be offline, or censored in your country.
- skywhopper 5mo agoDNS is barely centralized. Is there an alternative global name lookup system that is less centralized without even worse downsides?
- pocksuppet 5mo agoBGP, but the names in question are limited to 128 bits, of which at most 48 will be looked up, and you don't get to choose which 48 bits are assigned to you.
- fc417fc802 5mo agoGNS is the obvious response here, in addition to the various blockchain based solutions. Nothing that enjoys widespread support or mindshare unfortunately. Even the current centralized ICANN flavor could be substantially more resilient if it instead handed out key fingerprints and semi-permanent addresses when queried. That way it would only ever need to be used as a fallback when the previously queried information failed to resolve.
- account42 4mo agoGP said it was a risk (and it is), not that there are better alternatives. Not all risks can be eliminated easily but you should still be aware of them.
- miki123211 4mo agoThe blockchain. The only thing a blockchain is good for is achieving decentralized consensus on what value a key points to, which is what DNS is. An alternative way of looking at this is that acquiring domains must be somewhat expensive by definition; either you enforce it at the system level, or you make it free, but then somebody will inevitably grab all the interesting ones and re-sell them to others. A blockchain is the only way to make decentralized financial infrastructure viable.
- lschueller 5mo agoI have a bad feeling, that the impact will be quite severe for some services, as monitoring, performance, and security services might get disrupted. and just cleaning up is a big mess.. Worst case, some ot will experience outage and / or damage. But maybe I am just overestimating the severity of this.
- Muromec 5mo ago>So a single configuration mistake in a single place wiped out external reachability of a major economy. And fuck nothing at all happened as a result.
- Our_Benefactors 5mo agoProve it? I’m sure many lifespans were lost to stress
- pinkgolem 5mo agoAs someone with oncall yesterday it was a fun experience, but you noticed quickly that everything .de was down and then it was just a waiting game. We had a short discussion about migrating to .com, but decided risk != reward as no one would know the new tld I assume there are a couple people working for denic who had a stressfull night..
- belorn 5mo agoI am reminded of the warning that zonemaster gives about putting your domain name servers on a single AS, as is common practice for many larger providers. A lot of people do not want others to see this as a problem since a single AS is a convenient configuration for routing, but it has the downside of being a single point of failure. Building redundant infrastructure that can withstand BGP and DNS configuration mistakes are not that simple but it can be done.
- walrus01 5mo agoAs the CPU/RAM resources to run an authoritative-only slave nameserver for a few domains are extremely minimal (mine run at a unix load of 0.01), it's a very wise idea to put your ns3 or something at a totally different service provider on another continent. It costs less than a cup of coffee per month.
- deleted 5mo ago[deleted]
- belorn 4mo agoFor a very long time, the computer club I was in operated a DNS server on a Pentium 75MHz and after the last major hardware upgrade it had a total of 110MB RAM memory and 2G disk space. It worked great except that before the upgrade it tended to run out of ram whenever there was a Linux kernel update, a problem we solved forever by populating all the ram slots with the maximum that the motherboard could handle to that nice 110 MB.
- psd1 4mo agoDid you populate the motherboard with the most it could handle, or the most you could assemble from a box of assorted sticks? Otherwise, 110MB would hint at a fascinating engineering culture at the motherboard manufacturer.
- walrus01 4mo agoIf I remember right there were certain very early pentium 3 processor competitors from VIA and other non-intel, non-AMD sources (with much worse performance) that had integrated onboard SVGA video, where the video RAM was shared with the system DRAM. Meaning that depending how you configured the video in the BIOS, you could have something like a 128GB RAM server "minus" 16GB RAM withheld for video, with like 112GB usable by the OS. But if this guy is talking about a pentium 75 MHz (socket 5 CPU) that's a totally different generation of stuff several generations before that.
- gerdesj 5mo ago"The internet's famous "routing around damage" isn't quite working here." DNS is a look up service that runs on the internet. Internet routing of IP packets is what the internet does and that is working fine (for a given value of fine). You remind me of someone using the term "the internet is down" that really means: "I've forgotten my wifi password".
- Woodi 5mo ago> So a single configuration mistake in a single place wiped out external reachability of a major economy. Real world beats sci-fi :) And isn't it why we love IT ? And hate it too, because of "peoples in charge"...
- otabdeveloper4 5mo ago> The internet's famous "routing around damage" ...is only for Pentagon networks and military stuff. It's not for us normal people. (We get Cloudflare and FAANG bullshit instead.)
- zelon88 5mo agoThis is actually startlingly true. Every FAANG company has their own fiber backbone. Why invest the internet that everyone uses when you can invest in your own private internet and then sell that instead?
- profmonocle 5mo agoIt's not like the long-haul fiber not owned by FAANG is a public utility, at least not in most places. Traffic that goes over "the Internet" traverses some mix of your ISP's fiber, fiber belonging to some other ISP they have a deal with, then fiber belong to some ISP they have a deal with, etc. All those ISPs are being paid to provide service, they can invest in their own networks.
- account42 4mo agoAnd we all know that ISPs are famous for investing in timely infrastructure upgrades.
- number6 5mo agoThere is the kritis (critical infrastructure law) law, which trys to enforce some standards to make things not as brittle.
- beeforpork 4mo ago... wiped out external reachability of a major economy ... internal reachability (from Germany to .de domains), too... :-)))
- miki123211 4mo agoThe more interesting question is, could a political adversary do this to a country on purpose, and how hard would that be?
- throw0101c 4mo ago> So a single configuration mistake in a single place wiped out external reachability of a major economy. No different than a bunch of BGP issues we've seen over the years. And you don't even need DNSSEC for DNS to break things: reminder of the October 2025 AWS outage: * https://www.akamai.com/blog/security/when-cloud-breaks-lessons-aws-outage https://www.akamai.com/blog/security/when-cloud-breaks-lesso...
- dlopes7 5mo agoI love how I work with IT for 20 years and don't understand a single acronym here other than DNSSEC
- walrus01 5mo agoTo be fair, advanced real world knowledge of public/private key PKIs (x.509 or other), things like root CAs, are a fairly esoteric and very specialized field of study. There's people whose regular day jobs are nothing but doing stuff with PKI infrastructure and their depth of knowledge on many other non-PKI subjects is probably surface level only.
- hannob 5mo agoI know quite a bit about PKI and X.509, and I can tell you that much: the overlap with how DNSSEC works is limited.
- silisili 5mo agoAs is the overlap between DNSSEC and DNS itself, to be honest. I once worked at the level of administering DNSSEC for 300+ TLDs. It's its own world. When that company was winding down, I tried to continue in the field but the most common response (outside of no response, of course), was 'we already have a DNS team/vendor/guy.' And well, then things like this happen. I won't throw stones though, it's a lot to learn and can be incredibly brittle.
- hathawsh 5mo agoIs that actually true, though? Even though it's not really my job, I find myself debugging certificates and keys at least once a month, and that's after automating as much as possible with certbot and cloud certificates. PKI always seems to demand attention.
- walrus01 5mo agoIn my initial comment, I meant more in terms of complexity and planning from the perspective of the people who are running the public/private key infrastructure on the other side/upstream of what you're doing as a letsencrypt end user. Broadly similar general concept to the team responsible for the DNSSSEC signing keys for an entire ccTLD. Yeah a x509 PKI / root CA is a very different thing than DNSSSEC but they have a number of general logical similarities in that the chain of trust ultimately comes down to a "do not fuck this up" single point of failure.
- 1vuio0pswjnm7 4mo agoCurious how this comment was originally marked as "dead" but is now on top I saw it at bottom of thread and vouched for it. Usually when I "vouch" nothing happens I always ignore the RRSIG lines in zone files. To me it's not "DNS data", it's cruft But DNSSEC has its true believers. I'm just not one of them