11 ms·

DigiCert Revocation Incident (CNAME Domain Validation)

141 points by vitaliyf 2y ago

deleted 2y ago
[deleted]
kevinday 2y ago
https://bugzilla.mozilla.org/show_bug.cgi?id=1910322 https://bugzilla.mozilla.org/show_bug.cgi?id=1910322 for more background. The short story is that when doing CNAME based validation, they were supposed to put an underscore at the start of the random string for you to add to your DNS records. They still generated sufficiently random strings but didn't include a _ before it which is in violation of the RFC. The rationale is that some sites might do something like give you control of yourusername.example.com and they don't want to make it possible for random users to register the random string and be able to manipulate it. If you don't allow users to generate anything that causes a hostname to appear with a leading underscore, they can't pass the domain validation.
- tialaramex 2y ago
  Also, while a DNS name can have an underscore a host name, even in DNS, cannot have this character. So if you have a user named "haha_funny" you already aren't allowed to give them the hostname "haha_funny.somesite.example" - and on some system it will just silently not work because it's invalid. So even if you are completely oblivious to this work, and don't care about security at all, your "Give everybody a hostname" code should already avoid underscore characters as desired because otherwise stuff breaks. Several current systems use DNS names (but not hostnames) which feature underscores but it's pretty unlikely that you've got (for example) a service where users can pick their own TCP/IP service name and port and issued appropriate records for it in DNS. If you have done this weird thing you probably want to use the existing mechanism (in DNS of course, the CAA record) to tell most CAs that they should not issue for your names even if they think they've received permission. You can then cut a suitable deal with a for-profit CA to do whatever crazy extra checks you want (e.g. Meta's CA has to contact actual people in the appropriate security team at Meta, so that "mistakes" which give somebody a certificate for facebook.com never happen without some pretty drastic real world errors).
  - userbinator 2y ago
    So if you have a user named "haha_funny" you already aren't allowed to give them the hostname "haha_funny.somesite.example" - and on some system it will just silently not work because it's invalid. Not long ago I actually did come across a site that had an underscore in its domain name, and it worked both for me and apparently Google, because it indexed and showed a (relevant) page from that site. I only remembered it was on a *.tripod.com subdomain, and can't find that exact site now since I don't remember what I was searching for (it was a highly obscure and technical topic), but there do appear to be others there with underscores, e.g.: http://computer_collector.tripod.com/ http://computer_collector.tripod.com/ http://hattori_striker.tripod.com/ http://hattori_striker.tripod.com/ http://forgotten_dark_angel.tripod.com/ http://forgotten_dark_angel.tripod.com/
    - deleted 2y ago
      [deleted]
    - tialaramex 2y ago
      There are. There shouldn't be, but there are.
    - jesprenj 2y ago
      Google also indexed my site http://_.4a.si http://_.4a.si as seen here http://google.com/search?q=site:_.4a.si http://google.com/search?q=site:_.4a.si
      - rvnx 2y ago
        A live proof that CNAME records starting with _ exist.
    - pests 2y ago
      In 2019 the CAs agreed not to issue certs to underscored subdomains making this less useful. As evidenced by all your links being http. (as an aside, it looks really weird seeing a bare http link in the wild - crazy that was the old norm!)
agwa 2y ago
> The underscore prefix ensures that the random value cannot collide with an actual domain name that uses the same random value. While the odds of that happening are practically negligible, the validation is still deemed as non-compliant if it does not include the underscore prefix. That's not the rationale for mandating the underscore prefix. The actual reason is so services that allow users to create DNS records at subdomains (e.g. dynamic DNS services) can block users from registering subdomains starting with an underscore. It serves the same purpose that /.well-known does. For example, if an attacker requests a certificate for dyndns.example and DigiCert gives them a record without an underscore prefix like da39a3ee5e6b4b0d3255bfef95601890afd80709.dyndns.example, they can register that subdomain with the dynamic DNS provider, publish the required record, and get the certificate for dyndns.example. It doesn't matter how much entropy DigiCert put in the record name. I definitely commend DigiCert for pledging to revoke the certificates within 24 hours and not having a delayed revocation or trying to language lawyer their way to a 5 day revocation as other CAs have tried. Nevertheless, this post severely minimizes the security impact of their mistake, and provides an excellent example of why CAs should always be required to strictly adhere to the rules and not be permitted to excuse noncompliance based on their own security analysis.
- michaelt 2y ago
  > For example, if an attacker requests a certificate for dyndns.example Shouldn't that get caught by the Public Suffix List? I would hope DigiCert has checks in place to prevent someone domain-validating ownership of the entire of co.uk under any circumstances :) (They should still revoke the mis-issued certificates though)
  - remus 2y ago
    > Shouldn't that get caught by the Public Suffix List? PSL is a best-effort sort of thing, so it's good but not definitive. It would be dangerous to rely on it when issuing certs imo.
  - nickf 2y ago
    PSL has a couple of sections - ICANN and PRIVATE. PRIVATE can be a little more flexible/ignorable. If they implement a hard rule, then occasionally they'd have to make exceptions when the real Dyn comes along and wants (legitimately) a wildcard for their name.
jiggawatts 2y ago
I just want to call out both CrowdStrike and DigiCert for being one of "those" companies that insist on publishing critical support information behind a login with the clock ticking on a global outage of their own making. There are no polite words that I can use to accurately convey the depth of my disappointment at this kind of inconsiderate behaviour during a crisis, so I won't say anything more.
- Cyphase 2y ago
  Are you referring to the list of affected certificates?
- Brian_K_White 2y ago
  What critical support information? What global outage? How are these two events or companies remotely equivalent? If I'm not a Digicert customer, what do I care about the details of how to redo a validation on Digicert? If I am a Digicert customer I have been emailed already and I will obviously have to log in to do anything at all with my domain. They say this affects 0.4% of Digicert customers who are what % of the world? Actually not even 0.4% of Digicert customers, but 0.4% of those particular validations. What does that actually work out to? Just who all is actually down? I fail to see any equivalence.
  - deleted 2y ago
    [deleted]
    - deleted 2y ago
      [deleted]
- freeone3000 2y ago
  If you’re not a customer, your domain isn’t affected.
  - deleted 2y ago
    [deleted]
256_ 2y ago
> While we had regression testing in place, those tests failed to alert us to the change in functionality because the regression tests were scoped to workflows and functionality instead of the content/structure of the random value. [...] > Unfortunately, no reviews were done to compare the legacy random value implementations with the random value implementations in the new system for every scenario. In other words, they didn't do proper testing. At the bottom of the article they suggest they're going to improve it.
ratg13 2y ago
24h notice to change certificates in who knows how many systems, at the worlds largest companies, while everyone is on vacation. This will be interesting.
- devrand 2y ago
  Respect to them for actually abiding by the BRs. Most CAs just shrug [1] and [2] say [3] it's [4] too [5] complicated [6], or just lie and claim planes will start crashing [7]. It's really disheartening that publicly trusted CAs just ignore their contractual obligations however they see fit. Ideally these companies should have response plans in place to prioritize certificate rotation. They can use this as a fire drill for what would happen if there were a key compromise. Alternatively, if companies cannot handle the rotation, then they likely should re-evaluate if WebPKI is even appropriate for their use-case. [1]: https://bugzilla.mozilla.org/show_bug.cgi?id=1885568 https://bugzilla.mozilla.org/show_bug.cgi?id=1885568 [2]: https://bugzilla.mozilla.org/show_bug.cgi?id=1898848 https://bugzilla.mozilla.org/show_bug.cgi?id=1898848 [3]: https://bugzilla.mozilla.org/show_bug.cgi?id=1910237 https://bugzilla.mozilla.org/show_bug.cgi?id=1910237 [4]: https://bugzilla.mozilla.org/show_bug.cgi?id=1896053 https://bugzilla.mozilla.org/show_bug.cgi?id=1896053 [5]: https://bugzilla.mozilla.org/show_bug.cgi?id=1896553 https://bugzilla.mozilla.org/show_bug.cgi?id=1896553 [6]: https://bugzilla.mozilla.org/show_bug.cgi?id=1877388 https://bugzilla.mozilla.org/show_bug.cgi?id=1877388 [7]: https://bugzilla.mozilla.org/show_bug.cgi?id=1903066#c48 https://bugzilla.mozilla.org/show_bug.cgi?id=1903066#c48
  - hg35h4 2y ago
    "Alternatively, if companies cannot handle the rotation, then they likely should re-evaluate if WebPKI is even appropriate for their use-case." I hate hearing this awful take, as if every IT organization has the same neat and tidy systems deployed as they do. Never had to deal with 3rd party SaaS vendors certificate pinning requiring service tickets to change, don't have any hardware devices or appliance based software images each with their own web interface to update certs... Yes companies should have a plan to do their minimum yearly certificate rotates. Yes those companies should have a security plan to rotate affected certificate issues, but in those cases the business users are ok with an outage to remediate a real security issue. But what happened here is that Digicert invalided the entire domain's worth of certs. All those service.companyname.com certs or duplicates under that domain validation were affected in bulk. In some companies there could be thousands of certs under that domain. Digicert screwed up their system implementation and made their customers suffer. "It's really disheartening that publicly trusted CAs just ignore their contractual obligations however they see fit." It's also disheartening to see browsers in the CA consortium ignore the CA resolutions as well. Like how everyone voted for 2 year certs and Apple did their own thing anyways. Any punishment for Apple come? So why pick on the others?
Apfel 2y ago
Is this a potential cause of the current Azure outages hitting western europe? I know DigiCert are used by Azure extensively...
- agwa 2y ago
  Unlikely. Microsoft operates their own CAs. Some of their CAs have been cross-signed by a DigiCert root, but Microsoft is responsible for the domain validation. I don't think they extensively use certificates issued directly by a DigiCert CA.
  - nickf 2y ago
    They have almost 300 in the affected list from DigiCert, so who knows?!
olliej 2y ago
One of the impacted companies filed a restraining order, because they believe their incompetence is more important than basic functionality of the PKI. Can't wait to hear how they expect to respond if they ever have encounter a cert compromise or actual misissuance, maybe they'll demand 24 hour revocation in that case? Honestly my opinion is that this should trigger the company being banned by all CAs. The company in question is Alegeus Technologies LLC: https://www.courtlistener.com/docket/68995396/alegeus-technologies-llc-v-digicert/ https://www.courtlistener.com/docket/68995396/alegeus-techno... From basic googling it looks like a healthcare provider, so exactly the kind of company you would want to have shitty IT and security infrastructure. A++ work. Absolutely stellar.
notemaker 2y ago
Can someone explain why this issue deserves a 24h notice? Seems more reasonable to me to have a much longer deprecation notice.
- alex34778 2y ago
  As far as I can tell, this issue would be a problem where all of the following conditions are met: 1. Tenants are allowed to create arbitrary subdomains with arbitrary CNAME values 2. Tenants are not authorized to act on behalf of the TLD directly, only on their respective subdomain 3. Tenants are ostensibly prevented from TLD cert issuance by being explicitly blocked from creating subdomains that start with underscores For most entities these conditions probably do not hold true anyway. But it could conceivably apply to certain free/dynamic dns providers, for example afraid.org and noip both allow arbitrary CNAMEs (though I checked my noip account and it wouldn't work anyway because of length limits on subdomains). I would guess that in act fact there are very few entities in existence for which this actually represents a potential threat against them, since it requires a very specific delineation of zone authorizations, but there might be a few. For most of Alegeus customers I doubt any of this applies, though, they're probably lucky to know their GoDaddy login to add any sort of DNS record, let alone have a whole system in place for less privileged users to create arbitrary CNAME records subject to controls over the use of underscores.

141 points by vitaliyf 2y ago

deleted 2y ago

[deleted]

https://bugzilla.mozilla.org/show_bug.cgi?id=1910322 https://bugzilla.mozilla.org/show_bug.cgi?id=1910322 for more background. The short story is that when doing CNAME based validation, they were supposed to put an underscore at the start of the random string for you to add to your DNS records. They still generated sufficiently random strings but didn't include a _ before it which is in violation of the RFC. The rationale is that some sites might do something like give you control of yourusername.example.com and they don't want to make it possible for random users to register the random string and be able to manipulate it. If you don't allow users to generate anything that causes a hostname to appear with a leading underscore, they can't pass the domain validation.

tialaramex 2y ago
Also, while a DNS name can have an underscore a host name, even in DNS, cannot have this character. So if you have a user named "haha_funny" you already aren't allowed to give them the hostname "haha_funny.somesite.example" - and on some system it will just silently not work because it's invalid. So even if you are completely oblivious to this work, and don't care about security at all, your "Give everybody a hostname" code should already avoid underscore characters as desired because otherwise stuff breaks. Several current systems use DNS names (but not hostnames) which feature underscores but it's pretty unlikely that you've got (for example) a service where users can pick their own TCP/IP service name and port and issued appropriate records for it in DNS. If you have done this weird thing you probably want to use the existing mechanism (in DNS of course, the CAA record) to tell most CAs that they should not issue for your names even if they think they've received permission. You can then cut a suitable deal with a for-profit CA to do whatever crazy extra checks you want (e.g. Meta's CA has to contact actual people in the appropriate security team at Meta, so that "mistakes" which give somebody a certificate for facebook.com never happen without some pretty drastic real world errors).
- userbinator 2y ago
  So if you have a user named "haha_funny" you already aren't allowed to give them the hostname "haha_funny.somesite.example" - and on some system it will just silently not work because it's invalid. Not long ago I actually did come across a site that had an underscore in its domain name, and it worked both for me and apparently Google, because it indexed and showed a (relevant) page from that site. I only remembered it was on a *.tripod.com subdomain, and can't find that exact site now since I don't remember what I was searching for (it was a highly obscure and technical topic), but there do appear to be others there with underscores, e.g.: http://computer_collector.tripod.com/ http://computer_collector.tripod.com/ http://hattori_striker.tripod.com/ http://hattori_striker.tripod.com/ http://forgotten_dark_angel.tripod.com/ http://forgotten_dark_angel.tripod.com/
  - deleted 2y ago
    [deleted]
  - tialaramex 2y ago
    There are. There shouldn't be, but there are.
  - jesprenj 2y ago
    Google also indexed my site http://_.4a.si http://_.4a.si as seen here http://google.com/search?q=site:_.4a.si http://google.com/search?q=site:_.4a.si
    - rvnx 2y ago
      A live proof that CNAME records starting with _ exist.
  - pests 2y ago
    In 2019 the CAs agreed not to issue certs to underscored subdomains making this less useful. As evidenced by all your links being http. (as an aside, it looks really weird seeing a bare http link in the wild - crazy that was the old norm!)

agwa 2y ago

> The underscore prefix ensures that the random value cannot collide with an actual domain name that uses the same random value. While the odds of that happening are practically negligible, the validation is still deemed as non-compliant if it does not include the underscore prefix. That's not the rationale for mandating the underscore prefix. The actual reason is so services that allow users to create DNS records at subdomains (e.g. dynamic DNS services) can block users from registering subdomains starting with an underscore. It serves the same purpose that /.well-known does. For example, if an attacker requests a certificate for dyndns.example and DigiCert gives them a record without an underscore prefix like da39a3ee5e6b4b0d3255bfef95601890afd80709.dyndns.example, they can register that subdomain with the dynamic DNS provider, publish the required record, and get the certificate for dyndns.example. It doesn't matter how much entropy DigiCert put in the record name. I definitely commend DigiCert for pledging to revoke the certificates within 24 hours and not having a delayed revocation or trying to language lawyer their way to a 5 day revocation as other CAs have tried. Nevertheless, this post severely minimizes the security impact of their mistake, and provides an excellent example of why CAs should always be required to strictly adhere to the rules and not be permitted to excuse noncompliance based on their own security analysis.

michaelt 2y ago
> For example, if an attacker requests a certificate for dyndns.example Shouldn't that get caught by the Public Suffix List? I would hope DigiCert has checks in place to prevent someone domain-validating ownership of the entire of co.uk under any circumstances :) (They should still revoke the mis-issued certificates though)
- remus 2y ago
  > Shouldn't that get caught by the Public Suffix List? PSL is a best-effort sort of thing, so it's good but not definitive. It would be dangerous to rely on it when issuing certs imo.
- nickf 2y ago
  PSL has a couple of sections - ICANN and PRIVATE. PRIVATE can be a little more flexible/ignorable. If they implement a hard rule, then occasionally they'd have to make exceptions when the real Dyn comes along and wants (legitimately) a wildcard for their name.

jiggawatts 2y ago

I just want to call out both CrowdStrike and DigiCert for being one of "those" companies that insist on publishing critical support information behind a login with the clock ticking on a global outage of their own making. There are no polite words that I can use to accurately convey the depth of my disappointment at this kind of inconsiderate behaviour during a crisis, so I won't say anything more.

Cyphase 2y ago
Are you referring to the list of affected certificates?
Brian_K_White 2y ago
What critical support information? What global outage? How are these two events or companies remotely equivalent? If I'm not a Digicert customer, what do I care about the details of how to redo a validation on Digicert? If I am a Digicert customer I have been emailed already and I will obviously have to log in to do anything at all with my domain. They say this affects 0.4% of Digicert customers who are what % of the world? Actually not even 0.4% of Digicert customers, but 0.4% of those particular validations. What does that actually work out to? Just who all is actually down? I fail to see any equivalence.
- deleted 2y ago
  [deleted]
  - deleted 2y ago
    [deleted]
freeone3000 2y ago
If you’re not a customer, your domain isn’t affected.
- deleted 2y ago
  [deleted]

256_ 2y ago

> While we had regression testing in place, those tests failed to alert us to the change in functionality because the regression tests were scoped to workflows and functionality instead of the content/structure of the random value. [...] > Unfortunately, no reviews were done to compare the legacy random value implementations with the random value implementations in the new system for every scenario. In other words, they didn't do proper testing. At the bottom of the article they suggest they're going to improve it.

ratg13 2y ago

24h notice to change certificates in who knows how many systems, at the worlds largest companies, while everyone is on vacation. This will be interesting.

devrand 2y ago
Respect to them for actually abiding by the BRs. Most CAs just shrug [1] and [2] say [3] it's [4] too [5] complicated [6], or just lie and claim planes will start crashing [7]. It's really disheartening that publicly trusted CAs just ignore their contractual obligations however they see fit. Ideally these companies should have response plans in place to prioritize certificate rotation. They can use this as a fire drill for what would happen if there were a key compromise. Alternatively, if companies cannot handle the rotation, then they likely should re-evaluate if WebPKI is even appropriate for their use-case. [1]: https://bugzilla.mozilla.org/show_bug.cgi?id=1885568 https://bugzilla.mozilla.org/show_bug.cgi?id=1885568 [2]: https://bugzilla.mozilla.org/show_bug.cgi?id=1898848 https://bugzilla.mozilla.org/show_bug.cgi?id=1898848 [3]: https://bugzilla.mozilla.org/show_bug.cgi?id=1910237 https://bugzilla.mozilla.org/show_bug.cgi?id=1910237 [4]: https://bugzilla.mozilla.org/show_bug.cgi?id=1896053 https://bugzilla.mozilla.org/show_bug.cgi?id=1896053 [5]: https://bugzilla.mozilla.org/show_bug.cgi?id=1896553 https://bugzilla.mozilla.org/show_bug.cgi?id=1896553 [6]: https://bugzilla.mozilla.org/show_bug.cgi?id=1877388 https://bugzilla.mozilla.org/show_bug.cgi?id=1877388 [7]: https://bugzilla.mozilla.org/show_bug.cgi?id=1903066#c48 https://bugzilla.mozilla.org/show_bug.cgi?id=1903066#c48
- hg35h4 2y ago
  "Alternatively, if companies cannot handle the rotation, then they likely should re-evaluate if WebPKI is even appropriate for their use-case." I hate hearing this awful take, as if every IT organization has the same neat and tidy systems deployed as they do. Never had to deal with 3rd party SaaS vendors certificate pinning requiring service tickets to change, don't have any hardware devices or appliance based software images each with their own web interface to update certs... Yes companies should have a plan to do their minimum yearly certificate rotates. Yes those companies should have a security plan to rotate affected certificate issues, but in those cases the business users are ok with an outage to remediate a real security issue. But what happened here is that Digicert invalided the entire domain's worth of certs. All those service.companyname.com certs or duplicates under that domain validation were affected in bulk. In some companies there could be thousands of certs under that domain. Digicert screwed up their system implementation and made their customers suffer. "It's really disheartening that publicly trusted CAs just ignore their contractual obligations however they see fit." It's also disheartening to see browsers in the CA consortium ignore the CA resolutions as well. Like how everyone voted for 2 year certs and Apple did their own thing anyways. Any punishment for Apple come? So why pick on the others?

Apfel 2y ago

Is this a potential cause of the current Azure outages hitting western europe? I know DigiCert are used by Azure extensively...

agwa 2y ago
Unlikely. Microsoft operates their own CAs. Some of their CAs have been cross-signed by a DigiCert root, but Microsoft is responsible for the domain validation. I don't think they extensively use certificates issued directly by a DigiCert CA.
- nickf 2y ago
  They have almost 300 in the affected list from DigiCert, so who knows?!

olliej 2y ago

One of the impacted companies filed a restraining order, because they believe their incompetence is more important than basic functionality of the PKI. Can't wait to hear how they expect to respond if they ever have encounter a cert compromise or actual misissuance, maybe they'll demand 24 hour revocation in that case? Honestly my opinion is that this should trigger the company being banned by all CAs. The company in question is Alegeus Technologies LLC: https://www.courtlistener.com/docket/68995396/alegeus-technologies-llc-v-digicert/ https://www.courtlistener.com/docket/68995396/alegeus-techno... From basic googling it looks like a healthcare provider, so exactly the kind of company you would want to have shitty IT and security infrastructure. A++ work. Absolutely stellar.

notemaker 2y ago

Can someone explain why this issue deserves a 24h notice? Seems more reasonable to me to have a much longer deprecation notice.

alex34778 2y ago
As far as I can tell, this issue would be a problem where all of the following conditions are met: 1. Tenants are allowed to create arbitrary subdomains with arbitrary CNAME values 2. Tenants are not authorized to act on behalf of the TLD directly, only on their respective subdomain 3. Tenants are ostensibly prevented from TLD cert issuance by being explicitly blocked from creating subdomains that start with underscores For most entities these conditions probably do not hold true anyway. But it could conceivably apply to certain free/dynamic dns providers, for example afraid.org and noip both allow arbitrary CNAMEs (though I checked my noip account and it wouldn't work anyway because of length limits on subdomains). I would guess that in act fact there are very few entities in existence for which this actually represents a potential threat against them, since it requires a very specific delineation of zone authorizations, but there might be a few. For most of Alegeus customers I doubt any of this applies, though, they're probably lucky to know their GoDaddy login to add any sort of DNS record, let alone have a whole system in place for less privileged users to create arbitrary CNAME records subject to controls over the use of underscores.