8 ms·

Microsoft failed to rotate certificates for winget CDN on time

111 points by yjh0502 4y ago

fsckboy 4y ago
reminds me of that time the regular guy, might've been a student, re-registered hotmail.com just to get his email working again after Microsoft let it expire. oh, looks like it was either hotmail.co.uk or passport.com https://slashdot.org/story/99/12/25/114201/microsoft-hotmailpassport-service-interruptedupdated https://slashdot.org/story/99/12/25/114201/microsoft-hotmail... from https://whoapi.com/blog/5-all-time-domain-expirations-in-internets-history/ https://whoapi.com/blog/5-all-time-domain-expirations-in-int...
vbezhenar 4y ago
While I appreciate TLS, this thing with certificate expiration is one of the biggest sources of downtime IMO. Something should be done about it. May be throw error not permanently but in a some probabilistic way. Like if 1 year certificate expired, after 3 months 25% of connections would fail. It'll allow eventually to find out about problem but it'll allow for connections to somewhat work, with few retries here an there. Expired certificate is not compromised certificate and should not be treated like one. Often next certificate is issued with the same private key. Especially with short-lived letsencrypt certificates. Despite all the evangelists assurances, certbot is not always easy to set up. After letsencrypt gained popularity, the percentage of small websites with expired certificates significantly increased IMO.
- tbrownaw 4y ago
  I don't like intermittent bugs.
- TedDoesntTalk 4y ago
  We already have CRL lists. So Why do we need certificate expiration?
  - 8organicbits 4y ago
    When a certificate expires, it can be removed from the CRL. If certificates never expire then the CRL grows without bound. Also, checking CRL is implemented in different ways. Some checks may be "soft", where a connection failure to the CRL is ignored. You probably want this anyway, if the CRL goes offline you don't want the internet to break. An expiry check, on the other hand, works as long as your clock is accurate.
    - moremetadata 4y ago
      You are forgetting something. If you want to reinstall some old software, lets say MS Small Business Server 2000 or Small Business Server 2003 today, the certificates in the installation files prevent the installation of said software. So you wouldnt even get as far as being able to remove any certs. Your only recourse is change the system date and time back to before the certificates in installation files would have expired. Besides being a stealth way to prevent old software from being reinstalled, it narrows down the window of opportunity for hackers. I used to automatically issue certs for my own servers which lasted 24hr's because if a hacker had got in to my system without me knowing which is a real possibility, at least an expired cert being used by someone else would highlight this problem. As it happened, despite locking everything down to packet level and controlling the packets, my devices were just prevented from getting online. My ISP at the time TalkTalk had a very responsive system, issuing new IP address every 2 seconds in a bid to prevent me from hosting a website, with a domain name using dynamic ip address domain name service. There is way more surveillance than most people realise at least here in the UK.
      - t0mas88 4y ago
        What? 2 seconds? That sounds crazy. It either breaks ongoing connections or wastes addresses since you would have to keep your old one until all connections are closed.
        derefr 4y ago
        It probably wasn't really "issuing a new IP address" per se, but rather CGNAT, where your apparent IP from the perspective of an IP reflector would be the IP of whichever NAT gateway your outgoing connection had been round-robin-load-balanced onto. Under CGNAT, you don't really have any single public IP; or rather, in another sense, you (and 100k other people) "have" all N public IPs at once — just like devices on a NATed home network all share the one IP address assigned to the gateway router in front of the NAT, and would all "have" multiple addresses if that gateway-router were multi-homed.
- linsomniac 4y ago
  Making silent, intermittent failures for 3 months sounds so, so much worse than just having it 100% fail at expiration. What are we trying to fix here?
- physicles 4y ago
  I'd take it as a canary for how much the site owner actually cares about ops and security. If you can't be bothered to take the day or so to set up certbot and monitoring for when your certs are ~15 days from expiration, then that's very likely not the only thing you've cut corners on.
josephg 4y ago
I think the default certificate expiration time (2 years) is a terrible idea. Its long enough that there's a good chance whoever registered the cert last time has left the team or the company. Its long enough that I've forgotten how to generate a certificate with openssl on the command line. And its long enough that each time, I (and everyone else) can justify not bothering to automate the process. But 2 years is still short enough that if you have a couple domains, remembering to renew them is an ongoing hassle! Letsencrypt certificates last 90 days, and they recommend renewing them every 60 days. This is a much better duration, because it encourages the entire ecosystem - developers and admins - to set up processes which automate renewal. And if the automated renewal process fails, letsencrypt starts emailing you about it to let you know your certificate is about to expire. (And you have enough time to fix it). https://letsencrypt.org/2015/11/09/why-90-days.html https://letsencrypt.org/2015/11/09/why-90-days.html
- Uvix 4y ago
  The maximum expiration time is now down to 13 months, for certs that need to be valid in a browser. And if you want to cycle yours more frequently, you can. But there's enough places that can't set up automated processes that trying to make it 90 days for everyone would be a lot of pain and a lot of broken sites.
  - josephg 4y ago
    > But there's enough places that can't set up automated processes Why can't they be automated? And anyway, this is the exact problem that short expiration times avoid! Systems that aren't set up for automation, and rely on someone once a year remembering some creaky, error prone process to get a new cert. Much better to force short expiration times so manual cert renewal is a thing of the past.
    - voytec 4y ago
      DV is not the only kind of certificates validation. I don't want to have to go through the OV/EV validation process several times a year, nor to validate 4 certificate issuances a year in advance. But if I wanted to, I can do so even now without being forced - request new certificate during it's validity period, and revoke the former one.
aliswe 4y ago
If only there was a cloud based solution by a large company for managing certificates automatically! (Azure Front Door)
- Suirtimed 4y ago
  The certificate was renewed, but it wasn't deployed correctly. We're looking into the root cause.
  - aliswe 4y ago
    I'm just thankful we switched to afd and never have to think about certs (neither issuing nor deploying) again. You should check it out!
Suirtimed 4y ago
We're working on renewing the certificate. It looks like this is the first report: https://github.com/microsoft/winget-cli/issues/2956 https://github.com/microsoft/winget-cli/issues/2956
Jamie9912 4y ago
Aaaand it's still not fixed. I think this just goes to show how much red tape there is around processes at Microsoft.
- janoc 4y ago
  You realize it is Sunday, right? And that probably a lot of relevant people are not in the office today, even if the techops grunts who are on call are scrambling.
  - linker3000 4y ago
    Day of the week is irrelevant for an organisation that big. If you can't escalate a major incident like this to a designated person - even if you have to wake them up - your process and business continuity plans are seriously flawed.
- Suirtimed 4y ago
  https://github.com/microsoft/winget-cli/issues/2956#issuecomment-1426968138 https://github.com/microsoft/winget-cli/issues/2956#issuecom... We updated the certificate about an hour before this post. It takes 6 - 8 hours for the certificate to fully propagate.
SillyUsername 4y ago
Amateurs.
8n4vidtmkvmk 4y ago
I think they've been bad stewards of that project since they stole it from that guy. it had promise but they refused to add basic features
zinekeller 4y ago
Looking at crt.sh (https://crt.sh/?q=cdn.winget.microsoft.com https://crt.sh/?q=cdn.winget.microsoft.com), it seems that the certificate is issued automatically anyways but for some reason the updated certificate is not applied correctly. A bad screwup really, but more of did someone forget to check their logs for deployment errors rather than the common case of someone forgetting to manually update the certificate.
sqldba 4y ago
ITT: Lots of people feigning ignorance about how companies work, or have literally never worked for a large company before. And hilariously thinking Microsoft would spin up a critical incident team for a free open-source product. I'm rolling on the floor laughing.
kasajian 4y ago
Why would anyone standardize on winget when there's chocolatey and it works everywhere you can run Windows software? Recently I decided to install Windows Server on one of my PC instead of Windows 10/11. I thought, why not attempt to use winget and/or the Windows App Store to install basic software to see how far that takes me. Immediate dead end. Apparently there's no access from the App Store on Windows Server. You have to have your IT set it up. And all the documentation refer to using winget from the App Store. :/ Microsoft, I can't even with you right now. With Windows Server 2022, apparently you can get winget to work, but you have to install something else or other that's in preview. For real, guys. Please get with the program. Possible workarounds: Anything that's a workaround is already a non-starter for me because I'm not trying to experiment to see what I can get to work. I'm trying to move to using a new norm so I don't get left behind. If Microsoft was pushing winget as the new norm for global silent command-line installations of all things Windows, as an alternative to Chocolately, great! I would have given it a try. Other than that, imma just wait until y'all get your act together, <i>someday-maybe</i>.
- spixy 4y ago
  because winget is built-in
- jonathantf2 4y ago
  Not sure why you'd be trying to run Server on a workstation?
no_identd 4y ago
Microsoft's certificate management skills have gone down the drain anyway, so this doesn't surprise me. I have a long standing support case open with them about how they ship one of their more obscure tools signed with the wrong code signing certificate (one signed by their PKI for Azure INTERNAL usage, which should have everyone a tiny bit worried), and I've pretty much given up on trying to get the quite-obviously-on-an-H1B developer (which I mention only to explain that this PROBABLY leads to a perverse incentive to sweep things under the rug), or any of the Indian support agents involved to comprehend that this isn't just inconvenient when one has AppLocker in place, but also that it violates Microsoft's internal policies (which I know for a fact that it does), and MSRC ignored my email about it, too, so… par for the course. ¯\_(ツ)_/¯

111 points by yjh0502 4y ago

reminds me of that time the regular guy, might've been a student, re-registered hotmail.com just to get his email working again after Microsoft let it expire. oh, looks like it was either hotmail.co.uk or passport.com https://slashdot.org/story/99/12/25/114201/microsoft-hotmailpassport-service-interruptedupdated https://slashdot.org/story/99/12/25/114201/microsoft-hotmail... from https://whoapi.com/blog/5-all-time-domain-expirations-in-internets-history/ https://whoapi.com/blog/5-all-time-domain-expirations-in-int...

vbezhenar 4y ago

While I appreciate TLS, this thing with certificate expiration is one of the biggest sources of downtime IMO. Something should be done about it. May be throw error not permanently but in a some probabilistic way. Like if 1 year certificate expired, after 3 months 25% of connections would fail. It'll allow eventually to find out about problem but it'll allow for connections to somewhat work, with few retries here an there. Expired certificate is not compromised certificate and should not be treated like one. Often next certificate is issued with the same private key. Especially with short-lived letsencrypt certificates. Despite all the evangelists assurances, certbot is not always easy to set up. After letsencrypt gained popularity, the percentage of small websites with expired certificates significantly increased IMO.

tbrownaw 4y ago
I don't like intermittent bugs.
TedDoesntTalk 4y ago
We already have CRL lists. So Why do we need certificate expiration?
- 8organicbits 4y ago
  When a certificate expires, it can be removed from the CRL. If certificates never expire then the CRL grows without bound. Also, checking CRL is implemented in different ways. Some checks may be "soft", where a connection failure to the CRL is ignored. You probably want this anyway, if the CRL goes offline you don't want the internet to break. An expiry check, on the other hand, works as long as your clock is accurate.
  - moremetadata 4y ago
    You are forgetting something. If you want to reinstall some old software, lets say MS Small Business Server 2000 or Small Business Server 2003 today, the certificates in the installation files prevent the installation of said software. So you wouldnt even get as far as being able to remove any certs. Your only recourse is change the system date and time back to before the certificates in installation files would have expired. Besides being a stealth way to prevent old software from being reinstalled, it narrows down the window of opportunity for hackers. I used to automatically issue certs for my own servers which lasted 24hr's because if a hacker had got in to my system without me knowing which is a real possibility, at least an expired cert being used by someone else would highlight this problem. As it happened, despite locking everything down to packet level and controlling the packets, my devices were just prevented from getting online. My ISP at the time TalkTalk had a very responsive system, issuing new IP address every 2 seconds in a bid to prevent me from hosting a website, with a domain name using dynamic ip address domain name service. There is way more surveillance than most people realise at least here in the UK.
    - t0mas88 4y ago
      What? 2 seconds? That sounds crazy. It either breaks ongoing connections or wastes addresses since you would have to keep your old one until all connections are closed.
      - derefr 4y ago
        It probably wasn't really "issuing a new IP address" per se, but rather CGNAT, where your apparent IP from the perspective of an IP reflector would be the IP of whichever NAT gateway your outgoing connection had been round-robin-load-balanced onto. Under CGNAT, you don't really have any single public IP; or rather, in another sense, you (and 100k other people) "have" all N public IPs at once — just like devices on a NATed home network all share the one IP address assigned to the gateway router in front of the NAT, and would all "have" multiple addresses if that gateway-router were multi-homed.
linsomniac 4y ago
Making silent, intermittent failures for 3 months sounds so, so much worse than just having it 100% fail at expiration. What are we trying to fix here?
physicles 4y ago
I'd take it as a canary for how much the site owner actually cares about ops and security. If you can't be bothered to take the day or so to set up certbot and monitoring for when your certs are ~15 days from expiration, then that's very likely not the only thing you've cut corners on.

josephg 4y ago

I think the default certificate expiration time (2 years) is a terrible idea. Its long enough that there's a good chance whoever registered the cert last time has left the team or the company. Its long enough that I've forgotten how to generate a certificate with openssl on the command line. And its long enough that each time, I (and everyone else) can justify not bothering to automate the process. But 2 years is still short enough that if you have a couple domains, remembering to renew them is an ongoing hassle! Letsencrypt certificates last 90 days, and they recommend renewing them every 60 days. This is a much better duration, because it encourages the entire ecosystem - developers and admins - to set up processes which automate renewal. And if the automated renewal process fails, letsencrypt starts emailing you about it to let you know your certificate is about to expire. (And you have enough time to fix it). https://letsencrypt.org/2015/11/09/why-90-days.html https://letsencrypt.org/2015/11/09/why-90-days.html

Uvix 4y ago
The maximum expiration time is now down to 13 months, for certs that need to be valid in a browser. And if you want to cycle yours more frequently, you can. But there's enough places that can't set up automated processes that trying to make it 90 days for everyone would be a lot of pain and a lot of broken sites.
- josephg 4y ago
  > But there's enough places that can't set up automated processes Why can't they be automated? And anyway, this is the exact problem that short expiration times avoid! Systems that aren't set up for automation, and rely on someone once a year remembering some creaky, error prone process to get a new cert. Much better to force short expiration times so manual cert renewal is a thing of the past.
  - voytec 4y ago
    DV is not the only kind of certificates validation. I don't want to have to go through the OV/EV validation process several times a year, nor to validate 4 certificate issuances a year in advance. But if I wanted to, I can do so even now without being forced - request new certificate during it's validity period, and revoke the former one.

aliswe 4y ago

If only there was a cloud based solution by a large company for managing certificates automatically! (Azure Front Door)

Suirtimed 4y ago
The certificate was renewed, but it wasn't deployed correctly. We're looking into the root cause.
- aliswe 4y ago
  I'm just thankful we switched to afd and never have to think about certs (neither issuing nor deploying) again. You should check it out!

Suirtimed 4y ago

We're working on renewing the certificate. It looks like this is the first report: https://github.com/microsoft/winget-cli/issues/2956 https://github.com/microsoft/winget-cli/issues/2956

Jamie9912 4y ago

Aaaand it's still not fixed. I think this just goes to show how much red tape there is around processes at Microsoft.

janoc 4y ago
You realize it is Sunday, right? And that probably a lot of relevant people are not in the office today, even if the techops grunts who are on call are scrambling.
- linker3000 4y ago
  Day of the week is irrelevant for an organisation that big. If you can't escalate a major incident like this to a designated person - even if you have to wake them up - your process and business continuity plans are seriously flawed.
Suirtimed 4y ago
https://github.com/microsoft/winget-cli/issues/2956#issuecomment-1426968138 https://github.com/microsoft/winget-cli/issues/2956#issuecom... We updated the certificate about an hour before this post. It takes 6 - 8 hours for the certificate to fully propagate.

SillyUsername 4y ago

Amateurs.

8n4vidtmkvmk 4y ago

I think they've been bad stewards of that project since they stole it from that guy. it had promise but they refused to add basic features

zinekeller 4y ago

Looking at crt.sh (https://crt.sh/?q=cdn.winget.microsoft.com https://crt.sh/?q=cdn.winget.microsoft.com), it seems that the certificate is issued automatically anyways but for some reason the updated certificate is not applied correctly. A bad screwup really, but more of did someone forget to check their logs for deployment errors rather than the common case of someone forgetting to manually update the certificate.

sqldba 4y ago

ITT: Lots of people feigning ignorance about how companies work, or have literally never worked for a large company before. And hilariously thinking Microsoft would spin up a critical incident team for a free open-source product. I'm rolling on the floor laughing.

kasajian 4y ago

Why would anyone standardize on winget when there's chocolatey and it works everywhere you can run Windows software? Recently I decided to install Windows Server on one of my PC instead of Windows 10/11. I thought, why not attempt to use winget and/or the Windows App Store to install basic software to see how far that takes me. Immediate dead end. Apparently there's no access from the App Store on Windows Server. You have to have your IT set it up. And all the documentation refer to using winget from the App Store. :/ Microsoft, I can't even with you right now. With Windows Server 2022, apparently you can get winget to work, but you have to install something else or other that's in preview. For real, guys. Please get with the program. Possible workarounds: Anything that's a workaround is already a non-starter for me because I'm not trying to experiment to see what I can get to work. I'm trying to move to using a new norm so I don't get left behind. If Microsoft was pushing winget as the new norm for global silent command-line installations of all things Windows, as an alternative to Chocolately, great! I would have given it a try. Other than that, imma just wait until y'all get your act together, <i>someday-maybe</i>.

spixy 4y ago
because winget is built-in
jonathantf2 4y ago
Not sure why you'd be trying to run Server on a workstation?

no_identd 4y ago

Microsoft's certificate management skills have gone down the drain anyway, so this doesn't surprise me. I have a long standing support case open with them about how they ship one of their more obscure tools signed with the wrong code signing certificate (one signed by their PKI for Azure INTERNAL usage, which should have everyone a tiny bit worried), and I've pretty much given up on trying to get the quite-obviously-on-an-H1B developer (which I mention only to explain that this PROBABLY leads to a perverse incentive to sweep things under the rug), or any of the Indian support agents involved to comprehend that this isn't just inconvenient when one has AppLocker in place, but also that it violates Microsoft's internal policies (which I know for a fact that it does), and MSRC ignored my email about it, too, so… par for the course. ¯\_(ツ)_/¯