8 ms·

GitHub and BitBucket's SSL Provider's Cert has expired

72 points by hiroprot 12y ago

huntaub 12y ago
I just got hit with this issue. There doesn't seem to be any information on DigiCert's site or Github's. edit: For some reason, deleting the expired DigiCert certificate from Keychain (and restarting Chrome) allowed it to find a valid chain to the Github certificate. I would recommend doing this if you want to get to Github without turning off SSL. edit2: (Or they just fixed it and I restarted Chrome.) Can anyone confirm that it works now (without deleting the Intermediate Cert)?
- bdon 12y ago
  https://twitter.com/flavmartins/status/493102588410494976 https://twitter.com/flavmartins/status/493102588410494976
- hiroprot 12y ago
  It doesn't for me.
Chronic29 12y ago
Has the royal family been informed?
jpdlla 12y ago
I'm always surprised at how often this tends to happen to many startups/companies
bonf 12y ago
bitbucket as well
- jpdlla 12y ago
  I thought you might have been joking. Weird.
  - bonf 12y ago
    I thought someone was after my private repositories
- hiroprot 12y ago
  Their SSL chain includes an expired DigiCert certificate as well.
- maximveksler 12y ago
  iwantmydomain as well https://twitter.com/mvxlr/status/493111861131227137 https://twitter.com/mvxlr/status/493111861131227137
pknerd 12y ago
I just upgraded the Cert from DigiCert Website. It's workable for 2038 now. Enough time. Who knows Github exist by that time or not.
flavmartins 12y ago
I work with the Certificate Provider DigiCert and confirm that the expired intermediate is a deprecated certificate no longer used in installs. Some sites still have it installed on their server or users might have it installed on their local machine. If you are on Chrome, follow @huntaub's suggestion and remove the expired certificate from keychain and restart. We've been notifying customers of the expiration and have Technical Support in the office 24 hours to help the sites who need help updating the certificate. We're also reaching out to the sites we see having issues online.
STRML 12y ago
Looks like digicert itself screwed up - getting an invalid certificate error on digicert.com. Their twitter feed says they are in contact with GitHub, DigitalOcean, Namecheap, Stripe, Pingdom, and so on. This was a big error, and even they made the mistake on their own root domain.
dzink 12y ago
I'm seeing the same issue on both GitHub and Heroku today as well. "Cannot connect to the real www.heroku.com Something is currently interfering with your secure connection to www.heroku.com. Try to reload this page in a few minutes or after switching to a new network. "
abritishguy 12y ago
What a fuck up. How did that go unnoticed.
- flavmartins 12y ago
  It was noticed, but tested across a number of platforms did not show errors except for Android < v3. The issue is with a weird Mac OS X chain issue that causes a chain to be downloaded to the login keystore in Keychain. Mac forces it to be used when validating the certificate chain. Most users have removed the cert and everything is working as it should. Tracking down how and why that happens on Mac OS X is tough. Reaching Apple engineers has not been extremely successful. Not Apple's fault. Usually SSL Root Chain groups are distributed with organizations so it's not always clear who to go to.
deleted 12y ago
[deleted]
D4AHNGM 12y ago
Is this only an issue for those using Google Chrome? I haven't had any SSL issues with Github all day using Firefox.
- maximveksler 12y ago
  No, git fetch from MacOSX 10.10 command line as well: Maxims-MacBook-Air:walk maximveksler$ git up Fetching origin fatal: unable to access ‘https://github.com/maximveksler/walk.git/': SSL certificate problem: Invalid certificate chain error: Could not fetch origin `git fetch` failed
rsanheim 12y ago
A fix to remove the expired cert right now: https://twitter.com/aarongraves/status/493116549599739905 https://twitter.com/aarongraves/status/493116549599739905 Pretty sure this is on Digicert's side, but we (at GitHub) are investigating to make sure of that.
pknerd 12y ago
Download your required certificate from here and it should work like charm https://www.digicert.com/digicert-root-certificates.htm https://www.digicert.com/digicert-root-certificates.htm
- alecco 12y ago
  This is the right solution. "DigiCert High Assurance EV Root CA" (Try test link before downloading). Add to KeyChain, restart browser.
pknerd 12y ago
This certificate expiration also caused not using Github and HomeBrew from CLI.
relix 12y ago
A side-project I'm working on will alert you when SSL certificates are about to expire, preventing these things from happening. It'll also show you a overview of all the expiration dates of your certificates and domains, updated automatically. It's not live yet, but if you're interested you can sign up for the launch mail here: http://www.domainsquire.com http://www.domainsquire.com
- teach 12y ago
  Such a thing would have been very helpful to me. My domain's SSL cert expired and I got a new one but didn't notice that my provider had shipped a different, newer intermediate certificate. For three weeks I was showing a great big THIS CONNECTION IS UNTRUSTED screen to Firefox users and didn't know it.
  - relix 12y ago
    It surprises me how common it is, which made me build Domainsquire to fix it. As today's event shows it even happens to the big boys, and this is a predictable, pre-emptively solvable downtime event, so it really shouldn't happen.
- rayshan 12y ago
  Thank you for building this. Signed up.
  - relix 12y ago
    Cheers!
ab 12y ago
If this is anything like the issues we've seen at Stripe, the problem is probably an obsolete cross-signed root in your login keychain. It's caused by a certificate with CN="DigiCert High Assurance EV Root CA" but signed by some other authority rather than being self-signed. It's not clear to us how these are getting into people's login keychains, as they're not present on a fresh install. Typically servers will present their certificate and intermediates but not the root, under the assumption that browsers must already have the root in their CA store. So for DigiCert that would probably be all the certs up to but not including "DigiCert High Assurance EV Root CA". You can see the presented cert chain using `openssl s_client -showcerts ...` or the Certification Paths section of the Qualys SSL Labs Test: https://www.ssllabs.com/ssltest/analyze.html?d=github.com https://www.ssllabs.com/ssltest/analyze.html?d=github.com Do you see an expired "DigiCert High Assurance EV Root CA" certificate in your login keychain? If so, delete it. If not, something weirder may be going on.
- nknighthb 12y ago
  That is interesting, it's in my login keychain, as well, alongside "DigiCert Global CA" that expired on the 14th. The most obvious answer would be that it's being installed by some widely-used piece of software, but I don't know what. Pity the "Date Modified" column is empty, and I don't think there's really a log of what added things to the keychain.
  - ab 12y ago
    I'd love to hear theories for what might have installed it. If anyone still has that certificate, it would be helpful to export it and email it to support@stripe.com. We've worked around the issue for now by not using EV certificates, which isn't a great solution.
    - nknighthb 12y ago
      I've sent both the recently expired DigiCert certs. Could be just about anything. In my case, my keychain has followed me from one Mac to the next since before the cert that expired today was ever issued, so it could have ended up in there anytime in the last ~8 years from anything I might have had installed dating back to my PowerBook G4... Virtualization software might be a candidate. Actually, I just had another thought: Steam. And when I just tried to go to https://store.steampowered.com/ https://store.steampowered.com/, guess what certificate is in the trust chain?
flavmartins 12y ago
For a full explanation on the legacy intermediate explanation and affected users see DigiCert's post: https://blog.digicert.com/expired-intermediate-certificate/ https://blog.digicert.com/expired-intermediate-certificate/
- jcheng 12y ago
  Thanks, very helpful, though I think affected users are going to want to use this link instead: http://blog.digicert.com/expired-intermediate-certificate/ http://blog.digicert.com/expired-intermediate-certificate/
zizee 12y ago
Hah! I'm working on a side project to solve this problem: http://www.renewalmonitor.com/ http://www.renewalmonitor.com/ The idea is that the service will monitor things like domains and ssl expiry dates and then alert you in an increasingly obnoxious manner as the expiration date gets closer. My MVP has just needs a few more finishing touches and then I'll send it live. In the meantime, you can signup on the waiting list. Cheers.
gianpaj 12y ago
step tutorial http://gianpaj.com/post/93100630815/cant-access-github-on-chrome-os-x http://gianpaj.com/post/93100630815/cant-access-github-on-ch...
robermiranda 12y ago
not sure why, but i had to remove all the certificates and download its from here https://www.digicert.com/digicert-root-certificates.htm https://www.digicert.com/digicert-root-certificates.htm
joefiorini 12y ago
I'm having this issue as well. I deleted all digicert certificates from my keychain just in case. Still couldn't get to Github. I can get to the DigiCert Root Certificates download page, but it gives me an invalid certificate warning. It looks like the same issue as Github. I really, really don't feel comfortable downloading a ROOT CERTIFICATE with an SSL warning on the page. Who knows what could be compromised in this case? I'm going to try a couple other things first; I'd like to hear from a security expert, should we find this scary or just a small hiccup?
- joefiorini 12y ago
  I had an outstanding OSX update, installed that and rebooted. One of the two fixed it for me. Note: I DID NOT have to install the the root certificates, and if anyone else gets an SSL warning from DigiCert's root cert download site, I strongly recommend against downloading anything from there.

72 points by hiroprot 12y ago

I just got hit with this issue. There doesn't seem to be any information on DigiCert's site or Github's. edit: For some reason, deleting the expired DigiCert certificate from Keychain (and restarting Chrome) allowed it to find a valid chain to the Github certificate. I would recommend doing this if you want to get to Github without turning off SSL. edit2: (Or they just fixed it and I restarted Chrome.) Can anyone confirm that it works now (without deleting the Intermediate Cert)?

bdon 12y ago
https://twitter.com/flavmartins/status/493102588410494976 https://twitter.com/flavmartins/status/493102588410494976
hiroprot 12y ago
It doesn't for me.

Chronic29 12y ago

Has the royal family been informed?

jpdlla 12y ago

I'm always surprised at how often this tends to happen to many startups/companies

bonf 12y ago

bitbucket as well

jpdlla 12y ago
I thought you might have been joking. Weird.
- bonf 12y ago
  I thought someone was after my private repositories
hiroprot 12y ago
Their SSL chain includes an expired DigiCert certificate as well.
maximveksler 12y ago
iwantmydomain as well https://twitter.com/mvxlr/status/493111861131227137 https://twitter.com/mvxlr/status/493111861131227137

pknerd 12y ago

I just upgraded the Cert from DigiCert Website. It's workable for 2038 now. Enough time. Who knows Github exist by that time or not.

flavmartins 12y ago

I work with the Certificate Provider DigiCert and confirm that the expired intermediate is a deprecated certificate no longer used in installs. Some sites still have it installed on their server or users might have it installed on their local machine. If you are on Chrome, follow @huntaub's suggestion and remove the expired certificate from keychain and restart. We've been notifying customers of the expiration and have Technical Support in the office 24 hours to help the sites who need help updating the certificate. We're also reaching out to the sites we see having issues online.

STRML 12y ago

Looks like digicert itself screwed up - getting an invalid certificate error on digicert.com. Their twitter feed says they are in contact with GitHub, DigitalOcean, Namecheap, Stripe, Pingdom, and so on. This was a big error, and even they made the mistake on their own root domain.

dzink 12y ago

I'm seeing the same issue on both GitHub and Heroku today as well. "Cannot connect to the real www.heroku.com Something is currently interfering with your secure connection to www.heroku.com. Try to reload this page in a few minutes or after switching to a new network. "

abritishguy 12y ago

What a fuck up. How did that go unnoticed.

flavmartins 12y ago
It was noticed, but tested across a number of platforms did not show errors except for Android < v3. The issue is with a weird Mac OS X chain issue that causes a chain to be downloaded to the login keystore in Keychain. Mac forces it to be used when validating the certificate chain. Most users have removed the cert and everything is working as it should. Tracking down how and why that happens on Mac OS X is tough. Reaching Apple engineers has not been extremely successful. Not Apple's fault. Usually SSL Root Chain groups are distributed with organizations so it's not always clear who to go to.

deleted 12y ago

[deleted]

D4AHNGM 12y ago

Is this only an issue for those using Google Chrome? I haven't had any SSL issues with Github all day using Firefox.

maximveksler 12y ago
No, git fetch from MacOSX 10.10 command line as well: Maxims-MacBook-Air:walk maximveksler$ git up Fetching origin fatal: unable to access ‘https://github.com/maximveksler/walk.git/': SSL certificate problem: Invalid certificate chain error: Could not fetch origin `git fetch` failed

rsanheim 12y ago

A fix to remove the expired cert right now: https://twitter.com/aarongraves/status/493116549599739905 https://twitter.com/aarongraves/status/493116549599739905 Pretty sure this is on Digicert's side, but we (at GitHub) are investigating to make sure of that.

pknerd 12y ago

Download your required certificate from here and it should work like charm https://www.digicert.com/digicert-root-certificates.htm https://www.digicert.com/digicert-root-certificates.htm

alecco 12y ago
This is the right solution. "DigiCert High Assurance EV Root CA" (Try test link before downloading). Add to KeyChain, restart browser.

pknerd 12y ago

This certificate expiration also caused not using Github and HomeBrew from CLI.

relix 12y ago

A side-project I'm working on will alert you when SSL certificates are about to expire, preventing these things from happening. It'll also show you a overview of all the expiration dates of your certificates and domains, updated automatically. It's not live yet, but if you're interested you can sign up for the launch mail here: http://www.domainsquire.com http://www.domainsquire.com

teach 12y ago
Such a thing would have been very helpful to me. My domain's SSL cert expired and I got a new one but didn't notice that my provider had shipped a different, newer intermediate certificate. For three weeks I was showing a great big THIS CONNECTION IS UNTRUSTED screen to Firefox users and didn't know it.
- relix 12y ago
  It surprises me how common it is, which made me build Domainsquire to fix it. As today's event shows it even happens to the big boys, and this is a predictable, pre-emptively solvable downtime event, so it really shouldn't happen.
rayshan 12y ago
Thank you for building this. Signed up.
- relix 12y ago
  Cheers!

ab 12y ago

If this is anything like the issues we've seen at Stripe, the problem is probably an obsolete cross-signed root in your login keychain. It's caused by a certificate with CN="DigiCert High Assurance EV Root CA" but signed by some other authority rather than being self-signed. It's not clear to us how these are getting into people's login keychains, as they're not present on a fresh install. Typically servers will present their certificate and intermediates but not the root, under the assumption that browsers must already have the root in their CA store. So for DigiCert that would probably be all the certs up to but not including "DigiCert High Assurance EV Root CA". You can see the presented cert chain using `openssl s_client -showcerts ...` or the Certification Paths section of the Qualys SSL Labs Test: https://www.ssllabs.com/ssltest/analyze.html?d=github.com https://www.ssllabs.com/ssltest/analyze.html?d=github.com Do you see an expired "DigiCert High Assurance EV Root CA" certificate in your login keychain? If so, delete it. If not, something weirder may be going on.

nknighthb 12y ago
That is interesting, it's in my login keychain, as well, alongside "DigiCert Global CA" that expired on the 14th. The most obvious answer would be that it's being installed by some widely-used piece of software, but I don't know what. Pity the "Date Modified" column is empty, and I don't think there's really a log of what added things to the keychain.
- ab 12y ago
  I'd love to hear theories for what might have installed it. If anyone still has that certificate, it would be helpful to export it and email it to support@stripe.com. We've worked around the issue for now by not using EV certificates, which isn't a great solution.
  - nknighthb 12y ago
    I've sent both the recently expired DigiCert certs. Could be just about anything. In my case, my keychain has followed me from one Mac to the next since before the cert that expired today was ever issued, so it could have ended up in there anytime in the last ~8 years from anything I might have had installed dating back to my PowerBook G4... Virtualization software might be a candidate. Actually, I just had another thought: Steam. And when I just tried to go to https://store.steampowered.com/ https://store.steampowered.com/, guess what certificate is in the trust chain?

flavmartins 12y ago

For a full explanation on the legacy intermediate explanation and affected users see DigiCert's post: https://blog.digicert.com/expired-intermediate-certificate/ https://blog.digicert.com/expired-intermediate-certificate/

jcheng 12y ago
Thanks, very helpful, though I think affected users are going to want to use this link instead: http://blog.digicert.com/expired-intermediate-certificate/ http://blog.digicert.com/expired-intermediate-certificate/

zizee 12y ago

Hah! I'm working on a side project to solve this problem: http://www.renewalmonitor.com/ http://www.renewalmonitor.com/ The idea is that the service will monitor things like domains and ssl expiry dates and then alert you in an increasingly obnoxious manner as the expiration date gets closer. My MVP has just needs a few more finishing touches and then I'll send it live. In the meantime, you can signup on the waiting list. Cheers.

gianpaj 12y ago

step tutorial http://gianpaj.com/post/93100630815/cant-access-github-on-chrome-os-x http://gianpaj.com/post/93100630815/cant-access-github-on-ch...

robermiranda 12y ago

not sure why, but i had to remove all the certificates and download its from here https://www.digicert.com/digicert-root-certificates.htm https://www.digicert.com/digicert-root-certificates.htm

joefiorini 12y ago

I'm having this issue as well. I deleted all digicert certificates from my keychain just in case. Still couldn't get to Github. I can get to the DigiCert Root Certificates download page, but it gives me an invalid certificate warning. It looks like the same issue as Github. I really, really don't feel comfortable downloading a ROOT CERTIFICATE with an SSL warning on the page. Who knows what could be compromised in this case? I'm going to try a couple other things first; I'd like to hear from a security expert, should we find this scary or just a small hiccup?

joefiorini 12y ago
I had an outstanding OSX update, installed that and rebooted. One of the two fixed it for me. Note: I DID NOT have to install the the root certificates, and if anyone else gets an SSL warning from DigiCert's root cert download site, I strongly recommend against downloading anything from there.