8 ms·
Heartbleed Update v3
- zoowar 12y agoI don't know akamai's relationship with open source projects, but it sounds like they had local changes and tried to karma whore them as, "trust us, our customers were protected, and we contribute to open source." I hope I'm wrong, or they deserve the public shaming they are receiving. Ultimately, we need more engagement from commercial organizations benefiting from open source projects.
- Artemis2 12y agoGood politics: they had custom code for parts of OpenSSL, they published it when it seemed needed, open-source community told them what was wrong, they fixed it and apologized. No bullshit.
- glimcat 12y agoAlso: the way this played out makes an awesome story for the next time Akamai experiences internal pushback against participating in open source. "Remember that one time it saved our bacon" is a political goldmine.
- notatoad 12y agoThem and everybody else. "Remember that time akamai ran vulnerable code in production for 13 years and the bug got patched two days after they open-sourced it" should be able to drive open source contributions at all kinds of companies.
- Consultant32452 12y agoIt won't matter. The response will be, "Without releasing the source no one would've ever found the bug. Now think of how many bugs were found that haven't been responsibly disclosed!"
- sobkas 12y agoThey are only in problem now, because they have released the patch. If no one outside the company saw that patch, they could spot the problem and fix it without anyone noticing and creating a PR disaster. They probably wouldn't, but they could. That is one argument someone could use against "participating in open source". It's a bit to late for saving bacon, patch should be publicly reviewed before it went into production. So no bacon saved, but there is a hope for a smoke detector.
- orblivion 12y agoI just want to say I don't think you deserve to get voted down (as you currently have been). I was thinking the same thing about this story - it's very interesting because it highlights the (utilitarian) philosophical battle around Open Source. It's really not totally clear in which case they were safer. Your position should at least be considered.
- ghshephard 12y agoIf they had not released the patch, then it's entirely possible that black hats would have been able to make off with the keys from their customers - and that would have been a real disaster.
- madaxe_again 12y agoNope. People will remember it as "remember all that work as a result of OpenSSL?"
- roel_v 12y agoLook, I like open source as much as the next guy, but this is myopic - one might as well say 'remember that one time we used open source stuff and we had to spend weeks scrambling to fix things?'. Please let's not fool ourselves into believing things that simply aren't true - it detracts from the things we can leverage.
- sjtgraham 12y agoI'm so grateful we have people like Willem Pinckaers uncovering these things.
- hrrsn 12y agoGood response from Akamai. No beating around the bush there.
- mycodebreaks 12y agoYes, they've been clear and honest in their communication.
- daeken 12y agoNo, this was not a good response. It was well written and took responsibility for bad code, but it failed to address the elephant in the room: why were keys not immediately rotated? I've said this a number of times lately, but it bears repeating that the golden rule of security is: If a compromise occurs, assume that everything is compromised. Even if their approach was tested to be perfect, they shouldn't have made the assumption that that actually reflected reality. They were extraordinarily irresponsible. There's simply no other way to slice it.
- blahedo 12y agoNo, that goes too far the other way. If I have legitimate reason to believe that an exploit does not affect me (especially if it's because I acknowledged its possibility and programmed defensively against it), why would I automatically act as if I were breached too? Your argument is totally correct if I am simply unsure---if I don't know whether an exploit affects me but I suspect it might, I assume the worst and act accordingly. But in this case Akamai had good reason to believe they were unaffected. (And, when someone demonstrated otherwise, they responded responsibly.)
- daeken 12y ago> If I have legitimate reason to believe that an exploit does not affect me (especially if it's because I acknowledged its possibility and programmed defensively against it), why would I automatically act as if I were breached too? If you have a memory read out of a process that contains sensitive data, you should always assume that the sensitive data was breached, even if you "programmed defensively against it". Period. No exceptions. They had good reason to believe they were unaffected, which is why they shouldn't have hit the "oh shit, the world is ending" button. But the decision to not roll their keys in the face of a breach was a massively irresponsible one. We have how many years of major bugs to look at? And in how many cases are the likes of Akamai (that is, organizations with theoretical mitigations in place) actually safe when they think they are? Always, always assume your protections are faulty in a way that a motivated attacker will figure out.
- sobkas 12y agoI would like to know how many security related patches they are keeping for themselves? And I think not only Akamai did this. I hope that this situation will prove that security needs a bit more patch reviews and a bit less of secret sauce.
- sweettea 12y agoI suspect their patches are oriented toward paranoia. Their particular patch (keeping what they believed to be all critical values in a special region of memory used only for critical things) had no obvious necessity -- it didn't obviously provide more security -- but was just paranoia. Paranoia isn't always necessary, or even desirable; hence, the remainder of the patches have no obvious external utility, so they presumably haven't released them. <p> They may also have performance improvements, conceivably, and those have a more obvious rationale for keeping secret.
- kevinr 12y agoAs Andy tweeted, we built that particular patch to keep unencrypted secrets off disk -- it actually was intended to provide tangible security benefit, it wasn't just paranoia: https://twitter.com/csoandy/status/455307255895060480 https://twitter.com/csoandy/status/455307255895060480
- Serow225 12y agoIf you folks have used this technique for years as I believe it has been stated, any particular reason why it wasn't contributed back to OpenSSL earlier?
- dsl 12y agoRight before this all came out our Akamai rep came on site and swore up and down they were not vulnerable. They need to quickly communicate internally as well that a vulnerability existed and send account managers back out to work on high priority cert rollovers.
- anonymouse01 12y agoOur rep said the same thing even when we insisted they rotate our certs on day 1. Of course we're still waiting. I don't know who's decision it was to not reissue all certs on day 1, but this is an epically bad decision bordering on gross negligence. Heads should roll.
- Yver 12y agoThe part I still don't get because I haven't seen it discussed so far is that their original patch mentionned they had been using a variant of the patch "for a decade" but I never got why they didn't seek to get it merged upstream a decade ago. Or why they released "a variant" of their patch instead of the real thing to begin with.
- forrestthewoods 12y agoThis does not inspire confidence in akamai for me. These days people on the internet seem to get super excited when various companies write post mortems on how they screwed up. Yay, they screwed up and were honest about it!!! I find those to be wildly overrated. I'd rather there no be a screw-up to begin with. And I don't particularly care if you come clean or not. I know bugs are written and bugs happen. It's an unfortunate fact of life. And anyone who posts their code will likely have holes poked in it by the community. But the akamai thing really sets of warning flags. The fact they thought it would work and it doesn't, at all, does not sit right.
- chubot 12y agoWhat's interesting is that I just watched a video that strongly relates to this problem last night (from DJB at CCC): https://www.youtube.com/watch?v=IuSnY_O8DqQ https://www.youtube.com/watch?v=IuSnY_O8DqQ http://facthacks.cr.yp.to/ http://facthacks.cr.yp.to/ It is only part of the video, but they talk about searching Google and finding partial private keys, e.g. ---- BEGIN PRIVATE KEY ---. Even with text missing, it is trivial to recover the entire key, because of redundancy that Akamai neglected to take into account. I never really thought about how those ASCII strings are encoded, but it's actually the 6 or so values encoded. If some are missing, you can recover the others. In retrospect, why shouldn't the private keyh have RFC 822-like key value format so this is obvious? I don't see why it has to be encoded at all; the format is unnecessarily obscure. The rest of this talk is also great. Most of it is accessible to non-cryptographers (like me), as long as you understand basic arithmetic and have an undergrad-level overview of public key crypto (which they give). They provide very readable and runnable Sage scripts. I really like the batch GCD attacks. Very elegant and effective.
- jonhohle 12y agoGreat talk, thanks for the link. I've used DJB's software for years, but this was the first talk I've seen.
- beambot 12y agoWillem's letter to Akamai addressed three areas where their patch was likely insufficient: (1) First question is then, are 'p', 'q' and 'd' the only sensitive parameters? NO. (The Chinese Remainder Theorem parameters are included in OpenSSL private keys by default and they are enough to recover the private key). (2) Second question: Does OpenSSL ever copy the private key, or parts, after the key has been read? YES. (3) Third question: Does OpenSSL allocate temporary variables for intermediates when performing operations with a private key. Can they be used to recover the private key? YES and YES. (Willem's post submitted to HN: https://news.ycombinator.com/item?id=7582990 https://news.ycombinator.com/item?id=7582990 ) This update from Akamai only addresses (1), AFAICT.
- mynameisvlad 12y agoWhy would their update need to address more than just (1)? That's the only part of the problem that they have to deal with. The OpenSSL team should be the one answering (2) and (3), since it's OpenSSL that copies the PK/parts after it's been read, or allocate temp vars for intermediates. If Akamai chooses to patch those two problems and then releases it to open source like their initial patch, then they can answer them, but otherwise, what's the point?
- Danieru 12y ago(2) & (3) are not bugs for OpenSSL because only Akamai is trying to segregate the key data. In upstream OpenSSL the intermediates are stored in the same memory pool as the key data. Thus the intermediates do not expose privileged data anywhere it is not already present. Without addressing issues 2 & 3 Akamai's patch is not a line of defense. It does not guard against dirty memory exploits like Heartbleed, it only increases the difficulty of exploitation.
- schneidmaster 12y agoThey also said > In parallel, we are evaluating the other claims made by the researcher, to understand what actions we can take to improve our customer protection. which seems to imply that they're at least looking into the other questions.
- Sami_Lehtinen 12y agoWhy people are only talking about SSL keys? Because Heartbleed allowed to fetch any data, that could have also included SSH keys and or any other authentication credentials / information the server got, for itself or to other services. As example if your site got Facebook API keys it's possible that those have been leaking too.
- jbg_ 12y agoUnless your web server makes SSH connections from some kind of in-process application, why would this bug allow SSH keys to leak?
- jevinskie 12y agoIs memory wiped with zeros by malloc/free in libc or in the kernel? If not, you're at risk.
- bzbarsky 12y agoYes, it is. Leaking memory contents across processes would be a pretty serious bug in the kernel. So when your process asks the kernel for memory (via sbrk or mmap) the memory it gets is zeroed out.
- tptacek 12y agoNo, your web server process memory does not include SSH keys.
- deleted 12y ago[deleted]
- daeken 12y agoWhen a company says "we're not changing our potentially compromised keys because we're protected by this magic box" and you punch a million holes in the magic box, your goal should be to make it well known that the protection is faulty. He accomplished that quickly, and now they're finally doing the right thing.
- deleted 12y ago[deleted]
- brians 12y agoWe're not doing anything we wouldn't have done in response to mail to security@akamai.com encrypted to pub 4096R/5B9283CE 2007-04-04 [expires: 2017-04-01] Key fingerprint = 4A76 FD9B 3603 C9F5 33F8 1490 49E0 5C45 5B92 83CE uid Akamai Security Team (General Public) <security@akamai.com> sub 2048R/DF985919 2013-03-29 [expires: 2015-03-29]
- daeken 12y agoIf the Heartbleed bug was still viable against your service, I'd say that he made the wrong call and should've emailed your team about it. But here, we're basically talking forensics. I've written at length about the decision to not roll the keys as soon as Heartbleed was discovered, so I won't repeat myself, but in short: him publicizing this, as well as the Cloudflare challenge immediately getting slapped down, should make it clear to everyone that if you have a memory read, you should assume everything is compromised. Hopefully this is a good thing for security in general.
- brians 12y agoThere I agree with you. Thanks.
- wdr1 12y agoInteresting to note that if Akamai hadn't open sourced their modifications, they never would have realized their certs actually had been compromised.
- joeybaker 12y agoThis is the perfect example to point people to when they ask why the company shouldn't just maintain an internal fork or write its own version of an open source project. Actually contributing the code you write is helps _everyone_.