5 ms·

Fighting Cargo Cult – An Incomplete SSL/TLS Bookmark Collection

80 points by danimo 12y ago

mrtksn 12y ago
With the rise of the cloud, now everybody manages a server. I am curious about how many of the admins actually know what are they doing. If I have to speak for myself, I set up my first ssl by following a guide that seemed to written by reasonably well informed stranger. It was my first exposure to the topic so I could not tell what I am doing until I break things and start reading deeper materials so that I can debug the problems by understanding what's going on. Turns out that the stranger who wrote the tutorial I followed was not well informed at all.
- danieltillett 12y ago
  A lot of sys admins are there by default. I would consider myself one of these - I would love to be able off load this work onto someone else who knows more than me, but there just isn't anyone.
  - danimo 12y ago
    Essentially this comes down to pressuring distros and server vendors do their homework finally ship with good examples/defaults. E.g. Microsoft IIS (!) has OCSP stapling enabled by default since ages. Apache? Most people still run 2.2, which isn't capable of OCSP stapling at all. Nginx is in a similar position. That said, the more sysadmins rely on <s>rotten</s>well-proven "Enterprise Linuxes" and "LTS" versions with old libraries and servers, the more security expertise is required from sysadmins to decide where to deviate from the distros default packages to meet current best practices. On the other hand, security is a moving target and knowing your (Open)SSL setup is as important as e.g. knowing your RoR setup. It's an inconvenient truth, because it requires learning new stuff. I don't see any alternative though, that's why I compiled this material. Finally, a remark on the "offloading" part: Security is the single thing where delegation becomes hard because it means delegating trust, as in: your private SSL/TLS keys. And that's quite some trust to delegate.
    - danieltillett 12y ago
      There is so much I agree with in your post. The more I learn in this area the less sure I am that I am doing the right things.
    - jarrett 12y ago
      > knowing your (Open)SSL setup is as important as e.g. knowing your RoR setup Very true, and yet it's so much harder to know your OpenSSL or other security-related setup. You learn your Rails setup well enough to make your application work, and hopefully well enough to make it performant. If you miss either of those goals, it's obvious to you. You know something's broken, and you grind away until you fix it. Not so with security. Your system can be "working" by all outward appearances yet be riddled with vulnerabilities. And you won't know it, so you won't see any reason to go and learn more. Nor would you know what you don't know, or where you need to learn more. That's the scary part.
- danimo 12y ago
  Yes, that's the exact problem that made me write this. What's particularly amazing is the amount of magical cipher suite strings shared throughout the web, most of which do not take in account PFS, or still prioritize RC4. All of that was acceptable at some point in time. Other cipher lists just don't make any sense at all, e.g. first removing a cipher (-RC4), then killing it (!RC4), all in one string with no benefit at all.
  - tombrossman 12y ago
    Thanks for writing this post, I've bookmarked it for reading later as it has plenty of links and I can see I have a lot more reading to do. If you edit or update your post, I hope you will Mozilla's excellent "Security/Server Side TLS" page at https://wiki.mozilla.org/Security/Server_Side_TLS https://wiki.mozilla.org/Security/Server_Side_TLS. This helped me get up to speed quickly and provided clear examples. As proof of how good the Mozilla docs are, I tested my personal website using the Qualys test you mention and received an A+ rating! This beat the A- rating for your site (though I freely admit I'm a total noob in this area - I'm copy/pasting and don't understand much about SSL). I guess this reinforces your point that good documentation is critical, and I hope more people find it at the Mozilla site. https://www.ssllabs.com/ssltest/analyze.html?d=tombrossman.com https://www.ssllabs.com/ssltest/analyze.html?d=tombrossman.c... https://www.ssllabs.com/ssltest/analyze.html?d=daniel.molkentin.net https://www.ssllabs.com/ssltest/analyze.html?d=daniel.molken...
    - danimo 12y ago
      That's an excellent reference with good explanations. I'll add it to the list to get away from the strong Ivan bias :-). The reason why I had A- only is that my openssl (Debian) doesn't seem provide all the ciphers required.
      - puetzk 12y ago
        It's probably your apache, actually; 2.2 can't do ECDHE, even though the openssl in Debian 7 can. Sadly, this means IE doesn't get forward secrecy, because it can't do do DHE with RSA keys, only DSS.
luminousbit 12y ago
I don't think this article is really helpful at all. The author attacks blog posts that state the current best-practices, but then goes on to recommend several books that are even more out of date than the blog posts. (Some of the books are 'theory' books and so presumably remain 'current' for longer, but not all of them). Second, instead of allowing sysadmins to find and follow simple, well-researched best-practices, the author instead wants each person to thoroughly research the annals of cryptography in order to then come to their 'own' conclusion. Most sysadmins won't do this. Or they will miss something or make rookie mistakes (this why you're always told not to roll your own cryptography). To put it another way: "you should not roll your own choices about which cryptography to use". So for the other 99% who just need to get it right and move on: find the current best practices from a reputable source (like Qualys) and use those.
- danimo 12y ago
  >The author attacks blog posts that state the current best-practices No, I'm attacking the fact that people blingly follow blog posts that have been, at some point, what their author believed were best practices. > But then goes on to recommend several books that are even more out of date than the blog posts.</cite> Which is fine, as long as they are read for what they are supposed to be: Either introductions or specializing on a specific topic. I wouldn't have chosen them otherwise. > Second, instead of allowing sysadmins to find and follow simple, well-researched best-practices, the author instead wants each person to thoroughly research the annals of cryptography in order to then come to their 'own' conclusion It is up to your own self-conception as a sysadmin as to how deep you want to dive. When interviewing (non-junior grade) sysadmins, I challenge them on security knowledge just as much as other skills. And I know I'm not alone with that. A certain degree of security awareness is not a "nice to have" typoe additional skill. It's vital for everyone that has machines connected to the internet. How far he/she can dive is solely limited by the economics and time constrains. Which is why sensible defaults are needed from vendors and distros (see my other response). > Most sysadmins won't do this. Which is a real problem, and again, there should be some effort remedying this, but currently there isn't. That's why I plea to sit down and at least learn about the basics. What "the basics" are obviously depends greatly on your educational background, but if AES, RC4 and PFS do not ring a bell, and you are a professional sysadmin who runs SSL-secured web servers, you are not worth your money. > (this why you're always told not to roll your own cryptography) Nobody said anything about rolling your own cryptography. But if you at least have read one of Ivan's books, you can at least make a /qualified decision/ on how credible a config proposed in a "random blog post" is, without having to come up with the complete solution by yourself. > find the current best practices from a reputable source (like Qualys) and use those. Which is ok -- if you run a small setup and you at least use SSLLabs to verify your setup. The more responsibility you have, and the more security is required, the more you should dive in and have a qualified opinion on how your SSL/TLS setup should look like.
kyberias 12y ago
Following incorrect guidelines is "cargo cult" now? Or are the guidelines themselves cargo cult? I'm confused.
- TheCowboy 12y ago
  Following an incorrect tutorial that is supposed to bestow security is as useful as following ritualistic acts intended to bestow wealth. There are a lot of steps and motions included in many resources, without a good reason behind them, that one follows. You basically have to assume a degree of faith that they have a purpose and actually achieve what the author purports they achieve.
- pessimizer 12y ago
  All "cargo cult" means is that you're doing something associated with an outcome that you want, rather than understanding the mechanism by which what you're doing would lead to the desired outcome. A common example: towns that have a lot of tourists have a lot of parking. Therefore if we tear down buildings to make more parking lots, we'll increase tourism! I'm pretty sure that what 99.9% of developers (including me) are doing with SSL is cargo cult.
faskfask 12y ago
I have no idea what I am doing. Is my Digital Ocean - Default Ubuntu based server safe? I know I login via the shell using ssh from my OS-X and DO does not administrate the droplet I guess. Any advice on where I can educate myself?
- raiyu 12y ago
  We have a couple of resources to help you with that in our community: https://www.digitalocean.com/community/questions/how-do-you-secure-your-new-server https://www.digitalocean.com/community/questions/how-do-you-... https://digitalocean.com/community/tags/security https://digitalocean.com/community/tags/security Thanks! Moisey Cofounder DigitalOcean
peterwwillis 12y ago
TLS 1.0 was created in 1999, and has approximately 97.7% website support worldwide. The only modern browser that doesn't support TLS 1.0 is IE6. Please, do the public a favor and remove the name "SSL" from your list. Also, I get that you probably just found out what a cargo cult is, but there's really no purpose in telling people about the phrase; it doesn't help them stay more secure, and it's basically just pop psychology applied to technology. "Current guides for securing your TLS servers" would make a more accurate title.
- danimo 12y ago
  About the Cargo Cult thing: Fair enough. I'm not a native speaker myself, but thought that it may be a well-enough-known idiom. Anyway, SSL is still what people know it under (plus, according to Wikipedia, TLS support was only added in Java 7, and there are still many Java 6 setups around which have to put up with SSL 3). So I'll stick with SSL/TLS for the time being.
  - puetzk 12y ago
    Re: "Cargo Cult" as a well-enough-known idiom: It's basically a shibboleth for "Are you a fan of Richard Feynman". I doubt very many people have actually heard of it directly, but his "Cargo Cult Science" commencement address (http://neurotheory.columbia.edu/~ken/cargo_cult.html http://neurotheory.columbia.edu/~ken/cargo_cult.html) is moderately famous.
- schwarze_pest 12y ago
  IE6 supports TLS 1.0
  - hsivonen 12y ago
    TLS 1.0 is not enabled by default in IE6, so for practical purposes, IE6 does not support TLS 1.0.
jrochkind1 12y ago
Okay, I understand what he means about cargo culting, and that to really make your server secure you have to actually know what you're doing. But okay, let's be realistic here. Let s look at the real thing that just happened to me. I went and used the awesome tester at ssllabs.com that OP recommends. It told me that my server doesn't support newer cipher protocols that it should support, and that it doesn't support forward secrecy. So, okay, I start googling on how to fix this. It looks like supporting forward secrecy is also an issue of cipher protocols, if I support the proper cipher protocols I'll support forward secrecy. Okay, how am I going to figure out which cipher protocols to configure as supported in apache? I'm to google and copy and paste someone elses apache configuration. That is, I'm going to cargo cult it. What else is possibly realistic for me to do? How many hours would it take for me to actually understand myself which ciphers are secure, which are preferred, and why? Not just how many hours would it take now, but how many _continual_ hours to keep up with developments? We all can't be experts at this level in this area -- that level of expertise in security is pretty much a role all by itself. And we can't all even have such experts on staff. We've _got_ to rely on external experts, anything else is unrealistic. Call that 'cargo culting' all you want, but what I need is an expert to tell me what my apache config should be -- sure I'm going to read up to get enough background to basically understand what they're talking about, but there's no way I'm going to become enough of an expert to make this determination on my own. No? Many use apache httpd. Seems to me apache httpd should ship with proper SSL configuration, and apache httpd team should provide notification channels for people to change configuration on their already-installed apache when best practices change. Figuring this out can't be a task for each individual 'customer', for basic use cases and routine needs, it should be the task of the 'vendor', no?
- sp332 12y ago
  I don't think that's true. Copy-and-pasting an apache config because it does what you want is different from copy-and-pasting an apache config and hoping that it does what you want.
- danimo 12y ago
  I've essentially answered this in https://news.ycombinator.com/item?id=7620756 https://news.ycombinator.com/item?id=7620756. Basically, this cannot be the sole task of Apache, as they rely on OpenSSL via mod_ssl. Some parts are apache specific (OCSP stapling), but others, just like cipher suites, purely depend on OpenSSL. But I agree that Distributors (as the "Vendors") need to come up with a suitable solution.

80 points by danimo 12y ago

With the rise of the cloud, now everybody manages a server. I am curious about how many of the admins actually know what are they doing. If I have to speak for myself, I set up my first ssl by following a guide that seemed to written by reasonably well informed stranger. It was my first exposure to the topic so I could not tell what I am doing until I break things and start reading deeper materials so that I can debug the problems by understanding what's going on. Turns out that the stranger who wrote the tutorial I followed was not well informed at all.

danieltillett 12y ago
A lot of sys admins are there by default. I would consider myself one of these - I would love to be able off load this work onto someone else who knows more than me, but there just isn't anyone.
- danimo 12y ago
  Essentially this comes down to pressuring distros and server vendors do their homework finally ship with good examples/defaults. E.g. Microsoft IIS (!) has OCSP stapling enabled by default since ages. Apache? Most people still run 2.2, which isn't capable of OCSP stapling at all. Nginx is in a similar position. That said, the more sysadmins rely on <s>rotten</s>well-proven "Enterprise Linuxes" and "LTS" versions with old libraries and servers, the more security expertise is required from sysadmins to decide where to deviate from the distros default packages to meet current best practices. On the other hand, security is a moving target and knowing your (Open)SSL setup is as important as e.g. knowing your RoR setup. It's an inconvenient truth, because it requires learning new stuff. I don't see any alternative though, that's why I compiled this material. Finally, a remark on the "offloading" part: Security is the single thing where delegation becomes hard because it means delegating trust, as in: your private SSL/TLS keys. And that's quite some trust to delegate.
  - danieltillett 12y ago
    There is so much I agree with in your post. The more I learn in this area the less sure I am that I am doing the right things.
  - jarrett 12y ago
    > knowing your (Open)SSL setup is as important as e.g. knowing your RoR setup Very true, and yet it's so much harder to know your OpenSSL or other security-related setup. You learn your Rails setup well enough to make your application work, and hopefully well enough to make it performant. If you miss either of those goals, it's obvious to you. You know something's broken, and you grind away until you fix it. Not so with security. Your system can be "working" by all outward appearances yet be riddled with vulnerabilities. And you won't know it, so you won't see any reason to go and learn more. Nor would you know what you don't know, or where you need to learn more. That's the scary part.
danimo 12y ago
Yes, that's the exact problem that made me write this. What's particularly amazing is the amount of magical cipher suite strings shared throughout the web, most of which do not take in account PFS, or still prioritize RC4. All of that was acceptable at some point in time. Other cipher lists just don't make any sense at all, e.g. first removing a cipher (-RC4), then killing it (!RC4), all in one string with no benefit at all.
- tombrossman 12y ago
  Thanks for writing this post, I've bookmarked it for reading later as it has plenty of links and I can see I have a lot more reading to do. If you edit or update your post, I hope you will Mozilla's excellent "Security/Server Side TLS" page at https://wiki.mozilla.org/Security/Server_Side_TLS https://wiki.mozilla.org/Security/Server_Side_TLS. This helped me get up to speed quickly and provided clear examples. As proof of how good the Mozilla docs are, I tested my personal website using the Qualys test you mention and received an A+ rating! This beat the A- rating for your site (though I freely admit I'm a total noob in this area - I'm copy/pasting and don't understand much about SSL). I guess this reinforces your point that good documentation is critical, and I hope more people find it at the Mozilla site. https://www.ssllabs.com/ssltest/analyze.html?d=tombrossman.com https://www.ssllabs.com/ssltest/analyze.html?d=tombrossman.c... https://www.ssllabs.com/ssltest/analyze.html?d=daniel.molkentin.net https://www.ssllabs.com/ssltest/analyze.html?d=daniel.molken...
  - danimo 12y ago
    That's an excellent reference with good explanations. I'll add it to the list to get away from the strong Ivan bias :-). The reason why I had A- only is that my openssl (Debian) doesn't seem provide all the ciphers required.
    - puetzk 12y ago
      It's probably your apache, actually; 2.2 can't do ECDHE, even though the openssl in Debian 7 can. Sadly, this means IE doesn't get forward secrecy, because it can't do do DHE with RSA keys, only DSS.

luminousbit 12y ago

I don't think this article is really helpful at all. The author attacks blog posts that state the current best-practices, but then goes on to recommend several books that are even more out of date than the blog posts. (Some of the books are 'theory' books and so presumably remain 'current' for longer, but not all of them). Second, instead of allowing sysadmins to find and follow simple, well-researched best-practices, the author instead wants each person to thoroughly research the annals of cryptography in order to then come to their 'own' conclusion. Most sysadmins won't do this. Or they will miss something or make rookie mistakes (this why you're always told not to roll your own cryptography). To put it another way: "you should not roll your own choices about which cryptography to use". So for the other 99% who just need to get it right and move on: find the current best practices from a reputable source (like Qualys) and use those.

danimo 12y ago
>The author attacks blog posts that state the current best-practices No, I'm attacking the fact that people blingly follow blog posts that have been, at some point, what their author believed were best practices. > But then goes on to recommend several books that are even more out of date than the blog posts.</cite> Which is fine, as long as they are read for what they are supposed to be: Either introductions or specializing on a specific topic. I wouldn't have chosen them otherwise. > Second, instead of allowing sysadmins to find and follow simple, well-researched best-practices, the author instead wants each person to thoroughly research the annals of cryptography in order to then come to their 'own' conclusion It is up to your own self-conception as a sysadmin as to how deep you want to dive. When interviewing (non-junior grade) sysadmins, I challenge them on security knowledge just as much as other skills. And I know I'm not alone with that. A certain degree of security awareness is not a "nice to have" typoe additional skill. It's vital for everyone that has machines connected to the internet. How far he/she can dive is solely limited by the economics and time constrains. Which is why sensible defaults are needed from vendors and distros (see my other response). > Most sysadmins won't do this. Which is a real problem, and again, there should be some effort remedying this, but currently there isn't. That's why I plea to sit down and at least learn about the basics. What "the basics" are obviously depends greatly on your educational background, but if AES, RC4 and PFS do not ring a bell, and you are a professional sysadmin who runs SSL-secured web servers, you are not worth your money. > (this why you're always told not to roll your own cryptography) Nobody said anything about rolling your own cryptography. But if you at least have read one of Ivan's books, you can at least make a /qualified decision/ on how credible a config proposed in a "random blog post" is, without having to come up with the complete solution by yourself. > find the current best practices from a reputable source (like Qualys) and use those. Which is ok -- if you run a small setup and you at least use SSLLabs to verify your setup. The more responsibility you have, and the more security is required, the more you should dive in and have a qualified opinion on how your SSL/TLS setup should look like.

kyberias 12y ago

Following incorrect guidelines is "cargo cult" now? Or are the guidelines themselves cargo cult? I'm confused.

TheCowboy 12y ago
Following an incorrect tutorial that is supposed to bestow security is as useful as following ritualistic acts intended to bestow wealth. There are a lot of steps and motions included in many resources, without a good reason behind them, that one follows. You basically have to assume a degree of faith that they have a purpose and actually achieve what the author purports they achieve.
pessimizer 12y ago
All "cargo cult" means is that you're doing something associated with an outcome that you want, rather than understanding the mechanism by which what you're doing would lead to the desired outcome. A common example: towns that have a lot of tourists have a lot of parking. Therefore if we tear down buildings to make more parking lots, we'll increase tourism! I'm pretty sure that what 99.9% of developers (including me) are doing with SSL is cargo cult.

faskfask 12y ago

I have no idea what I am doing. Is my Digital Ocean - Default Ubuntu based server safe? I know I login via the shell using ssh from my OS-X and DO does not administrate the droplet I guess. Any advice on where I can educate myself?

raiyu 12y ago
We have a couple of resources to help you with that in our community: https://www.digitalocean.com/community/questions/how-do-you-secure-your-new-server https://www.digitalocean.com/community/questions/how-do-you-... https://digitalocean.com/community/tags/security https://digitalocean.com/community/tags/security Thanks! Moisey Cofounder DigitalOcean

peterwwillis 12y ago

TLS 1.0 was created in 1999, and has approximately 97.7% website support worldwide. The only modern browser that doesn't support TLS 1.0 is IE6. Please, do the public a favor and remove the name "SSL" from your list. Also, I get that you probably just found out what a cargo cult is, but there's really no purpose in telling people about the phrase; it doesn't help them stay more secure, and it's basically just pop psychology applied to technology. "Current guides for securing your TLS servers" would make a more accurate title.

danimo 12y ago
About the Cargo Cult thing: Fair enough. I'm not a native speaker myself, but thought that it may be a well-enough-known idiom. Anyway, SSL is still what people know it under (plus, according to Wikipedia, TLS support was only added in Java 7, and there are still many Java 6 setups around which have to put up with SSL 3). So I'll stick with SSL/TLS for the time being.
- puetzk 12y ago
  Re: "Cargo Cult" as a well-enough-known idiom: It's basically a shibboleth for "Are you a fan of Richard Feynman". I doubt very many people have actually heard of it directly, but his "Cargo Cult Science" commencement address (http://neurotheory.columbia.edu/~ken/cargo_cult.html http://neurotheory.columbia.edu/~ken/cargo_cult.html) is moderately famous.
schwarze_pest 12y ago
IE6 supports TLS 1.0
- hsivonen 12y ago
  TLS 1.0 is not enabled by default in IE6, so for practical purposes, IE6 does not support TLS 1.0.

jrochkind1 12y ago

Okay, I understand what he means about cargo culting, and that to really make your server secure you have to actually know what you're doing. But okay, let's be realistic here. Let s look at the real thing that just happened to me. I went and used the awesome tester at ssllabs.com that OP recommends. It told me that my server doesn't support newer cipher protocols that it should support, and that it doesn't support forward secrecy. So, okay, I start googling on how to fix this. It looks like supporting forward secrecy is also an issue of cipher protocols, if I support the proper cipher protocols I'll support forward secrecy. Okay, how am I going to figure out which cipher protocols to configure as supported in apache? I'm to google and copy and paste someone elses apache configuration. That is, I'm going to cargo cult it. What else is possibly realistic for me to do? How many hours would it take for me to actually understand myself which ciphers are secure, which are preferred, and why? Not just how many hours would it take now, but how many _continual_ hours to keep up with developments? We all can't be experts at this level in this area -- that level of expertise in security is pretty much a role all by itself. And we can't all even have such experts on staff. We've _got_ to rely on external experts, anything else is unrealistic. Call that 'cargo culting' all you want, but what I need is an expert to tell me what my apache config should be -- sure I'm going to read up to get enough background to basically understand what they're talking about, but there's no way I'm going to become enough of an expert to make this determination on my own. No? Many use apache httpd. Seems to me apache httpd should ship with proper SSL configuration, and apache httpd team should provide notification channels for people to change configuration on their already-installed apache when best practices change. Figuring this out can't be a task for each individual 'customer', for basic use cases and routine needs, it should be the task of the 'vendor', no?

sp332 12y ago
I don't think that's true. Copy-and-pasting an apache config because it does what you want is different from copy-and-pasting an apache config and hoping that it does what you want.
danimo 12y ago
I've essentially answered this in https://news.ycombinator.com/item?id=7620756 https://news.ycombinator.com/item?id=7620756. Basically, this cannot be the sole task of Apache, as they rely on OpenSSL via mod_ssl. Some parts are apache specific (OCSP stapling), but others, just like cipher suites, purely depend on OpenSSL. But I agree that Distributors (as the "Vendors") need to come up with a suitable solution.