6 ms·

>What I read him as saying (and agree with) is that he's surprised that GnuTLS isn't tested against known-bad certificates in a simple integration test that doe

by cube13 13y ago

>What I read him as saying (and agree with) is that he's surprised that GnuTLS isn't tested against known-bad certificates in a simple integration test that doesn't require a Ph.D. to set up, as you imply. Define "known-bad" in a general enough way that a specific test can be created to cover the entire range of "bad" certs. That's quite difficult, and, probably isn't realistically possible to go through all the "known-bad" if you want your tests to run quickly. Realistically, all you can do is have regression tests to make sure that the found bugs aren't repeated in future releases.

jsmthrowaway 13y ago
My thoughts on this are more about the previous TLS disclosure, not this one, where an integration test that says "this invalid certificate should fail validation" probably would have caught it. But, this isn't my argument to make as I don't know enough about it, I just want to see the actual coherent argument made instead of a side tangent as we're seeing here.
- tptacek 13y ago
  What integration test? Writing that integration test implies that you know the bug is a possibility. Once you consider that possibility, a glance at the code tells you if it's there or not. It's not like this bug is likely to regress, either.
  - jsmthrowaway 13y ago
    Again, I'm talking about the "goto fail" one, not this one, in which a simple test of the library against an invalid certificate would have caught at least one case that I recall. And I think it's pretty obvious that "you might erroneously validate a bad certificate" is a bug that's, you know, a given.
    - tptacek 13y ago
      Feature: *Certificate Validation* In order to *keep NSA from reading my emai* As a *TLS X.509 validation library* I want *to never erroneously validate a certificate* What are the "Scenarios"? Given: *???* And: *???* When: *???* Then: *the certificate should be rejected* Remember, if we're switching topics to the "goto fail" bug: that bug didn't affect every instance of certificate validation. You had to be in a particular set of ciphersuites.
      - deleted 13y ago
        [deleted]
      - jsmthrowaway 13y ago
        I spent a few minutes replying to your original comment, then you ninja edited in the closing paragraph and invalidated mine, so this is where I end the conversation. I can't reply to an ever-shifting comment with someone who reads what they want to read out of my comments and invokes the NSA for no reason whatsoever.
        tptacek 13y ago
        You'd be a happier person if you didn't assume people were editing their comments just to make you look bad. You can be assured that if I had seen you respond to something I'd written, I'd have noted where I added to my original comment. But I agree that you & I are better off not discussing things.
        jsmthrowaway 13y ago
        I'm quite a happy person and thanks for the irrelevant concern trolling, based on a comment thread that's calling you out (do you always assume people that think you're wrong are unhappy or otherwise defective?). And no, I didn't assume those words you've placed in my fingers. I was, rather, annoyed that I invested a few minutes into a playing field that you'd already moved while I worked. If you'd permit me a brief bit of my own concern trolling: I remember when I looked forward to reading your comments, several years ago. Now I see your nickname and say "bah, again?" What changed? Was it me or you?
  - cube13 13y ago
    I'm a firm believer in regression tests for complex systems. Sure, this specific case is extremely unlikely to come up again, but in general, it's always good to have that coverage, especially if a large refactor happens later on.
    - tptacek 13y ago
      I'm definitely not arguing against regression testing, or really even against test suites for TLS (there should be more of them, and better ones). I'm suggesting that the discipline of aggressively testing code for coverage is unlikely to make as much of a dent as moving to a better language would, and since both are extremely costly changes to the way TLS stacks are developed, we might as well adopt the one that will make us safer.
      - comex 13y ago
        I claim that writing an extensive test suite for TLS is not nearly as difficult as switching people to a new language. Rewriting in a stricter language, maybe. But Haskell has a big runtime and garbage collection and its own compiler and would be a big pain to integrate everywhere that uses C TLS libraries; smaller compile-to-C languages might be easier, but who wants to use an experimental language to develop crypto code? ...And you'd still want to test it, because although functional programming style makes many bug classes less likely, it's not a panacea. You're still basically hoping that the developer doesn't make a single thinko. Compared to that: - Better unit testing is much easier to integrate into existing projects. Yes, it can only prevent a bug if the developer generally thought of the class of error, but at least it sort of forces them to spend some time thinking about possible failure cases, and can detect cases where their mental model was wrong. Also, it helps detect regressions: "goto fail" wasn't a strange edge case the developer didn't think of, it was a copy paste error which good unit tests could have caught. - Functional testing can be independent of the implementation and written by someone unrelated. They can only do so much in general, but they might have caught both of these bugs. Yes, audits are another option, but I'd say they should complement tests, not replace them. ed: oh, and if you want to be really intellectually rigorous, you could try to formally verify your C code; model could have bugs but could also be implementation independent. But I hear that's rather difficult...
leoc 13y ago
> Define "known-bad" in a general enough way that a specific test can be created to cover the entire range of "bad" certs. Any known-bad cert at all would have been quite sufficient to catch this bug apparently. A simple ARE WE ACCEPTING BAD CERTIFICATES LOL sanity-check would have found it, which is the kind of unit test it should be possible to think of in advance rather than in response to a specific bug found earlier. A little can go a long way. EDIT: Additionally, the difficulty of catching all bad certs is good reason to develop and continually update a torture-test of invalid certs (and valid ones) to test SSL clients against. The suite would be much too slow to check against once per recompile, but testing once before each point release should be useful enough...

by cube13 13y ago

jsmthrowaway 13y ago

My thoughts on this are more about the previous TLS disclosure, not this one, where an integration test that says "this invalid certificate should fail validation" probably would have caught it. But, this isn't my argument to make as I don't know enough about it, I just want to see the actual coherent argument made instead of a side tangent as we're seeing here.

tptacek 13y ago
What integration test? Writing that integration test implies that you know the bug is a possibility. Once you consider that possibility, a glance at the code tells you if it's there or not. It's not like this bug is likely to regress, either.
- jsmthrowaway 13y ago
  Again, I'm talking about the "goto fail" one, not this one, in which a simple test of the library against an invalid certificate would have caught at least one case that I recall. And I think it's pretty obvious that "you might erroneously validate a bad certificate" is a bug that's, you know, a given.
  - tptacek 13y ago
    Feature: *Certificate Validation* In order to *keep NSA from reading my emai* As a *TLS X.509 validation library* I want *to never erroneously validate a certificate* What are the "Scenarios"? Given: *???* And: *???* When: *???* Then: *the certificate should be rejected* Remember, if we're switching topics to the "goto fail" bug: that bug didn't affect every instance of certificate validation. You had to be in a particular set of ciphersuites.
    - deleted 13y ago
      [deleted]
    - jsmthrowaway 13y ago
      I spent a few minutes replying to your original comment, then you ninja edited in the closing paragraph and invalidated mine, so this is where I end the conversation. I can't reply to an ever-shifting comment with someone who reads what they want to read out of my comments and invokes the NSA for no reason whatsoever.
      - tptacek 13y ago
        You'd be a happier person if you didn't assume people were editing their comments just to make you look bad. You can be assured that if I had seen you respond to something I'd written, I'd have noted where I added to my original comment. But I agree that you & I are better off not discussing things.
        jsmthrowaway 13y ago
        I'm quite a happy person and thanks for the irrelevant concern trolling, based on a comment thread that's calling you out (do you always assume people that think you're wrong are unhappy or otherwise defective?). And no, I didn't assume those words you've placed in my fingers. I was, rather, annoyed that I invested a few minutes into a playing field that you'd already moved while I worked. If you'd permit me a brief bit of my own concern trolling: I remember when I looked forward to reading your comments, several years ago. Now I see your nickname and say "bah, again?" What changed? Was it me or you?
- cube13 13y ago
  I'm a firm believer in regression tests for complex systems. Sure, this specific case is extremely unlikely to come up again, but in general, it's always good to have that coverage, especially if a large refactor happens later on.
  - tptacek 13y ago
    I'm definitely not arguing against regression testing, or really even against test suites for TLS (there should be more of them, and better ones). I'm suggesting that the discipline of aggressively testing code for coverage is unlikely to make as much of a dent as moving to a better language would, and since both are extremely costly changes to the way TLS stacks are developed, we might as well adopt the one that will make us safer.
    - comex 13y ago
      I claim that writing an extensive test suite for TLS is not nearly as difficult as switching people to a new language. Rewriting in a stricter language, maybe. But Haskell has a big runtime and garbage collection and its own compiler and would be a big pain to integrate everywhere that uses C TLS libraries; smaller compile-to-C languages might be easier, but who wants to use an experimental language to develop crypto code? ...And you'd still want to test it, because although functional programming style makes many bug classes less likely, it's not a panacea. You're still basically hoping that the developer doesn't make a single thinko. Compared to that: - Better unit testing is much easier to integrate into existing projects. Yes, it can only prevent a bug if the developer generally thought of the class of error, but at least it sort of forces them to spend some time thinking about possible failure cases, and can detect cases where their mental model was wrong. Also, it helps detect regressions: "goto fail" wasn't a strange edge case the developer didn't think of, it was a copy paste error which good unit tests could have caught. - Functional testing can be independent of the implementation and written by someone unrelated. They can only do so much in general, but they might have caught both of these bugs. Yes, audits are another option, but I'd say they should complement tests, not replace them. ed: oh, and if you want to be really intellectually rigorous, you could try to formally verify your C code; model could have bugs but could also be implementation independent. But I hear that's rather difficult...

leoc 13y ago

> Define "known-bad" in a general enough way that a specific test can be created to cover the entire range of "bad" certs. Any known-bad cert at all would have been quite sufficient to catch this bug apparently. A simple ARE WE ACCEPTING BAD CERTIFICATES LOL sanity-check would have found it, which is the kind of unit test it should be possible to think of in advance rather than in response to a specific bug found earlier. A little can go a long way. EDIT: Additionally, the difficulty of catching all bad certs is good reason to develop and continually update a torture-test of invalid certs (and valid ones) to test SSL clients against. The suite would be much too slow to check against once per recompile, but testing once before each point release should be useful enough...