7 ms·

Can anyone comment on the claim "QuickCheck is shockingly more effective at finding bugs than unit tests"? I'd be interested in hearing other opinions.

ams6110 15y ago
I haven't used QuickCheck, but read about it. So take this with a grain of salt, but it seems to me that specifying properties that should hold for your functions and then letting QuickCheck throw all kinds of data at them would be more efficient at least, if not more effective, than manually writing a bunch of unit tests and coming up with your own test data.
pjscott 15y ago
It's surprising how often people overlook weird edge cases that QuickCheck finds, because it doesn't have as many preconceptions about what kinds of inputs are sensible.
tel 15y ago
It tends to be best for mathematical code because you can quickly think of invariances. In any situation where you can state a powerful invariance your code must always be consistent with it's both fast and thorough. The common example is that (= str (reverse (reverse sequences))) for all sequences. Note that this only tests invertibility. There are lots of ways reversal can be broken without violating this invariance. It's just really easy to write down this invariance and use it to rapidly check gross violations.
masklinn 15y ago
It's essentially a unit-fuzzing framework, so it can find holes in boundary conditions people generally forget to check, because it's in their blind spot or because "nobody would ever do that". QC does not care, and since it's injecting fuzzed data at the unit leve (~function, generally) it makes it quite easy to see precisely where the failure happens.
- eru 15y ago
  Also writing up the laws that the fuzzers checks, is a good exercise in thinking about your API. Read http://donsbot.wordpress.com/2007/05/01/roll-your-own-window-manager-part-1-defining-and-testing-a-model/ http://donsbot.wordpress.com/2007/05/01/roll-your-own-window... and http://donsbot.wordpress.com/2007/05/17/roll-your-own-window-manager-tracking-focus-with-a-zipper/ http://donsbot.wordpress.com/2007/05/17/roll-your-own-window... for some examples. Quick also not only gives you the failing cases, but tries to simplify and shrink them first.
exDM69 15y ago
It's worth noting that QuickCheck is used with functional code only. For non-functional code with side effects (e.g. database transactions), Haskell has unit test frameworks that are not too different from unit testing frameworks in other languages. What distingishes QuickCheck from regular unit testing is that you give it a function to test, some invariants that must hold and an input generator. Some of this is automated with compiler magic (where possible) to make simple tests really simple to write. QuickCheck will then grind through a bunch of random inputs and see if the invariants hold, finally showing you the failed cases.
- bos 15y ago
  > It's worth noting that QuickCheck is used with functional code only. That's actually not true. You can use QuickCheck with "normal" code that has side effects, too, and it's every bit as useful there.
tmoertel 15y ago
The claim is true: QuickCheck is surprisingly more effective at finding problems (and documenting expected behavior) than unit tests. When I first tried it, not long after the QuickCheck paper came out at ICFP 2000, it found problems that I never would have thought to test for. [1] (That's why I end up writing something like QuickCheck when I code in environments that don't already have one. [2] It's worth it.) The key to its effectiveness is that it doesn't suffer from limited human imagination. When I write test cases by hand, it's up to me to think of all the things that could go wrong and write test cases for them. But, being human, I have blind spots. Even when I try to be systematic about detecting edge cases and writing tests for them, I miss some. But when QuickCheck-like tools write the test cases for me, they can dream up corner cases like you wouldn't believe. And, even better, when I use QuickCheck, my test code doesn't end up looking like an enumeration of corner cases. Instead, it becomes clear, concise, and formal documentation. (For a good example of QuickCheck properties as documentation, see [3].) I just specify the intended general properties of my code, and QuickCheck generates the messy test cases behind the scenes, where they don't become visible noise. [1] http://blog.moertel.com/pages/seven-lessons-from-the-icfp-programming-contest http://blog.moertel.com/pages/seven-lessons-from-the-icfp-pr... [2] http://search.cpan.org/dist/Test-LectroTest/lib/Test/LectroTest/Tutorial.pod http://search.cpan.org/dist/Test-LectroTest/lib/Test/LectroT... [3] http://haskell.org/ghc/docs/latest/html/libraries/base/Text-ParserCombinators-ReadP.html#g:5 http://haskell.org/ghc/docs/latest/html/libraries/base/Text-...

ams6110 15y ago

I haven't used QuickCheck, but read about it. So take this with a grain of salt, but it seems to me that specifying properties that should hold for your functions and then letting QuickCheck throw all kinds of data at them would be more efficient at least, if not more effective, than manually writing a bunch of unit tests and coming up with your own test data.

pjscott 15y ago

It's surprising how often people overlook weird edge cases that QuickCheck finds, because it doesn't have as many preconceptions about what kinds of inputs are sensible.

tel 15y ago

It tends to be best for mathematical code because you can quickly think of invariances. In any situation where you can state a powerful invariance your code must always be consistent with it's both fast and thorough. The common example is that (= str (reverse (reverse sequences))) for all sequences. Note that this only tests invertibility. There are lots of ways reversal can be broken without violating this invariance. It's just really easy to write down this invariance and use it to rapidly check gross violations.

masklinn 15y ago

It's essentially a unit-fuzzing framework, so it can find holes in boundary conditions people generally forget to check, because it's in their blind spot or because "nobody would ever do that". QC does not care, and since it's injecting fuzzed data at the unit leve (~function, generally) it makes it quite easy to see precisely where the failure happens.

eru 15y ago
Also writing up the laws that the fuzzers checks, is a good exercise in thinking about your API. Read http://donsbot.wordpress.com/2007/05/01/roll-your-own-window-manager-part-1-defining-and-testing-a-model/ http://donsbot.wordpress.com/2007/05/01/roll-your-own-window... and http://donsbot.wordpress.com/2007/05/17/roll-your-own-window-manager-tracking-focus-with-a-zipper/ http://donsbot.wordpress.com/2007/05/17/roll-your-own-window... for some examples. Quick also not only gives you the failing cases, but tries to simplify and shrink them first.

exDM69 15y ago

It's worth noting that QuickCheck is used with functional code only. For non-functional code with side effects (e.g. database transactions), Haskell has unit test frameworks that are not too different from unit testing frameworks in other languages. What distingishes QuickCheck from regular unit testing is that you give it a function to test, some invariants that must hold and an input generator. Some of this is automated with compiler magic (where possible) to make simple tests really simple to write. QuickCheck will then grind through a bunch of random inputs and see if the invariants hold, finally showing you the failed cases.

bos 15y ago
> It's worth noting that QuickCheck is used with functional code only. That's actually not true. You can use QuickCheck with "normal" code that has side effects, too, and it's every bit as useful there.

tmoertel 15y ago

The claim is true: QuickCheck is surprisingly more effective at finding problems (and documenting expected behavior) than unit tests. When I first tried it, not long after the QuickCheck paper came out at ICFP 2000, it found problems that I never would have thought to test for. [1] (That's why I end up writing something like QuickCheck when I code in environments that don't already have one. [2] It's worth it.) The key to its effectiveness is that it doesn't suffer from limited human imagination. When I write test cases by hand, it's up to me to think of all the things that could go wrong and write test cases for them. But, being human, I have blind spots. Even when I try to be systematic about detecting edge cases and writing tests for them, I miss some. But when QuickCheck-like tools write the test cases for me, they can dream up corner cases like you wouldn't believe. And, even better, when I use QuickCheck, my test code doesn't end up looking like an enumeration of corner cases. Instead, it becomes clear, concise, and formal documentation. (For a good example of QuickCheck properties as documentation, see [3].) I just specify the intended general properties of my code, and QuickCheck generates the messy test cases behind the scenes, where they don't become visible noise. [1] http://blog.moertel.com/pages/seven-lessons-from-the-icfp-programming-contest http://blog.moertel.com/pages/seven-lessons-from-the-icfp-pr... [2] http://search.cpan.org/dist/Test-LectroTest/lib/Test/LectroTest/Tutorial.pod http://search.cpan.org/dist/Test-LectroTest/lib/Test/LectroT... [3] http://haskell.org/ghc/docs/latest/html/libraries/base/Text-ParserCombinators-ReadP.html#g:5 http://haskell.org/ghc/docs/latest/html/libraries/base/Text-...