10 ms·

The evil unit test.

77 points by wtfdeveloper 15y ago

MrEnigma 15y ago
I've found that the higher level the test (i.e. unit -> integration -> functional) the better they catch things, but the harder it is to figure out what broke. For instance we had a rule at a place I worked, where we needed to have 80% unit test coverage. And what is described in this article happened. Anytime we'd make a small refactor, we'd have to go update 3-4 test libraries. And if it was a big one, then you have a lot of tests to update. And since some were complex (or since there were a lot of them) people didn't dig into that much. When people finally did it sometimes showed the test wasn't even testing what it was supposed to anymore. The other issue is we had a lot of bugs around the integration of data. For instance if the DB class returned back an empty array vs null vs empty string. When we did the unit test we mocked it to return what we thought it should, which may not be the correct one. Integration tests better caught this, functional tests even more so. It seems a lot of shops don't use tests at all, now working at places that have done both, I'd rather error on the side of fewer tests, and do it smart instead of hitting a metric and assuming your code falls into line.
- wr1472 15y ago
  Sounds like your test were quite brittle, if refactoring application code causes tests to break. A good mental rule I use is: "test the what not the how". It also a good reason to program (and test) to interfaces than to the implementation, although you have to be careful with this as it is all too easy to go OTT in your use of design patterns. You have to know how to strike the right balance.
  - eaurouge 15y ago
    There are various kinds of tests, from unit tests to behavior driven testing. Refactoring application code may cause tests, especially unit tests, to break. If you're writing unit tests, by definition, you are testing the implementation - that's the point. It seems you're referring to behavioral (or black box) tests, which are very different.
    - wr1472 15y ago
      Agreed refactoring method signatures will cause tests to break, but they should be relatively trivial to fix. Black box tests are not limited to behavioral or system integration testing. you can black box at the unit level. In fact if you are doing proper TDD you will write the tests first, which help to define the interface of the class under test, make sure the test fails to ensure you are testing for something, and then finally write the implementation to make the test pass. Such an approach implies it is black box as you've defined your interface before your implementation. Remember you are testing what it does, not how it does it.
      - jsdalton 15y ago
        > Remember you are testing what it does, not how it does it. While I tend to agree with you (I strongly prefer black-box testing over white-box testing), this is a hotly debated topic and many people feel quite strongly the opposite. You can Google around and find many advocates for white box unit tests. Martin Fowler's article from several years ago I think does an excellent job of presenting a balanced look at both sides: http://martinfowler.com/articles/mocksArentStubs.html http://martinfowler.com/articles/mocksArentStubs.html
        wr1472 15y ago
        I've not come across the article before. The section on coupling tests to implementation I guess is the most pertinent part (http://martinfowler.com/articles/mocksArentStubs.html#CouplingTestsToImplementations http://martinfowler.com/articles/mocksArentStubs.html#Coupli...). > Coupling to the implementation also interferes with refactoring, since implementation changes are much more likely to break tests than with classic testing. Fowler acknowledges two different styles of tests - state/behaviour also can be termed as classic/mockist. He doesn't really advocate one over the other. I do mock when I have to, but only when there is no other way to setup my test. It is certainly not the default. I find that taking this approach does not make my tests brittle, and makes them robust enough to handle the majority of refactoring exercises. Of course YMMV.
  - keithnoizu 15y ago
    Exactly. You want to verify requirements not implementation details.Requirements change less frequently than implementation details.
- jrockway 15y ago
  For instance we had a rule at a place I worked, where we needed to have 80% unit test coverage. And what is described in this article happened. Anytime we'd make a small refactor, we'd have to go update 3-4 test libraries. And if it was a big one, then you have a lot of tests to update. And since some were complex (or since there were a lot of them) people didn't dig into that much. When people finally did it sometimes showed the test wasn't even testing what it was supposed to anymore. This just means your tests are bad. Write good tests and you won't have this problem. Integration and functional tests are good, but only as a quick sanity check. The nitty gritty details need to be handled by good unit tests; the functional tests only tell you things like "did the RPC protocol change" or "does the database cache actually cache something".
  - RandallBrown 15y ago
    It is really easy to write bad tests. That doesn't mean that you shouldn't write any, but just "writing good tests" can be really hard.
- capsule_toy 15y ago
  For me, unit tests are to assure that if I refactor the code, the places where the code is being used won't be affected by my changes. If I have to rewrite the tests, I've lost this assurance.
16s 15y ago
I used to oppose unit testing because I had a good C++ compiler with all warnings turned on and I foolishly thought that was all I needed and that unit testing was only for dynamic languages that did not have compilers. I'm a big fan now though. If I have written code that's hard to test, I re-factor the code so that it's testable. I've found that un-testable code is generally bad code (it works OK, but is hard to modify later). I also run my unit tests during the build (right after compilation), so I know right away if something fails. I've come to really like unit testing. I'm less afraid to change things and feel better (in general) about the quality of my code.
- franklindholm 15y ago
  I agree that Unit testing is a power full tool. But tools are just tools, and they can be misused. I see this sometimes at companies where they "make" their developers write unit tests, they are just writing them because they have to and this often shows in the quality of the tests. Everything is green in Jenkins so obviously the functional tester did something wrong :)
- 3wetwetw 15y ago
  C++ definitely also needs unit testing - even with untestable code (even when it's bad code, though not always, it may be easier to test it using something like Isolator++, rather than rewrite greenfield from scratch).
Swizec 15y ago
There is a time and place for unit tests - when you're creating algorithms. Most of the time, however, we are creating web apps and things that run at a much higher level of abstraction than the underlying algorithms. We should probably write tests at the same level of abstraction our solution is. Although, I guess, when you're writing medical software or something for a bank, you should be a bit more strict than when you are making the next best social sharing app. A guy I know once solved a bug by having a unit test for the md5 function from the standard library of the language at hand ... turns out a specific version was buggy. Either way, I prefer using integration/functional test. What I care about is that the right output flies in for a certain input. What I don't care about is how. So I don't test for that. Also, Unit tests must be created before the code they test. yes they should. It's very helpful to think about how your code will be used before you write it, helps you think it through as well.
- lyudmil 15y ago
  Usually the problem you'll get if you use a lot of functional/integration testing is that your test suite will run too slowly for you to run it often enough to validate your design. It ends up being something you run when you think you've completed a feature to make sure it works, but not after you've extracted a class to make your code cleaner. I have also had times when I moved away from unit testing in favor of higher-level tests, but ultimately found that the quality of the code got worse and the build time became unacceptable. I'm not trying to generalize here, I'm only talking about my code on a particular project at a particular point in time.
  - jerf 15y ago
    "Usually the problem you'll get if you use a lot of functional/integration testing is that your test suite will run too slowly..." I've had good luck attacking this problem directly. If your integration tests are sufficiently integration-y, often this slowness directly reflects the fact that your system is slow to the users as well! Or, failing that, that you've got too many dependencies hardwired into your system, or some other real problem. Following up on my other proximal post here, the question is ultimate not whether it's high or low level, it's whether it's fast or slow, period. I've written slow low-level tests because they fully tested a 10x10x10x10x10x10 state space accidentally, I've written (after some work on the underlying system) some very fast tests that literally simulate user input to the system via the web server at the highest possible level. The real key is that you can't really follow a testing recipe, you always have to be looking out for the cost-benefits. I suspect the other key is that you must come to internalize the idea that automated testing is really important and if that means spending a man-week to improve your testing environment is what you need to do, then do it. It's really OK.
dabit 15y ago
Was this a sarcastic post?
- Confusion 15y ago
  What makes you think that?
agentultra 15y ago
So basically: don't test the implementation and don't test the framework. Do test the API. Test the output. Good, smart tests should be a pleasure to work with. They should give you the confidence to change the implementation without mercy. They make the feedback cycle faster, bugs more visible, and documentation more thorough. The only time I've found unit tests to be a pain are when testing asynchronous, event-driven code. The tests aren't any more brittle, but they really need to be supported by good integration tests in order to find the kinds of bugs that can surface in such systems. I think the point about unit tests is that they should be the default and not the exception. 100% coverage is an asymptotic value. But if you write the test first you can get pretty darn close.
- dspillett 15y ago
  > So basically: don't test the implementation. I thought that was the point of unit tests... > Do test the API. Test the output. ... and that this would be integration testing. Am I drawing my mental lines separating the range of tests in the wrong places?
  - agentultra 15y ago
    I can't really say, but if you read The Clean Coder we might get on the same page. fwiw, what I mean when I say "don't test the implementation," is that your test should not test "how" a function or object performs its task when called. tldr; unit tests should test units -- the smallest amount of functionality possible A trivial example might be an object that retrieves some data from a persistent store and prints it out. It might have a "fetch" method whose argument is a key in the data store. You could easily write a test case that checks when "fetch" is called that the data is written to the standard output stream. It would pass and you could go home after a hard day of work. But if the requirements change and you need it to write to a socket instead and format the data for a binary stream -- you'll have to modify the "fetch" method and the tests. If we were clever though we'd realize that when we wrote the test to see that "fetch" output its result to standard out we were actually asking the object to do too much and our test was testing several different units of work that could be broken down. Ideally our hypothetical object would take a couple of callable objects that handle things like printing data to streams and formatting it. Then our test for fetch only has to test one small "unit" of functionality -- not several units (retrieving data, formatting it, printing it to a stream). hth
brown9-2 15y ago
A lot of these points apply to any sort of coding convention that a group agrees upon, not just unit testing.
prez 15y ago
You shall use a foreach every time you need to perform a loop. Works pretty well in Python. Though it doesn't invalidate the point the author tries to make.
- recursive 15y ago
  So you think while loops are unnecessary in python?
- cbs 15y ago
  Works pretty well in Python But xrange() in python is just a roundabout way to get back to having a regular for loop. for x in xrange(0,100) for(x=0, x<100, x++)
3wetwetw 15y ago
Absolutely - even those of us (like me - disclaimer: I work at Typemock) know that there is a time and a place for unit tests and TDD (not the same thing). In fact, we wrote about this before http://bitly.com/zdyJfl http://bitly.com/zdyJfl We're hosting a webinar on Wednesday about different kinds of testing and when a unit test is appropriate and when another kind of test, like an integration test, may be a better choice - http://bitly.com/xeYSYg http://bitly.com/xeYSYg. Unit tests have a time and place. There are times in which they are a must if you want to reduce technical debt and test your code - both if you're writing mission critical stuff or even when you're writing the latest social sharing app (remember the Fail Whale? We all hate that!). But, no, you don't need TDD on greenfield code all the time and you shouldn't have to write tests for logic.
owenjones 15y ago
The title and content don't correlate. This would be like having an article titled "The Evil Hammer" and then in the article explaining how "Hammer's aren't actually evil, just you shouldn't use them in situations where a saw would be more appropriate." I think anyone falling into these pitfalls just aren't writing good enough tests, don't dissuade them from furthering their testing skills, promote skillful testing. And change the title.
chaostheory 15y ago
I don't feel that this article is relevant for most programmers using newer languages such as Ruby or Python. There's a simple change in the way I use JUnit that I learned from using Ruby (RSpec, Cucumber) about 4-5 years ago: Don't test methods or functions. Instead test expected behavior e.g. a project requirement. Then you can pretty much avoid every problem listed in the article.
peteretep 15y ago
I wrote a similar article not too long ago: http://www.writemoretests.com/2011/09/test-driven-development-give-me-break.html http://www.writemoretests.com/2011/09/test-driven-developmen...
fuzzylizard 15y ago
I agree with only one point in the article; the ability to delete tests. Tests of all kinds should be living, breathing code, just like the code in the rest of your application. As such, tests, like production code, can become outdated and obsolete and need to be updated, changed, and sometimes deleted. However, I do not agree with the idea that tests are evil. If your tests are failing, or breaking regularly, or are hard to write, then your tests are trying to tell you something. Tests are a direct mirror of the state of your code. If your tests are brittle and hard to manage, don't delete the tests, fix your code. If you need to instantiate a tonne of objects in order to make your tests work, fix your methods dependencies and learn to test your methods in isolation of its dependencies. Any production code needs tests, this is non-negotiable in today's development eco-system. But I get tired of people who do not understand the purpose of tests complaining that tests are evil. If there is a problem with your tests, then your tests are telling you there is a problem with your code. Clean up your code in order to clean up your tests. However, remember that as code changes, your tests will need to change as well. It is okay to edit, delete, change tests to reflect the current state of the application. Just change the tests first to express how you want to interact with a specific bit of code and then change the code, not the other way around.
keithnoizu 15y ago
Sounds like a bit of a hyperbole, Blanket statements like 100% coverage, unit test per method are not too helpful, and it's important to understand the point where you will start to see diminishing returns with unit tests but that said well written unit tests with ~ 85-95 coverage by line free of conditional logic (use custom asserts) with meaningful test names/descriptions aren't exactly going to put you in league with beezlebub.
hashfold 15y ago
there are really nice comments below in this thread on why testing functionality is better than writing unit test cases to test the functions/methods/code. e.g. wr1472: test the what not the how MrEnigma: the higher level the test (i.e. unit -> integration -> functional) the better they catch things Keithnoizu: verify requirements not implementation details we understand that the unit testing is trying to test the functionality and making sure that the acceptance criteria is also met at the same time. I would like to mention one more point is that we should also try to write unit test cases not just only test the functionality but also try to do a good code coverage. unit test cases are tools for: 1. functionality testing 2. comply with acceptance criteria 3. a tool to help good code coverage 4. be iterative. be backward compatible as long as it could. (anyway to add logic based on version of the code change? need to think on this). yes it makes it little bulky but will work and avoid needing code cleanup and redesigns. let unit tests help come up with min 80% of code coverage.
lmm 15y ago
>You shall use a foreach every time you need to perform a loop. Yes >You will never access your database if you are not using an ORM. Hell yes. (Not necessarily disagreeing with his conclusions)
- tikhonj 15y ago
  What about while loops? That's the problem with blanket statements: they're very rarely 100% correct.
jwatte 15y ago
Unit tests, acceptance tests, and integration tests are different things. Unit tests test that the implementation behaves as intended, and generally grope around private internals. These change when implementation changes. Acceptance tests test that the public API is implemented correctly. These change when the API/interface change. Integration tests test that the end system implements requirements correctly, and change when requirements change. Most TDD hate comes from not separating these correctly, or not having suitable tools for each type, or from (gasp!) not actually having a clear understanding of requirements, interface, or implementation.
Peaker 15y ago
I find TDD to be somewhat redundant when I code Haskell. The types cover more ground than typical unit tests -- and if you follow a few simple principles (avoid partial functions and some of the worse part of the libraries), you can trust the compiled code more than you trust UT'd code. Production-quality Haskell does usually employ tests, and those are a joy to write (with QuickCheck), but there's little need to spend time writing tests at the early coding phases. When doing large refactorings, you can also trust the type system to guide you and find virtually all bugs introduced.
tikhonj 15y ago
Actually, his 2x3 example is not very good. The fact that 2x3 is 2 three times is the what, not the how; in a perfect world you would just assert that nxm == n + n + ... n m times for all n and all positive m. This is, coincidentally, where property based testing like QuickCheck comes in. Edit: Just pretend 2x3 has an asterisk instead of an x :)
- prodigal_erik 15y ago
  (FYI, 2 * 3 is okay; HN preserves * followed by whitespace.)
  - tikhonj 15y ago
    Ah, thanks. That was probably the issue, because some of my asterisks worked but others didn't. I'll keep it in mind for the future.
jrockway 15y ago
Many people write bad tests. So stop doing that, but don't stop writing tests. The reason why every public method should be tested is because changing how a public methods works is going to break everything using that public method. If you must make a change, fix the test along with the method. It broke to remind you to fix the consumers of your API. Good tests don't depend on implementation. Your tests should tests interfaces instead of objects and anything implementing the interface should pass the tests. If it doesn't, your interface is wrong or your test is wrong.
yason 15y ago
The thing with unit tests is that they test units. A unit of code might be a simple algorithm inside one function but more often than not the unit is a more complex interaction that is better covered by a smart test on a whole lot higher level. I prefer fewer tests that try to touch most of the essential features in one go, rather than several low-level tests that test simple if not idiotically limited cases. I prefer fewer tests and high code coverage: by being smart you can do a lot of testing even with a single test if only it touches the strategically most sensitive parts of your program. Also, most low-level stuff you will test implicitly by the rest of your code. Lots of your higher level stuff would fail to work if your lower level functionality was crappy. Thus, the basic utilities tend to straighten themselves (bug-wise) early on. Lines of code have a cost and lots of tests drive up your lines. Less code, smarter code. Less tests, smarter tests.

77 points by wtfdeveloper 15y ago

I've found that the higher level the test (i.e. unit -> integration -> functional) the better they catch things, but the harder it is to figure out what broke. For instance we had a rule at a place I worked, where we needed to have 80% unit test coverage. And what is described in this article happened. Anytime we'd make a small refactor, we'd have to go update 3-4 test libraries. And if it was a big one, then you have a lot of tests to update. And since some were complex (or since there were a lot of them) people didn't dig into that much. When people finally did it sometimes showed the test wasn't even testing what it was supposed to anymore. The other issue is we had a lot of bugs around the integration of data. For instance if the DB class returned back an empty array vs null vs empty string. When we did the unit test we mocked it to return what we thought it should, which may not be the correct one. Integration tests better caught this, functional tests even more so. It seems a lot of shops don't use tests at all, now working at places that have done both, I'd rather error on the side of fewer tests, and do it smart instead of hitting a metric and assuming your code falls into line.

wr1472 15y ago
Sounds like your test were quite brittle, if refactoring application code causes tests to break. A good mental rule I use is: "test the what not the how". It also a good reason to program (and test) to interfaces than to the implementation, although you have to be careful with this as it is all too easy to go OTT in your use of design patterns. You have to know how to strike the right balance.
- eaurouge 15y ago
  There are various kinds of tests, from unit tests to behavior driven testing. Refactoring application code may cause tests, especially unit tests, to break. If you're writing unit tests, by definition, you are testing the implementation - that's the point. It seems you're referring to behavioral (or black box) tests, which are very different.
  - wr1472 15y ago
    Agreed refactoring method signatures will cause tests to break, but they should be relatively trivial to fix. Black box tests are not limited to behavioral or system integration testing. you can black box at the unit level. In fact if you are doing proper TDD you will write the tests first, which help to define the interface of the class under test, make sure the test fails to ensure you are testing for something, and then finally write the implementation to make the test pass. Such an approach implies it is black box as you've defined your interface before your implementation. Remember you are testing what it does, not how it does it.
    - jsdalton 15y ago
      > Remember you are testing what it does, not how it does it. While I tend to agree with you (I strongly prefer black-box testing over white-box testing), this is a hotly debated topic and many people feel quite strongly the opposite. You can Google around and find many advocates for white box unit tests. Martin Fowler's article from several years ago I think does an excellent job of presenting a balanced look at both sides: http://martinfowler.com/articles/mocksArentStubs.html http://martinfowler.com/articles/mocksArentStubs.html
      - wr1472 15y ago
        I've not come across the article before. The section on coupling tests to implementation I guess is the most pertinent part (http://martinfowler.com/articles/mocksArentStubs.html#CouplingTestsToImplementations http://martinfowler.com/articles/mocksArentStubs.html#Coupli...). > Coupling to the implementation also interferes with refactoring, since implementation changes are much more likely to break tests than with classic testing. Fowler acknowledges two different styles of tests - state/behaviour also can be termed as classic/mockist. He doesn't really advocate one over the other. I do mock when I have to, but only when there is no other way to setup my test. It is certainly not the default. I find that taking this approach does not make my tests brittle, and makes them robust enough to handle the majority of refactoring exercises. Of course YMMV.
- keithnoizu 15y ago
  Exactly. You want to verify requirements not implementation details.Requirements change less frequently than implementation details.
jrockway 15y ago
For instance we had a rule at a place I worked, where we needed to have 80% unit test coverage. And what is described in this article happened. Anytime we'd make a small refactor, we'd have to go update 3-4 test libraries. And if it was a big one, then you have a lot of tests to update. And since some were complex (or since there were a lot of them) people didn't dig into that much. When people finally did it sometimes showed the test wasn't even testing what it was supposed to anymore. This just means your tests are bad. Write good tests and you won't have this problem. Integration and functional tests are good, but only as a quick sanity check. The nitty gritty details need to be handled by good unit tests; the functional tests only tell you things like "did the RPC protocol change" or "does the database cache actually cache something".
- RandallBrown 15y ago
  It is really easy to write bad tests. That doesn't mean that you shouldn't write any, but just "writing good tests" can be really hard.
capsule_toy 15y ago
For me, unit tests are to assure that if I refactor the code, the places where the code is being used won't be affected by my changes. If I have to rewrite the tests, I've lost this assurance.

16s 15y ago

I used to oppose unit testing because I had a good C++ compiler with all warnings turned on and I foolishly thought that was all I needed and that unit testing was only for dynamic languages that did not have compilers. I'm a big fan now though. If I have written code that's hard to test, I re-factor the code so that it's testable. I've found that un-testable code is generally bad code (it works OK, but is hard to modify later). I also run my unit tests during the build (right after compilation), so I know right away if something fails. I've come to really like unit testing. I'm less afraid to change things and feel better (in general) about the quality of my code.

franklindholm 15y ago
I agree that Unit testing is a power full tool. But tools are just tools, and they can be misused. I see this sometimes at companies where they "make" their developers write unit tests, they are just writing them because they have to and this often shows in the quality of the tests. Everything is green in Jenkins so obviously the functional tester did something wrong :)
3wetwetw 15y ago
C++ definitely also needs unit testing - even with untestable code (even when it's bad code, though not always, it may be easier to test it using something like Isolator++, rather than rewrite greenfield from scratch).

Swizec 15y ago

There is a time and place for unit tests - when you're creating algorithms. Most of the time, however, we are creating web apps and things that run at a much higher level of abstraction than the underlying algorithms. We should probably write tests at the same level of abstraction our solution is. Although, I guess, when you're writing medical software or something for a bank, you should be a bit more strict than when you are making the next best social sharing app. A guy I know once solved a bug by having a unit test for the md5 function from the standard library of the language at hand ... turns out a specific version was buggy. Either way, I prefer using integration/functional test. What I care about is that the right output flies in for a certain input. What I don't care about is how. So I don't test for that. Also, Unit tests must be created before the code they test. yes they should. It's very helpful to think about how your code will be used before you write it, helps you think it through as well.

lyudmil 15y ago
Usually the problem you'll get if you use a lot of functional/integration testing is that your test suite will run too slowly for you to run it often enough to validate your design. It ends up being something you run when you think you've completed a feature to make sure it works, but not after you've extracted a class to make your code cleaner. I have also had times when I moved away from unit testing in favor of higher-level tests, but ultimately found that the quality of the code got worse and the build time became unacceptable. I'm not trying to generalize here, I'm only talking about my code on a particular project at a particular point in time.
- jerf 15y ago
  "Usually the problem you'll get if you use a lot of functional/integration testing is that your test suite will run too slowly..." I've had good luck attacking this problem directly. If your integration tests are sufficiently integration-y, often this slowness directly reflects the fact that your system is slow to the users as well! Or, failing that, that you've got too many dependencies hardwired into your system, or some other real problem. Following up on my other proximal post here, the question is ultimate not whether it's high or low level, it's whether it's fast or slow, period. I've written slow low-level tests because they fully tested a 10x10x10x10x10x10 state space accidentally, I've written (after some work on the underlying system) some very fast tests that literally simulate user input to the system via the web server at the highest possible level. The real key is that you can't really follow a testing recipe, you always have to be looking out for the cost-benefits. I suspect the other key is that you must come to internalize the idea that automated testing is really important and if that means spending a man-week to improve your testing environment is what you need to do, then do it. It's really OK.

dabit 15y ago

Was this a sarcastic post?

Confusion 15y ago
What makes you think that?

agentultra 15y ago

So basically: don't test the implementation and don't test the framework. Do test the API. Test the output. Good, smart tests should be a pleasure to work with. They should give you the confidence to change the implementation without mercy. They make the feedback cycle faster, bugs more visible, and documentation more thorough. The only time I've found unit tests to be a pain are when testing asynchronous, event-driven code. The tests aren't any more brittle, but they really need to be supported by good integration tests in order to find the kinds of bugs that can surface in such systems. I think the point about unit tests is that they should be the default and not the exception. 100% coverage is an asymptotic value. But if you write the test first you can get pretty darn close.

dspillett 15y ago
> So basically: don't test the implementation. I thought that was the point of unit tests... > Do test the API. Test the output. ... and that this would be integration testing. Am I drawing my mental lines separating the range of tests in the wrong places?
- agentultra 15y ago
  I can't really say, but if you read The Clean Coder we might get on the same page. fwiw, what I mean when I say "don't test the implementation," is that your test should not test "how" a function or object performs its task when called. tldr; unit tests should test units -- the smallest amount of functionality possible A trivial example might be an object that retrieves some data from a persistent store and prints it out. It might have a "fetch" method whose argument is a key in the data store. You could easily write a test case that checks when "fetch" is called that the data is written to the standard output stream. It would pass and you could go home after a hard day of work. But if the requirements change and you need it to write to a socket instead and format the data for a binary stream -- you'll have to modify the "fetch" method and the tests. If we were clever though we'd realize that when we wrote the test to see that "fetch" output its result to standard out we were actually asking the object to do too much and our test was testing several different units of work that could be broken down. Ideally our hypothetical object would take a couple of callable objects that handle things like printing data to streams and formatting it. Then our test for fetch only has to test one small "unit" of functionality -- not several units (retrieving data, formatting it, printing it to a stream). hth

brown9-2 15y ago

A lot of these points apply to any sort of coding convention that a group agrees upon, not just unit testing.

prez 15y ago

You shall use a foreach every time you need to perform a loop. Works pretty well in Python. Though it doesn't invalidate the point the author tries to make.

recursive 15y ago
So you think while loops are unnecessary in python?
cbs 15y ago
Works pretty well in Python But xrange() in python is just a roundabout way to get back to having a regular for loop. for x in xrange(0,100) for(x=0, x<100, x++)

3wetwetw 15y ago

Absolutely - even those of us (like me - disclaimer: I work at Typemock) know that there is a time and a place for unit tests and TDD (not the same thing). In fact, we wrote about this before http://bitly.com/zdyJfl http://bitly.com/zdyJfl We're hosting a webinar on Wednesday about different kinds of testing and when a unit test is appropriate and when another kind of test, like an integration test, may be a better choice - http://bitly.com/xeYSYg http://bitly.com/xeYSYg. Unit tests have a time and place. There are times in which they are a must if you want to reduce technical debt and test your code - both if you're writing mission critical stuff or even when you're writing the latest social sharing app (remember the Fail Whale? We all hate that!). But, no, you don't need TDD on greenfield code all the time and you shouldn't have to write tests for logic.

owenjones 15y ago

The title and content don't correlate. This would be like having an article titled "The Evil Hammer" and then in the article explaining how "Hammer's aren't actually evil, just you shouldn't use them in situations where a saw would be more appropriate." I think anyone falling into these pitfalls just aren't writing good enough tests, don't dissuade them from furthering their testing skills, promote skillful testing. And change the title.

chaostheory 15y ago

I don't feel that this article is relevant for most programmers using newer languages such as Ruby or Python. There's a simple change in the way I use JUnit that I learned from using Ruby (RSpec, Cucumber) about 4-5 years ago: Don't test methods or functions. Instead test expected behavior e.g. a project requirement. Then you can pretty much avoid every problem listed in the article.

peteretep 15y ago

I wrote a similar article not too long ago: http://www.writemoretests.com/2011/09/test-driven-development-give-me-break.html http://www.writemoretests.com/2011/09/test-driven-developmen...

fuzzylizard 15y ago

I agree with only one point in the article; the ability to delete tests. Tests of all kinds should be living, breathing code, just like the code in the rest of your application. As such, tests, like production code, can become outdated and obsolete and need to be updated, changed, and sometimes deleted. However, I do not agree with the idea that tests are evil. If your tests are failing, or breaking regularly, or are hard to write, then your tests are trying to tell you something. Tests are a direct mirror of the state of your code. If your tests are brittle and hard to manage, don't delete the tests, fix your code. If you need to instantiate a tonne of objects in order to make your tests work, fix your methods dependencies and learn to test your methods in isolation of its dependencies. Any production code needs tests, this is non-negotiable in today's development eco-system. But I get tired of people who do not understand the purpose of tests complaining that tests are evil. If there is a problem with your tests, then your tests are telling you there is a problem with your code. Clean up your code in order to clean up your tests. However, remember that as code changes, your tests will need to change as well. It is okay to edit, delete, change tests to reflect the current state of the application. Just change the tests first to express how you want to interact with a specific bit of code and then change the code, not the other way around.

keithnoizu 15y ago

Sounds like a bit of a hyperbole, Blanket statements like 100% coverage, unit test per method are not too helpful, and it's important to understand the point where you will start to see diminishing returns with unit tests but that said well written unit tests with ~ 85-95 coverage by line free of conditional logic (use custom asserts) with meaningful test names/descriptions aren't exactly going to put you in league with beezlebub.

hashfold 15y ago

there are really nice comments below in this thread on why testing functionality is better than writing unit test cases to test the functions/methods/code. e.g. wr1472: test the what not the how MrEnigma: the higher level the test (i.e. unit -> integration -> functional) the better they catch things Keithnoizu: verify requirements not implementation details we understand that the unit testing is trying to test the functionality and making sure that the acceptance criteria is also met at the same time. I would like to mention one more point is that we should also try to write unit test cases not just only test the functionality but also try to do a good code coverage. unit test cases are tools for: 1. functionality testing 2. comply with acceptance criteria 3. a tool to help good code coverage 4. be iterative. be backward compatible as long as it could. (anyway to add logic based on version of the code change? need to think on this). yes it makes it little bulky but will work and avoid needing code cleanup and redesigns. let unit tests help come up with min 80% of code coverage.

lmm 15y ago

>You shall use a foreach every time you need to perform a loop. Yes >You will never access your database if you are not using an ORM. Hell yes. (Not necessarily disagreeing with his conclusions)

tikhonj 15y ago
What about while loops? That's the problem with blanket statements: they're very rarely 100% correct.

jwatte 15y ago

Unit tests, acceptance tests, and integration tests are different things. Unit tests test that the implementation behaves as intended, and generally grope around private internals. These change when implementation changes. Acceptance tests test that the public API is implemented correctly. These change when the API/interface change. Integration tests test that the end system implements requirements correctly, and change when requirements change. Most TDD hate comes from not separating these correctly, or not having suitable tools for each type, or from (gasp!) not actually having a clear understanding of requirements, interface, or implementation.

Peaker 15y ago

I find TDD to be somewhat redundant when I code Haskell. The types cover more ground than typical unit tests -- and if you follow a few simple principles (avoid partial functions and some of the worse part of the libraries), you can trust the compiled code more than you trust UT'd code. Production-quality Haskell does usually employ tests, and those are a joy to write (with QuickCheck), but there's little need to spend time writing tests at the early coding phases. When doing large refactorings, you can also trust the type system to guide you and find virtually all bugs introduced.

tikhonj 15y ago

Actually, his 2x3 example is not very good. The fact that 2x3 is 2 three times is the what, not the how; in a perfect world you would just assert that nxm == n + n + ... n m times for all n and all positive m. This is, coincidentally, where property based testing like QuickCheck comes in. Edit: Just pretend 2x3 has an asterisk instead of an x :)

prodigal_erik 15y ago
(FYI, 2 * 3 is okay; HN preserves * followed by whitespace.)
- tikhonj 15y ago
  Ah, thanks. That was probably the issue, because some of my asterisks worked but others didn't. I'll keep it in mind for the future.

jrockway 15y ago

Many people write bad tests. So stop doing that, but don't stop writing tests. The reason why every public method should be tested is because changing how a public methods works is going to break everything using that public method. If you must make a change, fix the test along with the method. It broke to remind you to fix the consumers of your API. Good tests don't depend on implementation. Your tests should tests interfaces instead of objects and anything implementing the interface should pass the tests. If it doesn't, your interface is wrong or your test is wrong.

yason 15y ago

The thing with unit tests is that they test units. A unit of code might be a simple algorithm inside one function but more often than not the unit is a more complex interaction that is better covered by a smart test on a whole lot higher level. I prefer fewer tests that try to touch most of the essential features in one go, rather than several low-level tests that test simple if not idiotically limited cases. I prefer fewer tests and high code coverage: by being smart you can do a lot of testing even with a single test if only it touches the strategically most sensitive parts of your program. Also, most low-level stuff you will test implicitly by the rest of your code. Lots of your higher level stuff would fail to work if your lower level functionality was crappy. Thus, the basic utilities tend to straighten themselves (bug-wise) early on. Lines of code have a cost and lots of tests drive up your lines. Less code, smarter code. Less tests, smarter tests.