7 ms·

The article makes some fair points for performing end-to-end tests. There is however a benefit of unit tests over end-to-end tests that is not mentioned: code l

The article makes some fair points for performing end-to-end tests. There is however a benefit of unit tests over end-to-end tests that is not mentioned: code locality. The cost of fixing a discovered bug seems to grow exponentially with the amount of code in which the bug may appear. Unit tests typically cover a small part of the code. If a unit test fails, isolating and fixing the bug is typically cheap. On the other hand, if an end-to-end test fails (due to the same bug) then isolating and fixing the bug can be quite costly because there are many different parts of the code that need to be checked. To me, code locality is the main advantage of writing unit tests. At the end of the day, there is a spectrum of tests going from unit tests to end-to-end tests. The spectrum represents several trade-offs such code locality vs. coverage. In my experience, the most economical approach is to write a balanced mix of tests the lie along this spectrum.

lbill 8y ago
You are right! I work on test automation for end-to-end testing. I dare say my work is very useful and has prevented a lot of bugs from hitting our prod. But at my workplace we also all agree on the fact that it is often quite painful to locate the cause of the bugs that I report, precisely because we don't have enough unit tests on our old code. Since we live in a world of limited budget and time to spend, I agree with the conclusion of the article: "use unit tests where it makes sense". It is what we try to implement on our new code (and when refactoring legacy code).
- deleted 8y ago
  [deleted]
mosselman 8y ago
I hear this argument a lot, but I do not think it is valid. What happens is that you simply put in the time before a bug occurs with every unit test that you write. Which isn't very efficient, seeing as not all of your units will lead to bugs. So yes, once you have all of your Units covered 100% it will be easier to find bugs, but you have invested a lot of time in order to get here and in order to keep the unit-test-suite maintained.
- Klathmon 8y ago
  It's a trade off of when you want to spend the time. Sure, it's less efficient to write unit tests ahead of time for every case, but in a lot of cases fixing a bug faster when it's discovered is much more important than even double the hours spent during less "pressing" times.
  - skohan 8y ago
    But I think the argument that the article is making is that it's not only the case that you're moving the time spent fixing bugs, (i.e. a greedy vs lazy strategy), the issue is also that the baseline maintenance cost of the code base is increased by orders of magnitude when unit tests are considered. I tend to agree - in my personal experience in organizations with strict TDD culture, a perverse incentive often emerges to preserve existing flawed architecture over obviously better solutions just because it's so painful to deal with all the tests. One of software development's most powerful properties is the ability to iterate quickly: it's foolish to prioritize dogmatic beliefs about testing over that quality.
    - Klathmon 8y ago
      Yeah I absolutely agree that in many cases tests become more of burden than a help, but I think that isn't a problem with testing but with it's application. It's just an extension of your code, if you are going to throw some code away to change an interface, then throw the tests away too. If you are afraid to do that because of the time spent, then you probably spent too much time on writing tests. While I am dogmatic about tests, I also believe that around 50% code coverage is normally enough in most application codebases. Cover the important parts, the parts that are hard to test manually, the "core" pieces that lots of other areas rely on, and some tests for bugs that you want to prevent happening again. If you want to quickly iterate, go for it! You shouldn't have all that many tests in the parts you are changing frequently. But to change a core aspect of the codebase, or a really complicated aspect of it, then the extra work of rewriting the tests shouldn't be all that bad. I'd shorten your last statement. It's foolish to prioritize dogmatic beliefs. Everything has a time and a place, and moderation is key.
StavrosK 8y ago
That's exactly what the article is saying. Also, your point about unit tests assumes that tests cost the same to write. Sure, a unit test will have ten times better locality than an integration test, but you have to write ten of them to catch a bug. Also, unit tests don't catch bugs where they're very likely to crop up: the boundaries.
- conradk 8y ago
  You have to write 10 of them if they are not part of your workflow. If you TDD (which is applicable in a lot of cases in my experience with web development), then instead of testing with Curl or a web browser, you write tests. This takes the same amount of time upfront, so the additional cost of writing the tests is actually 0. Not sure about other types of development, but in web development, TDD can be a good way to have automated tests without the additional cost.
  - StavrosK 8y ago
    Yes, if you've already written the tests, the tests will take no time to write. That doesn't mean they were free the first time around.
    - conradk 8y ago
      What I mean is that compared to testing in browser, writing unit tests and using those tests instead of the browser takes about the same amount of time. So doing tests in browser or in TDD ends up taking the same amount of time upfront. So testing in browser is as expensive as writing unit tests with TDD.
      - szatkus 8y ago
        If you're talking about emulating browser requests by tests you are already at integration tests territory.
        conradk 8y ago
crdoconnor 8y ago
>At the end of the day, there is a spectrum of tests going from unit tests to end-to-end tests. The spectrum represents several trade-offs such code locality vs. coverage. In my experience, the most economical approach is to write a balanced mix of tests the lie along this spectrum. IME unit tests work acceptably in one very specific scenario and fail pretty badly in all others. That scenario being: 1) You're surrounding a self contained block of code that interacts "with the outside world" via a code API. 2) That code API is a very stable and clean abstraction. 3) It has minimal interactions with modules outside of it and those interactions that it does have are tightly scoped (i.e. minimal to zero mock objects are required to write the test). 4) The logic of the code is relatively complex and most bugs that crop up are logical in nature (e.g. off by one, things getting swapped around, incorrect calculations, wrong behavior with negative numbers). Meanwhile, integration tests (at varying levels) work well for pretty much every case apart from this and still work okay for this type of code. They make much more sense as a go-to default. I've also worked on several projects where there was little to no code that it actually made sense to unit test. It's not uncommon that an entire codebase is predicated mainly on hooking systems together and doing some shallow calculations. IMHO, having zero unit tests in that environment is actually desirable. The worst unit tests I've seen have been written when two or more of those preconditions have failed. They would fail constantly, require massive maintenance and, somewhat comically, almost never fail in the presence of an actual bug.
- Mouse47 8y ago
  This is spot on IMO. I often see people touting the benefits of unit tests during a refactoring...but 95% of the time refactoring involves modifying class APIs since the hardest part of development is getting the object model right. Unit tests only assist refactoring when you don't modify the APIs - in other cases they are a burden.
  - cliffy 8y ago
    Yes, if you change a unit's interface you will have to also change any code relying on said interface. That's a maintenance cost of unit tests. But it doesn't follow that changing a unit's interface means unit tests suddenly become just a burden. Ideally unit tests are, well, testing a bunch of core functionality of the unit under test. You adapt them to the new interface. Then you're back to having a quick, automatic sanity check you can run against the unit whenever you have to make a change. I don't understand people bemoaning this 'cost' of unit tests when the benefits they provide typically far outweigh the costs. It's possible broader functional/integration tests have a better ROI in certain situations, but they come with a maintenance cost as well.
deleted 8y ago
[deleted]
marcosdumay 8y ago
> The cost of fixing a discovered bug seems to grow exponentially with the amount of code in which the bug may appear. That's completely at odds with my experience. I find that for local bugs, the cost of locating them grows with O(log n) of the amount of code. And for non-local bugs (interface bugs, system bugs, incompatible specs...) unit tests don't catch them anyway.
JamesBarney 8y ago
> The cost of fixing a discovered bug seems to grow exponentially with the amount of code in which the bug may appear. I don't think this is true. Fixing a bug is comprised of four parts. 1. Understanding and reproducing the bug. 2. Finding the code responsible for the bug. 3. Coming up with code that fixes the bug. 4. Verifying that your fixing code does not introduce any new bugs. #2 is the only one that could even theoretically be exponential. The only bugs I've found where I've spent days on 2, were hard to reproduce, intermittent, race condition bugs, which units tests aren't very good at finding anyway.
calyth2018 8y ago
At the practical level, I find that most end-to-end test tells you something failed, but not which part failed. A unit test, or a component test, would tell you that something failed, and it's in this general area, which narrows down the search quite a bit. They're both useful, but I've seen far more problems with people arguing that end-to-end test is more than what they need, while a bad conversion from seconds to nanoseconds would be caught quickly if a unit test was actually written.

lbill 8y ago

You are right! I work on test automation for end-to-end testing. I dare say my work is very useful and has prevented a lot of bugs from hitting our prod. But at my workplace we also all agree on the fact that it is often quite painful to locate the cause of the bugs that I report, precisely because we don't have enough unit tests on our old code. Since we live in a world of limited budget and time to spend, I agree with the conclusion of the article: "use unit tests where it makes sense". It is what we try to implement on our new code (and when refactoring legacy code).

deleted 8y ago
[deleted]

mosselman 8y ago

I hear this argument a lot, but I do not think it is valid. What happens is that you simply put in the time before a bug occurs with every unit test that you write. Which isn't very efficient, seeing as not all of your units will lead to bugs. So yes, once you have all of your Units covered 100% it will be easier to find bugs, but you have invested a lot of time in order to get here and in order to keep the unit-test-suite maintained.

Klathmon 8y ago
It's a trade off of when you want to spend the time. Sure, it's less efficient to write unit tests ahead of time for every case, but in a lot of cases fixing a bug faster when it's discovered is much more important than even double the hours spent during less "pressing" times.
- skohan 8y ago
  But I think the argument that the article is making is that it's not only the case that you're moving the time spent fixing bugs, (i.e. a greedy vs lazy strategy), the issue is also that the baseline maintenance cost of the code base is increased by orders of magnitude when unit tests are considered. I tend to agree - in my personal experience in organizations with strict TDD culture, a perverse incentive often emerges to preserve existing flawed architecture over obviously better solutions just because it's so painful to deal with all the tests. One of software development's most powerful properties is the ability to iterate quickly: it's foolish to prioritize dogmatic beliefs about testing over that quality.
  - Klathmon 8y ago
    Yeah I absolutely agree that in many cases tests become more of burden than a help, but I think that isn't a problem with testing but with it's application. It's just an extension of your code, if you are going to throw some code away to change an interface, then throw the tests away too. If you are afraid to do that because of the time spent, then you probably spent too much time on writing tests. While I am dogmatic about tests, I also believe that around 50% code coverage is normally enough in most application codebases. Cover the important parts, the parts that are hard to test manually, the "core" pieces that lots of other areas rely on, and some tests for bugs that you want to prevent happening again. If you want to quickly iterate, go for it! You shouldn't have all that many tests in the parts you are changing frequently. But to change a core aspect of the codebase, or a really complicated aspect of it, then the extra work of rewriting the tests shouldn't be all that bad. I'd shorten your last statement. It's foolish to prioritize dogmatic beliefs. Everything has a time and a place, and moderation is key.

StavrosK 8y ago

That's exactly what the article is saying. Also, your point about unit tests assumes that tests cost the same to write. Sure, a unit test will have ten times better locality than an integration test, but you have to write ten of them to catch a bug. Also, unit tests don't catch bugs where they're very likely to crop up: the boundaries.

conradk 8y ago
You have to write 10 of them if they are not part of your workflow. If you TDD (which is applicable in a lot of cases in my experience with web development), then instead of testing with Curl or a web browser, you write tests. This takes the same amount of time upfront, so the additional cost of writing the tests is actually 0. Not sure about other types of development, but in web development, TDD can be a good way to have automated tests without the additional cost.
- StavrosK 8y ago
  Yes, if you've already written the tests, the tests will take no time to write. That doesn't mean they were free the first time around.
  - conradk 8y ago
    What I mean is that compared to testing in browser, writing unit tests and using those tests instead of the browser takes about the same amount of time. So doing tests in browser or in TDD ends up taking the same amount of time upfront. So testing in browser is as expensive as writing unit tests with TDD.
    - szatkus 8y ago
      If you're talking about emulating browser requests by tests you are already at integration tests territory.
      - conradk 8y ago

crdoconnor 8y ago

>At the end of the day, there is a spectrum of tests going from unit tests to end-to-end tests. The spectrum represents several trade-offs such code locality vs. coverage. In my experience, the most economical approach is to write a balanced mix of tests the lie along this spectrum. IME unit tests work acceptably in one very specific scenario and fail pretty badly in all others. That scenario being: 1) You're surrounding a self contained block of code that interacts "with the outside world" via a code API. 2) That code API is a very stable and clean abstraction. 3) It has minimal interactions with modules outside of it and those interactions that it does have are tightly scoped (i.e. minimal to zero mock objects are required to write the test). 4) The logic of the code is relatively complex and most bugs that crop up are logical in nature (e.g. off by one, things getting swapped around, incorrect calculations, wrong behavior with negative numbers). Meanwhile, integration tests (at varying levels) work well for pretty much every case apart from this and still work okay for this type of code. They make much more sense as a go-to default. I've also worked on several projects where there was little to no code that it actually made sense to unit test. It's not uncommon that an entire codebase is predicated mainly on hooking systems together and doing some shallow calculations. IMHO, having zero unit tests in that environment is actually desirable. The worst unit tests I've seen have been written when two or more of those preconditions have failed. They would fail constantly, require massive maintenance and, somewhat comically, almost never fail in the presence of an actual bug.

Mouse47 8y ago
This is spot on IMO. I often see people touting the benefits of unit tests during a refactoring...but 95% of the time refactoring involves modifying class APIs since the hardest part of development is getting the object model right. Unit tests only assist refactoring when you don't modify the APIs - in other cases they are a burden.
- cliffy 8y ago
  Yes, if you change a unit's interface you will have to also change any code relying on said interface. That's a maintenance cost of unit tests. But it doesn't follow that changing a unit's interface means unit tests suddenly become just a burden. Ideally unit tests are, well, testing a bunch of core functionality of the unit under test. You adapt them to the new interface. Then you're back to having a quick, automatic sanity check you can run against the unit whenever you have to make a change. I don't understand people bemoaning this 'cost' of unit tests when the benefits they provide typically far outweigh the costs. It's possible broader functional/integration tests have a better ROI in certain situations, but they come with a maintenance cost as well.

deleted 8y ago

[deleted]

marcosdumay 8y ago

> The cost of fixing a discovered bug seems to grow exponentially with the amount of code in which the bug may appear. That's completely at odds with my experience. I find that for local bugs, the cost of locating them grows with O(log n) of the amount of code. And for non-local bugs (interface bugs, system bugs, incompatible specs...) unit tests don't catch them anyway.

JamesBarney 8y ago

> The cost of fixing a discovered bug seems to grow exponentially with the amount of code in which the bug may appear. I don't think this is true. Fixing a bug is comprised of four parts. 1. Understanding and reproducing the bug. 2. Finding the code responsible for the bug. 3. Coming up with code that fixes the bug. 4. Verifying that your fixing code does not introduce any new bugs. #2 is the only one that could even theoretically be exponential. The only bugs I've found where I've spent days on 2, were hard to reproduce, intermittent, race condition bugs, which units tests aren't very good at finding anyway.

calyth2018 8y ago

At the practical level, I find that most end-to-end test tells you something failed, but not which part failed. A unit test, or a component test, would tell you that something failed, and it's in this general area, which narrows down the search quite a bit. They're both useful, but I've seen far more problems with people arguing that end-to-end test is more than what they need, while a bad conversion from seconds to nanoseconds would be caught quickly if a unit test was actually written.