5 ms·

Show HN: FlakyBot – Identify and suppress flaky tests

27 points by ankitdce 5y ago

ankitdce 5y ago
Hi HN, we are Spriha and Ankit building Flakybot is a tool to automatically identify and suppress test-flakiness so that developers are better able to trust their test results. Most CI systems leave it up to teams to manually identify and debug test flakiness. Since most CI systems today don’t handle test reruns, teams just end up with manually rerunning tests that are flaky. Ultimately, tribal knowledge gets built over time where certain tests are known to be flaky, but the flakiness isn’t specifically addressed. Our solution, Flakybot, removes one of the hardest parts of the problem: identifying flaky tests in the first place. We ingest test artifacts from CI systems, and note when builds are healthy (so that we can mark them as “known-good builds” to use while testing for flakiness). This helps automatically identify flakiness, and proactively offer mitigation strategies, both in the short term and long term. You can read more about this here: https://ritzy-angelfish-3de.notion.site/FlakyBot-How-it-works-5ab7e20ca01e4efc93732897501e52b4 https://ritzy-angelfish-3de.notion.site/FlakyBot-How-it-work... We’re in the early stages of development and are opening up Flakybot for private beta to companies that have serious test-flakiness issues. The CI systems we currently support are Jenkins, CircleCI and BuildKite, but if your team uses a different CI and has very serious test-flakiness problems, sign up anyway and we’ll reach out. During the private beta, we’ll work closely with our users to ensure their test flakiness issues are resolved before we open it up more broadly.
ncmncm 5y ago
What I want is a tool to make flaky tests fail reliably. They won't be fixed until they start actually preventing commits. If somebody deletes a test, that is on that person. I don't want a tool automatically suppressing testing.
- ankitdce 5y ago
  That's very interesting feedback. We certainly don't have a way to force simulate failure. A related capability we are working on is to also rerun the identified flaky tests X times so they pass. This depends on the capabilities of the test runner, so it will work with specific ones first (cypress, pytest, etc). That way you still make sure that flaky tests pass instead of supressing.
  - ncmncm 5y ago
    It should run the flaky tests N times, and report failure if any run fails. Hiding failures is not doing us a favor.
- berkay 5y ago
  What if it was easy to see why a test is flaky, compare failed/successful test runs like a code diff? Would that be useful? This is what we're building at Thundra (foresight product), instrument the tests as well as backend services to enable devs to quickly diagnose failing/flaky tests. Would appreciate any feedback you may have, here or privately.
  - ncmncm 5y ago
    It would be helpful to be able to present diffs of log output between successful and failing runs of a test. This is tricky to implement, for several reasons. Log output is normally timestamped, making every line unique. Those parts of log lines would need to be ignored when comparing between runs. Log output ordering is often indeterminate, particularly when a test has multiple threads, or interacts with an external service. Often the order of events logged is an essential feature of the difference between a successful and failed run. But some or most order differences are just incidental. The number of logged events may vary incidentally, or significantly. Explaining all these differences in detail to the test system would be too hard. So, the system needs to discover as much as possible of this for itself, and represent these discoveries symbolically. Then, allow a test to be annotated to override default judgments about the diagnostic significance of these features.
olafmol 5y ago
Congrats on getting Flakybot into private beta! For people that prefer to minimize the number of moving parts, CircleCI now also has built-in Flaky test detection: https://circleci.com/blog/introducing-test-insights-with-flaky-test-detection/ https://circleci.com/blog/introducing-test-insights-with-fla... Of course this is coupled to our CircleCI platform, so if you want to stay agnostic for sure check out Flakybot :) NB I work for CircleCI
gitgud 5y ago
Tests can fail for a multitude of "flakey" reasons, that aren't due to 'source code' changes. - Network connections failed - Run out of memory - Duration timeouts - Setting up the test infrastructure failed All these can contribute to flakiness, and could be detected and reported by a bot like this. Making this a useful idea! "Your test failed, but it was consider a flakey fail [Network Flakiness]" - FlakyBot
parthi 5y ago
We've been relying on manual testing so far. We're just starting to think about unit tests and integration tests. We don't know where to start. Would be cool if you could provide guidance on setting up good testing practices in the first place so that we avoid flaky tests all together.
- ankitdce 5y ago
  Yeh flaky tests generally creep up in a big service due to several issues. There are some best practices to avoid the tests that requires some discipline and good oversight! We wrote some stuff around it: https://www.flakybot.com/blog/five-causes-for-flaky-tests https://www.flakybot.com/blog/five-causes-for-flaky-tests This is by no means an exhaustive list, but our goal with FlakyBot is to get better at identifying root causes as we identify flakiness across the systems.
tossaway9000 5y ago
How about you fix the flaky tests? Am I insane for thinking that? The whole concept of "just reboot it" or "re run it again" and "fixing" the problem is at least one reason the modern world sits on a mountain of complete garbage software.
- ankitdce 5y ago
  Haha great point. Well from what we have learned from our users is "fixing" test typically end up with "delete most of them". Fixing tests can be time consuming effort. Another way to think about it is, whether Flaky tests are worth keeping? At some point if the tests fail often, do these really add value. And we think - it does. If you are able to identify flakiness from real failure and reduce noise, you can still avoid real failures.
  - rio517 5y ago
    Wow. That works like really poor technical leadership. Fixing flaky tests (as opposed to deleting them) is indeed time consuming, but it is a far cheaper choice than getting to the point your test suite is untrustworthy. There may be a point where the cost of ownership for a specific test exceeds its utility, but the way to resolve that is usually to reevaluate your code and supporting tests. Suppressing flaky tests seems a very unwise choice. Perhaps under extreme circumstances and with unhealthy code bases there may be a case for this, but I struggle to imagine it.
    - ankitdce 5y ago
      That is a fair argument. Not all organizations have the bandwidth to measure and manage stability of builds. Some companies build internal tools / dev productivity team for this purpose. There are always right intentions to comment out the flaky test with the mindset of coming back to it, but it is also a very low priority item in most cases when you have to ship new features. Fixing flaky tests can very commonly take longer than writing new tests.
- wink 5y ago
  Let me give you the example of a test that hadn't ever failed on a dev machine or on staging or prod, just on the flaky CI infrastructure. Yes, I'm mostly agreeing with you that the tests should be fixed, but I have seen ones that were perfectly fine (given the constraints) and what should have been fixed was the CI.
quickthrower2 5y ago
Nice. I’m for us building a rerun bot and I’m trying to advocate for it at work but not everyone agrees. Nice to pull this out as a service. Well done.

27 points by ankitdce 5y ago

Hi HN, we are Spriha and Ankit building Flakybot is a tool to automatically identify and suppress test-flakiness so that developers are better able to trust their test results. Most CI systems leave it up to teams to manually identify and debug test flakiness. Since most CI systems today don’t handle test reruns, teams just end up with manually rerunning tests that are flaky. Ultimately, tribal knowledge gets built over time where certain tests are known to be flaky, but the flakiness isn’t specifically addressed. Our solution, Flakybot, removes one of the hardest parts of the problem: identifying flaky tests in the first place. We ingest test artifacts from CI systems, and note when builds are healthy (so that we can mark them as “known-good builds” to use while testing for flakiness). This helps automatically identify flakiness, and proactively offer mitigation strategies, both in the short term and long term. You can read more about this here: https://ritzy-angelfish-3de.notion.site/FlakyBot-How-it-works-5ab7e20ca01e4efc93732897501e52b4 https://ritzy-angelfish-3de.notion.site/FlakyBot-How-it-work... We’re in the early stages of development and are opening up Flakybot for private beta to companies that have serious test-flakiness issues. The CI systems we currently support are Jenkins, CircleCI and BuildKite, but if your team uses a different CI and has very serious test-flakiness problems, sign up anyway and we’ll reach out. During the private beta, we’ll work closely with our users to ensure their test flakiness issues are resolved before we open it up more broadly.

ncmncm 5y ago

What I want is a tool to make flaky tests fail reliably. They won't be fixed until they start actually preventing commits. If somebody deletes a test, that is on that person. I don't want a tool automatically suppressing testing.

ankitdce 5y ago
That's very interesting feedback. We certainly don't have a way to force simulate failure. A related capability we are working on is to also rerun the identified flaky tests X times so they pass. This depends on the capabilities of the test runner, so it will work with specific ones first (cypress, pytest, etc). That way you still make sure that flaky tests pass instead of supressing.
- ncmncm 5y ago
  It should run the flaky tests N times, and report failure if any run fails. Hiding failures is not doing us a favor.
berkay 5y ago
What if it was easy to see why a test is flaky, compare failed/successful test runs like a code diff? Would that be useful? This is what we're building at Thundra (foresight product), instrument the tests as well as backend services to enable devs to quickly diagnose failing/flaky tests. Would appreciate any feedback you may have, here or privately.
- ncmncm 5y ago
  It would be helpful to be able to present diffs of log output between successful and failing runs of a test. This is tricky to implement, for several reasons. Log output is normally timestamped, making every line unique. Those parts of log lines would need to be ignored when comparing between runs. Log output ordering is often indeterminate, particularly when a test has multiple threads, or interacts with an external service. Often the order of events logged is an essential feature of the difference between a successful and failed run. But some or most order differences are just incidental. The number of logged events may vary incidentally, or significantly. Explaining all these differences in detail to the test system would be too hard. So, the system needs to discover as much as possible of this for itself, and represent these discoveries symbolically. Then, allow a test to be annotated to override default judgments about the diagnostic significance of these features.

olafmol 5y ago

Congrats on getting Flakybot into private beta! For people that prefer to minimize the number of moving parts, CircleCI now also has built-in Flaky test detection: https://circleci.com/blog/introducing-test-insights-with-flaky-test-detection/ https://circleci.com/blog/introducing-test-insights-with-fla... Of course this is coupled to our CircleCI platform, so if you want to stay agnostic for sure check out Flakybot :) NB I work for CircleCI

gitgud 5y ago

Tests can fail for a multitude of "flakey" reasons, that aren't due to 'source code' changes. - Network connections failed - Run out of memory - Duration timeouts - Setting up the test infrastructure failed All these can contribute to flakiness, and could be detected and reported by a bot like this. Making this a useful idea! "Your test failed, but it was consider a flakey fail [Network Flakiness]" - FlakyBot

parthi 5y ago

We've been relying on manual testing so far. We're just starting to think about unit tests and integration tests. We don't know where to start. Would be cool if you could provide guidance on setting up good testing practices in the first place so that we avoid flaky tests all together.

ankitdce 5y ago
Yeh flaky tests generally creep up in a big service due to several issues. There are some best practices to avoid the tests that requires some discipline and good oversight! We wrote some stuff around it: https://www.flakybot.com/blog/five-causes-for-flaky-tests https://www.flakybot.com/blog/five-causes-for-flaky-tests This is by no means an exhaustive list, but our goal with FlakyBot is to get better at identifying root causes as we identify flakiness across the systems.

tossaway9000 5y ago

How about you fix the flaky tests? Am I insane for thinking that? The whole concept of "just reboot it" or "re run it again" and "fixing" the problem is at least one reason the modern world sits on a mountain of complete garbage software.

ankitdce 5y ago
Haha great point. Well from what we have learned from our users is "fixing" test typically end up with "delete most of them". Fixing tests can be time consuming effort. Another way to think about it is, whether Flaky tests are worth keeping? At some point if the tests fail often, do these really add value. And we think - it does. If you are able to identify flakiness from real failure and reduce noise, you can still avoid real failures.
- rio517 5y ago
  Wow. That works like really poor technical leadership. Fixing flaky tests (as opposed to deleting them) is indeed time consuming, but it is a far cheaper choice than getting to the point your test suite is untrustworthy. There may be a point where the cost of ownership for a specific test exceeds its utility, but the way to resolve that is usually to reevaluate your code and supporting tests. Suppressing flaky tests seems a very unwise choice. Perhaps under extreme circumstances and with unhealthy code bases there may be a case for this, but I struggle to imagine it.
  - ankitdce 5y ago
    That is a fair argument. Not all organizations have the bandwidth to measure and manage stability of builds. Some companies build internal tools / dev productivity team for this purpose. There are always right intentions to comment out the flaky test with the mindset of coming back to it, but it is also a very low priority item in most cases when you have to ship new features. Fixing flaky tests can very commonly take longer than writing new tests.
wink 5y ago
Let me give you the example of a test that hadn't ever failed on a dev machine or on staging or prod, just on the flaky CI infrastructure. Yes, I'm mostly agreeing with you that the tests should be fixed, but I have seen ones that were perfectly fine (given the constraints) and what should have been fixed was the CI.

quickthrower2 5y ago

Nice. I’m for us building a rerun bot and I’m trying to advocate for it at work but not everyone agrees. Nice to pull this out as a service. Well done.