7 ms·

> What I've noticed in practice however, is that occasionally, this process will allow an upgrade to a dependency that will pass the automated build and test st

> What I've noticed in practice however, is that occasionally, this process will allow an upgrade to a dependency that will pass the automated build and test step, but introduce the wildest runtime error into the application. Usually at the time when we aim to deliver something. Sounds like dependabot is very useful for uncovering insufficient test coverage or missing integration tests :)

dynamite-ready 4y ago
That would be a shallow reading, however. The last two major runtime issues wer actually one that broke the test runner and ignored a number of tests. And another runtime error was a Python Django specific sub dependency that broke the admin interface, which obviously, we don't explicitly test. On the other hand, very recently, we had to abort a release, because of an outdated dependency that Dependabot DID actually raise. Which is why I don't want to throw the baby out with the bathwater, as one or two people have suggested. But I can say that I think that the reality of working with Dependabot, is not very well reflected in popular online articles.
- viraptor 4y ago
  Most test runners have an option (or can be easily modified) to fail when 0 or less than X tests have been run. You should use it for situations like this.
  - lucasyvas 4y ago
    But what if there's also a bug where that feature doesn't work :)
    - viraptor 4y ago
      What if there's a bug where your CI ignores all steps? There's always some situation possible and we can go all the way to nasal demons. You have to accept the risk at some point.
- damienwebdev 4y ago
  > Python Django specific sub dependency, that broke the admin interface, which obviously, we don't explicitly test. There's your problem.
- cr3ative 4y ago
  > the admin interface, which obviously, we don't explicitly test This... is not obvious. It broke a part of your software which you care about. You care, it caused a problem, so it should be tested.
  - dynamite-ready 4y ago
    You routinely write unit tests for dependent packages in your apps?
    - manfre 4y ago
      This is not testing some other package. It's testing functionality provided by your app that happens to rely upon a 3rd party package. The 3rd party package has its tests, but doesn't know or care about the specific integration environment of your site. With regards to writing tests of the Django admin, you need to do it if your site is customizing and/or depending upon it.
    - 4ndrewl 4y ago
      But once you've imported that package it's part of _your_ codebase.
      - KronisLV 4y ago
        While this is true, many will probably disagree, just because they don't want to consider the maintenance burden that external dependencies will introduce. So between choosing to write everything themselves (and getting nothing done), writing tests against dependencies (and getting little done due to the overhead), or claiming that external dependencies should have tests of their own, many will pick the latter. Then again, in a world where create-react-app results in 180 MB of dependencies and about 1500 modules (probably different numbers now, using some older ones from my blog post), auditing security is an uphill battle, not even talking about actual testing. The situation in the back end development, isn't that much better either, to be honest, because once you look into the complexity of any framework like Spring, Laravel, Django, Rails etc., it becomes apparent that creating a fully featured framework like that is a huge undertaking. That said, you should at least test the bits where the external dependency is integrated with your codebase.
- Narretz 4y ago
  Was it an update to the test runner or test specific packages that broke the test runner? I would ignore infrastructure/testability/tooling packages in dependabot and do them manually to prevent these errors.
- dignati 4y ago
  > one that broke the test runner, and ignored a number of tests That's unfortunate! For the project I'm working on, we've "solved" that by showing the number of test and the difference to the number of tests that ran on main. FWIW, at previous jobs, upgrading Java dependencies was a major pain because they were all outdated and the latest versions introduced too many breaking changes for us. At my current job, we pretty much instantly merge all PRs from dependabot because we trust our CI. Upgrades rarely introduce problems and if they do, they are easy to fix.

dynamite-ready 4y ago

That would be a shallow reading, however. The last two major runtime issues wer actually one that broke the test runner and ignored a number of tests. And another runtime error was a Python Django specific sub dependency that broke the admin interface, which obviously, we don't explicitly test. On the other hand, very recently, we had to abort a release, because of an outdated dependency that Dependabot DID actually raise. Which is why I don't want to throw the baby out with the bathwater, as one or two people have suggested. But I can say that I think that the reality of working with Dependabot, is not very well reflected in popular online articles.

viraptor 4y ago
Most test runners have an option (or can be easily modified) to fail when 0 or less than X tests have been run. You should use it for situations like this.
- lucasyvas 4y ago
  But what if there's also a bug where that feature doesn't work :)
  - viraptor 4y ago
    What if there's a bug where your CI ignores all steps? There's always some situation possible and we can go all the way to nasal demons. You have to accept the risk at some point.
damienwebdev 4y ago
> Python Django specific sub dependency, that broke the admin interface, which obviously, we don't explicitly test. There's your problem.
cr3ative 4y ago
> the admin interface, which obviously, we don't explicitly test This... is not obvious. It broke a part of your software which you care about. You care, it caused a problem, so it should be tested.
- dynamite-ready 4y ago
  You routinely write unit tests for dependent packages in your apps?
  - manfre 4y ago
    This is not testing some other package. It's testing functionality provided by your app that happens to rely upon a 3rd party package. The 3rd party package has its tests, but doesn't know or care about the specific integration environment of your site. With regards to writing tests of the Django admin, you need to do it if your site is customizing and/or depending upon it.
  - 4ndrewl 4y ago
    But once you've imported that package it's part of _your_ codebase.
    - KronisLV 4y ago
      While this is true, many will probably disagree, just because they don't want to consider the maintenance burden that external dependencies will introduce. So between choosing to write everything themselves (and getting nothing done), writing tests against dependencies (and getting little done due to the overhead), or claiming that external dependencies should have tests of their own, many will pick the latter. Then again, in a world where create-react-app results in 180 MB of dependencies and about 1500 modules (probably different numbers now, using some older ones from my blog post), auditing security is an uphill battle, not even talking about actual testing. The situation in the back end development, isn't that much better either, to be honest, because once you look into the complexity of any framework like Spring, Laravel, Django, Rails etc., it becomes apparent that creating a fully featured framework like that is a huge undertaking. That said, you should at least test the bits where the external dependency is integrated with your codebase.
Narretz 4y ago
Was it an update to the test runner or test specific packages that broke the test runner? I would ignore infrastructure/testability/tooling packages in dependabot and do them manually to prevent these errors.
dignati 4y ago
> one that broke the test runner, and ignored a number of tests That's unfortunate! For the project I'm working on, we've "solved" that by showing the number of test and the difference to the number of tests that ran on main. FWIW, at previous jobs, upgrading Java dependencies was a major pain because they were all outdated and the latest versions introduced too many breaking changes for us. At my current job, we pretty much instantly merge all PRs from dependabot because we trust our CI. Upgrades rarely introduce problems and if they do, they are easy to fix.