6 ms·

Deprecations via warnings don't work for Python libraries

63 points by scolby33 9mo ago

theamk 9mo ago
Deprecations via warnings don't reliably work anywhere, in general. If you are a good developer, you'll have extensive unit test coverage and CI. You never see the unit test output (unless they fail) - so warnings go unnoticed. If you are a bad developer, you have no idea what you are doing and you ignore all warnings unless program crashes.
- SethMLarson 9mo ago
  Author here! Agreed that are different levels of "engaged" from users, which is okay. The concerning part of this finding is that even dependent users that I know to be highly engaged didn't respond to the deprecation warnings, so they're not working for even the most engaged users.
- eternityforest 9mo ago
  Why is it that CI tools don't make warnings visible? Why are they ignored by default in the first place? Seems like that should be a rather high priority.
  - metadat 9mo ago
    Most of the time they don't matter and aren't an immediate problem. The Business doesn't care about warnings, they want working software NOW.
  - bluGill 9mo ago
    It isn't that easy. If you have a new warning on upgrade you probably want to work on it "next week", but that means you need to ignore it for a bit. Or you might still want to support a really old version without the new API and so you can't fix it now.
    - Hizonner 9mo ago
      > If you have a new warning on upgrade you probably want to work on it "next week", but that means you need to ignore it for a bit. So you create a bug report or an issue or a story or whatever you happen to call it, and you make sure it gets tracked, and you schedule it with the rest of your work. That's not the same thing as "ignoring" it.
      - bluGill 9mo ago
        And you always have something more important/interesting to do and so never get around to it.
        Hizonner 9mo ago
        ... which means that when the axe falls, the results are 100 percent your fault.
  - michaelt 9mo ago
    > Why is it that CI tools don't make warnings visible? A developer setting up CI decides to start an ubuntu 24.04 container and run 'apt-get install npm' This produces 3,600 lines of logging (5.4 log lines per package, 668 packages) and 22 warnings (all warnings about man page creation being skipped) Then they decide "Nobody's going to read all that, and the large volume might bury important information. I think I'll hide console output for processes that don't fail." Now your CI doesn't show warnings.
- optionalsquid 9mo ago
  You can turn warnings into errors with the `-Werror` option. I personally use that in CI runs, along with the `-X dev` option to enable additional runtime checks. Though that wont solve the author's problem, since most Python devs don't use either of those options
  - Y-bar 9mo ago
    In PHP I don’t think there is a native way to convert E_DEPRECATED into E_ERROR, but the most common testing framework has a quick way of doing the same. https://docs.phpunit.de/en/12.5/configuration.html#the-failondeprecation-attribute https://docs.phpunit.de/en/12.5/configuration.html#the-failo...
- rimunroe 9mo ago
  > If you are a good developer, you'll have extensive unit test coverage and CI. You never see the unit test output (unless they fail) - so warnings go unnoticed. In my opinion test suites should treat any output other than the reporter saying that a test passed as a test failure. In JavaScript I usually have part of my test harness record calls to the various console methods. At the end of each test it checks to see if any calls to those methods were made, and if they were it fails the tests and logs the output. Within tests if I expect or want some code to produce a message, I wrap the invocation of that code in a helper which requires two arguments: a function to call and an expected output. If the code doesn't output a matching message, doesn't output anything, or outputs something else then the helper throws and explains what went wrong. Otherwise it just returns the result of the called function: let result = silenceWarning(() => user.getV1ProfileId(), /getV1ProfileId has been deprecated/); expect(result).toBe('foo'); This is dead simple code in most testing frameworks. It makes maintaining and working with the test suite becomes much easier as when something starts behaving differently it's immediately obvious rather than being hidden in a sea of noise. It makes working with dependencies easier because it forces you to acknowledge things like deprecation warnings when they get introduced and either solve them there or create an upgrade plan.
- ploxiln 9mo ago
  When I update python version, python packages, container image, etc for a service, I take a quick look at CI output, in addition to the all the other checks I do (like a couple basic real-world-usage end-to-end usage tests), to "smoke test" whether something not caught by outright CI failure caused some subtle problem. So, I do often see deprecation warnings in CI output, and fix them. Am I a bad developer? I think the mistake here is making some warnings default-hidden. The developer who cares about the user running their the app in a terminal can add a line of code to suppress them for users, and be more aware of this whole topic as a result (and have it more evident near the entrypoint of the program, for later devs to see also). I think that making warnings error or hidden removes warnings as a useful tool. But this is an old argument: Who should see Python warnings? (2017) https://lwn.net/Articles/740804/ https://lwn.net/Articles/740804/
- Hizonner 9mo ago
  If you are a good developer, you consider warnings to be errors until proven otherwise.
  - hiq 9mo ago
    What does a good developer do when working in a codebase with hundreds of warnings? Or are you only considering a certain warnings?
    - Hizonner 9mo ago
      Why does your codebase generate hundreds of warnings, given that every time one initially appeared, you should have stamped it out (or specifically marked that one warning to be ignored)? Start with one line of code that doesn't generate a warning. Add a second line of code that doesn't generate a warning...
      - stackskipton 9mo ago
        Because most people are working at Failure/Feature factories where they might work on something and at last minute, they find out something is now warning. If they work on fixing it, the PM will screaming about time slippage and be like "I want you to work on X, not Y which can wait". 2 Years later, you have hundreds of warning.
        Hizonner 9mo ago
        You found that out at the last minute. So then you did a release. It's no longer the last minute. Now what's your excuse for the next release? If your management won't resource your project to the point where you can assure that the software is correct, you might want to see if you can find the free time to look for another job. You'll have to do that anyway when they either tank the company, or lay you off next time they feel they need to cut more costs.
      - hiq 9mo ago
        It's rare that I work on a project I myself started. If I start working on an existing codebase, the warnings might be there already. Then what do I do? I'm also referring to all the warnings you might get if you use an existing library. If the requirements entail that I use this library, should I just silence them all? But I'm guessing you might be talking about more specific warnings. Yes I do fix lints specific to my new code before I commit it, but a lot of warnings might still be logged at runtime, and I may have no control over them.
- mrweasel 9mo ago
  There was this one library we depended on, it was sort of in limbo during the Python 2 -> 3 migration. During that period is was maintained by this one person who'd just delete older versions when never ones became available. In one year I think we had three or four instances where our CI and unit tests just broke randomly one day, because the APIs had changed and the old version of the library had been yanked. In hindsight it actually helped us, because in frustrations we ended up setting up our own Python package repo and started to pay more attention to our dependencies.
- wang_li 9mo ago
  That's why you, very early on, release code that slows the API down once it has been deprecated. Every place you issue a deprecation warning, you also sleep 60. Problem solved.
  - skeledrew 9mo ago
    A solid idea. There should be a package for this.
superkuh 9mo ago
>We ended up adding the APIs back and creating a hurried release to fix the issue. So it was entirely possible to keep the software working with these. Why change/remove them in the first place? Is the benefit of of the new abstraction greater than the downside of requiring everyone using the software to re-write theirs?
- integralid 9mo ago
  OS software maintainers don't like maintaining legacy ugly APIs forever and want to refactor/remove legacy code to keep themselves sane and the project maintainable.
  - superkuh 9mo ago
    Yep. Or you can see it as, "This software doesn't really care about the users and their use cases. It prioritizes making things look pretty and easier on the dev side over maintaining functionality." Or in the worse but fairly common OSS case, CADT, but that doesn't seem to apply in this context.
  - LegionMammal978 9mo ago
    Every public API change is a cost on the user: for an extreme example, if every library I ever used renamed half its APIs every year to align with the latest ontology, then there would hardly be any point in saving my scripts, since I'd have to be constantly rewriting them all. Of course, the reality is hardly ever as bad as that, but I'd say having to deal with trivial API changes is a reasonable basis for a user to dislike a given project or try to avoid using it. It's up to the maintainers how friendly they want to be toward existing user code, and whether they pursue mitigating options like bundling migrations into less-common bigger updates.
  - mxey 9mo ago
    It’s like 5 lines of code https://github.com/urllib3/urllib3/pull/3732/files https://github.com/urllib3/urllib3/pull/3732/files
- memco 9mo ago
traverseda 9mo ago
Wait, does urlib not use semvar? Don't remove APIs on minor releases people. A major release doesn't have to be a problem or a major redesign, you can do major release 400 for all I care, just don't break things on minor releases. Lots of things not using semvar that I always just assumed did.
- Bratmon 9mo ago
  +1 The industry has a solution for the exact problem the urllib is having (semver). Urllib just actively refuses to use it.
- philipwhiuk 9mo ago
  Python itself doesn't use semver.
  - skeledrew 9mo ago
    Because everyone is afraid of a v4, after the 2-3 debacle. And there are things which need to be culled every once in a while to keep the stdlib fresh.
    - 7bit 9mo ago
      Python is culling stuff all the time, but that doesn't warrant a major version jump. You are probably right about Pythons careful approach of when to ship v4, but for the wrong reasons. Python 3 was necessary not for the removal of functions, but because of the syntax changes e.g., moving print from a statement to a method.
- deleted 9mo ago
  [deleted]
- CaliforniaKarl 9mo ago
  This, I think, is the crux of the matter. As an example, I always knew urllib3 as one of the foundational packages that Requests uses. And I was curious, what versions of urllib3 does Requests pull in? Well, according to https://github.com/psf/requests/blob/main/setup.cfg https://github.com/psf/requests/blob/main/setup.cfg, it's this: urllib3>=1.21.1,<3 That is exactly the kind of dependency specification I would expect to see for a package that is using semver: The current version of urllib3 is 2.x, so with semver, you set up your dependencies to avoid the next major-version number (in this case, 3). So, it seems to me that even the Requests folks assumed urllib3 was using semver.
hartator 9mo ago
I think they are misreading the situation. The devil is in the details. It seems `getHeaders` v. `headers` is non-security, non-performance related issue. Why people should spend time fixing these?
- CaliforniaKarl 9mo ago
  If you (either directly or from SerpApi) are supporting the urllib3 folks (through a Tidelift subscription), then yes, that is a valid point. Otherwise, I'd say that's a very brave comment you are making.
  - shadowgovt 9mo ago
    "Brave" in what sense? It's a legitimate question why an API is having a pattern deprecated and removed for what appears to be pattern reasons.
  - ipaddr 9mo ago
    If he gave money valid point if not he must be brave?
- attractivechaos 9mo ago
  Even if getHeaders() has security/performance concerns, the better solution is to make it an alias to the newer headers.get() in this case. Keeping the old API is a small hassle to a handful of developers but breaking existing code puts a much bigger burden on a lot more users.
  - nodesocket 9mo ago
    Ya, why not just alias old api calls to the new if implementation details changed?
Revisional_Sin 9mo ago
> Maybe the answer is to do away with advance notice and adopt SemVer with many major versions Yes.
jakub_g 9mo ago
Fixing deprecations is unfortunately the lowest prio of any kind of work for majority of the projects. Part of the problem is probably lack of pressure to do so it if the timeline is unclear. What if this is actually never removed? Why going through the pain? IMO telling "we deprecate now and let's see when we remove it" is counterproductive. A better way: deprecate now and tell "in 12 (or 24?) months this WILL be removed". After 12/24 months, cut a new semver-major release. People notice the semver-major through the dependency management tools at some point, an maybe they have a look at changelog. If they don't, at some point they may want to use a new feature, and finally be incentivised to update. If there's no incentive other than "do the right thing", it never gets done. Having said that, I think LLMs are really going to help with chores like this, if e.g. deprecations and migration steps are well documented. Alternative option: create a codemod CLI that fixes deprecations for the users, doing the right thing automatically. If migration is painless and quick, it's more likely people will do it.
- Hizonner 9mo ago
  > Fixing deprecations is unfortunately the lowest prio of any kind of work for majority of the projects. ... and the right answer to that is to make it entirely their problem. > Part of the problem is probably lack of pressure to do so it if the timeline is unclear. What if this is actually never removed? In this case, the warnings said exactly what release would remove the API. Didn't help. > Why going through the pain? Because you're not a feckless irresponsible idiot? I don't think it's an accident that the projects they said didn't react were an overcomplicated and ill-designed management layer for an overcomplicated and ill-designed container system, a move-fast-and-break-things techbro company, and what looks to be a consolation project for the not-too-bright. You probably get an extra measure of that if you're operating in the Python ecosystem, which is culturally all about half-assed, 80-percent-right-we-hope approaches. The right answer is to remove it when you say you're going to remove it, and let them pick up the pieces. It also helps if you design your API right to begin with, of course. But this is Python we're talking about again.
- CaliforniaKarl 9mo ago
  > After 12/24 months, cut a new semver-major release. People notice the semver-major through the dependency management tools at some point, an maybe they have a look at changelog. The urllib3 package doesn't use SemVer.
eddythompson80 9mo ago
Deprecations in all forms are always a shitshow. There isn’t a particular pattern that “just works”. Anybody that tells you about one, best case scenario, it just worked for them because of their consumer/user not because of the method itself. The best I have seen is a heavy handed in-editor strike through with warnings (assuming the code is actively being worked on) and even then it’s at best a 50/50 thing. 50% of the developers would feel that using an API with a strike through in the editor is wrong. And the other 50% will just say “I dunno, I copied it from there. What’s wrong with it??”
shadowgovt 9mo ago
The secret trick I've used on rare occasion, but when necessary, is the "ten second rule." Users don't notice a deprecation warning. But they might notice adding a "time.sleep(10)" immediately at the top of the function. And that gives them one last grace period to change out their software before it breaks-breaks.
- odie5533 9mo ago
  This will just waste CI compute and not solve anything.
  - shadowgovt 9mo ago
    It's worked in the past. But it does require someone at your org to care that CI times are spiking, which is not always a thing you can rely upon. In addition: if CI is the only place the issue shows up, and never in a user interaction... Why does that software exist in the first place? In that context, the slowdown may be serving as a useful signal to the project to drop the entire dependency. ETA: To be clear, I don't do this as a substitute for a regular deprecation cycle (clear documentation, clear language-supported warnings / annotations, clear timeline to deprecate); I do it in addition before the final yank that actually breaks end-users.
    - odie5533 9mo ago
      Most CI runs I see have more than a 10s variance.
  - pavel_lishin 9mo ago
    Are you saying you wouldn't notice if your CI suddenly started taking twice as long, ten times as long, a hundred times as long to run?
- minitech 9mo ago
  This is so much worse than just making the breaking change.
hiq 9mo ago
Wild (and I guess most of the time bad) idea: on top of the warnings, introduce a `sleep` in the deprecated functions. At every version, increase the sleep. Has this ever been considered? The problem with warnings is that they're not really observable: few people actually read these logs, most of the time. Making the deprecation observable means annoying the library users. The question is then: what's the smallest annoyance we can come up with, so that they still have a look?
- fpoling 9mo ago
  Yes, people do notice sleep. But it has to be on the scale of minutes or it will be ignored especially if it happens during a CI run.
Hackbraten 9mo ago
A deprecation warning is not actionable for typical end users. Why don't more warnings include calls to action? Instead of a warning that says, "The get_widget method in libfoo is deprecated and will be removed by November 30", the warning could say: "This app uses the deprecated get_widget method. Please report this bug to the app developer. Ask them to fix this issue so you can continue using this app after November 30."
jeffrallen 9mo ago
I prefer Go's solution to this problem. Just don't deprecate stuff. And to make that possible, slow down and design stuff you will be willing to support forever.
- mxey 9mo ago
  Go does deprecate stuff. It just never removes it.
- deleted 9mo ago
  [deleted]
plantain 9mo ago
# Deprecated APIs resp.getheader("Content-Length") # Recommended APIs resp.headers.get("Content-Length") Why inflict this change on tens of millions of users? It's such a nonsense tiny busywork change. Let sleeping dogs lie.
- gsnedders 9mo ago
  This is exactly the sort of breaking change that I really struggle to see the value of — maintaining the deprecated method seems incredibly unlikely to be a notable maintenance burden when it is literally just: @deprecated("Use response.headers.get(name) instead") def getheader(self, name): return self.headers.get(name) Like sure — deprecate it, which might have _some_ downstream cost, rather than having two non-deprecated ways to do the same thing, just to make it clear which one people should be using; but removing it has a much more significant cost on every downstream user, and the cost of maintenance of the old API seems like it should be almost nothing. (I also don't hate the thought of having a `DeprecationWarning` subclass like `IndefiniteDeprecationWarning` to make it clear that there's no plan to remove the deprecated function, which can thus be ignored/be non-fatal in CI etc.)
  - eglintondust 9mo ago
    There is value for the person maintaining this library cause they want it that way. If you develop a useful library and give it away for free then all power to you if you want to rearrange the furniture every 6 months. I'll roll with it.
    - mxey 9mo ago
      > If you develop a useful library and give it away for free then all power to you if you want to rearrange the furniture every 6 months. That would make it no longer a useful library
    - gsnedders 9mo ago
      You are, of course, correct. But the fact that they made a new release with it undeprecated shows they _do_ care about their users (direct and indirect), and at least from my point of view (both from the Python ecosystem and the browser ecosystem) this was a pretty foreseeable outcome.
akie 9mo ago
Start adding a 500ms delay to every invocation of a deprecated function. Incentives work!
- handoflixue 9mo ago
  Or even scale it up over time: start with 100ms of delay and crank it up the closer you get to deprecated
tgsovlerkhgsel 9mo ago
Deprecations don't work. Don't deprecate stuff without a really really good reason. The new API being cleaner is not a good reason. There are very few good reasons. If you deprecate something in a popular library, you're forcing millions of people to do work. Waste time that could be used for something better, possibly at a time of your choice, not theirs. It was emitting warnings for 3 years... so you think everyone should have to rewrite their software every 3 years? Especially for something like this. Only document it in a footnote, mark it as deprecated, etc - but don't remove the alias. Don't break stuff, unless, to quote a famous work, you think your users are scum. Do you think your users are scum? Why do you hate your users?
- jghn 9mo ago
  Open source developers have no such obligation. They’re doing things on their own time. But hey, I’m sure they’d be willing to give you a full refund for the amount you paid them to use their work
- deleted 9mo ago
  [deleted]
bigstrat2003 9mo ago
Honestly, I think that the author already found (and rejected the solution): > I could ask for more Python developers to run with warnings enabled, but solutions in the form of “if only we could all just” are a folly. I get where he's coming from. But the facts are: the language provides a tool to warn users, the library is using that tool, and users are choosing to turn that tool off. That's fine, but then they don't get to complain when stuff breaks without warning. It is the user's responsibility at that point, not the library maintainers'.
Incipient 9mo ago
I'm very confused on the debate of semver here - the fundamental principle seems very simple, and important. "give me all updates to my core version that's still compatible" Semver simply puts a 'protocol' to this - define your major version and off you go. While in practice you could go and search for each and every library you use to check when/how they do breaking versions, but semver just allows matching a single number across the board - it makes it more consistent and error proof.
- BugsJustFindMe 9mo ago
  Because it's not "error proof". It's not even "more error proof". Relying on updates not breaking if the maintainer doesn’t intend it means relying on software being bug-free, and it should be immediately obvious that that would be a very very stupid thing to believe. Letting software you rely on change on its own is always dangerous no matter how much someone pinky promises that you can trust them to have thought of everything. Semver numbers are determined by humans who aren't oracles and therefore make mistakes all the time about what changes will break something for someone else. It may get people to think a little bit about whether they _think_ they're breaking something, but it's never safe to rely on that judgement because there's a good chance that they're wrong. And given that it's never safe to rely on that judgment, the process of pretending is worse than worthless; the pretense is inherently dangerous, because people start to unthinkingly trust it. If you care about things working, you pin and vendor (npm left-pad) your dependencies and only update a package if you _need_ to and only after explicit evaluation. And if after reading that you think to yourself, "But BugsJustFindMe, I have unit tests that will catch semver mistakes!" I think you need to ask yourself what your unit tests tell you about semver code that they don't also tell you about non-semver code.
  - Incipient 9mo ago
    I would absolutely agree with your approach for large-scale saas/highly publicised applications, especially when you have a team/people dedicated to security/versioning. In small teams where you're focused on building features, being able to rely on semver for non-breaking versions has been very helpful/successful. I personally haven't encountered a breaking feature added to something that shouldn't be. This also applies to our docker containers, using python 3.13-slim-trixie for example, and letting the patch version sort itself out.
skeledrew 9mo ago
Instead of reverting the updates to appease those who've been ignoring the warnings, a way forward could be to gate the old API behind an environment variable, with a name like "TEMPORARILY_REENABLE_DEPRECATED_APIS". And maybe put an expiry on the variable so it resets occasionally. Otherwise the project just becomes mired and stagnant.

63 points by scolby33 9mo ago

Deprecations via warnings don't reliably work anywhere, in general. If you are a good developer, you'll have extensive unit test coverage and CI. You never see the unit test output (unless they fail) - so warnings go unnoticed. If you are a bad developer, you have no idea what you are doing and you ignore all warnings unless program crashes.

SethMLarson 9mo ago
Author here! Agreed that are different levels of "engaged" from users, which is okay. The concerning part of this finding is that even dependent users that I know to be highly engaged didn't respond to the deprecation warnings, so they're not working for even the most engaged users.
eternityforest 9mo ago
Why is it that CI tools don't make warnings visible? Why are they ignored by default in the first place? Seems like that should be a rather high priority.
- metadat 9mo ago
  Most of the time they don't matter and aren't an immediate problem. The Business doesn't care about warnings, they want working software NOW.
- bluGill 9mo ago
  It isn't that easy. If you have a new warning on upgrade you probably want to work on it "next week", but that means you need to ignore it for a bit. Or you might still want to support a really old version without the new API and so you can't fix it now.
  - Hizonner 9mo ago
    > If you have a new warning on upgrade you probably want to work on it "next week", but that means you need to ignore it for a bit. So you create a bug report or an issue or a story or whatever you happen to call it, and you make sure it gets tracked, and you schedule it with the rest of your work. That's not the same thing as "ignoring" it.
    - bluGill 9mo ago
      And you always have something more important/interesting to do and so never get around to it.
      - Hizonner 9mo ago
        ... which means that when the axe falls, the results are 100 percent your fault.
- michaelt 9mo ago
  > Why is it that CI tools don't make warnings visible? A developer setting up CI decides to start an ubuntu 24.04 container and run 'apt-get install npm' This produces 3,600 lines of logging (5.4 log lines per package, 668 packages) and 22 warnings (all warnings about man page creation being skipped) Then they decide "Nobody's going to read all that, and the large volume might bury important information. I think I'll hide console output for processes that don't fail." Now your CI doesn't show warnings.
optionalsquid 9mo ago
You can turn warnings into errors with the `-Werror` option. I personally use that in CI runs, along with the `-X dev` option to enable additional runtime checks. Though that wont solve the author's problem, since most Python devs don't use either of those options
- Y-bar 9mo ago
  In PHP I don’t think there is a native way to convert E_DEPRECATED into E_ERROR, but the most common testing framework has a quick way of doing the same. https://docs.phpunit.de/en/12.5/configuration.html#the-failondeprecation-attribute https://docs.phpunit.de/en/12.5/configuration.html#the-failo...
rimunroe 9mo ago
> If you are a good developer, you'll have extensive unit test coverage and CI. You never see the unit test output (unless they fail) - so warnings go unnoticed. In my opinion test suites should treat any output other than the reporter saying that a test passed as a test failure. In JavaScript I usually have part of my test harness record calls to the various console methods. At the end of each test it checks to see if any calls to those methods were made, and if they were it fails the tests and logs the output. Within tests if I expect or want some code to produce a message, I wrap the invocation of that code in a helper which requires two arguments: a function to call and an expected output. If the code doesn't output a matching message, doesn't output anything, or outputs something else then the helper throws and explains what went wrong. Otherwise it just returns the result of the called function: let result = silenceWarning(() => user.getV1ProfileId(), /getV1ProfileId has been deprecated/); expect(result).toBe('foo'); This is dead simple code in most testing frameworks. It makes maintaining and working with the test suite becomes much easier as when something starts behaving differently it's immediately obvious rather than being hidden in a sea of noise. It makes working with dependencies easier because it forces you to acknowledge things like deprecation warnings when they get introduced and either solve them there or create an upgrade plan.
ploxiln 9mo ago
When I update python version, python packages, container image, etc for a service, I take a quick look at CI output, in addition to the all the other checks I do (like a couple basic real-world-usage end-to-end usage tests), to "smoke test" whether something not caught by outright CI failure caused some subtle problem. So, I do often see deprecation warnings in CI output, and fix them. Am I a bad developer? I think the mistake here is making some warnings default-hidden. The developer who cares about the user running their the app in a terminal can add a line of code to suppress them for users, and be more aware of this whole topic as a result (and have it more evident near the entrypoint of the program, for later devs to see also). I think that making warnings error or hidden removes warnings as a useful tool. But this is an old argument: Who should see Python warnings? (2017) https://lwn.net/Articles/740804/ https://lwn.net/Articles/740804/
Hizonner 9mo ago
If you are a good developer, you consider warnings to be errors until proven otherwise.
- hiq 9mo ago
  What does a good developer do when working in a codebase with hundreds of warnings? Or are you only considering a certain warnings?
  - Hizonner 9mo ago
    Why does your codebase generate hundreds of warnings, given that every time one initially appeared, you should have stamped it out (or specifically marked that one warning to be ignored)? Start with one line of code that doesn't generate a warning. Add a second line of code that doesn't generate a warning...
    - stackskipton 9mo ago
      Because most people are working at Failure/Feature factories where they might work on something and at last minute, they find out something is now warning. If they work on fixing it, the PM will screaming about time slippage and be like "I want you to work on X, not Y which can wait". 2 Years later, you have hundreds of warning.
      - Hizonner 9mo ago
        You found that out at the last minute. So then you did a release. It's no longer the last minute. Now what's your excuse for the next release? If your management won't resource your project to the point where you can assure that the software is correct, you might want to see if you can find the free time to look for another job. You'll have to do that anyway when they either tank the company, or lay you off next time they feel they need to cut more costs.
    - hiq 9mo ago
      It's rare that I work on a project I myself started. If I start working on an existing codebase, the warnings might be there already. Then what do I do? I'm also referring to all the warnings you might get if you use an existing library. If the requirements entail that I use this library, should I just silence them all? But I'm guessing you might be talking about more specific warnings. Yes I do fix lints specific to my new code before I commit it, but a lot of warnings might still be logged at runtime, and I may have no control over them.
mrweasel 9mo ago
There was this one library we depended on, it was sort of in limbo during the Python 2 -> 3 migration. During that period is was maintained by this one person who'd just delete older versions when never ones became available. In one year I think we had three or four instances where our CI and unit tests just broke randomly one day, because the APIs had changed and the old version of the library had been yanked. In hindsight it actually helped us, because in frustrations we ended up setting up our own Python package repo and started to pay more attention to our dependencies.
wang_li 9mo ago
That's why you, very early on, release code that slows the API down once it has been deprecated. Every place you issue a deprecation warning, you also sleep 60. Problem solved.
- skeledrew 9mo ago
  A solid idea. There should be a package for this.

superkuh 9mo ago

>We ended up adding the APIs back and creating a hurried release to fix the issue. So it was entirely possible to keep the software working with these. Why change/remove them in the first place? Is the benefit of of the new abstraction greater than the downside of requiring everyone using the software to re-write theirs?

integralid 9mo ago
OS software maintainers don't like maintaining legacy ugly APIs forever and want to refactor/remove legacy code to keep themselves sane and the project maintainable.
- superkuh 9mo ago
  Yep. Or you can see it as, "This software doesn't really care about the users and their use cases. It prioritizes making things look pretty and easier on the dev side over maintaining functionality." Or in the worse but fairly common OSS case, CADT, but that doesn't seem to apply in this context.
- LegionMammal978 9mo ago
  Every public API change is a cost on the user: for an extreme example, if every library I ever used renamed half its APIs every year to align with the latest ontology, then there would hardly be any point in saving my scripts, since I'd have to be constantly rewriting them all. Of course, the reality is hardly ever as bad as that, but I'd say having to deal with trivial API changes is a reasonable basis for a user to dislike a given project or try to avoid using it. It's up to the maintainers how friendly they want to be toward existing user code, and whether they pursue mitigating options like bundling migrations into less-common bigger updates.
- mxey 9mo ago
  It’s like 5 lines of code https://github.com/urllib3/urllib3/pull/3732/files https://github.com/urllib3/urllib3/pull/3732/files
memco 9mo ago

traverseda 9mo ago

Wait, does urlib not use semvar? Don't remove APIs on minor releases people. A major release doesn't have to be a problem or a major redesign, you can do major release 400 for all I care, just don't break things on minor releases. Lots of things not using semvar that I always just assumed did.

Bratmon 9mo ago
+1 The industry has a solution for the exact problem the urllib is having (semver). Urllib just actively refuses to use it.
philipwhiuk 9mo ago
Python itself doesn't use semver.
- skeledrew 9mo ago
  Because everyone is afraid of a v4, after the 2-3 debacle. And there are things which need to be culled every once in a while to keep the stdlib fresh.
  - 7bit 9mo ago
    Python is culling stuff all the time, but that doesn't warrant a major version jump. You are probably right about Pythons careful approach of when to ship v4, but for the wrong reasons. Python 3 was necessary not for the removal of functions, but because of the syntax changes e.g., moving print from a statement to a method.
deleted 9mo ago
[deleted]
CaliforniaKarl 9mo ago
This, I think, is the crux of the matter. As an example, I always knew urllib3 as one of the foundational packages that Requests uses. And I was curious, what versions of urllib3 does Requests pull in? Well, according to https://github.com/psf/requests/blob/main/setup.cfg https://github.com/psf/requests/blob/main/setup.cfg, it's this: urllib3>=1.21.1,<3 That is exactly the kind of dependency specification I would expect to see for a package that is using semver: The current version of urllib3 is 2.x, so with semver, you set up your dependencies to avoid the next major-version number (in this case, 3). So, it seems to me that even the Requests folks assumed urllib3 was using semver.

hartator 9mo ago

I think they are misreading the situation. The devil is in the details. It seems `getHeaders` v. `headers` is non-security, non-performance related issue. Why people should spend time fixing these?

CaliforniaKarl 9mo ago
If you (either directly or from SerpApi) are supporting the urllib3 folks (through a Tidelift subscription), then yes, that is a valid point. Otherwise, I'd say that's a very brave comment you are making.
- shadowgovt 9mo ago
  "Brave" in what sense? It's a legitimate question why an API is having a pattern deprecated and removed for what appears to be pattern reasons.
- ipaddr 9mo ago
  If he gave money valid point if not he must be brave?
attractivechaos 9mo ago
Even if getHeaders() has security/performance concerns, the better solution is to make it an alias to the newer headers.get() in this case. Keeping the old API is a small hassle to a handful of developers but breaking existing code puts a much bigger burden on a lot more users.
- nodesocket 9mo ago
  Ya, why not just alias old api calls to the new if implementation details changed?

Revisional_Sin 9mo ago

> Maybe the answer is to do away with advance notice and adopt SemVer with many major versions Yes.

jakub_g 9mo ago

Fixing deprecations is unfortunately the lowest prio of any kind of work for majority of the projects. Part of the problem is probably lack of pressure to do so it if the timeline is unclear. What if this is actually never removed? Why going through the pain? IMO telling "we deprecate now and let's see when we remove it" is counterproductive. A better way: deprecate now and tell "in 12 (or 24?) months this WILL be removed". After 12/24 months, cut a new semver-major release. People notice the semver-major through the dependency management tools at some point, an maybe they have a look at changelog. If they don't, at some point they may want to use a new feature, and finally be incentivised to update. If there's no incentive other than "do the right thing", it never gets done. Having said that, I think LLMs are really going to help with chores like this, if e.g. deprecations and migration steps are well documented. Alternative option: create a codemod CLI that fixes deprecations for the users, doing the right thing automatically. If migration is painless and quick, it's more likely people will do it.

Hizonner 9mo ago
> Fixing deprecations is unfortunately the lowest prio of any kind of work for majority of the projects. ... and the right answer to that is to make it entirely their problem. > Part of the problem is probably lack of pressure to do so it if the timeline is unclear. What if this is actually never removed? In this case, the warnings said exactly what release would remove the API. Didn't help. > Why going through the pain? Because you're not a feckless irresponsible idiot? I don't think it's an accident that the projects they said didn't react were an overcomplicated and ill-designed management layer for an overcomplicated and ill-designed container system, a move-fast-and-break-things techbro company, and what looks to be a consolation project for the not-too-bright. You probably get an extra measure of that if you're operating in the Python ecosystem, which is culturally all about half-assed, 80-percent-right-we-hope approaches. The right answer is to remove it when you say you're going to remove it, and let them pick up the pieces. It also helps if you design your API right to begin with, of course. But this is Python we're talking about again.
CaliforniaKarl 9mo ago
> After 12/24 months, cut a new semver-major release. People notice the semver-major through the dependency management tools at some point, an maybe they have a look at changelog. The urllib3 package doesn't use SemVer.

eddythompson80 9mo ago

Deprecations in all forms are always a shitshow. There isn’t a particular pattern that “just works”. Anybody that tells you about one, best case scenario, it just worked for them because of their consumer/user not because of the method itself. The best I have seen is a heavy handed in-editor strike through with warnings (assuming the code is actively being worked on) and even then it’s at best a 50/50 thing. 50% of the developers would feel that using an API with a strike through in the editor is wrong. And the other 50% will just say “I dunno, I copied it from there. What’s wrong with it??”

shadowgovt 9mo ago

The secret trick I've used on rare occasion, but when necessary, is the "ten second rule." Users don't notice a deprecation warning. But they might notice adding a "time.sleep(10)" immediately at the top of the function. And that gives them one last grace period to change out their software before it breaks-breaks.

odie5533 9mo ago
This will just waste CI compute and not solve anything.
- shadowgovt 9mo ago
  It's worked in the past. But it does require someone at your org to care that CI times are spiking, which is not always a thing you can rely upon. In addition: if CI is the only place the issue shows up, and never in a user interaction... Why does that software exist in the first place? In that context, the slowdown may be serving as a useful signal to the project to drop the entire dependency. ETA: To be clear, I don't do this as a substitute for a regular deprecation cycle (clear documentation, clear language-supported warnings / annotations, clear timeline to deprecate); I do it in addition before the final yank that actually breaks end-users.
  - odie5533 9mo ago
    Most CI runs I see have more than a 10s variance.
- pavel_lishin 9mo ago
  Are you saying you wouldn't notice if your CI suddenly started taking twice as long, ten times as long, a hundred times as long to run?
minitech 9mo ago
This is so much worse than just making the breaking change.

hiq 9mo ago

Wild (and I guess most of the time bad) idea: on top of the warnings, introduce a `sleep` in the deprecated functions. At every version, increase the sleep. Has this ever been considered? The problem with warnings is that they're not really observable: few people actually read these logs, most of the time. Making the deprecation observable means annoying the library users. The question is then: what's the smallest annoyance we can come up with, so that they still have a look?

fpoling 9mo ago
Yes, people do notice sleep. But it has to be on the scale of minutes or it will be ignored especially if it happens during a CI run.

Hackbraten 9mo ago

A deprecation warning is not actionable for typical end users. Why don't more warnings include calls to action? Instead of a warning that says, "The get_widget method in libfoo is deprecated and will be removed by November 30", the warning could say: "This app uses the deprecated get_widget method. Please report this bug to the app developer. Ask them to fix this issue so you can continue using this app after November 30."

jeffrallen 9mo ago

I prefer Go's solution to this problem. Just don't deprecate stuff. And to make that possible, slow down and design stuff you will be willing to support forever.

mxey 9mo ago
Go does deprecate stuff. It just never removes it.
deleted 9mo ago
[deleted]

plantain 9mo ago

# Deprecated APIs resp.getheader("Content-Length") # Recommended APIs resp.headers.get("Content-Length") Why inflict this change on tens of millions of users? It's such a nonsense tiny busywork change. Let sleeping dogs lie.

gsnedders 9mo ago
This is exactly the sort of breaking change that I really struggle to see the value of — maintaining the deprecated method seems incredibly unlikely to be a notable maintenance burden when it is literally just: @deprecated("Use response.headers.get(name) instead") def getheader(self, name): return self.headers.get(name) Like sure — deprecate it, which might have _some_ downstream cost, rather than having two non-deprecated ways to do the same thing, just to make it clear which one people should be using; but removing it has a much more significant cost on every downstream user, and the cost of maintenance of the old API seems like it should be almost nothing. (I also don't hate the thought of having a `DeprecationWarning` subclass like `IndefiniteDeprecationWarning` to make it clear that there's no plan to remove the deprecated function, which can thus be ignored/be non-fatal in CI etc.)
- eglintondust 9mo ago
  There is value for the person maintaining this library cause they want it that way. If you develop a useful library and give it away for free then all power to you if you want to rearrange the furniture every 6 months. I'll roll with it.
  - mxey 9mo ago
    > If you develop a useful library and give it away for free then all power to you if you want to rearrange the furniture every 6 months. That would make it no longer a useful library
  - gsnedders 9mo ago
    You are, of course, correct. But the fact that they made a new release with it undeprecated shows they _do_ care about their users (direct and indirect), and at least from my point of view (both from the Python ecosystem and the browser ecosystem) this was a pretty foreseeable outcome.

akie 9mo ago

Start adding a 500ms delay to every invocation of a deprecated function. Incentives work!

handoflixue 9mo ago
Or even scale it up over time: start with 100ms of delay and crank it up the closer you get to deprecated

tgsovlerkhgsel 9mo ago

Deprecations don't work. Don't deprecate stuff without a really really good reason. The new API being cleaner is not a good reason. There are very few good reasons. If you deprecate something in a popular library, you're forcing millions of people to do work. Waste time that could be used for something better, possibly at a time of your choice, not theirs. It was emitting warnings for 3 years... so you think everyone should have to rewrite their software every 3 years? Especially for something like this. Only document it in a footnote, mark it as deprecated, etc - but don't remove the alias. Don't break stuff, unless, to quote a famous work, you think your users are scum. Do you think your users are scum? Why do you hate your users?

jghn 9mo ago
Open source developers have no such obligation. They’re doing things on their own time. But hey, I’m sure they’d be willing to give you a full refund for the amount you paid them to use their work
deleted 9mo ago
[deleted]

bigstrat2003 9mo ago

Honestly, I think that the author already found (and rejected the solution): > I could ask for more Python developers to run with warnings enabled, but solutions in the form of “if only we could all just” are a folly. I get where he's coming from. But the facts are: the language provides a tool to warn users, the library is using that tool, and users are choosing to turn that tool off. That's fine, but then they don't get to complain when stuff breaks without warning. It is the user's responsibility at that point, not the library maintainers'.

Incipient 9mo ago

I'm very confused on the debate of semver here - the fundamental principle seems very simple, and important. "give me all updates to my core version that's still compatible" Semver simply puts a 'protocol' to this - define your major version and off you go. While in practice you could go and search for each and every library you use to check when/how they do breaking versions, but semver just allows matching a single number across the board - it makes it more consistent and error proof.

BugsJustFindMe 9mo ago
Because it's not "error proof". It's not even "more error proof". Relying on updates not breaking if the maintainer doesn’t intend it means relying on software being bug-free, and it should be immediately obvious that that would be a very very stupid thing to believe. Letting software you rely on change on its own is always dangerous no matter how much someone pinky promises that you can trust them to have thought of everything. Semver numbers are determined by humans who aren't oracles and therefore make mistakes all the time about what changes will break something for someone else. It may get people to think a little bit about whether they _think_ they're breaking something, but it's never safe to rely on that judgement because there's a good chance that they're wrong. And given that it's never safe to rely on that judgment, the process of pretending is worse than worthless; the pretense is inherently dangerous, because people start to unthinkingly trust it. If you care about things working, you pin and vendor (npm left-pad) your dependencies and only update a package if you _need_ to and only after explicit evaluation. And if after reading that you think to yourself, "But BugsJustFindMe, I have unit tests that will catch semver mistakes!" I think you need to ask yourself what your unit tests tell you about semver code that they don't also tell you about non-semver code.
- Incipient 9mo ago
  I would absolutely agree with your approach for large-scale saas/highly publicised applications, especially when you have a team/people dedicated to security/versioning. In small teams where you're focused on building features, being able to rely on semver for non-breaking versions has been very helpful/successful. I personally haven't encountered a breaking feature added to something that shouldn't be. This also applies to our docker containers, using python 3.13-slim-trixie for example, and letting the patch version sort itself out.

skeledrew 9mo ago

Instead of reverting the updates to appease those who've been ignoring the warnings, a way forward could be to gate the old API behind an environment variable, with a name like "TEMPORARILY_REENABLE_DEPRECATED_APIS". And maybe put an expiry on the variable so it resets occasionally. Otherwise the project just becomes mired and stagnant.