12 ms·

Rye and Uv: August Is Harvest Season for Python Packaging

254 points by keybits 2y ago

keybits 2y ago
Uv's latest release was discussed yesterday: https://news.ycombinator.com/item?id=41302475 https://news.ycombinator.com/item?id=41302475 The linked post is the author of Rye's take on that.
bananapub 2y ago
I really like this framing - lots of incremental work by lots of people over time got us to the point where ~a few people at one company can radically improve the situation with a medium amount of work.
deleted 2y ago
[deleted]
dirtbag__dad 2y ago
TY to the astral team for making my quality of life so much better and Armin for being brave enough to pass the torch. Strong +1 on a one tool wins approach - I am so tired of burning time on local dev setup, everything from managing a monorepo (many packages that import into each other) to virtual environments and PYTHONPATH (I’ve been at it for like 8 years now and I still can’t grok how to “just avoid” issues like those across all pkg managers, woof!) I am really excited to see what’s next. Especially looking for a mypy replacement and perhaps something that gives compiling python a “native” feeling thing we can do
alanfranz 2y ago
There’s one problem left with those tools: authority. They’re not pypa endorsed, that’s what makes those different from cargo. At the same time pypa wasn’t able to provide a comprehensive solution over the years, and python packaging and development tools multiplied - just 3-4 years ago poetry and pipenv seemed to solve python packaging problems in a way that pip+virtualenv couldn’t. We need pypa to now jump on the astral.sh ship - but will they do that without a certain amount of control?
- the_mitsuhiko 2y ago
  As a somewhat outside observer I don't quite understand what the PyPA actually is. In some sense there is an unclear number of participants and it's not even sure to which degree the PyPA relates to core python or the PSF. I think the real endorsement that could help would be the core Python project itself. In a perfect world the official Python tutorial would start with "here is how you install python" and it starts by installing uv, the same way as the official Rust docs point to rustup and cargo. I hope strongly that the PSF will manage to establish some sort of relationship with Astral which would enable that to eventually be a reality.
  - crabbone 2y ago
    Here are few things that... well, won't really work well: * Python package installation, package format and loading of modules are defective. The design is bad. It means that no implementation blessed by PSF or not isn't going to solve the packaging problem. So, there's no point to ask PSF or PyPA to adopt any external tool. If the external tools are better than pip in some way, then it could be the speed or memory footprint etc. They will not solve the conceptual problems, because they don't have the authority to do that. * PyPA and PSF, but maybe to a lesser extent, are populated today by delusional mediocre coders who have no idea where the ship is going or how to stir it. They completely lack vision and understanding of the problems they are to deal with. They add "features", but they don't know if those features are needed, and, in most cases it's just noise and bloat. From a perspective of someone who has to deal with fruits of their labor, they just ensure that my job of "someone who fixes Python packaging issues" will never go away. So... as "someone who fixes Python packaging issues" I kind of welcome the new level of hell coming from Astral. From where I stand, it is pouring more gas into a big dumpster fire. Just one more tool written in a non-mainstream, non-standardized, quickly evolving language, impossible to debug without a ton of instrumentation, with the source code hard to write and hard to understand. It's just another deposit towards my job security.
    - simonw 2y ago
      "Python package installation, package format and loading of modules are defective" How are they defective?
fvdessen 2y ago
I was looking this morning at migrating our software from poetry to uv at my company, due to poetry's slowness. And so far i've been reading a lot of doc and not getting a lot things done. I did the previous migration to poetry as well which was vastly simpler. So far it seems that poetry tried to make a simple package manager that works like any other, while uv is keeping quite a bit of the python package insanity around.
- deleted 2y ago
  [deleted]
- burntsushi 2y ago
  Are there any specifics you can share about what you found difficult to do in uv?
- bvrmn 2y ago
  At least uv doesn't lead to an absolute clusterfuck of poetry conflicting with virtualenv. Or broken package.toml format with minor poetry changes. Or amazingly dumb "sources" which doesn't work for transitive dependencies and leads to even longer resolve times in case of multiple indexes.
  - seabrookmx 2y ago
    How does it conflict with virtualenv? Last I checked it's default behavior was to automagically create a virtualenv for you if you run it from system python (a bit weird but ok) or if you were in an virtualenv, it would use it. For our projects we use pyenv-virtualenv (so we can have specific python versions per project) and then poetry "just works" (though can be slow, hence rye, uv and friends).
- orbisvicis 2y ago
  To which insanities are you referring?
  - bschwindHN 2y ago
    Probably the entirety of the current python package management ecosystem. From the outside, it all looks insane but I suppose most users are Stockholm-syndrome-d into thinking everything is fine.
PaulHoule 2y ago
My take is that this one of the most interesting topics there is right now but a below average article about that topic.
- simonw 2y ago
  Are you saying that because this was more of an opinion piece? As the author of Rye, a package that was heavily influential on the development of uv, Armin is very well positioned to share his opinions about this.
  - deleted 2y ago
    [deleted]
lmeyerov 2y ago
After the whole npm VC rugpull + Microsoft acquisition, and OpenAI showing legal non-profit status is toothless marketing to VC-path-entangled leaders, I'm reluctant to cede critical path language infra to these kinds of organizations. Individual contributors to these are individually great (and often exceptional!), but financial alignment at the organizational level is corrupted out of the gate. Fast forward 1-4 years, and the organization is what matters. "Die a hero or live long enough to become the villain." So fast lint, type checking, code scans, PR assistants, yes, we can swap these whenever. But install flow & package repo, no. That is unfortunate given the state of pip and conda... But here we are.
- crabbone 2y ago
  That battle has already been lost for Python. Microsoft owns Python, they just don't make it public. This is how I came to believe this is the case: Few years ago I wanted to write Python bindings to kubectl. I discovered that in order for that to work cross-platform, I need to make CGo use the same compiler on all platforms as does Python. Unfortunately, on MS Windows, CGO uses MINGW while Python uses MSVC. I wrote to Python dev. mailing list (which still existed at that time) and asked why did they choose to use a proprietary compiler for their "open-source" project. The answer I received in a round-about way was that MSVC was a historical choice, which cannot be presently changed because MS provides Python Foundation with free infrastructure to run CI and builds, and it also provides developers to work on Python (i.e. MS employees get paid by MS to work on Python interpreter). And that they are under orders not to drop MS tools from the toolchain. Year after year the situation was getting worse. Like in a lot of similar projects, success created a lot of ground for mediocre nobodies to reach positions of power. Python foundation and satellite projects like PyPA started to be populated by people whose way into these positions was not through contributing any useful code, but rather writing pages of code of conduct. This code of conduct and never-ending skirmishes around controlling positions eventually led to some old-timers leaving or being outright kicked out (latest such event was the ban of Tim, the guy who, beside other things, wrote Tim sort, which is a somewhat famous feature of Python). Year after year MS was pushing its usual agenda they do in every project they get their hands on: add crapload of useless features for the sake of advertising. Make the project swing every way possible, but mostly follow the fashion trends as hard as possible. This is how Python is now devoted to adding as much of ML-style types as possible (in the language with a completely different type system...), AoT compilation and JIT (in the language that's half of the time used to dynamically glue native libraries...) and so on. Essentially, making it a C#, but without curly braces. MS is smart enough to understand that publicly announcing their ownership of Python will scare a lot of people away from the technology, so they don't advertise it much. But they keep working on ensuring developers' dependency on their tooling, and eventually they will come to collect on their investment.
ris 2y ago
I wouldn't blame people if they sat this round out and waited for 2026's iteration of "Python Package Managers: we've really solved it this time!" (still a happy Nix user)
- nurettin 2y ago
  Should have ended with "I use Nix, btw"
  - intelVISA 2y ago
    It's important to remind people that all this is solved, by Nix, btw.
    - maleldil 2y ago
      For Python, specifically? As in, pyproject.toml-compatible tooling, with virtual environments and the like? I've see Nix being used for compiled languages, and maybe I'm missing something, but I haven't really seen it used to manage real-world projects. How do I manage a modern Python project with nix? How do I publish once to PyPI?
      - createaccount99 2y ago
        Agree. Don't think gp has any idea what they're saying. Using nix to install python dependencies is a disaster, it's only good for the python versions themselves.
        maleldil 2y ago
        > it's only good for the python versions themselves Versioning Python isn't hard. pyenv, asdf, mise, now uv... I honestly don't see what Nix brings to the Python ecosystem. I can see using it to version Python if you already use it, but that's it.
        2y ago
zokier 2y ago
I know python packaging has it's issues, but so far I personally have gotten pretty far with plain pip. The biggest shift to me was switching from original virtualenv to built-in venv module. On the other hand, if I wanted to be really serious about dependency management, I'd steal a page from FAANG, and build a monorepo and avoid all this hassle with package managers.
- drawnwren 2y ago
  I would really encourage you to try your hand at at a monorepo. I manage a python monorepo in prod and dependency management is hell. Poetry has some newer features that I am looking at trying to implement, but the state of the ecosystem wrt big monorepos is horrible.
  - akdor1154 2y ago
    I'm in a similar boat - uv's workspaces look tantalising.
  - crabbone 2y ago
    [flagged]
    - miohtama 2y ago
      Generally all Python software use third party packages which are fetched from PyPi package site. These packages are not stored in monorepo.
      - crabbone 2y ago
        This is maybe common, but this contradicts the definition of monorepo. You just use the word incorrectly. "Mono" means "one". If you pull packages from elsewhere, that stops being "mono". There's really no difference in this situation between your team publishing multiple packages from multiple repositories and then assembling them together for the purpose of deployment, or doing so, but with the third-party packages.
orbisvicis 2y ago
With declarative tooling - unlike setup.py - we've lost the ability to install to locations not managed by Python. Most of my projects used to have a python-code config file that I could include both in my code and in setup.py. Last I checked none of that is now possible. Want to install a system binary - ship an RPM or a flatpak. I don't understand how we could lose so much flexibility and yet gain so little in return. P.S I've only ever encountered minor dependency issues in my admittedly small projects using just pip and venv.
- OJFord 2y ago
  That's entirely appropriate imo - system packager should be running pip or whatever and then putting the output where it wants, it's not for setuptools or whatever to declare that. What path would you put in setup.py anyway? A different one for different distro preferences, a different one again for Windows, for macOS?
- heavyset_go 2y ago
  You can still use the setuptools backend when using declarative tooling, I use it in projects I can't fully port to pyproject.toml.
bkor 2y ago
For those interested in uv (instead of pip), uv massively sped up the release process for Home Assistant. The time needed to make a release went down from ~2.5 hours to ~20 minutes. See https://developers.home-assistant.io/blog/2024/04/03/build-images-with-uv/ https://developers.home-assistant.io/blog/2024/04/03/build-i... for details. I'm just a HA user btw.
- forrestthewoods 2y ago
  How the hell was pip taking over an hour to produce an image for a non-compiler language? I’m only a casual Python user. But wtf was it doing and why did it take so long? That’s bonkers.
  - cqqxo4zV46cp 2y ago
    Python is slow compared to Rust, obviously. Beyond that, pip is at this point carrying a bunch of legacy decisions because the ludicrously large number of hard left turns the Python packaging ecosystem has taken over the last 20 years. Home Assistant is an absolute behemoth of a project, especially with regard to dependencies. Dependency resolution across a project of that size is nuts. There are probably few currently projects that’d see as big an improvement aa HA.
    - forrestthewoods 2y ago
      Any specifics on what those legacy decisions are? I'm just curious. Have no idea. I'm a very casual Python user.
      - mkesper 2y ago
        If no hard versions are given, but only e.g. <= 2.1, pip will download EVERY SINGLE VERSION until 2.1 to look for metadata. That easily can take hours if it happens multiple times.
        bazzargh 2y ago
the__alchemist 2y ago
The churn is interesting. In 2019, I made a python version-manager and dependency manager written in rust. I gave up after it seemed like no one wanted to use it. Everyone not satisfied with Pip was on Poetry or Pipenv; I made the one I did because they both had user-interface problems; of the sort I would run into immediately. (I believe Poetry would default to Python 2, and not give you a choice by default, or something to that effect). Now there is a new batch. The biggest challenge was dealing with older pacakges that used non-standard packaging and ran arbitrary code; generally ones that didn't have wheels. From the article: > As of the most recent release, uv also gained a lot of functionality that previously required Rye such as manipulating pyproject.toml files, workspace support, local package references and script installation. It now also can manage Python installations for you so it's getting much closer. These are all things that dead project I wrote could do.
- toyg 2y ago
  The harsh reality is that this sort of tooling requires a lot of publicity to be successful. Nobody is going to try an unknown package manager from some random developer, but people will enthusiastically adopt anything pushed by a "famous" developer like Ronacher.
- vaylian 2y ago
  Not all good things gain traction. It definitely helped that Armin is well-known in the Python community. Where did you advertise your new tool?
  - the__alchemist 2y ago
    I think I posted on reddit, discord, and here. Didn't really do much advertising, which is obviously a room elephant!
remram 2y ago
So is Rye deprecated? The puzzling rye/uv relationship is what had stopped me from trying out with tool so far.
- mixmastamyk 2y ago
  It was a prototype. Lessons/code folded into uv.
- jsmeaton 2y ago
  Rye was started by Armin as a collection of existing tools (mostly) with a frontend similar to cargo. Then Astral came out with uv which aims to be a frontend into a collection of their own tools similar to cargo. Armin and Astral agreed for Astral to take over Rye some time during uv development with (I assume) the goal for uv to fully replace Rye. Use uv. As of 0.3.0 it covers most of rye now anyway. Especially if you’re writing projects and not consumable libs/apps (I haven’t used uv for anything other than package management so far).
billdueber 2y ago
I'm kind of interested in this space -- can anyone point me at an article that goes over why this is harder for python than it seems to be for, e.g., ruby? Is there something inherent about the way code is imported in python that makes it less tractable? Or is it just that the python world has never quite all come together on something that works well enough? (Note that I can certainly complain about how `bundler` works in ruby, but these discussions in python-land seem to go way beyond my quibbles with the ruby ecosystem)
- mixmastamyk 2y ago
  It's not. Just that Guido never cared about packaging, so it was left to a ragtag unpaid motley crew to piece together and later learn from industry practices that solidified a decade or two after they started.
- 0cf8612b2e1e 2y ago
  The big problem is that Python is used in a lot of different contexts and has too many valid ways in which people are able to build, compile, release non-Pythonic extensions in their package. Original way for packaging code was effectively an executable script which made some things easy, but other things much harder. Modern efforts are trying to limit the flexibility of the package definitions without breaking all of this legacy code. I unfairly think of Ruby as only for Ruby on Rails, so only ever dealing with the web domain. Here is a story about the nightmare it takes to compile the Fortran code that supports much of SciPy (backbone numerical computing library with algorithms for wide swaths of disciplines) https://news.ycombinator.com/item?id=38196412 https://news.ycombinator.com/item?id=38196412
- Kwpolska 2y ago
  Python's import system, as it is, makes some things slightly more complicated due to the need to use virtual environments (copies of the interpreter with separate library paths) as opposed to something like node_modules. This could easily be changed, if the powers-that-be actually cared about packaging. Packaging is handled by a separate group, which likes design by committee, and which enables the creation of separate third party tools instead of standardising on one good tool (uv, rye, and poetry, are all popular tools made by people who are explicitly not members of the packaging group).
atdt 2y ago
Armin advocates for 'uv' to dominate the space, but acknowledges it could be rug-pulled due to its VC backing. His solution to this potential issue is that it's "very forkable." But doesn't forking inherently lead to further fragmentation, the very problem he wants to solve? Any tool hoping to dominate the Python packaging landscape must be community-driven and community-controlled, IMO.
- thatxliner 2y ago
  Isn’t npm VC-backed?
  - simonw 2y ago
    It was until it got acquired by Microsoft/GitHub.
- eviks 2y ago
  Forking doesn't inherently lead to further fragmentation: the level of fragmentation post forking can still be much lower than before consolidating on the rug-pulled tool (also, how many more decades does this imaginary community need to create a great dominant tool?)
- Kwpolska 2y ago
  It may be easily forkable due to the licence choice (MIT or Apache), but the choice of Rust limits the number of people who can actually contribute.
insane_dreamer 2y ago
How well are the CUDA and Jax related packages supported by uv?
- sooheon 2y ago
  jax itself is just pip installable, so this should be drop-in. CUDA is it's own headache, and that's where mamba (now pixi[1]) really shines. [1]: https://pixi.sh/ https://pixi.sh/
  - insane_dreamer 2y ago
    looking at the uv docs it seems it doesn't support conda packages. That's a non-starter for us as we need both conda and pypi (some packages are in one, some in the other). So we'd probably look at pixi as a possible replacement for conda.
    - maleldil 2y ago
      If you really need conda, then you're on a _different_ nightmare than standard Python packaging.
      - insane_dreamer 2y ago
        indeed we are; but CUDA+Jax, which we need, isn't properly supported on pypi
      - insane_dreamer 2y ago
        so UV and PIXI are both nice in that they provide per-project environments (though using conda envs works just fine for us), but they don't solve the actual python packaging problem in that they still depend on either the pypi or conda packages (and neither support both)
        maleldil 2y ago
        Yes. They are tools to manage the user side of packaging (i.e. installing, managing versions, locking, environments) but can't do anything to fix the problematic ecosystem. That is a much harder problem to solve.
incorrecthorse 2y ago
Those people seem to have a passion for developing package managers (instead of just seeing it as a tool that needs to do the job), and as long as it is the case, I don't see how we wouldn't end up with one new package manager every year.
- the_mitsuhiko 2y ago
  Who are "those people"?
- rtpg 2y ago
  People aren’t writing new package managers for rust every other week. JavaScript still sees some but less. Perhaps people are trying to solve real pain points, and by getting closer to solving them things feel nicer!
objectified 2y ago
At first I was excited to see that a new tool would solve the Python "packaging" problem. But upon further reading, I realized that this was about _package management_, not so much about packaging a Python application that I've built. Personally I haven't had many problems with package management in Python. While the ecosystem has some shortcomings (no namespaces!), pip generally works just fine for me. What really annoys me about Python, is the fact that I cannot easily wrap my application in an executable and ship it somewhere. More often than not, I see git clones and virtualenv creation being done in production, often requiring more connectivity than needed on the target server, and dev dependencies being present on the OS. All in all, that's a horrible idea from a security viewpoint. Until that problem is fixed, I'll prefer different languages for anything that requires some sort of end user/production deployment.
- the_mitsuhiko 2y ago
  > What really annoys me about Python, is the fact that I cannot easily wrap my application in an executable and ship it somewhere. You are not wrong, but let's unpack this. What you're saying is that there is a need to make it easy for another person to run your application. What is needed for that? Well you need a way for the application to make its way to the user and to find some Python there and for that process to be transparent to the user. That's one of the reasons why I wanted Rye (and uv does the same) to be able to install Python and not in a way where it fucks up your system in the process. The evolved version of this is to make the whole thing including uv be a thing you can do automatically. You can even today already (if you want to go nuts) have a fully curl to bash installer that installs uv/rye and your app into a temporary location just for your app and never break your user's system. It would be nice to eventually make that process entire transparent and not require network access, to come with a .msi for windows etc. However the per-requisite for this is that a tool like uv can arbitrarily place a pre-compiled Python and all the dependencies that you need at the right location for the platform of your user. The cherry on the top that uv could deliver at one point is that fully packaged thing and it will be very nice. But even prior to this, building a command line tool with Python today will no longer be an awful experience for your users which I think is a good first step. Either via uvx or if you want you can hide uv entirely away.
BerislavLopac 2y ago
Just a reminder of the article that should be mandatory reading for anyone interested in Python packaging tools: https://news.ycombinator.com/item?id=40045318 https://news.ycombinator.com/item?id=40045318
BodyCulture 2y ago
Is there any chance that computer scientists will analyse the software distribution situation for several language eco systems and finally find a general solution so that we can stop wasting so much time with these things? It feels like we were driving cars since 50 years and still haven’t figured out a way to distribute gas. Is there any research going on? The situation is totally crazy, especially for python. I would like to see this done by top scientists. I would love to never have to spend any time again on the newest packaging tool. What is the core of this problem and why is it not solved?

254 points by keybits 2y ago

Uv's latest release was discussed yesterday: https://news.ycombinator.com/item?id=41302475 https://news.ycombinator.com/item?id=41302475 The linked post is the author of Rye's take on that.

bananapub 2y ago

I really like this framing - lots of incremental work by lots of people over time got us to the point where ~a few people at one company can radically improve the situation with a medium amount of work.

deleted 2y ago

[deleted]

dirtbag__dad 2y ago

TY to the astral team for making my quality of life so much better and Armin for being brave enough to pass the torch. Strong +1 on a one tool wins approach - I am so tired of burning time on local dev setup, everything from managing a monorepo (many packages that import into each other) to virtual environments and PYTHONPATH (I’ve been at it for like 8 years now and I still can’t grok how to “just avoid” issues like those across all pkg managers, woof!) I am really excited to see what’s next. Especially looking for a mypy replacement and perhaps something that gives compiling python a “native” feeling thing we can do

alanfranz 2y ago

There’s one problem left with those tools: authority. They’re not pypa endorsed, that’s what makes those different from cargo. At the same time pypa wasn’t able to provide a comprehensive solution over the years, and python packaging and development tools multiplied - just 3-4 years ago poetry and pipenv seemed to solve python packaging problems in a way that pip+virtualenv couldn’t. We need pypa to now jump on the astral.sh ship - but will they do that without a certain amount of control?

the_mitsuhiko 2y ago
As a somewhat outside observer I don't quite understand what the PyPA actually is. In some sense there is an unclear number of participants and it's not even sure to which degree the PyPA relates to core python or the PSF. I think the real endorsement that could help would be the core Python project itself. In a perfect world the official Python tutorial would start with "here is how you install python" and it starts by installing uv, the same way as the official Rust docs point to rustup and cargo. I hope strongly that the PSF will manage to establish some sort of relationship with Astral which would enable that to eventually be a reality.
- crabbone 2y ago
  Here are few things that... well, won't really work well: * Python package installation, package format and loading of modules are defective. The design is bad. It means that no implementation blessed by PSF or not isn't going to solve the packaging problem. So, there's no point to ask PSF or PyPA to adopt any external tool. If the external tools are better than pip in some way, then it could be the speed or memory footprint etc. They will not solve the conceptual problems, because they don't have the authority to do that. * PyPA and PSF, but maybe to a lesser extent, are populated today by delusional mediocre coders who have no idea where the ship is going or how to stir it. They completely lack vision and understanding of the problems they are to deal with. They add "features", but they don't know if those features are needed, and, in most cases it's just noise and bloat. From a perspective of someone who has to deal with fruits of their labor, they just ensure that my job of "someone who fixes Python packaging issues" will never go away. So... as "someone who fixes Python packaging issues" I kind of welcome the new level of hell coming from Astral. From where I stand, it is pouring more gas into a big dumpster fire. Just one more tool written in a non-mainstream, non-standardized, quickly evolving language, impossible to debug without a ton of instrumentation, with the source code hard to write and hard to understand. It's just another deposit towards my job security.
  - simonw 2y ago
    "Python package installation, package format and loading of modules are defective" How are they defective?

fvdessen 2y ago

I was looking this morning at migrating our software from poetry to uv at my company, due to poetry's slowness. And so far i've been reading a lot of doc and not getting a lot things done. I did the previous migration to poetry as well which was vastly simpler. So far it seems that poetry tried to make a simple package manager that works like any other, while uv is keeping quite a bit of the python package insanity around.

deleted 2y ago
[deleted]
burntsushi 2y ago
Are there any specifics you can share about what you found difficult to do in uv?
bvrmn 2y ago
At least uv doesn't lead to an absolute clusterfuck of poetry conflicting with virtualenv. Or broken package.toml format with minor poetry changes. Or amazingly dumb "sources" which doesn't work for transitive dependencies and leads to even longer resolve times in case of multiple indexes.
- seabrookmx 2y ago
  How does it conflict with virtualenv? Last I checked it's default behavior was to automagically create a virtualenv for you if you run it from system python (a bit weird but ok) or if you were in an virtualenv, it would use it. For our projects we use pyenv-virtualenv (so we can have specific python versions per project) and then poetry "just works" (though can be slow, hence rye, uv and friends).
orbisvicis 2y ago
To which insanities are you referring?
- bschwindHN 2y ago
  Probably the entirety of the current python package management ecosystem. From the outside, it all looks insane but I suppose most users are Stockholm-syndrome-d into thinking everything is fine.

PaulHoule 2y ago

My take is that this one of the most interesting topics there is right now but a below average article about that topic.

simonw 2y ago
Are you saying that because this was more of an opinion piece? As the author of Rye, a package that was heavily influential on the development of uv, Armin is very well positioned to share his opinions about this.
- deleted 2y ago
  [deleted]

lmeyerov 2y ago

After the whole npm VC rugpull + Microsoft acquisition, and OpenAI showing legal non-profit status is toothless marketing to VC-path-entangled leaders, I'm reluctant to cede critical path language infra to these kinds of organizations. Individual contributors to these are individually great (and often exceptional!), but financial alignment at the organizational level is corrupted out of the gate. Fast forward 1-4 years, and the organization is what matters. "Die a hero or live long enough to become the villain." So fast lint, type checking, code scans, PR assistants, yes, we can swap these whenever. But install flow & package repo, no. That is unfortunate given the state of pip and conda... But here we are.

crabbone 2y ago
That battle has already been lost for Python. Microsoft owns Python, they just don't make it public. This is how I came to believe this is the case: Few years ago I wanted to write Python bindings to kubectl. I discovered that in order for that to work cross-platform, I need to make CGo use the same compiler on all platforms as does Python. Unfortunately, on MS Windows, CGO uses MINGW while Python uses MSVC. I wrote to Python dev. mailing list (which still existed at that time) and asked why did they choose to use a proprietary compiler for their "open-source" project. The answer I received in a round-about way was that MSVC was a historical choice, which cannot be presently changed because MS provides Python Foundation with free infrastructure to run CI and builds, and it also provides developers to work on Python (i.e. MS employees get paid by MS to work on Python interpreter). And that they are under orders not to drop MS tools from the toolchain. Year after year the situation was getting worse. Like in a lot of similar projects, success created a lot of ground for mediocre nobodies to reach positions of power. Python foundation and satellite projects like PyPA started to be populated by people whose way into these positions was not through contributing any useful code, but rather writing pages of code of conduct. This code of conduct and never-ending skirmishes around controlling positions eventually led to some old-timers leaving or being outright kicked out (latest such event was the ban of Tim, the guy who, beside other things, wrote Tim sort, which is a somewhat famous feature of Python). Year after year MS was pushing its usual agenda they do in every project they get their hands on: add crapload of useless features for the sake of advertising. Make the project swing every way possible, but mostly follow the fashion trends as hard as possible. This is how Python is now devoted to adding as much of ML-style types as possible (in the language with a completely different type system...), AoT compilation and JIT (in the language that's half of the time used to dynamically glue native libraries...) and so on. Essentially, making it a C#, but without curly braces. MS is smart enough to understand that publicly announcing their ownership of Python will scare a lot of people away from the technology, so they don't advertise it much. But they keep working on ensuring developers' dependency on their tooling, and eventually they will come to collect on their investment.

ris 2y ago

I wouldn't blame people if they sat this round out and waited for 2026's iteration of "Python Package Managers: we've really solved it this time!" (still a happy Nix user)

nurettin 2y ago
Should have ended with "I use Nix, btw"
- intelVISA 2y ago
  It's important to remind people that all this is solved, by Nix, btw.
  - maleldil 2y ago
    For Python, specifically? As in, pyproject.toml-compatible tooling, with virtual environments and the like? I've see Nix being used for compiled languages, and maybe I'm missing something, but I haven't really seen it used to manage real-world projects. How do I manage a modern Python project with nix? How do I publish once to PyPI?
    - createaccount99 2y ago
      Agree. Don't think gp has any idea what they're saying. Using nix to install python dependencies is a disaster, it's only good for the python versions themselves.
      - maleldil 2y ago
        > it's only good for the python versions themselves Versioning Python isn't hard. pyenv, asdf, mise, now uv... I honestly don't see what Nix brings to the Python ecosystem. I can see using it to version Python if you already use it, but that's it.
        2y ago

zokier 2y ago

I know python packaging has it's issues, but so far I personally have gotten pretty far with plain pip. The biggest shift to me was switching from original virtualenv to built-in venv module. On the other hand, if I wanted to be really serious about dependency management, I'd steal a page from FAANG, and build a monorepo and avoid all this hassle with package managers.

drawnwren 2y ago
I would really encourage you to try your hand at at a monorepo. I manage a python monorepo in prod and dependency management is hell. Poetry has some newer features that I am looking at trying to implement, but the state of the ecosystem wrt big monorepos is horrible.
- akdor1154 2y ago
  I'm in a similar boat - uv's workspaces look tantalising.
- crabbone 2y ago
  [flagged]
  - miohtama 2y ago
    Generally all Python software use third party packages which are fetched from PyPi package site. These packages are not stored in monorepo.
    - crabbone 2y ago
      This is maybe common, but this contradicts the definition of monorepo. You just use the word incorrectly. "Mono" means "one". If you pull packages from elsewhere, that stops being "mono". There's really no difference in this situation between your team publishing multiple packages from multiple repositories and then assembling them together for the purpose of deployment, or doing so, but with the third-party packages.

orbisvicis 2y ago

With declarative tooling - unlike setup.py - we've lost the ability to install to locations not managed by Python. Most of my projects used to have a python-code config file that I could include both in my code and in setup.py. Last I checked none of that is now possible. Want to install a system binary - ship an RPM or a flatpak. I don't understand how we could lose so much flexibility and yet gain so little in return. P.S I've only ever encountered minor dependency issues in my admittedly small projects using just pip and venv.

OJFord 2y ago
That's entirely appropriate imo - system packager should be running pip or whatever and then putting the output where it wants, it's not for setuptools or whatever to declare that. What path would you put in setup.py anyway? A different one for different distro preferences, a different one again for Windows, for macOS?
heavyset_go 2y ago
You can still use the setuptools backend when using declarative tooling, I use it in projects I can't fully port to pyproject.toml.

bkor 2y ago

For those interested in uv (instead of pip), uv massively sped up the release process for Home Assistant. The time needed to make a release went down from ~2.5 hours to ~20 minutes. See https://developers.home-assistant.io/blog/2024/04/03/build-images-with-uv/ https://developers.home-assistant.io/blog/2024/04/03/build-i... for details. I'm just a HA user btw.

forrestthewoods 2y ago
How the hell was pip taking over an hour to produce an image for a non-compiler language? I’m only a casual Python user. But wtf was it doing and why did it take so long? That’s bonkers.
- cqqxo4zV46cp 2y ago
  Python is slow compared to Rust, obviously. Beyond that, pip is at this point carrying a bunch of legacy decisions because the ludicrously large number of hard left turns the Python packaging ecosystem has taken over the last 20 years. Home Assistant is an absolute behemoth of a project, especially with regard to dependencies. Dependency resolution across a project of that size is nuts. There are probably few currently projects that’d see as big an improvement aa HA.
  - forrestthewoods 2y ago
    Any specifics on what those legacy decisions are? I'm just curious. Have no idea. I'm a very casual Python user.
    - mkesper 2y ago
      If no hard versions are given, but only e.g. <= 2.1, pip will download EVERY SINGLE VERSION until 2.1 to look for metadata. That easily can take hours if it happens multiple times.
      - bazzargh 2y ago

the__alchemist 2y ago

The churn is interesting. In 2019, I made a python version-manager and dependency manager written in rust. I gave up after it seemed like no one wanted to use it. Everyone not satisfied with Pip was on Poetry or Pipenv; I made the one I did because they both had user-interface problems; of the sort I would run into immediately. (I believe Poetry would default to Python 2, and not give you a choice by default, or something to that effect). Now there is a new batch. The biggest challenge was dealing with older pacakges that used non-standard packaging and ran arbitrary code; generally ones that didn't have wheels. From the article: > As of the most recent release, uv also gained a lot of functionality that previously required Rye such as manipulating pyproject.toml files, workspace support, local package references and script installation. It now also can manage Python installations for you so it's getting much closer. These are all things that dead project I wrote could do.

toyg 2y ago
The harsh reality is that this sort of tooling requires a lot of publicity to be successful. Nobody is going to try an unknown package manager from some random developer, but people will enthusiastically adopt anything pushed by a "famous" developer like Ronacher.
vaylian 2y ago
Not all good things gain traction. It definitely helped that Armin is well-known in the Python community. Where did you advertise your new tool?
- the__alchemist 2y ago
  I think I posted on reddit, discord, and here. Didn't really do much advertising, which is obviously a room elephant!

remram 2y ago

So is Rye deprecated? The puzzling rye/uv relationship is what had stopped me from trying out with tool so far.

mixmastamyk 2y ago
It was a prototype. Lessons/code folded into uv.
jsmeaton 2y ago
Rye was started by Armin as a collection of existing tools (mostly) with a frontend similar to cargo. Then Astral came out with uv which aims to be a frontend into a collection of their own tools similar to cargo. Armin and Astral agreed for Astral to take over Rye some time during uv development with (I assume) the goal for uv to fully replace Rye. Use uv. As of 0.3.0 it covers most of rye now anyway. Especially if you’re writing projects and not consumable libs/apps (I haven’t used uv for anything other than package management so far).

billdueber 2y ago

I'm kind of interested in this space -- can anyone point me at an article that goes over why this is harder for python than it seems to be for, e.g., ruby? Is there something inherent about the way code is imported in python that makes it less tractable? Or is it just that the python world has never quite all come together on something that works well enough? (Note that I can certainly complain about how `bundler` works in ruby, but these discussions in python-land seem to go way beyond my quibbles with the ruby ecosystem)

mixmastamyk 2y ago
It's not. Just that Guido never cared about packaging, so it was left to a ragtag unpaid motley crew to piece together and later learn from industry practices that solidified a decade or two after they started.
0cf8612b2e1e 2y ago
The big problem is that Python is used in a lot of different contexts and has too many valid ways in which people are able to build, compile, release non-Pythonic extensions in their package. Original way for packaging code was effectively an executable script which made some things easy, but other things much harder. Modern efforts are trying to limit the flexibility of the package definitions without breaking all of this legacy code. I unfairly think of Ruby as only for Ruby on Rails, so only ever dealing with the web domain. Here is a story about the nightmare it takes to compile the Fortran code that supports much of SciPy (backbone numerical computing library with algorithms for wide swaths of disciplines) https://news.ycombinator.com/item?id=38196412 https://news.ycombinator.com/item?id=38196412
Kwpolska 2y ago
Python's import system, as it is, makes some things slightly more complicated due to the need to use virtual environments (copies of the interpreter with separate library paths) as opposed to something like node_modules. This could easily be changed, if the powers-that-be actually cared about packaging. Packaging is handled by a separate group, which likes design by committee, and which enables the creation of separate third party tools instead of standardising on one good tool (uv, rye, and poetry, are all popular tools made by people who are explicitly not members of the packaging group).

atdt 2y ago

Armin advocates for 'uv' to dominate the space, but acknowledges it could be rug-pulled due to its VC backing. His solution to this potential issue is that it's "very forkable." But doesn't forking inherently lead to further fragmentation, the very problem he wants to solve? Any tool hoping to dominate the Python packaging landscape must be community-driven and community-controlled, IMO.

thatxliner 2y ago
Isn’t npm VC-backed?
- simonw 2y ago
  It was until it got acquired by Microsoft/GitHub.
eviks 2y ago
Forking doesn't inherently lead to further fragmentation: the level of fragmentation post forking can still be much lower than before consolidating on the rug-pulled tool (also, how many more decades does this imaginary community need to create a great dominant tool?)
Kwpolska 2y ago
It may be easily forkable due to the licence choice (MIT or Apache), but the choice of Rust limits the number of people who can actually contribute.

insane_dreamer 2y ago

How well are the CUDA and Jax related packages supported by uv?

sooheon 2y ago
jax itself is just pip installable, so this should be drop-in. CUDA is it's own headache, and that's where mamba (now pixi[1]) really shines. [1]: https://pixi.sh/ https://pixi.sh/
- insane_dreamer 2y ago
  looking at the uv docs it seems it doesn't support conda packages. That's a non-starter for us as we need both conda and pypi (some packages are in one, some in the other). So we'd probably look at pixi as a possible replacement for conda.
  - maleldil 2y ago
    If you really need conda, then you're on a _different_ nightmare than standard Python packaging.
    - insane_dreamer 2y ago
      indeed we are; but CUDA+Jax, which we need, isn't properly supported on pypi
    - insane_dreamer 2y ago
      so UV and PIXI are both nice in that they provide per-project environments (though using conda envs works just fine for us), but they don't solve the actual python packaging problem in that they still depend on either the pypi or conda packages (and neither support both)
      - maleldil 2y ago
        Yes. They are tools to manage the user side of packaging (i.e. installing, managing versions, locking, environments) but can't do anything to fix the problematic ecosystem. That is a much harder problem to solve.

incorrecthorse 2y ago

Those people seem to have a passion for developing package managers (instead of just seeing it as a tool that needs to do the job), and as long as it is the case, I don't see how we wouldn't end up with one new package manager every year.

the_mitsuhiko 2y ago
Who are "those people"?
rtpg 2y ago
People aren’t writing new package managers for rust every other week. JavaScript still sees some but less. Perhaps people are trying to solve real pain points, and by getting closer to solving them things feel nicer!

objectified 2y ago

At first I was excited to see that a new tool would solve the Python "packaging" problem. But upon further reading, I realized that this was about _package management_, not so much about packaging a Python application that I've built. Personally I haven't had many problems with package management in Python. While the ecosystem has some shortcomings (no namespaces!), pip generally works just fine for me. What really annoys me about Python, is the fact that I cannot easily wrap my application in an executable and ship it somewhere. More often than not, I see git clones and virtualenv creation being done in production, often requiring more connectivity than needed on the target server, and dev dependencies being present on the OS. All in all, that's a horrible idea from a security viewpoint. Until that problem is fixed, I'll prefer different languages for anything that requires some sort of end user/production deployment.

the_mitsuhiko 2y ago
> What really annoys me about Python, is the fact that I cannot easily wrap my application in an executable and ship it somewhere. You are not wrong, but let's unpack this. What you're saying is that there is a need to make it easy for another person to run your application. What is needed for that? Well you need a way for the application to make its way to the user and to find some Python there and for that process to be transparent to the user. That's one of the reasons why I wanted Rye (and uv does the same) to be able to install Python and not in a way where it fucks up your system in the process. The evolved version of this is to make the whole thing including uv be a thing you can do automatically. You can even today already (if you want to go nuts) have a fully curl to bash installer that installs uv/rye and your app into a temporary location just for your app and never break your user's system. It would be nice to eventually make that process entire transparent and not require network access, to come with a .msi for windows etc. However the per-requisite for this is that a tool like uv can arbitrarily place a pre-compiled Python and all the dependencies that you need at the right location for the platform of your user. The cherry on the top that uv could deliver at one point is that fully packaged thing and it will be very nice. But even prior to this, building a command line tool with Python today will no longer be an awful experience for your users which I think is a good first step. Either via uvx or if you want you can hide uv entirely away.

BerislavLopac 2y ago

Just a reminder of the article that should be mandatory reading for anyone interested in Python packaging tools: https://news.ycombinator.com/item?id=40045318 https://news.ycombinator.com/item?id=40045318

BodyCulture 2y ago

Is there any chance that computer scientists will analyse the software distribution situation for several language eco systems and finally find a general solution so that we can stop wasting so much time with these things? It feels like we were driving cars since 50 years and still haven’t figured out a way to distribute gas. Is there any research going on? The situation is totally crazy, especially for python. I would like to see this done by top scientists. I would love to never have to spend any time again on the newest packaging tool. What is the core of this problem and why is it not solved?