11 ms·

Conda: A package management disaster?

161 points by osdotsystem 2y ago

orf 2y ago
Impossible to read on mobile, least of all because of the lack of any word breaking.
- anakaine 2y ago
  Seconded. How about we don't write a blog trashing an implementation of something when our own design is missing some very basic accessibility and ux features. Though I agree with the premise, Conda is an absolute pest when you start customising an environment with a number of packages. Dependency resolution hell.
  - osdotsystem 2y ago
    All the articles are from the Python mailing list, like the blog selected a mail (Added the source in italics below the title)
- bangaladore 2y ago
  Even on Desktop. When 30% of your page is junk, it makes you wonder about the other 30%...
- emmelaich 2y ago
  You might the linked src. https://mail.python.org/pipermail/python-list/2024-May/912306.html https://mail.python.org/pipermail/python-list/2024-May/91230... [oof, never mind, it's worse in some ways]
- foepys 2y ago
  How does one even do that? Browsers try quite hard to not break words and make text readable by default.
  - pphysch 2y ago
    The author put "word-break:break-all" on a parent element of the text content, which itself is a <ul> containing <p>, probably for "layout" purposes. Methinks some CSS education is desperately needed.
  - deathanatos 2y ago
    The blog author explicitly requested it, with `word-break: break-all`. Now why you would do that … IDK. Weirdly, it's the second post on HN this quarter to do it, from a completely different site. Makes me wonder if there's some viral piece of CSS advice out there …? (And nobody looks at their site…?) Bad LLM output?
- deleted 2y ago
  [deleted]
- kelnos 2y ago
  I see this on Firefox on desktop too. I usually try not to criticize the presentation of things posted here, but this is completely unreadable. I've tried several times to get through the first couple paragraphs, but it's just not worth the extra mental effort.
- Finnucane 2y ago
  I gave up after a few lines. Why would you do this?
- osdotsystem 2y ago
  fixed!
alsodumb 2y ago
As someone with admittedly no formal CS education, I've been using conda for all of my grad school and never managed to break it. I create a virtual environment for every project. I install almost all packages with pip, except for any binaries or CUDA related things from conda. I always exported the conda yaml file and managed to reproduce the code/environment including the Python version. I've seen a lot of posts over time praising poetry and other tools and complaining about conda but I could never relate to any of them. Am i doing something wrong? Or something right?
- jszymborski 2y ago
  God forbid you should require conda-forge and more than three packages lest the dependency resolver take longer than the heat death of the planet to complete.
  - fransje26 2y ago
    Install mamba first?
    - jszymborski 2y ago
      Mamba is indeed a lot better. I personally just don't bother with conda and stick to pip + venv.
- thangngoc89 2y ago
  1. You need to run export manual while other tools you mentioned would create it automatically (the lock file) 2. Distinguishes between direct dependencies (packages you added yourself) and indirect dependencies (packages of the packages)
- theamk 2y ago
  You are doing something right, author does some pretty unusual things: - Setup custom kernels in Jupyter Notebook - Hardlink the environments, then install same packages via pip in one and conda in others - install conda inside conda (!!!) and enter nested environment - Use tox within conda I believe as long as you treat the environments as "cattle" (if it goes bad, remove it and re-create from yaml file), you should not have any problems. It's clearly not the case of for the post's author though.
kussenverboten 2y ago
I prefer uv now. https://github.com/astral-sh/uv https://github.com/astral-sh/uv
zefrieira 2y ago
I think Pixi mostly solves the main issues of conda by forcing users to have project-specific environments. It also solves environments incredibly fast, so it’s really quick to create new projects/environments. https://pixi.sh/ https://pixi.sh/
viraptor 2y ago
> The traditional setup.py install command may install multiple versions of the same package into the same directory Wait, what? In what situation would that ever happen? Especially given the directories for packages are not versioned, so setuptools should never do two different versions in any way.
shadowgovt 2y ago
I think Python had a pretty good idea in standardizing a packaging protocol and then allowing competing implementations, but I would have preferred a single "blessed" solution. More than one package management option in an ecosystem always adds some kind of "can't get there from here" friction and an additional maintenance burden on package maintainers. poetry has been working well enough for me as of late, but it'd be nice if I didn't have to pick.
SamPatt 2y ago
Conda is the only package manager I've used on Ubuntu that intermittently and inexplicably gets stuck when installing or uninstalling. It will sometimes resolve itself if left alone for hours, but often won't. I avoid it as much as possible.
- StableAlkyne 2y ago
  It's because of the SAT solver for dependencies. Unlike Pip, it keeps track of every package you installed and goes out of its way to avoid installing incompatible packages. Why go through all this trouble? Because originally it was meant to be a basic "scientific Python" distribution, and needed to be strict around what's installed for reproducibility reasons. It's IMO overkill for most users, and I suspect most scientific users don't care either - most of the time I see grads and researchers just say "fuck it" and use Pip whenever Conda refuses to get done in a timely fashion. And the ones who do care about reproducibility are using R anyway, since there's a perception those libraries are "more correct" (read: more faithful to the original publication) than Pythonland. And TBH I can't blame them when the poster child of it is Sklearn's RandomForestRegressor not even being correctly named - it's bagged trees under the default settings, and you don't get any indication of this unless you look at that specific kwarg in the docs. Personally, I use Conda not for reproducibility, but so all of my projects have independent environments without having to mess with containers
  - Balinares 2y ago
    > Personally, I use Conda not for reproducibility, but so all of my projects have independent environments without having to mess with containers A perfectly reasonable goal, yup! Thankfully not one that, in fact, requires conda. Automated per-project environments are increasingly the default way of doing things in Python, thank goodness. It's been a long time coming.
  - MathMonkeyMan 2y ago
    Wow, TIL: https://www.anaconda.com/blog/understanding-and-improving-condas-performance https://www.anaconda.com/blog/understanding-and-improving-co... Neat idea, but sounds like a lot of work.
smcleod 2y ago
I honestly have no idea why anyone still uses Conda, it's a right pain in the ass. Python package management in general is a nightmare, but whenever I run up a project that uses Conda I immediately disregard it and use uv / pyenv.
The_Colonel 2y ago
It's rare to see something as systematically broken as Python package/dependencies ecosystem. What I don't understand - what makes this so difficult to solve in Python? It seems that many other platforms solved this a long time ago - maven 2.0 was released almost 20 years ago. While it wasn't / isn't by no means perfect, its fundamentals were decent already back then. One thing which I think messed this up from the beginning was applying the Unix philosophy with several/many individual tools as opposed to one cohesive system - requirements.txt, setuptools, pip, pipx, pipenv, venv... were always woefully inadequate, but produced a myriad of possible combinations to support. It seems like simplicity was the main motivation for such design, but these certainly seems like examples of being too simplistic for the job. I recently tried to run a Python app (after having a couple of years break from Python) which used conda and I got lost there quickly. Project README described using conda, mamba, anaconda, conda-forge, mini-forge, mini-conda ... In the end, nothing I tried worked.
- pmarreck 2y ago
  Nix solves it for me. Takes a bit more effort upfront, but the payoff is "Python dependency determinism," which is pretty much unachievable in any other way, so...
  - reactordev 2y ago
    The answer is not Yet Another Tool In The Chain. Python community itself needs to address this. Because if they don’t then you’ll have requirements.txt, setuptools, pyproject, pip, pipx, pipenv, pyenv, venv, nix.
    - The_Colonel 2y ago
      Agreed. Often there's a quite tight coupling between the core platform devs and package management - node.js has its npm, rust cargo, go has one as well and for the most part it seems to have worked out fine for them. Java and .NET (and I think PHP) are different in the sense that the package management systems have no relation to the platform developers, but industry standards (maven, gradle, NuGET, Composer) still appeared and are widely accepted. But with Python it seems completely fractured - everyone tries to solve it their own way, with nothing becoming a truly widely used solution. More involvement from the Python project could make a difference. From my perspective, this mess is currently Python's biggest problem and should be prioritized accordingly.
pmarreck 2y ago
I feel like a major selling point of Nix is "solving the Python dependency-hell problem" (as well as that of pretty much every other stack) I've seen so many issues with different Python venvs from different Python project directories stepping on each others' dependencies somehow (probably because there are some global ones) that the fact that I can now just stick a basic and barely-modified-per-project Python flake.nix file in each one and be always guaranteed to have the entirely of the same dependencies available when I run it 6 months later is a win.
- drawnwren 2y ago
  Do you have a publicly available copy of your flake?
  - jyap 2y ago
    I started using Devenv which utilizes Nix. You might want to check that out. https://devenv.sh/ https://devenv.sh/
jkrubin 2y ago
All thoughts and opinions about conda aside, it’s the only sane way (on several platforms) to install gdalbins + gdal-python-bindings. I don’t mind conda. It has a lot of caveats and weird quirks
- cozzyd 2y ago
  dnf install python3-gdal?
j0057 2y ago
This seems to be an aggregation of some posts on python-list. Basically, extra-random opinions. I'll offer mine: I won't say that Python packaging is generally excellent, but it's gotten much better over the years. The pyproject.toml is a godsend, there's the venv module built-in to Python, pip will by default no longer install package outside of a venv. Dependency groups are being added, meaning that the requirements.txt files can also be specified in the project.toml. Documentation is pretty good, especially if you avoid blog posts from 5+ years ago.
- jszymborski 2y ago
  pip + venv or just using Poetry usually is 100% headache-free for me. Conda, however, is usually a great way to ensure I have an awful time.
BrenBarn 2y ago
Nothing in the "article" seems to support the title. A lot of it is just about Python packaging in general, or about problems when mixing conda- and pip-installed packages. In my experience conda is enormously superior to the standard Python packaging tools.
- Balinares 2y ago
  If we're doing anecdotal evidence, then mine is that conda is by far the worst of the main Python packaging solutions in use. The absurd slowness and incompatibility with the entire rest of the Python world are only the visible tip of that iceberg. To the best of my ability to tell, conda largely exists to make up for endemic deficiencies in Windows software distribution toolchains (not Python specific) and sadly it's not even good at that either. Mind you, glad it works for you. Warms my grey heart to know there's some balance in this universe. :)
  - maxnoe 2y ago
    The "absurd slowness" is gone since more than a year when it switched to using the libmamba solver.
    - Balinares 2y ago
      We must be living in slightly different universes. (I'm in the Berenstain one. Things are not amazing here TBH.) I'll grant you that libmamba is faster, but we're still talking "tragicomic, leaning tragic" which is not in fact a qualitative improvement because we're still in the territory of switching to a different task while resolution, eventually, occurs. Another poster mentioned pixi elsewhere in the thread. I'll need to look into that.
icameron 2y ago
Conda: a package manager disaster that became paid license required for companies over 200 employees. It worked 5 years ago, we can no longer legally use it
- rbanffy 2y ago
  It kind of limits its own blast radius that way.
These335 2y ago
Can somebody please eli5 why it is so unanimously accepted that Python's package management is terrible? For personal projects venv + requirements.txt has never caused problems for me. For work projects we use poetry because of an assumption that we would need something better but I remain unconvinced (nothing was causing a problem for that decision to be made).
- gdiamos 2y ago
  It's popular enough that it causes pain for a lot of people. Coming from C++, IMO, it is vastly better.
  - dccsillag 2y ago
    Well, yes, but that's an extremely low bar!!
- invaliduser 2y ago
  venv + requirements.txt has worked for every single python project I made for the last 2 years (I'm new to python). Only issue I had was when using a newish python version and not having a specific library released yet for this new version, but downgrading python solved this. Being new to the ecosystem I have no clue why people would use Conda and why it matters. I tried it, but was left bewildered, not understanding the benefits.
  - dagw 2y ago
    I have no clue why people would use Conda The big thing to realise is that when Conda first was released it was the only packaging solution that truly treated Windows as a first class citizen and for a long time was really the only way to easily install python packages on Windows. This got it a huge following in the scientific community where many people don't have a solid programming/computer background and generally still ran Windows on their desktops. Conda also not only manages your python interpreter and python libraries, it manages your entire dependency chain down to the C level in a cross platform way. If a python library is a wrapper around a C library then pip generally won't also install the C library, Conda (often) will. If you have two different projects that need two different versions of GDAL or one needs OpenBLAS and one that needs MKL, or two different versions of CUDA then Conda (attempts to) solve that in a way that transparently works on Windows, Linux and MacOS. Using venv + requirements.txt you're out of luck and will have to fall back on doing everything in its own docker container. Conda lets you mix private and public repos as well as mirroring public packages on-perm in a transparent way much smoother than pip, and has tools for things like audit logging, find grained access control, package signing and centralised controls and policy management. Conda also has support for managing multi-language projects. Does your python project need nodejs installed to build the front-end? Conda can also manage your nodejs install. Using R for some statistical analysis in some part of your data pipeline? Conda will mange your R install. Using a Java library for something? Conda will make sure everybody has the right version of Java installed. Also, it at least used to be common for people writing numeric and scientific libraries to release Conda packages first and then only eventually publish on PyPi once the library was 'done' (which could very well be never). So if you wanted the latest cutting edge packages in many fields you needed Conda. Now there are obviously a huge class a projects where none of these features are needed and mean nothing. If you don't need Conda, then Conda is no longer the best answer. But there are still a lot of niche things Conda still does better than any other tool.
japanuspus 2y ago
The problem conda solved that nothing had solved before was installing binary dependencies on MS Windows. Before conda, getting a usable scipy install up and running on MS Windows was a harrowing experience. And having two independent installations was basically impossible. The real hard work that went into conda was reverse engineering all the nooks and crannies of the DLL loading heuristics, to allow it to ensure that you loaded what you intended. If you are working on macOS and deploying to some *nix in the cloud, you are unlikely to find any value in this. But in ten years as lead on a large tool that was deployed to personal (Windows) laptops in a corporate environment, I did not find anything that beat conda.
- dist-epoch 2y ago
  This was mostly because most scientific packages didn't provide Windows binary builds for many years. Today you can just "pip install scipy" on Windows at it will just work.
  - wruza 2y ago
    Oh right, recently I started learning classic ML and “just” tried to install tensorflow, which, itself or through one of dependencies, stopped providing windows binaries since x.y.z and so my python has to be downgraded to 3.a and then other dependencies stop installing. Eventually I managed to find a proper version intersection for everything together with some shady repo, but it felt like one more requirement and I’ll get overconstrained.
    - disgruntledphd2 2y ago
      Tensorflow is the worst. Basically every time my python env was borked (with multiple incompatible versions of Numpy) it was down to tensorflow.
      - coredog64 2y ago
        You said it. I was working with an official Google library that used TF and it didn’t work at all with 3.12. I spent a day building the wheels for 3.12 only to find there was a bug with dataclasses. :| I can’t recall the library, but there was another major project that just deprecated TF because it was the cause of so many build problems.
oivey 2y ago
Besides the horrendous formatting, some stuff in this article seem incorrect or irrelevant. Like, is this even possible? > A single Anaconda distribution may have multiple NumPy versions installed at the same time, although only one will be available to the Python process (note that this means that sub-processes created in this Python process won’t necessarily have the same version of NumPy!). I’m pretty sure there’s not, but maybe there is some insane way to cause subprocesses to do this. Besides that, under the authors definition, different Python virtualenvs also install multiple copies of libraries in the same way conda does. The comments about Jupyter also seem very confused. It’s hard to make heads or tails of exactly what the author is saying. There might be some misunderstandings of how Jupyter kernels select environments. > Final warning: no matter how ridiculous this is: the current directory in Python is added to the module lookup path, and it precedes every other lookup location. If, accidentally, you placed a numpy.py in the current directory of your Python process – that is going to be the numpy module you import. This has nothing to do with conda.
BiteCode_dev 2y ago
Conda used to be a life saver when years and years ago, compiled extensions were hard to install because you had to compile them yourself. Nowadays, thanks to wheels being numerous and robust, the appeal of anaconda is disappearing for most users except for some exotic mixes. conda itself now causes more trouble than it solves as it's slow, and lives in its own incompatible world. But anaconda solves a different problem now that nobody else solves, and that's managing Python for big corporation. This is worth a lot of money to big structures that need to control packages origin, permissions, updates, and so on, at scale. So it thrives there.
dannyz 2y ago
While `uv` works amazingly well I think a lot of people don't realize that installing packages through conda (or let's say the conda-forge ecosystem) has technical advantages compared to wheels/pypi. When you install the numpy wheel through `uv` you are likely installing a pre-compiled binary that bundles openblas inside of it. When you install numpy through conda-forge, it dynamically links against a dummy blas package that can be substituted for mkl, openblas, accelerate, whatever you prefer on your system. It's a much better solution to be able to rely on a separate package rather than having to bundle every dependency. Then lets say you install scipy. Scipy also has to bundle openblas in their wheel, and now you have two copies of openblas sitting around. They don't conflict, but this quickly becomes an odd thing to have to do.
- KolenCh 2y ago
  In this sense I personally prefer pixi because of this. It is pixi like but resolves using conda channels like conda, and similar to conda it supports PyPI packages via uv. With a background in scientific computing where many of the dependencies I managed are compiled, conda packages gives me much more control. P.S. I’d like to point out to others to differentiate between package index and package managers. PyPI is an index (that hosts packages in a predefined format) while pip, poetry, uv are package managers that resolve and build your environments using the index. Similarly but a bit more confusingly, conda can be understood as the index, hosted by anaconda but can also be hosted elsewhere, with different “channels” (kinda like a GitHub organization) where conda-forge is a popular one built by communities. Conda is also a reference implementation of a package manager that uses anaconda channels to resolve. Mamba is an independent, performant, drop in replacement of conda. And pixi is a different one with a different interface by the author of mamba. Even more confusingly, there are distributions. Distributions come with a set of predefined packages together with the package manager such that you just start running things immediately (sort of like a TeXLive distribution in relation to the package manager tlmgr.) there are anaconda distributions (if you installed anaconda instead of installing conda, that’s what you get), but also Intels distribution for Python, mini forge, mambaforge, etc.
- fire_lake 2y ago
  Is there a reason this behaviour couldn’t be implemented in uv? Is this beyond what the pyproject.toml spec supports?
teekert 2y ago
People here focus on Python, but to me, a bioinformatician, conda is much more, it provides 99.99% of the tools I need. Like bwa, samtools, rsem, salmon, fastqc, R. And many, many obscure tools.
- GuestFAUniverse 2y ago
  And then somebody tries to install mamba via conda and that house of cards reveals itself.
  - teekert 2y ago
    I install Snakemake via miniforge which uses mambaforge to make its own envs. Biology is messy ;)
- deleted 2y ago
  [deleted]
- mbreese 2y ago
  I wish you luck with tracking down versions of software used when you're writing papers... especially if you're using multiple conda environments. This is pretty much the example used in the article -- version mismatches. But, I think this illustrates the problem very well. Conda isn't just used for Python. It's used for general tools and libraries that Python scripts depend on. They could be C/C++ that needs to be compiled. It could be a Cython library. It could be... When you're trying to be a package manager that operates on-top of the operating system's package manager, you're always going to have issues. And that is why Conda is such a mess, it's trying to do too much. Installation issues are one of the reason why I stopped writing so many projects in Python. For now, I'm only doing smaller scripts in Python. Anything larger than a module gets written in something else. People here have mentioned Rust as an example of a language with a solid dependency toolchain. I've used more Go, which similarly has had dependency management tooling from the begining. By and large, these languages aren't trying to bring in C libraries that need to be compiled and linked into Python accessible code (it's probably possible, but not the main use-case). For Python code though, when I do need to import a package, I always start with a fresh venv virtual environment, install whatever libraries are needed in that venv, and then always run the python from that absolute path (ex: `venv/bin/python3 script.py`). This has solved 99% of my dependency issues. If you can separate yourself from the system python as much as possible, you're 90% of the way there. Side rant: Which, is why I think there is a problem with Python to begin with -- *nix OSes all include a system level Python install. Dependencies only become a problem when you're installing libraries in a global path. If you can have separate dependency trees for individual projects, you're largely safe. It's not very storage efficient, but that's a different issue.
ur-whale 2y ago
I tried Conda a number of time over the years, regretted it every time. These days, when I absolutely have to use it because some obscure piece of software can't run unless Conda, I install it in a VM so that: - I protect my working system from the damage of installing Conda on it - I can throw the whole garbage fire away without long term brain damage to my system once I'm done
jujube3 2y ago
[flagged]
- deleted 2y ago
  [deleted]
prpl 2y ago
conda was for scientific python, but had to solve for everything below python to make that work. There was no generic binary solution before python for multiple architectures and operating systems.
benreesman 2y ago
I strongly suspect that there is about to be a spike in Python packaging discussion over and above the high ambient baseline. uv is here to kick ass and chew bubblegum. And it’s all out of gum.
nodivbyzero 2y ago
The Five Demons of Python Packaging That Fuel Our Persistent Nightmare: https://youtu.be/qA7NVwmx3gw?si=QbchrYvCEp8aazvL https://youtu.be/qA7NVwmx3gw?si=QbchrYvCEp8aazvL
whywhywhywhy 2y ago
Really the issue is python itself, it shouldn't be treating it's installs and packages as something that's linked and intertwined to the base operating system. People like to complain about node packages but never seen people have the trouble with them that they have with python.
- macinjosh 2y ago
  This is spot on. Running some python projects on nixos is a nightmare because of this model. Especially if it’s ML related.
- theamk 2y ago
  What do you do though if you want to import code written in C++? Especially complex, dependency-heavy like CUDA/ML stuff? You can just give up and say that "The proper way to do this is to use the Nvidia CUDA toolkit to write your cuda app in C++ and then invoke it as a separate process from node" [0]. That apparently works for node, but Python wants much more. If you actually want to use high-performance native code in your slow compiled language, then no solution is going to be very good, that's because the problem is inherently hard. You can rely on host OS as much as possible - if OS is known, provide binaries; if it's unknown, provide source code and hope user has C/C++/Rust/Fortran compilers to build it. That's what uv, pip, etc.. do. You can create your own parallel OS, bringing your own copy of every math libray, as well as CUDA even if there are perfectly good versions installed on the system - that's what conda/minconda does. You can implement as much as possible in your own language, so there is much less need to use "high-performance native language" - that's what Rust and Go do. Sadly, that's not an option for Python. [0] https://stackoverflow.com/questions/20875456/how-can-i-use-cuda-with-nodejs https://stackoverflow.com/questions/20875456/how-can-i-use-c...
  - whywhywhywhy 2y ago
    >What do you do though if you want to import code written in C++? Especially complex, dependency-heavy like CUDA/ML stuff? No other CUDA software has to put it's claws deep into the os to function. I'm not even convinced Python does, seems it just dumps all the 500-900meg DLLs into the torch folder. It's more the insistence that the libraries have to exist in this place that isn't related to the project that's the issue. Node also handles what you're talking about without issue and without thinking it needs to be part of the operating system like Python does.
kristianp 2y ago
Are most people having problems with python packages using Windows? It's been mentioned a couple of times in this thread, but not that often.
sznio 2y ago
The domain got parked.
mdaniel 2y ago
I hope you read it while it was available, because the domain has expired Domain Name: pyherald.com Registry Domain ID: 2663190918_DOMAIN_COM-VRSN Registrar WHOIS Server: whois.namesilo.com Registrar URL: https://www.namesilo.com/ Updated Date: 2024-12-21T07:00:00Z Creation Date: 2021-12-21T07:00:00Z Registrar Registration Expiration Date: 2024-12-21T07:00:00Z https://web.archive.org/web/20241220211119/https://pyherald.com/articles/16_12_2024/ https://web.archive.org/web/20241220211119/https://pyherald.... is the most recent snap
- btrettel 2y ago
  Looks like the webpage is on GitHub too: https://github.com/pyherald/pyherald/blob/main/docs/articles/16_12_2024/index.html https://github.com/pyherald/pyherald/blob/main/docs/articles...
- osdotsystem 2y ago
  Back online

161 points by osdotsystem 2y ago

orf 2y ago

Impossible to read on mobile, least of all because of the lack of any word breaking.

anakaine 2y ago
Seconded. How about we don't write a blog trashing an implementation of something when our own design is missing some very basic accessibility and ux features. Though I agree with the premise, Conda is an absolute pest when you start customising an environment with a number of packages. Dependency resolution hell.
- osdotsystem 2y ago
  All the articles are from the Python mailing list, like the blog selected a mail (Added the source in italics below the title)
bangaladore 2y ago
Even on Desktop. When 30% of your page is junk, it makes you wonder about the other 30%...
emmelaich 2y ago
You might the linked src. https://mail.python.org/pipermail/python-list/2024-May/912306.html https://mail.python.org/pipermail/python-list/2024-May/91230... [oof, never mind, it's worse in some ways]
foepys 2y ago
How does one even do that? Browsers try quite hard to not break words and make text readable by default.
- pphysch 2y ago
  The author put "word-break:break-all" on a parent element of the text content, which itself is a <ul> containing <p>, probably for "layout" purposes. Methinks some CSS education is desperately needed.
- deathanatos 2y ago
  The blog author explicitly requested it, with `word-break: break-all`. Now why you would do that … IDK. Weirdly, it's the second post on HN this quarter to do it, from a completely different site. Makes me wonder if there's some viral piece of CSS advice out there …? (And nobody looks at their site…?) Bad LLM output?
deleted 2y ago
[deleted]
kelnos 2y ago
I see this on Firefox on desktop too. I usually try not to criticize the presentation of things posted here, but this is completely unreadable. I've tried several times to get through the first couple paragraphs, but it's just not worth the extra mental effort.
Finnucane 2y ago
I gave up after a few lines. Why would you do this?
osdotsystem 2y ago
fixed!

alsodumb 2y ago

As someone with admittedly no formal CS education, I've been using conda for all of my grad school and never managed to break it. I create a virtual environment for every project. I install almost all packages with pip, except for any binaries or CUDA related things from conda. I always exported the conda yaml file and managed to reproduce the code/environment including the Python version. I've seen a lot of posts over time praising poetry and other tools and complaining about conda but I could never relate to any of them. Am i doing something wrong? Or something right?

jszymborski 2y ago
God forbid you should require conda-forge and more than three packages lest the dependency resolver take longer than the heat death of the planet to complete.
- fransje26 2y ago
  Install mamba first?
  - jszymborski 2y ago
    Mamba is indeed a lot better. I personally just don't bother with conda and stick to pip + venv.
thangngoc89 2y ago
1. You need to run export manual while other tools you mentioned would create it automatically (the lock file) 2. Distinguishes between direct dependencies (packages you added yourself) and indirect dependencies (packages of the packages)
theamk 2y ago
You are doing something right, author does some pretty unusual things: - Setup custom kernels in Jupyter Notebook - Hardlink the environments, then install same packages via pip in one and conda in others - install conda inside conda (!!!) and enter nested environment - Use tox within conda I believe as long as you treat the environments as "cattle" (if it goes bad, remove it and re-create from yaml file), you should not have any problems. It's clearly not the case of for the post's author though.

kussenverboten 2y ago

I prefer uv now. https://github.com/astral-sh/uv https://github.com/astral-sh/uv

zefrieira 2y ago

I think Pixi mostly solves the main issues of conda by forcing users to have project-specific environments. It also solves environments incredibly fast, so it’s really quick to create new projects/environments. https://pixi.sh/ https://pixi.sh/

viraptor 2y ago

> The traditional setup.py install command may install multiple versions of the same package into the same directory Wait, what? In what situation would that ever happen? Especially given the directories for packages are not versioned, so setuptools should never do two different versions in any way.

shadowgovt 2y ago

I think Python had a pretty good idea in standardizing a packaging protocol and then allowing competing implementations, but I would have preferred a single "blessed" solution. More than one package management option in an ecosystem always adds some kind of "can't get there from here" friction and an additional maintenance burden on package maintainers. poetry has been working well enough for me as of late, but it'd be nice if I didn't have to pick.

SamPatt 2y ago

Conda is the only package manager I've used on Ubuntu that intermittently and inexplicably gets stuck when installing or uninstalling. It will sometimes resolve itself if left alone for hours, but often won't. I avoid it as much as possible.

StableAlkyne 2y ago
It's because of the SAT solver for dependencies. Unlike Pip, it keeps track of every package you installed and goes out of its way to avoid installing incompatible packages. Why go through all this trouble? Because originally it was meant to be a basic "scientific Python" distribution, and needed to be strict around what's installed for reproducibility reasons. It's IMO overkill for most users, and I suspect most scientific users don't care either - most of the time I see grads and researchers just say "fuck it" and use Pip whenever Conda refuses to get done in a timely fashion. And the ones who do care about reproducibility are using R anyway, since there's a perception those libraries are "more correct" (read: more faithful to the original publication) than Pythonland. And TBH I can't blame them when the poster child of it is Sklearn's RandomForestRegressor not even being correctly named - it's bagged trees under the default settings, and you don't get any indication of this unless you look at that specific kwarg in the docs. Personally, I use Conda not for reproducibility, but so all of my projects have independent environments without having to mess with containers
- Balinares 2y ago
  > Personally, I use Conda not for reproducibility, but so all of my projects have independent environments without having to mess with containers A perfectly reasonable goal, yup! Thankfully not one that, in fact, requires conda. Automated per-project environments are increasingly the default way of doing things in Python, thank goodness. It's been a long time coming.
- MathMonkeyMan 2y ago
  Wow, TIL: https://www.anaconda.com/blog/understanding-and-improving-condas-performance https://www.anaconda.com/blog/understanding-and-improving-co... Neat idea, but sounds like a lot of work.

smcleod 2y ago

I honestly have no idea why anyone still uses Conda, it's a right pain in the ass. Python package management in general is a nightmare, but whenever I run up a project that uses Conda I immediately disregard it and use uv / pyenv.

The_Colonel 2y ago

It's rare to see something as systematically broken as Python package/dependencies ecosystem. What I don't understand - what makes this so difficult to solve in Python? It seems that many other platforms solved this a long time ago - maven 2.0 was released almost 20 years ago. While it wasn't / isn't by no means perfect, its fundamentals were decent already back then. One thing which I think messed this up from the beginning was applying the Unix philosophy with several/many individual tools as opposed to one cohesive system - requirements.txt, setuptools, pip, pipx, pipenv, venv... were always woefully inadequate, but produced a myriad of possible combinations to support. It seems like simplicity was the main motivation for such design, but these certainly seems like examples of being too simplistic for the job. I recently tried to run a Python app (after having a couple of years break from Python) which used conda and I got lost there quickly. Project README described using conda, mamba, anaconda, conda-forge, mini-forge, mini-conda ... In the end, nothing I tried worked.

pmarreck 2y ago
Nix solves it for me. Takes a bit more effort upfront, but the payoff is "Python dependency determinism," which is pretty much unachievable in any other way, so...
- reactordev 2y ago
  The answer is not Yet Another Tool In The Chain. Python community itself needs to address this. Because if they don’t then you’ll have requirements.txt, setuptools, pyproject, pip, pipx, pipenv, pyenv, venv, nix.
  - The_Colonel 2y ago
    Agreed. Often there's a quite tight coupling between the core platform devs and package management - node.js has its npm, rust cargo, go has one as well and for the most part it seems to have worked out fine for them. Java and .NET (and I think PHP) are different in the sense that the package management systems have no relation to the platform developers, but industry standards (maven, gradle, NuGET, Composer) still appeared and are widely accepted. But with Python it seems completely fractured - everyone tries to solve it their own way, with nothing becoming a truly widely used solution. More involvement from the Python project could make a difference. From my perspective, this mess is currently Python's biggest problem and should be prioritized accordingly.

pmarreck 2y ago

I feel like a major selling point of Nix is "solving the Python dependency-hell problem" (as well as that of pretty much every other stack) I've seen so many issues with different Python venvs from different Python project directories stepping on each others' dependencies somehow (probably because there are some global ones) that the fact that I can now just stick a basic and barely-modified-per-project Python flake.nix file in each one and be always guaranteed to have the entirely of the same dependencies available when I run it 6 months later is a win.

drawnwren 2y ago
Do you have a publicly available copy of your flake?
- jyap 2y ago
  I started using Devenv which utilizes Nix. You might want to check that out. https://devenv.sh/ https://devenv.sh/

jkrubin 2y ago

All thoughts and opinions about conda aside, it’s the only sane way (on several platforms) to install gdalbins + gdal-python-bindings. I don’t mind conda. It has a lot of caveats and weird quirks

cozzyd 2y ago
dnf install python3-gdal?

j0057 2y ago

This seems to be an aggregation of some posts on python-list. Basically, extra-random opinions. I'll offer mine: I won't say that Python packaging is generally excellent, but it's gotten much better over the years. The pyproject.toml is a godsend, there's the venv module built-in to Python, pip will by default no longer install package outside of a venv. Dependency groups are being added, meaning that the requirements.txt files can also be specified in the project.toml. Documentation is pretty good, especially if you avoid blog posts from 5+ years ago.

jszymborski 2y ago
pip + venv or just using Poetry usually is 100% headache-free for me. Conda, however, is usually a great way to ensure I have an awful time.

BrenBarn 2y ago

Nothing in the "article" seems to support the title. A lot of it is just about Python packaging in general, or about problems when mixing conda- and pip-installed packages. In my experience conda is enormously superior to the standard Python packaging tools.

Balinares 2y ago
If we're doing anecdotal evidence, then mine is that conda is by far the worst of the main Python packaging solutions in use. The absurd slowness and incompatibility with the entire rest of the Python world are only the visible tip of that iceberg. To the best of my ability to tell, conda largely exists to make up for endemic deficiencies in Windows software distribution toolchains (not Python specific) and sadly it's not even good at that either. Mind you, glad it works for you. Warms my grey heart to know there's some balance in this universe. :)
- maxnoe 2y ago
  The "absurd slowness" is gone since more than a year when it switched to using the libmamba solver.
  - Balinares 2y ago
    We must be living in slightly different universes. (I'm in the Berenstain one. Things are not amazing here TBH.) I'll grant you that libmamba is faster, but we're still talking "tragicomic, leaning tragic" which is not in fact a qualitative improvement because we're still in the territory of switching to a different task while resolution, eventually, occurs. Another poster mentioned pixi elsewhere in the thread. I'll need to look into that.

icameron 2y ago

Conda: a package manager disaster that became paid license required for companies over 200 employees. It worked 5 years ago, we can no longer legally use it

rbanffy 2y ago
It kind of limits its own blast radius that way.

These335 2y ago

Can somebody please eli5 why it is so unanimously accepted that Python's package management is terrible? For personal projects venv + requirements.txt has never caused problems for me. For work projects we use poetry because of an assumption that we would need something better but I remain unconvinced (nothing was causing a problem for that decision to be made).

gdiamos 2y ago
It's popular enough that it causes pain for a lot of people. Coming from C++, IMO, it is vastly better.
- dccsillag 2y ago
  Well, yes, but that's an extremely low bar!!
invaliduser 2y ago
venv + requirements.txt has worked for every single python project I made for the last 2 years (I'm new to python). Only issue I had was when using a newish python version and not having a specific library released yet for this new version, but downgrading python solved this. Being new to the ecosystem I have no clue why people would use Conda and why it matters. I tried it, but was left bewildered, not understanding the benefits.
- dagw 2y ago
  I have no clue why people would use Conda The big thing to realise is that when Conda first was released it was the only packaging solution that truly treated Windows as a first class citizen and for a long time was really the only way to easily install python packages on Windows. This got it a huge following in the scientific community where many people don't have a solid programming/computer background and generally still ran Windows on their desktops. Conda also not only manages your python interpreter and python libraries, it manages your entire dependency chain down to the C level in a cross platform way. If a python library is a wrapper around a C library then pip generally won't also install the C library, Conda (often) will. If you have two different projects that need two different versions of GDAL or one needs OpenBLAS and one that needs MKL, or two different versions of CUDA then Conda (attempts to) solve that in a way that transparently works on Windows, Linux and MacOS. Using venv + requirements.txt you're out of luck and will have to fall back on doing everything in its own docker container. Conda lets you mix private and public repos as well as mirroring public packages on-perm in a transparent way much smoother than pip, and has tools for things like audit logging, find grained access control, package signing and centralised controls and policy management. Conda also has support for managing multi-language projects. Does your python project need nodejs installed to build the front-end? Conda can also manage your nodejs install. Using R for some statistical analysis in some part of your data pipeline? Conda will mange your R install. Using a Java library for something? Conda will make sure everybody has the right version of Java installed. Also, it at least used to be common for people writing numeric and scientific libraries to release Conda packages first and then only eventually publish on PyPi once the library was 'done' (which could very well be never). So if you wanted the latest cutting edge packages in many fields you needed Conda. Now there are obviously a huge class a projects where none of these features are needed and mean nothing. If you don't need Conda, then Conda is no longer the best answer. But there are still a lot of niche things Conda still does better than any other tool.

japanuspus 2y ago

The problem conda solved that nothing had solved before was installing binary dependencies on MS Windows. Before conda, getting a usable scipy install up and running on MS Windows was a harrowing experience. And having two independent installations was basically impossible. The real hard work that went into conda was reverse engineering all the nooks and crannies of the DLL loading heuristics, to allow it to ensure that you loaded what you intended. If you are working on macOS and deploying to some *nix in the cloud, you are unlikely to find any value in this. But in ten years as lead on a large tool that was deployed to personal (Windows) laptops in a corporate environment, I did not find anything that beat conda.

dist-epoch 2y ago
This was mostly because most scientific packages didn't provide Windows binary builds for many years. Today you can just "pip install scipy" on Windows at it will just work.
- wruza 2y ago
  Oh right, recently I started learning classic ML and “just” tried to install tensorflow, which, itself or through one of dependencies, stopped providing windows binaries since x.y.z and so my python has to be downgraded to 3.a and then other dependencies stop installing. Eventually I managed to find a proper version intersection for everything together with some shady repo, but it felt like one more requirement and I’ll get overconstrained.
  - disgruntledphd2 2y ago
    Tensorflow is the worst. Basically every time my python env was borked (with multiple incompatible versions of Numpy) it was down to tensorflow.
    - coredog64 2y ago
      You said it. I was working with an official Google library that used TF and it didn’t work at all with 3.12. I spent a day building the wheels for 3.12 only to find there was a bug with dataclasses. :| I can’t recall the library, but there was another major project that just deprecated TF because it was the cause of so many build problems.

oivey 2y ago

Besides the horrendous formatting, some stuff in this article seem incorrect or irrelevant. Like, is this even possible? > A single Anaconda distribution may have multiple NumPy versions installed at the same time, although only one will be available to the Python process (note that this means that sub-processes created in this Python process won’t necessarily have the same version of NumPy!). I’m pretty sure there’s not, but maybe there is some insane way to cause subprocesses to do this. Besides that, under the authors definition, different Python virtualenvs also install multiple copies of libraries in the same way conda does. The comments about Jupyter also seem very confused. It’s hard to make heads or tails of exactly what the author is saying. There might be some misunderstandings of how Jupyter kernels select environments. > Final warning: no matter how ridiculous this is: the current directory in Python is added to the module lookup path, and it precedes every other lookup location. If, accidentally, you placed a numpy.py in the current directory of your Python process – that is going to be the numpy module you import. This has nothing to do with conda.

BiteCode_dev 2y ago

Conda used to be a life saver when years and years ago, compiled extensions were hard to install because you had to compile them yourself. Nowadays, thanks to wheels being numerous and robust, the appeal of anaconda is disappearing for most users except for some exotic mixes. conda itself now causes more trouble than it solves as it's slow, and lives in its own incompatible world. But anaconda solves a different problem now that nobody else solves, and that's managing Python for big corporation. This is worth a lot of money to big structures that need to control packages origin, permissions, updates, and so on, at scale. So it thrives there.

dannyz 2y ago

While `uv` works amazingly well I think a lot of people don't realize that installing packages through conda (or let's say the conda-forge ecosystem) has technical advantages compared to wheels/pypi. When you install the numpy wheel through `uv` you are likely installing a pre-compiled binary that bundles openblas inside of it. When you install numpy through conda-forge, it dynamically links against a dummy blas package that can be substituted for mkl, openblas, accelerate, whatever you prefer on your system. It's a much better solution to be able to rely on a separate package rather than having to bundle every dependency. Then lets say you install scipy. Scipy also has to bundle openblas in their wheel, and now you have two copies of openblas sitting around. They don't conflict, but this quickly becomes an odd thing to have to do.

KolenCh 2y ago
In this sense I personally prefer pixi because of this. It is pixi like but resolves using conda channels like conda, and similar to conda it supports PyPI packages via uv. With a background in scientific computing where many of the dependencies I managed are compiled, conda packages gives me much more control. P.S. I’d like to point out to others to differentiate between package index and package managers. PyPI is an index (that hosts packages in a predefined format) while pip, poetry, uv are package managers that resolve and build your environments using the index. Similarly but a bit more confusingly, conda can be understood as the index, hosted by anaconda but can also be hosted elsewhere, with different “channels” (kinda like a GitHub organization) where conda-forge is a popular one built by communities. Conda is also a reference implementation of a package manager that uses anaconda channels to resolve. Mamba is an independent, performant, drop in replacement of conda. And pixi is a different one with a different interface by the author of mamba. Even more confusingly, there are distributions. Distributions come with a set of predefined packages together with the package manager such that you just start running things immediately (sort of like a TeXLive distribution in relation to the package manager tlmgr.) there are anaconda distributions (if you installed anaconda instead of installing conda, that’s what you get), but also Intels distribution for Python, mini forge, mambaforge, etc.
fire_lake 2y ago
Is there a reason this behaviour couldn’t be implemented in uv? Is this beyond what the pyproject.toml spec supports?

teekert 2y ago

People here focus on Python, but to me, a bioinformatician, conda is much more, it provides 99.99% of the tools I need. Like bwa, samtools, rsem, salmon, fastqc, R. And many, many obscure tools.

GuestFAUniverse 2y ago
And then somebody tries to install mamba via conda and that house of cards reveals itself.
- teekert 2y ago
  I install Snakemake via miniforge which uses mambaforge to make its own envs. Biology is messy ;)
deleted 2y ago
[deleted]
mbreese 2y ago
I wish you luck with tracking down versions of software used when you're writing papers... especially if you're using multiple conda environments. This is pretty much the example used in the article -- version mismatches. But, I think this illustrates the problem very well. Conda isn't just used for Python. It's used for general tools and libraries that Python scripts depend on. They could be C/C++ that needs to be compiled. It could be a Cython library. It could be... When you're trying to be a package manager that operates on-top of the operating system's package manager, you're always going to have issues. And that is why Conda is such a mess, it's trying to do too much. Installation issues are one of the reason why I stopped writing so many projects in Python. For now, I'm only doing smaller scripts in Python. Anything larger than a module gets written in something else. People here have mentioned Rust as an example of a language with a solid dependency toolchain. I've used more Go, which similarly has had dependency management tooling from the begining. By and large, these languages aren't trying to bring in C libraries that need to be compiled and linked into Python accessible code (it's probably possible, but not the main use-case). For Python code though, when I do need to import a package, I always start with a fresh venv virtual environment, install whatever libraries are needed in that venv, and then always run the python from that absolute path (ex: `venv/bin/python3 script.py`). This has solved 99% of my dependency issues. If you can separate yourself from the system python as much as possible, you're 90% of the way there. Side rant: Which, is why I think there is a problem with Python to begin with -- *nix OSes all include a system level Python install. Dependencies only become a problem when you're installing libraries in a global path. If you can have separate dependency trees for individual projects, you're largely safe. It's not very storage efficient, but that's a different issue.

ur-whale 2y ago

I tried Conda a number of time over the years, regretted it every time. These days, when I absolutely have to use it because some obscure piece of software can't run unless Conda, I install it in a VM so that: - I protect my working system from the damage of installing Conda on it - I can throw the whole garbage fire away without long term brain damage to my system once I'm done

jujube3 2y ago

[flagged]

deleted 2y ago
[deleted]

prpl 2y ago

conda was for scientific python, but had to solve for everything below python to make that work. There was no generic binary solution before python for multiple architectures and operating systems.

benreesman 2y ago

I strongly suspect that there is about to be a spike in Python packaging discussion over and above the high ambient baseline. uv is here to kick ass and chew bubblegum. And it’s all out of gum.

nodivbyzero 2y ago

The Five Demons of Python Packaging That Fuel Our Persistent Nightmare: https://youtu.be/qA7NVwmx3gw?si=QbchrYvCEp8aazvL https://youtu.be/qA7NVwmx3gw?si=QbchrYvCEp8aazvL

whywhywhywhy 2y ago

Really the issue is python itself, it shouldn't be treating it's installs and packages as something that's linked and intertwined to the base operating system. People like to complain about node packages but never seen people have the trouble with them that they have with python.

macinjosh 2y ago
This is spot on. Running some python projects on nixos is a nightmare because of this model. Especially if it’s ML related.
theamk 2y ago
What do you do though if you want to import code written in C++? Especially complex, dependency-heavy like CUDA/ML stuff? You can just give up and say that "The proper way to do this is to use the Nvidia CUDA toolkit to write your cuda app in C++ and then invoke it as a separate process from node" [0]. That apparently works for node, but Python wants much more. If you actually want to use high-performance native code in your slow compiled language, then no solution is going to be very good, that's because the problem is inherently hard. You can rely on host OS as much as possible - if OS is known, provide binaries; if it's unknown, provide source code and hope user has C/C++/Rust/Fortran compilers to build it. That's what uv, pip, etc.. do. You can create your own parallel OS, bringing your own copy of every math libray, as well as CUDA even if there are perfectly good versions installed on the system - that's what conda/minconda does. You can implement as much as possible in your own language, so there is much less need to use "high-performance native language" - that's what Rust and Go do. Sadly, that's not an option for Python. [0] https://stackoverflow.com/questions/20875456/how-can-i-use-cuda-with-nodejs https://stackoverflow.com/questions/20875456/how-can-i-use-c...
- whywhywhywhy 2y ago
  >What do you do though if you want to import code written in C++? Especially complex, dependency-heavy like CUDA/ML stuff? No other CUDA software has to put it's claws deep into the os to function. I'm not even convinced Python does, seems it just dumps all the 500-900meg DLLs into the torch folder. It's more the insistence that the libraries have to exist in this place that isn't related to the project that's the issue. Node also handles what you're talking about without issue and without thinking it needs to be part of the operating system like Python does.

kristianp 2y ago

Are most people having problems with python packages using Windows? It's been mentioned a couple of times in this thread, but not that often.

sznio 2y ago

The domain got parked.

mdaniel 2y ago

I hope you read it while it was available, because the domain has expired Domain Name: pyherald.com Registry Domain ID: 2663190918_DOMAIN_COM-VRSN Registrar WHOIS Server: whois.namesilo.com Registrar URL: https://www.namesilo.com/ Updated Date: 2024-12-21T07:00:00Z Creation Date: 2021-12-21T07:00:00Z Registrar Registration Expiration Date: 2024-12-21T07:00:00Z https://web.archive.org/web/20241220211119/https://pyherald.com/articles/16_12_2024/ https://web.archive.org/web/20241220211119/https://pyherald.... is the most recent snap

btrettel 2y ago
Looks like the webpage is on GitHub too: https://github.com/pyherald/pyherald/blob/main/docs/articles/16_12_2024/index.html https://github.com/pyherald/pyherald/blob/main/docs/articles...
osdotsystem 2y ago
Back online