7 ms·

As someone with admittedly no formal CS education, I've been using conda for all of my grad school and never managed to break it. I create a virtual environmen

As someone with admittedly no formal CS education, I've been using conda for all of my grad school and never managed to break it. I create a virtual environment for every project. I install almost all packages with pip, except for any binaries or CUDA related things from conda. I always exported the conda yaml file and managed to reproduce the code/environment including the Python version. I've seen a lot of posts over time praising poetry and other tools and complaining about conda but I could never relate to any of them. Am i doing something wrong? Or something right?

jszymborski 2y ago
God forbid you should require conda-forge and more than three packages lest the dependency resolver take longer than the heat death of the planet to complete.
- fransje26 2y ago
  Install mamba first?
  - jszymborski 2y ago
    Mamba is indeed a lot better. I personally just don't bother with conda and stick to pip + venv.
thangngoc89 2y ago
1. You need to run export manual while other tools you mentioned would create it automatically (the lock file) 2. Distinguishes between direct dependencies (packages you added yourself) and indirect dependencies (packages of the packages)
theamk 2y ago
You are doing something right, author does some pretty unusual things: - Setup custom kernels in Jupyter Notebook - Hardlink the environments, then install same packages via pip in one and conda in others - install conda inside conda (!!!) and enter nested environment - Use tox within conda I believe as long as you treat the environments as "cattle" (if it goes bad, remove it and re-create from yaml file), you should not have any problems. It's clearly not the case of for the post's author though.
- fluorinerocket 2y ago
  Yep nuke the bad env and start over. Conda is great only problem are when a package is not available on conda forge or you have to compile and install with setup.py. But then you can blow the env away and start over.
throwawaymaths 2y ago
i think you got lucky and fell into best practices on your first go > except for any binaries or CUDA related things from conda doing the default thing with cuda related python packages used to often result in "fuck it, reinstall linux". admittedly, i dont know how it is now. i have one machine that runs python with a gpu and it runs only one python program.
- disgruntledphd2 2y ago
  > doing the default thing with cuda related python packages used to often result in "fuck it, reinstall linux" From about 2014-17 you are correct, but it appears (on ubuntu at least), that it mostly works now. Maybe I've just gotten better at dealing with the pain though...
duped 2y ago
My experience with conda is that its fine if you're the original author of whatever you're using it for and never share it with anyone else. But as a professional I usually have to pull in someone else's work and make it function on a completely different machine/environment. I've only had negative experiences with conda for that reason. IME the hard job of package management is not getting software to work in one location, but allowing that software to be moved somewhere else and used in the same way. Poetry solves that problem, conda doesn't. Poetry isn't perfect, but it's working in an imperfect universe and at least gets the basics (lockfiles) correct to where packages can be semi-reproducible. There's another rant to be had at the very existence of venvs as part of the solution, but that's neither poetry or anaconda's fault.
- throwawaymaths 2y ago
  imagine being a beginner to programming and being told "use venvs" or worse, imagine being a longtime user of shells but not python and then being presented a venv as a solution to the problem that for some reason python doesn't stash deps in a subdirectory of your project
  - wakawaka28 2y ago
    Beginners in Python typically don't need venvs. They can just install a few libraries (or no libraries even) to get started. If you truly need venvs then you're either past the initial learning phase or you're learning how to run Python apps instead of learning Python itself. For some libraries, it is not acceptable to stash the dependencies for every single toy app you use. I don't know how much space TensorFlow or PyQt use but I'm guessing most people don't want to install those in many venvs.
    - throwawaymaths 2y ago
      i remember reading somewhere (on twitter iirc) an amateur sex survey statistician who decided she needed to use python to analyze her dataset, being guided toward setting up venvs pretty early on by her programmer friends and getting extremely frustrated.
bean-weevil 2y ago
As someone with a formal computer science, half of my friends who work in other sciences have asked me to help them fix their broken conda environments
maurosilber 2y ago
I had the same experience. But you should try pixi, which is to conda what uv is to pip.
- akdor1154 2y ago
  Isn't uv to conda what uv is to pip?
  - LarsDu88 2y ago
    `uv` is not a drop-in replacement for `conda` in the sense that `conda` also handles non-python dependencies, has its own distinct api server for packages, and has its own packaging yaml standard. `pixi` basically covers `conda` while using the same solver as `uv` and is written in Rust like `uv`. Now is it a good idea to have python's package management tool handle non-python packages? I think that's debateable. I personally am in favor of a world where `uv` is simply the final python package management solution. Wrote an article on it here: https://dublog.net/blog/so-many-python-package-managers/ https://dublog.net/blog/so-many-python-package-managers/
    - Balinares 2y ago
      Bookmarking. Thanks for sharing the link, looks like a great overview of that particular tragic landscape. :) Also crossing fingers that uv ends up being the last one standing when the comprehensive amounts of dust here settle. But until then, I'll look into pixi, on the off chance it minimizes some of my workplace sorrows.
    - immmmmm 2y ago
      I have been using pixi for half a year and it has been fantastic. It’s fast, takes yml files as an input (which is super convenient) and super intuitive Quite surprised it isn’t more popular
    - traversaro 2y ago
      I am not sure pixi uses the same solver of uv, at least in general. pixi uses resolvo (https://github.com/mamba-org/resolvo https://github.com/mamba-org/resolvo) for conda packages, while uv (that in turns uses pubgrub https://github.com/pubgrub-rs/pubgrub https://github.com/pubgrub-rs/pubgrub) for pip packages.
fluorinerocket 2y ago
Same but I try to use conda to install everything first, and only use pip as a last resort. If pip only installs the package and no dependency it's fine
rcxdude 2y ago
This is exactly the kind of thing that causes python package nightmares. Pip is barely aware of packages it's installed itself, let alone packages from other package managers and especially other package repositories. Mixing conda and pip is 100% doing it wrong (not that there's an easy way to do it right, but stick to one or the other, I would generally recommend just using pip, the reasons for conda's existance are mostly irrelevant now)
- skeledrew 2y ago
  I still run into cases where a pip install that fails due to some compile issue works fine via conda. It's still very relevant. It's pip that should be switched out for something like poetry.
  - rcxdude 2y ago
    poetry vs pip does very little for compilation-related install failures. Most likely the difference is whether you are getting a binary package or not, and conda's repository may have a binary package that pypi does not (but also vice-versa: nowadays pypi has decent binary packages, previously conda gained a lot of popularity because it had them while pypi generally did not, especially on windows). But the main badness comes from mixing them in the same environment (or mixing pypi packages with linux distribution packages, i.e. pip installing outside of a virtual environment). (I do agree pip is still pretty lackluster, but the proposed replacements don't really get to the heart of the problem and seem to lack staying power. I'm in 'wait and see' mode on most of them)
    - skeledrew 2y ago
      Oh I meant that poetry could be a general replacement for pip (actually it used it in it's backend) because it does a great job managing dependencies and projects in general.
- whywhywhywhy 2y ago
  Works absolutely fine as possible with Python using conda to manage the environments and python versions and pip to install the packages.

jszymborski 2y ago

God forbid you should require conda-forge and more than three packages lest the dependency resolver take longer than the heat death of the planet to complete.

fransje26 2y ago
Install mamba first?
- jszymborski 2y ago
  Mamba is indeed a lot better. I personally just don't bother with conda and stick to pip + venv.

thangngoc89 2y ago

1. You need to run export manual while other tools you mentioned would create it automatically (the lock file) 2. Distinguishes between direct dependencies (packages you added yourself) and indirect dependencies (packages of the packages)

theamk 2y ago

You are doing something right, author does some pretty unusual things: - Setup custom kernels in Jupyter Notebook - Hardlink the environments, then install same packages via pip in one and conda in others - install conda inside conda (!!!) and enter nested environment - Use tox within conda I believe as long as you treat the environments as "cattle" (if it goes bad, remove it and re-create from yaml file), you should not have any problems. It's clearly not the case of for the post's author though.

fluorinerocket 2y ago
Yep nuke the bad env and start over. Conda is great only problem are when a package is not available on conda forge or you have to compile and install with setup.py. But then you can blow the env away and start over.

throwawaymaths 2y ago

i think you got lucky and fell into best practices on your first go > except for any binaries or CUDA related things from conda doing the default thing with cuda related python packages used to often result in "fuck it, reinstall linux". admittedly, i dont know how it is now. i have one machine that runs python with a gpu and it runs only one python program.

disgruntledphd2 2y ago
> doing the default thing with cuda related python packages used to often result in "fuck it, reinstall linux" From about 2014-17 you are correct, but it appears (on ubuntu at least), that it mostly works now. Maybe I've just gotten better at dealing with the pain though...

duped 2y ago

My experience with conda is that its fine if you're the original author of whatever you're using it for and never share it with anyone else. But as a professional I usually have to pull in someone else's work and make it function on a completely different machine/environment. I've only had negative experiences with conda for that reason. IME the hard job of package management is not getting software to work in one location, but allowing that software to be moved somewhere else and used in the same way. Poetry solves that problem, conda doesn't. Poetry isn't perfect, but it's working in an imperfect universe and at least gets the basics (lockfiles) correct to where packages can be semi-reproducible. There's another rant to be had at the very existence of venvs as part of the solution, but that's neither poetry or anaconda's fault.

throwawaymaths 2y ago
imagine being a beginner to programming and being told "use venvs" or worse, imagine being a longtime user of shells but not python and then being presented a venv as a solution to the problem that for some reason python doesn't stash deps in a subdirectory of your project
- wakawaka28 2y ago
  Beginners in Python typically don't need venvs. They can just install a few libraries (or no libraries even) to get started. If you truly need venvs then you're either past the initial learning phase or you're learning how to run Python apps instead of learning Python itself. For some libraries, it is not acceptable to stash the dependencies for every single toy app you use. I don't know how much space TensorFlow or PyQt use but I'm guessing most people don't want to install those in many venvs.
  - throwawaymaths 2y ago
    i remember reading somewhere (on twitter iirc) an amateur sex survey statistician who decided she needed to use python to analyze her dataset, being guided toward setting up venvs pretty early on by her programmer friends and getting extremely frustrated.

bean-weevil 2y ago

As someone with a formal computer science, half of my friends who work in other sciences have asked me to help them fix their broken conda environments

maurosilber 2y ago

I had the same experience. But you should try pixi, which is to conda what uv is to pip.

akdor1154 2y ago
Isn't uv to conda what uv is to pip?
- LarsDu88 2y ago
  `uv` is not a drop-in replacement for `conda` in the sense that `conda` also handles non-python dependencies, has its own distinct api server for packages, and has its own packaging yaml standard. `pixi` basically covers `conda` while using the same solver as `uv` and is written in Rust like `uv`. Now is it a good idea to have python's package management tool handle non-python packages? I think that's debateable. I personally am in favor of a world where `uv` is simply the final python package management solution. Wrote an article on it here: https://dublog.net/blog/so-many-python-package-managers/ https://dublog.net/blog/so-many-python-package-managers/
  - Balinares 2y ago
    Bookmarking. Thanks for sharing the link, looks like a great overview of that particular tragic landscape. :) Also crossing fingers that uv ends up being the last one standing when the comprehensive amounts of dust here settle. But until then, I'll look into pixi, on the off chance it minimizes some of my workplace sorrows.
  - immmmmm 2y ago
    I have been using pixi for half a year and it has been fantastic. It’s fast, takes yml files as an input (which is super convenient) and super intuitive Quite surprised it isn’t more popular
  - traversaro 2y ago
    I am not sure pixi uses the same solver of uv, at least in general. pixi uses resolvo (https://github.com/mamba-org/resolvo https://github.com/mamba-org/resolvo) for conda packages, while uv (that in turns uses pubgrub https://github.com/pubgrub-rs/pubgrub https://github.com/pubgrub-rs/pubgrub) for pip packages.

fluorinerocket 2y ago

Same but I try to use conda to install everything first, and only use pip as a last resort. If pip only installs the package and no dependency it's fine

rcxdude 2y ago

This is exactly the kind of thing that causes python package nightmares. Pip is barely aware of packages it's installed itself, let alone packages from other package managers and especially other package repositories. Mixing conda and pip is 100% doing it wrong (not that there's an easy way to do it right, but stick to one or the other, I would generally recommend just using pip, the reasons for conda's existance are mostly irrelevant now)

skeledrew 2y ago
I still run into cases where a pip install that fails due to some compile issue works fine via conda. It's still very relevant. It's pip that should be switched out for something like poetry.
- rcxdude 2y ago
  poetry vs pip does very little for compilation-related install failures. Most likely the difference is whether you are getting a binary package or not, and conda's repository may have a binary package that pypi does not (but also vice-versa: nowadays pypi has decent binary packages, previously conda gained a lot of popularity because it had them while pypi generally did not, especially on windows). But the main badness comes from mixing them in the same environment (or mixing pypi packages with linux distribution packages, i.e. pip installing outside of a virtual environment). (I do agree pip is still pretty lackluster, but the proposed replacements don't really get to the heart of the problem and seem to lack staying power. I'm in 'wait and see' mode on most of them)
  - skeledrew 2y ago
    Oh I meant that poetry could be a general replacement for pip (actually it used it in it's backend) because it does a great job managing dependencies and projects in general.
whywhywhywhy 2y ago
Works absolutely fine as possible with Python using conda to manage the environments and python versions and pip to install the packages.