9 ms·

Show HN: Wetlands – a lightweight Python library for managing Conda environments

30 points by arthursw 1y ago

When building a plugin system for an application, avoiding dependency conflicts is critical. To address this, I created Wetlands – a lightweight Conda environment manager. Wetlands not only simplifies the creation of isolated Conda environments with specific dependencies, but also allows you to run arbitrary Python code within those environments and retrieve the results. It uses the multiprocessing.connection and pickle modules for inter-process communication. Additionally, one can easily use shared memory between the environments, making data exchange more efficient. Docs: https://arthursw.github.io/wetlands/latest/ https://arthursw.github.io/wetlands/latest/ Source: https://github.com/arthursw/wetlands/ https://github.com/arthursw/wetlands/ I’d really appreciate any feedback. Thanks!

mushufasa 1y ago
> Wetlands not only simplifies the creation of isolated Conda environments with specific dependencies, but also allows you to run I've been using Conda for 10 years as my default package manager on my devices (not pipenv or poetry etc). I started because it was "the way" for data science but I kept with it because the syntax is really intuitive to me (conda create, conda activate). I'm not sure what problem you are solving here -- the issues with conda IMO are that it is overkill for the rest of the python community, so conda-forge has gradually declined and I typically create a conda environment then use pip for the latest libraries. Managing the conda environments though is not my issue -- that part works so well that I keep with it. If you could explain why you created this and what problems you are solving with an example, that would be helpful. All package managers are aimed at "avoiding dependency conflicts" so that doesn't really communicate to me what this is and what real problem it solves.
- N1H1L 1y ago
  I used to use conda, but have switched entirely to uv now
  - oezi 1y ago
    As someone new to Python: what was ever the appeal of conda that uv doesn't satisfy?
    - aeroevan 1y ago
      conda doesn't just package python libraries, but also the C/Fortran/other bits that the scipy stack often depended on. With the rise of binary wheels that is less needed though
    - kjkjadksj 1y ago
      Fundamentally it is a fresh usr/bin per environment with all that can go into that. Not just python tooling. R packages. Binaries. All of that. Env can be exported as a yaml file and trivially shared without appending some header to all scripts you write.
    - jaimergp 1y ago
      I think it's more about tool X vs Y, but about ecosystems and packaging approaches; in other words Python packaging (which has tools like pip, uv or poetry) vs conda packaging (which has tools like conda itself, mamba or pixi). https://pypackaging-native.github.io/ https://pypackaging-native.github.io/ is an excellent starting point to learn about the limitations on Python packaging for native dependencies and compiled extensions.
    - aldanor 1y ago
      Distributing non Python packages via the same channel that Python packages may depend on. E.g, h5py depending on libhdf5.
    - N1H1L 1y ago
      As someone using Python from 2017, there was no uv then, and conda worked fine but was glacially slow.
mrweasel 1y ago
> If the user doesn't have pixi or micromamba installed, Wetlands will download and set it up automatically. Please don't. Never have a tool that automatically reaches out onto the internet to get a binary and then run it. Just let the user know that they need to install either pixi or micromamba. It's inherently unsafe and you don't know what will be put into those binaries in the future. Maybe it's because I don't have a use case for this, but I don't really get what this is for. It's interesting, but I'm not really sure where I'd use it.
- deleted 1y ago
  [deleted]
barapa 1y ago
Why do people use Conda instead of uv?
- martinky24 1y ago
  Because conda has been around for years and uv is more or less brand new. It's pretty much as simple as that.
- ElectricalUnion 1y ago
  The conda ecosystem was a early adopter of standardized binary packages. Now it's mostly behind us, but there used to be a time where pypi didn't have wheels (a 2012 thing), or manylinux wheels (a 2016 thing) for most libraries. pip install was a world of pain if you didn't have the "correct source packages" in your system. And now several of those projects built back then, they're no longer projects but deployed systems, might as well stick to what is working.
  - krapht 1y ago
    Conda is still the recommended way to work with Intel's Python distribution, so there's a reason for it to live on my work computer. The fastest graph library I know, graph-tool, needs to be installed with Conda too.
    - zzzeek 1y ago
      that appears to be because the maintainers have chosen not to release any files https://pypi.org/project/graph-tool/ https://pypi.org/project/graph-tool/ that's not a limitation of pip, that's a limitation of the maintainers of graph-tool
      - krapht 1y ago
        I agree, and yet here we are. If it wasn't built with automake I would even consider getting wheels built on the projects behalf, but I can't make heads or tails of M4. EDIT: But let's not pretend getting cibuildwheel going for every supported variant of PythonMajVer/OS/(32/64) bit is an easy lift for small open source projects. Conda is still more ergonomic than this, although it's slowly dying because big projects have the manpower to stand up and maintain the CI pipeline for PyPI, so Conda is less and less necessary.
jpecar 1y ago
Why not just call it swamp? It would better describe the python ecosystem mess ;) Jokes aside, this feels very meta: package manager for a package manager for a package manager. Reminds me of the old RFC1925: "you can always add another layer of abstraction". That RFC also says "perfection has been reached not when there is nothing left to add, but when there is nothing left to take away". And as a hpc admin, I'm not offering my users any help with conda and let them suffer on their own. Instead I'm showing them the greener pastures with spack, easybuild and eessi whenever I can. And they're slowly migrating over.
- vindex10 1y ago
  it was also my first thought on abstractions of abstractions, thanks for sharing :) could you elaborate a bit more on why HPC world is special when it comes to configuring the environment? I always feel it is a typical problem in software development, to separate operating system env from the application env. do you use spack / easybuild on your personal computer, for example if you need to install a package that is not part of the distribution?
  - tgamblin 1y ago
    I do for macOS and Linux :). Windows support is also coming along. There isn’t anything particularly special about the HPC world other than the need for many different configurations of the same software for many different CPU and GPU architectures. You might want to have several versions of the same application installed at once, with different options or optimizations enabled. Spack enables that. Environments are one way to keep the different software stacks separate (though, like nix, spack uses a store model to keep installs in separate directories, as well).
  - deleted 1y ago
    [deleted]
- arthursw 1y ago
  When I started the project, Spack was not available on Windows so it was not an option. Now I should reconsider this. Thanks!
superkuh 1y ago
A python dependency manager manager manager. Truly we live in an age of unprecedented code abstraction and complexity. And I love that you install and manage wetlands itself with pip. An ouroboros matrioska of code.
chillpenguin 1y ago
The fact that this exists shows that there is a serious problem in the python ecosystem. I'm sure it solves a real problem, so I'm not knocking the author. It's more of a "state of our industry" problem.
- whalesalad 1y ago
  I maintain there is no issue. It's really not hard. conda is a smell for me though. pyenv is all you need. it manages python versions and python virtual environments. you can create and destroy them just as easily as git branches. pyenv + good ol' requirements.txt is really all you need. if your env dictates containers, it's even easier to work with. FROM python:version and done.
  - tuckerman 1y ago
    The issue is you think pyenv has solved everything, someone else thinks poetry solves everything, I think uv solves everything, and someone else is apt installing things. And then there is installing torch and cuda... I think having a very widely accepted and supported default would let the community rally around it and improve the experience and documentation (though I am not a fan of the centralized hosting that npm and cargo push, I much prefer the go approach, but we are already there with PyPI)
    - whalesalad 1y ago
      uv + poetry are a higher level in the stack than something like pyenv. pip, uv, poetry are all analagous. they ensure the correct packages are installed. we have some internal apps that devs decided to start with poetry and it has some nice ergonomics ... but on the other hand I find the simplicity of a requirements file to be so ... simple. People get caught up on the file, too, but really its just a convention. you can call it whatever you want, deps.txt, deps.foo, packages.bar ... its just a plaintext file with newline delimited packages. since its "just a file" and "just unix" this has the added perk of being able to cat many files together and use that with pip. it's all just lines of text, pip does not care. pyenv + pip works. pyenv + poetry works. pyenv + uv works. those become inconsequential decisions you can make on a case by case basis and try or forget as needed.
arthursw 1y ago
Thanks for your comments! I fully agree that the Python ecosystem is (overly) complex, and Conda is generally not necessary for Python tools. I made this library for a workflow management system, which can use any tool packaged with Conda, not just Python tools. The tools can be binaries made in C++, Java programs, or anything Conda can containerize. Note that Docker is not an option because it cannot be installed automatically on all platforms (and because of performances on non-Linux OS). My users do not have to worry about command lines to install tools since Wetlands is installed in the workflow management system. Each tool is installed when the user executes a workflow using it. In the bio-image analysis and medical imaging communities —as well as many others— scientists are often unfamiliar with the Python ecosystem and the concept of virtual environments. However, they rely heavily on a wide range of tools, each with numerous dependencies written in various languages. Applications with a built-in package management system like Wetlands greatly simplify their workflow by handling the complex task of setting up environments for these tools behind the scenes. For example, Napari is an excellent viewer for multi-dimensional images written in Python which can be easily extended via plugins. There are hundreds of plugins, to do things like image denoising, registration, segmentation, particle tracking, etc. Plugins depend on tools (like Segment-Anything-Model, Cellpose, Stardist, etc.) which cannot be installed in the same environment. Wetlands can come to the rescue and isolate each plugin in its own environment. I hope the purpose of Wetlands is clearer now :)

30 points by arthursw 1y ago

mushufasa 1y ago

> Wetlands not only simplifies the creation of isolated Conda environments with specific dependencies, but also allows you to run I've been using Conda for 10 years as my default package manager on my devices (not pipenv or poetry etc). I started because it was "the way" for data science but I kept with it because the syntax is really intuitive to me (conda create, conda activate). I'm not sure what problem you are solving here -- the issues with conda IMO are that it is overkill for the rest of the python community, so conda-forge has gradually declined and I typically create a conda environment then use pip for the latest libraries. Managing the conda environments though is not my issue -- that part works so well that I keep with it. If you could explain why you created this and what problems you are solving with an example, that would be helpful. All package managers are aimed at "avoiding dependency conflicts" so that doesn't really communicate to me what this is and what real problem it solves.

N1H1L 1y ago
I used to use conda, but have switched entirely to uv now
- oezi 1y ago
  As someone new to Python: what was ever the appeal of conda that uv doesn't satisfy?
  - aeroevan 1y ago
    conda doesn't just package python libraries, but also the C/Fortran/other bits that the scipy stack often depended on. With the rise of binary wheels that is less needed though
  - kjkjadksj 1y ago
    Fundamentally it is a fresh usr/bin per environment with all that can go into that. Not just python tooling. R packages. Binaries. All of that. Env can be exported as a yaml file and trivially shared without appending some header to all scripts you write.
  - jaimergp 1y ago
    I think it's more about tool X vs Y, but about ecosystems and packaging approaches; in other words Python packaging (which has tools like pip, uv or poetry) vs conda packaging (which has tools like conda itself, mamba or pixi). https://pypackaging-native.github.io/ https://pypackaging-native.github.io/ is an excellent starting point to learn about the limitations on Python packaging for native dependencies and compiled extensions.
  - aldanor 1y ago
    Distributing non Python packages via the same channel that Python packages may depend on. E.g, h5py depending on libhdf5.
  - N1H1L 1y ago
    As someone using Python from 2017, there was no uv then, and conda worked fine but was glacially slow.

mrweasel 1y ago

> If the user doesn't have pixi or micromamba installed, Wetlands will download and set it up automatically. Please don't. Never have a tool that automatically reaches out onto the internet to get a binary and then run it. Just let the user know that they need to install either pixi or micromamba. It's inherently unsafe and you don't know what will be put into those binaries in the future. Maybe it's because I don't have a use case for this, but I don't really get what this is for. It's interesting, but I'm not really sure where I'd use it.

deleted 1y ago
[deleted]

barapa 1y ago

Why do people use Conda instead of uv?

martinky24 1y ago
Because conda has been around for years and uv is more or less brand new. It's pretty much as simple as that.
ElectricalUnion 1y ago
The conda ecosystem was a early adopter of standardized binary packages. Now it's mostly behind us, but there used to be a time where pypi didn't have wheels (a 2012 thing), or manylinux wheels (a 2016 thing) for most libraries. pip install was a world of pain if you didn't have the "correct source packages" in your system. And now several of those projects built back then, they're no longer projects but deployed systems, might as well stick to what is working.
- krapht 1y ago
  Conda is still the recommended way to work with Intel's Python distribution, so there's a reason for it to live on my work computer. The fastest graph library I know, graph-tool, needs to be installed with Conda too.
  - zzzeek 1y ago
    that appears to be because the maintainers have chosen not to release any files https://pypi.org/project/graph-tool/ https://pypi.org/project/graph-tool/ that's not a limitation of pip, that's a limitation of the maintainers of graph-tool
    - krapht 1y ago
      I agree, and yet here we are. If it wasn't built with automake I would even consider getting wheels built on the projects behalf, but I can't make heads or tails of M4. EDIT: But let's not pretend getting cibuildwheel going for every supported variant of PythonMajVer/OS/(32/64) bit is an easy lift for small open source projects. Conda is still more ergonomic than this, although it's slowly dying because big projects have the manpower to stand up and maintain the CI pipeline for PyPI, so Conda is less and less necessary.

jpecar 1y ago

Why not just call it swamp? It would better describe the python ecosystem mess ;) Jokes aside, this feels very meta: package manager for a package manager for a package manager. Reminds me of the old RFC1925: "you can always add another layer of abstraction". That RFC also says "perfection has been reached not when there is nothing left to add, but when there is nothing left to take away". And as a hpc admin, I'm not offering my users any help with conda and let them suffer on their own. Instead I'm showing them the greener pastures with spack, easybuild and eessi whenever I can. And they're slowly migrating over.

vindex10 1y ago
it was also my first thought on abstractions of abstractions, thanks for sharing :) could you elaborate a bit more on why HPC world is special when it comes to configuring the environment? I always feel it is a typical problem in software development, to separate operating system env from the application env. do you use spack / easybuild on your personal computer, for example if you need to install a package that is not part of the distribution?
- tgamblin 1y ago
  I do for macOS and Linux :). Windows support is also coming along. There isn’t anything particularly special about the HPC world other than the need for many different configurations of the same software for many different CPU and GPU architectures. You might want to have several versions of the same application installed at once, with different options or optimizations enabled. Spack enables that. Environments are one way to keep the different software stacks separate (though, like nix, spack uses a store model to keep installs in separate directories, as well).
- deleted 1y ago
  [deleted]
arthursw 1y ago
When I started the project, Spack was not available on Windows so it was not an option. Now I should reconsider this. Thanks!

superkuh 1y ago

A python dependency manager manager manager. Truly we live in an age of unprecedented code abstraction and complexity. And I love that you install and manage wetlands itself with pip. An ouroboros matrioska of code.

chillpenguin 1y ago

The fact that this exists shows that there is a serious problem in the python ecosystem. I'm sure it solves a real problem, so I'm not knocking the author. It's more of a "state of our industry" problem.

whalesalad 1y ago
I maintain there is no issue. It's really not hard. conda is a smell for me though. pyenv is all you need. it manages python versions and python virtual environments. you can create and destroy them just as easily as git branches. pyenv + good ol' requirements.txt is really all you need. if your env dictates containers, it's even easier to work with. FROM python:version and done.
- tuckerman 1y ago
  The issue is you think pyenv has solved everything, someone else thinks poetry solves everything, I think uv solves everything, and someone else is apt installing things. And then there is installing torch and cuda... I think having a very widely accepted and supported default would let the community rally around it and improve the experience and documentation (though I am not a fan of the centralized hosting that npm and cargo push, I much prefer the go approach, but we are already there with PyPI)
  - whalesalad 1y ago
    uv + poetry are a higher level in the stack than something like pyenv. pip, uv, poetry are all analagous. they ensure the correct packages are installed. we have some internal apps that devs decided to start with poetry and it has some nice ergonomics ... but on the other hand I find the simplicity of a requirements file to be so ... simple. People get caught up on the file, too, but really its just a convention. you can call it whatever you want, deps.txt, deps.foo, packages.bar ... its just a plaintext file with newline delimited packages. since its "just a file" and "just unix" this has the added perk of being able to cat many files together and use that with pip. it's all just lines of text, pip does not care. pyenv + pip works. pyenv + poetry works. pyenv + uv works. those become inconsequential decisions you can make on a case by case basis and try or forget as needed.

arthursw 1y ago

Thanks for your comments! I fully agree that the Python ecosystem is (overly) complex, and Conda is generally not necessary for Python tools. I made this library for a workflow management system, which can use any tool packaged with Conda, not just Python tools. The tools can be binaries made in C++, Java programs, or anything Conda can containerize. Note that Docker is not an option because it cannot be installed automatically on all platforms (and because of performances on non-Linux OS). My users do not have to worry about command lines to install tools since Wetlands is installed in the workflow management system. Each tool is installed when the user executes a workflow using it. In the bio-image analysis and medical imaging communities —as well as many others— scientists are often unfamiliar with the Python ecosystem and the concept of virtual environments. However, they rely heavily on a wide range of tools, each with numerous dependencies written in various languages. Applications with a built-in package management system like Wetlands greatly simplify their workflow by handling the complex task of setting up environments for these tools behind the scenes. For example, Napari is an excellent viewer for multi-dimensional images written in Python which can be easily extended via plugins. There are hundreds of plugins, to do things like image denoising, registration, segmentation, particle tracking, etc. Plugins depend on tools (like Segment-Anything-Model, Cellpose, Stardist, etc.) which cannot be installed in the same environment. Wetlands can come to the rescue and isolate each plugin in its own environment. I hope the purpose of Wetlands is clearer now :)