8 ms·

Almost 2 decades of working with python. I create a venv. Pip install and keep my direct deps in requirements.txt That's it. Never understood all these python

Almost 2 decades of working with python. I create a venv. Pip install and keep my direct deps in requirements.txt That's it. Never understood all these python dependency management problems dramas. Recently, I started using pyproject.toml as well which makes the whole thing more compact. I make lots of python packages too. Either I go setup.py or sometimes I like to use flit for no specific reason. I haven't ever felt the need for something like uv. I'm good with pip.

rcxdude 2y ago
This is more or less my experience, but I think in part it took a while for pip to actually get into a usable position, hence some of the proliferation of other options.
mrbungie 2y ago
This. And also try always to fix the version of the requirements, and that's it. Never had a problem making reproducible builds doing so.
- Fethbita 2y ago
  I had issues with exactly this method. One of my dependencies was pulled off to a paid model so my project no longer worked.
  - mrbungie 2y ago
    Anyone remember the leftpad fiasco in the node ecosystem? That could happen in any dependency system that allows owners to unpublish dependencies and that's one risk users must weigh when adding them.
- Alir3z4 2y ago
  Yeah, I assume pinning the version is something everyone does? Or probably many just don't and will have those "python deps management is a mess drama". TBH, I've seen tutorials or even some companies simply do `pip freeze > requirements.txt` :shrug: which is a mess.
Borealid 2y ago
That doesn't work well (enough) if you have one project that requires Python <3.10 and another that requires Python >=3.10. To really pin everything you'd need to use something like asdf, on top of poetry or a manual virtualenv. Otherwise you get your colleagues complaining that pip install failed with mysterious errors.
- marky1991 2y ago
  I don't get it, you ought to be building a different venv per project anyway. (Of course, I don't distribute most of my projects, so I just dump them all in the global install and don't worry about it)
  - redserk 2y ago
    Some libraries break across different versions of Python for a variety of reasons. Pinning python version with asdf (in conjunction with a venv) gets you reasonably far in ensuring a certain project works across a lot of people in a team.
    - nyrikki 2y ago
      Venvs solve that problem, they don't cause it. You activate the venv per project to get out of the global namespace problem. As python x.nn versions are major with the nn part, it is like trying to mix c11 with c19.... things will break if you don't respect that.
      - Borealid 2y ago
        A venv does not actually install a different Python interpreter. It's bound to the Python version that created it. You cannot make a Python 2.7 venv using a Python 3 interpreter. You need Python 3.10 to create a Python 3.10 venv. There are plenty of situations where the Python interpreter version matters. As a non-exhaustive list, you have libraries that compile code, non-Python languages that link to cpython, build scripts that do different things depending on wheel/setup/other-bundled-stuff, Python code that uses removed compat shims like importlib-metadata... If you haven't run into one of those situations yet, congratulations. I've been through this already, and making a reproducible environment does require first installing a pinned version of the Python interpreter and THEN setting up a venv using that particular interpreter.
atoav 2y ago
Then you deploy to an old debian and everything falls apart.
- Alir3z4 2y ago
  Not really. `pyproject.toml` let's you set the min python version. If not met, it won't install. Regardless, majority of the times, deployment is done via Docker.
  - atoav 2y ago
    I was reacting to a comment that said a dependencies.txt and a venv was enough, so in the model I critisized there is no pyproject.toml > Regardless, majority of the times, deployment is done via Docker. What, generally? In your peer circle? I'd say [citation needed] — docker has the problematic habit of inducing more moveable parts that can bite you, so I know many who — if given the choice — would rather deploy python projects without it. Having deployed many python application on actual bare metal and VMs alike I'd say the ratio of Docker VS just Python is more like 1:8.
thingification 2y ago
That might be fine in your context. People's problems are real, though. What they're almost always missing is separating the source code from the compiled output ("lock files"). Pick a tool to help with that, commit both files to your ("one's") project, problem solved. People end up committing either one or the other, not both, but: - You need the source code, else your project is hard to update ("why did they pick these versions exactly?" - the answer is the source code). - You need the compiled pinned versions in the lock file, else if dependencies are complicated or fast-moving or a project goes unmaintained, installing it becomes a huge mindless boring timesink (hello machine learning, all three counts). Whenever I see people complaining about python dependencies, most of the time it seems just that somebody lacked this concept, or didn't know how to do it with python, or are put off by too many choices? That plus that ML projects are moving quickly and may have heavy "system" dependencies (CUDA).
- thingification 2y ago
  To be more concrete: In the source code - e.g. requirements.in (in the case of pip-tools or uv's clone of that: uv pip compile + uv pip sync), one lists the names of the projects one's application depends on, with a few version constraints explained with comments (`someproject <= 5.3 # right now spamalyzer doesn't seem to work with 5.4`). In the compiled output - i.e. the lock files (pip-tools or uv pip sync/compile use requirements.txt for this) one makes sure every version is pinned to one specific version, to form a set of versions that work together. A tool (like uv pip compile) will generate the lock files from the source code, picking versions that are declared (in PyPI metadata) should work together. My advice: pip-tools (pip-compile + pip-sync) does this very nicely - even better, uv's clone of pip-tools (uv pip compile + uv pip sync), which runs faster. Goes nicely with: - pyproject.toml (project config / metadata) - plain old setuptools (works fine, doesn't change: great) - requirements.in: the source for pip-tools (that's all pip-tools does: great! uv has a faster clone) - pyenv to install python versions for you (that's all it does: great! again uv has a faster clone) - virtualenv to make separate sandboxed sets of installed python libraries (that's all it does: great! again uv has a faster clone) - maybe a few tiny bash scripts, maybe a Makefile or similar just as a way to list out some canned commands - actually write down the commands you run in your README PS: the point of `uv pip sync` over `uv pip install -r requirements.txt` is that the former will uninstall packages that aren't explicitly listed in requirements.txt. uv also has a poetry-like do-everything 'managed' everything-is-glued-together framework (OK you can see my bias). Personally I don't understand the benefits of that over its nice re-implementations of existing unix-y tools, except I guess for popularizing python lockfiles - but can't we just market the idea "lock your versions"? The idea is the good part!
gkhartman 2y ago
That's been my experience too. The main complaint I hear about this workflow is that venvs can't be moved without breaking. I just rebuild my venv in each new new location, but that rebuild time can add up for projects with many large scientific packages. Uv solved that pain point for me, since it provides a "pip install" implementation that runs in a fraction of the time.

rcxdude 2y ago

This is more or less my experience, but I think in part it took a while for pip to actually get into a usable position, hence some of the proliferation of other options.

mrbungie 2y ago

This. And also try always to fix the version of the requirements, and that's it. Never had a problem making reproducible builds doing so.

Fethbita 2y ago
I had issues with exactly this method. One of my dependencies was pulled off to a paid model so my project no longer worked.
- mrbungie 2y ago
  Anyone remember the leftpad fiasco in the node ecosystem? That could happen in any dependency system that allows owners to unpublish dependencies and that's one risk users must weigh when adding them.
Alir3z4 2y ago
Yeah, I assume pinning the version is something everyone does? Or probably many just don't and will have those "python deps management is a mess drama". TBH, I've seen tutorials or even some companies simply do `pip freeze > requirements.txt` :shrug: which is a mess.

Borealid 2y ago

That doesn't work well (enough) if you have one project that requires Python <3.10 and another that requires Python >=3.10. To really pin everything you'd need to use something like asdf, on top of poetry or a manual virtualenv. Otherwise you get your colleagues complaining that pip install failed with mysterious errors.

marky1991 2y ago
I don't get it, you ought to be building a different venv per project anyway. (Of course, I don't distribute most of my projects, so I just dump them all in the global install and don't worry about it)
- redserk 2y ago
  Some libraries break across different versions of Python for a variety of reasons. Pinning python version with asdf (in conjunction with a venv) gets you reasonably far in ensuring a certain project works across a lot of people in a team.
  - nyrikki 2y ago
    Venvs solve that problem, they don't cause it. You activate the venv per project to get out of the global namespace problem. As python x.nn versions are major with the nn part, it is like trying to mix c11 with c19.... things will break if you don't respect that.
    - Borealid 2y ago
      A venv does not actually install a different Python interpreter. It's bound to the Python version that created it. You cannot make a Python 2.7 venv using a Python 3 interpreter. You need Python 3.10 to create a Python 3.10 venv. There are plenty of situations where the Python interpreter version matters. As a non-exhaustive list, you have libraries that compile code, non-Python languages that link to cpython, build scripts that do different things depending on wheel/setup/other-bundled-stuff, Python code that uses removed compat shims like importlib-metadata... If you haven't run into one of those situations yet, congratulations. I've been through this already, and making a reproducible environment does require first installing a pinned version of the Python interpreter and THEN setting up a venv using that particular interpreter.

atoav 2y ago

Then you deploy to an old debian and everything falls apart.

Alir3z4 2y ago
Not really. `pyproject.toml` let's you set the min python version. If not met, it won't install. Regardless, majority of the times, deployment is done via Docker.
- atoav 2y ago
  I was reacting to a comment that said a dependencies.txt and a venv was enough, so in the model I critisized there is no pyproject.toml > Regardless, majority of the times, deployment is done via Docker. What, generally? In your peer circle? I'd say [citation needed] — docker has the problematic habit of inducing more moveable parts that can bite you, so I know many who — if given the choice — would rather deploy python projects without it. Having deployed many python application on actual bare metal and VMs alike I'd say the ratio of Docker VS just Python is more like 1:8.

thingification 2y ago

That might be fine in your context. People's problems are real, though. What they're almost always missing is separating the source code from the compiled output ("lock files"). Pick a tool to help with that, commit both files to your ("one's") project, problem solved. People end up committing either one or the other, not both, but: - You need the source code, else your project is hard to update ("why did they pick these versions exactly?" - the answer is the source code). - You need the compiled pinned versions in the lock file, else if dependencies are complicated or fast-moving or a project goes unmaintained, installing it becomes a huge mindless boring timesink (hello machine learning, all three counts). Whenever I see people complaining about python dependencies, most of the time it seems just that somebody lacked this concept, or didn't know how to do it with python, or are put off by too many choices? That plus that ML projects are moving quickly and may have heavy "system" dependencies (CUDA).

thingification 2y ago
To be more concrete: In the source code - e.g. requirements.in (in the case of pip-tools or uv's clone of that: uv pip compile + uv pip sync), one lists the names of the projects one's application depends on, with a few version constraints explained with comments (`someproject <= 5.3 # right now spamalyzer doesn't seem to work with 5.4`). In the compiled output - i.e. the lock files (pip-tools or uv pip sync/compile use requirements.txt for this) one makes sure every version is pinned to one specific version, to form a set of versions that work together. A tool (like uv pip compile) will generate the lock files from the source code, picking versions that are declared (in PyPI metadata) should work together. My advice: pip-tools (pip-compile + pip-sync) does this very nicely - even better, uv's clone of pip-tools (uv pip compile + uv pip sync), which runs faster. Goes nicely with: - pyproject.toml (project config / metadata) - plain old setuptools (works fine, doesn't change: great) - requirements.in: the source for pip-tools (that's all pip-tools does: great! uv has a faster clone) - pyenv to install python versions for you (that's all it does: great! again uv has a faster clone) - virtualenv to make separate sandboxed sets of installed python libraries (that's all it does: great! again uv has a faster clone) - maybe a few tiny bash scripts, maybe a Makefile or similar just as a way to list out some canned commands - actually write down the commands you run in your README PS: the point of `uv pip sync` over `uv pip install -r requirements.txt` is that the former will uninstall packages that aren't explicitly listed in requirements.txt. uv also has a poetry-like do-everything 'managed' everything-is-glued-together framework (OK you can see my bias). Personally I don't understand the benefits of that over its nice re-implementations of existing unix-y tools, except I guess for popularizing python lockfiles - but can't we just market the idea "lock your versions"? The idea is the good part!

gkhartman 2y ago

That's been my experience too. The main complaint I hear about this workflow is that venvs can't be moved without breaking. I just rebuild my venv in each new new location, but that rebuild time can add up for projects with many large scientific packages. Uv solved that pain point for me, since it provides a "pip install" implementation that runs in a fraction of the time.