8 ms·

I'm surprised at the number of people here complaining about venvs in Python. There are lots of warts when it comes to package management in Python, but the bui

I'm surprised at the number of people here complaining about venvs in Python. There are lots of warts when it comes to package management in Python, but the built-in venv support has been rock solid in Python 3 for a long time now. Most of the complaints here ironically are from people using a bunch of tooling in lieu of, or as a replacement for vanilla python venvs and then hitting issues associated with those tools. We've been using vanilla python venvs across our company for many years now, and in all our CI/CD pipelines and have had zero issues on the venv side of things. And this is while using libraries like numpy, scipy, torch/torchvision, etc.

Jackevansevo 4y ago
Likewise, I think people have a negative first experience because it doesn't work exactly like node, throw their toys out the pram and complain on HN for the rest of time. Guess in taking this stance we're both part of the problem... \s
aflag 4y ago
It's incredibly lacking in features. PyPI doesn't even properly index packages, making pip go into this dependency resolution he'll trying to find a set of versions that will work for you. It works for simple cases with few dependencies/not a lot of pinning. But if your needs are a bit more complex it certainly shows its rough edges. I actually find it amazing that they python community puts up with that. But I suppose fixing it is not that pressing now the language is widely adopted. It's not going to be anyone's priority to mess with that. It's high risk low rewards sort of project.
- Groxx 4y ago
  What does that have to do with venvs? I agree the packaging and distribution setup in python is an absolute mess, but that's entirely unrelated to venvs. It's like bringing up how python uses whitespace instead of curly-braces.
  - oneepic 4y ago
    I think the GP comment might have caused some confusion since it mentioned both package management and venvs very close together.
  - deleted 4y ago
    [deleted]
  - lmm 4y ago
    venvs are the recommended workaround for the fact that python packaging and distribution is a mess of global state. Lanugages with working packaging and distribution don't generally bother with anything venv-like.
    - Groxx 4y ago
      Sure, but that's like 99% pip. Venvs are patching it (quite effectively), not causing it.
- whalesalad 4y ago
  I've been writing Python for a looong time. I have pushed out thousands and thousands of deployments across probably 40+ distinct Python codebases and only once or twice have I ever encountered a showstopper dependency resolution issue. At the end of the day you should want to have fine grained control over your deps and frankly there are many times where a decision cannot be automatically made by a package manager. Pip gets beat on hard but it puts in work all day every day and rarely skips a beat. It's entirely free and developed with open source contributions. Areas where I have felt a lot of pain is with legacy Ruby projects/bundler. Don't get me started on golang. Can pip be made better? Sure. Should we have an attitude of disgust towards it? Heck no!
  - rxhernandez 4y ago
    > only once or twice have I ever encountered a showstopper dependency resolution issue I've encountered them with other languages and they're the sort of thing where one time is more than enough to make me feel like it could get me fired; they're Never (with a capital N) okay imo
  - crabbone 4y ago
    > once or twice have I ever encountered a showstopper dependency resolution issue. Hahaha... (rolls on the floor) Do you want to know why that is? No seriously? I'm not laughing at you as much as I'm laughing at Python now, but hey, well, anyways, do you want to know why that happened to you? I know you don't. But I'll tell you anyways! Until quite recently, pip didn't give a rat's ass if the dependencies it installed were consistent. It would blink a message in the long stream of vomit it spills on the screen saying something like "you have package X installed of version Y, but package Z wants X of version Q, which will not be installed". And happily streamed more garbage to your screen. It was an issue that was filed against pip for something like 12 years until it got resolved about a year or so ago. Even after it got resolved a lot of people tried to upgrade, saw that that would "break" their deployment, and rolled back to the latest broken version. Things are sort of improving gradually since then, but we are light years away from the system working properly, and I know you don't want to know why, but I'll tell you anyways! So, when for whatever reason pip doesn't find a dependency it thinks you need, a lot of packages, when they roll out their "releases", they upload also what Python calls "source release". Which should have never been treated as an installation option, but it is, and is treated like that by default. So, what will happen once pip finally gives up on finding a match, right, you guessed it! -- It's going to try to build it! Installing build dependencies along the way. What you get in the end is anyone's guess, but most likely, it's something broken because the developers who made this release didn't make a release specifically for your version. Don't despair. There's a flag you can use with pip install that should prevent it from trying to build stuff. But two bad things will happen to you if you use it: in any non-trivial project your dependencies will irreparably break. And, who knows if that flag is implemented correctly... nobody in the real world is using that. So, who knows, maybe it'll format your hard drive along the way.
- crabbone 4y ago
  I hate PyPI probably even more than you do, but venv doesn't do that. All it does is write a handful of files and make a bunch of symlinks. It doesn't deal with installation of packages.
  - aflag 4y ago
    Ok. Fair. Venvs are great, unless you want to install packages on them.
winrid 4y ago
Because even with --copy it creates all kinds of symlinks, and if you're using pyenv, hard coded paths to the python binary which can break from CI to installation. If you're using docker then it's a lot easier I guess.
- mikepurvis 4y ago
  It also quietly reuses the stdlib of whatever python you start from. Which mostly doesn’t matter in real world usage, but can be quite surprising if you ever get into your head the idea that that venv is portable.
9dev 4y ago
Oh, yeah? It’s working great? Like figuring out which packages your application actually uses? Or having separate development and production dependencies? Upgrading outdated libraries? Having taken a deep-dive into refactoring a large python app, I can confidently say that package management in python is a pain compared to other interpreted languages.
- rowanseymour 4y ago
  Virtual environments aren't package management. For example we use Poetry for package management - it supports separate dev and prod dependencies, upgrading etc. It generates a virtual environment.
  - 9dev 4y ago
    The distinction feels entirely academic to me. Managing packages means having a sane way to define the dependencies of software projects, so they can be checked into version control and be installed reproducibly later and/or elsewhere. I don’t know which problem python intended to solve by separating the two, but it doesn’t occur often in contemporary software engineering work. Having said that, the point you make is valid and Poetry is a good option, but it feels so maddening having to learn about like seven different tools which all do more or less the same but not quite, and everyone and their mother having an opinion on which is the best. Doesn’t help that there’s an arbitrary yet blurry line where package managers end and environment managers begin.
- peterhil 4y ago
  I strongly agree with this, and I have been actively using Python since 2009. Trying top keep a Pygame/Numpy/Scipy project working has been a real struggle. I started it with Python 2 and ported to Python 3 some years ago. The whole Python 3 transition is a huge mess with every Python 3 point release breaking some things. No other interpreted language’s packaging system is so fucked up. On a positive note: Lately I've liked using pdm instead of pip, and things seem to work quite a lot better. I evaluated Poetry, Flit and something else also. I just commented about this on Twitter, when someone asked “Which programming language do you consider beginner's friendly?” https://twitter.com/peterhil/status/1633793218411126789 https://twitter.com/peterhil/status/1633793218411126789
black3r 4y ago
> Most of the complaints here ironically are from people using a bunch of tooling in lieu of, or as a replacement for vanilla python venvs and then hitting issues associated with those tools. That's because the vanilla python venvs feel like a genius idea but not thought out thoroughly, they feel as if there's something missing..., So there's naturally lots of attempts at improvements and people jump at those... And when you think about it in bigger depth, venvs are themselves just another one of the solutions used to fix the horrible mess that is python's management of packages and sys.path... The "Zen of Python" says "There should be one-- and preferably only one --obvious way to do it.", so I can't understand why it's nowhere near as easy when it comes to Python's package management...
- JohnFen 4y ago
  Honestly, virtual environments are one of the reasons why I prefer to avoid Python whenever I can.
  - earthboundkid 4y ago
    To me, the virtual environments are a symptom. The cause is people defending Python even when it’s not as good as the alternatives. Every language has flaws. Every language has things it can learn from other languages. Every language is a trade off of different features. But somehow Python packaging, despite being really unpleasant compared to other languages and quite stagnant, is defended very vigorously in these threads (which lol are constantly recurring, unlike other languages). Just last week, I tried to install a Python program from 2020. I failed. I think the problem is that it relied on Pandas and maybe Pandas from then doesn’t work? I have no idea what the real flaw was, but jeez, it’s annoying to have people act like there is no problem. Yes, there is a big problem! This is a dead parrot.
  - bombolo 4y ago
    why? Also, you don't have to use them.
    - 9dev 4y ago
whalesalad 4y ago
I've been using Python since like 2006, so maybe I just have that generational knowledge and battlefront experience... but whenever I come into threads like this I really feel like an imposter or a fish out of water. Like, am I using the same Python that everyone else is using? I echo your stance - the less overhead and additional tooling the better. A simple requirements.txt file and pip is all I need.
- crabbone 4y ago
  Lol. You put "simple" and "requirements.txt" unironically next to each other... I mean, I think you genuinely believe that what you suggest is simple... so, I won't pretend to not understand how you might think that. I'll explain: There's simplicity in performing and simplicity of understanding the process. It's simple to make more humans, it's very hard to understand how humans work. When you think about using pip with requirements.txt you are doing the simple to perform part, but you have no idea what stands behind that. Unfortunately for you, what stands behind that is ugly and not at all simple. Well, you may say that sometimes it's necessary... but, in this case it's not. It's a product of multiple subsequent failures of people working on this system. Series of mistakes, misunderstandings, bad designs which set in motions processes that in retrospect became impossible to revert. There aren't good ways to use Python, but even with what we have today, pip + requirements.txt is not anywhere near the best you can do, if you want simplicity. Do you want to know what's actually simple? Here: Store links to Wheels of your dependencies in a file. You can even call it requirements.txt if you so want. Use curl or equivalent to download those wheels and extract them into what Python calls "platlib" (finding it is left as an exercise for the reader) removing everything in scripts and data catalogues. If you feel adventurous, you can put scripts into the same directory where Python binary is installed, but I wouldn't do that if I were you. Years of being in infra roles taught me that this is the most reliable way to have nightly builds running quietly and avoiding various "infra failures" due to how poorly Python infra tools behave.
  - akprasad 4y ago
    What are specific problems you have with pip + requirements.txt, and why do you believe storing links to wheels is more reliable? Your comment makes your conclusion clear, but I don't follow your argument.
crabbone 4y ago
The most important part about venv is that you shouldn't need it. The very fact that it exists is a problem. It is a wrong fix to a problem that was left unfixed because of it. The problem is fundamental in Python in that its runtime doesn't have a concept of a program or a library or a module (not to be confused with Python's modules, which is a special built-in type) etc. The only thing that exists in Python is a "Python system", i.e. an installation of Python with some packages. Python systems aren't built to be shared between programs (especially so because it's undefined what a program is in Python), but, by any plausible definition of a program, venv doesn't help to solve the problem. This is also amplified by a bunch of tools that simply ignore venvs existence. Here are some obvious problems venv doesn't even pretend to solve: * A Python native module linking with shared objects outside of Python's lib subtree. Most comically, you can accidentally link a python module in one installation of Python with Python from a "wrong" location (and also a wrong version). And then wonder how it works on your computer in your virtual environment, but not on the server. * venvs provides no compile-time isolation. If you are building native Python modules, you are going to use system-wide installed headers, and pray that your system headers are compatible with the version of Python that's going to load your native modules. * venv doesn't address PYTHONPATH or any "tricks" various popular libraries (s.a. pytest and setuptools) like to play with the path where Python searches for loadable code. So much so that people using these tools often use them contrary to how they should be used (probably in most cases that's what happens). Ironically, often even the authors of the tools don't understand the adverse effects of how the majority is using their tools in combination with venv. * It's become a fashion to use venv when distributing Python programs (eg. there are tools that help you build DEB or RPM packages that rely on venv) and of course, a lot of bad things happen because of that. But, really, like I said before: it's not because of venv, it's because venv is the wrong fix for the actual problem. The problem nobody in Python community is bold enough to address.
- dkarl 3y ago
  > The most important part about venv is that you shouldn't need it. The very fact that it exists is a problem. It is a wrong fix to a problem that was left unfixed because of it. What Python needs is a tool that understands your project structure and dependencies so the rest of your tools don't have to. In other languages, that's called a build tool, which is why people have a hard time understanding that Python needs one.
hot_gril 4y ago
I've never used anything but vanilla Python venvs, and no they don't work reliably. What does is a Docker container. I keep hearing excuses for it, but the prevalence of Dockerfiles in GitHub Python projects says it all. This is somehow way less of an issue in NodeJS, maybe because local environments were always the default way to install things.
- neuronexmachina 4y ago
  > This is somehow way less of an issue in NodeJS, maybe because local environments were always the default way to install things. There's also NodeJS's ability for dependencies to simultaneously use conflicting sub-dependencies.
  - hot_gril 4y ago
    Yeah, you can't have two deps use different versions of the same sub-dep, cause they flatten everything instead of having a tree. In practice I rarely have issues with this except in React-Native, where it's a common problem, but then again RN is doing some crazy stuff to begin with. Often just force install deps and things work anyway. Side note, there are way too many React/React-Native "router" type packages, and at least one of them breaks its entire API every update (I think https://reactrouter.com/en/main/upgrading/v5 https://reactrouter.com/en/main/upgrading/v5, how are they on version 6 of this). It's so bad that you can't even Google things anymore cause of the naming conflicts.
TheRealPomax 4y ago
Except when you try to move it, or copy it to a different location. This _almost_ made sense back when it was its own script, but it hasn't made sense for years, and the obstinacy to just sit down and fix this has been bafflingly remarkable. ("why not make everyone install their own venv and run pip install?" because, and here's the part that's going to blow your mind: because they shouldn't have to. The vast majority of packages don't need compilation, you just put the files in the right libs dir, and done. Your import works. Checking this kind of thing into version control, or moving it across disks, etc. etc. should be fine and expected. Python yelling about dependencies that do need to be (re)compile for your os/python combination should be the exception, not the baseline)
- haskellandchill 4y ago
  > Except when you try to move it, or copy it to a different location. The article says it is explicitly not designed for that: "One point I would like to make is how virtual environments are designed to be disposable and not relocatable."
  - TheRealPomax 4y ago
    Good job, you spotted the exact problem: that's what they were originally for, and that makes no sense in 2023 (or 2020, or the day venv functionality was added into main line Python) where literally everything a venv does is trivially achieved in a new location if it didn't hardcode everything relating to paths. There is literally nothing about a venv that somehow magically makes it impossible to still work after relocation. Breaking the venv on relocation was a conscious choice that has been insisted on to this day for no good reason other than "a long history of not bothering to fix this nonsense is all the justification we need to continue not fixing this nonsense".
    - nodemaker 4y ago
      Jeez just clone a venv with venv cloning tools
renewiltord 4y ago
The annoying thing with vanilla venvs (which are principally what I use) is that when I activate a venv, I can no longer `nvim` in that directory because that venv is not going to have `python-neovim` installed. This kind of state leakage is unpleasant for me to work with.
nl 4y ago
I agree with this 100%. Simple venv works reliably. The only gotcha I've had is to make sure you deactivate and reactivate the virtual environment after installing Jupyter (or iPython). Otherwise (depending on your shell) you might have the executable path to "jupyter" cached by your shell so "jupyter notebook" will be using different dependencies to what you think. Even comparatively experienced programmers don't see to know this and it causes a lot of subtle programs. Here's some details on how bash caches paths: https://unix.stackexchange.com/questions/5609/how-do-i-clear-bashs-cache-of-paths-to-executables https://unix.stackexchange.com/questions/5609/how-do-i-clear...
smeagull 4y ago
My complaints stem from libraries/OSes requiring different tools. So conda is sometimes required, and pip is also sometimes required, and some provide documentation only for pipenv rather than venv. And then you've got Jupyter, which needs to be configured for each environment. On top of that there are some large libraries that need to only be installed once per system because they're large, which you can do but does mess with dependency resolution, and god help you if you have multiple shadowing versions of the same library installed. I wish it was simpler. I agree the underlying system is solid, but the fact that it doesn't solve some issues means we have multiple standards layered on top, which is itself a problem. And great if you've been using vanilla venvs. Good for those that can. If I want hardware support for Apple's hardware I need to use fucking conda. Heaven help me if I want to combine that in a project with something that only uses pip.
atoav 4y ago
I agree with the statement that venvs are usable and fine. However, they do not come without their pitfalls in the greater picture of development and deployment of python software. It very often not as simple as going to your target system, cloning the repo and running a single line command that gives you the running software. This is what e.g. Rust's cargo would do. The problem with python venvs is that when problems occur, they require a lot of deep knowledge very fast and that deep knowledge will not be available to the typical beginner. Even I as a decade long python dev will occasionally encounter stuff that takes longer to resolve than needed.
emptysongglass 4y ago
But why bother? Just use PDM in PEP-582 mode [1] which handles packages the same way as project-scoped Node packages. Virtual environments are just a stop-gap that persisted for long enough for a whole ecosystem of tooling to support for them. It doesn't make them less bad, just less frustrating to deal with. [1] https://pdm.fming.dev/latest/usage/pep582/ https://pdm.fming.dev/latest/usage/pep582/

Jackevansevo 4y ago

Likewise, I think people have a negative first experience because it doesn't work exactly like node, throw their toys out the pram and complain on HN for the rest of time. Guess in taking this stance we're both part of the problem... \s

aflag 4y ago

It's incredibly lacking in features. PyPI doesn't even properly index packages, making pip go into this dependency resolution he'll trying to find a set of versions that will work for you. It works for simple cases with few dependencies/not a lot of pinning. But if your needs are a bit more complex it certainly shows its rough edges. I actually find it amazing that they python community puts up with that. But I suppose fixing it is not that pressing now the language is widely adopted. It's not going to be anyone's priority to mess with that. It's high risk low rewards sort of project.

Groxx 4y ago
What does that have to do with venvs? I agree the packaging and distribution setup in python is an absolute mess, but that's entirely unrelated to venvs. It's like bringing up how python uses whitespace instead of curly-braces.
- oneepic 4y ago
  I think the GP comment might have caused some confusion since it mentioned both package management and venvs very close together.
- deleted 4y ago
  [deleted]
- lmm 4y ago
  venvs are the recommended workaround for the fact that python packaging and distribution is a mess of global state. Lanugages with working packaging and distribution don't generally bother with anything venv-like.
  - Groxx 4y ago
    Sure, but that's like 99% pip. Venvs are patching it (quite effectively), not causing it.
whalesalad 4y ago
I've been writing Python for a looong time. I have pushed out thousands and thousands of deployments across probably 40+ distinct Python codebases and only once or twice have I ever encountered a showstopper dependency resolution issue. At the end of the day you should want to have fine grained control over your deps and frankly there are many times where a decision cannot be automatically made by a package manager. Pip gets beat on hard but it puts in work all day every day and rarely skips a beat. It's entirely free and developed with open source contributions. Areas where I have felt a lot of pain is with legacy Ruby projects/bundler. Don't get me started on golang. Can pip be made better? Sure. Should we have an attitude of disgust towards it? Heck no!
- rxhernandez 4y ago
  > only once or twice have I ever encountered a showstopper dependency resolution issue I've encountered them with other languages and they're the sort of thing where one time is more than enough to make me feel like it could get me fired; they're Never (with a capital N) okay imo
- crabbone 4y ago
  > once or twice have I ever encountered a showstopper dependency resolution issue. Hahaha... (rolls on the floor) Do you want to know why that is? No seriously? I'm not laughing at you as much as I'm laughing at Python now, but hey, well, anyways, do you want to know why that happened to you? I know you don't. But I'll tell you anyways! Until quite recently, pip didn't give a rat's ass if the dependencies it installed were consistent. It would blink a message in the long stream of vomit it spills on the screen saying something like "you have package X installed of version Y, but package Z wants X of version Q, which will not be installed". And happily streamed more garbage to your screen. It was an issue that was filed against pip for something like 12 years until it got resolved about a year or so ago. Even after it got resolved a lot of people tried to upgrade, saw that that would "break" their deployment, and rolled back to the latest broken version. Things are sort of improving gradually since then, but we are light years away from the system working properly, and I know you don't want to know why, but I'll tell you anyways! So, when for whatever reason pip doesn't find a dependency it thinks you need, a lot of packages, when they roll out their "releases", they upload also what Python calls "source release". Which should have never been treated as an installation option, but it is, and is treated like that by default. So, what will happen once pip finally gives up on finding a match, right, you guessed it! -- It's going to try to build it! Installing build dependencies along the way. What you get in the end is anyone's guess, but most likely, it's something broken because the developers who made this release didn't make a release specifically for your version. Don't despair. There's a flag you can use with pip install that should prevent it from trying to build stuff. But two bad things will happen to you if you use it: in any non-trivial project your dependencies will irreparably break. And, who knows if that flag is implemented correctly... nobody in the real world is using that. So, who knows, maybe it'll format your hard drive along the way.
crabbone 4y ago
I hate PyPI probably even more than you do, but venv doesn't do that. All it does is write a handful of files and make a bunch of symlinks. It doesn't deal with installation of packages.
- aflag 4y ago
  Ok. Fair. Venvs are great, unless you want to install packages on them.

winrid 4y ago

Because even with --copy it creates all kinds of symlinks, and if you're using pyenv, hard coded paths to the python binary which can break from CI to installation. If you're using docker then it's a lot easier I guess.

mikepurvis 4y ago
It also quietly reuses the stdlib of whatever python you start from. Which mostly doesn’t matter in real world usage, but can be quite surprising if you ever get into your head the idea that that venv is portable.

9dev 4y ago

Oh, yeah? It’s working great? Like figuring out which packages your application actually uses? Or having separate development and production dependencies? Upgrading outdated libraries? Having taken a deep-dive into refactoring a large python app, I can confidently say that package management in python is a pain compared to other interpreted languages.

rowanseymour 4y ago
Virtual environments aren't package management. For example we use Poetry for package management - it supports separate dev and prod dependencies, upgrading etc. It generates a virtual environment.
- 9dev 4y ago
  The distinction feels entirely academic to me. Managing packages means having a sane way to define the dependencies of software projects, so they can be checked into version control and be installed reproducibly later and/or elsewhere. I don’t know which problem python intended to solve by separating the two, but it doesn’t occur often in contemporary software engineering work. Having said that, the point you make is valid and Poetry is a good option, but it feels so maddening having to learn about like seven different tools which all do more or less the same but not quite, and everyone and their mother having an opinion on which is the best. Doesn’t help that there’s an arbitrary yet blurry line where package managers end and environment managers begin.
peterhil 4y ago
I strongly agree with this, and I have been actively using Python since 2009. Trying top keep a Pygame/Numpy/Scipy project working has been a real struggle. I started it with Python 2 and ported to Python 3 some years ago. The whole Python 3 transition is a huge mess with every Python 3 point release breaking some things. No other interpreted language’s packaging system is so fucked up. On a positive note: Lately I've liked using pdm instead of pip, and things seem to work quite a lot better. I evaluated Poetry, Flit and something else also. I just commented about this on Twitter, when someone asked “Which programming language do you consider beginner's friendly?” https://twitter.com/peterhil/status/1633793218411126789 https://twitter.com/peterhil/status/1633793218411126789

black3r 4y ago

> Most of the complaints here ironically are from people using a bunch of tooling in lieu of, or as a replacement for vanilla python venvs and then hitting issues associated with those tools. That's because the vanilla python venvs feel like a genius idea but not thought out thoroughly, they feel as if there's something missing..., So there's naturally lots of attempts at improvements and people jump at those... And when you think about it in bigger depth, venvs are themselves just another one of the solutions used to fix the horrible mess that is python's management of packages and sys.path... The "Zen of Python" says "There should be one-- and preferably only one --obvious way to do it.", so I can't understand why it's nowhere near as easy when it comes to Python's package management...

JohnFen 4y ago
Honestly, virtual environments are one of the reasons why I prefer to avoid Python whenever I can.
- earthboundkid 4y ago
  To me, the virtual environments are a symptom. The cause is people defending Python even when it’s not as good as the alternatives. Every language has flaws. Every language has things it can learn from other languages. Every language is a trade off of different features. But somehow Python packaging, despite being really unpleasant compared to other languages and quite stagnant, is defended very vigorously in these threads (which lol are constantly recurring, unlike other languages). Just last week, I tried to install a Python program from 2020. I failed. I think the problem is that it relied on Pandas and maybe Pandas from then doesn’t work? I have no idea what the real flaw was, but jeez, it’s annoying to have people act like there is no problem. Yes, there is a big problem! This is a dead parrot.
- bombolo 4y ago
  why? Also, you don't have to use them.
  - 9dev 4y ago

whalesalad 4y ago

I've been using Python since like 2006, so maybe I just have that generational knowledge and battlefront experience... but whenever I come into threads like this I really feel like an imposter or a fish out of water. Like, am I using the same Python that everyone else is using? I echo your stance - the less overhead and additional tooling the better. A simple requirements.txt file and pip is all I need.

crabbone 4y ago
Lol. You put "simple" and "requirements.txt" unironically next to each other... I mean, I think you genuinely believe that what you suggest is simple... so, I won't pretend to not understand how you might think that. I'll explain: There's simplicity in performing and simplicity of understanding the process. It's simple to make more humans, it's very hard to understand how humans work. When you think about using pip with requirements.txt you are doing the simple to perform part, but you have no idea what stands behind that. Unfortunately for you, what stands behind that is ugly and not at all simple. Well, you may say that sometimes it's necessary... but, in this case it's not. It's a product of multiple subsequent failures of people working on this system. Series of mistakes, misunderstandings, bad designs which set in motions processes that in retrospect became impossible to revert. There aren't good ways to use Python, but even with what we have today, pip + requirements.txt is not anywhere near the best you can do, if you want simplicity. Do you want to know what's actually simple? Here: Store links to Wheels of your dependencies in a file. You can even call it requirements.txt if you so want. Use curl or equivalent to download those wheels and extract them into what Python calls "platlib" (finding it is left as an exercise for the reader) removing everything in scripts and data catalogues. If you feel adventurous, you can put scripts into the same directory where Python binary is installed, but I wouldn't do that if I were you. Years of being in infra roles taught me that this is the most reliable way to have nightly builds running quietly and avoiding various "infra failures" due to how poorly Python infra tools behave.
- akprasad 4y ago
  What are specific problems you have with pip + requirements.txt, and why do you believe storing links to wheels is more reliable? Your comment makes your conclusion clear, but I don't follow your argument.

crabbone 4y ago

The most important part about venv is that you shouldn't need it. The very fact that it exists is a problem. It is a wrong fix to a problem that was left unfixed because of it. The problem is fundamental in Python in that its runtime doesn't have a concept of a program or a library or a module (not to be confused with Python's modules, which is a special built-in type) etc. The only thing that exists in Python is a "Python system", i.e. an installation of Python with some packages. Python systems aren't built to be shared between programs (especially so because it's undefined what a program is in Python), but, by any plausible definition of a program, venv doesn't help to solve the problem. This is also amplified by a bunch of tools that simply ignore venvs existence. Here are some obvious problems venv doesn't even pretend to solve: * A Python native module linking with shared objects outside of Python's lib subtree. Most comically, you can accidentally link a python module in one installation of Python with Python from a "wrong" location (and also a wrong version). And then wonder how it works on your computer in your virtual environment, but not on the server. * venvs provides no compile-time isolation. If you are building native Python modules, you are going to use system-wide installed headers, and pray that your system headers are compatible with the version of Python that's going to load your native modules. * venv doesn't address PYTHONPATH or any "tricks" various popular libraries (s.a. pytest and setuptools) like to play with the path where Python searches for loadable code. So much so that people using these tools often use them contrary to how they should be used (probably in most cases that's what happens). Ironically, often even the authors of the tools don't understand the adverse effects of how the majority is using their tools in combination with venv. * It's become a fashion to use venv when distributing Python programs (eg. there are tools that help you build DEB or RPM packages that rely on venv) and of course, a lot of bad things happen because of that. But, really, like I said before: it's not because of venv, it's because venv is the wrong fix for the actual problem. The problem nobody in Python community is bold enough to address.

dkarl 3y ago
> The most important part about venv is that you shouldn't need it. The very fact that it exists is a problem. It is a wrong fix to a problem that was left unfixed because of it. What Python needs is a tool that understands your project structure and dependencies so the rest of your tools don't have to. In other languages, that's called a build tool, which is why people have a hard time understanding that Python needs one.

hot_gril 4y ago

I've never used anything but vanilla Python venvs, and no they don't work reliably. What does is a Docker container. I keep hearing excuses for it, but the prevalence of Dockerfiles in GitHub Python projects says it all. This is somehow way less of an issue in NodeJS, maybe because local environments were always the default way to install things.

neuronexmachina 4y ago
> This is somehow way less of an issue in NodeJS, maybe because local environments were always the default way to install things. There's also NodeJS's ability for dependencies to simultaneously use conflicting sub-dependencies.
- hot_gril 4y ago
  Yeah, you can't have two deps use different versions of the same sub-dep, cause they flatten everything instead of having a tree. In practice I rarely have issues with this except in React-Native, where it's a common problem, but then again RN is doing some crazy stuff to begin with. Often just force install deps and things work anyway. Side note, there are way too many React/React-Native "router" type packages, and at least one of them breaks its entire API every update (I think https://reactrouter.com/en/main/upgrading/v5 https://reactrouter.com/en/main/upgrading/v5, how are they on version 6 of this). It's so bad that you can't even Google things anymore cause of the naming conflicts.

TheRealPomax 4y ago

Except when you try to move it, or copy it to a different location. This _almost_ made sense back when it was its own script, but it hasn't made sense for years, and the obstinacy to just sit down and fix this has been bafflingly remarkable. ("why not make everyone install their own venv and run pip install?" because, and here's the part that's going to blow your mind: because they shouldn't have to. The vast majority of packages don't need compilation, you just put the files in the right libs dir, and done. Your import works. Checking this kind of thing into version control, or moving it across disks, etc. etc. should be fine and expected. Python yelling about dependencies that do need to be (re)compile for your os/python combination should be the exception, not the baseline)

haskellandchill 4y ago
> Except when you try to move it, or copy it to a different location. The article says it is explicitly not designed for that: "One point I would like to make is how virtual environments are designed to be disposable and not relocatable."
- TheRealPomax 4y ago
  Good job, you spotted the exact problem: that's what they were originally for, and that makes no sense in 2023 (or 2020, or the day venv functionality was added into main line Python) where literally everything a venv does is trivially achieved in a new location if it didn't hardcode everything relating to paths. There is literally nothing about a venv that somehow magically makes it impossible to still work after relocation. Breaking the venv on relocation was a conscious choice that has been insisted on to this day for no good reason other than "a long history of not bothering to fix this nonsense is all the justification we need to continue not fixing this nonsense".
  - nodemaker 4y ago
    Jeez just clone a venv with venv cloning tools

renewiltord 4y ago

The annoying thing with vanilla venvs (which are principally what I use) is that when I activate a venv, I can no longer `nvim` in that directory because that venv is not going to have `python-neovim` installed. This kind of state leakage is unpleasant for me to work with.

nl 4y ago

I agree with this 100%. Simple venv works reliably. The only gotcha I've had is to make sure you deactivate and reactivate the virtual environment after installing Jupyter (or iPython). Otherwise (depending on your shell) you might have the executable path to "jupyter" cached by your shell so "jupyter notebook" will be using different dependencies to what you think. Even comparatively experienced programmers don't see to know this and it causes a lot of subtle programs. Here's some details on how bash caches paths: https://unix.stackexchange.com/questions/5609/how-do-i-clear-bashs-cache-of-paths-to-executables https://unix.stackexchange.com/questions/5609/how-do-i-clear...

smeagull 4y ago

My complaints stem from libraries/OSes requiring different tools. So conda is sometimes required, and pip is also sometimes required, and some provide documentation only for pipenv rather than venv. And then you've got Jupyter, which needs to be configured for each environment. On top of that there are some large libraries that need to only be installed once per system because they're large, which you can do but does mess with dependency resolution, and god help you if you have multiple shadowing versions of the same library installed. I wish it was simpler. I agree the underlying system is solid, but the fact that it doesn't solve some issues means we have multiple standards layered on top, which is itself a problem. And great if you've been using vanilla venvs. Good for those that can. If I want hardware support for Apple's hardware I need to use fucking conda. Heaven help me if I want to combine that in a project with something that only uses pip.

atoav 4y ago

I agree with the statement that venvs are usable and fine. However, they do not come without their pitfalls in the greater picture of development and deployment of python software. It very often not as simple as going to your target system, cloning the repo and running a single line command that gives you the running software. This is what e.g. Rust's cargo would do. The problem with python venvs is that when problems occur, they require a lot of deep knowledge very fast and that deep knowledge will not be available to the typical beginner. Even I as a decade long python dev will occasionally encounter stuff that takes longer to resolve than needed.

emptysongglass 4y ago

But why bother? Just use PDM in PEP-582 mode [1] which handles packages the same way as project-scoped Node packages. Virtual environments are just a stop-gap that persisted for long enough for a whole ecosystem of tooling to support for them. It doesn't make them less bad, just less frustrating to deal with. [1] https://pdm.fming.dev/latest/usage/pep582/ https://pdm.fming.dev/latest/usage/pep582/