10 ms·
My Python Development Environment, 2020 Edition
- mlthoughts2018 7y agoThis is so painful to see compared to using conda.
- whalesalad 7y ago1. The author of this post helped to create the Django framework and runs a successful Python consultancy. 2. Conda is not used as much as you might think... it's really only used within the data science community.
- denhaus 7y agoBeing in the data science community myself, I prefer straight venv + pip to conda. It’s simpler for me to manage errors. I only use conda when I have to.
- whalesalad 7y agoYeah I don't know a single person who chooses to use conda.
- kirubakaran 7y agoHi whalesalad, good to meet you! Now that you know me, you can never say that again anymore :-) Although, tbf, I only use conda for my machine learning related projects. I've tried using pip for that but was at risk of massive hair loss.
- analog31 7y agoIn the scientific community, there is a widespread "just use anaconda" message. Many people spend their entire lives inside anaconda, and equate it with Python.
- Daishiman 7y agoConda is ideal if you need to support Windows environments with large, compiled scientific programming libraries.
- tcpekin 7y agoOn the other hand, almost everyone I know who does scientific python in grad school loves miniconda...
- lsiebert 7y agoAnd anyway, you should tell people to install mini conda, not conda, unless they know they'll need everything it includes by default.
- whalesalad 7y agoYet another reason to avoid it entirely.
- mlthoughts2018 7y ago1. Argument from authority doesn’t mean anything to me. I also don’t believe creating Django or running a Python consultancy endow someone with especially useful opinions of Python packaging tooling. (Not that the author isn’t knowledgeable, just you seem to think there’s an A implies B relationship between those two items and having good opinions about Python packaging, and there’s not). 2. Conda is quite widely used outside of data science. It’s for example part of Anaconda enterprise offerings used by huge banks, government agencies, universities, etc., on large projects often with no use cases related to data science. Conda itself has no logical connection with data science, it’s just a package & environment manager. In each of my last 4 jobs, 2 at large Fortune 500 ecommerce companies, conda has been the environment manager used for all internal Python development. Still use pip a lot within conda envs, but conda is the one broader constant.
- m000z0rz 7y ago> huge banks, government agencies, universities > large Fortune 500 e-commerce companies Sorry, but argument from authority doesn't mean anything to me. In all seriousness though, you literally did not provide any logical reasons to think conda is better.
- mlthoughts2018 7y agoGiving a counterexample is not argument from authority. I did not respond to the parent comment to discuss any feature of conda, only to dispel the wrong claim that only mostly data science projects rely on it.
- diminoten 7y agoI love when people claim to not care about arguments from authority. "Sorry doc, I don't care about your medical degree or years of experience!" Yes you do.
- misnome 7y ago<20 minutes later> "Sorry, your dependencies could not be resolved"
- dang 7y agoPlease don't break HN's guidelines by being snarky or putting down others' work in a shallow way. If you know more or have a different perspective to offer, try sharing some of what you know so we can all learn something!
- deleted 7y ago[deleted]
- mrfusion 7y agoWhy pipx vs just using pip?
- tedivm 7y agoWith pipx when you install things they go into isolated environments. With pip you're just installing things globally. This difference is important due to dependencies- if you have two different CLI tools you want to install but they have conflicting dependencies then pip is going to put at least one of them into an unusable state, while pipx will allow them to both coexist on the same system.
- Znafon 7y agoYou can get this with standard pip by setting the PIP_REQUIRE_VIRTUALENV env variable.
- acdha 7y agoThat just gives you a warning. It doesn’t automatically create a virtualenv and link the bin scripts into the path.
- AdrienLemaire 7y agoI haven't used pipx, but as far as I understand, pipx = pip + venv. If your pip executable is in a virtualenv, the "globally installed" is locally installed. pipx, poetry, pipenv and co are still nice wrappers to have, I suppose. It just feel less useful now that most of my projects are dockerized.
- yrro 7y agopipx looks nice. Is there any way to persuade it to install 'wheel' before it installs the desired package? That way 'pipx install foo' can download and install wheels rather than downloading source distributions and building/installing them...
- takeda 7y ago
- ninetax 7y agoCurious to hear other's experiences with pipenv vs poetry. Has anyone made the switch?
- kndjckt 7y agoI switched from pipenv to poetry over 1 year ago. I love it! The main reasoning was so that I could easily build and publish packages to a private repository and then easily import packages from both pypi and the private repository. Happy to answer more questions.
- thehesiod 7y agoI'd like to use poetry however ran into https://github.com/sdispater/poetry/issues/1554 https://github.com/sdispater/poetry/issues/1554 We have a custom pypi server and need all requests to go through it, however haven't figured a way to make poetry always use our index server for all modules instead of pypi.org
- _AzMoo 7y agoAdd a second source and it will prioritise that over pypi. [[tool.poetry.source]] name = "my-repo-name" url = "https://myrepo.url/"
- timothycrosley 7y agoSwitched to poetry and couldn't be happier. From my experience, poetry wins hands down. It managed to replace flit, remove duplicate dependencies, and maintain stability across machines. All while using the standard pyproject.toml configuration file.
- franey 7y agoI've used both and recommend Poetry. It's got a larger feature set (it can be used to manage packages _and_ publish packages), it's faster, and it's more actively developed (measured by releases). Pipenv's last release was 2018-11-26, and Poetry has been publishing pre-releases as recently as three days ago to prepare for v1.0.0. I did a quick comparison here[0], and I'm planning to do an update with the latest version of Poetry. [0] https://johnfraney.ca/posts/2019/03/06/pipenv-poetry-benchmarks-ergonomics/ https://johnfraney.ca/posts/2019/03/06/pipenv-poetry-benchma...
- Lucasoato 7y agoThis article is great, those are viable solutions for sure. One of the alternatives is conda: it's common among data scientists, but many of its features (isolation between environments, you can keep private repository off the internet) meet enterprise needs.
- analog31 7y agoIn addition to the popular conda, it's worth checking out WinPython for scientific use. Each WinPython installation is an isolated environment that resides in a folder. To move an installation to another computer, just copy the folder. To completely remove it from your system, delete the folder. I find it useful to keep a WinPython installation on a flash drive in my pocket. I can plug it into somebody's computer and run my own stuff, without worrying that I'm going to bollix up their system.
- snorkasaurus 7y agoI like pip-tools for venv requirements management, but I don't see it mentioned much.
- globular-toast 7y agoI use pip-tools. It fits in nicely as an additional component to the standard toolset (pip and virtualenv). But most people probably do not need to freeze environments so it's great to be able to not use it for most projects.
- gravypod 7y agoThe Dockerfile that's provided looks like it would be very slow to build. I always try to make Dockerfiles that install deps and then install my python package (usually just copy in the code and set PYTHONPATH) to fully take advantage of the docker build cache. When you have lots of services it really reduces the time it takes to iterate with `docker-compose up -d --build`-like setups.
- dang 7y agoRelated from 2018: https://news.ycombinator.com/item?id=16439270 https://news.ycombinator.com/item?id=16439270
- nunez 7y ago> Although Docker meets all these requirements, I don't really like using it. I find it slow, frustrating, and overkill for my purposes. How so? I've been using Docker for development for years now and haven't experienced this EXCEPT with some slowness I experienced with Docker Compose upon upgrading to MacOS Catalina (which turned out to be bug with PyInstaller, not Docker or Docker Compose). This is on a Mac, btw; I hear that Docker on Linux is blazing fast. I personally would absolutely leverage Docker for the setup being described here: multiple versions with lots of environmental differences between each other. That's what Docker was made for!
- jsmeaton 7y agoThe build step for installing or upgrading a package can be a killer with nontrivial projects.
- maksimum 7y agoIt seems like long builds are either (a) necessary or (b) user error. (a) If you have a tree of dependencies and you change the root, you should rebuild everything that depends on it to make sure it's still compatible. (b) if you placed your application into one of the initial Dockerfile layers, but then you're installing dependencies that don't depend on you, it's user error. What's the situation where your application needs to go first in the Dockerfile, and then you need to put a bunch of stuff that doesn't depend on your application?
- jacobian 7y agoI would love to read a blog post covering how to do this! My experience has been that it's significantly more effort to meet my requirements with Docker, and that I spend a _lot_ of time waiting on Docker builds, or trying to debug finicky issues in my Dockerfile/docker-compose setup. I'm sure all of these things have fixes -- of course they do! But I find the toolset challenging, and the documentation difficult to follow. I'd love to learn what I'm missing, but I also need to balance that against Getting Shit Done.
- marmada 7y agoDoes anyone else think this reflects badly on Python? The fact that the author has to use a bunch of different tools to manage Python versions/projects is intimidating. I don't say this out of negativity for the sake of negativity. Earlier today, I was trying to resurrect an old Python project that was using pipenv. "pipenv install" gave me an error about accepting 1 argument, but 3 were provided. Then I switched to Poetry. Poetry kept on detecting my Python2 installation, but not my Python3 installation. It seemed like I had to use pyenv, which I didn't want to use, since that's another tool to use and setup on different machines. I gave up and started rewriting the project (web scraper) in Node.js with Puppeteer.
- lone_haxx0r 7y agoYes. As someone who has never dove deep into python, but has had some contact with it: the package manager ecosystem is the #1 thing keeping me away from it. npm sucks and all, but at least it just works and doesn't get in my way as much.
- kart23 7y agoComing the opposite way: having started programming on python and only started doing node.js stuff recently, npm is awesome.
- GuyOnMySpace 7y ago100% agree. Coming from the scientific computing realm, I've only ever done "real" code in Python until a couple of weeks ago when I was forced to do some work in Typescript. Once I got over the initial worries about the best method to install npm to avoid Python-esque problems and gave nvm a try, I was very pleasantly surprised by the package management process and was able to just get on with the work. I've tried various Python env management solutions in the past (mostly leaning towards conda), but had recently settled on just using separate LXC/LXD containers for each project.
- heyoni 7y agoWhat does npm do that python can’t? I’m curious.
- Evidlo 7y agoFor those new or unfamiliar with python, I think the best solution is the simplest: pip and virtualenv
- ramraj07 7y agoSeriously. The author is bending over backwards to accommodate poetry from every direction, from the stupidest installation instructions I've heard, to "can't transfer from requirements.txt" to "it doesn't work well with docker but doable". Like what exactly does it add that's worth all this complexity? Make you a maitai every hour?
- Daishiman 7y agoDid you even read the article?
- Twirrim 7y agoI did, certainly don't see what specific value poetry is providing from what he said.
- raihansaputra 7y agoAFAIK, pipenv and poetry also pins down the dependencies requirements to the same versions, while pip don't (?).
- yjftsjthsd-h 7y agohttps://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html > Please don't comment on whether someone read an article. "Did you even read the article? It mentions that" can be shortened to "The article mentions that."
- thesuperbigfrog 7y agopip and virtualenv are my preferred solution too because they are simple and easy to use. However, you can run into issues if you are using different versions of Python, or Python on different operating systems.
- madelyn 7y agoI have never understood the need for all the different tools surrounding Python packaging, development environments, or things like Pipenv. For years, I have used Virtualenv and a script to create a virtual environment in my project folder. It's as simple as a node_modules folder, the confusion around it is puzzling to me. Nowadays, using setuptools to create packages is really easy too, there's a great tutorial on the main Python site. It's not as easy as node.js, sure, but there's tools like Cookiecutter to remove the boilerplate from new packages. requirements.txt files aren't very elegant, but they work well enough. And with docker, all this is is even easier. The python + docker story is really nice. Honestly I just love these basic tools and how they let me do my job without worrying about are they the latest and greatest. My python setup has been stable for years and I am so productive with it.
- heyoni 7y agoYou’re absolutely right. I use pipenv almost entirely because it can activate a local environment automatically when entering a folder in terminal. That and the fact that my virtual folder lives in ~/.local let’s me work directly in Dropbox. Nothing I couldn’t live without.
- Twirrim 7y agoI'm firmly set on virtualenv with virtualenvwrapper for some convenience functions. Need a new space for a project? mkvirtualenv -p /path/to/python projectname (-p only if I'm not using the default configured in the virtualenv config file, which is rare) From there it's just "workon projectname" and just "deactivate" when I'm done (or "workon otherprojectname") It has been stable and working for ages now. I just don't see any strong incentive to change.
- babayega2 7y agoI have been doing this starting Ubuntu 14.04. it's been stable even when I upgraded now to 18.04 which has python 3 as default. The only downside compared with tool such as pipenv is the automatic update of packages that pipenv can offer and it's ability to be integrated into CI/CD pipelines.
- 7y ago
- notus 7y agoI recommend asdf for version management if you use more than one programming language
- rhizome31 7y agoSeconded. Just to be clear asdf manages interpreters, not project dependencies. It actually uses pyenv under the hood to manage Python versions. I use it to manage Elixir and Python versions and it works rather well. I also find its CLI interface well designed and consistent.
- pritambaral 7y agohttps://common-lisp.net/project/asdf/ https://common-lisp.net/project/asdf/ ?
- rhizome31 7y agohttps://asdf-vm.com/ https://asdf-vm.com/
- globular-toast 7y agoThe Common Lisp build system?
- diminoten 7y ago> Governance: the lead of Pipenv was someone with a history of not treating his collaborators well. That gave me some serious concerns about the future of the project, and of my ability to get bugs fixed. Doesn't seem fair. You're not abandoning requests, are you?
- maw 7y agoRequests made it to a stable, usable state. Sadly, pipenv never did.
- oefrha 7y agoJust noticed requests moved from kennethreitz/requests to psf/requests. Interesting. Edit: From https://www.python.org/psf/github/ https://www.python.org/psf/github/, > ... we have created a GitHub organization, @psf, to support and protect projects that have outgrown ownership by their original author.
- eximius 7y agoI would generally reach for conda instead of this, but they seem quite comparable in aggregate. And, given that I've been trying NixOS lately and had loads of trouble and failing to get Conda to work, I will definitely give this setup a try. (I haven't quite embraced the nix-shell everything solution. It still has trouble with some things. My current workaround is a Dockerfile and a requirements.txt file, which does work...)
- ausjke 7y agopython -m venv venv source venv/bin/activate pip install -U pip pip install whatever # <do you stuff here> deactivate no need any third-party tools, venv is built-in the above steps always worked for me out of the box.
- snypox 7y agoBut how about replacing all of these commands with two words? poetry install
- ausjke 7y agowhich is another layer of abstraction and dependency that I do not really need, e.g poetry no longer maintained, poetry(or whatever) has an urgent bugfix,etc
- sjustinas 7y agoDo you not consider pip third-party? It does not come with the standard Python distribution. Although `ensurepip` does.
- oefrha 7y agoNo, pip has been available in standard Python distributions since 3.4.[1] Distributions that don't come with pip (e.g. some Linux distro packages) are non-standard. [1] https://docs.python.org/3/whatsnew/3.4.html#whatsnew-pep-453 https://docs.python.org/3/whatsnew/3.4.html#whatsnew-pep-453
- radicallib 7y agoIs this developer actually relevant? Who cares what their stack is. It sounds like more marketing over substance. If the developer of click or flask explains their stack, yes, I will listen. Not sure why I care about a non-developer explaining hipster tools to me that are not what people are using.
- packetslave 7y agoYou mean other than being one of the 3 original authors of Django?
- Klonoar 7y ago...yes, this developer is relevant, lol Is this comment just trolling? Load the main page of his site. > I'm a software developer, co-creator of Django, and an experienced engineering leader. I previously ran teams at 18F and Heroku. I'm currently taking new clients through my consultancy, REVSYS.
- kovek 7y agoI'll pay anyone who can assist me with my Python setup. Is there a service like this, where one can find a developer on demand?
- SpaceL10n 7y agoFiverr?
- abcininin 7y agoI have been using a consistent setup that hasn't yet failed me for the past 2 years. 1. Install Anaconda to your home user directory . 2. create environment using (conda create --name myenv python=3.6) . 3. Switch to the environment using (conda activate myenv) . 4. Use (conda install mypackage), (pip install mypackage) in that priority order . 5. Export environment using (conda env export > conda_env.yaml) . 6. Environment can be created on an other system using (conda env create -f conda_env.yaml) . Anaconda: https://www.anaconda.com/distribution/#download-section https://www.anaconda.com/distribution/#download-section . Dockerized Anaconda: https://docs.anaconda.com/anaconda/user-guide/tasks/docker/ https://docs.anaconda.com/anaconda/user-guide/tasks/docker/ .
- bmer 7y agoI can vouch for this. Anaconda is especially good for simulation/data stuff (based on the focus on which packages are included by default). One pain point though: getting it to work with Sublime Text 3 requires you to set the `CONDA_DLL_SEARCH_MODIFICATION_ENABLE` environment variable to `1` on Windows. Not a flaw of Anaconda: it just pays attention to how to with multiple Python installations on Windows.
- paulproteus 7y agoHi kovek! You might like https://www.codementor.io/ https://www.codementor.io/ . I admit I'm a mentor there, and I've made about twenty bucks helping people, but anyway it was super fun helping poeple. :)
- slig 7y agoI've had success finding a great Python dev on /r/ForHire
- deleted 7y ago[deleted]
- nsomaru 7y agoI’ve been manually deploying my projects for years. Can anyone comment on the Docker learning & troubleshooting story for python?
- maksimum 7y agoDocker + setuptools/pip + python is great for development and production. Docker is definitely worth learning, and is pretty easy to learn.
- slig 7y agoStart with `docker` and learn the basics concepts: difference between image and container, layers, etc. Copy a Python `Dockerfile` and see that it works. After a while you'll get the hang of it and will be able to know what to google and how to navigate the docker manual. Pythonspeed.com has some good protips once you understand the basics. You'll get confident and from there learning `docker-compose` is a breeze.
- wp381640 7y agoIf you're going to be using pyenv + poetry you should be aware of #571 that causes issues with activating the virtualenv https://github.com/sdispater/poetry/issues/571 https://github.com/sdispater/poetry/issues/571 the OP himself has a fix for this in his own dotfiles repo: https://github.com/jacobian/dotfiles/commit/e7889c5954daacfe0988fc05ff9e8e87eb1241b7 https://github.com/jacobian/dotfiles/commit/e7889c5954daacfe...
- jacobian 7y agoHa, I'd forgotten about that. Thanks for the reminder. (Though, how'd you find that? Mildly creepy that you know more about my dotfiles than I do!)
- wp381640 7y agoI landed on that issue a while ago and your pull request was linked so I ripped your solution and added it to my own dotfiles :)
- f4stjack 7y agoEeeehhh I think I will be downvoted to hell and back for this but after I read the article I had the feeling of "why are you making this feel more complex than it needs to be?" I mean compared to Java and C# I have a MUCH MORE EASIER time to set up my development environment. Installing Python, if I am on a Windows box I mean, is enough to satisfy a lot of the requirements. I then clone the repo of the project and source venv/bin/activate pip install -r requirements.txt is enough to get me to start coding.
- slig 7y ago> "why are you making this feel more complex than it needs to be?" Because it's more complex if you have projects on multiple Python versions and if you want to lock your Python packages to specific versions. (Pip can bite you when different packages have different requirements for the same lib).
- perlgeek 7y ago> On Linux, the system Python is used by the OS itself, so if you hose your Python you can hose your system. I never manged to hose the OS Python on Linux, by sticking to a very simple rule: DON'T BE ROOT. Don't work as root, don't run `sudo`. On Linux, I use the system python + virtualenv. Good enough. When I need a different python version, I use docker (or podman, which is an awesome docker replacement in context of development environments) + virtualenv in the container. (Why virtualenv in the container? Because I use them outside the container as well, and IMHO it can't hurt to be consistent).
- fock 7y agoapparently in the 2020s and the age of containers it needs a perlgeek to explain basic principles of workstation usage since the 2000s to all the hip kids on the block.
- xchaotic 7y agoI love Python syntax, but I still haven't found a sufficiently popular way that can deploy my code in the same set of setting s as my dev box (other than literally shipping a VM). So setting up a dev env is one problem, but deploying it so that the prod env is the same and works the same is another.
- eivarv 7y agoWhy not just use conda for envs and deps (or env-specific pip), and install youtube-dl etc. via your platform's package manager?
- cosmic_quanta 7y agoIn my experience, conda breaks quite often. Most recently, conda has changed the location where it stores DLLs (e.g. for PyQt), which broke pyinstaller-based workflows. In principle, it's a good idea; in practice, I'm not satisfied. On Windows, it's an easy solution, especially for packages that depend on non-python dependencies (e.g. hdf5).
- MasterScrat 7y agoMy main problem with conda is that it's super slow.
- rullopat 7y agoI like Python has a language, but when I see how clean are the tools of other similar languages, for example Ruby, compared to the clusterfk of the Python ecosystem, it just make me want to close the terminal. I'm always wondering how it became the language #1 on StackOverflow.
- frou_dh 7y agoMy sole use of Python is writing plugins (mostly single-user: me) for Sublime Text. It feels pretty comfy to effectively be on an island and far away from the hustle and bustle of the industrial Python tooling.
- luord 7y agoSo far using docker and setup.py files is working for me, I've never felt they were particularly slow, so I'll keep using them. I gotta give poetry a try, though.
- anonu 7y agoThere are two things that I find a bit elusive with Python: 1. Highlight to run 2. Remoting into a kernel Both features are somewhat related. I want to be able to fire up a Python Kernel on a remote server. I want to be able to connect to it easily (not having to ssh tunnel over 6 different ports). I want connect my IDE to it and easily send commands and view data objects remotely. Spyder does all this but its not great. You have to run a custom kernel to be able to view variables locally. Finally, I want to be able to connect to a Nameko or Flask instance as I would any remote kernel and hot-swap code out as needed.
- schainks 7y agoI've moved to ASDF and haven't really looked back. It's working well with low fuss, and supporting far more than just python on my machine.