7 ms·

Show HN: Python requirements for Humans

78 points by Socketubs 11y ago

JulianWasTaken 11y ago
This is a nice idea in theory, but setup.py and requirements.txt are not the same, and you cannot particularly generate one from the other: https://caremad.io/2013/07/setup-vs-requirement/ https://caremad.io/2013/07/setup-vs-requirement/ Also, unfortunately pip's parser for requirements.txt isn't public, like much of its internals at the moment, which means AIUI it's likely that the parsing code here is going to be brittle if it uses it, or miss edge cases if it doesn't. (I haven't read this carefully yet though).
- yeukhon 11y ago
  ds' article has been referenced many times and I have had multiple discussions on #python myself and trust me, no one agree on a single approach. > Also, unfortunately pip's parser for requirements.txt isn't public I am not sure what you mean by "isn't public." The API is "private" or the parser is proprietary? or the API is not isolated to its own (like Ansible's command line parser is stuck inside a giant function). If the second I would be super surprise. But I am with the repo author. I think we just need one format, one way to do things. The implementation however, needs to be discussed as the example code is pretty redundant in the effort, would be much nicer if we can just provide the path in setup.py and pip reads off setup.py and consume requirements.txt. For god sake this is dependency declaration, ain't package management.
  - pietroalbini 11y ago
    pip is not meant to be used as an API, and there is no backward compatibility guarantee, AFAIK. There is an ongoing effort to extract all its logic into separated packages [1], thought. [1] https://github.com/pypa/packaging https://github.com/pypa/packaging
- thristian 10y ago
  As somebody who deploys a lot of business-critical Python apps at $EMPLOYER, I don't even write a requirements.txt. During development and integration testing, I want to get the latest version of my dependencies that I should be compatible with, so I use SemVer matching (somepackage~=1.2.3) in setup.py. For deployment, I want to use the exact dependency packages that the code was tested against. So after automated testing is complete, it runs "pip wheel" to build wheels of my package and all its dependencies and put them into a directory, then that is the artefact I use to recreate the package in production. Much more reliable than requirement.txt, since it doesn't assume that PyPI or internet access is available, and it doesn't care if upstream authors silently update packages without changing the version number. The last scenario is unit-testing, because sometimes tests have extra dependencies the rest of the code doesn't share, like a mock library or other test helper. In theory, these extra things could be put in setup.py's "tests_require" option, but then they'd only be installed if you run tests with "python setup.py test", and they'd also be installed with "easy_install", which is terrible. I'm thinking maybe I should use setuptools' "feature" syntax, so you only get the test dependencies if you install with the [test] feature.
  - TheCowboy 10y ago
    Another way to do this, that you or someone may be interested in, is by using pip-tools. You write a requirements.in text file where you list libraries by just their name, and then run "pip-compile requirements.in" and it will output a requirements.txt. It also has a function called "pip-sync" which will then install/update everything in the requirements.txt file. https://github.com/nvie/pip-tools https://github.com/nvie/pip-tools
haldean 11y ago
How would you install a package that depended on this? You can't install it's dependencies automatically until you've got one of the dependencies installed.
- Socketubs 11y ago
  Like I say in README[0], you have to put `requirements.py` beside your `setup.py` [0]: https://github.com/socketubs/requirements#usage https://github.com/socketubs/requirements#usage
mpdehaan2 11y ago
Ironically, the package to make your package requirements easier to write has additional requirements not in the core distribution :) But more seriously, good idea in thinking that things should already be this way. The dual maintainance of the dependency list between setup.py and requirements.txt unfortunately leads to a lot of packages being left out of one or the other, or a package that works great from source that has a broken setup.py (which usually applies to everything I do). A setup.py command for installing deps from setup.py would, to me, seem a bit more logical - but things already exist the way they do. In the end though, it's a bit weird to deviate from the standard norms of package installation, at least in Python circles, which may introduce some confusion -- contrast this with Javascript circles that replace "the one true way" about every 3 months :)
- Socketubs 11y ago
  Thank you for your feeback. Same reasons that encourage me to build `requirements` :) I know this isn't perfect for now, `requirements` could evolve!
whalesalad 11y ago
I wish the Python community would adopt a Bundler/Gemfile style. The project.clj concept is also a fantastic approach. Environments should just be maps of data that you can modify or compose on the fly.
reidrac 11y ago
I see a problem with that "License is MIT". That's not how software is licensed and if the author wants this to be used, the legal part must be flawless. Who's the copyright holder? How can I contact that person? What's the copyright year? Licensing software with MIT licence is trivial: http://choosealicense.com/licenses/mit/ http://choosealicense.com/licenses/mit/ Depending on the project, if I really care about the legal status of the code I use, I may contact the author to clarify this kind of thing, but sometimes it is too much hassle. Just look at it as a critical bug in your software, and fix it. On a more personal note I started to dislike the "* for humans", it's been overused.
- deleted 11y ago
  [deleted]
- Socketubs 11y ago
  I'll add license file in repository. Thank you.
guitarbill 10y ago
Python dependency management is a hard problem, but better than most languages [citation needed]. And `pip` and `setup.py` have emerged over several years, with several influences merged in (remember distutils?). Honestly, I wish you'd picked a different tag-line though (riffing on `requests` no doubt). Unlike `requests`, your solution only works for a subset of deployment situations, because - as already pointed out - `setup.py` and `requirements.txt` are for different things. One of the best examples for this I've seen is to use both to deploy to a server with no internet connectivity. For development the dependencies are installed from `setup.py`. Then, before deploying, all dependencies are downloaded via `pip download`. Put the dependencies on the server, finally, use `requirements.txt` with `--no-index` and `--find-links` to install. Definitely an interesting setup, but needs must. Unfortunately, your solution doesn't support `--no-index`, `--find-links` and a few others. You may want to have a look at tools like `pbr` (Python Build Reasonableness) [1], which has an interesting way of dealing with some hard problems. It also shows how to use `setup_requires` so you don't have to have `requirements.py` hanging around in your repo. [1] http://docs.openstack.org/developer/pbr/ http://docs.openstack.org/developer/pbr/
- Socketubs 10y ago
  Thank you! I definitely need to have a look at `pdr`, it looks very interesting. `requirements.py` doesn't stand to be "the" universal solution to Python dependency problem. It's more like a sugar syntax for "simple" setups.
  - guitarbill 10y ago
    That's fair. If you could get installation via `setup_requires` to work, so people don't need a (soon to be out-of-date) copy of `requirements.py` in their repo, that'd be a win imo.
- taion 10y ago
  I split my time pretty evenly between the Python and Node ecosystems, and I've found that Python dependency management is extremely poor compared to Node dependency management. For example, Python's lack of support for nested dependencies means that it can be extremely difficult to use small "utility" libraries like six as subdependencies without running into potential problems from conflicting version requirements from other dependencies. Additionally, Python's management of different types of dependencies is very weak, specifically with regard to setup.py and requirements.txt. npm has (among other things) very nice explicit concepts of development dependencies and regular dependencies, in addition to application shrink-wraps, which gives library maintainers very easy ways to split out different kinds of dependencies, or lock down all dependencies and subdependencies, using the same set of tools. While I can say that the current version of pip, especially in conjunction with pip-tools, is significantly better than earlier iterations of Python packaging, I strongly hold that Python packaging is substantially worse than at least one other prominent example.

78 points by Socketubs 11y ago

This is a nice idea in theory, but setup.py and requirements.txt are not the same, and you cannot particularly generate one from the other: https://caremad.io/2013/07/setup-vs-requirement/ https://caremad.io/2013/07/setup-vs-requirement/ Also, unfortunately pip's parser for requirements.txt isn't public, like much of its internals at the moment, which means AIUI it's likely that the parsing code here is going to be brittle if it uses it, or miss edge cases if it doesn't. (I haven't read this carefully yet though).

yeukhon 11y ago
ds' article has been referenced many times and I have had multiple discussions on #python myself and trust me, no one agree on a single approach. > Also, unfortunately pip's parser for requirements.txt isn't public I am not sure what you mean by "isn't public." The API is "private" or the parser is proprietary? or the API is not isolated to its own (like Ansible's command line parser is stuck inside a giant function). If the second I would be super surprise. But I am with the repo author. I think we just need one format, one way to do things. The implementation however, needs to be discussed as the example code is pretty redundant in the effort, would be much nicer if we can just provide the path in setup.py and pip reads off setup.py and consume requirements.txt. For god sake this is dependency declaration, ain't package management.
- pietroalbini 11y ago
  pip is not meant to be used as an API, and there is no backward compatibility guarantee, AFAIK. There is an ongoing effort to extract all its logic into separated packages [1], thought. [1] https://github.com/pypa/packaging https://github.com/pypa/packaging
thristian 10y ago
As somebody who deploys a lot of business-critical Python apps at $EMPLOYER, I don't even write a requirements.txt. During development and integration testing, I want to get the latest version of my dependencies that I should be compatible with, so I use SemVer matching (somepackage~=1.2.3) in setup.py. For deployment, I want to use the exact dependency packages that the code was tested against. So after automated testing is complete, it runs "pip wheel" to build wheels of my package and all its dependencies and put them into a directory, then that is the artefact I use to recreate the package in production. Much more reliable than requirement.txt, since it doesn't assume that PyPI or internet access is available, and it doesn't care if upstream authors silently update packages without changing the version number. The last scenario is unit-testing, because sometimes tests have extra dependencies the rest of the code doesn't share, like a mock library or other test helper. In theory, these extra things could be put in setup.py's "tests_require" option, but then they'd only be installed if you run tests with "python setup.py test", and they'd also be installed with "easy_install", which is terrible. I'm thinking maybe I should use setuptools' "feature" syntax, so you only get the test dependencies if you install with the [test] feature.
- TheCowboy 10y ago
  Another way to do this, that you or someone may be interested in, is by using pip-tools. You write a requirements.in text file where you list libraries by just their name, and then run "pip-compile requirements.in" and it will output a requirements.txt. It also has a function called "pip-sync" which will then install/update everything in the requirements.txt file. https://github.com/nvie/pip-tools https://github.com/nvie/pip-tools

haldean 11y ago

How would you install a package that depended on this? You can't install it's dependencies automatically until you've got one of the dependencies installed.

Socketubs 11y ago
Like I say in README[0], you have to put `requirements.py` beside your `setup.py` [0]: https://github.com/socketubs/requirements#usage https://github.com/socketubs/requirements#usage

mpdehaan2 11y ago

Ironically, the package to make your package requirements easier to write has additional requirements not in the core distribution :) But more seriously, good idea in thinking that things should already be this way. The dual maintainance of the dependency list between setup.py and requirements.txt unfortunately leads to a lot of packages being left out of one or the other, or a package that works great from source that has a broken setup.py (which usually applies to everything I do). A setup.py command for installing deps from setup.py would, to me, seem a bit more logical - but things already exist the way they do. In the end though, it's a bit weird to deviate from the standard norms of package installation, at least in Python circles, which may introduce some confusion -- contrast this with Javascript circles that replace "the one true way" about every 3 months :)

Socketubs 11y ago
Thank you for your feeback. Same reasons that encourage me to build `requirements` :) I know this isn't perfect for now, `requirements` could evolve!

whalesalad 11y ago

I wish the Python community would adopt a Bundler/Gemfile style. The project.clj concept is also a fantastic approach. Environments should just be maps of data that you can modify or compose on the fly.

reidrac 11y ago

I see a problem with that "License is MIT". That's not how software is licensed and if the author wants this to be used, the legal part must be flawless. Who's the copyright holder? How can I contact that person? What's the copyright year? Licensing software with MIT licence is trivial: http://choosealicense.com/licenses/mit/ http://choosealicense.com/licenses/mit/ Depending on the project, if I really care about the legal status of the code I use, I may contact the author to clarify this kind of thing, but sometimes it is too much hassle. Just look at it as a critical bug in your software, and fix it. On a more personal note I started to dislike the "* for humans", it's been overused.

deleted 11y ago
[deleted]
Socketubs 11y ago
I'll add license file in repository. Thank you.

guitarbill 10y ago

Python dependency management is a hard problem, but better than most languages [citation needed]. And `pip` and `setup.py` have emerged over several years, with several influences merged in (remember distutils?). Honestly, I wish you'd picked a different tag-line though (riffing on `requests` no doubt). Unlike `requests`, your solution only works for a subset of deployment situations, because - as already pointed out - `setup.py` and `requirements.txt` are for different things. One of the best examples for this I've seen is to use both to deploy to a server with no internet connectivity. For development the dependencies are installed from `setup.py`. Then, before deploying, all dependencies are downloaded via `pip download`. Put the dependencies on the server, finally, use `requirements.txt` with `--no-index` and `--find-links` to install. Definitely an interesting setup, but needs must. Unfortunately, your solution doesn't support `--no-index`, `--find-links` and a few others. You may want to have a look at tools like `pbr` (Python Build Reasonableness) [1], which has an interesting way of dealing with some hard problems. It also shows how to use `setup_requires` so you don't have to have `requirements.py` hanging around in your repo. [1] http://docs.openstack.org/developer/pbr/ http://docs.openstack.org/developer/pbr/

Socketubs 10y ago
Thank you! I definitely need to have a look at `pdr`, it looks very interesting. `requirements.py` doesn't stand to be "the" universal solution to Python dependency problem. It's more like a sugar syntax for "simple" setups.
- guitarbill 10y ago
  That's fair. If you could get installation via `setup_requires` to work, so people don't need a (soon to be out-of-date) copy of `requirements.py` in their repo, that'd be a win imo.
taion 10y ago
I split my time pretty evenly between the Python and Node ecosystems, and I've found that Python dependency management is extremely poor compared to Node dependency management. For example, Python's lack of support for nested dependencies means that it can be extremely difficult to use small "utility" libraries like six as subdependencies without running into potential problems from conflicting version requirements from other dependencies. Additionally, Python's management of different types of dependencies is very weak, specifically with regard to setup.py and requirements.txt. npm has (among other things) very nice explicit concepts of development dependencies and regular dependencies, in addition to application shrink-wraps, which gives library maintainers very easy ways to split out different kinds of dependencies, or lock down all dependencies and subdependencies, using the same set of tools. While I can say that the current version of pip, especially in conjunction with pip-tools, is significantly better than earlier iterations of Python packaging, I strongly hold that Python packaging is substantially worse than at least one other prominent example.