6 ms·

How are you hosting the wheels internally? Are you still using an internal pypi instance but with wheels instead of sdists? I've been looking at doing somethin

How are you hosting the wheels internally? Are you still using an internal pypi instance but with wheels instead of sdists? I've been looking at doing something similar in our environment but there's so many options I haven't figured out what the best and most straightforward way might be.

SEJeff 11y ago
Your easiest bet is probably devpi, but you can download the cheeseshop code that powers pypi as well. I'm using cheeseshop, but some people swear by warehouse, which is supposedly a legacy-free version for running pypi eventually. If you don't care about the search api, you can also just enable an directory listing index page and use any web server. Pip will do the right thing when given the right incantation of magical arguments and you make a prayer to the pip gods.
benesch 11y ago
Shameless plug: I had the exact same question at my last gig, and wrote up a quick open-source tool to build wheels and dump them to an S3 bucket. [0] Usage is as simple as mkwheelhouse mybucket.mycorp.co scipy numpy which will automatically build and upload those wheels for your current architecture and dump them into mybucket.mycorp.co. It builds a pip-compatible index, too, so you can just tell pip to search your wheelhouse first before falling back to PyPI: pip install --find-links mybucket.mycorp.co scipy numpy If you need to build for several OSes, you can run mkwheelhouse against the same bucket from each OS. The downside of this approach is you can't host private packages, because you need to enable public website hosting. (Although, VPC endpoints might have changed this!) But the simplicity of this approach plus the massive speedup of not needing to constantly recompile scipy was totally worth it. [0]: https://github.com/whoopinc/mkwheelhouse https://github.com/whoopinc/mkwheelhouse
soofaloofa 11y ago
Another shameless plug: At my previous job I wrote a basic PyPI server running on App Engine with packages stored in Cloud Storage. You get the benefits and drawbacks of Google Cloud Platform. https://github.com/vendasta/cloudpypi/tree/master https://github.com/vendasta/cloudpypi/tree/master
jaza 11y ago
More plugging unashamedly: I wrote a guide for what is probably the quickest and simplest way to get started with a private pypi instance - just Apache with directory autoindex, and wheels that are manually-uploaded to the server: https://gist.github.com/Jaza/fcea493dd0ba6ebf09d3 https://gist.github.com/Jaza/fcea493dd0ba6ebf09d3
Kaizyn 11y ago
Use PyPICloud or DevPI to host your built binaries. Wheels have another advantage that you don't install any dev tools in your production environments (no need for compiler tools). The only annoying thing is that there isn't any way to distinguish between different Linux distributions, so hopefully all your servers are standardized.

SEJeff 11y ago

Your easiest bet is probably devpi, but you can download the cheeseshop code that powers pypi as well. I'm using cheeseshop, but some people swear by warehouse, which is supposedly a legacy-free version for running pypi eventually. If you don't care about the search api, you can also just enable an directory listing index page and use any web server. Pip will do the right thing when given the right incantation of magical arguments and you make a prayer to the pip gods.

benesch 11y ago

Shameless plug: I had the exact same question at my last gig, and wrote up a quick open-source tool to build wheels and dump them to an S3 bucket. [0] Usage is as simple as mkwheelhouse mybucket.mycorp.co scipy numpy which will automatically build and upload those wheels for your current architecture and dump them into mybucket.mycorp.co. It builds a pip-compatible index, too, so you can just tell pip to search your wheelhouse first before falling back to PyPI: pip install --find-links mybucket.mycorp.co scipy numpy If you need to build for several OSes, you can run mkwheelhouse against the same bucket from each OS. The downside of this approach is you can't host private packages, because you need to enable public website hosting. (Although, VPC endpoints might have changed this!) But the simplicity of this approach plus the massive speedup of not needing to constantly recompile scipy was totally worth it. [0]: https://github.com/whoopinc/mkwheelhouse https://github.com/whoopinc/mkwheelhouse

soofaloofa 11y ago

Another shameless plug: At my previous job I wrote a basic PyPI server running on App Engine with packages stored in Cloud Storage. You get the benefits and drawbacks of Google Cloud Platform. https://github.com/vendasta/cloudpypi/tree/master https://github.com/vendasta/cloudpypi/tree/master

jaza 11y ago

More plugging unashamedly: I wrote a guide for what is probably the quickest and simplest way to get started with a private pypi instance - just Apache with directory autoindex, and wheels that are manually-uploaded to the server: https://gist.github.com/Jaza/fcea493dd0ba6ebf09d3 https://gist.github.com/Jaza/fcea493dd0ba6ebf09d3

Kaizyn 11y ago

Use PyPICloud or DevPI to host your built binaries. Wheels have another advantage that you don't install any dev tools in your production environments (no need for compiler tools). The only annoying thing is that there isn't any way to distinguish between different Linux distributions, so hopefully all your servers are standardized.