5 ms·

It seems surprising that the architecture is spread over so many different service providers. CDN from Fastly, servers from Rackspace, file storage on AWS, data

It seems surprising that the architecture is spread over so many different service providers. CDN from Fastly, servers from Rackspace, file storage on AWS, database from Heroku, ElasticSearch from Elastic, log storage on Dreamhost, DNS from Dyn etc. It seems like a lot of accounts and effort to manage all these, while AWS, Rackspace and Heroku could host most of the required parts on their own. Is there a technical reason for using each of these services from these providers? Or is it just easier to get smaller donations from multiple supporters than all from one source?

donaldstufft 10y ago
It's a few things. One of the simplest reasons is as you identified, it's easier to get smaller donations from multiple people than it is to get one large donation from a single company (although we do have large donations too ranging from 30,000/month and going downwards from there-- for all the PSF infra not just PyPI). Another part of that is a lot of this has grown organically over time and we sought out donations from different providers based on our need at the time. In addition, I can't think of a single company that actually provides everything we need except for maybe Amazon/AWS. On top of all of that the more we centralize our donations onto a single company, the more important a single company becomes to PyPI and the larger the amount of Risk we take on is. It would be a lot harder to find a replacement for all of the things we need all at once than it would be to find a replacement for just a single service. All in all, managing these accounts is not particularly hard (though in some part that's likely because the set of people who has access to any of one of these is pretty static). Most of them provide some sort of standardize API access that doesn't really change based on who is providing said thing (in general, we attempt to rely as much as possible on "Hosted X" where X is some OSS thing we could possibly run ourselves or switch to someone else's "Hosted X" if need be. It's not mandatory but the harder it would be to switch the more we factor that into our decision (for instance, our use of S3 is pretty simple so we don't worry about their proprietary API because it wouldn't be difficult to modify the code to do it differently).
nathancahill 10y ago
You pay a significant premium for AWS/Heroku type services.
bluedino 10y ago
Another issue would be if Company X donated all the resources to run the project, they could theoretically try and influence the project in certain ways. Imagine if Chipmaker X funded some project and wouldn't pay for architecture Y or Z testing or build machines, and when outside providers offered help, Chipmaker X could threaten to remove funding.

donaldstufft 10y ago

It's a few things. One of the simplest reasons is as you identified, it's easier to get smaller donations from multiple people than it is to get one large donation from a single company (although we do have large donations too ranging from 30,000/month and going downwards from there-- for all the PSF infra not just PyPI). Another part of that is a lot of this has grown organically over time and we sought out donations from different providers based on our need at the time. In addition, I can't think of a single company that actually provides everything we need except for maybe Amazon/AWS. On top of all of that the more we centralize our donations onto a single company, the more important a single company becomes to PyPI and the larger the amount of Risk we take on is. It would be a lot harder to find a replacement for all of the things we need all at once than it would be to find a replacement for just a single service. All in all, managing these accounts is not particularly hard (though in some part that's likely because the set of people who has access to any of one of these is pretty static). Most of them provide some sort of standardize API access that doesn't really change based on who is providing said thing (in general, we attempt to rely as much as possible on "Hosted X" where X is some OSS thing we could possibly run ourselves or switch to someone else's "Hosted X" if need be. It's not mandatory but the harder it would be to switch the more we factor that into our decision (for instance, our use of S3 is pretty simple so we don't worry about their proprietary API because it wouldn't be difficult to modify the code to do it differently).

nathancahill 10y ago

You pay a significant premium for AWS/Heroku type services.

bluedino 10y ago

Another issue would be if Company X donated all the resources to run the project, they could theoretically try and influence the project in certain ways. Imagine if Chipmaker X funded some project and wouldn't pay for architecture Y or Z testing or build machines, and when outside providers offered help, Chipmaker X could threaten to remove funding.