6 ms·
That's not on a per-host basis. Shopify's design is, quite fortunately, one that partitions really well, as each store is completely independent of each other.
by mabbo 3y ago
That's not on a per-host basis. Shopify's design is, quite fortunately, one that partitions really well, as each store is completely independent of each other.
Each store can be assigned to one pod, each pod can have as many hosts as it takes to optimize the use of a database instance, and then you can add more pods as the need arises.
Edit: to be clear, that's not to say Rails can't scale. It can. It's just that it doesn't need to- you can scale anything with enough partitioning.
- choilive 3y agoSure.. but isn't that a database problem and not a rails problem?
- mabbo 3y agoNo, they both matter. The database can handle X shops per partition, and the rails host can handle Y shops per partition. If rails were half as fast, you'd need twice as many rails hosts (but no more databases).
- phamilton 3y ago> but no more databases Sort of. Twice as many rails hosts means more DB connections which generally means more load/memory on the DB or more load/memory on the external connection pooler. It's only a bit of incremental load, but it's easy to overlook how many other systems need to run to make Rails scale.
- brianwawok 3y agoNo he’s his examples he’s talking 50 stores over 10 machines vs 50 stores over 5 machines. Both would require the same DB count, but the second would save on server costs for stores.
- mabbo 3y agoI think the above means (if I do get the point) that databases scale in terms of both requests per second and number of active connections. Having 1000 connections doing 1 request per second isn't the same as having 1 connection doing 1000 requests per second. But, and I could be wrong, the larger factor is the number of requests per second.
- brianwawok 3y agoThis entirely depends on the database you are using, and if you are using a connection proxy in front of your database.
- cowsandmilk 3y agoWell, Shopify is investing in optimizing Ruby, so they believe Ruby was the component with the most opportunity for improvement. And the results are showing there was indeed lots of places where improvements could be made.
- elif 3y agoThat's like saying the leading F1 team is investing in optimizing side mirrors, therefore they believe side mirrors are the component with the most opportunity for improvement. The fact is that performance oriented organizations optimize everything unless they have math telling them it isn't worth optimizing. The "weakest link" belief is pure conjecture
- pmontra 3y agoI understand that it's a metaphor but F1 teams did really optimize side mirrors more than once, leading to controversies Mercedes 2022 https://www.the-race.com/formula-1/mercedes-2022-f1-car-makes-ferrari-fear-spaceship-mirrors/ https://www.the-race.com/formula-1/mercedes-2022-f1-car-make... Williams 2019 https://www.autosport.com/f1/news/williams-modifying-front-suspension-and-mirrors-to-ensure-legality-5280436/5280436/ https://www.autosport.com/f1/news/williams-modifying-front-s... Ferrari 2018 https://www.autosport.com/f1/news/how-ferraris-formula-1-mirrors-became-a-talking-point-5320415/5320415/ https://www.autosport.com/f1/news/how-ferraris-formula-1-mir...
- elif 3y agoOf course, most teams do. Some teams have had 2 revisions to side mirrors this season.. and of course it didn't decide the WC, which is my point. Organizations don't merely optimize "the bottleneck" they optimize everything worth optimizing.
- alberth 3y agoNo one disputes that Ruby is slow. But you picking “mirrors” in your analogy makes it sounds like a premature optimization. The reason why perf isn’t typically an issue with Rails is the design pattern is to leverage heavy caching. The use of caching is to address the slowness of Ruby.
- maccard 3y ago>That's not on a per-host basis. I'm sorry, but this is one of the silliest nit picks ive ever seen on this site. Of course 1 million rps and 3TB/s isn't coming from a single host. 3 TB/s is far beyond the throughput of any network I've ever seen, considered or thought of, short of maybe a data center (and I don't work in that domain). 1.3 million requests per second is far beyond the capacity of pretty much any hardware available right now.
- block_dagger 3y agoThe second paragraph explains the argument. Seems spot on and relevant to me.
- dotancohen 3y agoThe point is, it's not Rails that is scaling here.
- paulddraper 3y agoThere are of course a lot of things that cross store partitions, e.g. ShopPay.