15 ms·

It would be useful in this article to hear about what content is acceptable in a merge request. For example: can these all go straight to queue because they use

It would be useful in this article to hear about what content is acceptable in a merge request. For example: can these all go straight to queue because they use feature flags? Are commits a "single piece of work", etc. Not to sound like a downer but this is really an article about fixing a broken process because not running CI on branches before merging to master goes against best practices. Would have loved to actually hear about their work process as this whole article could be summed up as "not running CI on branches before merging their commits to master is a great way to ruin master".

latortuga 7y ago
Well, the problem is that master is a bottleneck. Trying to build CI on every branch before merging to master just won't work with the scale they are dealing with. At 1000 developers, the rate of PRs coming in makes it impossible to determine what current master will be when the PR is ready to merge (i.e. when the branch has a green build). It's also wasteful to build each branch against current master because what is "current" will not be when the branch is ready to merge. Perhaps this problem is what microservices are meant to solve. When you can't coherently integrate code fast enough, attack the bottleneck (master) by splitting it (multiple services).
- scrollaway 7y ago
  > Perhaps this problem is what microservices are meant to solve. Kinda. Microservices have always been an organizational solution; they're a way to shard your company's work output. Usually that's API contracts, but whatever mechanisms are bottlenecked on the work output is affected, including how many concurrent builds are running due to how many people are touching the code at the same time.
- tudelo 7y ago
  Yet it seems like large companies mostly prefer monorepos, so while it takes investment to have such a monorepo, it seems the benefits are worth the investment.
  - peterwwillis 7y ago
    This is appeal to accomplishment fallacy. Because large companies have a lot of money, whatever they do must be great. But this is false - they do what they do because they are large companies, not because it is a good idea. At scale, managing complexity can require either a lot of coordination, or a lot of careful planning. Large companies (especially tech companies) don't do either well, so they pick architectures that remove choices, and iterate on them until they are workable. And they have the money and workforce to do it.
  - anon73044 7y ago
    Google, Microsoft, Facebook and Twitter prefer monorepos but this is not indicative of most large orgs. You'll notice that those listed have had to customize or creat new vcs's to meet their needs. https://news.ycombinator.com/item?id=17605371 https://news.ycombinator.com/item?id=17605371 https://news.ycombinator.com/item?id=11789182 https://news.ycombinator.com/item?id=11789182 https://medium.com/@maoberlehner/monorepos-in-the-wild-33c6eb246cb9 https://medium.com/@maoberlehner/monorepos-in-the-wild-33c6e...
- carlisle_ 7y ago
  This paper might be of interest to you on this very subject: https://eng.uber.com/research/keeping-master-green-at-scale/ https://eng.uber.com/research/keeping-master-green-at-scale/
- adrianN 7y ago
  Microservices don't really help with this. They just force you to think about your interfaces, but you should do that in a monolith too. If you interfaces are reasonably stable, merging is unlikely to break master if the branch was green before, if your interfaces change rapidly you get problems with microservices too, just one level higher up, where you try to integrate them into a usable product.
  - gav 7y ago
    One of the things I think microservices does help with is thinking about systems being composed of components that are being developed at different velocities and different tolerances of risk. Imagine an e-commerce site broken into a bunch of services including search and checkout. The search team is making updates daily, trying to improve ranking and drive conversion. The checkout team (assuming that the site is mature and has hit some design equilibrium) may only be releasing changes every couple of months, and if a bug is introduced, the financial impact is a lot higher. By not bundling the outputs of very different teams together, you can help those that want to "move fast and break things" with their "moving fast" goal, and de-risk breaking everything by reducing the surface area of changes. Microservices-based architectures are a way to reduce friction caused by the structure of your organization and is one outcome of an Inverse Conway Maneuver.
  - isaacaggrey 7y ago
    They do help if a single team of 5-7 developers own a set of microservices; it's unlikely you will have tons of PRs to merge all at once in a single repository with a smaller team. Granted, the ownership is is a bit more clear when talking about a self-contained system that a team owns: https://scs-architecture.org/vs-ms.html https://scs-architecture.org/vs-ms.html In the SCS literature, you would integrate via async mechanisms across SCSes, provide versioned interfaces, and enforce via consumer-driven contract testing like Pact: https://pact.io https://pact.io
- jrockway 7y ago
  We didn't have a merge queue at Google. You rebased if there was a merge conflict, ran through CI again, and hoped there wasn't another merge conflict. I think I ran into merge conflicts maybe once a year, if that. I think the success of this system breaks down into several parts: 1) Yup, microservices. You could submit your proto change, which would affect all clients, before actually implementing the code that used the new feature. (Or after, in the case of renaming some field from foo to deprecated_foo and refactoring the clients to stop using that field.) That means you could wrangle that change without having to worry about it affecting your actual feature. (Typically proto changes did not cause any breakages since people were very conservative about what changes they would make. Nobody renames all the fields, invalidating dependent code, or renumbers the fields, invalidating all existing messages. You COULD do those things, but nobody ever did.) 2) Clear dependencies in the build system. The CI system only had to run a small set of tests for most changes, because it knew exactly what tests the change would affect. You had to go way out of your way to depend on code without informing the build system. This is very different from every CI system that I've seen outside of Google, which seem to default to running everything and hoping your programming language or build system magically tracks dependencies. It doesn't; Docker for example will happily use random images that it thinks haven't changed, without actually checking if it has changed. (Consider building your app on top of golang:latest. Go is updated, and docker may or may not pull that new base image. Meanwhile, docker will happily clear its build cache if you edit README.md and no code. The result is that 50% of the time you waste 10 minutes rebuilding stuff that didn't change, and 50% of the time you get an outdated build. And nobody seems to care at all!) 3) Being careful about keeping changes small. I don't know what the average CL size is, but I would aim for 100 lines changed rather than 1000 lines changed. This is something that surprised me post-Google, people go away and work for a week and you have a 2000 line PR to review. These are tough to merge and were relatively rare in my experience at Google. It is not always possible to make every change small, but that should be the norm. Figure out how much work you can do in a day, and try to make a CL/PR that is that size. A lot can churn in a week. A lot less churns in a day. If you respected steps 1 and 2, that means your tests will run fast and it's unlikely that your merge will fail between CI and actually merging. If you have 2000 lines of code across 8 services... you'll probably never get it merged. But I am sure that I have successfully merged ginormous changes before, it's just more work. All in all, my takeaway from this article is that Shopify is huge but I'm surprised that specialized merge tooling was necessary. I wonder what the underlying problem is; do they really have a 1000 developer monolith? Do they not use a proper build system like Bazel?
- username90 7y ago
  > Trying to build CI on every branch before merging to master just won't work with the scale they are dealing with. Google does it with 50 times the developer count. > At 1000 developers, the rate of PRs coming in makes it impossible to determine what current master will be when the PR is ready to merge (i.e. when the branch has a green build). True, it is impossible to catch all errors like this, but you can catch almost every error by building and testing it against current master and then merge it with the master 20 minutes later when the build is done. I have seen maybe one build breakage a year being introduced due to this in projects I've worked on, so it isn't a big deal.
  - greiskul 7y ago
    > building and testing it against current master and then merge it with the master 20 minutes later when the build is done. And I'm pretty sure that is the way Google does it too. Test a commit against current master, if tests are green commit. Then run tests against master again (and I think this stage might not run for every single commit) to see if anything broke on the rare times there was an actual conflict. If that run was red, which should be rare, then you can have the system do a bisect to find the offending commit, or just run all the ones that haven't been individually tested.
  - iamweswilson 7y ago
    For even better accuracy you can use a tool that will run tests against speculative merge states. Zuul[1] is an open source project that supports it out of the box. [1] https://zuul-ci.org/docs/zuul/user/gating.html https://zuul-ci.org/docs/zuul/user/gating.html
  - randomidiot666 7y ago
    You have no idea how Google solved it. Basically everyone with a Monorepo (except Google) implements it as a cargo cult best practice. Mindlessly copying Google without understanding how Google actually does it.
    - deleted 7y ago
      [deleted]
- marcosdumay 7y ago
  This is the problem external libraries were created to solve, in a time when it was a much harder problem. Microservices are the same kind of solution, with the same gains and costs for this specific problem.
- bob1029 7y ago
  By virtue of having a queue of PRs that need to test & merge, you could pipeline this thing out pretty substantially. The implication here being that a queue must be processed in-order, so you will ultimately have a perfect sequence of future commits to speculate against, and can incrementally build up each hypothetical future master state for a test build on one of any number of parallel build agents. As the queue depth grows, you would see higher and higher throughput.
tekstar 7y ago
Shopify always runs CI on branches before merging to master. Everything this article describes is in addition to that, in order to deal with the problems the article talks about at "merge to master" time, like 2 merged PRs failing or a stale PR that passed on branch but fails on master due changes. At this scale you need to be deploying constantly, otherwise deploys are hundreds of commits large and its impossible to triage - what PR in the deploy broke something, is it even safe to rollback, etc. That is the primary reason to automate deploys and manage the deploy queue.
- dkoston 7y ago
  Sorry if my comment was unclear. I consider the queue to be a “branch” as well. Many people use a “develop” branch instead of a queue in this instance. The queue appears designed to allow arbitrary selection rather than merging in order (though the new solution with CD seems generally in order) Totally agree that CD is required with this many commits. It’s commonplace on teams with many fewer developers. Was surprised to see you folks roll your own workflows rather than using other systems. Would also be interesting to see if you tag commits that go to master in instrumentation systems so you have visibility into production metrics and can correlate them with what code was running at the time.
  - wvanbergen 7y ago
    Generally our metrics and exception reports are tagged with the sha and the deploy stage.
    - dkoston 7y ago
      Good to hear, that’ll make change management less of a chore. I think the main thing that was missing for me is the rationale behind building this system rather than building a workflow in one of the existing CI/CD tools. Was there a throughout bottleneck in existing tools? Was there something custom about your workflow that wasn’t supported elsewhere? I may be wrong but the workflow you landed upon seems pretty common so I’m curious as to why the need to build and maintain a tool in house for this?
rb808 7y ago
Literally the definition of CI is to run on master or release branches a few times a day, not on every dev branch. https://en.wikipedia.org/wiki/Continuous_integration https://en.wikipedia.org/wiki/Continuous_integration
- judge2020 7y ago
  The definition is sourced from here[0], and says "Each check-in is then verified by an automated build, allowing teams to detect problems early.". It's not a hard rule to "not run on every dev branch". "continuous integration" is often just "npm ci && npm run build", and sometimes "npm run test" (or similar for your language). For products that don't make any remote API calls (or when they use a faker service), most of this is done on the same machine and costs very little to do on every commit, making it easier to precisely define which commit broke something. 0: https://www.thoughtworks.com/continuous-integration https://www.thoughtworks.com/continuous-integration
- dkoston 7y ago
  Rather than being pedantic about the definition, maybe you could share your experiences with why that’s superior to validating each branch? Being dogmatic about a definition rather than experimenting with works best in production at your business seems illogical.
jacktli 7y ago
Hi, Author here! Pull requests are our unit of work, and the queue was created to support all pull requests. We do have feature flags as a tool, but we let our developers make the judgment call on how their changes should be rolled out.
- dkoston 7y ago
  Interesting. It seems like you have a very flexible process of how to launch code which could contribute to issues with visibility and rollbacks. I’m curious as to why you had a queue instead of a develop branch before moving to CD? Was this to allow arbitrary commits to be launched to production rather than getting them batched by time?
  - byroot 7y ago
    The queue is simply an automated "develop" branch.
    - dkoston 7y ago
      From what I gathered in the article, that’s the case now but before the queue required manual merges.
      - byroot 7y ago
        No, even with v1, the merge weren't manual. A bot would merge for you, but directly into master. Now the bot merges into a temporary that is fast-forwarded as the new master if CI validates it.
        dkoston 7y ago
        Interesting. Would you say this is more of a decision based around the constraints of using GitHub or more of the ideal process for Shopify’s needs? I’m curious because the article doesn’t mention the core reasons that you chose to write your own CD tool versus the other options that exist. The workflow you describe seems readily available in most tools. Perhaps the throughput was causing other options to break?
lreeves 7y ago
We (Shopify) still run the full CI on each development branch as the article mentions: > We check if Branch CI has passed and if the pull request has been approved by a reviewer before adding the pull request to the queue
peterwwillis 7y ago
> Trying to build CI on every branch before merging to master just won't work with the scale they are dealing with. At 1000 developers, the rate of PRs coming in makes it impossible to determine what current master will be when the PR is ready to merge (i.e. when the branch has a green build). It's also wasteful to build each branch against current master because what is "current" will not be when the branch is ready to merge. I'm starting to think most CI problems are just people not looking at the problem the right way. Here is the problem re-worded: - When a PR has a green light and someone hits 'merge', it locks anything else from being to merge to master, and you merge your PR. When it finishes merging and deploying, now all the other PRs waiting have to rebuild themselves to see if they will merge with this new state of master. So 100s of PRs are rebuilding every time you merge one PR, and there's constant CI churn. Here is why that problem exists: - The system was designed for 1000 developers to all be writing to the same code base. Here is how you solve that: - Don't let 1000 developers all write to the same code base. Break the code down into discrete components that different small teams manage. The only bottleneck for that code base is that small team. This small team is often called the two-pizza team, and their discrete components are often called microservices.
- robocat 7y ago
  Google don't solve it your way: https://news.ycombinator.com/item?id=21586180 https://news.ycombinator.com/item?id=21586180
  - peterwwillis 7y ago
    Yes, that's correct, Google invented its own proprietary distributed object store and distributed version control system and distributed Linux-only filesystem and distributed build-and-test-system to work with a single SDLC that its entire company must follow strictly to release anything, just so it could keep using a single repository. What's your point?
    - robocat 7y ago
      Clearly given those costs, Google really believe in mono-repo, and presumably they have tried to back it up with internal stats? Although hard to get stats without control group - maybe control group could be acquisitions?

latortuga 7y ago

Well, the problem is that master is a bottleneck. Trying to build CI on every branch before merging to master just won't work with the scale they are dealing with. At 1000 developers, the rate of PRs coming in makes it impossible to determine what current master will be when the PR is ready to merge (i.e. when the branch has a green build). It's also wasteful to build each branch against current master because what is "current" will not be when the branch is ready to merge. Perhaps this problem is what microservices are meant to solve. When you can't coherently integrate code fast enough, attack the bottleneck (master) by splitting it (multiple services).

scrollaway 7y ago
> Perhaps this problem is what microservices are meant to solve. Kinda. Microservices have always been an organizational solution; they're a way to shard your company's work output. Usually that's API contracts, but whatever mechanisms are bottlenecked on the work output is affected, including how many concurrent builds are running due to how many people are touching the code at the same time.
tudelo 7y ago
Yet it seems like large companies mostly prefer monorepos, so while it takes investment to have such a monorepo, it seems the benefits are worth the investment.
- peterwwillis 7y ago
  This is appeal to accomplishment fallacy. Because large companies have a lot of money, whatever they do must be great. But this is false - they do what they do because they are large companies, not because it is a good idea. At scale, managing complexity can require either a lot of coordination, or a lot of careful planning. Large companies (especially tech companies) don't do either well, so they pick architectures that remove choices, and iterate on them until they are workable. And they have the money and workforce to do it.
- anon73044 7y ago
  Google, Microsoft, Facebook and Twitter prefer monorepos but this is not indicative of most large orgs. You'll notice that those listed have had to customize or creat new vcs's to meet their needs. https://news.ycombinator.com/item?id=17605371 https://news.ycombinator.com/item?id=17605371 https://news.ycombinator.com/item?id=11789182 https://news.ycombinator.com/item?id=11789182 https://medium.com/@maoberlehner/monorepos-in-the-wild-33c6eb246cb9 https://medium.com/@maoberlehner/monorepos-in-the-wild-33c6e...
carlisle_ 7y ago
This paper might be of interest to you on this very subject: https://eng.uber.com/research/keeping-master-green-at-scale/ https://eng.uber.com/research/keeping-master-green-at-scale/
adrianN 7y ago
Microservices don't really help with this. They just force you to think about your interfaces, but you should do that in a monolith too. If you interfaces are reasonably stable, merging is unlikely to break master if the branch was green before, if your interfaces change rapidly you get problems with microservices too, just one level higher up, where you try to integrate them into a usable product.
- gav 7y ago
  One of the things I think microservices does help with is thinking about systems being composed of components that are being developed at different velocities and different tolerances of risk. Imagine an e-commerce site broken into a bunch of services including search and checkout. The search team is making updates daily, trying to improve ranking and drive conversion. The checkout team (assuming that the site is mature and has hit some design equilibrium) may only be releasing changes every couple of months, and if a bug is introduced, the financial impact is a lot higher. By not bundling the outputs of very different teams together, you can help those that want to "move fast and break things" with their "moving fast" goal, and de-risk breaking everything by reducing the surface area of changes. Microservices-based architectures are a way to reduce friction caused by the structure of your organization and is one outcome of an Inverse Conway Maneuver.
- isaacaggrey 7y ago
  They do help if a single team of 5-7 developers own a set of microservices; it's unlikely you will have tons of PRs to merge all at once in a single repository with a smaller team. Granted, the ownership is is a bit more clear when talking about a self-contained system that a team owns: https://scs-architecture.org/vs-ms.html https://scs-architecture.org/vs-ms.html In the SCS literature, you would integrate via async mechanisms across SCSes, provide versioned interfaces, and enforce via consumer-driven contract testing like Pact: https://pact.io https://pact.io
jrockway 7y ago
We didn't have a merge queue at Google. You rebased if there was a merge conflict, ran through CI again, and hoped there wasn't another merge conflict. I think I ran into merge conflicts maybe once a year, if that. I think the success of this system breaks down into several parts: 1) Yup, microservices. You could submit your proto change, which would affect all clients, before actually implementing the code that used the new feature. (Or after, in the case of renaming some field from foo to deprecated_foo and refactoring the clients to stop using that field.) That means you could wrangle that change without having to worry about it affecting your actual feature. (Typically proto changes did not cause any breakages since people were very conservative about what changes they would make. Nobody renames all the fields, invalidating dependent code, or renumbers the fields, invalidating all existing messages. You COULD do those things, but nobody ever did.) 2) Clear dependencies in the build system. The CI system only had to run a small set of tests for most changes, because it knew exactly what tests the change would affect. You had to go way out of your way to depend on code without informing the build system. This is very different from every CI system that I've seen outside of Google, which seem to default to running everything and hoping your programming language or build system magically tracks dependencies. It doesn't; Docker for example will happily use random images that it thinks haven't changed, without actually checking if it has changed. (Consider building your app on top of golang:latest. Go is updated, and docker may or may not pull that new base image. Meanwhile, docker will happily clear its build cache if you edit README.md and no code. The result is that 50% of the time you waste 10 minutes rebuilding stuff that didn't change, and 50% of the time you get an outdated build. And nobody seems to care at all!) 3) Being careful about keeping changes small. I don't know what the average CL size is, but I would aim for 100 lines changed rather than 1000 lines changed. This is something that surprised me post-Google, people go away and work for a week and you have a 2000 line PR to review. These are tough to merge and were relatively rare in my experience at Google. It is not always possible to make every change small, but that should be the norm. Figure out how much work you can do in a day, and try to make a CL/PR that is that size. A lot can churn in a week. A lot less churns in a day. If you respected steps 1 and 2, that means your tests will run fast and it's unlikely that your merge will fail between CI and actually merging. If you have 2000 lines of code across 8 services... you'll probably never get it merged. But I am sure that I have successfully merged ginormous changes before, it's just more work. All in all, my takeaway from this article is that Shopify is huge but I'm surprised that specialized merge tooling was necessary. I wonder what the underlying problem is; do they really have a 1000 developer monolith? Do they not use a proper build system like Bazel?
username90 7y ago
> Trying to build CI on every branch before merging to master just won't work with the scale they are dealing with. Google does it with 50 times the developer count. > At 1000 developers, the rate of PRs coming in makes it impossible to determine what current master will be when the PR is ready to merge (i.e. when the branch has a green build). True, it is impossible to catch all errors like this, but you can catch almost every error by building and testing it against current master and then merge it with the master 20 minutes later when the build is done. I have seen maybe one build breakage a year being introduced due to this in projects I've worked on, so it isn't a big deal.
- greiskul 7y ago
  > building and testing it against current master and then merge it with the master 20 minutes later when the build is done. And I'm pretty sure that is the way Google does it too. Test a commit against current master, if tests are green commit. Then run tests against master again (and I think this stage might not run for every single commit) to see if anything broke on the rare times there was an actual conflict. If that run was red, which should be rare, then you can have the system do a bisect to find the offending commit, or just run all the ones that haven't been individually tested.
- iamweswilson 7y ago
  For even better accuracy you can use a tool that will run tests against speculative merge states. Zuul[1] is an open source project that supports it out of the box. [1] https://zuul-ci.org/docs/zuul/user/gating.html https://zuul-ci.org/docs/zuul/user/gating.html
- randomidiot666 7y ago
  You have no idea how Google solved it. Basically everyone with a Monorepo (except Google) implements it as a cargo cult best practice. Mindlessly copying Google without understanding how Google actually does it.
  - deleted 7y ago
    [deleted]
marcosdumay 7y ago
This is the problem external libraries were created to solve, in a time when it was a much harder problem. Microservices are the same kind of solution, with the same gains and costs for this specific problem.
bob1029 7y ago
By virtue of having a queue of PRs that need to test & merge, you could pipeline this thing out pretty substantially. The implication here being that a queue must be processed in-order, so you will ultimately have a perfect sequence of future commits to speculate against, and can incrementally build up each hypothetical future master state for a test build on one of any number of parallel build agents. As the queue depth grows, you would see higher and higher throughput.

tekstar 7y ago

Shopify always runs CI on branches before merging to master. Everything this article describes is in addition to that, in order to deal with the problems the article talks about at "merge to master" time, like 2 merged PRs failing or a stale PR that passed on branch but fails on master due changes. At this scale you need to be deploying constantly, otherwise deploys are hundreds of commits large and its impossible to triage - what PR in the deploy broke something, is it even safe to rollback, etc. That is the primary reason to automate deploys and manage the deploy queue.

dkoston 7y ago
Sorry if my comment was unclear. I consider the queue to be a “branch” as well. Many people use a “develop” branch instead of a queue in this instance. The queue appears designed to allow arbitrary selection rather than merging in order (though the new solution with CD seems generally in order) Totally agree that CD is required with this many commits. It’s commonplace on teams with many fewer developers. Was surprised to see you folks roll your own workflows rather than using other systems. Would also be interesting to see if you tag commits that go to master in instrumentation systems so you have visibility into production metrics and can correlate them with what code was running at the time.
- wvanbergen 7y ago
  Generally our metrics and exception reports are tagged with the sha and the deploy stage.
  - dkoston 7y ago
    Good to hear, that’ll make change management less of a chore. I think the main thing that was missing for me is the rationale behind building this system rather than building a workflow in one of the existing CI/CD tools. Was there a throughout bottleneck in existing tools? Was there something custom about your workflow that wasn’t supported elsewhere? I may be wrong but the workflow you landed upon seems pretty common so I’m curious as to why the need to build and maintain a tool in house for this?

rb808 7y ago

Literally the definition of CI is to run on master or release branches a few times a day, not on every dev branch. https://en.wikipedia.org/wiki/Continuous_integration https://en.wikipedia.org/wiki/Continuous_integration

judge2020 7y ago
The definition is sourced from here[0], and says "Each check-in is then verified by an automated build, allowing teams to detect problems early.". It's not a hard rule to "not run on every dev branch". "continuous integration" is often just "npm ci && npm run build", and sometimes "npm run test" (or similar for your language). For products that don't make any remote API calls (or when they use a faker service), most of this is done on the same machine and costs very little to do on every commit, making it easier to precisely define which commit broke something. 0: https://www.thoughtworks.com/continuous-integration https://www.thoughtworks.com/continuous-integration
dkoston 7y ago
Rather than being pedantic about the definition, maybe you could share your experiences with why that’s superior to validating each branch? Being dogmatic about a definition rather than experimenting with works best in production at your business seems illogical.

jacktli 7y ago

Hi, Author here! Pull requests are our unit of work, and the queue was created to support all pull requests. We do have feature flags as a tool, but we let our developers make the judgment call on how their changes should be rolled out.

dkoston 7y ago
Interesting. It seems like you have a very flexible process of how to launch code which could contribute to issues with visibility and rollbacks. I’m curious as to why you had a queue instead of a develop branch before moving to CD? Was this to allow arbitrary commits to be launched to production rather than getting them batched by time?
- byroot 7y ago
  The queue is simply an automated "develop" branch.
  - dkoston 7y ago
    From what I gathered in the article, that’s the case now but before the queue required manual merges.
    - byroot 7y ago
      No, even with v1, the merge weren't manual. A bot would merge for you, but directly into master. Now the bot merges into a temporary that is fast-forwarded as the new master if CI validates it.
      - dkoston 7y ago
        Interesting. Would you say this is more of a decision based around the constraints of using GitHub or more of the ideal process for Shopify’s needs? I’m curious because the article doesn’t mention the core reasons that you chose to write your own CD tool versus the other options that exist. The workflow you describe seems readily available in most tools. Perhaps the throughput was causing other options to break?

lreeves 7y ago

We (Shopify) still run the full CI on each development branch as the article mentions: > We check if Branch CI has passed and if the pull request has been approved by a reviewer before adding the pull request to the queue

peterwwillis 7y ago

> Trying to build CI on every branch before merging to master just won't work with the scale they are dealing with. At 1000 developers, the rate of PRs coming in makes it impossible to determine what current master will be when the PR is ready to merge (i.e. when the branch has a green build). It's also wasteful to build each branch against current master because what is "current" will not be when the branch is ready to merge. I'm starting to think most CI problems are just people not looking at the problem the right way. Here is the problem re-worded: - When a PR has a green light and someone hits 'merge', it locks anything else from being to merge to master, and you merge your PR. When it finishes merging and deploying, now all the other PRs waiting have to rebuild themselves to see if they will merge with this new state of master. So 100s of PRs are rebuilding every time you merge one PR, and there's constant CI churn. Here is why that problem exists: - The system was designed for 1000 developers to all be writing to the same code base. Here is how you solve that: - Don't let 1000 developers all write to the same code base. Break the code down into discrete components that different small teams manage. The only bottleneck for that code base is that small team. This small team is often called the two-pizza team, and their discrete components are often called microservices.

robocat 7y ago
Google don't solve it your way: https://news.ycombinator.com/item?id=21586180 https://news.ycombinator.com/item?id=21586180
- peterwwillis 7y ago
  Yes, that's correct, Google invented its own proprietary distributed object store and distributed version control system and distributed Linux-only filesystem and distributed build-and-test-system to work with a single SDLC that its entire company must follow strictly to release anything, just so it could keep using a single repository. What's your point?
  - robocat 7y ago
    Clearly given those costs, Google really believe in mono-repo, and presumably they have tried to back it up with internal stats? Although hard to get stats without control group - maybe control group could be acquisitions?