8 ms·

The Continuous Delivery Test

87 points by sourceless 4y ago

UweSchmidt 4y ago
> 9. Does your Infrastructure as Code live alongside the service it hosts? That means minor changes to some test infrastructure have to go through the strict review process, because the gitlab-ci.yaml is in the part of the main code. Last time this happened to me I found it annoying, and I don't think the code quality guardians care about some CI config anyway.
- davidmurdoch 4y ago
  Changes to CI config for release automation are probably vetted and manually tested more than anything else in the projects I work on.
- xorcist 4y ago
  I will be first in line to suggest to not split repos unless required. It is my experience that deployment, admin and node provisioning (and everything that is generally put under the not always useful IaaC umbrella term) is one of those exceptions. Often documentation is, too. It all depends on what the release flow looks like. A useful rule of thumb when components can share a repository is when the branch model is identical. If that's the case for codified release and provisioning processes and the code that is shipped, then it is likely that you are not taking full advantage of what that deployment and provisioning code can do. If not anything else, then at least test versions must deploy production code, and test code deploy production versions. Again, in my experience across multiple organizations, it is a good idea to keep that code both forwards and backwards compatible. It should be natural that is sometimes says things like "if build number greater than x then set parameter y". While it may feel a bit dirty for some, that type of logic is what deployment code does, and it is much more maintainable than keeping branches around for the same result. If I get to say one thing to new devops/release engineer type people, that's probably it. That, and the value of clear and concise commit messages. Which goes double for this type of code.
dijit 4y ago
This feels like yet more cargo-culting. Even the first item on this list is hotly contested, with Facebook/Microsoft/Google and Co. using centralised mono-repos, google itself using a server model similar to Perforce called Piper. To be clear here, I’m not saying they’re right, but with it feels un-scientific to make a blanket statement that they’re wrong.
- theoldlove 4y ago
  You can easily have a monorepo with a distributed control system.
  - dijit 4y ago
    Yes. But I’m referring to the article: > 1. Do you use a distributed version control system? Edit: the parent originally said “distributed build system”
    - chrisoverzero 4y ago
      Read this as “Do you use git?” This is opposed to svn, cvs, something home-grown, or passing thumb drives back and forth among developers.
      - maccard 4y ago
        Hg, svn, perforce, plastic are all feasible alternatives to git.
    - Morgawr 4y ago
      Just because it's a monorepo it doesn't mean it's centralized?
      - michaelt 4y ago
        Yes. It’s the accompanying suggestion of using github or gitlab that means it’s centralised.
alkonaut 4y ago
Extremely web centric list. Any other development (mobile, desktop, games, embedded, …) will find a lot of these steps very weird.
- kqr 4y ago
  Really? I challenge you to find any items other than 7 and perhaps 8 that don't apply to the types of development you mention. Essentially the only thing that's different with the types of development you mention is that "final deployment to production" looks different, as it usually involves more or less physically transporting artifacts to your customer. But the rest is just the same. You should be doing trunk-based development, practise something like code review, gate on integration tests, feature flag, spin up virtual production clones, etc. In other words, the fact that getting software to your end users is a clumsy process does not preclude you from doing every other step of the process with short feedback loops.
  - maccard 4y ago
    Using distributed version control isn't applicable to most people working in games. 9) is also debatable - requiring someone to clone the entire application to make an infra change to a testing environment. Adding tickets to commit messages isn't necessarily a requirement - some work (at least in my area) is prototypey and maybe ill defined (the task might be to define it). Being able to deploy from your own machine is a double edged sword; the situation you need this is an absolute last resort. Enabling deployments from dev machines means credentials to environments, write access to infra, and likely skirting around normal processes.
    - KronisLV 4y ago
      > Using distributed version control isn't applicable to most people working in games. Are you sure about this? I mean, code absolutely should be versioned and if you can afford the storage, you should also version all of your assets, like models and audio. Even Git has specialized functionality for storing binary assets: https://git-lfs.github.com/ https://git-lfs.github.com/ Unity also seems to be pushing their Plastic SCM as well for this more specialized use case: https://www.plasticscm.com/ https://www.plasticscm.com/ Either way, not using version control for any collaborative project is just asking for issues. Not using distributed version control in particular might just make things more annoying, as anyone who has ever worked with SVN might attest to.
bhawks 4y ago
> 11. Do you include ticket IDs in your commits or branches? This is one of the things that sound 'good' on the surface and are totally worthless in practice (especially when practicing minimal commits). In the wild this devolves into a meaningless umbrella issue (Deliver Feature Foo) or 1 liner issues that sometimes did not even match the pr contents. Just use the commit message and in pr review enforce commit message norms.
- kqr 4y ago
  My previous team used to have ticket IDs on every commit but just decided to drop that requirement for commits that are truly self-contained and documented properly in the commit message. That approach makes a lot of sense to me. Link the commit to the ticket when it enhances future reading, but not out of blind application of principle.
- whakim 4y ago
  It's incredibly easy to simply add the ticket number to branch names (or commits if you're rebasing your branches when you merge). Many times the ticket isn't helpful, but "just make the commit/branch self-explanatory" ignores the fact that non-developers - product folks, designers, etc. - are far more likely to leave useful context in the ticket itself.
  - lelandfe 4y ago
    Yes, exactly. I treat my commit history like I'm preparing for a detective to investigate years down the line :) Great commit messages are just one part of the timeline. Ensuring there's a link between the commits/PR and the ticket lets the future detective get more information (frequently, that's "why did we do this").
  - kqr 4y ago
    When there is a ticket, yes, adding the number is trivial and probably worth it. Where the rule breaks down, in my opinion, is when you have to create a ticket and replicate the commit message in it any time you stumble over a code maintainability issue and fix it in stride.
satyrnein 4y ago
I'd like to see a workflow that meets all the criteria. I was surprised to see the item about deploying directly from your own machine; that seems to contradict the other goals which point to automated pipelines that deploy on merge.
- TheP1000 4y ago
  This is a very common pattern especially with the cdk. Having a locally deployed stack makes development in the cloud so much easier.
- kqr 4y ago
  I feel like a lot of people misread that point. It's not about deploying to production (that sounds nuts!) It's about spinning up and deploying to a full dev environment. This is in part to not have to run everything locally, but mainly to exercise deploy automation regularly.
etamponi 4y ago
7. 7! This is by far the one that resonates with me the most. But for some reasons SREs think the opposite is true: the longest it takes to get to prod, the better. What a pain.
- ryanklee 4y ago
  I'm an SRE and currently on a mission to get code into prod as fast and safely as possible. Instead of painting such a broad, generic negative picture, maybe articulate an actual position to argue against.
  - deleted 4y ago
    [deleted]
  - etamponi 4y ago
    You are right -- I was referring specifically to SREs in FAANG, which enforce a mandate of maximum 1 deploy to prod per day -- in some case even 1 per week, or less. To be honest, I think there is a threshold, perhaps expressed in terms of risk of losing money, at which it is more beneficial to delay deploys. But applying the same deployment rules to a multi-billion-dollar revenue machine and to a smaller project with perhaps a few thousand weekly users is ridiculous.
    - ryanklee 4y ago
      Thanks for the clarification, I can see how that might be frustrating.
zaat 4y ago
9. Does your Infrastructure as Code live alongside the service it hosts? No, the IaC is configuration, and configuration should be separated from the application. in organizations that implemented GitOps, the IaC files in git are not only a copy of the configuration, it is the source of truth from which the configuration is copied to the systems. Unless all the developers who can commit code should have permission to change production configuration, the IaC repo should be separated from the code repo.
- wxdisk 4y ago

87 points by sourceless 4y ago

> 9. Does your Infrastructure as Code live alongside the service it hosts? That means minor changes to some test infrastructure have to go through the strict review process, because the gitlab-ci.yaml is in the part of the main code. Last time this happened to me I found it annoying, and I don't think the code quality guardians care about some CI config anyway.

davidmurdoch 4y ago
Changes to CI config for release automation are probably vetted and manually tested more than anything else in the projects I work on.
xorcist 4y ago
I will be first in line to suggest to not split repos unless required. It is my experience that deployment, admin and node provisioning (and everything that is generally put under the not always useful IaaC umbrella term) is one of those exceptions. Often documentation is, too. It all depends on what the release flow looks like. A useful rule of thumb when components can share a repository is when the branch model is identical. If that's the case for codified release and provisioning processes and the code that is shipped, then it is likely that you are not taking full advantage of what that deployment and provisioning code can do. If not anything else, then at least test versions must deploy production code, and test code deploy production versions. Again, in my experience across multiple organizations, it is a good idea to keep that code both forwards and backwards compatible. It should be natural that is sometimes says things like "if build number greater than x then set parameter y". While it may feel a bit dirty for some, that type of logic is what deployment code does, and it is much more maintainable than keeping branches around for the same result. If I get to say one thing to new devops/release engineer type people, that's probably it. That, and the value of clear and concise commit messages. Which goes double for this type of code.

dijit 4y ago

This feels like yet more cargo-culting. Even the first item on this list is hotly contested, with Facebook/Microsoft/Google and Co. using centralised mono-repos, google itself using a server model similar to Perforce called Piper. To be clear here, I’m not saying they’re right, but with it feels un-scientific to make a blanket statement that they’re wrong.

theoldlove 4y ago
You can easily have a monorepo with a distributed control system.
- dijit 4y ago
  Yes. But I’m referring to the article: > 1. Do you use a distributed version control system? Edit: the parent originally said “distributed build system”
  - chrisoverzero 4y ago
    Read this as “Do you use git?” This is opposed to svn, cvs, something home-grown, or passing thumb drives back and forth among developers.
    - maccard 4y ago
      Hg, svn, perforce, plastic are all feasible alternatives to git.
  - Morgawr 4y ago
    Just because it's a monorepo it doesn't mean it's centralized?
    - michaelt 4y ago
      Yes. It’s the accompanying suggestion of using github or gitlab that means it’s centralised.

alkonaut 4y ago

Extremely web centric list. Any other development (mobile, desktop, games, embedded, …) will find a lot of these steps very weird.

kqr 4y ago
Really? I challenge you to find any items other than 7 and perhaps 8 that don't apply to the types of development you mention. Essentially the only thing that's different with the types of development you mention is that "final deployment to production" looks different, as it usually involves more or less physically transporting artifacts to your customer. But the rest is just the same. You should be doing trunk-based development, practise something like code review, gate on integration tests, feature flag, spin up virtual production clones, etc. In other words, the fact that getting software to your end users is a clumsy process does not preclude you from doing every other step of the process with short feedback loops.
- maccard 4y ago
  Using distributed version control isn't applicable to most people working in games. 9) is also debatable - requiring someone to clone the entire application to make an infra change to a testing environment. Adding tickets to commit messages isn't necessarily a requirement - some work (at least in my area) is prototypey and maybe ill defined (the task might be to define it). Being able to deploy from your own machine is a double edged sword; the situation you need this is an absolute last resort. Enabling deployments from dev machines means credentials to environments, write access to infra, and likely skirting around normal processes.
  - KronisLV 4y ago
    > Using distributed version control isn't applicable to most people working in games. Are you sure about this? I mean, code absolutely should be versioned and if you can afford the storage, you should also version all of your assets, like models and audio. Even Git has specialized functionality for storing binary assets: https://git-lfs.github.com/ https://git-lfs.github.com/ Unity also seems to be pushing their Plastic SCM as well for this more specialized use case: https://www.plasticscm.com/ https://www.plasticscm.com/ Either way, not using version control for any collaborative project is just asking for issues. Not using distributed version control in particular might just make things more annoying, as anyone who has ever worked with SVN might attest to.

bhawks 4y ago

> 11. Do you include ticket IDs in your commits or branches? This is one of the things that sound 'good' on the surface and are totally worthless in practice (especially when practicing minimal commits). In the wild this devolves into a meaningless umbrella issue (Deliver Feature Foo) or 1 liner issues that sometimes did not even match the pr contents. Just use the commit message and in pr review enforce commit message norms.

kqr 4y ago
My previous team used to have ticket IDs on every commit but just decided to drop that requirement for commits that are truly self-contained and documented properly in the commit message. That approach makes a lot of sense to me. Link the commit to the ticket when it enhances future reading, but not out of blind application of principle.
whakim 4y ago
It's incredibly easy to simply add the ticket number to branch names (or commits if you're rebasing your branches when you merge). Many times the ticket isn't helpful, but "just make the commit/branch self-explanatory" ignores the fact that non-developers - product folks, designers, etc. - are far more likely to leave useful context in the ticket itself.
- lelandfe 4y ago
  Yes, exactly. I treat my commit history like I'm preparing for a detective to investigate years down the line :) Great commit messages are just one part of the timeline. Ensuring there's a link between the commits/PR and the ticket lets the future detective get more information (frequently, that's "why did we do this").
- kqr 4y ago
  When there is a ticket, yes, adding the number is trivial and probably worth it. Where the rule breaks down, in my opinion, is when you have to create a ticket and replicate the commit message in it any time you stumble over a code maintainability issue and fix it in stride.

satyrnein 4y ago

I'd like to see a workflow that meets all the criteria. I was surprised to see the item about deploying directly from your own machine; that seems to contradict the other goals which point to automated pipelines that deploy on merge.

TheP1000 4y ago
This is a very common pattern especially with the cdk. Having a locally deployed stack makes development in the cloud so much easier.
kqr 4y ago
I feel like a lot of people misread that point. It's not about deploying to production (that sounds nuts!) It's about spinning up and deploying to a full dev environment. This is in part to not have to run everything locally, but mainly to exercise deploy automation regularly.

etamponi 4y ago

7. 7! This is by far the one that resonates with me the most. But for some reasons SREs think the opposite is true: the longest it takes to get to prod, the better. What a pain.

ryanklee 4y ago
I'm an SRE and currently on a mission to get code into prod as fast and safely as possible. Instead of painting such a broad, generic negative picture, maybe articulate an actual position to argue against.
- deleted 4y ago
  [deleted]
- etamponi 4y ago
  You are right -- I was referring specifically to SREs in FAANG, which enforce a mandate of maximum 1 deploy to prod per day -- in some case even 1 per week, or less. To be honest, I think there is a threshold, perhaps expressed in terms of risk of losing money, at which it is more beneficial to delay deploys. But applying the same deployment rules to a multi-billion-dollar revenue machine and to a smaller project with perhaps a few thousand weekly users is ridiculous.
  - ryanklee 4y ago
    Thanks for the clarification, I can see how that might be frustrating.

zaat 4y ago

9. Does your Infrastructure as Code live alongside the service it hosts? No, the IaC is configuration, and configuration should be separated from the application. in organizations that implemented GitOps, the IaC files in git are not only a copy of the configuration, it is the source of truth from which the configuration is copied to the systems. Unless all the developers who can commit code should have permission to change production configuration, the IaC repo should be separated from the code repo.

wxdisk 4y ago