7 ms·

Moving Fast and Securing Things

375 points by Chris911 8y ago

JepZ 8y ago
And I thought 'security' itself is friction ;-) No seriously, I was wondering if that tool has a CLI interface? Might make it more accessable for some devs.
boffinism 8y ago
> The process of deploying code to production is very simple, and takes about ten minutes total. This results in a life cycle in which we deploy code to production approximately 100 times per day. What? They spend 1000 minutes out of every 1440 deploying to production? The deployment process is occurring over 16 hours out of every 24? Am I the only one who is nonplussed by this? EDIT: Ok I get it, I get it. I guess I always worked in much smaller companies where CD meant deploying about 10 times a day tops. TIL big companies are big.
- m0meni 8y ago
  https://en.wikipedia.org/wiki/Continuous_delivery https://en.wikipedia.org/wiki/Continuous_delivery
- deleted 8y ago
  [deleted]
- maaark 8y ago
  C'mon dude, it obviously doesn't require everyone in the company to spend 10 minutes for each deployment, it takes one person 10 minutes to deploy one thing.
  - nathancahill 8y ago
    Probably not even. One person starts a deploy, after 10 minutes they can deploy again.
- xyzzy_plugh 8y ago
  I think you're assuming that each deploy is somehow disruptive or blocking, as if systems enter some "deploy mode". This may be true of a fleet overall, but is not when you take an instance into account. Instead imagine each deploy as an edge, and imagine them to be near instantaneous. With respect to an instance, and a user, and the observable effects of a deploy, this paints a more accurate picture. 100 times a day means one deploy every 14.4 minutes. How many times a day do you think Amazon deploys changes? Or Facebook? Or Google?
  - electrum 8y ago
    Deployment at Facebook: https://code.facebook.com/posts/270314900139291/rapid-release-at-massive-scale/ https://code.facebook.com/posts/270314900139291/rapid-releas...
- wgerard 8y ago
  Not a slack employee but worked at a company with similar CD views: (Likely) various groups of people are deploying to production throughout the day. Out of those 100 deploys, an individual is probably only involved in 1 or 2 a day. As soon as you're ready to deploy your code, you queue up and see it all the way through to production along with probably a few other people doing the same thing. The actual "change the servers over to the new production code" process is usually instantaneous or extremely quick, the 10 minutes is mostly spent testing/building/etc. People (including myself) enjoy this because you can push very small incremental changes to production, which significantly reduces the chance of confounding errors or major issues. Note that this is would be a Sisyphean task if your company doesn't have great logging/metrics reporting/testing/etc.
  - toomuchtodo 8y ago
    It’s usually a Sisyphean task. Everyone wants to look and act cutting edge (“but Netflix!”) but nobody wants to make the necessary investments in the tooling, org structure, and management ability/support that is required to support that sort of deployment cadence (if your org focuses on who broke something instead of the process, and management doesn’t want to change that culture, all hope is already lost [based on experience in a large enterprise, YMMV]). There are some legitimate needs for continuous deployment, the rest of it is cargo culting.
    - wgerard 8y ago
      > There are some legitimate needs for continuous deployment, the rest of it is cargo culting. Maybe, but I wouldn't go that far. Small companies already often do CD, because there's rarely a rigid deploy schedule. It's a practice people understand and feel the benefits of immediately. If you ask someone who moved from a small startup to a huge company what their biggest complaints are, I bet "longer/stricter deploy process" comes up 8/10 times. When I think of cargo cult programming I think more of TDD or Agile: Practices that people aren't familiar with and often implement without understanding the benefits or reasoning.
      - 8y ago
- hoorayimhelping 8y ago
  When you have a team of 200 people, deploying 100 times a day isn't that big of a number. A culture of continuous deployment is often hard to fathom for people who've never worked at a company with one. Everything, down to what you write and how you write it, is influenced by being able to deploy it and see its effects almost instantly. These aren't huge, sweeping changes being deployed. They're small pieces of larger feature sets. It's more like: Deploy a conditional statement with logging and confirm from the logs that it works; next, deploy the view that you're testing with a feature flag toggled in a way that you and the pm can see it that is properly called from the conditional from before; when things look good, deploy some controller code that handles form requests from the view, etc. You deploy small changes piecemeal and so spread out the risk over a larger period of time. It makes identifying issues with a new piece of code almost trivial. Needing to debug 30 lines of code is so much less harrowing than needing to look over 900.
  - Gravityloss 8y ago
    Also depends what you are working on...
- stephengillie 8y ago
  When I started at a real estate webhost in 2013, there was a "push Dev to Prod" script, and a scheduled task to run the script every 60 seconds. So we technically had 1440 pushes per day, though most developers were only working during daylight hours. So having more pushes per day isn't necessarily the metric to maximize. Quality of code changes for each push is important, and this is where automated testing can be very valuable. The goal is for automated testing to be a "gatekeeper of bad code". But even this system isn't perfect, and it's possible to deploy things that pass tests but still have show-stopping bugs. Or for the code to cause your tests to misbehave - I'm seeing this now with Tape.js on Travis, where Tape sees my S3 init calls as extra tests. Then my build fails because - of the 2 tests specified, 3 failed, and another 4 passed.
- jsjohnst 8y ago
  You’re assuming a monolithic app, maybe there’s twenty services that get deployed five times a day.
jrochkind1 8y ago
this is really cool.
ejcx 8y ago
I love this! If you're a part of a security team and you are not automating your processes and procedures then your team is going to drown. You must automate. It seems like some simple checklist app but having a non Jira process that takes only a few minutes is so valuable, and "security reviews" and "threat models" as part of your SDLC take insane amounts of time and honestly aren't super helpful.
- lightedman 8y ago
  "If you're a part of a security team and you are not automating your processes and procedures then your team is going to drown. You must automate." Except we just saw automation fail with two big companies - Twitter and GitHub.
- insensible 8y ago
  What I think is brilliant here is that that sort of work can take place separately and provide feedback to these checklists when problems or deficiencies are found. Basically the security checklists are a deliverable that can be iterated on independently while they still benefit from the existing one.
wpietri 8y ago
One of the things I think about when analyzing organizational behavior is where something falls on the supportive vs controlling spectrum. It's really impressive how much they're on the supportive end here. When organizations scale up, and especially when they're dealing with risks, it's easy for them to shift toward the controlling end of things. This is especially true when internally people can score points by assigning or shifting blame. Controlling and blaming are terrible for creative work, though. And they're also terrible for increasing safety beyond a certain pretty low level. (For those interested, I strongly recommend Sidney Dekker's "Field Guide to Understanding Human Error" [1], a great book on how to investigate airplane accidents, and how blame-focused approaches deeply harm real safety efforts.) So it's great to see Slack finding a way to scale up without losing something that has allowed them to make such a lovely product. [1] https://www.amazon.com/Field-Guide-Understanding-Human-Error/dp/0754648257 https://www.amazon.com/Field-Guide-Understanding-Human-Error...
- dvtrn 8y ago
  Having recently escaped from a "control and blame" environment, this is also horrible for releases as left unchecked, more energy is expended trying to double-down on architecting for perfection in fault tolerance. Risk aversion goes through the roof cripples decision making, and before you know it your entire team of developers have become full time maintenance coders, you stop innovating and spend cycles creating imaginary problems for yourself and begin slowly sinking. We had a guy who more or less appointed himself manager when previous engineering manager decided he couldn't deal with the environment anymore, his insistence on controlling everything resulted in a conscious decision to destroy the engineering wiki and knowledge base and forced everyone to funnel through him-creating a single source of truth. Once his mind was made up on something, he would berate other engineers, other developers and team members to get what he wanted. Features stopped being developed, things began to fail chronically, and because senior leadership weren't made up of tech people-they all deferred to him-and once they decided to officially make him engineering manager (for no reason other than he had been on the team the longest-because people were beginning to wise up and quit the company), the entire engineering department of 12 people except for 2 quit because no one wanted to work for him. Imagine my schadenfreude after leaving that environment to find out they were forced to close after years of failing to innovate, resulting in the market catching up and passing them. Never in my adult life have I seen a company inflict so many wounds on itself and then be shocked when competitors start plucking customers off like grapes.
vasilakisfil 8y ago
I am in favor of checklists for certain critical tasks, even if they are repetitive and/or boring. I think checklists are underrated.
- jimktrains2 8y ago
  They were/are helpful in making medicine safer too. https://www.newyorker.com/magazine/2007/12/10/the-checklist https://www.newyorker.com/magazine/2007/12/10/the-checklist https://www.ncbi.nlm.nih.gov/pubmed/24116973 https://www.ncbi.nlm.nih.gov/pubmed/24116973
  - kendallpark 8y ago
    Same with aviation. http://qualitysafety.bmj.com/content/24/7/428 http://qualitysafety.bmj.com/content/24/7/428
  - ranger207 8y ago
    The New Yorker article was later expanded into an entire book of the same name.
    - ErikAugust 8y ago
      The Checklist Manifesto
- ggregoire 8y ago
  I am in favor of (check)lists for everything. I am actually surprised about how few developers take notes and make list. It’s one of the most important part of my workflow.
  - peterwwillis 8y ago
    Exactly. Checklists are great for non-critical tasks. Hospital triage uses the same checklist for pretty much every new patient. Triage is boring, but the fact that it is checklisted is what helps drive efficiency in the ER. Similar boring processes in a tech business can be checklisted to increase efficiency, and the checklist itself can be iterated over. Everything from designing a new feature, to troubleshooting an error, to addressing a customer support ticket, to getting access to a new resource should use checklists.
mikekey 8y ago
Well written and timely for me. I would like to see this capable of something other than Jira though :/
- coldacid 8y ago
  There's instructions for working with Trello in the repo's README, but so far it seems to be just that or JIRA Enterprise (not Cloud).
deleted 8y ago
[deleted]
punnerud 8y ago
I like the addition question if you are using C/C++: «We confirm that we really, really need to use a non-memory-safe language.". PHP/Python/C/C++ get Medium Risk directly, Low Risk: WebApp/API/MessageServer/iOS/Android/Electron/WindowsPhone
maccard 8y ago
> At the start of 2015, Slack had 100 employees. Today, we’re over 800 people! That's a lot of people...
- corrigible 8y ago
  Seems like their client memory usage scales with their headcount
mbid 8y ago
A security app written in PHP. Nice touch.
deleted 8y ago
[deleted]
hhaidar 8y ago
The company I work for has been offering an enterprise level service like for about 8 years now: https://www.securitycompass.com/sdelements/ https://www.securitycompass.com/sdelements/
pdfpro 8y ago
Not sure why they build go SDL, when the exact same thing already exists, and is light years ahead: SD Elements.
spydum 8y ago
So glad they finally published this, saw the OWASP AppSec talk, was eagerly awaiting it. However - I would want to caution: I think this model works because Slack has a self-described "culture of developer trust". I tend to think, they hire bright engineers and ensure they are equipped to do the right thing. I believe the vast majority of organizations are NOT ready for this. I direly want them to be, but simple fact is there are too many mediocre developers, and they can't be trusted without guardrails (and some straight up need babysitters).

375 points by Chris911 8y ago

JepZ 8y ago

And I thought 'security' itself is friction ;-) No seriously, I was wondering if that tool has a CLI interface? Might make it more accessable for some devs.

boffinism 8y ago

> The process of deploying code to production is very simple, and takes about ten minutes total. This results in a life cycle in which we deploy code to production approximately 100 times per day. What? They spend 1000 minutes out of every 1440 deploying to production? The deployment process is occurring over 16 hours out of every 24? Am I the only one who is nonplussed by this? EDIT: Ok I get it, I get it. I guess I always worked in much smaller companies where CD meant deploying about 10 times a day tops. TIL big companies are big.

m0meni 8y ago
https://en.wikipedia.org/wiki/Continuous_delivery https://en.wikipedia.org/wiki/Continuous_delivery
deleted 8y ago
[deleted]
maaark 8y ago
C'mon dude, it obviously doesn't require everyone in the company to spend 10 minutes for each deployment, it takes one person 10 minutes to deploy one thing.
- nathancahill 8y ago
  Probably not even. One person starts a deploy, after 10 minutes they can deploy again.
xyzzy_plugh 8y ago
I think you're assuming that each deploy is somehow disruptive or blocking, as if systems enter some "deploy mode". This may be true of a fleet overall, but is not when you take an instance into account. Instead imagine each deploy as an edge, and imagine them to be near instantaneous. With respect to an instance, and a user, and the observable effects of a deploy, this paints a more accurate picture. 100 times a day means one deploy every 14.4 minutes. How many times a day do you think Amazon deploys changes? Or Facebook? Or Google?
- electrum 8y ago
  Deployment at Facebook: https://code.facebook.com/posts/270314900139291/rapid-release-at-massive-scale/ https://code.facebook.com/posts/270314900139291/rapid-releas...
wgerard 8y ago
Not a slack employee but worked at a company with similar CD views: (Likely) various groups of people are deploying to production throughout the day. Out of those 100 deploys, an individual is probably only involved in 1 or 2 a day. As soon as you're ready to deploy your code, you queue up and see it all the way through to production along with probably a few other people doing the same thing. The actual "change the servers over to the new production code" process is usually instantaneous or extremely quick, the 10 minutes is mostly spent testing/building/etc. People (including myself) enjoy this because you can push very small incremental changes to production, which significantly reduces the chance of confounding errors or major issues. Note that this is would be a Sisyphean task if your company doesn't have great logging/metrics reporting/testing/etc.
- toomuchtodo 8y ago
  It’s usually a Sisyphean task. Everyone wants to look and act cutting edge (“but Netflix!”) but nobody wants to make the necessary investments in the tooling, org structure, and management ability/support that is required to support that sort of deployment cadence (if your org focuses on who broke something instead of the process, and management doesn’t want to change that culture, all hope is already lost [based on experience in a large enterprise, YMMV]). There are some legitimate needs for continuous deployment, the rest of it is cargo culting.
  - wgerard 8y ago
    > There are some legitimate needs for continuous deployment, the rest of it is cargo culting. Maybe, but I wouldn't go that far. Small companies already often do CD, because there's rarely a rigid deploy schedule. It's a practice people understand and feel the benefits of immediately. If you ask someone who moved from a small startup to a huge company what their biggest complaints are, I bet "longer/stricter deploy process" comes up 8/10 times. When I think of cargo cult programming I think more of TDD or Agile: Practices that people aren't familiar with and often implement without understanding the benefits or reasoning.
    - 8y ago
hoorayimhelping 8y ago
When you have a team of 200 people, deploying 100 times a day isn't that big of a number. A culture of continuous deployment is often hard to fathom for people who've never worked at a company with one. Everything, down to what you write and how you write it, is influenced by being able to deploy it and see its effects almost instantly. These aren't huge, sweeping changes being deployed. They're small pieces of larger feature sets. It's more like: Deploy a conditional statement with logging and confirm from the logs that it works; next, deploy the view that you're testing with a feature flag toggled in a way that you and the pm can see it that is properly called from the conditional from before; when things look good, deploy some controller code that handles form requests from the view, etc. You deploy small changes piecemeal and so spread out the risk over a larger period of time. It makes identifying issues with a new piece of code almost trivial. Needing to debug 30 lines of code is so much less harrowing than needing to look over 900.
- Gravityloss 8y ago
  Also depends what you are working on...
stephengillie 8y ago
When I started at a real estate webhost in 2013, there was a "push Dev to Prod" script, and a scheduled task to run the script every 60 seconds. So we technically had 1440 pushes per day, though most developers were only working during daylight hours. So having more pushes per day isn't necessarily the metric to maximize. Quality of code changes for each push is important, and this is where automated testing can be very valuable. The goal is for automated testing to be a "gatekeeper of bad code". But even this system isn't perfect, and it's possible to deploy things that pass tests but still have show-stopping bugs. Or for the code to cause your tests to misbehave - I'm seeing this now with Tape.js on Travis, where Tape sees my S3 init calls as extra tests. Then my build fails because - of the 2 tests specified, 3 failed, and another 4 passed.
jsjohnst 8y ago
You’re assuming a monolithic app, maybe there’s twenty services that get deployed five times a day.

jrochkind1 8y ago

this is really cool.

ejcx 8y ago

I love this! If you're a part of a security team and you are not automating your processes and procedures then your team is going to drown. You must automate. It seems like some simple checklist app but having a non Jira process that takes only a few minutes is so valuable, and "security reviews" and "threat models" as part of your SDLC take insane amounts of time and honestly aren't super helpful.

lightedman 8y ago
"If you're a part of a security team and you are not automating your processes and procedures then your team is going to drown. You must automate." Except we just saw automation fail with two big companies - Twitter and GitHub.
insensible 8y ago
What I think is brilliant here is that that sort of work can take place separately and provide feedback to these checklists when problems or deficiencies are found. Basically the security checklists are a deliverable that can be iterated on independently while they still benefit from the existing one.

wpietri 8y ago

One of the things I think about when analyzing organizational behavior is where something falls on the supportive vs controlling spectrum. It's really impressive how much they're on the supportive end here. When organizations scale up, and especially when they're dealing with risks, it's easy for them to shift toward the controlling end of things. This is especially true when internally people can score points by assigning or shifting blame. Controlling and blaming are terrible for creative work, though. And they're also terrible for increasing safety beyond a certain pretty low level. (For those interested, I strongly recommend Sidney Dekker's "Field Guide to Understanding Human Error" [1], a great book on how to investigate airplane accidents, and how blame-focused approaches deeply harm real safety efforts.) So it's great to see Slack finding a way to scale up without losing something that has allowed them to make such a lovely product. [1] https://www.amazon.com/Field-Guide-Understanding-Human-Error/dp/0754648257 https://www.amazon.com/Field-Guide-Understanding-Human-Error...

dvtrn 8y ago
Having recently escaped from a "control and blame" environment, this is also horrible for releases as left unchecked, more energy is expended trying to double-down on architecting for perfection in fault tolerance. Risk aversion goes through the roof cripples decision making, and before you know it your entire team of developers have become full time maintenance coders, you stop innovating and spend cycles creating imaginary problems for yourself and begin slowly sinking. We had a guy who more or less appointed himself manager when previous engineering manager decided he couldn't deal with the environment anymore, his insistence on controlling everything resulted in a conscious decision to destroy the engineering wiki and knowledge base and forced everyone to funnel through him-creating a single source of truth. Once his mind was made up on something, he would berate other engineers, other developers and team members to get what he wanted. Features stopped being developed, things began to fail chronically, and because senior leadership weren't made up of tech people-they all deferred to him-and once they decided to officially make him engineering manager (for no reason other than he had been on the team the longest-because people were beginning to wise up and quit the company), the entire engineering department of 12 people except for 2 quit because no one wanted to work for him. Imagine my schadenfreude after leaving that environment to find out they were forced to close after years of failing to innovate, resulting in the market catching up and passing them. Never in my adult life have I seen a company inflict so many wounds on itself and then be shocked when competitors start plucking customers off like grapes.

vasilakisfil 8y ago

I am in favor of checklists for certain critical tasks, even if they are repetitive and/or boring. I think checklists are underrated.

jimktrains2 8y ago
They were/are helpful in making medicine safer too. https://www.newyorker.com/magazine/2007/12/10/the-checklist https://www.newyorker.com/magazine/2007/12/10/the-checklist https://www.ncbi.nlm.nih.gov/pubmed/24116973 https://www.ncbi.nlm.nih.gov/pubmed/24116973
- kendallpark 8y ago
  Same with aviation. http://qualitysafety.bmj.com/content/24/7/428 http://qualitysafety.bmj.com/content/24/7/428
- ranger207 8y ago
  The New Yorker article was later expanded into an entire book of the same name.
  - ErikAugust 8y ago
    The Checklist Manifesto
ggregoire 8y ago
I am in favor of (check)lists for everything. I am actually surprised about how few developers take notes and make list. It’s one of the most important part of my workflow.
- peterwwillis 8y ago
  Exactly. Checklists are great for non-critical tasks. Hospital triage uses the same checklist for pretty much every new patient. Triage is boring, but the fact that it is checklisted is what helps drive efficiency in the ER. Similar boring processes in a tech business can be checklisted to increase efficiency, and the checklist itself can be iterated over. Everything from designing a new feature, to troubleshooting an error, to addressing a customer support ticket, to getting access to a new resource should use checklists.

mikekey 8y ago

Well written and timely for me. I would like to see this capable of something other than Jira though :/

coldacid 8y ago
There's instructions for working with Trello in the repo's README, but so far it seems to be just that or JIRA Enterprise (not Cloud).

deleted 8y ago

[deleted]

punnerud 8y ago

I like the addition question if you are using C/C++: «We confirm that we really, really need to use a non-memory-safe language.". PHP/Python/C/C++ get Medium Risk directly, Low Risk: WebApp/API/MessageServer/iOS/Android/Electron/WindowsPhone

maccard 8y ago

> At the start of 2015, Slack had 100 employees. Today, we’re over 800 people! That's a lot of people...

corrigible 8y ago
Seems like their client memory usage scales with their headcount

mbid 8y ago

A security app written in PHP. Nice touch.

deleted 8y ago

[deleted]

hhaidar 8y ago

The company I work for has been offering an enterprise level service like for about 8 years now: https://www.securitycompass.com/sdelements/ https://www.securitycompass.com/sdelements/

pdfpro 8y ago

Not sure why they build go SDL, when the exact same thing already exists, and is light years ahead: SD Elements.

spydum 8y ago

So glad they finally published this, saw the OWASP AppSec talk, was eagerly awaiting it. However - I would want to caution: I think this model works because Slack has a self-described "culture of developer trust". I tend to think, they hire bright engineers and ensure they are equipped to do the right thing. I believe the vast majority of organizations are NOT ready for this. I direly want them to be, but simple fact is there are too many mediocre developers, and they can't be trusted without guardrails (and some straight up need babysitters).