5 ms·

Just imagine the amount of stress on this people, hope the money really worth it.

mov31tmov31t 5y ago
It shouldn't be too stressful. Well-managed companies blame processes rather than people, and have systems set up to communicate rapidly when large-scale events occur. It can be sort of exciting, but it's not like there is one person typing at a keyboard with a hundred managers breathing down their neck. These resolutions are collaborative, shared efforts.
- xorcist 5y ago
  > Well-managed companies blame processes rather than people, We're six hours without a route to their network, and counting. I think we can safely rule out well-managed.
- rvnx 5y ago
  "it's not like there is one person typing at a keyboard with a hundred managers breathing down their neck. These resolutions are collaborative, shared efforts" Well, you'd be surprised about how one person can bring everything down and/or save the day at Facebook, Cloudflare, Google, Gitlab, etc. Most people are observers/cheerleaders when there is an incident.
  - cromka 5y ago
    > Most people are observers/cheerleaders when there is an incident. Yeah, a typical fight/flight response.
    - SomeBoolshit 5y ago
      Or most people simply don't have anything useful to add or do during an incident.
      - ikiris 5y ago
        Taking all the available slots in the massive gvc warroom ain't much... but its honest work.
- thih9 5y ago
  > It shouldn't be too stressful. (...) it's not like there is one person typing at a keyboard with a hundred managers breathing down their neck Earlier comment mentioned that there is a bottleneck, and that people who are physically able to solve the issue are few and that they need to be informed what to do; being one of these people sounds pretty stressful to me. "but the people with physical access is separate (...) Part of this is also due to lower staffing in data centers due to pandemic measures", source: https://news.ycombinator.com/item?id=28749244 https://news.ycombinator.com/item?id=28749244
  - mov31tmov31t 5y ago
    Sure, but that's what conference calls are for. Most big tech companies automatically start a call for every large scale incident, and adjacent teams are expected to have a representative call in and contribute to identifying/remediating the issue. None of the people with physical access are individually responsible, and they should have a deep bench of advice and context to draw from.
- astridpeth 5y ago
  I'm not an IT Operations guy, but as a dev I always thought it was exciting when the IT guys had in their shoulders the destiny of the firm. I must be exciting.
  - donalhunt 5y ago
    You tend not to think about it… Most teams that handle incidents have well documented incident plans and playbooks. When something major happens you are mostly executing the plan (which has been designed and tested). There are always gotchas that require additional attention / hands but the general direction is usually clear.
- Ansil849 5y ago
  > Well-managed companies blame processes rather than people I feel like this just obfuscates the fact that individuals are ultimately responsible, and allows subpar employees to continue existing at an organization when their position could be filled by a more qualified employee. (Not talking about this Facebook incident in particular, but as a generalisation: not attributing individual fault allows faulty employees to thrive at the expense of more qualified ones).
  - nerdawson 5y ago
    By focusing on the process, lessons are learned and systems are put in place which leads to a cycle of improvement. When individuals are blamed instead, a culture of fear sets in and people hide / cover up their mistakes. Everybody loses as a result.
  - alex_sf 5y ago
    I don't think the comment you're replying to applies to your concern about subpar employees. We blame processes instead of people because people are fallible. We've spent millenia trying to correct people, and it rarely works to a sufficient level. It's better to create a process that makes it harder for humans to screw up.
    - Ansil849 5y ago
      Yes, absolutely, people make mistakes. But the thought I was trying to convey is that some people make a lot more mistakes than others, and by not attributing individual fault these people are allowed to thrive at the cost of having less error-prone people in their position. For example, someone who triple-checks every parameter that they input, versus someone who has a habit of just skimming or not checking at all. Yes the triple-checker will make mistakes too, but way less than the person who puts less effort in.
      - mynameisvlad 5y ago
        But that has nothing to do with blaming processes vs people. If the process in place means that someone has to triple check their numbers to make sure they’re correct, then it’s a broken process. Because even that person who triple checks is one time going to be woken up at 2:30am and won’t triple check because they want sleep. If the process lets you do something, then someone at some point in time, whether accidentally or maliciously, will cause that to happen. You can discipline that person, and they certainly won’t make the same mistake again, but what about their other 10 coworkers? Or the people on the 5 sister teams with similar access who didn’t even know the full details of what happened? If you blame the process and make improvements to ensure that triple checking isn’t required, then nobody will get into the situation in the first place. That is why you blame the process.
- ikiris 5y ago
  As one of the major responders to an incident analogous to this at a different fang... you're high, its still hella stressful.
- tristor 5y ago
  > It can be sort of exciting, but it's not like there is one person typing at a keyboard with a hundred managers breathing down their neck. As someone who formerly did Ops for many many years... this is not accurate. Even in a well organized company there are usually stakeholders at every level on IM calls so that they don't need to play "telephone" for status. For an incident of this size, it wouldn't be unusual to have C-level executives on the call. While those managers are mostly just quietly listening in on mute if they know what's good (e.g. don't distract the people doing the work to fix your problem), their mere presence can make the entire situation more tense and stressful for the person banging keyboards. If they decide to be chatty or belligerent, it makes everything 100x worse. I don't envy the SREs at Facebook today. Godspeed fellow Ops homies.
  - LordHumungous 5y ago
    C levels don't sit on the call with engineers. They aren't that dumb. Managers will communicate upward.
    - tristor 5y ago
      That greatly depends on the incident and the organization. I’ve personally been on numerous incident calls with C-level folks involved.
      - ikiris 5y ago
        Yeah hell, I've ended up with one of the big names as my comm's lead. That in itself was stressful, and became an example case later.
  - Salgat 5y ago
    I think it comes down to the comfort level of the worker. I remember when our production environment went down. The CTO was sitting with me just watching and I had no problem with it since he was completely supportive, wasn't trying to hurry me, just wanted to see how the process of fixing it worked. We knew it wasn't any specific person's fault, so no one had to feel the heat from the situation beyond just doing a decent job getting it back up.
- TrevorJ 5y ago
  >Well-managed companies To what extent does this include Facebook?
- aenis 5y ago
  Well, individuals will still stress, if anything, due to the feeling of bein personally responsible for inflicting damage. I know someone who accidentally added a rule 'reject access to * for all authenticated users' in some stupid system where the ACL ruleset itself was covered by this *, and this person nearly collapsed when she realized even admins were shut out of the system. It required getting low level access to the underlying software to reverse engineer its ACLs and hack into the system. Major financial institution. Shit like leaves people with actual trauma. As much as I hate fb, I really feel for the net ops guys trying to figure it all out, with the whole world watching (most of it with shadenfreude)
yupper32 5y ago
The stress for me usually goes away once the incident is fully escalated and there's a team with me working on the issue. I imagine that happened quite quick in this case...
- mrweasel 5y ago
  Exactly, the primary focus in situations like this, is to ensure that no one feel like they are alone, even if in the end it is one person who has to type in the right commands. Always be there, help them double check, help monitor, help make the calls to whomever needs to be informed, help debug. No one should ever be alone during a large incident.
tomjen3 5y ago
This is a one off event, not a chronic stress trigger. I find them envigorating personally, as long as everybody concerned understands that this is not good in the long run, and that you are not going to write your best code this way.

It shouldn't be too stressful. Well-managed companies blame processes rather than people, and have systems set up to communicate rapidly when large-scale events occur. It can be sort of exciting, but it's not like there is one person typing at a keyboard with a hundred managers breathing down their neck. These resolutions are collaborative, shared efforts.

xorcist 5y ago
> Well-managed companies blame processes rather than people, We're six hours without a route to their network, and counting. I think we can safely rule out well-managed.
rvnx 5y ago
"it's not like there is one person typing at a keyboard with a hundred managers breathing down their neck. These resolutions are collaborative, shared efforts" Well, you'd be surprised about how one person can bring everything down and/or save the day at Facebook, Cloudflare, Google, Gitlab, etc. Most people are observers/cheerleaders when there is an incident.
- cromka 5y ago
  > Most people are observers/cheerleaders when there is an incident. Yeah, a typical fight/flight response.
  - SomeBoolshit 5y ago
    Or most people simply don't have anything useful to add or do during an incident.
    - ikiris 5y ago
      Taking all the available slots in the massive gvc warroom ain't much... but its honest work.
thih9 5y ago
> It shouldn't be too stressful. (...) it's not like there is one person typing at a keyboard with a hundred managers breathing down their neck Earlier comment mentioned that there is a bottleneck, and that people who are physically able to solve the issue are few and that they need to be informed what to do; being one of these people sounds pretty stressful to me. "but the people with physical access is separate (...) Part of this is also due to lower staffing in data centers due to pandemic measures", source: https://news.ycombinator.com/item?id=28749244 https://news.ycombinator.com/item?id=28749244
- mov31tmov31t 5y ago
  Sure, but that's what conference calls are for. Most big tech companies automatically start a call for every large scale incident, and adjacent teams are expected to have a representative call in and contribute to identifying/remediating the issue. None of the people with physical access are individually responsible, and they should have a deep bench of advice and context to draw from.
astridpeth 5y ago
I'm not an IT Operations guy, but as a dev I always thought it was exciting when the IT guys had in their shoulders the destiny of the firm. I must be exciting.
- donalhunt 5y ago
  You tend not to think about it… Most teams that handle incidents have well documented incident plans and playbooks. When something major happens you are mostly executing the plan (which has been designed and tested). There are always gotchas that require additional attention / hands but the general direction is usually clear.
Ansil849 5y ago
> Well-managed companies blame processes rather than people I feel like this just obfuscates the fact that individuals are ultimately responsible, and allows subpar employees to continue existing at an organization when their position could be filled by a more qualified employee. (Not talking about this Facebook incident in particular, but as a generalisation: not attributing individual fault allows faulty employees to thrive at the expense of more qualified ones).
- nerdawson 5y ago
  By focusing on the process, lessons are learned and systems are put in place which leads to a cycle of improvement. When individuals are blamed instead, a culture of fear sets in and people hide / cover up their mistakes. Everybody loses as a result.
- alex_sf 5y ago
  I don't think the comment you're replying to applies to your concern about subpar employees. We blame processes instead of people because people are fallible. We've spent millenia trying to correct people, and it rarely works to a sufficient level. It's better to create a process that makes it harder for humans to screw up.
  - Ansil849 5y ago
    Yes, absolutely, people make mistakes. But the thought I was trying to convey is that some people make a lot more mistakes than others, and by not attributing individual fault these people are allowed to thrive at the cost of having less error-prone people in their position. For example, someone who triple-checks every parameter that they input, versus someone who has a habit of just skimming or not checking at all. Yes the triple-checker will make mistakes too, but way less than the person who puts less effort in.
    - mynameisvlad 5y ago
      But that has nothing to do with blaming processes vs people. If the process in place means that someone has to triple check their numbers to make sure they’re correct, then it’s a broken process. Because even that person who triple checks is one time going to be woken up at 2:30am and won’t triple check because they want sleep. If the process lets you do something, then someone at some point in time, whether accidentally or maliciously, will cause that to happen. You can discipline that person, and they certainly won’t make the same mistake again, but what about their other 10 coworkers? Or the people on the 5 sister teams with similar access who didn’t even know the full details of what happened? If you blame the process and make improvements to ensure that triple checking isn’t required, then nobody will get into the situation in the first place. That is why you blame the process.
ikiris 5y ago
As one of the major responders to an incident analogous to this at a different fang... you're high, its still hella stressful.
tristor 5y ago
> It can be sort of exciting, but it's not like there is one person typing at a keyboard with a hundred managers breathing down their neck. As someone who formerly did Ops for many many years... this is not accurate. Even in a well organized company there are usually stakeholders at every level on IM calls so that they don't need to play "telephone" for status. For an incident of this size, it wouldn't be unusual to have C-level executives on the call. While those managers are mostly just quietly listening in on mute if they know what's good (e.g. don't distract the people doing the work to fix your problem), their mere presence can make the entire situation more tense and stressful for the person banging keyboards. If they decide to be chatty or belligerent, it makes everything 100x worse. I don't envy the SREs at Facebook today. Godspeed fellow Ops homies.
- LordHumungous 5y ago
  C levels don't sit on the call with engineers. They aren't that dumb. Managers will communicate upward.
  - tristor 5y ago
    That greatly depends on the incident and the organization. I’ve personally been on numerous incident calls with C-level folks involved.
    - ikiris 5y ago
      Yeah hell, I've ended up with one of the big names as my comm's lead. That in itself was stressful, and became an example case later.
- Salgat 5y ago
  I think it comes down to the comfort level of the worker. I remember when our production environment went down. The CTO was sitting with me just watching and I had no problem with it since he was completely supportive, wasn't trying to hurry me, just wanted to see how the process of fixing it worked. We knew it wasn't any specific person's fault, so no one had to feel the heat from the situation beyond just doing a decent job getting it back up.
TrevorJ 5y ago
>Well-managed companies To what extent does this include Facebook?
aenis 5y ago
Well, individuals will still stress, if anything, due to the feeling of bein personally responsible for inflicting damage. I know someone who accidentally added a rule 'reject access to * for all authenticated users' in some stupid system where the ACL ruleset itself was covered by this *, and this person nearly collapsed when she realized even admins were shut out of the system. It required getting low level access to the underlying software to reverse engineer its ACLs and hack into the system. Major financial institution. Shit like leaves people with actual trauma. As much as I hate fb, I really feel for the net ops guys trying to figure it all out, with the whole world watching (most of it with shadenfreude)

yupper32 5y ago

The stress for me usually goes away once the incident is fully escalated and there's a team with me working on the issue. I imagine that happened quite quick in this case...

mrweasel 5y ago
Exactly, the primary focus in situations like this, is to ensure that no one feel like they are alone, even if in the end it is one person who has to type in the right commands. Always be there, help them double check, help monitor, help make the calls to whomever needs to be informed, help debug. No one should ever be alone during a large incident.

tomjen3 5y ago

This is a one off event, not a chronic stress trigger. I find them envigorating personally, as long as everybody concerned understands that this is not good in the long run, and that you are not going to write your best code this way.