5 ms·

I'm reminded of a time that an intern took down us-east1 on AWS, by modifying a configuration file they shouldn't have had access to. Amazon (somehow) did the c

• dudus 2y ago
  The difference in this case is intent. Did the employee have the intent to cause damage? If so just fire him/her.
  • danpalmer 2y ago
    Malicious intent to be precise. Well-intentioned attempts to demonstrate issues for the purposes of helping to fix should generally not be punished, unless there is a wider fallout than expected and that can be attributed to negligence.
• raihansaputra 2y ago
  afaik this was intentional in that they stopped training runs and changing parameters for other employee training runs, and even joined in on the debugging group trying to solve the "issues".
• grogenaut 2y ago
  From what I've seen in Amazon it's pretty consistent that they do not blame the messenger which is what they consider the person who messed up. Usually that person is the last in a long series of decisions that could have prevented the issue, and thus why blame them. That is unless the person is a) acting with malice, b) is repeatedly shown a pattern of willful ignorance. IIRC, when one person took down S3 with a manual command overriding the safeguards the action was not to fire them but to figure out why it was still a manual process without sign off. Say what you will about Amazon culture, the ability to make mistakes or call them out is pretty consistently protected.
  • tgavVs 2y ago
    > From what I've seen in Amazon it's pretty consistent that they do not blame the messenger which is what they consider the person who messed up Interesting that my experience has been the exact opposite. Whenever I’ve participated in COE discussions (incident analysis), questions have been focused on highlighting who made the mistake or who didn’t take the right precautions.
    • geon 2y ago
      Isn't that a necessary step in figuring out the issue and how t prevent it?
    • dockerd 2y ago
      That was not the idea of COE ever. Probably you were in bad org/team.
      • kelnos 2y ago
        Or maybe you were in an unusually good team? I always chuckle a little when the response to "I had a bad experience" is "I didn't, so you must be an outlier".
        • donavanm 2y ago
          No. The majority of teams and individuals are using it as intended, to understand and prevent future issues from process and tool defects. The complaints Ive heard are usually correlated with other indicators of a “bad”/punitive team culture, a lower level IC not understanding process or intent, or shades of opinion like “its a lot of work and I dont see the benefit. Ergo its malicious or naive.” I worked at aws for 13 years, was briefly in the reliability org that owns the COE (post incident analysis) tooling, and spent a lot if time on “ops” for about 5 years.
    • grogenaut 2y ago
      I've bar raised a ton of them. You do end up figuring out what actions by what operator caused what issues or didn't work well, but that's to diagnose what controls/processes/tools/metrics were missing. I always removed the actual people's name as part of the bar raising, well before publishing, usually before any manager sees it. Instead used Oncall 1, or Oncall for X team, Manager for X team. And that's mainly for the timeline. As a sibling said you were likely in a bad or or one that was using COEs punatively.
• kleton 2y ago
  It was one of the STEP interns that took down Google prod by modifying some config file by putting something erroneous into an automated tool. Everyone at the company was locked out, and someone had to physically access some machines in a datacenter to recover.
• EE84M3i 2y ago
  I'd like to learn more about the AWS incident, but when I google "us-east1 intern" I get this comment. Do you have a link?
  • rafram 2y ago
    Probably this: https://aws.amazon.com/message/41926/ https://aws.amazon.com/message/41926/
    • donavanm 2y ago
      No. That was operational modification of system state using existing tools. The “miss” was an intended subset filter that was not interpreted correctly. > an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. As of a while back that entire state management subsystem, which dates from the very beginning of AWS, has been replaced. Source: me. I was oncall for (some of) the incident management of that event.
• bawolff 2y ago
  There is a huge difference between someone making a mistake and someone intentionally sabotaging. You're not firing the person because they broke stuff, you are firing them because they tried to break stuff. If the attempt was a failure and caused no harm, you would still fire them. Its not about the damage they caused its that they wanted to cause damage.
  • ozim 2y ago
    But for damaging company assets on purpose firing is only first step. I do not see any mention of other legal action and article is shallow. It might’ve been that someone in command chain called it “malicious” to cover up his own mistakes. I think that is parent poster point while writing out Amazon story.
    • bawolff 2y ago
      Maybe, but without any other info, i kind of have to take the info provided at face value. Like obviously if the article is inaccurate the whole situation should be viewed differently.
    • andmarios 2y ago
      The article says: As well as firing the person in August, ByteDance said it had informed the intern's university and industry bodies about the incident.
  • xnavra50 2y ago
    [flagged]
    • Jensson 2y ago
      They were just fired, not put in prison or sued. Getting fired is a typical capitalist punishment, I'd bet way more engineers gets fired for mistakes in USA than China.
    • fragmede 2y ago
      Large powerful groups lying to save face is not a feature of communism, sadly. Stories about the CIA, FBI, and PG&E caught trying to do so come to mind, among others.
• deleted 2y ago
  [deleted]
• deleted 2y ago
  [deleted]
• noobermin 2y ago
  It's a Chinese company, saving face is far more important for them than "teaching lessons" to anyone, particularly employees who are probably considered expendable.
  • throw3828455 2y ago
    I always laugh when I see these predictable comments about "face" when talking about Asian companies, like they are so beholden to their culture they can't make individual judgments. I wonder if we applied this culture talk to Western companies how funny it would sound. The reason Facebook is firing so many people is because individualism "is far more important for them than 'teaching lessons' to anyone, particularly employees who are probably considered expendable."
    • simplify 2y ago
      I don't get it, aren't individual judgements made in the context of culture? How does your example sound funny?
      • gwervc 2y ago
        Every company worldwide, including US ones are trying to "save face" when anything bad happens. This is why we have corporate speech.
        • simplify 2y ago
          Certainly, but how they go about it differs depending on their values, no?
        • noobermin 2y ago
          Kindly, a lot of people are upset about my comment because they're American and have never worked with (or in particular, for) Chinese professionals. Losing face plays a very different role for them than mere embarrassment which is the closest analog in western contexts. For example read this[1]. Individualism does explain many aspects of American culture as do other cultural traits, such as puritanism, a focus on hypocrisy, etc. Face is just one of those aspects of Chinese culture westerns don't really understand unless they've had exposure to it. It however explains many things about modern Chinese culture that are hard to fathom otherwise. https://www.china-mike.com/chinese-culture/cult-of-face/ https://www.china-mike.com/chinese-culture/cult-of-face/
• donavanm 2y ago
  I worked at AWS for 13 years. I did “aws call leader” for 7 years, and worked in the reliability org when we rebuilt the coe tool. Ive personally blown up a service or two, and know other PEs whove done the same or larger. Ive never heard of an individual being terminated or meaningfully punished for making an earnest mistake, regardless of impact. I do know of people who were rapid term’d for malicious, or similar, actions like sharing internal information or (attempting to) subvert security controls. On the whole I did see Amazon “do the right thing” around improving process and tools; people are a fallible _part_ of a system, accountability requires authority, incremental improvements today over a hypothetical tomorrow.
  • zmgsabst 2y ago
    PAM debacle (17Q4) in Device Econ is a counter example. And that wasn’t even a mistake the SDEs made — they were punished for the economists being reckless and subsequently bullied out of the company, despite the SDEs trying to raise the alarm the whole time.
    • donavanm 2y ago
      Is that devices as in digital/alexa land? Never had too much overlap there. AWS and CDO were discrete for incident and problem management after ‘14 or soz
      • zmgsabst 2y ago
        Yeah — my point was Amazon is very large and standards vary. I won’t pretend I know the whole picture, but I’ve seen retaliation against SDEs multiple times. I’ve heard mixed things about CDO, positive things about AWS, but where I worked in Devices and FinTech were both wild… to the point FinTech (circa 2020) didn’t even use the PRFAQ/6-pager methodology. Much to the surprise of people in CDO I asked for advice.
• godelski 2y ago
  I think this is an important distinction and the answer is that it is hard to distinguish. People often bring up the Simple Sabotage Field Manual in situations like these and I think there's something that is often missed: the reason the techniques in here are effective is because they are difficult to differentiate from normal behavior. This creates plausible deniability for the saboteur. Acting too hastily could mean losing someone valuable for a genuine mistake. I'm saying I agree with the Amazon example. (You can also use saboteurs to your advantage if you recognize that they are hunting down and exploiting inefficiencies, but that's a whole other conversation) But my understanding of this case is that the actions do not appear like simple easy to make mistakes. As I understand, the claim was that the intern was modifying the weights of checkpoints for other peoples' training results in an effort to make their own work better. Mucking about in a checkpoint is not a very common thing to do, so should make someone suspicious in the first place. On top of this it appears he was exploiting weaknesses and injecting code to mess with peoples' optimizers, and to do things that do not have a reasonable explanation for. So as far as I can tell, not only was he touching files he shouldn't have been touching (and yes, shouldn't have had access to), he was taking steps to bypass the blocks there were in place and was messing with them in ways that are very difficult to explain away with "I thought this might be a good idea." (Things that explicitly look like a bad idea). If that is what in fact happened, I think it is not a reach to claim intentional sabotage. Because if it wasn't, then the actions are represent such a level of incompetence that they are a huge liability to anyone within reach. [0] https://www.cia.gov/static/5c875f3ec660e092cf893f60b4a288df/SimpleSabotage.pdf https://www.cia.gov/static/5c875f3ec660e092cf893f60b4a288df/...
• Aurornis 2y ago
  > If the intern "had no experience with the AI lab", is it the right thing to do to fire them, instead of admitting that there is a security/access fault internally? This wasn’t an accident, though. The intern had malicious intent and was intentionally trying to undermine other people’s work. This isn’t a case where blameless post-mortems apply. When someone is deliberately sabotaging other people’s work, they must be evicted from the company.