5 ms·

I have managed IT departments that managed hundreds of locations and thousands of computers running Windows XP and Windows Server 2003, no cloud at all. And I w

I have managed IT departments that managed hundreds of locations and thousands of computers running Windows XP and Windows Server 2003, no cloud at all. And I went through several similar outages (similar in impact on our operations, not cause or impact on others). Our first priority was to get the critical computers that operated machinery running - we did that hours (1-2) after the problem started. Then we played around with the servers and network for few weeks - but critical stuff was operable, albeit with lesser capacity and efficiency. And we were managing forests and waterways, not hospitals and human lives.

TeMPOraL 2y ago
That's all fine, but this time, no one could get those computers back up in the first few hours, since they were stuck in a boot loop. Plus, systems like hospitals had to be running all that time. Plus, at the scale this outage is reported to be - banks, stores, factories, phones, emergency services, CNC machines, networking, aircon - I imagine everyone was confused and trying to figure out if anything works. I'm happy nothing significant was hit over here in Poland; reading the main HN thread on the outage feels like reading war reports.
- throwaway3306a 2y ago
  If it's stuck in a boot loop, the first thing I do is call the local admins and tell them to take a fresh SSD and a Windows installation USB drive with them. Plug the new SSD, reinstall the OS and copy the files from the old one. Computer running in less than an hour. That's literally what we did to restart our forest logging machinery. Are human lives less critical than that?
  - saulpw 2y ago
    You might consider that things have changed in the past 20 years. Also that medicine operates differently than forest logging.
    - throwaway3306a 2y ago
      Things haven't changed in IT so much. I am not in ICT management anymore, but I write software for the modern enterprise systems and networks - I'm reasonably up to date. Ad medicine - hence my question, I'd really like to know what's the blocker. So far it seems the blocker is bad IT management, regulation and liability, not impossibility to perform the treatment.
      - j-bos 2y ago
        Your answers indicate that you have not worked in n environment heavily dependent on ever shifting physical world logistics. You might try talking to some coordinators on the ground of a hospital, rescue center, consteuction site, theme park, or military operation for insight.
        throwaway3306a 2y ago
        I talked to people in charge of the operations on a daily basis for years. I really don't think these considerations have changed that much since my times of leadership of an entire department managing just that.
        cowboylowrez 2y ago
        reminds me of a ticket I once worked on, no emergency or anything, I just needed to crank out a few pages of cobol, it took a little while to type it in, and the boss of the department wanting the ticket done came by and asked what the holdup was, "I'm working on it, will be ready in a bit", the boss asked "what, don't you just need to press a button or something?" she too "lead a department." hahaha

TeMPOraL 2y ago

That's all fine, but this time, no one could get those computers back up in the first few hours, since they were stuck in a boot loop. Plus, systems like hospitals had to be running all that time. Plus, at the scale this outage is reported to be - banks, stores, factories, phones, emergency services, CNC machines, networking, aircon - I imagine everyone was confused and trying to figure out if anything works. I'm happy nothing significant was hit over here in Poland; reading the main HN thread on the outage feels like reading war reports.

throwaway3306a 2y ago
If it's stuck in a boot loop, the first thing I do is call the local admins and tell them to take a fresh SSD and a Windows installation USB drive with them. Plug the new SSD, reinstall the OS and copy the files from the old one. Computer running in less than an hour. That's literally what we did to restart our forest logging machinery. Are human lives less critical than that?
- saulpw 2y ago
  You might consider that things have changed in the past 20 years. Also that medicine operates differently than forest logging.
  - throwaway3306a 2y ago
    Things haven't changed in IT so much. I am not in ICT management anymore, but I write software for the modern enterprise systems and networks - I'm reasonably up to date. Ad medicine - hence my question, I'd really like to know what's the blocker. So far it seems the blocker is bad IT management, regulation and liability, not impossibility to perform the treatment.
    - j-bos 2y ago
      Your answers indicate that you have not worked in n environment heavily dependent on ever shifting physical world logistics. You might try talking to some coordinators on the ground of a hospital, rescue center, consteuction site, theme park, or military operation for insight.
      - throwaway3306a 2y ago
        I talked to people in charge of the operations on a daily basis for years. I really don't think these considerations have changed that much since my times of leadership of an entire department managing just that.
        cowboylowrez 2y ago
        reminds me of a ticket I once worked on, no emergency or anything, I just needed to crank out a few pages of cobol, it took a little while to type it in, and the boss of the department wanting the ticket done came by and asked what the holdup was, "I'm working on it, will be ready in a bit", the boss asked "what, don't you just need to press a button or something?" she too "lead a department." hahaha