8 ms·

While I appreciate the effect this kind of downtime can have, I just don't understand these stories. Presumably it was planned in advance, so the patients know

While I appreciate the effect this kind of downtime can have, I just don't understand these stories. Presumably it was planned in advance, so the patients know the time of their appointment and the doctor knows what was planned, and everything necessary to physically perform the treatment is already prepared at the hospital. What's stopping them from doing it without filling it into a digital system? Why is it impossible to make a paper record and fill it into the computer system later? If somebody was literally dying, would they stand around the computer like confused characters in a The Sims game who can't find the door, instead of saving the life? And if not, why is this less urgent case different?

j-bos 2y ago
See some of the comments from affected medical staff on the main outage story, but the tldr is, tightly coupled systems.
- throwaway3306a 2y ago
  I get that, my point is, why is it absolutely necessary to use the computer system? Why don't they just knock on the door, go grab the medicine and tools, apply it, then fill it into the system later? I understand they would just postpone whatever can be postponed to save the headache, I don't get the stories about life/health threatening situations.
  - MostlyStable 2y ago
    I can imagine that for something like this procedure, which is an infusion of medication into the brain it sounds like?, that the "tools" to perform the procedure themselves are computer based or computer dependent. It might not be as simple as injecting a drug into an IV line. Note that I am not a doctor and have absolutely no specific knowledge beyond what is in the original article, but I am guessing at potential explanations. Additionally, the article states that there is some "wiffle [sic] room" around the timing of the infusions. So it may be that the delay is not quite as serious as the title makes it sound.
    - throwaway3306a 2y ago
      Presumably they would fix these computers first thing during the night from a backup? If not, is this really about CrowdStrike, and not about a hospital unable to keep their absolutely critical computers backed up and restored in a timely manner? Again, I understand that restoring a complex net of servers is hard and takes time. But they surely have local hospital IT admins for these absolutely critical computers who are always available on site and can do it individually - it's not like there will be more than a hundred of these at a particular hospital? Hack it a little if you have to, disable the SSO etc - all that can be fixed later.
      - advael 2y ago
        The unfortunate fact of the matter is that centralizing IT systems around large corporate products, including the on-prem software and any cloud services, necessarily means less local control of what can go wrong and how it can be mitigated, and thus often problems that simply can't be fixed, even by competent on-prem staff. Even when it is possible, it's often highly illegal, and most organizations do a lot to beat risk-aversion into everyone on their staff, and of course I mean aversion to risk of breaking rules or protocols, not risk like "someone dying" I think it's always a mistake to outsource control of a mission-critical system, but that is exactly what large tech companies have been encouraging every organization that will listen to them to do for decades now
        lambdaone 2y ago
        Absolutely. The risk being managed is the risk to the CEO/CTO's jobs, not the risk to life.
        throwaway3306a 2y ago
        I have trouble accepting that. Even if they had to unplug the computer from the network and disable SSO and antivirus in safe mode, it's possible to get the computer operational. Even if they had to reinstall the OS and the critical software from scratch. There are solutions, the question is - did they even try? If not, why? And is CrowdStrike really to blame if they didn't? I just don't think so.
  - anotherhue 2y ago
    It's because these computers are a means of corporate control. Policies and checks and procedures and whatever are all delivered through them. It's preferable, from the corporate perspective, to have everything fail temporarily than to relinquish this level of workforce management. If this is hard to imagine, just think of a Lyft driver from the perspective of Lyft Inc.
  - ciabattabread 2y ago
    Hotels have difficulty with paper and pen bookings when their computers are down. You expect a modern hospital to function in those circumstances?
    - dopylitty 2y ago
      Yes
    - pasquinelli 2y ago
      the hospital better function. what you're saying is, if the less important service fails, of course the more important one will fail too.
  - j-bos 2y ago
    Have you ever worked a job that requires high degree of physical world logistics? In times where the primary coordination mechanism is down, any action becomes much slower to implement and often at a direct cost to implementing other actions. With regard to this case, I don't know any specifics, but I can imagine tools require digital calibration, inventories not tracked outside digital systems, certain meds behind digital access control, and emergency response striained to the point where complicated non emergency procedures would be more risk than benefit.
    - throwaway3306a 2y ago
      I have managed IT departments that managed hundreds of locations and thousands of computers running Windows XP and Windows Server 2003, no cloud at all. And I went through several similar outages (similar in impact on our operations, not cause or impact on others). Our first priority was to get the critical computers that operated machinery running - we did that hours (1-2) after the problem started. Then we played around with the servers and network for few weeks - but critical stuff was operable, albeit with lesser capacity and efficiency. And we were managing forests and waterways, not hospitals and human lives.
      - TeMPOraL 2y ago
        That's all fine, but this time, no one could get those computers back up in the first few hours, since they were stuck in a boot loop. Plus, systems like hospitals had to be running all that time. Plus, at the scale this outage is reported to be - banks, stores, factories, phones, emergency services, CNC machines, networking, aircon - I imagine everyone was confused and trying to figure out if anything works. I'm happy nothing significant was hit over here in Poland; reading the main HN thread on the outage feels like reading war reports.
        throwaway3306a 2y ago
        If it's stuck in a boot loop, the first thing I do is call the local admins and tell them to take a fresh SSD and a Windows installation USB drive with them. Plug the new SSD, reinstall the OS and copy the files from the old one. Computer running in less than an hour. That's literally what we did to restart our forest logging machinery. Are human lives less critical than that?
  - nullindividual 2y ago
    From having seen the infusion process myself, I take it that it requires precision measurements over an extended period of time. This would seem unreasonable requirement for staff to perform. Again, from what I've seen, infusions are not just "throw it in an IV bag and wait".
    - throwaway3306a 2y ago
      If it requires a computer, why was that operationally critical computer not restored from a backup within hours after the problem started? This has nothing to do with CrowdStrike or other bugs - it could've simply failed hardware wise and the hospital should have been able to replace it immediately.
      - nullindividual 2y ago
        You have a naive view of how modern operations work, I must say. This shows when you suggest endpoints have backups. We're back to the mainframe/terminal times where all software is running on a web server or other centralized application server, which is also in a boot loop, somewhere else. Failed hardware is different, but hospitals likely have very few computers just 'lying around'. Especially the highly regulated machines, such as those which are attached to MRIs and the like. CFR 21 Part 11 was the bane of my existence. Software that can be installed and configured in a matter of minutes? That's a six month project, at least. Sure, backups are great, but then you've got a significant process to get it back up and running. These aren't early-2000 logging operations. I see you'll never be convinced, but this is how modern operations work. Being a hospital (or other industry with heavy government regulations) make operations that much worse.
        throwaway3306a 2y ago
        You misunderstood me, I am easily convinced that this is the case - what I don't get is how they could let it be the case.
        nullindividual 2y ago
- zdragnar 2y ago
  I can get stories like call centers, but I absolutely don't understand how life critical systems aren't air gapped and rigidly controlled. Fail safe is the only acceptable failure mode for any critical system. Crowdstrike failed here, but they're not the only thing that can go wrong with computers. Where is the redundancy?
  - TeMPOraL 2y ago
    Life-critical systems are air-gapped. Just no one considered systems running Epic to be life-critical. It turns out they are, probably more so than most. Also, air-gapping helps only so much when network dies and hospitals can't exchange patient information or send images from MRIs and X-rays to radiologists.
    - Rinzler89 2y ago
      >and hospitals can't exchange patient information or send images from MRIs and X-rays to radiologists My dentist literally took a photo of my x-ray with his phone and sent it to to my orthodontist via Whatsapp and everything went quick and smooth, much faster than the official channels. Solutions to get a job done quickly and efficiently in case of emergency always exist, they're just not "by the book".
      - czl 2y ago
        Imagine a news story about a dentist that violated HIPPA (or equivalent) laws because they used Whatsapp / Facebook to share medical records. Will this news story be about a hero vs someone who got into trouble?
        Rinzler89 2y ago
        How would he get in trouble? Hippa doesn't apply in Europe but GDPR, and I don't see how that would be in violation since my information was exchanged only between the two parties with my consent, on an encrypted channel. They would only get into trouble if that info would leak in an identifiable way to unauthorized third parties and would cause damages (here there's no punitive damages like in the US). And people here tend to guard their WhatsApp chats pretty well since it's what everyone uses and it also contains their private chats so in a sense it can even be more secure than the official medical channels which are just more burocratic but offer no actual guarantee of more data security.
colechristensen 2y ago
It’s not clear to me that this case is actually life threatening. They have a regular procedure which even in the article they say they have wiggle room for timing. If all of your computers go down your throughput is going to go down because other kinds of organization are going to be slower to do ad hoc… so you triage.
dsclough 2y ago
A nurse was unable to give my wife medication while in labor because the barcode on the bag of drugs wouldn’t scan. Fortunately we just had to wait another 20 minutes to get a new bag from the pharmacy but I can easily imagine a world where doctors are unable to perform procedures they are physically capable of doing because of liability surrounding not using the computer systems as intended. Epic particularly has really done a number on the healthcare system.
- throwaway3306a 2y ago
  I really, sincerely don't understand that. How does an unscannable barcode prevent a doctor/nurse from administering medicine they are holding in their hands?
  - simmerup 2y ago
    Because they’re accepting the liability of it going wrong if they make an unusual choice to disregard the error
    - throwaway3306a 2y ago
      So the problem really isn't CrowdStrike or any computer at all, but dumb policy or regulation?
    - advael 2y ago
      I think a good remedy would be to completely remove "normal procedure" as a defense against liability. Our legal standard should defend people who break protocols if they know they will result in harm, and prosecute people who don't, or prosecute the people who make the protocols in those cases. Law should supercede corporate policy, not treat it as a form of law
    - staticman2 2y ago
      That's not how liability works. There is no "I followed some written procedure when it didn't make sense to do so" defense to malpractice claims.
throwup238 2y ago
> What's stopping them from doing it without filling it into a digital system? Why is it impossible to make a paper record and fill it into the computer system later? It’s not filling in new data that’s the problem - every person involved in treatment needs to be able to access the patient’s medical records to check for contraindications. Allergies and drug interactions are a quick way to kill someone when injecting drugs directly into their veins even if they’re already in a hospital. At a major hospital there’s too many patients coming through and the data changes too frequently to keep paper backups.
- moralestapia 2y ago
  >Her children’s appointments were cancelled, the first they would miss in five years. They've been going every two weeks for the last five years. I doubt they wouldn't know what to do ...
  - throwup238 2y ago
    It’s a large children’s hospital with thousands of employees treating tens of thousands of kids a year, not some rural family doctor with a list of patients that can fit on a single sheet of A4. They’re not going to get the same staff every time and the staff isn’t going to memorize the charts of every patient.
    - deleted 2y ago
      [deleted]
renewiltord 2y ago
The truth is that many medical personnel are not agentic. They are human robots unable to act unless instructed to by a computer. The computer tells them when they can do something and they do it.

j-bos 2y ago

See some of the comments from affected medical staff on the main outage story, but the tldr is, tightly coupled systems.

throwaway3306a 2y ago
I get that, my point is, why is it absolutely necessary to use the computer system? Why don't they just knock on the door, go grab the medicine and tools, apply it, then fill it into the system later? I understand they would just postpone whatever can be postponed to save the headache, I don't get the stories about life/health threatening situations.
- MostlyStable 2y ago
  I can imagine that for something like this procedure, which is an infusion of medication into the brain it sounds like?, that the "tools" to perform the procedure themselves are computer based or computer dependent. It might not be as simple as injecting a drug into an IV line. Note that I am not a doctor and have absolutely no specific knowledge beyond what is in the original article, but I am guessing at potential explanations. Additionally, the article states that there is some "wiffle [sic] room" around the timing of the infusions. So it may be that the delay is not quite as serious as the title makes it sound.
  - throwaway3306a 2y ago
    Presumably they would fix these computers first thing during the night from a backup? If not, is this really about CrowdStrike, and not about a hospital unable to keep their absolutely critical computers backed up and restored in a timely manner? Again, I understand that restoring a complex net of servers is hard and takes time. But they surely have local hospital IT admins for these absolutely critical computers who are always available on site and can do it individually - it's not like there will be more than a hundred of these at a particular hospital? Hack it a little if you have to, disable the SSO etc - all that can be fixed later.
    - advael 2y ago
      The unfortunate fact of the matter is that centralizing IT systems around large corporate products, including the on-prem software and any cloud services, necessarily means less local control of what can go wrong and how it can be mitigated, and thus often problems that simply can't be fixed, even by competent on-prem staff. Even when it is possible, it's often highly illegal, and most organizations do a lot to beat risk-aversion into everyone on their staff, and of course I mean aversion to risk of breaking rules or protocols, not risk like "someone dying" I think it's always a mistake to outsource control of a mission-critical system, but that is exactly what large tech companies have been encouraging every organization that will listen to them to do for decades now
      - lambdaone 2y ago
        Absolutely. The risk being managed is the risk to the CEO/CTO's jobs, not the risk to life.
      - throwaway3306a 2y ago
        I have trouble accepting that. Even if they had to unplug the computer from the network and disable SSO and antivirus in safe mode, it's possible to get the computer operational. Even if they had to reinstall the OS and the critical software from scratch. There are solutions, the question is - did they even try? If not, why? And is CrowdStrike really to blame if they didn't? I just don't think so.
- anotherhue 2y ago
  It's because these computers are a means of corporate control. Policies and checks and procedures and whatever are all delivered through them. It's preferable, from the corporate perspective, to have everything fail temporarily than to relinquish this level of workforce management. If this is hard to imagine, just think of a Lyft driver from the perspective of Lyft Inc.
- ciabattabread 2y ago
  Hotels have difficulty with paper and pen bookings when their computers are down. You expect a modern hospital to function in those circumstances?
  - dopylitty 2y ago
    Yes
  - pasquinelli 2y ago
    the hospital better function. what you're saying is, if the less important service fails, of course the more important one will fail too.
- j-bos 2y ago
  Have you ever worked a job that requires high degree of physical world logistics? In times where the primary coordination mechanism is down, any action becomes much slower to implement and often at a direct cost to implementing other actions. With regard to this case, I don't know any specifics, but I can imagine tools require digital calibration, inventories not tracked outside digital systems, certain meds behind digital access control, and emergency response striained to the point where complicated non emergency procedures would be more risk than benefit.
  - throwaway3306a 2y ago
    I have managed IT departments that managed hundreds of locations and thousands of computers running Windows XP and Windows Server 2003, no cloud at all. And I went through several similar outages (similar in impact on our operations, not cause or impact on others). Our first priority was to get the critical computers that operated machinery running - we did that hours (1-2) after the problem started. Then we played around with the servers and network for few weeks - but critical stuff was operable, albeit with lesser capacity and efficiency. And we were managing forests and waterways, not hospitals and human lives.
    - TeMPOraL 2y ago
      That's all fine, but this time, no one could get those computers back up in the first few hours, since they were stuck in a boot loop. Plus, systems like hospitals had to be running all that time. Plus, at the scale this outage is reported to be - banks, stores, factories, phones, emergency services, CNC machines, networking, aircon - I imagine everyone was confused and trying to figure out if anything works. I'm happy nothing significant was hit over here in Poland; reading the main HN thread on the outage feels like reading war reports.
      - throwaway3306a 2y ago
        If it's stuck in a boot loop, the first thing I do is call the local admins and tell them to take a fresh SSD and a Windows installation USB drive with them. Plug the new SSD, reinstall the OS and copy the files from the old one. Computer running in less than an hour. That's literally what we did to restart our forest logging machinery. Are human lives less critical than that?
- nullindividual 2y ago
  From having seen the infusion process myself, I take it that it requires precision measurements over an extended period of time. This would seem unreasonable requirement for staff to perform. Again, from what I've seen, infusions are not just "throw it in an IV bag and wait".
  - throwaway3306a 2y ago
    If it requires a computer, why was that operationally critical computer not restored from a backup within hours after the problem started? This has nothing to do with CrowdStrike or other bugs - it could've simply failed hardware wise and the hospital should have been able to replace it immediately.
    - nullindividual 2y ago
      You have a naive view of how modern operations work, I must say. This shows when you suggest endpoints have backups. We're back to the mainframe/terminal times where all software is running on a web server or other centralized application server, which is also in a boot loop, somewhere else. Failed hardware is different, but hospitals likely have very few computers just 'lying around'. Especially the highly regulated machines, such as those which are attached to MRIs and the like. CFR 21 Part 11 was the bane of my existence. Software that can be installed and configured in a matter of minutes? That's a six month project, at least. Sure, backups are great, but then you've got a significant process to get it back up and running. These aren't early-2000 logging operations. I see you'll never be convinced, but this is how modern operations work. Being a hospital (or other industry with heavy government regulations) make operations that much worse.
      - throwaway3306a 2y ago
        You misunderstood me, I am easily convinced that this is the case - what I don't get is how they could let it be the case.
        nullindividual 2y ago
zdragnar 2y ago
I can get stories like call centers, but I absolutely don't understand how life critical systems aren't air gapped and rigidly controlled. Fail safe is the only acceptable failure mode for any critical system. Crowdstrike failed here, but they're not the only thing that can go wrong with computers. Where is the redundancy?
- TeMPOraL 2y ago
  Life-critical systems are air-gapped. Just no one considered systems running Epic to be life-critical. It turns out they are, probably more so than most. Also, air-gapping helps only so much when network dies and hospitals can't exchange patient information or send images from MRIs and X-rays to radiologists.
  - Rinzler89 2y ago
    >and hospitals can't exchange patient information or send images from MRIs and X-rays to radiologists My dentist literally took a photo of my x-ray with his phone and sent it to to my orthodontist via Whatsapp and everything went quick and smooth, much faster than the official channels. Solutions to get a job done quickly and efficiently in case of emergency always exist, they're just not "by the book".
    - czl 2y ago
      Imagine a news story about a dentist that violated HIPPA (or equivalent) laws because they used Whatsapp / Facebook to share medical records. Will this news story be about a hero vs someone who got into trouble?
      - Rinzler89 2y ago
        How would he get in trouble? Hippa doesn't apply in Europe but GDPR, and I don't see how that would be in violation since my information was exchanged only between the two parties with my consent, on an encrypted channel. They would only get into trouble if that info would leak in an identifiable way to unauthorized third parties and would cause damages (here there's no punitive damages like in the US). And people here tend to guard their WhatsApp chats pretty well since it's what everyone uses and it also contains their private chats so in a sense it can even be more secure than the official medical channels which are just more burocratic but offer no actual guarantee of more data security.

colechristensen 2y ago

It’s not clear to me that this case is actually life threatening. They have a regular procedure which even in the article they say they have wiggle room for timing. If all of your computers go down your throughput is going to go down because other kinds of organization are going to be slower to do ad hoc… so you triage.

dsclough 2y ago

A nurse was unable to give my wife medication while in labor because the barcode on the bag of drugs wouldn’t scan. Fortunately we just had to wait another 20 minutes to get a new bag from the pharmacy but I can easily imagine a world where doctors are unable to perform procedures they are physically capable of doing because of liability surrounding not using the computer systems as intended. Epic particularly has really done a number on the healthcare system.

throwaway3306a 2y ago
I really, sincerely don't understand that. How does an unscannable barcode prevent a doctor/nurse from administering medicine they are holding in their hands?
- simmerup 2y ago
  Because they’re accepting the liability of it going wrong if they make an unusual choice to disregard the error
  - throwaway3306a 2y ago
    So the problem really isn't CrowdStrike or any computer at all, but dumb policy or regulation?
  - advael 2y ago
    I think a good remedy would be to completely remove "normal procedure" as a defense against liability. Our legal standard should defend people who break protocols if they know they will result in harm, and prosecute people who don't, or prosecute the people who make the protocols in those cases. Law should supercede corporate policy, not treat it as a form of law
  - staticman2 2y ago
    That's not how liability works. There is no "I followed some written procedure when it didn't make sense to do so" defense to malpractice claims.

throwup238 2y ago

> What's stopping them from doing it without filling it into a digital system? Why is it impossible to make a paper record and fill it into the computer system later? It’s not filling in new data that’s the problem - every person involved in treatment needs to be able to access the patient’s medical records to check for contraindications. Allergies and drug interactions are a quick way to kill someone when injecting drugs directly into their veins even if they’re already in a hospital. At a major hospital there’s too many patients coming through and the data changes too frequently to keep paper backups.

moralestapia 2y ago
>Her children’s appointments were cancelled, the first they would miss in five years. They've been going every two weeks for the last five years. I doubt they wouldn't know what to do ...
- throwup238 2y ago
  It’s a large children’s hospital with thousands of employees treating tens of thousands of kids a year, not some rural family doctor with a list of patients that can fit on a single sheet of A4. They’re not going to get the same staff every time and the staff isn’t going to memorize the charts of every patient.
  - deleted 2y ago
    [deleted]

renewiltord 2y ago

The truth is that many medical personnel are not agentic. They are human robots unable to act unless instructed to by a computer. The computer tells them when they can do something and they do it.