8 ms·

Company was hosting cardiac patient monitoring on EC2

442 points by jln 15y ago

protomyth 15y ago
I really hope this is someone's idea of a joke, but the nature of this person's other questions tends to make me believe no. Sometime during the last couple of days, a lawyer was getting a lecture from IT on how the system works and is going to have a very, very bad year.
blantonl 15y ago
Actual quote in the Amazon support forums from this company: "This not just some social network website issue, but a serious threat to peoples lives!"
- kierank 15y ago
  I have a feeling (and I really really hope) this is a joke because of that sentence.
  - aheilbut 15y ago
    An event monitor isn't a life-critical system. Perhaps it messes up a study, and ticks off some cardiologists, but it's not as if somebody's implantable defibrillator is depending on EC2 being online.
  - pstack 15y ago
    I tend to suspect this entire thing, but as I've been working with enterprise teams across the board for over a decade from universities to companies that specialize in space exploration and governments and hospitals, I have discovered that there is sometimes a distinct lack of common sense and that often people with little or no skill end up in positions where they can make drastic decisions. I just have to believe that in this case, were it real, someone would have stopped the plan to be cloud-only before it reached that point. If this is real, it's almost sickening on two levels. The first and most obvious is the risk the company exposed clients to in their deployment. Second, the lack of communication to paying customers from Amazon. I'm a fan of Amazon and probably spend more money there than at any other company on the planet and am disappointed whenever I see a business that puts up a giant wall between their paying customers and the support they need. On the other hand, that comment leads us to a great idea for a startup: Social ECG Monitoring with a dash of Gamification. I even got all the flashy keywords in there!
- deleted 15y ago
  [deleted]
st0p 15y ago
That's not even stupid, that is moronic.
- lawnchair_larry 15y ago
  Why? Who else would you suggest for providing this type of service? Given the information we had a week ago, I can see how Amazon may have looked like the best solution for HA. Their site never goes down, after all.
  - leon_ 15y ago
    So what? Even if God himself would promise 100% uptime I'd better have 2 or 3 fallbacks for such a service.
  - st0p 15y ago
    Well, first of all if they had used more geographical regions (I don't know what the official amazon buzzword), they shouldn't have real bad problems. I don't know about you, but I believe mission critical systems should be hosted in at least 2 physically different data centers. When it's life critical, this should be a no brainer. Second of all, according to Amazon PR speak this failure should not be possible. Every person in IT worth their salt should know PR speak is not to be trusted. Being reliant on AWS for your average business app is completely okay though. However, hosting apps which might endanger peoples lives on black box technology over which you have little influence is moronic imo. Last but not least I seriously question the privacy implications of this. Hosting parts of your medical app in the cloud is fine and I don't know the details of this app, but I wouldn't be happy to know that my medical history is stored by some third party I don't entirely trust with such sensitive data.
    - lawnchair_larry 15y ago
      1) Amazon's outages took down multiple "availability zones" 2) Amazon is HIPAA certified, which means they adhere to the industry standard for storing private medical data. In this sense, they are as good as any other company that such data is being passed around.
sliverstorm 15y ago
Hmm... That would rather seem to be the monitoring company's own fault. Admittedly I haven't used ec2 for about 3 years, but I seem to recall there being an explicit disclaimer about how it wasn't supposed to be used for this kind of uptime-critical application. I want to say it even specifically mentioned medical stuff.
- zaidf 15y ago
  Agree. I'd avoid cloud for half as critical apps where I cannot CLEARLY picture where my servers are in a physical sense and how I can physically get a hold of them if shit hits the fan.
  - jrockway 15y ago
    How do you physically get a hold of your servers if the gas main under them blows up or a tornado sucks them up and drops them a hundred miles away? Servers are not physical things that you can just go get when there are problems. They are imaginary devices that can suddenly vaporize, leaving no trace. Plan your deployment accordingly, and you won't need to be writing emails saying that people are dying because your data center blew up. Your systems will silently switch over to some other data center and you'll file a claim with your insurance company to have the servers replaced on Monday morning. In other words, "the cloud" had nothing to do with this problem. It could happen if the server was physically attached to you or in a abandoned missle silo with an army of armed guards.
- gte910h 15y ago
  EC2 is fine for part of the servers. I'd honestly say you should have multiple locations of servers, managed via different means (so the single points of failure are not there). EC2 just shouldn't be ALL of the servers.
- madaxe 15y ago
  Nah, you can use EC2 for uptime-critical apps, you just have to a) not be a dipshit and b) stick your stuff in as many AZs as you can, and ELB it.
mkramlich 15y ago
"We'll gladly agree to take care of grandma as long as you understand we only guarantee a 99% uptime. Good? Good. Sign here."
nblavoie 15y ago
Wow, this is the worse idea I've ever seen since a freaking while. Poor him (or them) though.
arkitaip 15y ago
Surely this must be criminal? This is one area where atleast 99.9999% uptime should be mandatory.
chrisjsmith 15y ago
Epic fail for the company which is hosting on their kit. Whoever threw it on there is a total dick. There is nothing wrong with EC2 here. It's just not the right tool for the job. If they are monitoring cardiac patients, they should, at least for non-critical people, have the instrumentation cache the data and send when available over leased lines. If it is critical, it should be PROPER HARDWARE attached to the POTS network (like we have in the UK!) with multiple redundant networks over several distinct carriers and multiple monitoring stations. It sounds like their product is a) crap and b) dangerous if used for critical care.
- MichaelApproved 15y ago
  Ec2 is fine here. All systems have downtime. Its not having a backup system in place that's the problem.
  - chrisjsmith 15y ago
    Cardiac monitoring systems DO NOT have downtime believe it or not.
    - Bo102010 15y ago
      Do not believe.
    - ceejayoz 15y ago
      Bullshit. Leads come off, equipment malfunctions, patients fidget, and they definitely don't have redundant stuff on each patient.
      - muyuu 15y ago
        I'm going to guess grandparent means that they don't include "downtime" in the contract. Which would be outrageous, BTW.
    - jrockway 15y ago
      He's saying that any computer system hosting the data would have downtime. Your own custom data center can blow up just as easily as Amazon's. Now, if zero downtime is the requirement, then you just need to have a lot of data centers and well-tested failover procedures. It's all very simple, and EC2 can easily be one of those redundant data centers.
dasil003 15y ago
Troll?
MichaelApproved 15y ago
I hope no one dies because of this. Yet, the sad part is, it'll take someones death to get serious attention and help prevent this stupidity in the future. Of course, then things will swing wildly in the other direction with massive legislation and regulation thats designed to get a lawmaker reelected more than it is to save lives.
beck5 15y ago
This is the sort of service which should have a back up, its not reddit. If it can go wrong it will go wrong.
gte910h 15y ago
I see EC2 as a good piece of infrastructre to have as part of a failover plan. But should it be even 1/3 the providers for servers? No.
- bluesnowmonkey 15y ago
  Triply redundant hosting providers is really setting the bar high. Lot of 20/20 hindsight being put into play the last few days! In my experience, small companies typically have one hosting provider, off-site backups, and the probable but untested ability to bring up a fresh installation in a few hours. That's both common in the real world and good enough for most situations. It would get you through an outage like this.
  - gte910h 15y ago
    It's a medical device. It's not a imposition in that case.
zaidf 15y ago
This is where you wish you had a data center address and a cabinet number where you could pop in, grab your servers and move the heck out to another DC. I wonder if this will get more companies to maintain a non-cloud, vanilla setup on some dedicated or collocated boxes. May be we should have an annual day to bring down our primary servers and see how the backups do. The idea shouldn't be to confirm if everything works or not. It should be to determine what did work and did not work.
- sliverstorm 15y ago
  You know, that gets me thinking- wouldn't it be pretty damn slick if Amazon offered a way to plug your own hardware into the greater EC2 cluster remotely, so that if Big EC2 goes down, you still have copies of your machines running locally? Maybe you could even timeshare your machines out for some other instances for a discount on your service, similar to how you can send solar power back up the power grid for an electricity credit. Then ec2 really WOULD be a cloud!
  - ceejayoz 15y ago
    That sounds horrid. It'd mean my instances might be running on your ancient crap computer in a closet hooked up to dialup.
    - sliverstorm 15y ago
      You could always add min spec requirements for machines to join, increase redundancy of instances ec2-wide, add quality tiers that control what machines your instances go on, etc. Amazon could even simply benchmark your server and network connection, and compensate you as a function of that, which would drastically reduce the incentive to hook up old crappy hardware to the cloud. It's really the idea of being able to run copies of your OWN instances locally that intrigues me though.
      - ceejayoz 15y ago
        > It's really the idea of being able to run copies of your OWN instances locally that intrigues me though. No reason you can't already do this.
byoung2 15y ago
We are a monitoring company and are monitoring hundreds of cardiac patients at home. If this is a serious post and there really is a service that monitors cardiac patients from home, first you would have to deal with the unreliability of in-home broadband connections. What if a router needs to be reset, or the DSL goes out? Even if you ignore that, you would expect that any server setup will go down at some time. It has happened to Google, Facebook, Twitter, and now AWS and PlayStation Network. I remember when Rackspace went down a few years back (a truck crashed into the transformer that powered the datacenter - http://www.datacenterknowledge.com/archives/2007/11/13/truck-crash-knocks-rackspace-offline/ http://www.datacenterknowledge.com/archives/2007/11/13/truck...). All of Armenia was knocked offline when a woman cut a fiber optic line while digging for scrap metal. I'm not sure I'd trust the internet with any kind of life or death monitoring.
- zaidf 15y ago
  I don't think it's so much about trust as alternatives and best practices. Ideally you should not trust any system when running mission critical apps and always have multiple levels of backups and monitoring including humans on call.
- esrauch 15y ago
  I could be way off base here since I only know what I saw on the site, but it doesn't look like this is life-or-death monitoring. It looks like its long term monitoring for later review by a doctor. It doesn't seem at all likely that they have someone monitoring the incoming data in realtime for heart attacks, that would be far better suited by hooking up the system to dial 911. I think some IT person was freaking out that they had downtime and slightly exaggerated the life endangering part, they likely just lost information that may have been used in a future diagnostic manner.
  - dr_ 15y ago
    Seems like it was real time monitoring. Otherwise why not just use the traditional Holter monitor, which is later reviewed by a doctor?
csomar 15y ago
Isn't this a good start-up idea? A start-up that backup your website and deploy it to another hosting in case the one you are using is down.
- troels 15y ago
  How about a host-abstraction layer, so that you can deal with different providers through a uniform interface?
  - csomar 15y ago
    I'm actually thinking of that. You deploy to the service that take care of deployment, backups and transfer in down times. It would make up of it another kind of hosting company, though.
    - troels 15y ago
      Yes, it would essentially be a hosting company with multiple providers. That in it self is a single point of failure, but at least it would give some insurance against outage of the more physical kind.
cypherpunks 15y ago
I'd like to see more back story first. Medicine is about risk management. There are places where you need 1/100,000 reliability, and there are places where you need 1/1000, and there are even places where 1/10 is good enough. There are primary systems, and there are auxiliary systems. Without any background, it is really tough to tell where this fits. You can't blame the people for making the wrong decision until you know what the risk profile for the failure is.
recampbell 15y ago
Interesting quote from a company providing such services: "The Remote Health Monitoring System is the IT backbone that supports HealthFrontier’s entire portfolio of solutions, including the ecg@home™ and the microtel™/ecgAnywhere™. It captures data transmitted by the patient through USB, Bluetooth, or trans-telephonically. The RHMS™ then stores the information in a database, which can be accessed by the patient’s doctor through a web-based interface. Benefits include: Cloud computing: The RHMS is hosted on a network of servers across the US, which increases reliability and eliminates the need for a large capital investment in on-site hardware and maintenance." http://www.healthfrontier.com/rhms.php http://www.healthfrontier.com/rhms.php How many other companies provide home ECG monitoring which report over the web? I'm looking but having a hard time finding more. I will update this post as I do. [UPDATE] Other Home-based ECG monitoring services: Medtronic CareLink: http://www.medtronic.com/your-health/heart-failure/device/cardiac-resynchronization-therapy-defibrillators/carelink/index.htm http://www.medtronic.com/your-health/heart-failure/device/ca... [UPDATE] A more complete listing: http://www.medcompare.com/matrix/1475/Outpatient-Cardiac-Monitoring-(Telemetry)-Services.html http://www.medcompare.com/matrix/1475/Outpatient-Cardiac-Mon...
- yellowbkpk 15y ago
  This is very popular. There are several products that take ECG samples at timed intervals and submit them in several different ways. In addition these devices can be used to take 12-lead EKG measurements in transit (e.g. inside the ambulance) and send them to the doctor for analysis before the patient arrives. I'm very surprised that they're hosting these services on EC2. The company I worked for recently finished building a huge highly-redundant data center specifically designed for hosting this kind of medically-sensitive information. I'm willing to bet it was a bit more fortified than even Amazon's.
- jmm 15y ago
  My uncle owns a few elder care franchises and had me look into monitoring services as a funemployment project. So I've spent a bit more time with this than I'd like to admit... The company I was most impressed with was http://www.halomonitoring.com http://www.halomonitoring.com . The device seems to be designed well and the monitoring services (online portal for caregivers or family members) are impressive, comparatively. There's kind of a sea change happening with home health monitoring -- from the "I've fallen and can't get up" devices to more robust and interactive solutions. GE recently bought a company that provides institutional monitoring equipment and services. Lemme know if you want the full report :)
ceejayoz 15y ago
Later in the thread they say two instances are up but the third has most of the patients. If any of their patients die, they should be convicted of negligent manslaughter, not Amazon. Amazon explicitly warns that single instances can be lost. Any EC2 system needs redundancy, let alone a patient critical one!
logjam 15y ago
If this story is true (and I have some doubts), you'd think this would have been a no-brainer to not try to keep critical data like this in the cloud. But even for non-critical medical data, non-tech savvy medical folks (and this includes some physicians, although in general physicians are more tech-savvy than most managed care decision makers) are going to get into increasingly serious trouble if they don't think through these kinds of things very carefully. Driven by electronic health care mandates and incentives, companies are pushing portable devices for medical data gathering (portable as in tablet, ie without local data store), and some groups are going to be knee-jerk tempted into buying into this without thinking through the ramifications of not owning your own data -- and by that, I don't mean vague marketing-speak hand-waving that "you own your data" through the simple mechanism of password-protected access to your patient's data living on someone's server farm. By actually owning your patient's data, I mean strongly considering restricting your local practice to use of on-site media. If you don't, you'd better have complete confidence that putting your patient's data on the web reflects a reasonable standard of care/privacy, that retention policies don't bite you in six months when your cloud provider goes out of business (or changes hands or farms out data storage or backup to some "economical" third-party), that you're going to have access to your data when you need it, that your cloud provider is going to audit access and changes to your patient's data and report the results of that audit to you accurately and regularly, that legal requests for data are not acted upon without you or your patient's consent, that backups are subjected to the same levels of security, etc. etc. Electronic medical records are a good idea. It can be done right, as systems such as the Veteran's Administration's VISTA system have demonstrated. However, there is a fine line to walk here. Implementing EHR for the sake of government incentive, or for the convenience of insurance companies, or even the convenience of walking into an exam room carrying a light-weight tablet (which have their own disadvantages for data gathering) rather than a laptop with a hard-drive isn't going to cut it when you put your patients and your practice at risk of harm.
teilo 15y ago
Elastic Load Balancer, anyone? I have no sympathy for this company. They deserve to get sued for this. Yes, for something life threatening like this, EC2 is a bad idea, but they didn't even bother to take advantage of the geographic redundancy, much less something so basic as having backup AMIs ready in another AZ in your current region. Of course, a whole lot of companies are learning this now.
- gregburek 15y ago
  Unfortunately, this last AWS problem affected multiple AZs in the us-east region. The OP very well may have had an alternate AZ failover plan, but like Quora, Sencha, Reddit, FourSquare and Heroku, they probably kept it region specific. As for backing up to multiple regions, I can imagine them thinking that sending everything over the public internet as being a bad idea. However, not having a multi-region/multi-provider failover plan was a worse one.
- kinofcain 15y ago
  ELB doesn't work across regions, the failure at EC2 affected multiple Availability zones in the Virginia Region. ELB might have helped, but it's likely that some people had all their "availabiliy" zones knocked out. In fact, it's looking like all the automatic-failovers from people hosting on those data centers compounded the problem when everyone's scripts tried to recover at the same time – what was referred to in a previous article as a "bank run". But yeah, if you're doing heart monitoring, you need to have your servers replicated across a wide geographic area.
jamesbkel 15y ago
I wish they offered some more detail on how important/life-threatening this is. Are we talking monitoring as in "detect signs of impending/occurring cardiac event and notify emergency services"? Or is this monitoring in a longitudinal sense, assisting physicians to look at trends in their patients? If it is the later case, is data stored on the device until it's uploaded, or is it simply lost if the connection fails? Still, not really forgivable to be 100% AWS, but it's a very different story if this is primarily a recording device vs. a monitoring device.
MSDjunior 15y ago
who was the IT professional that decided to rely soley on the AWS cloud without any kind of "just in case" solution....with people's lives at stake? lol. I think I'm gonna subscribe to the hoax theory
tybris 15y ago
Those people should be prosecuted for utter incompetence.
forgotAgain 15y ago
This is too incredible to be true. But if it is true then I would expect the company in question to be out of business quite soon. A company that put's a life critical system on AWS shows a total disregard for their patients well being.
- jauer 15y ago
  I've seen things much worse than this with medical records and data. Up to a few months ago we had a ASP.NET shared hosting customer that was doing some kind of data relay web service for medical imaging. No encryption. Patient data in full view on the server. No redundancy. Apparently it was used for outsourcing imagery review or something. If it didn't work doctors would have to drive in from home which slowed down the diagnostic process. "Mission critical" on a $30 a month shared hosting plan. Very much not HIPPA compliant.
renegadedev 15y ago
I recently had the "good" fortune of being at the receiving end of a sales pitch from one cloud hosting service (not Amazon). The pitch centered more around flexibility (ability to add CPUs, RAM, storage, network nodes) rather than reliability issues as the one EC2 is facing right now. All questions about reliability were brushed aside with blanket statements like "it's in the cloud" as if that was an end-all-be-all. After much prodding the sales rep admitted that the hosting was all in one location (low lying hurricane prone area) and if we desired redundancy we would have to mirror our apps at a different geographical location. As with any other product, don't believe the sales pitch. If it sounds too good to be true, it probably is. Always use redundancy specially for critical apps.
_3u10 15y ago
This reminds me of doing tech support back during the dot bomb. Everyday we'd have day traders phone in complaining that their internet was down and they were losing thousands of dollars. Then I'd advise them to switch to their backup connection, then they'd say they didn't have one. Finally, I'd ask why if they were running a business that could generate / lose thousands of dollars in a few minutes why they had chosen a residential internet service instead of opting for the business package and why they did not have a backup connection as neither the business packages or residential packages came with a SLA that was suitable for the type of risk inherent in their business. If their data is so important why are they hosting it on the absolute cheapest and shittiest servers they can find without a backup solution in place? If they read the TOS for EC2 it probably says not to run critical infrastructure on it. Perhaps for sensitive data upon which people's lives depend they might want to look into something like:http://h20223.www2.hp.com/nonstopcomputing/cache/76385-0-0-0-121.html?jumpid=go/integritynonstop http://h20223.www2.hp.com/nonstopcomputing/cache/76385-0-0-0... Yes, it costs more than EC2, but there's a reason for that. Wait until this company finds out that if their values exceed what MySQL expects it will silently truncate it.
stick 15y ago
This is a combination of posters not understanding what the technology is used for and the tech guy exaggerating the urgency of the situation. Remote cardiac monitoring isn't for people who are imminently going to die of catastrophic heart attacks or who risk dropping into a fatal arrhythmia (wonky heart rhythm). In fact, there isn't even solid evidence that it saves lives. What it is used for is investigating patients who have vague symptoms which might be related to transient changes in their heart beat. These are people who come to their doctor at the age of 54 and say "I felt my heart beating really quickly and felt kind of faint" or "I felt kind of dizzy and then passed out. It's happened twice in the last month." The more traditional way to investigate these patients are with what's called a Holter monitor, as alluded to by a previous poster. These are little belt packs you carry around while wired-up and that record your ECG for 24-48 hours at a time. The main weakness? You said it: the device only has a 24-48 hour windows to capture the weird, often rare, rhythm. There are different ways remote cardiac monitoring systems report their results and I'm not sure which this particular company was using, but it doesn't really matter. Some of them only report weird stretches, others only report events when the patient says they're feeling symptom X, others are reporting continuously. Take-home message: this is not life-or-death data. When doctors (at least those who are allowed to keep practicing) think a patient needs critical cardiac monitoring, they admit them to hospital. This was a tech guy, looking to jump the queue by trying to raise a red flag because his servers were being used for--OMFG!--cardiac monitoring. A lot of my doctor buddies use similar strategies when they're caught speeding by the police. "I just got called in to the hospital!" …to see a patient with a really nasty rash.
- mamp 15y ago
  From the website I think this analysis is correct. Holters store the 24 hour data on the machine as well. The 12 leads are not continuous but set pieces which also store locally. If they were providing real-time monitoring with alerts then this would be more serious. This seems to be a backup system, ironically itself without backups.
- GrandMasterBirt 15y ago
  a) what stick said. Not such a critical system. b) Good to know from responses that amazon's paid support is crap. They'll officially tell you "were on it". c) People are bashing this person for relying on amazon not failing all at once. I would be surprised if they hosted on amazon relying on each individual node being up 99.98% of the time. That is crazy. You build on cloud computing knowing that each one node can fail at any moment, but you build it to fall over to another node. From experience, sometimes nodes just... die... or get really unstable. d) People bashing the dude vs offering future advice. I'm sure if his company is in deep shit, he's well aware of it.
FreshCode 15y ago
Bullshit, linkbait post. After reading the whole thing, I would gladly spend karma to downvote this.
- lell 15y ago
  Are you referring to the link in the title? It is true that the company was hosting cardiac patient monitoring on EC2. This was not life threatening infrastructure, but that is not implied by the HN title. How is it linkbait? The poster on Amazon's forum misrepresented the severity of the infrastructure and tries to backpedal later on in the thread. This is an example of someone using EC2 and fucking up, firstly for hosting production level infrastructure without backups, and secondly for misrepresenting the importance of the infrastructure in an attempt to get more technical support, making them look liable for some sort of medical malpractice. The poster probably panicked when they realised their first mistake, and thought they were showing that they were trying to get back in control. I don't think the poster was totally stupid, maybe she/he didn't understand the risks of EC2, or maybe it wasn't even their decision to have no backups --- in which case they should have pressed harder at meetings or refused to continue to work on the project. And then they panicked. We can all learn something from this.
yaix 15y ago
Just wow! How difficult can it be to have a backup dedi server with the DB mirrored running at some other provider to quickly switch to, just in case? Even many simple web sites have a config like that. Clouds have been around for decades, they are still new and may fail.
rbanffy 15y ago
So, they experienced a Schrödinger situation...
code_duck 15y ago
This is just supplementary monitoring, not a true life-or-death situation. Everyone (the company involved included) is welcome to stop blowing it out of proportion any time.

442 points by jln 15y ago

I really hope this is someone's idea of a joke, but the nature of this person's other questions tends to make me believe no. Sometime during the last couple of days, a lawyer was getting a lecture from IT on how the system works and is going to have a very, very bad year.

blantonl 15y ago

Actual quote in the Amazon support forums from this company: "This not just some social network website issue, but a serious threat to peoples lives!"

kierank 15y ago
I have a feeling (and I really really hope) this is a joke because of that sentence.
- aheilbut 15y ago
  An event monitor isn't a life-critical system. Perhaps it messes up a study, and ticks off some cardiologists, but it's not as if somebody's implantable defibrillator is depending on EC2 being online.
- pstack 15y ago
  I tend to suspect this entire thing, but as I've been working with enterprise teams across the board for over a decade from universities to companies that specialize in space exploration and governments and hospitals, I have discovered that there is sometimes a distinct lack of common sense and that often people with little or no skill end up in positions where they can make drastic decisions. I just have to believe that in this case, were it real, someone would have stopped the plan to be cloud-only before it reached that point. If this is real, it's almost sickening on two levels. The first and most obvious is the risk the company exposed clients to in their deployment. Second, the lack of communication to paying customers from Amazon. I'm a fan of Amazon and probably spend more money there than at any other company on the planet and am disappointed whenever I see a business that puts up a giant wall between their paying customers and the support they need. On the other hand, that comment leads us to a great idea for a startup: Social ECG Monitoring with a dash of Gamification. I even got all the flashy keywords in there!
deleted 15y ago
[deleted]

st0p 15y ago

That's not even stupid, that is moronic.

lawnchair_larry 15y ago
Why? Who else would you suggest for providing this type of service? Given the information we had a week ago, I can see how Amazon may have looked like the best solution for HA. Their site never goes down, after all.
- leon_ 15y ago
  So what? Even if God himself would promise 100% uptime I'd better have 2 or 3 fallbacks for such a service.
- st0p 15y ago
  Well, first of all if they had used more geographical regions (I don't know what the official amazon buzzword), they shouldn't have real bad problems. I don't know about you, but I believe mission critical systems should be hosted in at least 2 physically different data centers. When it's life critical, this should be a no brainer. Second of all, according to Amazon PR speak this failure should not be possible. Every person in IT worth their salt should know PR speak is not to be trusted. Being reliant on AWS for your average business app is completely okay though. However, hosting apps which might endanger peoples lives on black box technology over which you have little influence is moronic imo. Last but not least I seriously question the privacy implications of this. Hosting parts of your medical app in the cloud is fine and I don't know the details of this app, but I wouldn't be happy to know that my medical history is stored by some third party I don't entirely trust with such sensitive data.
  - lawnchair_larry 15y ago
    1) Amazon's outages took down multiple "availability zones" 2) Amazon is HIPAA certified, which means they adhere to the industry standard for storing private medical data. In this sense, they are as good as any other company that such data is being passed around.

sliverstorm 15y ago

Hmm... That would rather seem to be the monitoring company's own fault. Admittedly I haven't used ec2 for about 3 years, but I seem to recall there being an explicit disclaimer about how it wasn't supposed to be used for this kind of uptime-critical application. I want to say it even specifically mentioned medical stuff.

zaidf 15y ago
Agree. I'd avoid cloud for half as critical apps where I cannot CLEARLY picture where my servers are in a physical sense and how I can physically get a hold of them if shit hits the fan.
- jrockway 15y ago
  How do you physically get a hold of your servers if the gas main under them blows up or a tornado sucks them up and drops them a hundred miles away? Servers are not physical things that you can just go get when there are problems. They are imaginary devices that can suddenly vaporize, leaving no trace. Plan your deployment accordingly, and you won't need to be writing emails saying that people are dying because your data center blew up. Your systems will silently switch over to some other data center and you'll file a claim with your insurance company to have the servers replaced on Monday morning. In other words, "the cloud" had nothing to do with this problem. It could happen if the server was physically attached to you or in a abandoned missle silo with an army of armed guards.
gte910h 15y ago
EC2 is fine for part of the servers. I'd honestly say you should have multiple locations of servers, managed via different means (so the single points of failure are not there). EC2 just shouldn't be ALL of the servers.
madaxe 15y ago
Nah, you can use EC2 for uptime-critical apps, you just have to a) not be a dipshit and b) stick your stuff in as many AZs as you can, and ELB it.

mkramlich 15y ago

"We'll gladly agree to take care of grandma as long as you understand we only guarantee a 99% uptime. Good? Good. Sign here."

nblavoie 15y ago

Wow, this is the worse idea I've ever seen since a freaking while. Poor him (or them) though.

arkitaip 15y ago

Surely this must be criminal? This is one area where atleast 99.9999% uptime should be mandatory.

chrisjsmith 15y ago

Epic fail for the company which is hosting on their kit. Whoever threw it on there is a total dick. There is nothing wrong with EC2 here. It's just not the right tool for the job. If they are monitoring cardiac patients, they should, at least for non-critical people, have the instrumentation cache the data and send when available over leased lines. If it is critical, it should be PROPER HARDWARE attached to the POTS network (like we have in the UK!) with multiple redundant networks over several distinct carriers and multiple monitoring stations. It sounds like their product is a) crap and b) dangerous if used for critical care.

MichaelApproved 15y ago
Ec2 is fine here. All systems have downtime. Its not having a backup system in place that's the problem.
- chrisjsmith 15y ago
  Cardiac monitoring systems DO NOT have downtime believe it or not.
  - Bo102010 15y ago
    Do not believe.
  - ceejayoz 15y ago
    Bullshit. Leads come off, equipment malfunctions, patients fidget, and they definitely don't have redundant stuff on each patient.
    - muyuu 15y ago
      I'm going to guess grandparent means that they don't include "downtime" in the contract. Which would be outrageous, BTW.
  - jrockway 15y ago
    He's saying that any computer system hosting the data would have downtime. Your own custom data center can blow up just as easily as Amazon's. Now, if zero downtime is the requirement, then you just need to have a lot of data centers and well-tested failover procedures. It's all very simple, and EC2 can easily be one of those redundant data centers.

dasil003 15y ago

Troll?

MichaelApproved 15y ago

I hope no one dies because of this. Yet, the sad part is, it'll take someones death to get serious attention and help prevent this stupidity in the future. Of course, then things will swing wildly in the other direction with massive legislation and regulation thats designed to get a lawmaker reelected more than it is to save lives.

beck5 15y ago

This is the sort of service which should have a back up, its not reddit. If it can go wrong it will go wrong.

gte910h 15y ago

I see EC2 as a good piece of infrastructre to have as part of a failover plan. But should it be even 1/3 the providers for servers? No.

bluesnowmonkey 15y ago
Triply redundant hosting providers is really setting the bar high. Lot of 20/20 hindsight being put into play the last few days! In my experience, small companies typically have one hosting provider, off-site backups, and the probable but untested ability to bring up a fresh installation in a few hours. That's both common in the real world and good enough for most situations. It would get you through an outage like this.
- gte910h 15y ago
  It's a medical device. It's not a imposition in that case.

zaidf 15y ago

This is where you wish you had a data center address and a cabinet number where you could pop in, grab your servers and move the heck out to another DC. I wonder if this will get more companies to maintain a non-cloud, vanilla setup on some dedicated or collocated boxes. May be we should have an annual day to bring down our primary servers and see how the backups do. The idea shouldn't be to confirm if everything works or not. It should be to determine what did work and did not work.

sliverstorm 15y ago
You know, that gets me thinking- wouldn't it be pretty damn slick if Amazon offered a way to plug your own hardware into the greater EC2 cluster remotely, so that if Big EC2 goes down, you still have copies of your machines running locally? Maybe you could even timeshare your machines out for some other instances for a discount on your service, similar to how you can send solar power back up the power grid for an electricity credit. Then ec2 really WOULD be a cloud!
- ceejayoz 15y ago
  That sounds horrid. It'd mean my instances might be running on your ancient crap computer in a closet hooked up to dialup.
  - sliverstorm 15y ago
    You could always add min spec requirements for machines to join, increase redundancy of instances ec2-wide, add quality tiers that control what machines your instances go on, etc. Amazon could even simply benchmark your server and network connection, and compensate you as a function of that, which would drastically reduce the incentive to hook up old crappy hardware to the cloud. It's really the idea of being able to run copies of your OWN instances locally that intrigues me though.
    - ceejayoz 15y ago
      > It's really the idea of being able to run copies of your OWN instances locally that intrigues me though. No reason you can't already do this.

byoung2 15y ago

We are a monitoring company and are monitoring hundreds of cardiac patients at home. If this is a serious post and there really is a service that monitors cardiac patients from home, first you would have to deal with the unreliability of in-home broadband connections. What if a router needs to be reset, or the DSL goes out? Even if you ignore that, you would expect that any server setup will go down at some time. It has happened to Google, Facebook, Twitter, and now AWS and PlayStation Network. I remember when Rackspace went down a few years back (a truck crashed into the transformer that powered the datacenter - http://www.datacenterknowledge.com/archives/2007/11/13/truck-crash-knocks-rackspace-offline/ http://www.datacenterknowledge.com/archives/2007/11/13/truck...). All of Armenia was knocked offline when a woman cut a fiber optic line while digging for scrap metal. I'm not sure I'd trust the internet with any kind of life or death monitoring.

zaidf 15y ago
I don't think it's so much about trust as alternatives and best practices. Ideally you should not trust any system when running mission critical apps and always have multiple levels of backups and monitoring including humans on call.
esrauch 15y ago
I could be way off base here since I only know what I saw on the site, but it doesn't look like this is life-or-death monitoring. It looks like its long term monitoring for later review by a doctor. It doesn't seem at all likely that they have someone monitoring the incoming data in realtime for heart attacks, that would be far better suited by hooking up the system to dial 911. I think some IT person was freaking out that they had downtime and slightly exaggerated the life endangering part, they likely just lost information that may have been used in a future diagnostic manner.
- dr_ 15y ago
  Seems like it was real time monitoring. Otherwise why not just use the traditional Holter monitor, which is later reviewed by a doctor?

csomar 15y ago

Isn't this a good start-up idea? A start-up that backup your website and deploy it to another hosting in case the one you are using is down.

troels 15y ago
How about a host-abstraction layer, so that you can deal with different providers through a uniform interface?
- csomar 15y ago
  I'm actually thinking of that. You deploy to the service that take care of deployment, backups and transfer in down times. It would make up of it another kind of hosting company, though.
  - troels 15y ago
    Yes, it would essentially be a hosting company with multiple providers. That in it self is a single point of failure, but at least it would give some insurance against outage of the more physical kind.

cypherpunks 15y ago

I'd like to see more back story first. Medicine is about risk management. There are places where you need 1/100,000 reliability, and there are places where you need 1/1000, and there are even places where 1/10 is good enough. There are primary systems, and there are auxiliary systems. Without any background, it is really tough to tell where this fits. You can't blame the people for making the wrong decision until you know what the risk profile for the failure is.

recampbell 15y ago

Interesting quote from a company providing such services: "The Remote Health Monitoring System is the IT backbone that supports HealthFrontier’s entire portfolio of solutions, including the ecg@home™ and the microtel™/ecgAnywhere™. It captures data transmitted by the patient through USB, Bluetooth, or trans-telephonically. The RHMS™ then stores the information in a database, which can be accessed by the patient’s doctor through a web-based interface. Benefits include: Cloud computing: The RHMS is hosted on a network of servers across the US, which increases reliability and eliminates the need for a large capital investment in on-site hardware and maintenance." http://www.healthfrontier.com/rhms.php http://www.healthfrontier.com/rhms.php How many other companies provide home ECG monitoring which report over the web? I'm looking but having a hard time finding more. I will update this post as I do. [UPDATE] Other Home-based ECG monitoring services: Medtronic CareLink: http://www.medtronic.com/your-health/heart-failure/device/cardiac-resynchronization-therapy-defibrillators/carelink/index.htm http://www.medtronic.com/your-health/heart-failure/device/ca... [UPDATE] A more complete listing: http://www.medcompare.com/matrix/1475/Outpatient-Cardiac-Monitoring-(Telemetry)-Services.html http://www.medcompare.com/matrix/1475/Outpatient-Cardiac-Mon...

yellowbkpk 15y ago
This is very popular. There are several products that take ECG samples at timed intervals and submit them in several different ways. In addition these devices can be used to take 12-lead EKG measurements in transit (e.g. inside the ambulance) and send them to the doctor for analysis before the patient arrives. I'm very surprised that they're hosting these services on EC2. The company I worked for recently finished building a huge highly-redundant data center specifically designed for hosting this kind of medically-sensitive information. I'm willing to bet it was a bit more fortified than even Amazon's.
jmm 15y ago
My uncle owns a few elder care franchises and had me look into monitoring services as a funemployment project. So I've spent a bit more time with this than I'd like to admit... The company I was most impressed with was http://www.halomonitoring.com http://www.halomonitoring.com . The device seems to be designed well and the monitoring services (online portal for caregivers or family members) are impressive, comparatively. There's kind of a sea change happening with home health monitoring -- from the "I've fallen and can't get up" devices to more robust and interactive solutions. GE recently bought a company that provides institutional monitoring equipment and services. Lemme know if you want the full report :)

ceejayoz 15y ago

Later in the thread they say two instances are up but the third has most of the patients. If any of their patients die, they should be convicted of negligent manslaughter, not Amazon. Amazon explicitly warns that single instances can be lost. Any EC2 system needs redundancy, let alone a patient critical one!

logjam 15y ago

If this story is true (and I have some doubts), you'd think this would have been a no-brainer to not try to keep critical data like this in the cloud. But even for non-critical medical data, non-tech savvy medical folks (and this includes some physicians, although in general physicians are more tech-savvy than most managed care decision makers) are going to get into increasingly serious trouble if they don't think through these kinds of things very carefully. Driven by electronic health care mandates and incentives, companies are pushing portable devices for medical data gathering (portable as in tablet, ie without local data store), and some groups are going to be knee-jerk tempted into buying into this without thinking through the ramifications of not owning your own data -- and by that, I don't mean vague marketing-speak hand-waving that "you own your data" through the simple mechanism of password-protected access to your patient's data living on someone's server farm. By actually owning your patient's data, I mean strongly considering restricting your local practice to use of on-site media. If you don't, you'd better have complete confidence that putting your patient's data on the web reflects a reasonable standard of care/privacy, that retention policies don't bite you in six months when your cloud provider goes out of business (or changes hands or farms out data storage or backup to some "economical" third-party), that you're going to have access to your data when you need it, that your cloud provider is going to audit access and changes to your patient's data and report the results of that audit to you accurately and regularly, that legal requests for data are not acted upon without you or your patient's consent, that backups are subjected to the same levels of security, etc. etc. Electronic medical records are a good idea. It can be done right, as systems such as the Veteran's Administration's VISTA system have demonstrated. However, there is a fine line to walk here. Implementing EHR for the sake of government incentive, or for the convenience of insurance companies, or even the convenience of walking into an exam room carrying a light-weight tablet (which have their own disadvantages for data gathering) rather than a laptop with a hard-drive isn't going to cut it when you put your patients and your practice at risk of harm.

teilo 15y ago

Elastic Load Balancer, anyone? I have no sympathy for this company. They deserve to get sued for this. Yes, for something life threatening like this, EC2 is a bad idea, but they didn't even bother to take advantage of the geographic redundancy, much less something so basic as having backup AMIs ready in another AZ in your current region. Of course, a whole lot of companies are learning this now.

gregburek 15y ago
Unfortunately, this last AWS problem affected multiple AZs in the us-east region. The OP very well may have had an alternate AZ failover plan, but like Quora, Sencha, Reddit, FourSquare and Heroku, they probably kept it region specific. As for backing up to multiple regions, I can imagine them thinking that sending everything over the public internet as being a bad idea. However, not having a multi-region/multi-provider failover plan was a worse one.
kinofcain 15y ago
ELB doesn't work across regions, the failure at EC2 affected multiple Availability zones in the Virginia Region. ELB might have helped, but it's likely that some people had all their "availabiliy" zones knocked out. In fact, it's looking like all the automatic-failovers from people hosting on those data centers compounded the problem when everyone's scripts tried to recover at the same time – what was referred to in a previous article as a "bank run". But yeah, if you're doing heart monitoring, you need to have your servers replicated across a wide geographic area.

jamesbkel 15y ago

I wish they offered some more detail on how important/life-threatening this is. Are we talking monitoring as in "detect signs of impending/occurring cardiac event and notify emergency services"? Or is this monitoring in a longitudinal sense, assisting physicians to look at trends in their patients? If it is the later case, is data stored on the device until it's uploaded, or is it simply lost if the connection fails? Still, not really forgivable to be 100% AWS, but it's a very different story if this is primarily a recording device vs. a monitoring device.

MSDjunior 15y ago

who was the IT professional that decided to rely soley on the AWS cloud without any kind of "just in case" solution....with people's lives at stake? lol. I think I'm gonna subscribe to the hoax theory

tybris 15y ago

Those people should be prosecuted for utter incompetence.

forgotAgain 15y ago

This is too incredible to be true. But if it is true then I would expect the company in question to be out of business quite soon. A company that put's a life critical system on AWS shows a total disregard for their patients well being.

jauer 15y ago
I've seen things much worse than this with medical records and data. Up to a few months ago we had a ASP.NET shared hosting customer that was doing some kind of data relay web service for medical imaging. No encryption. Patient data in full view on the server. No redundancy. Apparently it was used for outsourcing imagery review or something. If it didn't work doctors would have to drive in from home which slowed down the diagnostic process. "Mission critical" on a $30 a month shared hosting plan. Very much not HIPPA compliant.

renegadedev 15y ago

I recently had the "good" fortune of being at the receiving end of a sales pitch from one cloud hosting service (not Amazon). The pitch centered more around flexibility (ability to add CPUs, RAM, storage, network nodes) rather than reliability issues as the one EC2 is facing right now. All questions about reliability were brushed aside with blanket statements like "it's in the cloud" as if that was an end-all-be-all. After much prodding the sales rep admitted that the hosting was all in one location (low lying hurricane prone area) and if we desired redundancy we would have to mirror our apps at a different geographical location. As with any other product, don't believe the sales pitch. If it sounds too good to be true, it probably is. Always use redundancy specially for critical apps.

_3u10 15y ago

This reminds me of doing tech support back during the dot bomb. Everyday we'd have day traders phone in complaining that their internet was down and they were losing thousands of dollars. Then I'd advise them to switch to their backup connection, then they'd say they didn't have one. Finally, I'd ask why if they were running a business that could generate / lose thousands of dollars in a few minutes why they had chosen a residential internet service instead of opting for the business package and why they did not have a backup connection as neither the business packages or residential packages came with a SLA that was suitable for the type of risk inherent in their business. If their data is so important why are they hosting it on the absolute cheapest and shittiest servers they can find without a backup solution in place? If they read the TOS for EC2 it probably says not to run critical infrastructure on it. Perhaps for sensitive data upon which people's lives depend they might want to look into something like:http://h20223.www2.hp.com/nonstopcomputing/cache/76385-0-0-0-121.html?jumpid=go/integritynonstop http://h20223.www2.hp.com/nonstopcomputing/cache/76385-0-0-0... Yes, it costs more than EC2, but there's a reason for that. Wait until this company finds out that if their values exceed what MySQL expects it will silently truncate it.

stick 15y ago

This is a combination of posters not understanding what the technology is used for and the tech guy exaggerating the urgency of the situation. Remote cardiac monitoring isn't for people who are imminently going to die of catastrophic heart attacks or who risk dropping into a fatal arrhythmia (wonky heart rhythm). In fact, there isn't even solid evidence that it saves lives. What it is used for is investigating patients who have vague symptoms which might be related to transient changes in their heart beat. These are people who come to their doctor at the age of 54 and say "I felt my heart beating really quickly and felt kind of faint" or "I felt kind of dizzy and then passed out. It's happened twice in the last month." The more traditional way to investigate these patients are with what's called a Holter monitor, as alluded to by a previous poster. These are little belt packs you carry around while wired-up and that record your ECG for 24-48 hours at a time. The main weakness? You said it: the device only has a 24-48 hour windows to capture the weird, often rare, rhythm. There are different ways remote cardiac monitoring systems report their results and I'm not sure which this particular company was using, but it doesn't really matter. Some of them only report weird stretches, others only report events when the patient says they're feeling symptom X, others are reporting continuously. Take-home message: this is not life-or-death data. When doctors (at least those who are allowed to keep practicing) think a patient needs critical cardiac monitoring, they admit them to hospital. This was a tech guy, looking to jump the queue by trying to raise a red flag because his servers were being used for--OMFG!--cardiac monitoring. A lot of my doctor buddies use similar strategies when they're caught speeding by the police. "I just got called in to the hospital!" …to see a patient with a really nasty rash.

mamp 15y ago
From the website I think this analysis is correct. Holters store the 24 hour data on the machine as well. The 12 leads are not continuous but set pieces which also store locally. If they were providing real-time monitoring with alerts then this would be more serious. This seems to be a backup system, ironically itself without backups.
GrandMasterBirt 15y ago
a) what stick said. Not such a critical system. b) Good to know from responses that amazon's paid support is crap. They'll officially tell you "were on it". c) People are bashing this person for relying on amazon not failing all at once. I would be surprised if they hosted on amazon relying on each individual node being up 99.98% of the time. That is crazy. You build on cloud computing knowing that each one node can fail at any moment, but you build it to fall over to another node. From experience, sometimes nodes just... die... or get really unstable. d) People bashing the dude vs offering future advice. I'm sure if his company is in deep shit, he's well aware of it.

FreshCode 15y ago

Bullshit, linkbait post. After reading the whole thing, I would gladly spend karma to downvote this.

lell 15y ago
Are you referring to the link in the title? It is true that the company was hosting cardiac patient monitoring on EC2. This was not life threatening infrastructure, but that is not implied by the HN title. How is it linkbait? The poster on Amazon's forum misrepresented the severity of the infrastructure and tries to backpedal later on in the thread. This is an example of someone using EC2 and fucking up, firstly for hosting production level infrastructure without backups, and secondly for misrepresenting the importance of the infrastructure in an attempt to get more technical support, making them look liable for some sort of medical malpractice. The poster probably panicked when they realised their first mistake, and thought they were showing that they were trying to get back in control. I don't think the poster was totally stupid, maybe she/he didn't understand the risks of EC2, or maybe it wasn't even their decision to have no backups --- in which case they should have pressed harder at meetings or refused to continue to work on the project. And then they panicked. We can all learn something from this.

yaix 15y ago

Just wow! How difficult can it be to have a backup dedi server with the DB mirrored running at some other provider to quickly switch to, just in case? Even many simple web sites have a config like that. Clouds have been around for decades, they are still new and may fail.

rbanffy 15y ago

So, they experienced a Schrödinger situation...

code_duck 15y ago

This is just supplementary monitoring, not a true life-or-death situation. Everyone (the company involved included) is welcome to stop blowing it out of proportion any time.