7 ms·

Fatal Dose – Radiation Deaths linked to AECL Computer Errors (1994)

120 points by agopinath 12y ago

kohanz 12y ago
The Therac-25 case study is a tragic one, but fortunately it is not forgotten. I work on medical devices (and have worked on radiotherapy devices previously) and the standards for quality systems and regulatory hurdles (which I occasionally see bemoaned here on HN) are there with good reason. In fact, Therac-25 is often cited when training new hires on quality (as required with any ISO-13485 compliant QMS).
- lostlogin 12y ago
  Diagnostic imaging guy here - we point our recruits to this, balding patients when doing diagnostic tests shouldn't happen. http://www.ajnr.org/content/31/1/2.full http://www.ajnr.org/content/31/1/2.full
- throwwit 12y ago
  I hope there are sensor mechanisms that confirm and/or failsafe the exposures nowadays.
- noir_lord 12y ago
  Definitely not forgotten, I wrote an essay on software safety when I was at college (not Uni, UK meaning) and the Therac 25 was a big chunk of it. The research I did has always stuck with me because of the suffering the patients where exposed to combined with a company attitude of "admit only what we are forced to so we don't hurt sales". Nothing I write is safety critical (though it was a field I was fascinated with when I was younger) though so I can sleep at night.
gnaffle 12y ago
Thanks, I hadn't read this before. For those that haven't read it, here's Levesons article on the Therac-25: http://sunnyday.mit.edu/papers/therac.pdf http://sunnyday.mit.edu/papers/therac.pdf
- thaumasiotes 12y ago
  This one made me think about public outrage against tobacco companies. One minor theme in this article is that AECL denied knowledge of any reports of Therac-25 malfunctions even when, looking at a timeline of publicly-known events, such ignorance might be described as "implausible". They don't seem to have been punished for this, and while I agree that it isn't laudable I also agree that it's not the greatest infraction. AECL really did care about the proper functioning of their machine. They really did look for problems. They cooperated with the FDA to a very great extent. It's hard to fault them for not thinking of testing "what if we enter incorrect configuration information, and then correct it within 8 seconds?" But tobacco companies are routinely vilified for sitting on cigarette mortality data, as if this was by itself enough to make them irredeemable. They didn't even get off with a light punishment, much less the zero punishment AECL received. I suspect the difference, in the minds of many, is that AECL was a benign company advancing a useful purpose, while tobacco companies sold a product whose only use was to kill the operator. But that was legal then and remains legal today -- how can it be the justification for punishing them extra-hard for otherwise minor problems? AECL's misrepresentedly-unsafe product didn't even kill the operator; it killed random sick people who trusted the hospitals.
  - gnaffle 12y ago
    Didn't the tobacco companies also spend money to discredit scientists and peer-reviewed articles and seed misinformation about the real risks of smoking, all while they were sitting on that mortality data? I think that was the real problem.
    - thaumasiotes 12y ago
      All of what you've said is also legal today.
jacobparker 12y ago
Related reading: http://www.amazon.ca/Set-Phasers-Stun-Design-Technology/dp/0963617885 http://www.amazon.ca/Set-Phasers-Stun-Design-Technology/dp/0...
lostlogin 12y ago
>>As a result of the Therac-25 accidents, the FDA now requires documentation on software for new medical and other products: a paper trail, in other words, that can be examined by an independent body and retraced for flaws.<< Anyone have any idea if this can be looked at by the end user? I'm not a radiation technologist of the flavour mentioned in the article, I'm on the diagnostic side. I use an MR scanner with numerous software bugs that I have reported but which remain. Similarly, the scanner can be made to display data which it says it is going to use in the next scan, but which it isn't. I suspected a bug and found the way to reproduce it. My last email listed 24 similar bugs (I've found more since) but other than a "thanks, we will forward this on" there has been no reply or comment. It is hard to imagine when this could be a safety issue, but it is a waste of valuable time, it is a waste of money and it's frustrating when I have gone to the trouble of working out the exact way of creating the issues. If anyone is interested, the interface is so god awful that instead of having an on off button or switch interface, the scanner gets the user to type 1 or 0 for on and off into a text field. Some fields take other values like 1, 2 and 3. Some take decimal values like 0 to 1 in 0.1 increments. There is no pattern to what the user is expected to type. Yuck. This data is not properly sanitized either, and you can make the scanner say its "doing" something it's not. Type in 1.999, and error message appears, the field corrects to 2.0 but the scanner does the thing that a setting of 1 would produce. These sorts of bugs occur all over the place. Edit: The "thanks" email is the most positive I've ever got, my previous reports were me with statements like "we have some very experienced users who haven't had this issue" when there were clear safety problems with earlier scanner implementations (The scanner was producing axial slices at a location different to where I asked for them to be, on a spine patient due in theatre - good luck operating on the correct vertebral level). Its FDA approved and its on the latest software release. I have undergone manufacturer training and have had additional training half a dozen times at my request and at the manufacturers request after my bug reports were met with "you're doing it wrong". I'm not, the software is buggy and I have some excellent and amazing screen shots and camera phone video of the bugs in action.
- vardump 12y ago
  Like others have told, any complaint you make must be processed by the company. If it endangers human safety, failure to do so is against FDA's rules and can have dire and expensive consequences to the company in question. Fixing bugs in a medical device is not slow because it'd take a long time to fix the bug itself. It's slow because there's a lot of paperwork (Device History Files, specifications, tons of reviews, etc.) and system testing that follows. That's why you'd only correct issues that do not endanger patient safety once there are enough or when they can be combined with new features in firmware. There's also always a risk that those fixed issues cause patient endangering bugs. It can take almost one year from implementing the fix until it is running on any significant number of medical (imaging) devices in question. FDA's rules are there for a reason. Although... someone should tell FDA git exists. It's silly to revision things manually in Word documents.
icco 12y ago
One of the more infamous classes in Computer Science at Cal Poly SLO is "Professional Responsibilities", is taught by Dr. Clark Turner. The class delves into Therac-25, and similar cases that have happened since. I found the class really interesting because it does make you question and think about the ethics of what you are building and what others have built. Knowing about, and thinking about, the ACM Code of Ethics, Stuxnet, Therac-25, the Windows Security Patch Policy, and other problems our programming culture have come across is important. Realizing that the code we write can affect people in both positive and negative ways on a long and short term scale is something that can change both your product and how you build a product.
davidrusu 12y ago
I had a professor read this case study in a lecture. It amazes me that merely one programmer was trusted with building the software for a radiation beam canon.
- snowwindwaves 12y ago
  I think a manager for a product would give as much work and responsibility to one person as possible if they say they can do it, and sometimes even when they say they don't know if they can do it but they'll try. An experienced manager might know how realistic the workload is and downsides of having only one person on task x but every manager sees the upside, fewer people = less cost.
- wglb 12y ago
  It makes me uncomfortable as well. But keep in mind that at that time, it may have well been the only programmer on the project, and that the manager was not likely qualified to read code.
noddingham 12y ago
We covered this in my CS courses as well. I feel bad if anyone comes out of a CS program and isn't exposed to the Therac-25 incident even if superficially.
mariodiana 12y ago
There was an article that appeared in the NY Times, a few years ago, that discusses the malfunctions of radiology equipment. There was one story, in particular, that stood out for me. It describes a, reportedly not unusual, malfunction/crash of a linear accelerator used for Intensity Modulated Radiation Therapy (IMRT): "An error message asked [the medical physicist operating the device] if she wanted to save her changes before the program aborted. She answered yes." How many programmers read that and cringe? I know I did. My guess is that the operating system being used for the device is some standard OS (Windows CE, maybe?) that is being repurposed to run the application and provide the GUI for the device. It's not that this is necessarily bad, but I would think the most important thing to do would be to strip the OS (or UI) of the various "user conveniences" that in a life or death situation could have all kinds of unintended consequences. If a person is coding or doing graphic design -- or typing up cooking recipes -- and a crash happens, it's a good thing to have the opportunity to save your work. If 1 teaspoon of butter gets changed to 1 tablespoon because of some kind of data corruption, big deal. So your cookies come out terrible! It's quite a different matter if the application is coordinating 120 moving parts to direct a radiation beam onto a human body. The article is here: http://www.nytimes.com/2010/01/24/health/24radiation.html?pagewanted=all&_r=0 http://www.nytimes.com/2010/01/24/health/24radiation.html?pa...
merraksh 12y ago
The Therac-25's software program, relatively crude by today's standards, probably contained 101000 lines of code. At one error for every 500 lines, that works out to the possibility of twenty errors. I'd say 200, not twenty.
- pathdependent 12y ago
  I think the article was OCR'd. There were a few other mistakes that were clearly misinterpreted characters. I think the 1 in the thousands place is actually a comma on the source.
  - merraksh 12y ago
    Good point. There are many hints that article was OCR'd, for instance an ".4ECL" instead of "AECL".
spectre256 12y ago
"A professor in computer engineering at the University of Toronto told me that, as a matter of course, his undergraduate students are warned about the risks of incrementing numbers in a computer program." As someone with a computer science degree who was warned of such risks and studied the Therac-25 in my classes, this sentence made me realize how far we have to go as professionals. Something seemingly so simple as incrementing a number, one of the most common things done in a program, can cause serious problems (of course we have more help with this now than in the mid 80's). Other people must read things like that and cement any distrust they have in computers and computer programmers. And they're probably right to.
blabby 12y ago
At the time of Therac-25, FDA was only budgeted to investigate 6 percent of device applications. Currently, the same mistakes made in the eighties with Therac-25 are being made in many radiation therapy devices. The two NY Times articles (Pulitzer Prize winning) in 2010 and 2011 describe some of the newer cases. What's shocking to me is that the incidents are always reported in isolation. People become briefly outraged, then the furor dies down until the next death. Many of the comments in this thread suggest that people can't or won't face the fact that this is a current, ongoing problem of great complexity. A couple of comments mentioned the coverage of Therac-25 in schools. Very little of what is taught in schools makes it into the programming of radiation therapy devices. History has shown that schooling is not a sufficient solution. Other comments claim (erroneously) that the FDA is attending to the problem. The FDA has been carefully defanged by the medical device lobby. The FDA has gotten smarter, but has nowhere near the funding to keep pace with its charge and never will. I wish I could say that I see some hope but I don't see it.
- lstamour 12y ago
  Why hasn't the hardware failsafe for overdoses become mandatory? Why don't we apply defense-in-depth to all worst-case scenarios involving deadly things? Of course, sometimes hospitals aren't logical, air circulation between rooms comes to mind. And here, I'm sure everyone just trusts the machines because they paid a lot of money for them and it's always worked in the past ...
  - kalleboo 12y ago
    > Why hasn't the hardware failsafe for overdoses become mandatory? Why don't we apply defense-in-depth to all worst-case scenarios involving deadly things? Because money
JoshTheGeek 12y ago
101000 lines, 500 lines per error gives about 200 errors, not 20
mbzastava 12y ago
I remember this case study being taught to us at Engineering Ethics 101 while I was in school for my BSEE. I clearly recall the question posed to the room of about 50 would-be electrical engineers. In the case study it was defined that the engineer working on the software was different than the engineer working on the hardware. The class was to take the hardware engineer's perspective for the exercise. The question was "WHO IS RESPONSIBLE?" Never mind the gravity of such a question being posed to a 18 year-old fresh out of high school. What shocked me was the professor's answer "THE SOFTWARE ENGINEER!" I'm no programmer. I nearly failed to achieve my BSEE due to the required programming courses. But never in my life would I imagine to rationalize this type of blame onto someone else, let alone have it taught to me in such a point-blank manner. I raised my hand and offered my objection. "HOW CAN YOU BLAME THE SOFTWARE ENGINEER, WHEN YOU DESIGNED THE MACHINE THAT WAS CAPABLE OF GENERATING ENOUGH RADIATION TO KILL A PERSON?" I further objected "YOU ALSO LEFT IT WITHOUT ANY HARDWARE-BASED FAILSAFE MEASURES, RELYING SOLELY ON THE SOFTWARE PERFORMANCE." Dead silence from the class as the professor responded in a quip "NO. IT IS THE SOFTWARE ENGINEER'S FAULT. MOVING ON..." EDIT: We had been given a photo-copy of an article on the accidents for the exercise. No where did I recall reading they had implemented a solution (this was in 2001) that was primarily hardware based. I would have loved to shove that in the professor's face.

120 points by agopinath 12y ago

The Therac-25 case study is a tragic one, but fortunately it is not forgotten. I work on medical devices (and have worked on radiotherapy devices previously) and the standards for quality systems and regulatory hurdles (which I occasionally see bemoaned here on HN) are there with good reason. In fact, Therac-25 is often cited when training new hires on quality (as required with any ISO-13485 compliant QMS).

lostlogin 12y ago
Diagnostic imaging guy here - we point our recruits to this, balding patients when doing diagnostic tests shouldn't happen. http://www.ajnr.org/content/31/1/2.full http://www.ajnr.org/content/31/1/2.full
throwwit 12y ago
I hope there are sensor mechanisms that confirm and/or failsafe the exposures nowadays.
noir_lord 12y ago
Definitely not forgotten, I wrote an essay on software safety when I was at college (not Uni, UK meaning) and the Therac 25 was a big chunk of it. The research I did has always stuck with me because of the suffering the patients where exposed to combined with a company attitude of "admit only what we are forced to so we don't hurt sales". Nothing I write is safety critical (though it was a field I was fascinated with when I was younger) though so I can sleep at night.

gnaffle 12y ago

Thanks, I hadn't read this before. For those that haven't read it, here's Levesons article on the Therac-25: http://sunnyday.mit.edu/papers/therac.pdf http://sunnyday.mit.edu/papers/therac.pdf

thaumasiotes 12y ago
This one made me think about public outrage against tobacco companies. One minor theme in this article is that AECL denied knowledge of any reports of Therac-25 malfunctions even when, looking at a timeline of publicly-known events, such ignorance might be described as "implausible". They don't seem to have been punished for this, and while I agree that it isn't laudable I also agree that it's not the greatest infraction. AECL really did care about the proper functioning of their machine. They really did look for problems. They cooperated with the FDA to a very great extent. It's hard to fault them for not thinking of testing "what if we enter incorrect configuration information, and then correct it within 8 seconds?" But tobacco companies are routinely vilified for sitting on cigarette mortality data, as if this was by itself enough to make them irredeemable. They didn't even get off with a light punishment, much less the zero punishment AECL received. I suspect the difference, in the minds of many, is that AECL was a benign company advancing a useful purpose, while tobacco companies sold a product whose only use was to kill the operator. But that was legal then and remains legal today -- how can it be the justification for punishing them extra-hard for otherwise minor problems? AECL's misrepresentedly-unsafe product didn't even kill the operator; it killed random sick people who trusted the hospitals.
- gnaffle 12y ago
  Didn't the tobacco companies also spend money to discredit scientists and peer-reviewed articles and seed misinformation about the real risks of smoking, all while they were sitting on that mortality data? I think that was the real problem.
  - thaumasiotes 12y ago
    All of what you've said is also legal today.

jacobparker 12y ago

Related reading: http://www.amazon.ca/Set-Phasers-Stun-Design-Technology/dp/0963617885 http://www.amazon.ca/Set-Phasers-Stun-Design-Technology/dp/0...

lostlogin 12y ago

>>As a result of the Therac-25 accidents, the FDA now requires documentation on software for new medical and other products: a paper trail, in other words, that can be examined by an independent body and retraced for flaws.<< Anyone have any idea if this can be looked at by the end user? I'm not a radiation technologist of the flavour mentioned in the article, I'm on the diagnostic side. I use an MR scanner with numerous software bugs that I have reported but which remain. Similarly, the scanner can be made to display data which it says it is going to use in the next scan, but which it isn't. I suspected a bug and found the way to reproduce it. My last email listed 24 similar bugs (I've found more since) but other than a "thanks, we will forward this on" there has been no reply or comment. It is hard to imagine when this could be a safety issue, but it is a waste of valuable time, it is a waste of money and it's frustrating when I have gone to the trouble of working out the exact way of creating the issues. If anyone is interested, the interface is so god awful that instead of having an on off button or switch interface, the scanner gets the user to type 1 or 0 for on and off into a text field. Some fields take other values like 1, 2 and 3. Some take decimal values like 0 to 1 in 0.1 increments. There is no pattern to what the user is expected to type. Yuck. This data is not properly sanitized either, and you can make the scanner say its "doing" something it's not. Type in 1.999, and error message appears, the field corrects to 2.0 but the scanner does the thing that a setting of 1 would produce. These sorts of bugs occur all over the place. Edit: The "thanks" email is the most positive I've ever got, my previous reports were me with statements like "we have some very experienced users who haven't had this issue" when there were clear safety problems with earlier scanner implementations (The scanner was producing axial slices at a location different to where I asked for them to be, on a spine patient due in theatre - good luck operating on the correct vertebral level). Its FDA approved and its on the latest software release. I have undergone manufacturer training and have had additional training half a dozen times at my request and at the manufacturers request after my bug reports were met with "you're doing it wrong". I'm not, the software is buggy and I have some excellent and amazing screen shots and camera phone video of the bugs in action.

vardump 12y ago
Like others have told, any complaint you make must be processed by the company. If it endangers human safety, failure to do so is against FDA's rules and can have dire and expensive consequences to the company in question. Fixing bugs in a medical device is not slow because it'd take a long time to fix the bug itself. It's slow because there's a lot of paperwork (Device History Files, specifications, tons of reviews, etc.) and system testing that follows. That's why you'd only correct issues that do not endanger patient safety once there are enough or when they can be combined with new features in firmware. There's also always a risk that those fixed issues cause patient endangering bugs. It can take almost one year from implementing the fix until it is running on any significant number of medical (imaging) devices in question. FDA's rules are there for a reason. Although... someone should tell FDA git exists. It's silly to revision things manually in Word documents.

icco 12y ago

One of the more infamous classes in Computer Science at Cal Poly SLO is "Professional Responsibilities", is taught by Dr. Clark Turner. The class delves into Therac-25, and similar cases that have happened since. I found the class really interesting because it does make you question and think about the ethics of what you are building and what others have built. Knowing about, and thinking about, the ACM Code of Ethics, Stuxnet, Therac-25, the Windows Security Patch Policy, and other problems our programming culture have come across is important. Realizing that the code we write can affect people in both positive and negative ways on a long and short term scale is something that can change both your product and how you build a product.

davidrusu 12y ago

I had a professor read this case study in a lecture. It amazes me that merely one programmer was trusted with building the software for a radiation beam canon.

snowwindwaves 12y ago
I think a manager for a product would give as much work and responsibility to one person as possible if they say they can do it, and sometimes even when they say they don't know if they can do it but they'll try. An experienced manager might know how realistic the workload is and downsides of having only one person on task x but every manager sees the upside, fewer people = less cost.
wglb 12y ago
It makes me uncomfortable as well. But keep in mind that at that time, it may have well been the only programmer on the project, and that the manager was not likely qualified to read code.

noddingham 12y ago

We covered this in my CS courses as well. I feel bad if anyone comes out of a CS program and isn't exposed to the Therac-25 incident even if superficially.

mariodiana 12y ago

There was an article that appeared in the NY Times, a few years ago, that discusses the malfunctions of radiology equipment. There was one story, in particular, that stood out for me. It describes a, reportedly not unusual, malfunction/crash of a linear accelerator used for Intensity Modulated Radiation Therapy (IMRT): "An error message asked [the medical physicist operating the device] if she wanted to save her changes before the program aborted. She answered yes." How many programmers read that and cringe? I know I did. My guess is that the operating system being used for the device is some standard OS (Windows CE, maybe?) that is being repurposed to run the application and provide the GUI for the device. It's not that this is necessarily bad, but I would think the most important thing to do would be to strip the OS (or UI) of the various "user conveniences" that in a life or death situation could have all kinds of unintended consequences. If a person is coding or doing graphic design -- or typing up cooking recipes -- and a crash happens, it's a good thing to have the opportunity to save your work. If 1 teaspoon of butter gets changed to 1 tablespoon because of some kind of data corruption, big deal. So your cookies come out terrible! It's quite a different matter if the application is coordinating 120 moving parts to direct a radiation beam onto a human body. The article is here: http://www.nytimes.com/2010/01/24/health/24radiation.html?pagewanted=all&_r=0 http://www.nytimes.com/2010/01/24/health/24radiation.html?pa...

merraksh 12y ago

The Therac-25's software program, relatively crude by today's standards, probably contained 101000 lines of code. At one error for every 500 lines, that works out to the possibility of twenty errors. I'd say 200, not twenty.

pathdependent 12y ago
I think the article was OCR'd. There were a few other mistakes that were clearly misinterpreted characters. I think the 1 in the thousands place is actually a comma on the source.
- merraksh 12y ago
  Good point. There are many hints that article was OCR'd, for instance an ".4ECL" instead of "AECL".

spectre256 12y ago

"A professor in computer engineering at the University of Toronto told me that, as a matter of course, his undergraduate students are warned about the risks of incrementing numbers in a computer program." As someone with a computer science degree who was warned of such risks and studied the Therac-25 in my classes, this sentence made me realize how far we have to go as professionals. Something seemingly so simple as incrementing a number, one of the most common things done in a program, can cause serious problems (of course we have more help with this now than in the mid 80's). Other people must read things like that and cement any distrust they have in computers and computer programmers. And they're probably right to.

blabby 12y ago

At the time of Therac-25, FDA was only budgeted to investigate 6 percent of device applications. Currently, the same mistakes made in the eighties with Therac-25 are being made in many radiation therapy devices. The two NY Times articles (Pulitzer Prize winning) in 2010 and 2011 describe some of the newer cases. What's shocking to me is that the incidents are always reported in isolation. People become briefly outraged, then the furor dies down until the next death. Many of the comments in this thread suggest that people can't or won't face the fact that this is a current, ongoing problem of great complexity. A couple of comments mentioned the coverage of Therac-25 in schools. Very little of what is taught in schools makes it into the programming of radiation therapy devices. History has shown that schooling is not a sufficient solution. Other comments claim (erroneously) that the FDA is attending to the problem. The FDA has been carefully defanged by the medical device lobby. The FDA has gotten smarter, but has nowhere near the funding to keep pace with its charge and never will. I wish I could say that I see some hope but I don't see it.

lstamour 12y ago
Why hasn't the hardware failsafe for overdoses become mandatory? Why don't we apply defense-in-depth to all worst-case scenarios involving deadly things? Of course, sometimes hospitals aren't logical, air circulation between rooms comes to mind. And here, I'm sure everyone just trusts the machines because they paid a lot of money for them and it's always worked in the past ...
- kalleboo 12y ago
  > Why hasn't the hardware failsafe for overdoses become mandatory? Why don't we apply defense-in-depth to all worst-case scenarios involving deadly things? Because money

JoshTheGeek 12y ago

101000 lines, 500 lines per error gives about 200 errors, not 20

mbzastava 12y ago

I remember this case study being taught to us at Engineering Ethics 101 while I was in school for my BSEE. I clearly recall the question posed to the room of about 50 would-be electrical engineers. In the case study it was defined that the engineer working on the software was different than the engineer working on the hardware. The class was to take the hardware engineer's perspective for the exercise. The question was "WHO IS RESPONSIBLE?" Never mind the gravity of such a question being posed to a 18 year-old fresh out of high school. What shocked me was the professor's answer "THE SOFTWARE ENGINEER!" I'm no programmer. I nearly failed to achieve my BSEE due to the required programming courses. But never in my life would I imagine to rationalize this type of blame onto someone else, let alone have it taught to me in such a point-blank manner. I raised my hand and offered my objection. "HOW CAN YOU BLAME THE SOFTWARE ENGINEER, WHEN YOU DESIGNED THE MACHINE THAT WAS CAPABLE OF GENERATING ENOUGH RADIATION TO KILL A PERSON?" I further objected "YOU ALSO LEFT IT WITHOUT ANY HARDWARE-BASED FAILSAFE MEASURES, RELYING SOLELY ON THE SOFTWARE PERFORMANCE." Dead silence from the class as the professor responded in a quip "NO. IT IS THE SOFTWARE ENGINEER'S FAULT. MOVING ON..." EDIT: We had been given a photo-copy of an article on the accidents for the exercise. No where did I recall reading they had implemented a solution (this was in 2001) that was primarily hardware based. I would have loved to shove that in the professor's face.