10 ms·

Failed intercept at Dhahran caused by a software error in handling of timestamps

205 points by sine 9y ago

OedipusRex 9y ago
That was a temporary fix, then a software patch was released. I also wouldn't call that a "software" fix.
avar 9y ago
Even better, the timeline: - February 11th: Vendor informed of the issue - February 25th: 28 people die because of the issue - February 26th: The vendor ships a fix I'd have loved to be a fly on the wall for that phonecall on the 25th (or early on the 26th).
- nathan_long 9y ago
  Oh, gosh. Really? It makes me kind of sick imagining that call.
  - sli 9y ago
    You're not even curious to see how it was dealt with and how the issue was expressed to the vendor? I'd never be in a meeting regarding deaths of users of my software, because I just make internal webapps, so I just cannot help but be curious as to how one of those meetings would go.
    - deleted 9y ago
      [deleted]
    - brookside 9y ago
      Sure but > I'd have loved to be a fly ... loving anything about that sad scenario seems impossible.
      - kaishiro 9y ago
        It's a figure of speech, not a show of approval.
      - Nomentatus 9y ago
        Even loving learning enough to avoid the next one? 'Cause that's what responder wants, to learn. "What the hell were they thinking?" is often the most pertinent knowledge of all.
    - hx2a 9y ago
      > I'd never be in a meeting regarding deaths of users of my software I know what that's like. About 20 years ago I was at a consulting firm supporting an electric and gas utility company. Among other things they had to do something called "markouts" which means they paint the ground at a location in a way that indicates exactly what infrastructure they have in the ground and precisely where it is. Markouts are a government organized thing. Before digging somewhere you can call a number and anybody that might possibly have infrastructure in the ground anywhere near your dig site is required to paint their markouts within a short time period. There are stiff fines if you "miss a markout." Anyhow there was a data problem with a markout. The field worker was sent to paint a markout at the corner of two streets that actually ran parallel to each other and didn't meet. Instead of calling it in and questioning the task he did nothing. Shortly after a construction worker put a backhoe through an electrical conduit with 15K volts. There was an explosion that was heard for many miles. The worker died the next day. He died painfully. > so I just cannot help but be curious as to how one of those meetings would go. Finger pointing, of course. Data was being fed back and forth between systems and eventually somebody else took the blame. The field worker who ignored the markout also was blamed. We did add something to our system so that that kind of data error would raise an exception. I learned a lot about care and diligence about data from this experience. Data errors are no joke.
    - muthdra 9y ago
      I'm not, because it would bore me. I see shit like that for breakfast when studying transportation. But if you don't do it, I say it's because of your own primal instincts, so you stuff them down...
- phonon 9y ago
  You missed this date-- Feb 21--notice goes out to users to avoid "very long run times". Users do not know what that means, and ignore warning. https://www.gao.gov/assets/220/215614.pdf https://www.gao.gov/assets/220/215614.pdf (page 9) "On February 21, 1991, the Patriot Project Office sent a message to Patriot users stating that very long run times could cause a shift in the range gate, resulting in the target being offset. The message also said a software change was being sent that would improve the system’s targeting. However, the message did not specify what constitutes very long run times. According to Army officials, they presumed that the users would not continuously run the batteries for such extended periods of time that the Patriot would fail to track targets. Therefore, they did not think that more detailed guidance was required."
  - macintux 9y ago
    That's terrible. Competent technical writing is criminally undervalued.
    - zeeZ 9y ago
      But there's also "presumed" and "did not think" in there. When there's a problem with your killing device you probably shouldn't use it until you've clarified what the problem is and don't just assume your end users will use it correctly. That's like saying "It's fine, the critical vulnerability patch will be applied on reboot", while in reality all your users just suspend to disk and move that annoying reboot nag window behind the task bar where it's out of sight.
      - mikeash 9y ago
        “You probably shouldn’t use it until you’ve clarified...” doesn’t work so well for a defensive system.
        lostlogin 9y ago
        How long was the off/on cycle? If it was short it would be reasonable to do it periodically. I don't think one can pin the blame on the vendor alone.
- kevinconaway 9y ago
  Per the GAO report[0] > According to Army officials, the delay in distributing the software from the United States to all Patriot locations was due to the time it took to arrange for air and ground transportation in a wartime environment. I'm not knowledgeable at all on how software for missile batteries was distributed in 1991 from the US to the Persian Gulf but 11 days doesn't seem unreasonable to me. [0] https://www.gao.gov/assets/220/215614.pdf https://www.gao.gov/assets/220/215614.pdf
- godelmachine 9y ago
  Who was the vendor? Aren't defense contractors required to be on their toes all the time? EDIT → Found it. The PATRIOT Project Office.
sharemywin 9y ago
I remember hearing about this in my numerical analysis class. 1. I remember hearing the system was only designed for XX operational hours but was being run over the operational spec. 2. The time was stored in base 10 so the calculation errors added up over time or something like that so if they had used some base 2 timing scheme it would haven't have had issues with rounding errors. My class was in the mid nineties so the details of my 25 year old memory is pretty hazy...at best.
- clw8 9y ago
  My recollection matches with yours, except I learned about it in the first week of Embedded Systems 101. If it isn't a standard part of the curriculum at every college embedded systems class, it should be! It really drove home the point that bad code can kill.
  - pilom 9y ago
    I learned about it in a Decision Analysis course and had a completely different point driven home. This wasn't bad code. It was code that was correctly written to a very well defined requirement ("System shall be operational for at most X hours before a reboot"). The code was written to a spec that was approved by the customer (the military). Unfortunately though, that requirement wasn't communicated to the end users.
    - michaelmrose 9y ago
      From the article "However, the timestamps of the two radar pulses being compared were converted to floating point differently: one correctly, the other introducing an error proportionate to the operation time so far" The code had a defect that effects its aim from turning it on but because it took 100 hours to drift by 1/3 of a second the problem wasn't apparent when rebooted regularly. If software can't continue to do basic math without manual intervention its defective. In fact everyone including the company that made it admits it's defective. Its possible your teacher picked a great example to illustrate a communication failure.
jasonmaydie 9y ago
The scud missile lead to their deaths, not the software. There's no absolute guarantee it would have intercepted it, plus rebooting a deployed machine regularly is an acceptable fix when it's live in the field
- rosser 9y ago
  That's a reductio fallacy. If you want to play that game, it was being deployed to that specific place that caused their deaths. Or was it enlisting in the first place? Maybe merely having been born? This is a strictly technical examination of the proximate cause of their deaths; it makes no claims about their ultimate cause. Whether or not a missile system with an accurate clock might have hit the target, it is unambiguous that this one missed specifically because of clock drift.
  - jasonmaydie 9y ago
    How so? The implication you and the article are asserting is that the clock error caused their deaths.. rather than the more accurate description "could have prevented death".
    - euyyn 9y ago
      It would depend on whether they were relying on it to work or not.
    - Vivtek 9y ago
      Well, it wasn't the missile that caused their deaths. Strictly speaking, it was the explosion of the missile. Well, wait. It wasn't the explosion - technically, it was the impact of the pressure wave on their bodies that caused ... well, no. Really, it was the fact that their organs stopped working after impact of the ... well. If you really want to be accurate, it was the fact that metabolism ceased to be practicable after their organs stopped working. Well, no, actually, the fact that their mental processes depended on their metabolism - that was really the cause of their... Well, no...
    - KindOne 9y ago
nathan_long 9y ago
> The Patriot missile battery at Dhahran had been in operation for 100 hours, by which time the system's internal clock had drifted by one-third of a second. Due to the missile's speed this was equivalent to a miss distance of 600 meters.
bertjk 9y ago
I've often wondered, considering the supposed low accuracy of Scud missiles, (wiki gives it a CEP of 450m) how much of the casualties from that incident were more due to the bad luck of the missile actually hitting its target.
- nerpderp83 9y ago
  If the Scud had been brought down earlier in it's trajectory it would have not been near people regardless of any randomness in it's landing.
- dnautics 9y ago
  many of the scuds "broke up in flight" or otherwise malfunctioned, too, so the actual effectiveness of patriot has been called into question.
brohoolio 9y ago
This is depressing. One of my middle school classmates had a brother killed in a SCUD strike.
tntn 9y ago
Despite other comments below, I think that the equivalence drawn between "failed to save" and "killed" reflects an interesting philosophical choice. I don't think that this equivalence is universally accepted, even by those who call thinking otherwise fallacious. If an EMT fails to save a victim of a car crash, did he/she kill the victim? If the dispatcher misspoke and gave the wrong cross street, delaying aid, did the dispatcher kill them?
- rxhernandez 9y ago
  In the medical device industry the company who made the device can be found at fault if a clinician makes a poor decision that leads to death based on a fault in the device. If the soldiers would have sought better cover or be otherwise saved in the case that there was no missile defense system was there then yes, some, if not most, of the blame lies on the software error.
mlazos 9y ago
The title of this post is misleading, they eventually supplied a software patch that fixed the clock drift. The Israelis proposed rebooting as a stopgap until the bug could be fixed.
- sctb 9y ago
  We've updated the submitted title from “Clock error lead to death of 28 Soldiers. Software fix: Reboot system regularly” to a representative phrase (edited for length) from the article. Submitters: please follow the guidelines by not editorializing titles. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
macawfish 9y ago
Little things do add up.
jimjimjim 9y ago
regarding the comments about bug killed people versus weapon killed people. There is no 1 answer, this argument is a result of black-white/yes-no/us-them single point of blame thinking. and it's terrible. the bug contributed to the loss of life.
deleted 9y ago
[deleted]
dredmorbius 9y ago
The inimitable comp.risks discussed this in 1992: http://catless.ncl.ac.uk/Risks/13/35#subj1.1 http://catless.ncl.ac.uk/Risks/13/35#subj1.1 http://catless.ncl.ac.uk/Risks/13/76#subj8.1 http://catless.ncl.ac.uk/Risks/13/76#subj8.1 And in 1997: http://catless.ncl.ac.uk/Risks/18/79#subj9.1 http://catless.ncl.ac.uk/Risks/18/79#subj9.1
otoburb 9y ago
This was a tragic and preventable loss. It's incredible that a software bug might have been the root cause. At the time, this incident really stuck out because it broke the illusion of our fabled Patriot missile shield protecting us. Civilian expats really believed the inflated Patriot interception rates parroted to us by mainstream media and our American military expat buddies. A large number of remaining expats who had stuck out the Gulf War to that point decided to pack it in and leave when word got out that the Dhahran barracks were hit. Although history shows that Iraq surrendered days after this incident, at the time there was heightened fear and confusion amongst the remaining expats, especially the non-Americans. We left on the last Lufthansa flight (crewed by military personnel) after hearing about this. Nostalgic edit: During the Gulf War embassies issued equipment and rations to expat citizens who chose to stay behind. Americans were issued full body suits (for adults and youths) due to the biological and chemical weapon payloads that Saddam boasted his SCUDs were carrying, along with MREs that tasted fabulous! In stark contrast, Commonwealth citizens were issued a bare gas mask (adult size only) and mono-flavour MREs that tasted like cardboard. The British embassy sticks out in my mind: with stern stone-faced expressions they admonished us all for not evacuating and thus endangering children in a war zone. In addition to the terrible rations and gas masks, they wordlessly gave us a stack of translucent stickers. When asked what they were for, embassy staff explained that in the event of the air siren going off, we should get under our sturdiest tables and don our gas masks (standard procedure), and then slap the stickers on. If the stickers changed colour, it meant we were in the presence of a biochemical agent and would have approximately 10 seconds before we died a horrific death. You kind of had to be there to appreciate the grim humour.
- celticninja 9y ago
  I mean I kind of understand the attitude of the British Embassy, it wasnt like trouble flared up overnight, the option to leave was there for a long time prior to the war beginning. Obviously it isnt the fault of the children who were kept there by their parents, but some responsibility needs to be borne by the expats that decided they were getting paid well enough to stay.
  - otoburb 9y ago
criley2 9y ago
This is bad, editorialized title that is not the title of the article. Mods should change this. The "software fix" was a software patch which corrected the clocking bug. The "software workaround" to use pre-fix was reboot. I hate editorialized, lying titles :(
- codazoda 9y ago
  Came here to mention that. The title needs a re-write but the story is interesting still.
leggomylibro 9y ago
I could be reading this wrong, but 1/3 of a second within 100 hours seems really good, like something you'd get from a temperature-controlled crystal oven. I don't mean to second-guess them in an area I know so little about, but if that was enough to cause a serious issue in the span of only a few days, shouldn't the devices be designed with a separate synchronization system, at least as a backup? Maybe GPS? Which brings up a sort of interesting question...would a Patriot missile system even have receivers for a weak public signal like GPS, or is it all self-contained?
- ajross 9y ago
  MIL-SPEC was indeed famous for overspecified components. So it's not terribly shocking that the oscillator on that board would operate really well as an isolated system. You probably don't need temperature control per se, a temperature compensation circuit could probably do that.
- GCU-Empiricist 9y ago
  As a former submariner who has had used clock for inertial navigation or for similar weapons systems 1/3 of a second over 100 hours is terrible.
  - leggomylibro 9y ago
    I mean, it's not an atomic clock, but I'm comparing it to the 32.768KHz RTC crystals I use with consumer microchips. If super-precise isolated accuracy were actually important, I assume they would use a rubidium or cesium oscillator.
    - grkvlt 9y ago
      1/3 of a second in 100 hours is basically 1ppm, or TXCO levels of accuracy, so pretty good i'd have thought, even for a submarine INS?
      - cocoablazing 9y ago
logfromblammo 9y ago
For doing a ballistic propagation, you apply a gravitational map in Earth-centered, Earth-fixed (ECEF) geodetic coordinates, then convert to Earth-centered rotating (ECR) geodetic coordinates, because that way you don't have to correct for the Coriolis effect. That ECEF-ECR conversion requires a time-of-day parameter. You can use a gravitational map that only accounts for latitude, but it isn't as precise. So using an accurate clock is really important if your intent is to hit a missile with a missile.
sjburt 9y ago
This is a completely misleading headline. The Patriot missile was not effective at destroying the Scud [0]. The DoD initially claimed successful intercepts when the missile detonated near the Scud, but it rarely, if ever, actually destroyed the warhead. The only reason there was an illusion of success was that the Scud was also spectacularly unreliable and often broke up on re-entry or failed to detonate. It is a complete falsehood to claim that the Patriot would have prevented this loss of life. [0] http://www.slate.com/articles/news_and_politics/war_stories/2003/03/patriot_games.html http://www.slate.com/articles/news_and_politics/war_stories/...
- deleted 9y ago
  [deleted]
tofof 9y ago
This particular bug is often taught in university compsci classes as "bug that killed people" is a good attention grabber -- the CS/EE analysis is sound; its truthfulness is only suspect because of the DoD's claimed successes. A more truthful "computer bugs that killed people" example would be the Therac-25 - a machine intended to treat cancer with tightly-focused radiation therapy. Six patients died as a result of massive overdoses of radiation, on the order of 20,000 rads. It was possible for the machine to end up in a state where it delivered full-power radiation without a hardware shield in place to protect the rest of the patient's body. No hardware interlocks were used to ensure that the full power mode was only usable with the shield in place - all safety features relied on software. In addition, the bug was only possible when an operator made a mistake in mode selection and then rapidly (proficiently) corrected it - the rapidity required prevented the bug from being discovered during slow, methodic, careful testing. See Hackaday's article Killed by a Machine (and associated HN discussion) or for the especially curious, a 49-page post-mortem for more detail: https://hackaday.com/2015/10/26/killed-by-a-machine-the-therac-25/ https://hackaday.com/2015/10/26/killed-by-a-machine-the-ther... https://news.ycombinator.com/item?id=12201147 https://news.ycombinator.com/item?id=12201147 http://sunnyday.mit.edu/papers/therac.pdf http://sunnyday.mit.edu/papers/therac.pdf
seorphates 9y ago
Reboot. Around the same time-frame we gathered the flag for a deployment (fleet admiral) and I was responsible for UNIX systems on the ship. Not long after coming aboard the command came down to reboot all of the systems at midnight, nightly (yes, only the UNIX systems). Being that "But Mister.." never really gets you too far in the military I just rode it iterating through any possible reason for the madness, nightly. I could never come up with a good one. Until now. (ok, perhaps not a "good" reason but crazy enough to count.) It now makes much more sense to me that a (terrible) mishap had occurred and possible prevention was only a reboot away. I can see how being exposed to that context at upper levels could easily cause one to latch onto any perceived preventative measures. I also once saw a short ntp time step across multiple clusters (yeh, simultaneously) shut down half of a wafer factory. Time is important.. but rebooting all your systems at midnight probably will not help you to control it. This especially if there are large, hot, fast objects flying around in the night sky and definitely, really, don't do ALL of them at the same time every day .. especially during, you know, battle. /pro-tip
- lostlogin 9y ago
  That's still not great logic. Think of all the crazy shit you have seen fix machines. If all the was implemented you would have users doing some truly bizarre things.
  - seorphates 9y ago
    Mm. That's on point. It is as illogical as having the means and knowledge for prevention and not applying it. The crazy shit (booting theater active operational assets) was implemented by authority. Not patching theater active assets leads to death.

205 points by sine 9y ago

That was a temporary fix, then a software patch was released. I also wouldn't call that a "software" fix.

avar 9y ago

Even better, the timeline: - February 11th: Vendor informed of the issue - February 25th: 28 people die because of the issue - February 26th: The vendor ships a fix I'd have loved to be a fly on the wall for that phonecall on the 25th (or early on the 26th).

nathan_long 9y ago
Oh, gosh. Really? It makes me kind of sick imagining that call.
- sli 9y ago
  You're not even curious to see how it was dealt with and how the issue was expressed to the vendor? I'd never be in a meeting regarding deaths of users of my software, because I just make internal webapps, so I just cannot help but be curious as to how one of those meetings would go.
  - deleted 9y ago
    [deleted]
  - brookside 9y ago
    Sure but > I'd have loved to be a fly ... loving anything about that sad scenario seems impossible.
    - kaishiro 9y ago
      It's a figure of speech, not a show of approval.
    - Nomentatus 9y ago
      Even loving learning enough to avoid the next one? 'Cause that's what responder wants, to learn. "What the hell were they thinking?" is often the most pertinent knowledge of all.
  - hx2a 9y ago
    > I'd never be in a meeting regarding deaths of users of my software I know what that's like. About 20 years ago I was at a consulting firm supporting an electric and gas utility company. Among other things they had to do something called "markouts" which means they paint the ground at a location in a way that indicates exactly what infrastructure they have in the ground and precisely where it is. Markouts are a government organized thing. Before digging somewhere you can call a number and anybody that might possibly have infrastructure in the ground anywhere near your dig site is required to paint their markouts within a short time period. There are stiff fines if you "miss a markout." Anyhow there was a data problem with a markout. The field worker was sent to paint a markout at the corner of two streets that actually ran parallel to each other and didn't meet. Instead of calling it in and questioning the task he did nothing. Shortly after a construction worker put a backhoe through an electrical conduit with 15K volts. There was an explosion that was heard for many miles. The worker died the next day. He died painfully. > so I just cannot help but be curious as to how one of those meetings would go. Finger pointing, of course. Data was being fed back and forth between systems and eventually somebody else took the blame. The field worker who ignored the markout also was blamed. We did add something to our system so that that kind of data error would raise an exception. I learned a lot about care and diligence about data from this experience. Data errors are no joke.
  - muthdra 9y ago
    I'm not, because it would bore me. I see shit like that for breakfast when studying transportation. But if you don't do it, I say it's because of your own primal instincts, so you stuff them down...
phonon 9y ago
You missed this date-- Feb 21--notice goes out to users to avoid "very long run times". Users do not know what that means, and ignore warning. https://www.gao.gov/assets/220/215614.pdf https://www.gao.gov/assets/220/215614.pdf (page 9) "On February 21, 1991, the Patriot Project Office sent a message to Patriot users stating that very long run times could cause a shift in the range gate, resulting in the target being offset. The message also said a software change was being sent that would improve the system’s targeting. However, the message did not specify what constitutes very long run times. According to Army officials, they presumed that the users would not continuously run the batteries for such extended periods of time that the Patriot would fail to track targets. Therefore, they did not think that more detailed guidance was required."
- macintux 9y ago
  That's terrible. Competent technical writing is criminally undervalued.
  - zeeZ 9y ago
    But there's also "presumed" and "did not think" in there. When there's a problem with your killing device you probably shouldn't use it until you've clarified what the problem is and don't just assume your end users will use it correctly. That's like saying "It's fine, the critical vulnerability patch will be applied on reboot", while in reality all your users just suspend to disk and move that annoying reboot nag window behind the task bar where it's out of sight.
    - mikeash 9y ago
      “You probably shouldn’t use it until you’ve clarified...” doesn’t work so well for a defensive system.
      - lostlogin 9y ago
        How long was the off/on cycle? If it was short it would be reasonable to do it periodically. I don't think one can pin the blame on the vendor alone.
kevinconaway 9y ago
Per the GAO report[0] > According to Army officials, the delay in distributing the software from the United States to all Patriot locations was due to the time it took to arrange for air and ground transportation in a wartime environment. I'm not knowledgeable at all on how software for missile batteries was distributed in 1991 from the US to the Persian Gulf but 11 days doesn't seem unreasonable to me. [0] https://www.gao.gov/assets/220/215614.pdf https://www.gao.gov/assets/220/215614.pdf
godelmachine 9y ago
Who was the vendor? Aren't defense contractors required to be on their toes all the time? EDIT → Found it. The PATRIOT Project Office.

sharemywin 9y ago

I remember hearing about this in my numerical analysis class. 1. I remember hearing the system was only designed for XX operational hours but was being run over the operational spec. 2. The time was stored in base 10 so the calculation errors added up over time or something like that so if they had used some base 2 timing scheme it would haven't have had issues with rounding errors. My class was in the mid nineties so the details of my 25 year old memory is pretty hazy...at best.

clw8 9y ago
My recollection matches with yours, except I learned about it in the first week of Embedded Systems 101. If it isn't a standard part of the curriculum at every college embedded systems class, it should be! It really drove home the point that bad code can kill.
- pilom 9y ago
  I learned about it in a Decision Analysis course and had a completely different point driven home. This wasn't bad code. It was code that was correctly written to a very well defined requirement ("System shall be operational for at most X hours before a reboot"). The code was written to a spec that was approved by the customer (the military). Unfortunately though, that requirement wasn't communicated to the end users.
  - michaelmrose 9y ago
    From the article "However, the timestamps of the two radar pulses being compared were converted to floating point differently: one correctly, the other introducing an error proportionate to the operation time so far" The code had a defect that effects its aim from turning it on but because it took 100 hours to drift by 1/3 of a second the problem wasn't apparent when rebooted regularly. If software can't continue to do basic math without manual intervention its defective. In fact everyone including the company that made it admits it's defective. Its possible your teacher picked a great example to illustrate a communication failure.

jasonmaydie 9y ago

The scud missile lead to their deaths, not the software. There's no absolute guarantee it would have intercepted it, plus rebooting a deployed machine regularly is an acceptable fix when it's live in the field

rosser 9y ago
That's a reductio fallacy. If you want to play that game, it was being deployed to that specific place that caused their deaths. Or was it enlisting in the first place? Maybe merely having been born? This is a strictly technical examination of the proximate cause of their deaths; it makes no claims about their ultimate cause. Whether or not a missile system with an accurate clock might have hit the target, it is unambiguous that this one missed specifically because of clock drift.
- jasonmaydie 9y ago
  How so? The implication you and the article are asserting is that the clock error caused their deaths.. rather than the more accurate description "could have prevented death".
  - euyyn 9y ago
    It would depend on whether they were relying on it to work or not.
  - Vivtek 9y ago
    Well, it wasn't the missile that caused their deaths. Strictly speaking, it was the explosion of the missile. Well, wait. It wasn't the explosion - technically, it was the impact of the pressure wave on their bodies that caused ... well, no. Really, it was the fact that their organs stopped working after impact of the ... well. If you really want to be accurate, it was the fact that metabolism ceased to be practicable after their organs stopped working. Well, no, actually, the fact that their mental processes depended on their metabolism - that was really the cause of their... Well, no...
  - KindOne 9y ago

nathan_long 9y ago

> The Patriot missile battery at Dhahran had been in operation for 100 hours, by which time the system's internal clock had drifted by one-third of a second. Due to the missile's speed this was equivalent to a miss distance of 600 meters.

bertjk 9y ago

I've often wondered, considering the supposed low accuracy of Scud missiles, (wiki gives it a CEP of 450m) how much of the casualties from that incident were more due to the bad luck of the missile actually hitting its target.

nerpderp83 9y ago
If the Scud had been brought down earlier in it's trajectory it would have not been near people regardless of any randomness in it's landing.
dnautics 9y ago
many of the scuds "broke up in flight" or otherwise malfunctioned, too, so the actual effectiveness of patriot has been called into question.

brohoolio 9y ago

This is depressing. One of my middle school classmates had a brother killed in a SCUD strike.

tntn 9y ago

Despite other comments below, I think that the equivalence drawn between "failed to save" and "killed" reflects an interesting philosophical choice. I don't think that this equivalence is universally accepted, even by those who call thinking otherwise fallacious. If an EMT fails to save a victim of a car crash, did he/she kill the victim? If the dispatcher misspoke and gave the wrong cross street, delaying aid, did the dispatcher kill them?

rxhernandez 9y ago
In the medical device industry the company who made the device can be found at fault if a clinician makes a poor decision that leads to death based on a fault in the device. If the soldiers would have sought better cover or be otherwise saved in the case that there was no missile defense system was there then yes, some, if not most, of the blame lies on the software error.

mlazos 9y ago

The title of this post is misleading, they eventually supplied a software patch that fixed the clock drift. The Israelis proposed rebooting as a stopgap until the bug could be fixed.

sctb 9y ago
We've updated the submitted title from “Clock error lead to death of 28 Soldiers. Software fix: Reboot system regularly” to a representative phrase (edited for length) from the article. Submitters: please follow the guidelines by not editorializing titles. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html

macawfish 9y ago

Little things do add up.

jimjimjim 9y ago

regarding the comments about bug killed people versus weapon killed people. There is no 1 answer, this argument is a result of black-white/yes-no/us-them single point of blame thinking. and it's terrible. the bug contributed to the loss of life.

deleted 9y ago

[deleted]

dredmorbius 9y ago

The inimitable comp.risks discussed this in 1992: http://catless.ncl.ac.uk/Risks/13/35#subj1.1 http://catless.ncl.ac.uk/Risks/13/35#subj1.1 http://catless.ncl.ac.uk/Risks/13/76#subj8.1 http://catless.ncl.ac.uk/Risks/13/76#subj8.1 And in 1997: http://catless.ncl.ac.uk/Risks/18/79#subj9.1 http://catless.ncl.ac.uk/Risks/18/79#subj9.1

otoburb 9y ago

This was a tragic and preventable loss. It's incredible that a software bug might have been the root cause. At the time, this incident really stuck out because it broke the illusion of our fabled Patriot missile shield protecting us. Civilian expats really believed the inflated Patriot interception rates parroted to us by mainstream media and our American military expat buddies. A large number of remaining expats who had stuck out the Gulf War to that point decided to pack it in and leave when word got out that the Dhahran barracks were hit. Although history shows that Iraq surrendered days after this incident, at the time there was heightened fear and confusion amongst the remaining expats, especially the non-Americans. We left on the last Lufthansa flight (crewed by military personnel) after hearing about this. Nostalgic edit: During the Gulf War embassies issued equipment and rations to expat citizens who chose to stay behind. Americans were issued full body suits (for adults and youths) due to the biological and chemical weapon payloads that Saddam boasted his SCUDs were carrying, along with MREs that tasted fabulous! In stark contrast, Commonwealth citizens were issued a bare gas mask (adult size only) and mono-flavour MREs that tasted like cardboard. The British embassy sticks out in my mind: with stern stone-faced expressions they admonished us all for not evacuating and thus endangering children in a war zone. In addition to the terrible rations and gas masks, they wordlessly gave us a stack of translucent stickers. When asked what they were for, embassy staff explained that in the event of the air siren going off, we should get under our sturdiest tables and don our gas masks (standard procedure), and then slap the stickers on. If the stickers changed colour, it meant we were in the presence of a biochemical agent and would have approximately 10 seconds before we died a horrific death. You kind of had to be there to appreciate the grim humour.

celticninja 9y ago
I mean I kind of understand the attitude of the British Embassy, it wasnt like trouble flared up overnight, the option to leave was there for a long time prior to the war beginning. Obviously it isnt the fault of the children who were kept there by their parents, but some responsibility needs to be borne by the expats that decided they were getting paid well enough to stay.
- otoburb 9y ago

criley2 9y ago

This is bad, editorialized title that is not the title of the article. Mods should change this. The "software fix" was a software patch which corrected the clocking bug. The "software workaround" to use pre-fix was reboot. I hate editorialized, lying titles :(

codazoda 9y ago
Came here to mention that. The title needs a re-write but the story is interesting still.

leggomylibro 9y ago

I could be reading this wrong, but 1/3 of a second within 100 hours seems really good, like something you'd get from a temperature-controlled crystal oven. I don't mean to second-guess them in an area I know so little about, but if that was enough to cause a serious issue in the span of only a few days, shouldn't the devices be designed with a separate synchronization system, at least as a backup? Maybe GPS? Which brings up a sort of interesting question...would a Patriot missile system even have receivers for a weak public signal like GPS, or is it all self-contained?

ajross 9y ago
MIL-SPEC was indeed famous for overspecified components. So it's not terribly shocking that the oscillator on that board would operate really well as an isolated system. You probably don't need temperature control per se, a temperature compensation circuit could probably do that.
GCU-Empiricist 9y ago
As a former submariner who has had used clock for inertial navigation or for similar weapons systems 1/3 of a second over 100 hours is terrible.
- leggomylibro 9y ago
  I mean, it's not an atomic clock, but I'm comparing it to the 32.768KHz RTC crystals I use with consumer microchips. If super-precise isolated accuracy were actually important, I assume they would use a rubidium or cesium oscillator.
  - grkvlt 9y ago
    1/3 of a second in 100 hours is basically 1ppm, or TXCO levels of accuracy, so pretty good i'd have thought, even for a submarine INS?
    - cocoablazing 9y ago

logfromblammo 9y ago

For doing a ballistic propagation, you apply a gravitational map in Earth-centered, Earth-fixed (ECEF) geodetic coordinates, then convert to Earth-centered rotating (ECR) geodetic coordinates, because that way you don't have to correct for the Coriolis effect. That ECEF-ECR conversion requires a time-of-day parameter. You can use a gravitational map that only accounts for latitude, but it isn't as precise. So using an accurate clock is really important if your intent is to hit a missile with a missile.

sjburt 9y ago

This is a completely misleading headline. The Patriot missile was not effective at destroying the Scud [0]. The DoD initially claimed successful intercepts when the missile detonated near the Scud, but it rarely, if ever, actually destroyed the warhead. The only reason there was an illusion of success was that the Scud was also spectacularly unreliable and often broke up on re-entry or failed to detonate. It is a complete falsehood to claim that the Patriot would have prevented this loss of life. [0] http://www.slate.com/articles/news_and_politics/war_stories/2003/03/patriot_games.html http://www.slate.com/articles/news_and_politics/war_stories/...

deleted 9y ago
[deleted]

tofof 9y ago

This particular bug is often taught in university compsci classes as "bug that killed people" is a good attention grabber -- the CS/EE analysis is sound; its truthfulness is only suspect because of the DoD's claimed successes. A more truthful "computer bugs that killed people" example would be the Therac-25 - a machine intended to treat cancer with tightly-focused radiation therapy. Six patients died as a result of massive overdoses of radiation, on the order of 20,000 rads. It was possible for the machine to end up in a state where it delivered full-power radiation without a hardware shield in place to protect the rest of the patient's body. No hardware interlocks were used to ensure that the full power mode was only usable with the shield in place - all safety features relied on software. In addition, the bug was only possible when an operator made a mistake in mode selection and then rapidly (proficiently) corrected it - the rapidity required prevented the bug from being discovered during slow, methodic, careful testing. See Hackaday's article Killed by a Machine (and associated HN discussion) or for the especially curious, a 49-page post-mortem for more detail: https://hackaday.com/2015/10/26/killed-by-a-machine-the-therac-25/ https://hackaday.com/2015/10/26/killed-by-a-machine-the-ther... https://news.ycombinator.com/item?id=12201147 https://news.ycombinator.com/item?id=12201147 http://sunnyday.mit.edu/papers/therac.pdf http://sunnyday.mit.edu/papers/therac.pdf

seorphates 9y ago

Reboot. Around the same time-frame we gathered the flag for a deployment (fleet admiral) and I was responsible for UNIX systems on the ship. Not long after coming aboard the command came down to reboot all of the systems at midnight, nightly (yes, only the UNIX systems). Being that "But Mister.." never really gets you too far in the military I just rode it iterating through any possible reason for the madness, nightly. I could never come up with a good one. Until now. (ok, perhaps not a "good" reason but crazy enough to count.) It now makes much more sense to me that a (terrible) mishap had occurred and possible prevention was only a reboot away. I can see how being exposed to that context at upper levels could easily cause one to latch onto any perceived preventative measures. I also once saw a short ntp time step across multiple clusters (yeh, simultaneously) shut down half of a wafer factory. Time is important.. but rebooting all your systems at midnight probably will not help you to control it. This especially if there are large, hot, fast objects flying around in the night sky and definitely, really, don't do ALL of them at the same time every day .. especially during, you know, battle. /pro-tip

lostlogin 9y ago
That's still not great logic. Think of all the crazy shit you have seen fix machines. If all the was implemented you would have users doing some truly bizarre things.
- seorphates 9y ago
  Mm. That's on point. It is as illogical as having the means and knowledge for prevention and not applying it. The crazy shit (booting theater active operational assets) was implemented by authority. Not patching theater active assets leads to death.