6 ms·

Reverse Engineer’s Perspective on the Boeing 787 ‘51 days’ Directive

187 points by trasz 4y ago

rcyeh 4y ago
tl;dr: 51 days is the wraparound point of a signed 6-byte counter running at 33 MHz, used to invalidate stale data from instruments.
- saratogacx 4y ago
  When I saw 51 days my first thought is it had to be a time rollover. Mainly because of this bug from long ago and how close the time spans are. https://www.cnet.com/culture/windows-may-crash-after-49-7-days/ https://www.cnet.com/culture/windows-may-crash-after-49-7-da...
- drewrv 4y ago
  This assumes there is no margin of error baked into the 51 day rule, which surprises me.
  - nimish 4y ago
    2^47/(32MHz) ~= 50.9 days Not much of a margin there.
    - zokier 4y ago
      2^47/33e6 = 49.36 days. The value is so much off that makes me suspect that this is not the correct analysis, or at least that there are additional factors at play
      - fargle 4y ago
        exactly. you could find a plausible clock rate that rolls over in ~51 days with any number of assumed counter widths. this is not indicative of anything but guessing. and there is no margin at all or negative margin, so clearly this is wrong. is the rollover period 52 days? 60? 90? the whole article is a bunch of unsubstantiated speculation dressed up with lots of facts and details to distract ones attention.
  - lamontcg 4y ago
    This communication is not Boeing communicating a required maintenance interval, they're communicating a problem. It wouldn't seem natural for me for Boeing to add a random hidden margin in a problem description. When it comes to the maintenance remedy, I don't know if Boeing would do this or airlines or the FAA. Presumably the mandatory maintenance reboot interval will be much shorter than 51 days.
- Gibbon1 4y ago
  All I have to say is if my firmware barf's after being up for 8.919 million years I won't care.
- taneq 4y ago
  I feel like even 51 minutes might be too long to wait before invalidating stale instrument data on an aeroplane...
rootusrootus 4y ago
Fascinating analysis. I know planes get used a lot, but I'm surprised that they go for such a long time without ever being powered down.
- Aperocky 4y ago
  51 days seems to be approximately how often my mac dies in kernel panic or starting to be bugged by persistent software problems that go away with a restart.
  - dcormier 4y ago
    I’m at 356 days of uptime on my MacBook Pro. ¯\_(ツ)_/¯
    - assttoasstmgr 4y ago
      So you never install security updates? Because all Apple updates require a reboot due to their SIP "update the frozen image offline" nonsense.
      - Rebelgecko 4y ago
        IIRC security updates always required a reboot even before SIP existed
    - Waterluvian 4y ago
      Awww ya jinxed it. And only nine days from retirement.
    - rootusrootus 4y ago
      Sadly I'm at 2 days, 12:41 myself. I don't get many kernel panics, but this most recent reboot was in fact a panic, coincidentally. Googled the error and it came up as something that happens with M1 Mac Minis while they're sleeping. But while my machine has a M1, it is an MBP and not a Mini. And it was not sleeping. Ah well.
    - 4y ago
kreelman 4y ago
This is a really good analysis of the issue from the just the verbage from FAA. Well done.
userbinator 4y ago
For example, let’s imagine that the timestamp set by the transmitting ES is close to its wrap-around value. After performing the required calculation, the receiving ES obtains a timestamp that has already wrapped-around, so it would look like the message had been received before it was actually sent. Isn't it surprising that modulo arithmetic, as already employed successfully in TCP sequence numbers and the like, still seems to be incorrectly implemented today? What's more disappointing is seeing all the other incredible systemic complexity they've added, and yet the plane appears to have no mechanical backup instruments?
- steffan 4y ago
  To address the second part: > and yet the plane appears to have no mechanical backup instruments[?] This is unlikely in a modern aircraft because mechanical instruments to back up e.g., the artificial horizon / attitude indicator or directional gyro (DG) / heading indicator are: 1) Mechanically complex - the attitude indicator and DG make use of gyroscopes which rotate at up to 24,000 RPM along with other mechanisms. They are typically powered by vacuum or electric motors which consume relatively more power (or require vacuum lines and a vacuum pump) 2) Expensive to maintain - see (1) - they need to be serviced somewhat regularly (3) Heavier than their solid-state counterparts (4) Have [dramatically] different failure modes - instead of a display going dark, a DG will slowly drift as the gyroscope precesses, giving erroneous values. Same with the artificial horizon. This can lead to catastrophic results under instrument meteorologicalconditions (IMC) where the pilots rely solely on instruments to maintain essential things such as heading and level flight. (5) Because of (4) they require additional redundancy to ensure instruments can be cross-checked with one another. This compounds (2) and (3)
  - TillE 4y ago
    Plus the pilot stress of having to adjust to using dramatically different instruments when already in a difficult situation. It's just not a workable idea in general. There are checklists for stuff like instrument failure which can probably recover from a software bug like this.
deleted 4y ago
[deleted]
newsclues 4y ago
Why? Was it a cost issue? Or was there an expectation that a regular maintenance check would occur within this time frame that involved a reboot as part of the maintenance check for diagnostics?
junar 4y ago
Please add (2020) to the title.
acdanger 4y ago
Reminds me of the LAX Air Traffic Control Shutdown of 2004: https://m.slashdot.org/story/49885 https://m.slashdot.org/story/49885
Taniwha 4y ago
51 days is slightly more than 2^32 milliseconds?
- kelnos 4y ago
  If that were the issue, then they'd have to reboot it every 49.7 days, no? Waiting 51 days would trigger the problem they're trying to avoid.
tjr 4y ago
One thing to consider when looking at such things is that commercial avionics software systems are full of known limitations. I do not know if this particular 51-day limitation was intentional or not, but in general: Avionics software starts with writing comprehensive requirements. When the software itself is developed based on those requirements, it is then tested against the requirements, always in a real functioning airplane, but also often in smaller airplane-cockpit-like rigs and in purely simulated environments. Nobody is going to write a requirement that says "this avionics subsystem will function without error forever". Even if you thought you could make it happen, you can't test it. So there are going to be boundaries. You might say that the subsystem will function for X days. What happens after that? It may well run just fine for X+1 days, or 2X days, or 100X days. But it's only required to run for X days, and it's only tested and certified for running for X days. I could easily imagine that this particular subsystem was required and certified for some value of X <=51 days, and it just so happened that if the subsystem ran for over 51 days then it started to fail. Or, it could have been a genuine mistake. But even if the intended X wasn't 51 days, there almost certainly was some intended, finite value for X. We might say, "well, my laptop has run for three years without needing a reboot". Great! Is that a guaranteed, repeatable state of operation that the FAA would certify? Probably not. And besides that, do we really want to have to endure a three-year verification test? In most software, we are happy to say, "it should run indefinitely". For avionics software, that's insufficient. We instead say "it will run at least for some specific predetermined finite amount of time" and then back up that statement with certifiable evidence.
- Aloha 4y ago
  I work in a field that operates under similar development constraints. (Namely it's a mature product in a mature field with well defined requirements) Because if this I regularly get calls from my customers wondering why their system can't do X or Y in the B way instead of the A way, and I have a similar conversation. Wherein I have to explain "no, that wasn't part of your requirements 5 years ago, if you want to change it, you'll need to pay us for more development", that normally eliminates the requirement for whatever it was they wanted pretty quickly. Also, uptime is a factor, I've seen what windows looks like when it runs out of GDI objects, it's strange. But once you see it, you can explain to the customer the importance of regular reboot/restarts.
pixelfarmer 4y ago
I remember articles of the Airbus A350 requiring reboots every N days (150ish or so?). I remember the Patriot missile system required a reboot every 24 hours or so until they fixed the software defect which caused the time counting to drift. And I'm pretty sure there are many more such cases where devices fail if kept on for too long, even in spaces where you are supposed to fill out a lot of "paper"work + jump through a lot of defined processes like in avionics, medical, or automotive field, among a good few others (safety and all that).
- Yizahi 4y ago
  We had a bug years ago that after 50 days of uptime all network sessions dropped on our devices. Apparently it was a session timer overflow in a variable. I think it was unsigned int and time was in milliseconds.
- mormegil 4y ago
  149 hours, see https://en.wikipedia.org/wiki/List_of_software_bugs#Transportation https://en.wikipedia.org/wiki/List_of_software_bugs#Transpor...

187 points by trasz 4y ago

tl;dr: 51 days is the wraparound point of a signed 6-byte counter running at 33 MHz, used to invalidate stale data from instruments.

saratogacx 4y ago
When I saw 51 days my first thought is it had to be a time rollover. Mainly because of this bug from long ago and how close the time spans are. https://www.cnet.com/culture/windows-may-crash-after-49-7-days/ https://www.cnet.com/culture/windows-may-crash-after-49-7-da...
drewrv 4y ago
This assumes there is no margin of error baked into the 51 day rule, which surprises me.
- nimish 4y ago
  2^47/(32MHz) ~= 50.9 days Not much of a margin there.
  - zokier 4y ago
    2^47/33e6 = 49.36 days. The value is so much off that makes me suspect that this is not the correct analysis, or at least that there are additional factors at play
    - fargle 4y ago
      exactly. you could find a plausible clock rate that rolls over in ~51 days with any number of assumed counter widths. this is not indicative of anything but guessing. and there is no margin at all or negative margin, so clearly this is wrong. is the rollover period 52 days? 60? 90? the whole article is a bunch of unsubstantiated speculation dressed up with lots of facts and details to distract ones attention.
- lamontcg 4y ago
  This communication is not Boeing communicating a required maintenance interval, they're communicating a problem. It wouldn't seem natural for me for Boeing to add a random hidden margin in a problem description. When it comes to the maintenance remedy, I don't know if Boeing would do this or airlines or the FAA. Presumably the mandatory maintenance reboot interval will be much shorter than 51 days.
Gibbon1 4y ago
All I have to say is if my firmware barf's after being up for 8.919 million years I won't care.
taneq 4y ago
I feel like even 51 minutes might be too long to wait before invalidating stale instrument data on an aeroplane...

rootusrootus 4y ago

Fascinating analysis. I know planes get used a lot, but I'm surprised that they go for such a long time without ever being powered down.

Aperocky 4y ago
51 days seems to be approximately how often my mac dies in kernel panic or starting to be bugged by persistent software problems that go away with a restart.
- dcormier 4y ago
  I’m at 356 days of uptime on my MacBook Pro. ¯\_(ツ)_/¯
  - assttoasstmgr 4y ago
    So you never install security updates? Because all Apple updates require a reboot due to their SIP "update the frozen image offline" nonsense.
    - Rebelgecko 4y ago
      IIRC security updates always required a reboot even before SIP existed
  - Waterluvian 4y ago
    Awww ya jinxed it. And only nine days from retirement.
  - rootusrootus 4y ago
    Sadly I'm at 2 days, 12:41 myself. I don't get many kernel panics, but this most recent reboot was in fact a panic, coincidentally. Googled the error and it came up as something that happens with M1 Mac Minis while they're sleeping. But while my machine has a M1, it is an MBP and not a Mini. And it was not sleeping. Ah well.
  - 4y ago

kreelman 4y ago

This is a really good analysis of the issue from the just the verbage from FAA. Well done.

userbinator 4y ago

For example, let’s imagine that the timestamp set by the transmitting ES is close to its wrap-around value. After performing the required calculation, the receiving ES obtains a timestamp that has already wrapped-around, so it would look like the message had been received before it was actually sent. Isn't it surprising that modulo arithmetic, as already employed successfully in TCP sequence numbers and the like, still seems to be incorrectly implemented today? What's more disappointing is seeing all the other incredible systemic complexity they've added, and yet the plane appears to have no mechanical backup instruments?

steffan 4y ago
To address the second part: > and yet the plane appears to have no mechanical backup instruments[?] This is unlikely in a modern aircraft because mechanical instruments to back up e.g., the artificial horizon / attitude indicator or directional gyro (DG) / heading indicator are: 1) Mechanically complex - the attitude indicator and DG make use of gyroscopes which rotate at up to 24,000 RPM along with other mechanisms. They are typically powered by vacuum or electric motors which consume relatively more power (or require vacuum lines and a vacuum pump) 2) Expensive to maintain - see (1) - they need to be serviced somewhat regularly (3) Heavier than their solid-state counterparts (4) Have [dramatically] different failure modes - instead of a display going dark, a DG will slowly drift as the gyroscope precesses, giving erroneous values. Same with the artificial horizon. This can lead to catastrophic results under instrument meteorologicalconditions (IMC) where the pilots rely solely on instruments to maintain essential things such as heading and level flight. (5) Because of (4) they require additional redundancy to ensure instruments can be cross-checked with one another. This compounds (2) and (3)
- TillE 4y ago
  Plus the pilot stress of having to adjust to using dramatically different instruments when already in a difficult situation. It's just not a workable idea in general. There are checklists for stuff like instrument failure which can probably recover from a software bug like this.

deleted 4y ago

[deleted]

newsclues 4y ago

Why? Was it a cost issue? Or was there an expectation that a regular maintenance check would occur within this time frame that involved a reboot as part of the maintenance check for diagnostics?

junar 4y ago

Please add (2020) to the title.

acdanger 4y ago

Reminds me of the LAX Air Traffic Control Shutdown of 2004: https://m.slashdot.org/story/49885 https://m.slashdot.org/story/49885

Taniwha 4y ago

51 days is slightly more than 2^32 milliseconds?

kelnos 4y ago
If that were the issue, then they'd have to reboot it every 49.7 days, no? Waiting 51 days would trigger the problem they're trying to avoid.

tjr 4y ago

One thing to consider when looking at such things is that commercial avionics software systems are full of known limitations. I do not know if this particular 51-day limitation was intentional or not, but in general: Avionics software starts with writing comprehensive requirements. When the software itself is developed based on those requirements, it is then tested against the requirements, always in a real functioning airplane, but also often in smaller airplane-cockpit-like rigs and in purely simulated environments. Nobody is going to write a requirement that says "this avionics subsystem will function without error forever". Even if you thought you could make it happen, you can't test it. So there are going to be boundaries. You might say that the subsystem will function for X days. What happens after that? It may well run just fine for X+1 days, or 2X days, or 100X days. But it's only required to run for X days, and it's only tested and certified for running for X days. I could easily imagine that this particular subsystem was required and certified for some value of X <=51 days, and it just so happened that if the subsystem ran for over 51 days then it started to fail. Or, it could have been a genuine mistake. But even if the intended X wasn't 51 days, there almost certainly was some intended, finite value for X. We might say, "well, my laptop has run for three years without needing a reboot". Great! Is that a guaranteed, repeatable state of operation that the FAA would certify? Probably not. And besides that, do we really want to have to endure a three-year verification test? In most software, we are happy to say, "it should run indefinitely". For avionics software, that's insufficient. We instead say "it will run at least for some specific predetermined finite amount of time" and then back up that statement with certifiable evidence.

Aloha 4y ago
I work in a field that operates under similar development constraints. (Namely it's a mature product in a mature field with well defined requirements) Because if this I regularly get calls from my customers wondering why their system can't do X or Y in the B way instead of the A way, and I have a similar conversation. Wherein I have to explain "no, that wasn't part of your requirements 5 years ago, if you want to change it, you'll need to pay us for more development", that normally eliminates the requirement for whatever it was they wanted pretty quickly. Also, uptime is a factor, I've seen what windows looks like when it runs out of GDI objects, it's strange. But once you see it, you can explain to the customer the importance of regular reboot/restarts.

pixelfarmer 4y ago

I remember articles of the Airbus A350 requiring reboots every N days (150ish or so?). I remember the Patriot missile system required a reboot every 24 hours or so until they fixed the software defect which caused the time counting to drift. And I'm pretty sure there are many more such cases where devices fail if kept on for too long, even in spaces where you are supposed to fill out a lot of "paper"work + jump through a lot of defined processes like in avionics, medical, or automotive field, among a good few others (safety and all that).

Yizahi 4y ago
We had a bug years ago that after 50 days of uptime all network sessions dropped on our devices. Apparently it was a session timer overflow in a variable. I think it was unsigned int and time was in milliseconds.
mormegil 4y ago
149 hours, see https://en.wikipedia.org/wiki/List_of_software_bugs#Transportation https://en.wikipedia.org/wiki/List_of_software_bugs#Transpor...