16 ms·
Latest 737 Max Fault That Alarmed Test Pilots Rooted in Software
- pcl 7y ago> Because the fault was triggered by specific streams of erroneous flight data, a new software patch can be devised that monitors the computer for that highly unusual condition and prevents movement of the stabilizer when it occurs, one of the people said. Oh man. That sounds like a mess.
- teawrecks 7y agoBandaid fix if I've ever heard of one.
- User23 7y agoMy jaw literally dropped. It actually took me a moment to realize I was sitting gape-mouthed reading that. We're talking airplanes here, not some LAMP stack. If this is what we can expect going forward, the Jet Age isn't going to last another 50 years. This kind of software ought to be developed using a tool like TLA+.
- Aloha 7y agoWe effectively have a front row seat to the sausage making of software development, its not pleasant.
- ulfw 7y agoThat describes the whole 50+ years old 737 to it’s core.
- burfog 7y agoIt's proper. This is exactly how you do things with life-critical embedded software. You detect when things are crazy, which of course shouldn't happen but nevertheless does, and you react in a way that is hopefully not disastrous. The alternative is to ignore the problem. Sensors go bad. Actuators go bad. Voting hardware goes bad. Message routing goes bad. Even the RAM sometimes goes bad. When the hardware is failing ("specific streams of erroneous flight data") there isn't going to be a reliable solution. You can't even fully enumerate all the possible failures. How could you possibly guess that a message routing chip now flips bit 7 in every angle-of-attack measurement? Boeing's answer is standard: when the values look crazy, stop doing stuff that might make the situation worse.
- throwaway2048 7y agowhat happens when things are actually crazy and it causes a crash itself?
- Haga 7y agoOnly embedded guy here. You can't catch them all. Height radiation alone sees to that.
- BinaryIdiot 7y agoI suspect your take is correct but I understand the alarm from the previous poster as the way it's worded makes it sound like they'll look just for condition X, Y and Z and _then_ take action versus your explanation which looks for a lot of incoming, bad data and then handles it. This might just be a bad case of people talking past one another.
- tus88 7y agoThe way I read it they are not detecting malfunctioning sensors, but essentially a computer fault caused by it's inability to deal with erroneous input. Rather than fixing the computer, they are essentially wrapping a great big try/catch around it....good God.
- comex 7y agoThat interpretation seems like a stretch to me.
- aurelwu 7y agoThat sounds like adding a heuristic which works most of the time which is something I can see myself doing to cover some pathfinding issue in a game where it's sufficient if it works 99,9% of the time or when working on a simulation where I can look on the result data afterwards and can see if something looks highly unusual despite the heuristic and improve it iteratively - a luxury you don't get with air planes, at least not without casualties. Additionally given the very limited computing power of the hardware in the 737 adding even more software to it might create even more problems.
- thelittleone 7y agoWasn’t aware the compute power was low on the 737. Interested if you have some more info? Always thought they would spec a substantially more processing capability than required. I understand economics are a factor but surely it’s not through necessity to save power?
- Maakuth 7y agoHere's an article about it. According to this piece, 737 avionics run on a pair of 80286s. https://www.moonofalabama.org/2019/06/boeings-software-fix-for-the-737-max-problem-overwhelms-the-planes-computer.html https://www.moonofalabama.org/2019/06/boeings-software-fix-f...
- seymore_12 7y agoThe should at least get 1202 error..
- tyingq 7y agoIf true it uses a 286, for some relative comparison, the fastest 286 is 1.5 MIPS. A run of the mill current Arduino is probably 20 MIPS, and a Raspberry PI3 is 2451 MIPS.
- robin_reala 7y agoCan’t find the exact MIPS estimate for the Harris CS80C286 25Mhz 286, but given that the 1.5MIPS was quoted for the 10Mhz Intel one I’d guess it’s substantially faster. Although still slower than an Arduino :)
- xvf22 7y agoIt's one of those fixes where they add up the value of human life if something goes wrong and say 'OK we can handle that, ship it"
- alexis_fr 7y agoI wonder if the human life calculation worked well this time. As far as I see, Boeing lost more than the sum of the human lives; they also lost reputation for everything new they’ve designed in the last 7 years being corrupted, and they also engulfed the reputation of FAA with them, whose agents would fit the definition of “corrupted” by any people’s definition (I know, they are not, they just used agents of Boeing to inspect Boeing because they were understaffed), and the FAA showed the last step of failure by not admitting that the plane had to be stopped until a few days after the European agencies. In other words, even in financial terms, it cost more than damages. It may have cost the entire company. They “DeHavailland”’ed their company. Ever heard of DeHavailland? No? That’s probably to do with their 4 successive deintegrating planes that “CEOs have complete trust in.” It just died, as a name. The risk is high.
- lizmat 7y agohttps://en.wikipedia.org/wiki/De_Havilland_Comet https://en.wikipedia.org/wiki/De_Havilland_Comet
- iguy 7y agoExcept that, unlike de Havailland, Boeing is too big to fail. There's just no way the US will allow it to keel over, and leave Airbus the sole supplier. And as with the banks, you can bet that this implicit guarantee gets figured into what decisions people take.
- jacobush 7y agoThere is so much of U.K. which is incredible and depressing at the same time. Completely pummelled by war, (receiving no aid like West Germany did) they managed to, in 1949, release a Space Age passenger commercial airliner, defining the genre. It looks modern even today. Yet, fumbled the ball. They also had a space program, closed it down after their first satellite, which was a success. Concorde! (With France.) It goes on an on, I'm sure you have favourite examples. It's heart wrenching.
- tus88 7y agoWow a textbook example of a bandaid solution. Reminds me of something a novice programmer would do. When faced with a complex problem....nothing beats an even more complicated solution!
- goodcanadian 7y agoSounds like a running joke at my work. Any hardware problem or design failure: you can fix it in software!
- zwaps 7y agoDo you work at Boeing? ;}
- rasz 7y agoits industry standard https://en.wikipedia.org/wiki/Application_firewall https://en.wikipedia.org/wiki/Application_firewall https://en.wikipedia.org/wiki/Web_application_firewall https://en.wikipedia.org/wiki/Web_application_firewall
- minedwiz 7y agoTo be fair, any hardware running a WAF in Prod these days is going to have more compute power than a 737.
- zerogvt 7y agoWhen your design is a mess you're probably better off to scrap it altogether and start from a clean piece of paper. "Fixes" to a bad design are adding to the mess. And in this case the mess has already cost hundreds of lives. You cannot "patch" this...
- DanHulton 7y agoI see this sentiment a lot, but its kind of surprising. Traditional wisdom states the opposite - never do a full rewrite, you lose the years of buit-up fixes and edge cases and have to learn from scratch in too many places. Despite everything, I still don't think the solution to the 737 MAX problem is to start over. The base problem that Boeing has its actually their terrible engineering culture, the one that values shortcuts and profit over thoroughness and safety. Even if they started from a blank piece of paper, they'd have that engineering culture problem (and to be fair, it is far more likely an engineering management culture problem), and they'd still turn out a design that kills people.
- acqq 7y agoThe problem here is that what Boeing is selling ("737 that spends significantly less fuel and doesn't need recertification") simply can't be made safe. Whatever results from the marketing driven goal (selling something that doesn't and can't exist in reality) will remain unsafe as long as all the "already sold features" are preserved. So it's not if they should "start from clean" but just accept the need for the new certification and then do the changes that are really needed. It will be less profitable for Boeing but it righly should be this time! They are killing people otherwise.
- selectodude 7y agoThe funny thing is this plane already exists. It’s called the 757 and Boeing stopped making it 15 years ago because the airlines wanted stretched 737s far more than they wanted the clean sheet redesign.
- heisenbit 7y agoYep. At this point in time they should have plenty of test cases and regression tests which should catch such situations. While Boeing may feel they are doing a lot of testing in the simulator - there are way fewer simulators staffed with test pilots than there are 737 MAX out there. When such errors surface at this time in manual testing it calls into question the development process. And a lot of the trust - aka. certification - is based on that process. As I have mentioned here before: The angle of attack sensor was never intended for the purpose it is now used for - military aircraft may do such things but not as an afterthought and not connected with a control surface which has a lot of delay (trim wheel winding. Delay is a major factor in destabilizing control loops). Equivalent sensors are the airspeed pitot tubes but they are a much more robust concept. Even they have brought down airplanes. We lack sufficient real world (heat, humidity, snow) information about the quality of the sensor data even before it may get distorted by transmission and processing on its way to the control law. Cleaning and validating sensor data is tricky under the best of circumstances. It does not sound like they have a handle on it. Hi Boeing, what real world data from mounted angle of attack sensors have you analyzed? Why do you believe this data sufficiently represents climate zones and flight conditions (take-off, landing, turbulence, malfunction modes)? How do you clean and validate the sensor data? Please explain, because otherwise I won't step into one of these machines.
- michaelcampbell 7y agoCan you forward the data Airbus, your car or public transport vehicle manufacturer, Otis or equivalent elevator company obviously sent you for their respective operating environments?
- laythea 7y agoIt is. But don't worry, when it is found out that it is a mess, we have a plan. We can add a software patch to correct the bad software output. Not fix the bad software. But add some more to correct the output. Genius!
- Aloha 7y agoImagine if your bugfix process was broadcast to the world, every new IN added to spirateam et al, every new crash condition found by test/QA, every time the software failed in test, it got printed by news organizations the world over. This is world the 737MAX software people are living in, for this alone, they have my utmost respect.
- supercanuck 7y agoimagine if your bug has the potential to kill 300 people.
- gamedori 7y agoTo be fair, if another 150 people die that will be the end of the 737 program - and probably Boeing. 300 more deaths is highly unlikely.
- ulfw 7y agoSo one more plane crash is possible but two are not?? Huh?
- CamperBob2 7y agoYes. After one more plane crash, it will probably become uneconomical for carriers to continue operating the 737.
- Aloha 7y agoHow does a crash on the 737MAX effect operations on 737NG aircraft, or older ones - what suddenly makes it uneconomical?
- alexis_fr 7y agoBecause people are made aware that Boeing wasn’t serious with engineering since 2012. And the FAA paid Boeing consultants to inspect Boeing, which by all layman’s definition fall under the terms of “corrupted”. Who wouldn’t pay attention to the little mention in Google Flights with the brand of the plane?
- nocturnial 7y agoI really can't help to think that the design and production of this plane was rushed. They used a one sensor input to MCAS because the FAA wouldn't certify a two sensor input without requiring a level D certification which meant sim training for pilots. This latest revelation in the article only strengthens my belief it was rushed and possibly to the point that it was irresponsible.
- onli 7y agoThat's seriously overdiplomatic. The plane was rushed and it was constructed very badly. It was absolutely irresponsible to bring it into service, it's not a question anymore. It's a plane that crashes itself automatically into the ground and that killed over 300 people.
- marcyb5st 7y agoAgreed. However, the worst part is the FAA. Specifically, what baffles me is the massive hand-wave they did on this considering it is one of their mandates to check flight-worthiness of Airplanes/Airlines. I hope all the other agencies worldwide will double check all the prior certifications issued by the FAA, because all their credibility is gone. Especially considering the US was the one the last countries to ground the 737 MAX. I wonder what happened there. Bribes? Favors? Lobbying and ties? All of the above?
- jeremyjh 7y agoRegulatory capture. After their careers at the FAA they will go work at Boeing.
- linuxftw 7y agoDo you think regulatory capture actually exists? If so, why is the pharmaceutical industry immune?
- RockyMcNuts 7y ago
- anticristi 7y agoBefore upgrading to the latest Ubuntu, I like to wait for a few months until my friends confirm that it's for the better. I guess I would do the same for the MAX. I can't imagine what it must be like to be one of the first pilots to fly the MAX once the grounding is lifted. Can pilots refuse to fly a specific plane "for a few months, 'cause I have kids waiting for me home"?
- ninth_ant 7y agoThere were almost 400 delivered by the time it was grounded, some of them in service since 2017. Yes there were two high profile failures and this failure rate is extremely out of the norm by the very stringent standards of aviation — but these are not sudden-unavoidable-death machines that gets sensationalized.
- linuxftw 7y agoThat's exactly what they are. How many planes have malfunctioned in the same way and haven't nose-dived into the ground? So far, 0 planes that have had this issue haven't been lost (first plane had the issue, was recovered using non-standard procedures, then same plane killed everyone, even after sensor was replaced!)
- gomijacogeo 7y agoYou appear to be making a straw man argument that the planes are safe by claiming to refute the 'sudden-unavoidable-death machines' argument which no one is making. The probability of a total loss of an aircraft is basically proportional to either its flight hours or number of takeoffs and landings (which are (handwave) correlated (unless you are Aloha Airlines)). A fleet of thousands of planes (especially a 'mature' design), flying thousands of hours per year, in service for decades should have very few losses due to fundamental design issues. It would be nice to say it should be <1, but the reality is the stars sometimes align and several weaknesses conspire to make a fatal event. With MCAS, the 737 Max 8 was already at a rate of 2 events in less than a year with a fleet that was a tiny fraction (<6%) of its ultimate fleet size. Assuming nothing else changed, at full fleet size, that would be over 30 losses a year, or well over 1000 losses over the fleet's lifetime. They misengineered the system by at least 3 orders of magnitude. This is, at the very least, profound professional and managerial negligence. I'm not not yet convinced it's not also criminal. At the very least, all management who touched this subsystem or were part of the no-new-training-at-any-cost push should have already been shown the door up to and including the CEO. And yes, things _are_ being done and they are adding 'nines' to the reliability as quickly as they can, but again, they are starting from a subsystem that kills everybody on board every 800,000 flight hours or so. They appear to be dishonest with the public (and perhaps themselves) about how much risk is eliminated with each change and, similarly, appear to not be factoring in that some of these changes introduce other risks and failure modes. What we have here is a giant fucking mess and a management organization that fosters no confidence in its ability to navigate itself out of it. In the end, I do expect Boeing to somehow glue-gun 3 more 9's of reliability onto this airframe and stop bleeding valuation, but it won't be pretty and it probably won't be fast.
- s_T_e_v_o 7y agoOne thing I learned many years ago, you can solve infrastructure problems with software. If prior jets have multiple sensors and the new jet has one sensor, then software isn't going to help in this situation. Gyros worked back in 1911. why not use one to back up the failed sensor? seems like a simple fix that could be mounted anywhere on the plane.
- cesarb 7y ago> Gyros worked back in 1911. why not use one to back up the failed sensor? An angle of attack sensor measures something a gyroscope cannot measure: the angle of the plane relative to the wind. The amount of lift produced by the wings depends on this angle.
- hammerbrostime 7y agoSounds to some degree like a usability issue. Echoes of Three Mile Island.
- laythea 7y agoCan anyone in the know explain why, if ground clearance due to the new engines is the issue, why Boeing cannot just add a few inches to the wheel stems? This would raise the whole aircraft off the ground. Surely that wouldn't effect aerodynamics too much as the wheels are up most of the time.
- Glawen 7y agoBecause you then need longer and heavier landing gear, which you need to fit somewhere in the plane as they retract. This is a ripple effect where you probably need to redesign the complete airframe, from fuselage to the wings. They actually cleverly modified the landing gear to gain a bit of heigth, but it was not enough: https://youtu.be/F4IGl4OizM4 https://youtu.be/F4IGl4OizM4
- kjar 7y agoIt appears the 737 Max is a total failure. Pushed through by Boing to outpace Airbus, from physical design, attitude sensors, re-certification bypass imperative, and flight control software, finally killing nearly 600 people before being ground. In short I’m never flying in one!
- kyberias 7y agoAre there any public databases that would show plane model failure statistics so that I could choose how to fly?
- Someone1234 7y agoI'm surprised nobody is discussing this paragraph from the article: > The failure scenario was known previously and had been assessed in a safety analysis when the plane was certified before entering service in 2017. At that time, Boeing concluded that pilots could overcome the nose-down movement by performing a procedure to shut off the motor driving the stabilizer movement. This isn't a new fault. Boeing certified this as safe along with MCAS back when the aircraft first flew in 2017, using the same justifications ("pilots can overcome it"). They weren't going to fix it this time either, except test pilots ran across it in late-stage simulations monitored by the FAA and found it wasn't as easy to overcome as Boeing had been asserting in their cost-safety analysis (just like MCAS) and now the FAA are requiring a fix. Has Boeing learned anything from MCAS? The company has cultural problems vis-à-vis safety. I'm just glad the FAA are doing their job this time around.
- linuxftw 7y agoThe FAA is only doing their job this time around because the rest of the entire planet, literally, was grounding the aircraft while the FAA baselessly stated the aircraft was safe. This is all stuff that should have been investigated after the first crash, not the second. Has Boeing learned anything? Of course not. There are no executives facing jail time, or even fines! They're shameless frauds that outsourced their critical systems to the lowest bidder with predictable results.
- ricardobeat 7y agoI’ve only recently started following the air industry more closely, but isn’t it ironic that Boeing, the one on the “direct control” side vs computerized systems from AirBus, is the one now suffering from software and automation failures? I wonder if it might be a consequence of being a late adopter, and rushing the development of systems they never had before.
- ulfw 7y agoThe only reason Boeing is in the “direct control” business is because their best seller is from the 1960s and they don’t dare touch it. Their newer wide bodies 777 and 787 are fly-by-wire and in the latter case even fully electric to a fault.
- kitchenkarma 7y agoMove fast break things kind of doesn't work well with planes. Also too big to fail (fall) doesn't apply here.
- nimish 7y agoAu contraire, Boeing is most definitely too big too fail and will be bailed out somehow.
- breatheoften 7y agoI was just reading an article on New York Times that describes the process since 2005 whereby safety oversight responsibilities over Boeing were gradually moved in house — until by 2018 Boeing was self-certifying for 96% of safety testing. If they are doing 96% of the safety validation work — then they should bear 96% of the responsibility for safety process failures ... in this case, given the break down of safety validation capacity and what I’m increasingly viewing as essentially a coverup after the first and second crashes — 96% of the responsibility is and probably should be - enough to bring Boeing down ... I’m beginning to think that might be the only way to ensure responsibility for this failure gets allocated in a sufficiently accurate way to ensure this scenario isn’t likely to happen again ... allowing the faa to have the responsibility to bear the weight of this process breakdown would basically just allow Boeing to shift responsibility for this outcome off themselves and onto an organization which it seems has been deliberately engineered (by Boeing) to not have the capacity to perform effective oversight ...
- SkooterIn228 7y agoLaying off senior Boeing software engineers and outsourcing their work to India for $9/hour is really paying off for Boeing.