6 ms·
Boeing's 737 MAX Software Outsourced to $9-an-Hour Engineers (2019)
- jkic47 3y agoThe article seems to conflate engineering salaries in a low cost of living country with the quality of the software. As I understand it, the software performed exactly as designed, forcing the nose of the plane downwards to counter the upwards torque produced by the engines being offset relative to the centre of gravity of the plane. Mentour Pilot had a great episode on the two tragic accidents. It appears that Boeing did not expose the way the software was designed to behave, and the system silently turned itself back on even after the pilots disengaged it.
- silent_cal 3y agoIn my experience there is a correlation there. Low-cost outsourced code is generally pretty bad.
- Smeevy 3y agoNot that you intimated this, but I don't know that it's necessarily a problem with the people writing the code. What I've seen is that organizations that try to save money by offshoring are not providing adequate domain knowledge or usable requirements to the developers. Throw in a massive timezone difference and every misunderstanding winds up living in your system.
- silent_cal 3y agoThat could be. However I think the language barrier is also a significant factor. But of course there is nothing inherently problematic about offshore developers.
- cityofdelusion 3y agoI've had to deal with offshores my entire career and have even flown out to train them up. They are good people, but almost universally, they contract for U.S. companies via large body shops (you know who they are) that have awful pay, poor treatment of developers, and just a general culture of half-assing work. I've told various great offshore developers personally to leave as fast as possible and join literally any other organization for way more pay. Good developers have zero business working at those body shops that all the U.S. corps utilize.
- pfannkuchen 3y agoCulture issues do exist. IME different cultures have very different approaches to communicating e.g. “I don’t know” or “I made a mistake”. Not everyone in a particular culture acts the same, but it definitely affects the average outcome.
- initplus 3y agoIt’s not a problem with individual offshore engineers. The problem is the way the business they are employed by operate.
- rramadass 3y agoFalse. https://news.ycombinator.com/item?id=39700806 https://news.ycombinator.com/item?id=39700806
- silent_cal 3y agoMy experience is false? Ok..
- rramadass 3y agoCorrelation is the inference that you draw from your experience which can absolutely be wrong.
- silent_cal 3y agoCorrelation is the inference I draw from my experience? That's one of the more nonsensical things I've ever heard.
- rramadass 3y agoPerhaps a course in "Inferential Statistics/Statistical Inference" is in order here.
- silent_cal 3y agoI've taken several graduate courses in statistics. That's how I know this makes no sense at all.
- rramadass 3y ago> several graduate courses in statistics. Ha, Ha, Ha!
- uuddlrlrbaba 3y agoWeird take. The theme throughout has been cost cutting and poor oversight for the sole benefit of shareholder profits. A proper in house team would probably say, hey what happens when the sole sensor feeding our software is obstructed by a birthday balloon? Maybe we need an exponentially backoff instead of every 30 seconds. Maybe nose down actions shouldn't be automatically executed at low altitude. What we see playing out is the result of delegation and "not my problem", outsourcing safety critical systems to the lowest bidder...
- NullPrefix 3y agobirthday balloon? In front of a landing plane at 170mph?
- moosedev 3y agoGP is (quite reasonably) using a deliberately slightly silly example as a rhetorical flourish. I would do the same. Think of it of a stand-in or totem for all of the possibilities (from silly to very real) that leap into horrifying life in an engineer's brain when they learn that a critical system has a single point of failure.
- neuralRiot 3y agoIn a safety-critical design you have to take in account what will happen if what cannot happen happens.
- jiggawatts 3y agoYes. It happens. In a recent news clip a pilot was interviewed about the MAX and he raised the risk of bird strike and birthday balloons as something that “happens all the time”.
- uuddlrlrbaba 3y agoIt actually happens, similarly to bird strikes or collisions with essentially any object that can float or fly into the path of a plane. https://simpleflying.com/avianca-airbus-a319-balloon-strike/ https://simpleflying.com/avianca-airbus-a319-balloon-strike/ https://www.latimes.com/archives/la-xpm-1994-09-22-me-41720-story.html https://www.latimes.com/archives/la-xpm-1994-09-22-me-41720-... https://dronedj.com/2021/08/31/update-faa-says-envoy-air-passenger-plane-hit-balloon-not-drone/ https://dronedj.com/2021/08/31/update-faa-says-envoy-air-pas...
- hgfghj 3y agoMy experience is that contract work like this ends up being done by ordertakers; the contractors execute the design, no matter how stupid. Whereas, the best performing companies hire engineers who are empowered to understand the goals, understand the problem, and push back when appropriate. The results are much better, because the people working on each system, understand it deeply and care. A low-cost contractor isn’t allowed to care. They aren’t even allowed to talk to somebody who matters. This is a management disgrace.
- genman 3y agoI think that this a fair take, especially about the possible power structure.
- jiggawatts 3y agoOften this is the entire point. The in-house staff are seen as obstructionists, whereas outsourcers provide the equivalent of meat robots. Many managers far prefer the latter over the former.
- solarwindy 3y agoThey prefer the latter because the mere existence of the knowledgable, informed, involved engineer questions the need for the manager at all.
- gregors 3y agoAnd yet we seem to conflate salaries with competent leadership positions.
- jkic47 3y agoI think you are comparing apples and oranges here. An individual contributor is typically given a tangible task (eg: deliver a product that does x in y months for a cost of z). There are several qualified people they could choose from to achieve this task, which drives down the cost. I worked for a company with 10,000 engineers. Oversimplifying for effect, but their skills are not unique. A leader is given a far more intangible task (eg: gain 30% market share in z years for your division OR deliver 9M of product per day through 202X), and they are taking a huge risk. The team under them may not have the slightest clue how to do this. The interest rates can change. Currency fluctuates. They are paid to calmly chart a path through the chaos and deliver a result. At this level, there were about 60 of us, and the atmosphere gets rarer as you rise further in the ranks. They made absolutely certain we were kept super happy because we could create tangible value orders of magnitude above the average employee, and they REALLY did not want us working for a competitor.
- renewiltord 3y agoMCAS wasn't outsourced to India LOL. People here are gullible idiots hahaha.
- rednerrus 3y agoThe biggest problem with offshore engineering teams is their dedication to delivering exactly what you ask for. One of the qualities of a good software developer is to take the requirements and interpret what needs to happen then deliver appropriately. Delivering exactly what's written without thinking about the implications is a huge problem and one of the biggest issues I've run into working with offshore teams.
- struant 3y agoIf just deliver exactly what you are told to, you aren't doing any "engineering" at all.
- datavirtue 3y agoRight,I just spend a little extra time on prompts and get much better, faster, work done completely stress free.
- jkic47 3y agoI agree with you on the problem with offshoring, having done this myself in a previous role in a regulated industry. However, the offshore team are usually not intended to act as domain experts. In fact, they were very likely explicitly proscribed from interpreting the specifications handed to them to guard against them trying to act as domain experts and delivering something different from expectations. As such, they were (likely) not the ones who specified that MCAS should silently turn itself on after a pilot turned it off. That misjudgment probably lies with the engineering team who made that design decision, and it had tragic results.
- rramadass 3y ago> However, the offshore team are usually not intended to act as domain experts. In fact, they were very likely explicitly proscribed from interpreting the specifications handed to them to guard against them trying to act as domain experts and delivering something different from expectations. Finally! Thank you for stating this explicitly. In safety critical systems specifications are everything and is always done by a team of domain experts to an enormous amount of detail (with optional formal methods verification). The actual coder has to just use the chosen implementation language carefully to meet the specification with no personal interpretations; everything has to be explicit and documented.
- wannacboatmovie 3y ago> As I understand it, the software performed exactly as designed This is very likely correct. People on HN don't seem to understand how software is created in regulated industries outside the tech bubble. In a complex system such an aircraft, the behavior of the system is modeled and detailed requirements are generated. These models are created by the system engineers. The models and requirements are then handed off to the software engineers to implement the modeled behaviour in code. The software is then tested to see if the behaviour matches the models. So it doesn't matter how much the SWEs were paid, if the software met the requirements, and implemented the models as designed, then the software engineers did their jobs.
- adriancr 3y agoThe software engineers job is also to think about what happens in corner cases. What if sensor malfunctions?, how should software act?, will it crash the plane? If yes, then raise problem and refuse to implement / avoid killing people. Then you have QA which should have the same questions and test plan to make sure corner cases are covered. QA is also responsible. Then you have management which is supposed to make sure there are people skilled enough doing the above and the initial systems parts and encourage this environment. Then you have business people to ensure sufficient resources to hire competent people above. So what is happening is a massive problem everywhere...
- nine_zeros 3y ago> If yes, then raise problem and refuse to implement / avoid killing people. Often, software engineers are pipped, written up, berated, stack ranked - and churned via various management processes for doing exactly as asked for. Until the culture of the company aka ALL levels of management changes to full accountability without blaming the actual engineers, nothing will change. The current corporate structure just isn't set up to incentivize good work.
- applied_heat 3y agoPerhaps software engineers looking for accountability would like the same level of liability that is enjoyed by their peers in civil, mechanical, mining, and electrical engineering?
- karmakaze 3y agoYep, the problem with the software wasn't the implementation, it was the spec: to operate contrary to the expectations and experience of pilots. The problem was that they didn't outsource the design spec. I doubt an externally developed design would be so contrary, with a flippant we'll document it and include it in the short brush up training.
- datavirtue 3y agoThe umpteenth comment trilling on as though MCAS was built by contractors when it was explicitly stated in the article that it was developed in-house by Boeing employees.
- karmakaze 3y agoStill blame the ones defining the spec rather than implementing it.
- acchow 3y agoExperienced engineers don’t just build things to spec. They question the spec and the human experience of using the technology.
- rramadass 3y agoNot true. https://news.ycombinator.com/item?id=39700847 https://news.ycombinator.com/item?id=39700847
- zamfi 3y agoIs this more, or less, concerning than the growing trend to outsource software to $0.03 / 1000 token AIs?
- tledakis 3y agoDon't worry, with chatgpt the software is going to be safer. The plane won't even take off :)
- FirmwareBurner 3y ago"To save us time I tasked my AI, Anton, to debug the code, and Anton decided that the best way to get rid of all SW bugs was to delete the entire codebase, which is technically and statistically the right solution to what I tasked it to do." -Gilfoyle
- pipe01 3y agoThat's why we shouldn't ask AI to fix climate change
- darkhorn 3y agoOutsourcing is a very bad practice. Those softwares tend to be very low in quality.
- Madmallard 3y agoWait til Devin makes your next safety critical system. :)
- deleted 3y ago[deleted]
- aslihana 3y ago> recalled one manager saying at an all-hands meeting that Boeing didn’t need senior engineers because its products were mature. Shame. Industrial companies, if you decide as pincher, you will be doomed in your little world... as it always has been :)
- FirmwareBurner 3y agoMost big "industrial-like" product companies are pinchers when it comes to SW. They offshore a lot hence why they pay SW devs so little. And it mostly works for them. It works for them, because like Boeing said, they have old battles tested processes which usually means big mistakes can easily get caught ahead of time. And if big mistakes don't get caught they usually can't kill anyone and generate a huge public scandal.
- gostintheshell 3y agoI don't think it works for them, I think they are just entrenched with good moats. What I think actually happens is they get to prove all sorts of ridiculous things are good management and that all sorts of managers need to be coming up with complex ways to manage what would be obvious in a situation where reality was still visible underneath. In the end they pay a lot more than if they had competent individual contributors and less darwinian fights to the death in their org charts. But having a better organization is not really compatible with the incentive system involved.
- aslihana 3y agoI can easily say some of them calls themselves as "futuristic" pays too many unrequired attention and money much more than they pay technological infrastructure... what a oxymoron
- ametrau 3y ago[flagged]
- nuancebydefault 3y agoThe country is not what counts, there are good and bad engineers in any location. The hourly rate is a much more reliable statistic, though without a good quality control feedback loop anything can happen.
- ametrau 3y agoYou've got a point in general but I disagree in this situation. You can pay that to an outsourcing company in India expecting top talent but then get who knows. Also, there's a language barrier vs a local. Honestly, this is widely known (and really just common sense now).
- playingalong 3y agoThis is pure racism.
- ametrau 3y agoNot intended to be racist definitely. Also, india has many races from my experience there. This is more my from my own real life frustrations and experience. If we can't talk about a country, and certain issues there, then where are we?
- lencastre 3y agoFivrrr for airplane safety,… it’s like that BM episode, flywheel of death…
- fargle 3y ago> BM episode, flywheel of death what is this? do you have a link?
- fwungy 3y ago[flagged]
- methodical 3y agoNot that you're incorrect in all cases, but generally speaking, you do get what you pay for. SWEs who are charging $500/hr can charge $500/hr due to significant domain knowledge or overall experience, and as such their opinions and work are _worth_ that amount per hour to the market. SWEs who are charging $9/hr are either selling themselves extremely short, lack experience, or operate in an organization that doesn't allow them to negotiate their rate (in which case they should move to a new organization) and thus most likely generate lower quality results. To imply there's no relationship between the hourly rate and the quality of work is to pretend that there's no underlying reason why certain SWEs can charge premium hourly rates.
- zero_shift 3y agoIf xhe is that good and contracting for $15/hr, then xhe is extremely foolish. There are great engineers in non-Western countries. It's just that they realise talent is so rare, they can charge more than local rates.
- nottorp 3y ago> The Max became Boeing’s top seller soon after it was offered in 2011. Seriously? I mean, could you still order anything but a Max from Boeing in that size? And that being the most useful size, of course it became the top seller. Because it was the only offering.
- ilrwbwrkhv 3y agoMan these Indian bpo companies like Wipro Infosys HCl destroyed the future of India and gave the world the worst software ever.
- tremorscript 3y agoI am not surprised. When in doubt and when you have screwed things so badly, blame the people who don't look or speak or earn like you. Boeing did have $9 software engineers working for them. They were working on the accounting and ERP softwares. But as soon as 700 people lost their lives, apparently they were working on MCAS, a core system that only NASA engineers should have worked. I dont believe anything that Boeing says or does, the company is pure evil. If it's Boeing, I'm not going.
- rramadass 3y agoIt is absolutely a Boeing failure and not due to subcontractors/outsourcing as the clickbait title seems to imply. If you parcel out parts of your job in something as critical as Airplane Construction, it is your responsibility to validate all specifications/testing/verification/etc. If you treat "Airplane Engineering" as mere "Embedded Engineering" you need to be put out of business. The article explicitly says MCAS and "cockpit warning light not working" problems (which were the cause of the two crashes) were NOT outsourced. Key points; “Boeing was doing all kinds of things, everything you can imagine, to reduce cost, including moving work from Puget Sound, because we’d become very expensive here,” said Rick Ludtke, a former Boeing flight controls engineer laid off in 2017. “All that’s very understandable if you think of it from a business perspective. Slowly over time it appears that’s eroded the ability for Puget Sound designers to design. Rabin, the former software engineer, recalled one manager saying at an all-hands meeting that Boeing didn’t need senior engineers because its products were mature. “I was shocked that in a room full of a couple hundred mostly senior engineers we were being told that we weren’t needed,” said Rabin, who was laid off in 2015. “Engineering started becoming a commodity,” said Vance Hilderman, who co-founded a company called TekSci that supplied aerospace contract engineers and began losing work to overseas competitors in the early 2000s. The last point is the most damning; Airplane Engineering cannot be commodity engineering.
- RobertSchoenb 3y agoNo disagreements on your main points, but I would dispute calling that headline clickbait. We published this story in 2019 when software issues were the central issues at Boeing, so establishing how the company was coding was very newsworthy. I typically associate clickbait with bait-and-switch tactics, tricking readers into reading content unrelated to the headline. This article provides material directly related to its title. Robert Schoenberger, editor-in-chief, IndustryWeek
- rramadass 3y agoGlad to see that news websites (in this era of fake news) are willing to stand behind their stories; Much appreciated. While i understand your point, the headline gives the impression that Boeing's problems stem from outsourcing which then emboldens the racists to come out of the woodwork (as can be seen from some of the downvoted comments here) and start manipulating the conversations in a different and unproductive direction. Some of the folks commenting didn't even read the article before dunking on the outsourced country/engineers. It is highly frustrating when this sort of thing happens and hence my warning in the first line of my response to ignore the title and read the article. It would be great if you folks (maybe in collaboration with others sources/folks) could collate all the available info. on the disaster that is now "Boeing Engineering" (HW/SW) and publish a point-by-point synopsis of everything known until now. This would be a great learning lesson to the entire Industry on what not to do in pursuit of mere profits.
- reboot_boom 3y agoI think the big question is how did no FAA or whatever inspection show all those safety implications until now? What are they missing at other Aircraft manufacturers?
- lavezzi 3y agoWhat other aircraft manufacturers? Cessna?