5 ms·

I strongly believe that we will see an incident akin to Therac-25 in the near future. With as many people running YOLO mode on their agents as there are, Claude

I strongly believe that we will see an incident akin to Therac-25 in the near future. With as many people running YOLO mode on their agents as there are, Claude or Gemini is going to be hooked up to some real hardware that will end up killing someone. Personally, I've found even the latest batch of agents fairly poor at embedded systems, and I shudder at the thought of giving them the keys to the kingdom to say... a radiation machine.

the-grump 1y ago
The 737 MAX MCAS debacle was one such failure, albeit involving a wider system failure and not purely software. Agreed on the future but I think we were headed there regardless.
- jonplackett 1y ago
  Yeah reading this reminded me a lot of MCAS. Though MCAS was intentionally implemented and intentionally kept secret.
Maxion 1y ago
> Personally, I've found even the latest batch of agents fairly poor at embedded systems I mean even simple crud web apps where the data models are more complex, and where the same data has multiple structures, the LLMs get confused after the second data transformation (at the most). E.g. You take in data with field created_at, store it as created_on, and send it out to another system as last_modified.
SCdF 1y ago
The Horizon (UK Royal Mail accounting software) incident killed multiple postmasters through suicide, and bankrupted and destroyed the lives of dozens or hundreds more. The core takeaway developers should have from Therac-25 is not that this happens just on "really important" software, but that all software is important, and all software can kill, and you need to always care.
- hahn-kev 1y ago
  From what I've read about that incident I don't know what the devs could have done. The company sure was a problem but also the laws basically saying a computer can't be wrong. No dev can solve that problem.
  - sim7c00 1y ago
    as you point out this was a messup on a lot of levels. its an interesting effect tho not to be dismissed. how your software works and how its perceived and trusted can impact people psychologically.
  - fuckaj 1y ago
    Given whole truth testimony?
    - deleted 1y ago
      [deleted]
  - V__ 1y ago
    > Engineers are legally obligated to report unsafe conduct, activities or behaviours of others that could pose a risk to the public or the environment. [1] If software "engineers" want to be taken seriously, then they should also have the obligation to report unsafe/broken software and refuse to ship unsafe/broken software. The developers are just as much to blame as the post office: > Fujitsu was aware that Horizon contained software bugs as early as 1999 [2] [1] https://engineerscanada.ca/news-and-events/news/the-duty-to-report-as-an-engineer https://engineerscanada.ca/news-and-events/news/the-duty-to-... [2] https://en.wikipedia.org/wiki/British_Post_Office_scandal https://en.wikipedia.org/wiki/British_Post_Office_scandal
    - simulator5g 1y ago
      I don't think it's fair to blame individual developers for a systemic failure. Its not their fault there is no governing body to award or remove the title of "software engineer" and promote the concept of a software engineer refusing to do something without harming their career. Other engineering disciplines have laws, lobbied for by their governing body, that protect the ability of individual engineers to prevent higher-ups from making grave mistakes.
sim7c00 1y ago
talk to anyone in the industries about 'automation' on medical or critical infra devices and they will tell you NO. No touching our devices with your rubbish. i am pretty confident they wont let claude touch if it they dont even let deterministic automations run... that being said, maybe there are places. but this is always the sentiment i got. no automating, no scanning, no patching. device is delivered certified and any modifications will invalidate that. any changes need to be validated and certified. its a different world that makin apps thats for sure. not to say mistakes arent made and change doesnt happen, but i dont think people designing medical devices will be going yolo mode on their dev cycle anytime soon... give the folks in safety critical system engineering some credit..
- throwaway0261 1y ago
  > but i dont think people designing medical devices will be going yolo mode on their dev cycle anytime soon I don't have the same faith in corporate leadership as you, at least not when they see potentially huge savings by firing some of the expensive developers and using AI to write more of the code.
grues-dinner 1y ago
Non-agentic AI is already "killing" people by some definitions. There's a post about someone being talked into suicide on the front page right now, and they are 100% going to get used for something like health insurance and benefits where avoidable death is a very possible outcome. Self-driving cars are also full of "AI" and definitely have killed people already. Which is not to say that software hasn't killed people before (Horizon, Boeing, probably loads of industrial accidents and indirect process control failures leading to dangerous products, etc, etc). Hell, there's a suspicion that austerity is at least partly predicated on a buggy Excel spreadsheet, and with about 200k excess deaths in a decade (a decade not including Covid) in one country, even a small fraction of those being laid at the door of software is a lot of Theracs. AI will probably often skate away from responsibility in the same way that Horizon does: by being far enough removed and with enough murky causality that they can say "well, sure, it was a bug, but them killing themselves isn't our fault" I also find AI copilot things do not work well with embedded software. Again, people YOLOing embedded isn't new, but it might be about to get worse.
throwawayoldie 1y ago
They killed "only" about 350 people combined, but the two fatal crashes of the Boeing 737 MAX in 2018 and 2019 were due to poor quality software: https://en.wikipedia.org/wiki/Maneuvering_Characteristics_Augmentation_System https://en.wikipedia.org/wiki/Maneuvering_Characteristics_Au...

the-grump 1y ago

The 737 MAX MCAS debacle was one such failure, albeit involving a wider system failure and not purely software. Agreed on the future but I think we were headed there regardless.

jonplackett 1y ago
Yeah reading this reminded me a lot of MCAS. Though MCAS was intentionally implemented and intentionally kept secret.

Maxion 1y ago

> Personally, I've found even the latest batch of agents fairly poor at embedded systems I mean even simple crud web apps where the data models are more complex, and where the same data has multiple structures, the LLMs get confused after the second data transformation (at the most). E.g. You take in data with field created_at, store it as created_on, and send it out to another system as last_modified.

SCdF 1y ago

The Horizon (UK Royal Mail accounting software) incident killed multiple postmasters through suicide, and bankrupted and destroyed the lives of dozens or hundreds more. The core takeaway developers should have from Therac-25 is not that this happens just on "really important" software, but that all software is important, and all software can kill, and you need to always care.

hahn-kev 1y ago
From what I've read about that incident I don't know what the devs could have done. The company sure was a problem but also the laws basically saying a computer can't be wrong. No dev can solve that problem.
- sim7c00 1y ago
  as you point out this was a messup on a lot of levels. its an interesting effect tho not to be dismissed. how your software works and how its perceived and trusted can impact people psychologically.
- fuckaj 1y ago
  Given whole truth testimony?
  - deleted 1y ago
    [deleted]
- V__ 1y ago
  > Engineers are legally obligated to report unsafe conduct, activities or behaviours of others that could pose a risk to the public or the environment. [1] If software "engineers" want to be taken seriously, then they should also have the obligation to report unsafe/broken software and refuse to ship unsafe/broken software. The developers are just as much to blame as the post office: > Fujitsu was aware that Horizon contained software bugs as early as 1999 [2] [1] https://engineerscanada.ca/news-and-events/news/the-duty-to-report-as-an-engineer https://engineerscanada.ca/news-and-events/news/the-duty-to-... [2] https://en.wikipedia.org/wiki/British_Post_Office_scandal https://en.wikipedia.org/wiki/British_Post_Office_scandal
  - simulator5g 1y ago
    I don't think it's fair to blame individual developers for a systemic failure. Its not their fault there is no governing body to award or remove the title of "software engineer" and promote the concept of a software engineer refusing to do something without harming their career. Other engineering disciplines have laws, lobbied for by their governing body, that protect the ability of individual engineers to prevent higher-ups from making grave mistakes.

sim7c00 1y ago

talk to anyone in the industries about 'automation' on medical or critical infra devices and they will tell you NO. No touching our devices with your rubbish. i am pretty confident they wont let claude touch if it they dont even let deterministic automations run... that being said, maybe there are places. but this is always the sentiment i got. no automating, no scanning, no patching. device is delivered certified and any modifications will invalidate that. any changes need to be validated and certified. its a different world that makin apps thats for sure. not to say mistakes arent made and change doesnt happen, but i dont think people designing medical devices will be going yolo mode on their dev cycle anytime soon... give the folks in safety critical system engineering some credit..

throwaway0261 1y ago
> but i dont think people designing medical devices will be going yolo mode on their dev cycle anytime soon I don't have the same faith in corporate leadership as you, at least not when they see potentially huge savings by firing some of the expensive developers and using AI to write more of the code.

grues-dinner 1y ago

Non-agentic AI is already "killing" people by some definitions. There's a post about someone being talked into suicide on the front page right now, and they are 100% going to get used for something like health insurance and benefits where avoidable death is a very possible outcome. Self-driving cars are also full of "AI" and definitely have killed people already. Which is not to say that software hasn't killed people before (Horizon, Boeing, probably loads of industrial accidents and indirect process control failures leading to dangerous products, etc, etc). Hell, there's a suspicion that austerity is at least partly predicated on a buggy Excel spreadsheet, and with about 200k excess deaths in a decade (a decade not including Covid) in one country, even a small fraction of those being laid at the door of software is a lot of Theracs. AI will probably often skate away from responsibility in the same way that Horizon does: by being far enough removed and with enough murky causality that they can say "well, sure, it was a bug, but them killing themselves isn't our fault" I also find AI copilot things do not work well with embedded software. Again, people YOLOing embedded isn't new, but it might be about to get worse.

throwawayoldie 1y ago

They killed "only" about 350 people combined, but the two fatal crashes of the Boeing 737 MAX in 2018 and 2019 were due to poor quality software: https://en.wikipedia.org/wiki/Maneuvering_Characteristics_Augmentation_System https://en.wikipedia.org/wiki/Maneuvering_Characteristics_Au...