5 ms·

But there is still a difference here. Provenance and proper traceability would have allowed the subpostmasters to show their innocence and prove the system fail

by maweki 1y ago

But there is still a difference here. Provenance and proper traceability would have allowed the subpostmasters to show their innocence and prove the system failable. In the Therac-25 case, the killing was quite immediate and it would have happened even if the correct radiation dose was recorded.

scott_w 1y ago
I’m not sure it would. Remember that the prosecutors in this case were outright lying to the courts about the system! When you hit that point, it’s really hard to even get a clean audit trail out in the open any more!
SCdF 1y ago
I don't understand the distinction here. > Provenance and proper traceability would have allowed But there wasn't those things, so they couldn't, so they were driven to suicide. Bad software killed people. It being slow or fast doesn't seem to matter.
- maweki 1y ago
  Slow killing software can be made more secure by adding the possibility for human review. Fast killing software is too fast for that.
  - SCdF 1y ago
    I'm really trying to understand your point, but I am failing. It sounds like you're saying that you shouldn't care as much about the quality of "slow killing software" because in theory it can be made better in the future? But... it wasn't though? Horizon is a real software system that real developers like you and me built that really killed people. The absolutely terrible quality of it was known about. It was downplayed and covered up, including by the developers who were involved, not just the suits. I don't understand how a possible solution absolves the reality of what was built.
    - maweki 1y ago
      I teach the horizon post office scandal in my database courses. And my takeaway is, that software fails. And if people's lives are involved, an audit trail is paramount. In slowly killing software the audit trail might be faster than the killing. In fast killing software, the audit trail isn't.
      - SCdF 1y ago
        Yes, the audit trail that should exist is part of the package. Or more generically, Horizon should have had enough instrumentation, combined with adequate robustness, where they could detect the issues the lack of robustness caused, and resolve those issues without people dying. My core point is that if you're designing a system, *any system*, you should be thinking about what is required to produce safe software. It isn't just "well I don't work on medical devices that shoot radiation at people, so I don't need to worry"[1]. You still need to worry, you just solve those problems in different ways. It's not just deaths either, it's PII leakage, it's stalking and harassment enablement, it's privilege escalation, etc. [1] I have heard this, or a variation of this, from dozens of people over the my career. This is my core bug bear about Therac-25, is that it allows people to think this way, and divest themselves of responsibility. I am very happy to hear you are teaching a course about Horizon, because it's a much more grounded example that devs will hopefully see themselves in more. If your course is publicly available btw, I'd love to read it.
        maweki 1y ago
        It's just a course about database design and in the first seminar we look at different news stories that have something to do with databases, like trump putting some random Italian chef on an international sanction list should make us think about primary keys and identifying people. And the horizon post office scandal is the last and most poignant example that real people are affected by the systems we build and the design decisions we make. That sometimes easy to forget.

by maweki 1y ago

scott_w 1y ago

I’m not sure it would. Remember that the prosecutors in this case were outright lying to the courts about the system! When you hit that point, it’s really hard to even get a clean audit trail out in the open any more!

SCdF 1y ago

I don't understand the distinction here. > Provenance and proper traceability would have allowed But there wasn't those things, so they couldn't, so they were driven to suicide. Bad software killed people. It being slow or fast doesn't seem to matter.

maweki 1y ago
Slow killing software can be made more secure by adding the possibility for human review. Fast killing software is too fast for that.
- SCdF 1y ago
  I'm really trying to understand your point, but I am failing. It sounds like you're saying that you shouldn't care as much about the quality of "slow killing software" because in theory it can be made better in the future? But... it wasn't though? Horizon is a real software system that real developers like you and me built that really killed people. The absolutely terrible quality of it was known about. It was downplayed and covered up, including by the developers who were involved, not just the suits. I don't understand how a possible solution absolves the reality of what was built.
  - maweki 1y ago
    I teach the horizon post office scandal in my database courses. And my takeaway is, that software fails. And if people's lives are involved, an audit trail is paramount. In slowly killing software the audit trail might be faster than the killing. In fast killing software, the audit trail isn't.
    - SCdF 1y ago
      Yes, the audit trail that should exist is part of the package. Or more generically, Horizon should have had enough instrumentation, combined with adequate robustness, where they could detect the issues the lack of robustness caused, and resolve those issues without people dying. My core point is that if you're designing a system, *any system*, you should be thinking about what is required to produce safe software. It isn't just "well I don't work on medical devices that shoot radiation at people, so I don't need to worry"[1]. You still need to worry, you just solve those problems in different ways. It's not just deaths either, it's PII leakage, it's stalking and harassment enablement, it's privilege escalation, etc. [1] I have heard this, or a variation of this, from dozens of people over the my career. This is my core bug bear about Therac-25, is that it allows people to think this way, and divest themselves of responsibility. I am very happy to hear you are teaching a course about Horizon, because it's a much more grounded example that devs will hopefully see themselves in more. If your course is publicly available btw, I'd love to read it.
      - maweki 1y ago
        It's just a course about database design and in the first seminar we look at different news stories that have something to do with databases, like trump putting some random Italian chef on an international sanction list should make us think about primary keys and identifying people. And the horizon post office scandal is the last and most poignant example that real people are affected by the systems we build and the design decisions we make. That sometimes easy to forget.