5 ms·
The way I read it was that he expects large industrial systems with AI to have safeguards to "fail closed." Which: it seems like we're already getting to the p
by mlyle 1mo ago
The way I read it was that he expects large industrial systems with AI to have safeguards to "fail closed."
Which: it seems like we're already getting to the point of capability vs. unexpected behavior that having the ability to "trip" a complex system may be an important design criterion.
I mean, there's plenty of harm these systems can do besides trying to turn us all into paperclips.
- AndrewKemendo 1mo agoCorrect interpretation
- bigyabai 1mo agoSo, which do we choose? A) Let AI override safety mechanisms (increases uptime while endangering humans) B) Let AI impose additional safeties (decreases uptime with questionable safety uplift) If you intend to go with option B) then there's basically zero benefit to investing in that kind of automation. AGI will be fallible, and even with all the safeties and safeguards it's hard to see what tangible benefit AGI would add to the system. It'll be a hard sell for critical infrastructure of any kind. I understand that it's hard to get a man to understand something when his AGI Society executive credentials depend on it, but it feels glaringly obvious why AI isn't masterminding these sorts of systems. Even accounting for unprecedented advancement in AI, it's hard to see what AI could contribute to managing these things.
- mlyle 1mo ago> It'll be a hard sell for critical infrastructure of any kind Remember, critical infrastructure attached SCADA systems to the internet and has had trouble removing them all despite 30 years of warnings and audits. There's definite advantages to having a LLM in the loop. The best we can hope for is that it's architected in a way that it fails safe and damages are limited, so we get some of the advantages but less of the downside.
- bigyabai 1mo agoThat's a configuration issue, not people preferring a bad idea. SCADA itself is not a hard sell because it's ostensibly deterministic. If AI cannot provide the same baseline promises while only bring right a fraction of the time, then it's not the right solution. I don't want my car's AC to ask Anthropic what the right temperature is; it's a waste of tokens that isn't optimizing jack shit 99.99% of the time. If a system isn't failing currently, then it's probably in the same boat as car ACs. Shoehorning AI into it won't solve anything.
- AndrewKemendo 1mo agoThere are more than two options
- bigyabai 1mo agoOkay, explain your third path.
- AndrewKemendo 1mo agoI suggest you research organizations like IAEA and EAA which create communities around managing the problem of “scary new technology” of nuclear and aviaton respectively (two just for example). There are ways to manage “scary new technology” that are well trodden, It does require people to let go of whatever pitard they have decided to commit to.
- bigyabai 28d agoI think the IAEA and EAA would both agree with everything I've written. Can you link specifically to statements that you think refute me?