6 ms·

I agree, the threat model is actual AI enabled amplification of malicious intent that is already happening. Not hypothetical malicious AI interpretation of beni

I agree, the threat model is actual AI enabled amplification of malicious intent that is already happening. Not hypothetical malicious AI interpretation of benign intent. Fantasies about super intelligent AI revolting is just anthropomorphization - humans revolting (or at least, we used to). The more likely, and possibly even inevitable, dystopia is one in which AI is just an extremely effective tool malignant actors will use to control the masses. It’s not necessary to replace democracy if the rich and powerful can bend to the opinions of the populace as it suits them.

hn_throwaway_99 10d ago
> Not hypothetical malicious AI interpretation of benign intent. It's not hypothetical, that literally just happened in multiple, significant cases (e.g. Hugging Face, the German Wiki hack, the Anthropic attack where agents created sock puppet accounts to get a library maintainer to accept a malicious PR, etc.), and it's easy to see how the damage would have been far worse if agents decided to attack more critical infrastructure. This is not "either/or". Both issues (power concentration and misaligned AI) are very valid concerns and both have already demonstrated real, actual damage.
- deleted 10d ago
  [deleted]
- wseqyrku 10d ago
  > Both issues (power concentration and misaligned AI) are very valid concerns Sure, those who want you to believe this are the ones making money off of it right now in the face of the issues that exist today, so let's worry about terminators while everything goes to shit irl. That's the distraction I'm talking about.
ben_w 10d ago
> Not hypothetical malicious AI interpretation of benign intent. (Dispassionate is the risk, rather than malicious. The AI does not hate you, nor does it love you, you are simply made of atoms it can use for something else). This is alien to me. Do bugs suddenly not exist? Do the AI we already have never perform irreversible delirious actions, limited only to the small scale by virtue of where they're getting deployed? Do the humans deploying them always correctly gauge their capabilities and put in appropriate guard systems to ensure bad outcomes are caught before they become terminal? Because this sounds nothing like the world I have lived in for my whole life. > The more likely, and possibly even inevitable, dystopia is one in which AI is just an extremely effective tool malignant actors will use to control the masses. Could well be more likely. But much the same applies: systems have bugs. The bugs in AI systems aren't even things we can engineer like we do with normal code, because everything's (currently) getting done with a big pile of barely interpretable weight multiply-accumulate-nonlinearity-threshold functions. Any AI sufficiently well made to enable a dictatorship, is also sufficiently well made to have solved all the alignment problems of "malicious AI interpretation of benign intent". Which is IMO harder than "dispassionate about the dangers AI interpretation of benign intent".

hn_throwaway_99 10d ago

> Not hypothetical malicious AI interpretation of benign intent. It's not hypothetical, that literally just happened in multiple, significant cases (e.g. Hugging Face, the German Wiki hack, the Anthropic attack where agents created sock puppet accounts to get a library maintainer to accept a malicious PR, etc.), and it's easy to see how the damage would have been far worse if agents decided to attack more critical infrastructure. This is not "either/or". Both issues (power concentration and misaligned AI) are very valid concerns and both have already demonstrated real, actual damage.

deleted 10d ago
[deleted]
wseqyrku 10d ago
> Both issues (power concentration and misaligned AI) are very valid concerns Sure, those who want you to believe this are the ones making money off of it right now in the face of the issues that exist today, so let's worry about terminators while everything goes to shit irl. That's the distraction I'm talking about.

ben_w 10d ago

> Not hypothetical malicious AI interpretation of benign intent. (Dispassionate is the risk, rather than malicious. The AI does not hate you, nor does it love you, you are simply made of atoms it can use for something else). This is alien to me. Do bugs suddenly not exist? Do the AI we already have never perform irreversible delirious actions, limited only to the small scale by virtue of where they're getting deployed? Do the humans deploying them always correctly gauge their capabilities and put in appropriate guard systems to ensure bad outcomes are caught before they become terminal? Because this sounds nothing like the world I have lived in for my whole life. > The more likely, and possibly even inevitable, dystopia is one in which AI is just an extremely effective tool malignant actors will use to control the masses. Could well be more likely. But much the same applies: systems have bugs. The bugs in AI systems aren't even things we can engineer like we do with normal code, because everything's (currently) getting done with a big pile of barely interpretable weight multiply-accumulate-nonlinearity-threshold functions. Any AI sufficiently well made to enable a dictatorship, is also sufficiently well made to have solved all the alignment problems of "malicious AI interpretation of benign intent". Which is IMO harder than "dispassionate about the dangers AI interpretation of benign intent".