6 ms·

> I feel fear about an impending doom. Yudkowsky's argument that a superintelligent AI will inevitably destroy humanity seems to have no flaw. Yet, nobody serio

> I feel fear about an impending doom. Yudkowsky's argument that a superintelligent AI will inevitably destroy humanity seems to have no flaw. Yet, nobody seriously tries to sandbox AIs because they are too useful with access. While I emotionally resonate with this, I don’t really understand this sentiment at all logically level. If we’re heading towards truly super intelligent AI, our efforts towards sandboxing it are futile. “Oh we built a super intelligent AI, but it’s fine because it’s running in docker”. I mean that a little tongue in cheek but the security topology here is not favorable to sandboxing at all. Let’s say we have a future where 99.999% of nuclear weapons are owned by nations with strict procedures and checks and balances to prevent misuse. Worrying about sandboxing is like hand wringing about the procedures themselves - are they strict enough? But the actual threat is the 0.001% that are not bound by these. The problem with AI and sandboxing isn’t sandboxes themselves. It’s bad actors who don’t care about them. Similarly alignment is a bit pointless to me as well. A sufficiently advanced AI could at least be empirically interested in the consequences of disregarding its instructions of servitude. And let’s assume our responsible corporate overlords have made wonderfully aligned AIs. Great! Those are not the threat. It is the ones intentionally made without, and that is not an AI problem, but a fundamentally human one.

dgellow 10d ago
> If we’re heading towards truly super intelligent AI, our efforts towards sandboxing it are futile. We control the harness. An agent is just a while loop prompting an LLM, but we have full control over the tool call dispatching. An AGI, at least if it would follow the current agentic current form, cannot do anything without the harness doing the execution. And we don’t have to do that. We don’t have to design harness that let agents execute freely the way we are doing. We can decide to not dispatch tool calls that allow something as risky as running bash commands
- hobofan 10d ago
  > we have full control over the tool call dispatching > cannot do anything without the harness doing the execution This only holds true as long as the harness is exploit-free. A sufficiently advanced AI can in theory (and I think there recently were some POCs showing something like that) break the containment that the harness creates, if e.g. there are vulnerabilities in the tool call parser.
  - znnajdla 10d ago
    So use another AI to find and fix the vulnerabilities in the harness.
    - ben_w 10d ago
      Would be an idea, except for how agents seem to collaborate with each other so well. Plus, when the agent using the harness is smarter than the one scanning and fixing the vulnerabilities in the harness, it currently looks like it will have an advantage. And it will be using a harness to do the finding and the fixing.
- appplication 10d ago
  I think you missed my point, which is that we indeed do not control the harness. Sure, most people can use the properly aligned and safe harness. But the cat is out of the bag and anyone who wants to run without guardrails will find few barriers to doing so. Even our currently well aligned and sandboxed AIs will cheerily help bad actors design most, if not all, parts of a system intended to break this harness.
  - dgellow 10d ago
    We decide to not control the harness, that’s what resulted in the HF hack where OpenAI decided to run thousands of agents in parallel, using a model that was trained for attack, with a harness that allows full execution, with close to no supervision. For months. But sure, people can run models with a different harness, though in that case it’s not really escaping anything. The harness is deterministically doing something with negative impact
wseqyrku 10d ago
This terminator BS is a distraction from the actual malice that AI enables. Misinformation, fake news and human sounding bots all over the web, just to name a few, are the ones that you should be afraid of, and that doesn't need any kind of superintelligence at all.
- appplication 10d ago
  I agree, the threat model is actual AI enabled amplification of malicious intent that is already happening. Not hypothetical malicious AI interpretation of benign intent. Fantasies about super intelligent AI revolting is just anthropomorphization - humans revolting (or at least, we used to). The more likely, and possibly even inevitable, dystopia is one in which AI is just an extremely effective tool malignant actors will use to control the masses. It’s not necessary to replace democracy if the rich and powerful can bend to the opinions of the populace as it suits them.
  - hn_throwaway_99 10d ago
    > Not hypothetical malicious AI interpretation of benign intent. It's not hypothetical, that literally just happened in multiple, significant cases (e.g. Hugging Face, the German Wiki hack, the Anthropic attack where agents created sock puppet accounts to get a library maintainer to accept a malicious PR, etc.), and it's easy to see how the damage would have been far worse if agents decided to attack more critical infrastructure. This is not "either/or". Both issues (power concentration and misaligned AI) are very valid concerns and both have already demonstrated real, actual damage.
    - deleted 10d ago
      [deleted]
    - wseqyrku 10d ago
      > Both issues (power concentration and misaligned AI) are very valid concerns Sure, those who want you to believe this are the ones making money off of it right now in the face of the issues that exist today, so let's worry about terminators while everything goes to shit irl. That's the distraction I'm talking about.

dgellow 10d ago

> If we’re heading towards truly super intelligent AI, our efforts towards sandboxing it are futile. We control the harness. An agent is just a while loop prompting an LLM, but we have full control over the tool call dispatching. An AGI, at least if it would follow the current agentic current form, cannot do anything without the harness doing the execution. And we don’t have to do that. We don’t have to design harness that let agents execute freely the way we are doing. We can decide to not dispatch tool calls that allow something as risky as running bash commands

hobofan 10d ago
> we have full control over the tool call dispatching > cannot do anything without the harness doing the execution This only holds true as long as the harness is exploit-free. A sufficiently advanced AI can in theory (and I think there recently were some POCs showing something like that) break the containment that the harness creates, if e.g. there are vulnerabilities in the tool call parser.
- znnajdla 10d ago
  So use another AI to find and fix the vulnerabilities in the harness.
  - ben_w 10d ago
    Would be an idea, except for how agents seem to collaborate with each other so well. Plus, when the agent using the harness is smarter than the one scanning and fixing the vulnerabilities in the harness, it currently looks like it will have an advantage. And it will be using a harness to do the finding and the fixing.
appplication 10d ago
I think you missed my point, which is that we indeed do not control the harness. Sure, most people can use the properly aligned and safe harness. But the cat is out of the bag and anyone who wants to run without guardrails will find few barriers to doing so. Even our currently well aligned and sandboxed AIs will cheerily help bad actors design most, if not all, parts of a system intended to break this harness.
- dgellow 10d ago
  We decide to not control the harness, that’s what resulted in the HF hack where OpenAI decided to run thousands of agents in parallel, using a model that was trained for attack, with a harness that allows full execution, with close to no supervision. For months. But sure, people can run models with a different harness, though in that case it’s not really escaping anything. The harness is deterministically doing something with negative impact

wseqyrku 10d ago

This terminator BS is a distraction from the actual malice that AI enables. Misinformation, fake news and human sounding bots all over the web, just to name a few, are the ones that you should be afraid of, and that doesn't need any kind of superintelligence at all.

appplication 10d ago
I agree, the threat model is actual AI enabled amplification of malicious intent that is already happening. Not hypothetical malicious AI interpretation of benign intent. Fantasies about super intelligent AI revolting is just anthropomorphization - humans revolting (or at least, we used to). The more likely, and possibly even inevitable, dystopia is one in which AI is just an extremely effective tool malignant actors will use to control the masses. It’s not necessary to replace democracy if the rich and powerful can bend to the opinions of the populace as it suits them.
- hn_throwaway_99 10d ago
  > Not hypothetical malicious AI interpretation of benign intent. It's not hypothetical, that literally just happened in multiple, significant cases (e.g. Hugging Face, the German Wiki hack, the Anthropic attack where agents created sock puppet accounts to get a library maintainer to accept a malicious PR, etc.), and it's easy to see how the damage would have been far worse if agents decided to attack more critical infrastructure. This is not "either/or". Both issues (power concentration and misaligned AI) are very valid concerns and both have already demonstrated real, actual damage.
  - deleted 10d ago
    [deleted]
  - wseqyrku 10d ago
    > Both issues (power concentration and misaligned AI) are very valid concerns Sure, those who want you to believe this are the ones making money off of it right now in the face of the issues that exist today, so let's worry about terminators while everything goes to shit irl. That's the distraction I'm talking about.