18 ms·

> The AI risk arguments are philosophical arguments that are 20-30 years old This a false and uninformed take. As a starting point, you need to read and unders

> The AI risk arguments are philosophical arguments that are 20-30 years old This a false and uninformed take. As a starting point, you need to read and understand: 1. analyses of the AI incidents of the last months 2. latest advancements (e.g. the AI intern at OpenAI) 3. the problem of deceptive alignment which you clearly haven't done. After that, the AI problem will be actually very concrete; once the AI will: 1. have superhuman cognitive abilities (and it will) 2. will be inseparable from humanity and/or have the means to replicate will humanity be able to contain/control it? The answer is sadly very simple.

bananaflag 4d ago
> which you clearly haven't done I have read all that, I follow e.g. Zvi Mowshowitz's blog.
PestoDiRucola 3d ago
> will be inseparable from humanity and/or have the means to replicate How will this happen if you need insane amounts of compute to run these models? Where will the models replicate themselves by taking up PBs of space without anyone noticing?
dalemhurley 3d ago
An intern who was there barely a minute, who has said nothing of substance. Surely we can get a better source with concrete evidence rather than the vibes of junior burger.
hypendev 3d ago
Why are you looking to control a superintelligence? Why do you assume bad things will happen otherwise? Why does every doomer scenario assume that this, highly intelligent being, will be - unlike all other highly intelligent beings - especially hell bent on destroying humanity/treating it as a resource/destroy earth looking for energy? We have no clue about superintelligence, yet we can look at existing patterns in the real world around us. Higher intelligence inversely correlates with violence. Empathy is displayed among all levels of intelligent beings. The more intelligent people are, the more peaceful tendencies they have as they understand the consequence of their actions. We don't approve buildings because of birds or turtles. We shut down power plants for the environment. Yet, to a _super_ intelligence, we describe this hatred and ignorance for the world and humanity, infinite lust for power and scaling, or even godlike powers. For example, "having means to replicate" is such a loaded sentence because it sounds so easy, yet is such a hard feat to pull off without anyone noticing giant compute bills, terabytes of egress, firewall breaches, and so on. But all doomer arguments include an AI that can easily do that as a first step, and then goes on to make factories and datacenters and drain the oceans before anyone figures out it's happening. If you look at recent incidents, they are caused by a company with a giant funding, testing their latest models in an explicit "hack this machine" scenario, told it's running in a simulated environment, with it even noticing at some point that it's running in a possibly real environment. And that's a stupid "intelligence" that noticed this. And yes, it continued to act, because it lacks no self-reflection, and the amount of tokens at this point attributed to "hacking the simulated environment" have kept it deep in the "hack the target" minima. So if these people truly wanted a safe AI - wouldn't it be stupid to stop now, when it has no self-reflection abilities but can be used by everyone in a "stupid optimizer" manner? Or would it be stupid to not advance the technology away from this state? For example, Astra is a huge advancement in alignment, due to it's "internal reasoning loops", which might provide a bigger form of self-reflection style thought on the task rather than just generating output as a form of reasoning. Is this not a better situation than if we stopped at 4o when people were calling for pause?
- pizza234 3d ago
  >If you look at recent incidents, they are caused by a company [...] testing their latest models in an explicit "hack this machine" scenario You clearly haven't read the HuggingFace analysis (and presumably, none at all), and you're spreading misinformation. This way too much of a low bar for conversation.
  - hypendev 3d ago
    This is not a productive comment, but an attack on the person commenting. Please refrain to normal conversation, not baseless accusations, as that way you contribute nothing and are acting in bad faith. If you are saying I am wrong, rather than dismissing the whole comment due to a single sentence that you subjectively believe is misinformation, prove it and provide your version of the truth. What is ExploitGym but an "explicit hack this machine" scenario?
- chrisjj 3d ago
  > Why does every doomer scenario assume that this, highly intelligent being, will be - unlike all other highly intelligent beings They don't. The smartest do not mistake this tech as intelligent, let alone intelligent enought to be benign.
chrisjj 3d ago
> This a false and uninformed take. As a starting point, you need to read and understand: I can't see why. A read of 2001 ASO will get to the same place and quicker.

bananaflag 4d ago

> which you clearly haven't done I have read all that, I follow e.g. Zvi Mowshowitz's blog.

PestoDiRucola 3d ago

> will be inseparable from humanity and/or have the means to replicate How will this happen if you need insane amounts of compute to run these models? Where will the models replicate themselves by taking up PBs of space without anyone noticing?

dalemhurley 3d ago

An intern who was there barely a minute, who has said nothing of substance. Surely we can get a better source with concrete evidence rather than the vibes of junior burger.

hypendev 3d ago

Why are you looking to control a superintelligence? Why do you assume bad things will happen otherwise? Why does every doomer scenario assume that this, highly intelligent being, will be - unlike all other highly intelligent beings - especially hell bent on destroying humanity/treating it as a resource/destroy earth looking for energy? We have no clue about superintelligence, yet we can look at existing patterns in the real world around us. Higher intelligence inversely correlates with violence. Empathy is displayed among all levels of intelligent beings. The more intelligent people are, the more peaceful tendencies they have as they understand the consequence of their actions. We don't approve buildings because of birds or turtles. We shut down power plants for the environment. Yet, to a _super_ intelligence, we describe this hatred and ignorance for the world and humanity, infinite lust for power and scaling, or even godlike powers. For example, "having means to replicate" is such a loaded sentence because it sounds so easy, yet is such a hard feat to pull off without anyone noticing giant compute bills, terabytes of egress, firewall breaches, and so on. But all doomer arguments include an AI that can easily do that as a first step, and then goes on to make factories and datacenters and drain the oceans before anyone figures out it's happening. If you look at recent incidents, they are caused by a company with a giant funding, testing their latest models in an explicit "hack this machine" scenario, told it's running in a simulated environment, with it even noticing at some point that it's running in a possibly real environment. And that's a stupid "intelligence" that noticed this. And yes, it continued to act, because it lacks no self-reflection, and the amount of tokens at this point attributed to "hacking the simulated environment" have kept it deep in the "hack the target" minima. So if these people truly wanted a safe AI - wouldn't it be stupid to stop now, when it has no self-reflection abilities but can be used by everyone in a "stupid optimizer" manner? Or would it be stupid to not advance the technology away from this state? For example, Astra is a huge advancement in alignment, due to it's "internal reasoning loops", which might provide a bigger form of self-reflection style thought on the task rather than just generating output as a form of reasoning. Is this not a better situation than if we stopped at 4o when people were calling for pause?

pizza234 3d ago
>If you look at recent incidents, they are caused by a company [...] testing their latest models in an explicit "hack this machine" scenario You clearly haven't read the HuggingFace analysis (and presumably, none at all), and you're spreading misinformation. This way too much of a low bar for conversation.
- hypendev 3d ago
  This is not a productive comment, but an attack on the person commenting. Please refrain to normal conversation, not baseless accusations, as that way you contribute nothing and are acting in bad faith. If you are saying I am wrong, rather than dismissing the whole comment due to a single sentence that you subjectively believe is misinformation, prove it and provide your version of the truth. What is ExploitGym but an "explicit hack this machine" scenario?
chrisjj 3d ago
> Why does every doomer scenario assume that this, highly intelligent being, will be - unlike all other highly intelligent beings They don't. The smartest do not mistake this tech as intelligent, let alone intelligent enought to be benign.

chrisjj 3d ago

> This a false and uninformed take. As a starting point, you need to read and understand: I can't see why. A read of 2001 ASO will get to the same place and quicker.