15 ms·

I know people at METR and Anthropic. When they tell you they're worried the tech they're working on may kill everyone despite their best efforts, perhaps belie

by ben_w 4d ago

I know people at METR and Anthropic. When they tell you they're worried the tech they're working on may kill everyone despite their best efforts, perhaps believe them.

nullbio 4d ago
I'm sure it has nothing to do with their $500,000+ salaries and millions of dollars in equity. It's all solely because they're deeply concerned about next token prediction.
- phoghed 3d ago
  I think the “next token prediction” is too dismissive and reductive a framing of their capabilities at this point. Yes we all know that’s what they do, and guns just push a few grams of lead out of a pipe. It’s what you can do with that capability that is important. When you couldn’t count the R’s in strawberry it would have been a more effective statement. But a few short years later they are being used to solve millennium puzzles. What if the scaling continues? A model n years from now gets burned into silicon, a single company has millions of the chips, and in a few moments the system spend more time “thinking” than humans have ever spent thinking collectively? If it’s even possible I don’t think there’s anything we can do about it at this point. Cat’s out of the bag.
- azan_ 3d ago
  You really need to let your priors go if you still use this tired trope of next token prediction. It’s as useful for discussion as saying that human brain is made of fat, protein and carbohydrates - yeah that’s true, but it’s useless observation.
calgoo 4d ago
Or you know, stop working on it if its that dangerous? This whole thing of a bunch of employees saying that they are scared of building what they are building, but do it anyway because they are somehow going to make it different? Their model has been used in the planning of mass murdering in war as well as spying on the entire worlds population as well as helping ICE out in the US. They need to stop this BS fearmongering or actually stand up and do something about it. A government regulation is not the answer, especially when its done in a country that is run by a want to be dictator.
- int_19h 4d ago
  Anthropic specifically is basically saying that they believe it's even more dangerous if someone else gets to AGI before they do, so they have to either stop everyone or not stop themselves. I personally disagree with that take - and, as you note, it's hard to take seriously ethical wrangles from a company that literally sued the government in court to allow their models to be used by Palantir of all people. But if one genuinely believes that it's the robots themselves (rather than the people controlling the robots) that will kill us all, it's not inconsistent.
  - the_optimist 3d ago
    It’s a bit of a self-serving argument, don’t you think? And how does that relate to the ask for oligopoly licensing within global democracy? “We must build the nuclear bomb first in order to make sure no one else builds one.” This the most nonsense, disingenuous argument imaginable.
    - forgotone 3d ago
      The people at these companies are dumb as rocks. All “tech workers” are.
  - quickthrowman 3d ago
    > Anthropic specifically is basically saying that they believe it's even more dangerous if someone else gets to AGI before they do, so they have to either stop everyone or not stop themselves. That’s how people rationalize being a fentanyl dealer and selling a drug that can kill people, “Someone else will just sell them the drugs, might as well be me.”
  - folkrav 3d ago
    That's the Cold War nuclear arms race argument, not even disguised. The actual situation we all ended up in is both sides eventually having it, leading to a perpetual state of Mutually Assured Destruction. As you note, it may not be inconsistent with that they say they believe, but it's insanely inconsistent with what they actually are doing.
Tanjreeve 4d ago
Do you think they’re exaggerating (again) or do you think they should taken at face value and treated as hostile actors?
- ben_w 4d ago
  There's plenty of people who think greenhouse gas/global warming campaigns against fossil fuels are "exaggerating", that "earth was warm/the climate changed in the past", that a fee degrees isn't bad, that CO2 is good for plants. Are you likeminded? In this case, it's as if the oil and coal companies all said in the 60s and 70s "oh no, this research we did, it's all really bad; we need help to figure out how to transition away from this incredibly economically important input", rather than the observed reality where their entire PR campaign was approximately: there is no problem everything is fine and all critics are smelly hippies and/or communists; and/or hate the poor who are raised out of poverty by all the economic growth from the fossil fuel industry. What if the oil and coal companies were basically all pro nuclear, pro hyrdo, pro wind, pro solar, and believed in peak oil?
  - XRG 4d ago
    Here’s one crucial difference: there’s overwhelming evidence that human emissions have an effect on our climate. The mechanisms are generally well-understood and the research is widely disseminated and easily available to anyone that’s interested. With the ‘dangers’ touted by these insiders, it’s all “trust me bro”, hyperbole, and very little hard evidence. As such, a skeptical mind would question their motives.
    - ben_w 3d ago
      You're simultaneously overestimating what was observable in the 70s climate research, and ignoring all the actual research and evaluation test results for AI today. I don't expect people to be familiar with more than "trust me bro", but it's all right there for you to find with a search engine of choice. And, indeed, available for the LLMs themselves to explain to you in interrogative conversation.
schubidubiduba 4d ago
I'm sure the employees are (rightfully) worried. However, that does not at all preclude hidden motivations of the CEO behind acquiescing such worries.
- ben_w 3d ago
  Sure. I don't personally trust any of the CEOs, and I have yet to hear a single person (in general, not just on this topic) who trusts Altman.
owebmaster 4d ago
If that was the case they should all be in jail.
the_optimist 3d ago
It is an utterly silly premise that we should appoint insiders with permanent control. Megalomania is not evidence of either a problem or a solution.
- ben_w 3d ago
  The evidence is the acts the AI has already done.
  - the_optimist 3d ago
    All of the things it has done is just what people are already doing. Every single one. It helps, but there are already people doing this stuff. And we already have defenses against the existing attacks. We might have to defend better, but the real problem is “Who is setting the moral compass over generations.” One one hand, that will inherently fall to future generations. On the other, we are laying the groundwork. Imagine you use it to inflict trauma on your cruelest political enemy. Then next year they do that to you. That is war, and we already do it. But we don’t want people / governments in charge that are going to do this. God knows we have governments and individuals doing this historically, and this is probably the greatest source of historic instability. It might be the single best argument for open models — a unified frontier where no single exploit is going to represent capture. This is exactly the opposite of what Dario proposes.
quickthrowman 3d ago
If I really thought what my company was working on had a greater than 1% chance of ending human civilization I would feel obligated to destroy what my company was working on. Given they keep grinding away towards our alleged collective doom, I suspect it’s being overstated. Nobody knows what P(doom) actually is but I suspect it’s orders of magnitude closer to epsilon than 1. Recall Google’s Blake Lemoine who thought an old version of Gemini was sentient.
- ben_w 3d ago
  The people who think as you think indeed have left or never joined. The people still there necessarily think they can make a difference. I never applied to any of them because I didn't think I could make a difference. I currently have one idea that may help reduce risk; if I can turn that idea into research, I'll publish it for free for everyone. I don't expect it to be an important idea. > Recall Google’s Blake Lemoine who thought an old version of Gemini was sentient. Indeed. Current LLMs are sychopants boosting the users' own beliefs, I also think this causes researchers to have stronger beliefs than they had before. My own estimation happens to also be around this risk (0.1) over my lifetime, without using an LLM as a conversation partner in reaching this number. It is necessarily high-variance: we can't look at alternate realities. I base it on my expectation of how rapidly capabilities will increase the harm done when mistakes happen, vs. the chance that some instance of harm causes governments to change the law.
koverstreet 3d ago
"Trust us, really!" HAH
- ben_w 3d ago
  Rather the opposite.

by ben_w 4d ago

I know people at METR and Anthropic. When they tell you they're worried the tech they're working on may kill everyone despite their best efforts, perhaps believe them.

nullbio 4d ago

I'm sure it has nothing to do with their $500,000+ salaries and millions of dollars in equity. It's all solely because they're deeply concerned about next token prediction.

phoghed 3d ago
I think the “next token prediction” is too dismissive and reductive a framing of their capabilities at this point. Yes we all know that’s what they do, and guns just push a few grams of lead out of a pipe. It’s what you can do with that capability that is important. When you couldn’t count the R’s in strawberry it would have been a more effective statement. But a few short years later they are being used to solve millennium puzzles. What if the scaling continues? A model n years from now gets burned into silicon, a single company has millions of the chips, and in a few moments the system spend more time “thinking” than humans have ever spent thinking collectively? If it’s even possible I don’t think there’s anything we can do about it at this point. Cat’s out of the bag.
azan_ 3d ago
You really need to let your priors go if you still use this tired trope of next token prediction. It’s as useful for discussion as saying that human brain is made of fat, protein and carbohydrates - yeah that’s true, but it’s useless observation.

calgoo 4d ago

Or you know, stop working on it if its that dangerous? This whole thing of a bunch of employees saying that they are scared of building what they are building, but do it anyway because they are somehow going to make it different? Their model has been used in the planning of mass murdering in war as well as spying on the entire worlds population as well as helping ICE out in the US. They need to stop this BS fearmongering or actually stand up and do something about it. A government regulation is not the answer, especially when its done in a country that is run by a want to be dictator.

int_19h 4d ago
Anthropic specifically is basically saying that they believe it's even more dangerous if someone else gets to AGI before they do, so they have to either stop everyone or not stop themselves. I personally disagree with that take - and, as you note, it's hard to take seriously ethical wrangles from a company that literally sued the government in court to allow their models to be used by Palantir of all people. But if one genuinely believes that it's the robots themselves (rather than the people controlling the robots) that will kill us all, it's not inconsistent.
- the_optimist 3d ago
  It’s a bit of a self-serving argument, don’t you think? And how does that relate to the ask for oligopoly licensing within global democracy? “We must build the nuclear bomb first in order to make sure no one else builds one.” This the most nonsense, disingenuous argument imaginable.
  - forgotone 3d ago
    The people at these companies are dumb as rocks. All “tech workers” are.
- quickthrowman 3d ago
  > Anthropic specifically is basically saying that they believe it's even more dangerous if someone else gets to AGI before they do, so they have to either stop everyone or not stop themselves. That’s how people rationalize being a fentanyl dealer and selling a drug that can kill people, “Someone else will just sell them the drugs, might as well be me.”
- folkrav 3d ago
  That's the Cold War nuclear arms race argument, not even disguised. The actual situation we all ended up in is both sides eventually having it, leading to a perpetual state of Mutually Assured Destruction. As you note, it may not be inconsistent with that they say they believe, but it's insanely inconsistent with what they actually are doing.

Tanjreeve 4d ago

Do you think they’re exaggerating (again) or do you think they should taken at face value and treated as hostile actors?

ben_w 4d ago
There's plenty of people who think greenhouse gas/global warming campaigns against fossil fuels are "exaggerating", that "earth was warm/the climate changed in the past", that a fee degrees isn't bad, that CO2 is good for plants. Are you likeminded? In this case, it's as if the oil and coal companies all said in the 60s and 70s "oh no, this research we did, it's all really bad; we need help to figure out how to transition away from this incredibly economically important input", rather than the observed reality where their entire PR campaign was approximately: there is no problem everything is fine and all critics are smelly hippies and/or communists; and/or hate the poor who are raised out of poverty by all the economic growth from the fossil fuel industry. What if the oil and coal companies were basically all pro nuclear, pro hyrdo, pro wind, pro solar, and believed in peak oil?
- XRG 4d ago
  Here’s one crucial difference: there’s overwhelming evidence that human emissions have an effect on our climate. The mechanisms are generally well-understood and the research is widely disseminated and easily available to anyone that’s interested. With the ‘dangers’ touted by these insiders, it’s all “trust me bro”, hyperbole, and very little hard evidence. As such, a skeptical mind would question their motives.
  - ben_w 3d ago
    You're simultaneously overestimating what was observable in the 70s climate research, and ignoring all the actual research and evaluation test results for AI today. I don't expect people to be familiar with more than "trust me bro", but it's all right there for you to find with a search engine of choice. And, indeed, available for the LLMs themselves to explain to you in interrogative conversation.

schubidubiduba 4d ago

I'm sure the employees are (rightfully) worried. However, that does not at all preclude hidden motivations of the CEO behind acquiescing such worries.

ben_w 3d ago
Sure. I don't personally trust any of the CEOs, and I have yet to hear a single person (in general, not just on this topic) who trusts Altman.

owebmaster 4d ago

If that was the case they should all be in jail.

the_optimist 3d ago

It is an utterly silly premise that we should appoint insiders with permanent control. Megalomania is not evidence of either a problem or a solution.

ben_w 3d ago
The evidence is the acts the AI has already done.
- the_optimist 3d ago
  All of the things it has done is just what people are already doing. Every single one. It helps, but there are already people doing this stuff. And we already have defenses against the existing attacks. We might have to defend better, but the real problem is “Who is setting the moral compass over generations.” One one hand, that will inherently fall to future generations. On the other, we are laying the groundwork. Imagine you use it to inflict trauma on your cruelest political enemy. Then next year they do that to you. That is war, and we already do it. But we don’t want people / governments in charge that are going to do this. God knows we have governments and individuals doing this historically, and this is probably the greatest source of historic instability. It might be the single best argument for open models — a unified frontier where no single exploit is going to represent capture. This is exactly the opposite of what Dario proposes.

quickthrowman 3d ago

If I really thought what my company was working on had a greater than 1% chance of ending human civilization I would feel obligated to destroy what my company was working on. Given they keep grinding away towards our alleged collective doom, I suspect it’s being overstated. Nobody knows what P(doom) actually is but I suspect it’s orders of magnitude closer to epsilon than 1. Recall Google’s Blake Lemoine who thought an old version of Gemini was sentient.

ben_w 3d ago
The people who think as you think indeed have left or never joined. The people still there necessarily think they can make a difference. I never applied to any of them because I didn't think I could make a difference. I currently have one idea that may help reduce risk; if I can turn that idea into research, I'll publish it for free for everyone. I don't expect it to be an important idea. > Recall Google’s Blake Lemoine who thought an old version of Gemini was sentient. Indeed. Current LLMs are sychopants boosting the users' own beliefs, I also think this causes researchers to have stronger beliefs than they had before. My own estimation happens to also be around this risk (0.1) over my lifetime, without using an LLM as a conversation partner in reaching this number. It is necessarily high-variance: we can't look at alternate realities. I base it on my expectation of how rapidly capabilities will increase the harm done when mistakes happen, vs. the chance that some instance of harm causes governments to change the law.

koverstreet 3d ago

"Trust us, really!" HAH

ben_w 3d ago
Rather the opposite.