10 ms·

I cannot fathom how you can intensely work on a problem that you genuinely believe has a "10 to 25%" probability of causing immense harm or even present an exti

I cannot fathom how you can intensely work on a problem that you genuinely believe has a "10 to 25%" probability of causing immense harm or even present an extinction-level risk to humanity. How can you truly believe this and be ok with it?

mchinen 4d ago
I posted this comment on the last doom post: The early batch of Anthropic employees were mostly rationalist-adjacent AI safety folk that were almost uniformly claiming P_DOOM > .10 three years ago, so I believe them to be earnest. It's very interesting to me that besides the other small safety labs that don't actually produce frontier models, Anthropic manages to keep such a good reputation within that subculture compared to OpenAI. Despite having as crazy internal politics as OpenAI, they have converged quite a bit from the original vision of safety first through Darwinistic pressures. At least, it seems this way from the outside. I'm curious if the view from the inside is that different. edit: to be clear, my reading as an outsider is that Anthropic is seen as relatively better in the AI safety community, but has definitely dropped in absolute reputation too. This recent thread and the references show some of that: https://www.lesswrong.com/posts/6j3kBHdowGLCeqobg/dear-god-p https://www.lesswrong.com/posts/6j3kBHdowGLCeqobg/dear-god-p...
- moreati 4d ago
  > they have converged quite a bit from the original vision of safety first do you mean diverged? As in they've moved away from the original vision.
  - mchinen 4d ago
    Ah, I mean the AI companies converged to look more similar to each other, and at the same time diverged from the original vision for safety.
throwuxiytayq 4d ago
Do you genuinely believe people always act logically? I don't know a single person that does. Don't underestimate the human capability to hold several completely contradictory beliefs with zero self-reflection. There's physicists who believe in god, for fuck's sake. > How can you truly believe this and be ok with it? Are you saying they don't truly believe this, or that they aren't OK with it?
hdgvhicv 4d ago
If you believe you working will mean a lower probability than you quitting.
baq 4d ago
The line of thought goes like this: - p(doom) is 1 - but if we build it, it's only 0.25 - if it realizes, we've got a front row seat
swiftcoder 4d ago
The Manhattan Project succeeded, so clearly some people are ok working on this sort of thing. And with similar justifications: if we don't build the potentially world-ending bomb, someone else might beat us to it.
againstapples 4d ago
There must have been thousands of people who built or did work related to nuclear weapons in the Cold War, people are just really good at rationalising away probabilistic dangers with unclear consequences, especially when there’s a corresponding upside. The same is true for climate change, drug addiction, health problems etc. I also think some of them have become delusional and convinced themselves that AI is going to create some kind of transhumanist utopia, even Dario Armodei leans in this direction from time to time. I imagine the people in these labs spend much of their day talking to sycophantic AI models that will encourage their delusional ideas.
falserum 4d ago
My personal opinion, this doom-mongering about LLMs, is part of marketing campaign to exaggerate LLM capabilities. (Play being: Sociopaths with money will buy or invest if they believe LLMs are powerful things)

mchinen 4d ago

I posted this comment on the last doom post: The early batch of Anthropic employees were mostly rationalist-adjacent AI safety folk that were almost uniformly claiming P_DOOM > .10 three years ago, so I believe them to be earnest. It's very interesting to me that besides the other small safety labs that don't actually produce frontier models, Anthropic manages to keep such a good reputation within that subculture compared to OpenAI. Despite having as crazy internal politics as OpenAI, they have converged quite a bit from the original vision of safety first through Darwinistic pressures. At least, it seems this way from the outside. I'm curious if the view from the inside is that different. edit: to be clear, my reading as an outsider is that Anthropic is seen as relatively better in the AI safety community, but has definitely dropped in absolute reputation too. This recent thread and the references show some of that: https://www.lesswrong.com/posts/6j3kBHdowGLCeqobg/dear-god-p https://www.lesswrong.com/posts/6j3kBHdowGLCeqobg/dear-god-p...

moreati 4d ago
> they have converged quite a bit from the original vision of safety first do you mean diverged? As in they've moved away from the original vision.
- mchinen 4d ago
  Ah, I mean the AI companies converged to look more similar to each other, and at the same time diverged from the original vision for safety.

throwuxiytayq 4d ago

Do you genuinely believe people always act logically? I don't know a single person that does. Don't underestimate the human capability to hold several completely contradictory beliefs with zero self-reflection. There's physicists who believe in god, for fuck's sake. > How can you truly believe this and be ok with it? Are you saying they don't truly believe this, or that they aren't OK with it?

hdgvhicv 4d ago

If you believe you working will mean a lower probability than you quitting.

baq 4d ago

The line of thought goes like this: - p(doom) is 1 - but if we build it, it's only 0.25 - if it realizes, we've got a front row seat

swiftcoder 4d ago

The Manhattan Project succeeded, so clearly some people are ok working on this sort of thing. And with similar justifications: if we don't build the potentially world-ending bomb, someone else might beat us to it.

againstapples 4d ago

There must have been thousands of people who built or did work related to nuclear weapons in the Cold War, people are just really good at rationalising away probabilistic dangers with unclear consequences, especially when there’s a corresponding upside. The same is true for climate change, drug addiction, health problems etc. I also think some of them have become delusional and convinced themselves that AI is going to create some kind of transhumanist utopia, even Dario Armodei leans in this direction from time to time. I imagine the people in these labs spend much of their day talking to sycophantic AI models that will encourage their delusional ideas.

falserum 4d ago

My personal opinion, this doom-mongering about LLMs, is part of marketing campaign to exaggerate LLM capabilities. (Play being: Sociopaths with money will buy or invest if they believe LLMs are powerful things)