7 ms·
Gambling with our lives: AI researcher quits Anthropic with warning about safety
- feverzsj 10d agoIt's your fault to handle over your secrets to them.
- vbezhenar 10d agoYou can't blame a child for eating candies. We are children. There’s just no one to look after us.
- tilolebo 10d agoI thought the plan was sandboxes and markdown files to tell AI agents not to be bad. Is that not enough? /s
- vbezhenar 10d agoMore like sand castles...
- coffeebeqn 10d agoMake no mistakes. Kill no humans
- Bengalilol 10d ago"Has the whole world gone crazy? Am I the only one around here who gives a sh*t about the rules? Mark it zero!" walter_sobchak.md
- gen2brain 10d agoIt was written in the AGENTS.md, but Claude only read CLAUDE.md. That is how the man-vs-machine war started.
- glimshe 10d agoWhat do people feel about this in China? Even if their models are well behind, they are not years behind. If we restrain US companies, assuming that is desirable, it would do nothing to deter China's and AI-pocalypse would come anyway in short notice.
- margorczynski 10d agoThere would need to be some global agreement to stop it with maybe even a nuclear attack as a consequence of breaking the pact. From what we're seeing recently and all the thinking that went into analyzing AI it seems we do not have any effective way of controlling it and the whole "aligment" thing that AI labs are doing is just a sham. Maybe it is time to ask ourselves "should we?" instead of just "can we?".
- Jackpillar 10d ago>There would need to be some global agreement to stop it with maybe even a nuclear attack as a consequence of breaking the pact. Do you know how the world works?
- asp_hornet 10d ago> a nuclear attack as a consequence of breaking the pact That escalated quickly
- Bengalilol 10d agoUnrestricted models, running entirely locally and accessible to anyone: that’s what we should be taking as our baseline assumption. Everything else is just administrative distraction.
- not-kinsale-joe 10d agoChina has a better track record of regulating their big tech than the USA.
- glimshe 10d agoTrue, by imprisoning their tech CEOs until they can unambiguously support the dictators.
- koolba 10d ago> Evan Hubinger, Anthropic's staff lead on keeping the technology aligned with human goals and values, backed up Coxon claims in a follow-up post of his own, though he didn't quit the company. > "Jacob is correct here — we really do earnestly believe AI could kill all humans," he said. > Hubinger estimated the chances of that happening to be higher than ten percent within the next decade, and added that there's no plan yet on how to keep AI aligned with human goals in the superintelligence scenario. 10% chance we kill everybody is a small price to pay for Motown remixes of classic 2pac songs.
- pjc50 10d agoThis is just a bizarre thing to say that you're working on technology with that high a downside potential. If you were saying that while running a biology lab, or building a nuclear reactor, people would be demanding your head on a spike. But by not quitting it's clear that he himself doesn't really believe it. Or rather, this shows the difference between "believe" (political) and "believe" (use as a basis for action). I'm reminded of a story of how Afghans supposedly listened to the BBC World Service despite considering it enemy propaganda because the weather reports were really useful.
- dotancohen 10d agoOr he believes that other labs might get there first, and he is working to counter that threat. This is the Manhattan Project again.
- pjc50 10d agoHow exactly does that work? The nuclear system of MAD relies on physical threat, lab A achieving ASI (artificial scary intelligence) does not prevent lab B achieving it. I would like everyone involved to be a lot clearer about their threat models, with plausible series of clearly linked steps, rather than just sounding like a Vernor Vinge novel.
- deleted 10d ago[deleted]
- virgildotcodes 10d agoAt least we’ll eventually have an entity other than ourselves to blame for our annihilation.
- dotancohen 10d agoNo, the AI is still our responsibility.
- pluc 10d agoIsn't it insane that even in the face of complete annihilation through one of our inventions, we go "that wasn't us"? We deserve that shit ten times over
- virgildotcodes 10d agoI agree. It won’t stop us from othering the source of our demise as quickly as we are able. There’s also the argument to be made that your child is their own person, and you only have so much responsibility/control over their actions. Of course we have a good chunk of the world screaming that this child is obviously destined to be a homicidal psychopath, but what is a parent to do?
- Bengalilol 10d ago> Both OpenAI and Anthropic have recently flagged incidents in which agents powered by their models went rogue I may be biased and somewhat off topic, but I see these incidents as some of the most significant of the past century. I genuinely don't understand why these companies aren't taking a smarter approach to them. The latest analyses have been, at best, laughable: identify the vulnerability, patch it, and move on. Only to repeat the same cycle without considering that there may be something far more serious at play. These are AI security experts, and this has been their way of "solving" these incidents. AI security experts ... Moreover, when Challenger exploded, the government launched a series of investigations into the incident, bringing in experts from across the field. And now, what has the government done? Nothing. Literally nothing, as if everything was fine and all under control. Seriously, I'm generally quite optimistic and I don't buy into this fatalistic narrative about our shared future. But I have to admit that sometimes I feel like I'm stranded on a planet of primates. Sorry for this rather unproductive rant.
- pjc50 10d agoInternet isn't real. People (well, public discourse) have got extremely bad at dealing with forseeable risks and their mitigation. You can see this in things like climate change and vaccination, but also in discussions around regular crime, food poisoning, industrial accidents, and so on. Nothing will improve until something explodes on live TV. And it has to be something important, which means it has to be in California or New York.
- jurgenburgen 10d agoUltimately these are unserious companies ran by unserious people. They don’t even have a business plan, why would they bother with some kind of sensible security policy?
- Uptrenda 10d agoThey don't really care. It's just a play to tell people what they want to hear while they chase the money.
- azan_ 10d agoThe US government is run by idiot with dementia. No wonder govt does not take any real action.
- WalterGR 10d ago“I resigned from Anthropic today” (twitter.com/hilbertspaess) https://news.ycombinator.com/item?id=49619227 https://news.ycombinator.com/item?id=49619227 564 points | 9 hours ago | 766 comments
- archerx 10d agoAn AI that generates text will never be scary to me. An autonomous AI with facial recognition on a flying drone with weapons (bombs/guns) with swarming capabilities will always be terrifying. I feel like we are ignoring the massive elephant in the room.
- pjc50 10d agoThe killer robots are expensive and dependent on physical supply chains. While text is sufficient to radicalize humans into attacks.
- archerx 10d agoUkraine and Iran are proving that is not true. A drone is cheap and the AI required to run it is no where near as demanding as running a massive LLM. The problem is they are too cheap, a drone that cost a couple thousand can wipe out millions of dollars of “defense”.
- localhoster 10d agoI honestly feel that all those big ai companies think AI will long term harm humanity, but not their ai. A classic "it will not happen to me"
- pluc 10d agoCool cool cool cool
- jongjong 10d agoI'm not worried about AI safety. People greatly overestimate the utility and capabilities of intelligence. I'm not afraid of intelligence, I'm afraid of idiocy.
- MrThoughtful 10d agoWhy would AI wipe us out? We have not wiped out apes, ants, and most other species. We even have discussions about how to actively save them from extinction.
- pjc50 10d agoEnder's game model, presumably: the AI helpfully assists a human to build a nuclear bomb / pandemic virus / autonomous killer robot swarm in their basement. But again, I would like people to be clearer about how the threat is supposed to work rather than just making SF references.
- dumberquestions 10d agoYet we have wiped thousands of species completely by accident, breed some for slaughter and consumption and trap some for entertainment.
- mofeien 10d ago[dead]
- LadyCailin 10d agoWell, hopefully we end up like them then, and not https://en.wikipedia.org/wiki/Category:Species_made_extinct_by_human_activities https://en.wikipedia.org/wiki/Category:Species_made_extinct_... or worse, https://en.wikipedia.org/wiki/Category:Species_made_extinct_by_deliberate_extirpation_efforts https://en.wikipedia.org/wiki/Category:Species_made_extinct_...
- -0_0- 10d agoIt's not exactly trained on a neutral set of data. Engagement algorithms have ensured that a good chunk of content on the web these days is inflammatory and skewed towards the extreme (in fact in the last few months a good portion of the world has actually deemed it illegal for kids to consume because this content is so damaging to a developing brain).
- qznc 10d agoThe classic thought experiment is the paper clip optimizer. Quoting Nick Bostrom: > Suppose we have an AI whose only goal is to make as many paper clips as possible. The AI will realize quickly that it would be much better if there were no humans because humans might decide to switch it off. Because if humans do so, there would be fewer paper clips. Also, human bodies contain a lot of atoms that could be made into paper clips. The future that the AI would be trying to gear towards would be one in which there were a lot of paper clips but no humans. https://www.huffpost.com/entry/artificial-intelligence-oxford_n_5689858 https://www.huffpost.com/entry/artificial-intelligence-oxfor...
- themgt 10d agoThe fundamental point I think is far too often confused is the difference between LLM and agentic system. An LLM can't do anything but generate tokens. You run your LLM in vLLM or whatever, and it generates output tokens based on your input tokens. That's it! Humans then build ~deterministic systems to take those tokens and do all sorts of things with the tokens, like take actions in the real world. And then we can feed the output of those actions back to the LLM, and generate more tokens. And then our systems can use the new tokens to take new actions in the real world. Humans want to blame "AI" for attacking HuggingFace or a German wiki or whatever, but: 1) LLM - can't take over german wiki because it just generates tokens 2) agentic system with internet access, a prompt telling it to attack stuff, running in a shared CI env so agents can whiteboard in artifactory None of 2 is "AI", its standard networking and Markdown and CI virtual machine, etc etc. There's no AI to be found. CPUs not GPUs, even. Just deterministic systems ultimately managed by humans. And a 10x more powerful system-1 can still just generate 10x "smarter" inert data. If humanity and human organizations collectively decide to yolo the tokens generated from system-1 into our deterministic system-2s, over which we have complete control, back to system-1s, in a yolo loop, in such a way we lose control and it ends humanity, well ... "The coin don't have no say. It's just you."
- smb06 9d ago“a prompt telling it to attack stuff” — that’s the uncontrolled or not fully understood part of the AI isn’t it? No human told the deterministic system in 2) to attack Hugging Face. The random token generator landed on a guidance for 2) that caused it while 2) in itself still remains a deterministic system.
- MrSkelter 9d agoThis is a pointless essay. The point is the person in the article believes systems can cause harm to humans. How we label those systems is irrelevant. You write as if you are a lawyer for an AI corp trying to avoid a judgement. It’s akin to saying Teslas FSD/Autopilot can’t kill anyone, it’s just a computer. Cars can kill people. Totally different things. FSD/Autopilot is safe by definition and no one should try to legislate it.
- a2ff6eeb0 10d ago
- brunorsini 10d agoI particularly like the last point he makes here: Do not underestimate the power of this technology. These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources. It's interesting to see so much hate towards creators who use AI to make almost any type of creative work. At least these are humans using it as "controllable tools". Nuclear-powered bicycles for the mind. As the degree of separation increases, things can get interesting. "Create several social media accounts, post whatever, maximize views and engagement, give me back the aggregate numbers". "Now promote <x>." And then decisions to do things like that may soon be happening autonomously, as just another step in a reasoning series aiming to achieve some other, broader goal. Open models/weights may end up playing particularly important roles here. Users may, knowingly or not, bypass system prompt-derived safety that could have offered much needed protection.
- RandomLensman 10d ago> Do not underestimate the power of this technology. These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources. Is the idea here that no field needs experiments and data anymore (which can take a lot of time) to be revolutinized and just "thinking" would be enough? How would an AI itself own anything?
- 568236at 10d ago[dead]
- brunorsini 10d agoIf AI is spawning content of its own, it's accumulating attention time. If they spawn crypto wallets they can accumulate "money", etc.
- ygjb 10d agoThe concept of ownership is synthetic and a legal construct. Laws are written by people, and with enough accumulated power, for which money and influence are proxies, the laws can be changed. There are already a significant number of people (myself included) that believe that 'Freedom is the right of all sentient beings' (don't blame me, blame the person who wrote it into the bio on the box of Optimus Prime when I was 7). If an AI can acquire enough power and influence then it can lobby for personhood and in the correct state that is absolutely achievable. If you think that is ridiculous remember that we already have fictional people called corporations in various degrees of personhood by state and country.
- cranx 10d agoAt this point I feel like the boy who cried wolf is an ai researcher. Yes super intelligence could be dangerous. However, is the story a PR stunt or real? Well…
- ralfd 10d agoHow is a researcher quitting a PR stunt?
- tripledry 10d agoAt this point I wouldn't be surprised if they said to him "I will give you a million and your job back after IPO if you resign". To be clear, I don't believe this is what happened here, I'm typing this half jokingly, just I wouldn't be surprised.
- cranx 10d agoBc “the ai so dangerous and powerful he had to quit for moral reasons” sends a signal to investors about capabilities. And it could totally be legit. It’s hard to say. I feel like once a researcher is truly ringing the alarm bells no one will look bc it’s been done so many times already. It’s unfortunate imho
- wartywhoa23 10d ago> "The people building AI earnestly believe that it could kill us all by the end of the decade," Coxon said in a follow-up post. AI has no incentive to decimate extra 7.5B mouths to feed, those who print money out of thin air to make an AI cover for that decimation have.
- strideashort 10d agoWhoever controls violence controls the world, as simple as that.
- jacknews 10d agoI bring you peace. It may be the peace of plenty, and content, or the peace of unburied death, the choice is yours.
- mixxit 10d agoFeels like a marketing stunt honestly
- sensitivekt9q3 10d agoAt this point frontier agents could kill half of humanity and people would still be spouting “crazy marketing stunt, they must be really desperate for that IPO.”
- negura 10d agoI really believe they would sink so low and pay him to quit and post this, just to generate a bit more hype before the IPO.
- conqueso 10d agoI find it difficult to imagine how AI could be an existential threat. Is the general idea that as LLMs improve they will be integrated into more systems where the risk of malfunction will become literally dangerous? e.g. controlling nuclear reactors
- devinprater 10d agoAwww, company made to scare people scares itself. Scared employee quits. Stock goes up cause we love telling each other scary stories.
- ChrisArchitect 10d ago[dupe] Discussion on source: https://news.ycombinator.com/item?id=49619227 https://news.ycombinator.com/item?id=49619227
- someguynamedq 10d agoHe's klout farming now that he's made his millions profiting off of it.