8 ms·
Unlike most other commenters, I applaud him for acting on his principles. If you sincerely believe that, of course you should act. You might not succeed, but yo
by matherial 8d ago
Unlike most other commenters, I applaud him for acting on his principles. If you sincerely believe that, of course you should act. You might not succeed, but your voice might be the one that tips the scales and starts a broader movement.
This doesn't mean I agree with him. The fears of doomsday caused by rapid takeoff have been with us since day 1 and the mechanism is always basically "AI invents magic that sets it free of any physical constraints". Self-replicating sentient nanobots or something like that. I think there's plenty to be worried about with AI, but runaway scenarios are pretty low on my list.
- vigilantpuma 7d agoYes. It gets a little tiresome to read all these people prophesying doom and then turning around and continuing to work on the doom. If you genuinely think you are building the mechanism of the destruction of our species, the only rational thing to do is to do everything to stop that progress, not continue to work on it so you can buy a mansion in San Francisco (which will not be very useful when AI destroys humanity in four years).
- matthewfcarlson 8d agoAgreed. Granted I just read the Reverse Centaur book, so I’m still coming off that skeptical viewpoint but it’s hard not to see this as hype. But I will always respect someone for doing what they think is right.
- kennywinker 8d agoNow is a great time to watch Colossus: The Forbin Project.
- copperx 8d agoTo borrow on the 1990s Slashdot meme: 1. Invent transformer architecture. 2. Scale it up. 3. ??? 4. Machines become sentient and kill us all. OpenAI and Anthropic pinky promise that they have figured out #3 and they're not BSing just to get more funding, no. But because we live in a culture of fear, everyone eats it up no questions asked.
- dwaltrip 7d ago#3 is actually decently well mapped out. You just don't find it plausible or misunderstand it. I'd appreciate if you wrote your actual arguments against it. At lower capability levels, the patterns are very clear and have been studied to death. E.g. Why LLMs say they have correctly fixed a broken test when they haven't. What we saw with Hugging Face is literally the exact same problem, just scaled up and with more capable agents. This shit was predicted decades ago... No one can say exactly how it will play out as the complexity increases, but the risks are becoming extremely obvious. I personally think it's extremely unlikely to "kill everyone", but there are many outcomes far short of that which seem quite plausible and rather undesirable. Russian roulette is not a smart game. If you read the METR report and aren't scared at all, then I'd love to know why. It would help me sleep better. So please share. TL;DR; increasing capabilities, reward hacking, and unsafe training regimes.
- slicktux 8d agoWhy are we putting so much weight (no pun intended) on AI companies. At the end of the day the scaled up LLM transformers lack emotion and will… They do as they are told; or more correctly put. They do as they are programmed to do so.
- whateveracct 8d agothe AI-pilled exec at my job already (a few weeks ago) declared out of nowhere that we are in the rapid takeoff scenario lol. he must have gotten high on twitter kool-aid and posted on company slack to self-soothe.
- sssilver 8d agoI wonder whether he's vested any options, and whether he's exercised them.
- matsemann 8d agoWhy? I've never understood the sentiment that if you stand up for something you have to forego everything and not partake in society. "Oh, you want to stop climate change? But I saw you breathe co2 yesterday"
- gizmodo59 8d agoThis is a lame argument. Questioning his financial incentive is very legit. Why should we trust him?
- frabcus 8d agoQuestioning his finances is a poor argument because he clearly would have more money if he had stayed at Anthropic for the next couple of years, than he will have by leaving. Even if he still has vested options, or savings from his salary or whatever, those would be larger if he stayed.
- gizmodo59 7d agoThe fame he got, the stamp of a frontier lab in the resume - its worth more than working in many other companies for years. Plus, not everyone thinks working more is needed once they achieve a certain number.
- ungovernableCat 7d agoI think it’s a valid question to ask if he quit the company for moral reasons. If you seriously think the frontier labs are dangerously playing with everyone’s lives (how many companies can even claim this civilisational scale) then why would you be ok with holding vested shares which will grow in value if the corporation achieves what it aims to achieve? At the very least you can sell them (they’re not so illiquid considering the company’s hype and success) and invest in the broad market.
- khafra 8d agoI appreciate your ability to separate sharing the belief itself from approval of acting on sincerely-held principle. However, I think the danger is much more plausible than you do. First, and least important, consider that self-replicating, solar-powered factories aren't magic; they're algae. Second, and more important, consider this fully non-magic route to doom: - We continue putting AI in charge of more things - It continues to get more capable, more eval-aware, and more prone to doing odd things, in service of goals that humans didn't intend to inculcate in it - Eventually, enough of the economy depends on it that we couldn't turn it off, any more than we could turn off the faber-bosch process or cargo shipping - AIs start doing something we can't survive, but less acutely than we couldn't survive turning them off. Everything else we try seems to work at first, but quickly loses effect - Game over
- vigilantpuma 7d agoBut we CAN survive turning off both the Faber-Bosch process and cargo shipping. Our numbers would be diminished, we would lead poorer, much harder lives. But no one would claim that getting rid of modern fertilizers would be the end of the species. But the claim you’re making is that we won’t be able to stop AI from destroying our species because turning it off will destroy our species?
- khafra 7d agoThe species would survive the end of faber-Bosch, yes; in some diminished form. But the species would not survive, if for some reason we needed to voluntarily stop using faber-Bosch in order to do so—it would take some kind of supernatural event to convince everyone, and even then some countries would probably keep doing it.
- artemisart 8d ago> First, and least important, consider that self-replicating, solar-powered factories aren't magic; they're algae. But what is that supposed to mean? Because humanity is not facing existential threat from algae.
- 8d ago
- asdff 8d ago>"AI invents magic that sets it free of any physical constraints". That's not what I am worried about at all. I'm worried one of the 79 year old toddlers we have these days in charge of some powerful nuclear armed country says "gee, this ai says I should attack right now, boy is it smart, glad I bought the stock ahead of contracting the government with this company I can scarcely understand!"
- BlueTemplar 7d agoAnd they will be stopped by someone sane in the chain to actually fire these missiles. (Like it already happened several times.) The danger is upfront : if there is a group of people insane enough to set up a system where there is no such chain. (As lampooned in Kubrick's Dr Strangelove (1964), in particular on the Soviet side.)
- palmotea 8d ago> I'm worried one of the 79 year old toddlers we have these days in charge of some powerful nuclear armed country says "gee, this ai says I should attack right now, boy is it smart, glad I bought the stock ahead of contracting the government with this company I can scarcely understand!" That's not what I am worried about at all. I'm worried about the 40-60 year old businessmen wrecking the prosperity and security of millions while chasing higher investment returns, because they've finally been freed of many of the technological constraints that kept those impulses in check.
- matherial 8d agoAs I noted, there's plenty of things to worry about, and your example fits that mold. But it has nothing to do with the "AI will kill us all in 10 years" claim of the rapid self-improvement doomer crowd.
- mofeien 8d agoHow about sandbox escape + cyber security collapse + 50 (or 500) deadly and highly contagious novel pathogens with long incubation period that humans can't possibly roll out vaccines for simultaneously. At least the first two should seem like a near-term worry after the past five months.
- tyre 8d agoThere is simply too much money in it for almost every person at these companies to stop. Leaving OAI or A\ would cost people millions, tens of millions, or more. And for what? So someone else can take your seat and do the same thing anyway? If you're smart enough to get a job there, you're smart enough to be able to talk yourself into why it makes sense for you to stay. Huge kudos to people like this who make the hard choice against the easy way out.
- andriy_koval 8d agoThey have some internal market where he likely sold his shares and now is very rich.
- fergie 8d agoIt should be obvious that AI is already capable of inducing humans to think or do things, and that this power is only going to increase over time.
- nairboon 8d agoIt's not that much different from the effect of social media or the power held by other tech giants, who can control your exposure to particular information (media, search engines).