7 ms·

The HN discourse on this subject is incredibly discouraging to me. It's completely nonconstructive. I take the pace of AI development seriously. All of my frie

• red_green_yell 2mo ago
  Are they lazy dismissals? Or a simple observation that LLM scaling is hitting diminishing returns way way short of AGI. If the claim is (1) AGI coming and (2) we should so something about AGI, wouldn't it be smart to confirm (1) before worrying about (2)? Especially if certain companies have trillions of dollars of valuation that depend on society as a whole believing (1)?
  • bpascal 2mo ago
    i think part of the problem, though, is that we've already hit many of the warning signs that AGI is coming, and people have kept moving the goalposts. if you were to tell someone from ten years that there's an ai that has passed the turing test and has made novel mathematical discoveries, i think many would say AGI is already here.
    • versteegen 2mo ago
      The old observation that people tend to define AI as whatever computers can't do yet is as true as ever. It's getting a bit absurd, moving from demanding "general intelligence" to replicating human cognitive phenomenology (the experience of cognitive activities). Yet LLMs can already somewhat (confabulation-prone) introspect their own "internal unspoken thoughts" in their residual streams, quite fascinating.
    • marcus_holmes 2mo ago
      You're right, in that we would have said that. I wouldn't frame it as moving the goalposts in a bad way, though. We understand more now than we did then, so we have a better idea of what the actual goal is, and moving the goalposts to frame that goal is more rational than insisting the goalposts stay where they were ten years ago even though the entire pitch has changed since then. I'm straining the analogy, obviously.
      • gr_norm 2mo ago
        Moving the goalposts, so to speak, is how science works! We must of course update the things we think based on an improved understanding of how the world works. Only the deeply incurious could consistently demand that one adhere dogmatically to a prior way of understanding the world in the face of changing knowledge. It's important not to treat science as a game to be won (of which 'goalposts' are evocative), but as the collaborative effort that it is. Note well that I have not said LLMs are not useful, only that no amount of interaction with them has convinced me they remotely approach general intelligence. Their actual function of heuristically regurgitating all information ever known to mankind remains extremely useful in lots of domains. And by the way, it's not obvious an LLM could fool me (or the average person) over a sufficiently long period of time, exactly because they are heuristic machines. That they lack thought guided by underlying cognitive models seems to always leak out, in the end. I suspect pass rates by LLMs on long-range Turing tests (if there have been any conducted) might drop as people become acclimatized to them.
        • bpascal 2mo ago
          what do you mean by 'general intelligence'? i definitely agree that in comparison to humans, they have some definite weaknesses. however, i think they also have definite strengths. and i don't think their current weaknesses imply that much about a potential singularity or ai risk.
          • gr_norm 2mo ago
            Sorry, I should've used clearer language. I just mean intelligence in the sense humans possess it. Computers have always been able to exceed limited elements of human intelligence (say, at arithmetic), but the problem is to simultaneously match or exceed all of them. I specifically think that the important parts they currently lack make them a no-go for supposed existential risk.
  • roywiggins 2mo ago
    What if "way short of AGI" is still dangerous in ways we can't easily handle?
    • overfeed 2mo ago
      When did the pacers have this mass-revelation about dangers of AI? The timing is curious. Which takes priority: public safety or the free-market + national security? The AI labs hype in now biting them in the ass: why would China and the rest of the world deny their businesses a technology that "is bigger than the invention of electricity" or whatever hyperbole the snake-oil salesmen came up with when they were not breathlessly declaring that the first side to get AGI wins forever. Even if the pacers are right, it's like greenhouse emissions all over again; each country wants everyone else to cut emissions, but no one wants the economic hit of doing so.
• dmarcos 2mo ago
  Every single technology in history has had people claiming potential existential risks that didn’t materialize. Burden of proof is on people making the risk claims. Those have to be specific and unique to that technology to grant special actions.
  • red_green_yell 2mo ago
    See printing press and internet!
    • dmarcos 2mo ago
      And video games, democracy, novels, TV, cars, photography, cryptography, the Beatles… pick your favorite and read commentary / opinions of the time. Many thought that came with unique risks never seen before and must be stopped!
      • aesthesia 2mo ago
        How many of those concerns came from the people building the technologies? John Carmack wasn't up in arms about how Doom was going to destroy society. Charles Dickens didn't think novels rotted the brain. Johannes Gutenberg wasn't concerned about the effects of a free press.
        • dmarcos 2mo ago
          None of the previous concerns were true and nothing indicates current ones are. It’s extrapolation, hypothetical apocalyptic imagined scenarios and piles of assumptions on how the technology could evolve. What makes even less credible in the AI case is that there’s financial / power to be gained by those making the claims. And to your point. Don Quixote, the first modern and greatest novel of all times, is about a guy that lost touch with reality from reading too many books
          • aesthesia 2mo ago
            I mean, Cervantes was parodying medieval romances. That's what Don Quixote was reading, not modern novels. The point is, it should give you at least some pause when the people most familiar with a technology are among those making the strongest statements about its potential dangers. Extrapolation has worked extremely well in predicting what AI will be capable of over the past 5 years; how confident are you that it will fail before we reach apocalyptic scenarios?
• rossant 2mo ago
  This is serious. We're in uncharted territory. The geopolitical context is already complicated enough that we shouldn't be dismissive of such a rapid technological race. I'm glad AI leaders are taking it seriously. I'm surprised so many HN commenters aren't. On the other hand, I'm pessimistic about initiatives like this being effective on a global scale (though they could certainly work at the national level).
  • nostrebored 2mo ago
    we're always in uncharted territory. that's why most HN commenters think that this is hyperbole.
• deleted 2mo ago
  [deleted]
• philipkglass 2mo ago
  I know a lot more about chemistry and physics than I know about biology, so biological risk is still at least a "maybe" reason to be worried about model safety for me. I don't worry about other weapons of mass destruction developed with LLM assistance. Nuclear weapons are already controlled much more with political pressure and the physical difficulty of e.g. enriching uranium than they are controlled by secret-keeping. For all I know (I don't know a lot about biology), it's possible to develop a virus that spreads as easily as measles, but has the lethality and long induction period of HIV. If that's true, and if LLMs would make it easy to develop such a virus, I see how no-guardrails LLMs could be truly disastrous. The other justifications for restricting model training don't seem nearly as compelling. Worse, restrictions on model training have an infectious quality if they mean something more than "leading AI labs will voluntarily slow down together." (Like if the signatories mean that better models and software for building them should be regulated.) It's too dangerous to publish stronger models. So it's also too dangerous to publish training software. So it's also too dangerous to publish tools that allow you to build training software. (Claude's Fable won't help you write generic model training pipelines for this very reason.) If you keep following the implications of this safety argument, it's as broad an assault on the distribution of software and computing as has ever been proposed.
  • fidotron 2mo ago
    > For all I know (I don't know a lot about biology), it's possible to develop a virus that spreads as easily as measles, but has the lethality and long induction period of HIV. If that's true, and if LLMs would make it easy to develop such a virus, I see how no-guardrails LLMs could be truly disastrous. This is the specific example in the latest doomerist wave I have so much trouble with: by what mechanism is an LLM going to help you design and manufacture a novel bioweapon? We know they cannot do spatial reasoning, and if they were this good at bio stuff then drug research would be absolutely out of control. Similarly, how are you supposed to magically get access to everything required to actually manufacture the offensive organism in question? Are we all doing CRISPR in our kitchens now? If you are that concerned about this threat then . . . monitor biolabs, not AI. It is so detached from the real state of things as to be bizarre.
• ubertaco 2mo ago
  We're commenting on a site run by a Silicon Valley tech start-up accelerator. "Move fast and break things" is the accelerationist way, even if the "things" they break are a stable, functioning social order. As long as you IPO and cash out along the way, you can wreck everyone else's lives just fine, in the minds of this sort.
• atleastoptimal 2mo ago
  So many otherwise smart engineers, overrepresented on HN, have this weird mental block where they fail to take AI progress seriously. I think it's a fear that their status-defining skill could be commodified, so they adopt a mental framework that aims to dismantle the seeming legitimacy of any genuine leap in AI progress as some sort of mirage
  • Davidzheng 2mo ago
    Not just engineers - I see people attributing it to strong priors that nothing happens but in my opinion it's more some form of psychological defense mechanism against understanding there will be an intelligence vastly superior to humans which is a deeply uncomfortable thought (to many).
    • dullcrisp 2mo ago
      “Nothing ever happens” is like “you can’t beat the market.” It’s basically tautologically the strongest predictive model that we can have. A corollary is that something may happen, but it won’t be what you think.
      • 0xDEAFBEAD 2mo ago
        Interesting fact. Bryan Caplan is a economist who became famous for making bets on the future. He would take the "nothing ever happens" side of the bet, and he would almost always win. In 2023 he took the "nothing ever happens" side of an AI bet. Just 6 months later, he wrote: "When the answers change, I change my mind [...] Base rates have clearly failed me. I’m not conceding the bet, because I still think there’s a 10-15% chance I win via luck... make no mistake, this software truly is the exception that proves the rule." https://www.betonit.ai/p/gpt-retakes-my-midterm-and-gets-an https://www.betonit.ai/p/gpt-retakes-my-midterm-and-gets-an At this point, straightforward trend extrapolation suggests that AI continues to get better and better. Arguably, continued rapid AI progress is the "nothing ever happens" bet at this point. If AI progress stops, or slows dramatically, something will have happened.
• llm_nerd 2mo ago
  On the geopolitics, the American domiciled firms are completely in bed with the US military and US government, and have made it absolutely clear that they are committed to the domination of the US government. So when you say "geopolitical destabilization", you mean "only the US government is allowed to destabilize, which they have done for decades and wish to continue doing unfettered." And given what we have seen from that government -- piracy, terrorism, war-mongering, thievery, and just profound stupidity -- no thanks, in that case I'd like to see every American AI company utterly obliterated by any other country. If China becomes utterly dominate in AI, awesome.
• ls612 2mo ago
  "Many were increasingly of the opinion that they'd all made a big mistake in coming down from the trees in the first place. And some said that even the trees had been a bad move, and that no one should ever have left the oceans."
• deleted 2mo ago
  [deleted]
• JackFr 2mo ago
  If that’s truly the case, why aren’t they offering to be absorbed by national labs. That’s what we do for dangerous biological and nuclear work. Oh wait they want options and shares and to get rich. And there’s no natural moat. Which Federal regulation would bring. That’s why people think it’s disingenuous.
• txrx0000 2mo ago
  There is constructive discourse, but this kind of commentary doesn't get to the top: https://news.ycombinator.com/item?id=49078376 https://news.ycombinator.com/item?id=49078376 https://news.ycombinator.com/item?id=49077753 https://news.ycombinator.com/item?id=49077753 It's not a given that AI poses bio/political/cyber/economic risk in a way that warrants government intervention. Discourse should first revolve around whether it makes sense for the government to step in at all.
• yellowapple 2mo ago
  > It's very easy to make lazy, shallow dismissals that leaders like Dario/Demis/Sam are acting purely out of self-interest. It's very easy to make those dismissals because of how transparently obvious it is that self-interest is a significant factor, no matter how loudly they insist otherwise. > If AI does pose biological risk, or could destabilize geopolitics, or even just poses economic risks, surely a joint petition like this is a good thing? Similar attitudes hindered nuclear energy research and deployment for decades, in turn kneecapping our species' ability to obsolesce fossil fuels and avoid the global climate catastrophe we've now made inevitable.
• potsandpans 2mo ago
  What happens when a country doesn't agree to play along with this little decel game? The genie is already out of the bottle and now a bunch of people who helped release it think that the collective powers of the world are just going to put it back in. How far are the saftyists willing to go? Bombing data centers? Starting wars? Embargos? It's a terrible dilemma to be in. If the threats are real, it gives all the more incentive for some actor to race and try to grab it first. If not, then any punishment isn't proportional. Explain to a nations citizens under a 10 year embargo, "sorry about your quality of life, we all just got a little spooked by science fiction for a second." Folks try to reach for bad analogies: nuclear bombs and biological weapons. But conveniently omit that by the time the regulations took place for these things we had unequivocal proof of their destruction, scope and scale. So, what then? We're supposed to collectively stop innovating and put the keys to the kingdom in the hands of a few capitalists, a corrupt is administration and the ccp? I reject that these agreements should take place, on the grounds that the proposal is delusional and not founded in reality.
• alexashka 2mo ago
  > It's very easy to make lazy, shallow dismissals that leaders like Dario/Demis/Sam are acting purely out of self-interest Pointing out the endless stream of lies and self serving half truths these terrible people engage in helps the younger generation understand the world they were born into. It's constructive and helpful. It's why I'm replying to you. You interpreting it as 'shallow dismissal' is shallow dismissal on your end. You're the shallow dismisser, not the people calling terrible people terrible. As for your little petition. It's as good as eliminating plastic straws to save the environment. Terrible people with power they don't deserve doing pathetic little, tiny 'good' deeds to sleep better at night needs no explaining.
  • anon373839 2mo ago
    Agree with everything you wrote here. Every word. The GP strikes me as unbelievably credulous and trusting of the “leaders” in this space. I cannot fathom having that kind of unquestioning faith in people who have done absolutely nothing to deserve it.
• NitpickLawyer 2mo ago
  > We should want the AI Labs to make binding agreements to proceed safely and slower That's a red herring, IMO. Once the cat is out of the bag, there's no "slowing down", only "slowing up". What's even more egregious is that the people behind the "slowing down" camp are also the people who've had 10s of billions invested in their companies. It smells of "we got ours, now you can't follow". It reeks of regulatory capture, which is a proven net downside for everyone else. The thropics, and the oais, and the googs have billions to make it look like they comply (and just make it look like it), while the rest are ... what? Waiting for them to give us the privilege of prompting a thing, and maybe get a result? If we're good kids and eat our supper? The moment they started adding "nucular" to the acronyms of "safety alignment" you could tell this is just kabuki at best. There's absolutely nothing that a chatbot will tell you that will get you closer to anything nuclear. It's a joke. Meanwhile, companies hit by real-world attacks get guardrails. How's that for being safe?
  • txrx0000 2mo ago
    We have to stop comparing nuclear weapons with general intelligence. They are nothing alike. Nukes are weapons of mass destruction that can only destroy or deter. Intelligence is dual-use/multi-use, with no restrictions on strategy. If we look at all organisms on Earth, species that are more intelligent are also more prosperous. This is true from bacteria, to insects, to animals, to humanity. Intelligence itself is not harmful. There will be new adversarial games when everyone's smarter (bio/cyber offense/defense), but those games are always symmetrical in the long run, and there will also be new cooperative strategies. The reason some adversarial strategies appear to have an asymmetrical advantage right now is because we're too stupid, like how monkeys are too stupid to understand how humans work. Otherwise, we would not see more intelligent species being more prosperous as a general evolutionary trend.
    • 0xDEAFBEAD 2mo ago
      >Nukes are weapons of mass destruction that can only destroy. Intelligence is dual-use. Perhaps a better analogy, instead of "nukes", would be "nuclear technology". Just like intelligence, nuclear technology is dual-use: Capable of both energy generation which fuels prosperity, and also mass destruction. Just like intelligence, nuclear technology demands a healthy level of caution and vigilance. >If we look at all organisms on Earth, species that are more intelligent are also more prosperous. This is true from bacteria, to insects, to animals, to humanity. Intelligence itself is not harmful. Human intelligence has been quite harmful to the species we've driven extinct! AI intelligence could be similar for our species. >There will be new adversarial games when everyone's smarter (bio/cyber offense/defense), but those games are always symmetrical in the long run We've had multiple near misses with nuclear technology, such as the Cuban Missile Crisis. I think you're presuming an inevitability which just doesn't exist. Same way a person living in the year 1500 would have no hope of accurately predicting life in the year 2000, a person living today has no hope of accurately predicting where technological advances will take us, in the limit. This inherent uncertainty should, again, fuel caution and vigilance. Human technology is already so far beyond that of other species to make us a huge outlier. Even if intelligent species are more prosperous as a general evolutionary trend, it would be irresponsible to extrapolate that trend light years from the domain where most of your observations are. Almost all the species which have lived on Earth have gone extinct. It could happen to us too.
• overfeed 2mo ago
  > It's very easy to make lazy, shallow dismissals that leaders like Dario/Demis/Sam are acting purely out of self-interest. It's not a shallow dismissal if you have been following their - let's charitably call it "evolving" - public posture on AI legislation. It wasn't long ago the same people/companies successfully lobbied the Trump admin to quash state laws that threatened to "Pace" AI, and set ground-rules for safety. Back then, open source models were very far behind the frontier, and their spiel then was it was imperative not to impinge on AI progress with red-tape, lest America's adversaries catch up. I challenge you to explain what triggered the about-face. The simplest explanation is that open source/Chinese models have caught up with the frontier, and the change in opinion is tactical (and obviously self-interested), rather than a deeply held belief.
• soundworlds 2mo ago
  It really comes down to this: Sam and Dario are leading companies that who made the reckless decision to train on everyone's IP without proper process or consent. It has been proven they trained on pirated content. There is no question about this. Then, as soon as there are open models who threaten their businesses, they cry "distillation and IP theft". It's the last 3 years of pure hypocrisy and fear-baiting for investment. To be honest, I think people would now prefer the risk of misused open models, when the alternative is giving these guys more power.
  • 0xDEAFBEAD 2mo ago
    You're missing the point of the OP: "there is a real risk that capability development rapidly accelerates beyond our ability to understand or control the resulting systems... each company—and country—is under intense competitive pressure not to unilaterally slow that acceleration" >To be honest, I think people would now prefer the risk of misused open models, when the alternative is giving these guys more power. Then advocate a muscular regulatory regime which reduces the amount of power they have. Tell them: "You want regulation? OK. How about everyone whose IP you trained on gets some sort of veto option on your models? That should reduce the racing risk that you're talking about."
    • soundworlds 2mo ago
      I suppose I have personally lost faith in centralized institutions that would be implementing said regulation, partly because they let OpenAI and Anthropic (along with Suno, Audio, etc etc) get this far uninhibited. Centralized regulation simply can't keep up. My hopes now lay with a decentralized approach of open models, trusting people to adapt them to local needs and security, rather than a nationwide paternalism.
      • 0xDEAFBEAD 2mo ago
        I don't exactly see how "trusting people to adapt [open models] to local needs and security" solves the problem of "automating AI research" where "capability development rapidly accelerates beyond our ability to understand or control the resulting systems". If regulation has been too slow, that's not really an argument for making it even slower?
• GardenLetter27 2mo ago
  As a Europoor I've seen the devastating consequences of rampant safetyism first hand. Entire industries and energy security destroyed to satisfy the precautionary principle. We need to accelerate as much as possible and create abundance as fast as we can. Life is short.
  • 0xDEAFBEAD 2mo ago
    "This building is too cold. Time to set it on fire." Let's not swing from one extreme to the other. If American hypercapitalists, living in Silicon Valley of all places, tell you that you need to regulate, maybe regulation really is called for in these specific circumstances.
    • anon373839 2mo ago
      https://en.wikipedia.org/wiki/Regulatory_capture https://en.wikipedia.org/wiki/Regulatory_capture
      • 0xDEAFBEAD 2mo ago
        Well, we've been talking about the EU. EU regulation has not been especially favorable to US big tech. The EU is collecting more from fining US big tech than it collects in taxes from EU public tech companies: https://xcancel.com/levelsio/status/2080314960656159018 https://xcancel.com/levelsio/status/2080314960656159018 [Edit: I'm seeing some people dispute these numbers in the discussions around this infographic... regardless, the point stands that EU regulation has not been great for US big tech] I think you should at least consider the possibility that this letter is not primarily motivated by regulatory capture, and insiders at these AI companies realize something which outsiders don't. To me, the regulatory capture theory predicts that the only signatories on the letter should be big AI firms which can expect regulatory influence. But there are a number of signatories from smaller firms such as Thinking Machines, Core Automation, Inherent, and Prime Intellect.
        • anon373839 2mo ago
• tpm 2mo ago
  > All of my friends who work at AI Labs take it seriously. > If AI does pose biological risk, or could destabilize geopolitics, or even just poses economic risks, surely a joint petition like this is a good thing? If they seriously think what you just wrote then they should follow through with quitting en masse because that's currently the only way to slow the progress towards oblivion. Are they quitting? > We should be hoping that nations can make binding agreements. The big AI Labs are making sure international binding agreements are followed by financially supporting the notorious binding agreement creator and follower Donald Trump. That's how seriously they take this issue. Yeah let's have a constructive discourse and let's start that with dropping all hypocrisy, shall we.
• vessenes 2mo ago
  It is a very bad thing. Your friends do not have a solid understanding of geopolitics and are putting many, many people at risk worldwide, possibly for a generation or longer. The reason this petition is a bad thing is if it is effective it will create a global panopticon surveillance state, while INCREASING incentives to avoid that state and ‘defect’ in game theoretic terms. It is fucking begging for the worst people in the world to have a lot more time to create what will be the most valuable thing in the world, their very own ‘helpful only’ model. As conceived of by your friends, who I believe are all good people and want to help, they are locking what is already a very difficult game theoretic strategic scenario into a zero-trust game in which defection pays. It pays so amazingly well that they have to add both impossible and TERRIBLE penalties to the entire world to try and shake the game out of this zero-trust state. Begging Senators to force a slowdown on the world is going to result in … a slowdown on the US labs at best. It’s just a terrible idea with terrible first and second order consequences highly likely. What SHOULD be done instead is acknowledge the economic and warfare realities of this tech and try and construct a non-zero-trust game out of them. I’m not saying this is simple. In the interim while we hope for another Von Neumann to come forward, they MUST race, and any country that thinks its possible we’ll have some sort of lift-off and wants to be on the right side of it MUST also race, and MUST NOT politicize the idea of slowing down as a good thing.
  • veeti 2mo ago
    Did these guys miss who they elected for president again? No sane person anywhere else in the world is going to go along with said """governance""".
• HarHarVeryFunny 2mo ago
  It is hard to take people working at frontier labs now asking for a government mandated slow down seriously. These people are the ones creating this accelerated risk in the first place, trying to outdo each other to release the most powerful AI as quickly as possible, and additionally claiming credit (via "distillation") for recent progress by China. China is not the threat - you are. If you work for a frontier lab and are genuinely concerned about the pace of AI being faster than our ability to adapt to it, then you should either quit or lobby the management of your own company to slow down. You don't need the government to tell you to slow down - just do it - stop trying to leapfrog the competition and put a hold on any new releases for a year. You could try to form a pact (of the willing) of US companies agreeing not to release new updates for a given period of time, but in the end you (any/all of the frontier labs) can choose to do it yourself if you really care. This is just another version of Dario's insane "heads up, what I'm developing is going to destroy society" "responsible disclosure". It's not responsible in the least - if you actually believe you are creating such danger then stop doing that.
• QuantumGood 2mo ago
  > should want [others] to make binding agreements As Munger emphasized, others act according to incentives. First, clarify those incentives, then incentivize them if you can.