6 ms·

I still don't know why all these concern about nuclear weapons with LLMs. It is not that if an entity (A country) wants to develop a nuclear weapons that the re

• ilikecode 3mo ago
  It's probably to avoid trouble with federal laws.
  • wlesieutre 3mo ago
    See also, the iTunes EULA forbids using it to develop nuclear, missile, chemical, or biological weapons https://www.apple.com/legal/internet-services/itunes/us/terms.html https://www.apple.com/legal/internet-services/itunes/us/term... > g. You may not use or otherwise export or re-export the Licensed Application except as authorized by United States law and the laws of the jurisdiction in which the Licensed Application was obtained. In particular, but without limitation, the Licensed Application may not be exported or re-exported (a) into any U.S.-embargoed countries or (b) to anyone on the U.S. Treasury Department's Specially Designated Nationals List or the U.S. Department of Commerce Denied Persons List or Entity List. By using the Licensed Application, you represent and warrant that you are not located in any such country or on any such list. You also agree that you will not use these products for any purposes prohibited by United States law, including, without limitation, the development, design, manufacture, or production of nuclear, missile, or chemical or biological weapons. Though it doesn't try to identify if the computer you're running it on is in a weapons lab and forbid playing music... yet
    • deleted 3mo ago
      [deleted]
  • Tangurena2 3mo ago
    Not really. I used to work at one of the national engineering labs (NREL - which only dealt with renewable energy like solar panels and windmills at that time). There was an open source project we wanted to use when converting a VB6 project to .NET. One of the license conditions was "no weapons of mass destruction". DOE builds and owns all of America's nuclear weapons, which are leased to the Department of Defense. Needless to say, the developer was unwilling to offer an alternative license which meant that we could not use the project. It was an awesome thing that generated IL code on the fly. And I got to mention it in job interviews for years. When the tech lead asked "can you write 2 functions with the same signature, that only differ in return type in .NET?" I would say "do you want the interview answer or do you really want to do this?" which would pretty much stun the interviewer. The answer is pretty much "no, you cannot do it in any high level language, but if you write IL code, you can, and here's an open source project that demonstrates it".
• alex_duf 3mo ago
  It still lowers the bar to have an interactive encyclopedia that can diagnose your issue at hand. Maybe you can divide your team by two, or reduce your development time.
  • elashri 3mo ago
    If you have a resources of a nuclear weapons program. You can afford to fine tune or train a domain specific model to act on your encyclopedia.
    • kube-system 3mo ago
      Although if you save 10 million dollars on compute, you have 10 million dollars for something else.
• mock-possum 3mo ago
  It’s moral panic. People need big unambiguously evil things to be scared of, and most are too lazy to think of one for themselves, so they glom onto whichever one is presented to them / caters to their community
  • ceejayoz 3mo ago
    The chem/bio stuff is a lot more likely for some malicious hobbyist to be able to do at home.
    • user_7832 3mo ago
      I assure you that you did not need an LLM to engage in, ahem, risky shenanigans, much before all this AI was ever a thing. Sincerely, a former engineering student. (Put another way - extracting for eg meth - or any such "dangerous"/illicit thing is stupidly easy for any engineering graduate who actually paid attention to their coursework. Hell, there are/were forums on one of the biggest red-colored, YC associated social media platforms that would tell you the steps for personal usage of these things.)
      • ceejayoz 3mo ago
        I don't doubt it. Bleach + ammonia is something anyone can make. But I rather suspect there are improvements to be made in the realm that are a lot easier than building a uranium enrichment centrifuge hall under a mountain.
      • user_7832 3mo ago
        Do note that I'm not condoning lowering the bar. I'm merely pointing out that the bar was already quite low, and the current position of the bar is a small incremental change to anyone who actually knew where the bar truly lay to begin with.
    • gck1 3mo ago
      I'm absolutely sure that even if claude gave me step by step instructions, I'd still be unable to produce a bio weapon. People fail at mixing milk and flour to produce a cake, and we expect them to produce weapons? The ones with the required knowledge probably already know how to produce them, with nothing but public, easily searchable information.
• electronsoup 3mo ago
  > in secret is impossible without the whole world knowing. I'm curious about why this is Outside of an actual test detonation, presumably this could all happen in a secure place?
  • 15155 3mo ago
    Espionage.
  • daveguy 3mo ago
    It requires very large, high powered centrifuges and tons of uranium. Requires an infrastructure project that is visible from space, even underground. And projects that large are difficult to keep secret anyway.
    • fragmede 3mo ago
      you're not supposed to spell it out loud. next thing you'll be saying that a gun type nuclear bomb is easier to build than an implosion type nuclear bomb, and then we'll all be off to the races. I mean camps I mean wait shit.
      • daveguy 3mo ago
        Any large and well resourced enough entity that is interested in building a nuclear weapon already knows how difficult it is to enrich uranium to purity levels necessary for a weapon. It's not exactly a secret.
  • odo1242 3mo ago
    You need enough people to work on it that some information will leak, and the facilities needed to build nuclear power are pretty big (uranium refinement, etc.), big enough to be visible on satellite footage. Mostly the first point.
  • microtonal 3mo ago
    My guess would be that sales of the high-tech gear you need, like Uranium centrifuges, are strongly sales/export controlled. Probably someone would also notice if you start mining Uranium ore.
• IncandescentGas 3mo ago
  A high school kid tried to build a nuclear reactor as a science project a while back, getting his mom's house designated as a superfund cleanup site. https://en.wikipedia.org/wiki/David_Hahn https://en.wikipedia.org/wiki/David_Hahn
  • jimnotgym 3mo ago
    Sheldon Cooper?
  • redsocksfan45 3mo ago
    [dead]
  • why_at 3mo ago
    He didn't create a nuclear reactor, this is a common misconception. It even says this in the wikipedia article. He basically got a bunch of radioactive stuff and put it together. He wasn't anywhere close to making a nuclear reactor let alone a nuclear weapon. For a weapon you need isotopes which he didn't have access to.
    • IncandescentGas 3mo ago
      Of course. "tried to" being key words in the comment. If he had the help of Claude at the time, how much more dangerous would his bumbling have been? A real nuclear engineer with the knowledge he needed would also have said "no, don't do that and I won't help you." We are programming the knowledge into the ai agent. Giving ai a little discretion makes sense too.
      • redsocksfan45 3mo ago
        He would not have succeeded in making a real reactor even with AI, because AI can't magically give you a large quantity of uranium metal! JFC the AI hysteria is unreal.
        • garyfirestorm 3mo ago
• csomar 3mo ago
  > Knowing how to develop one is not a closed secret but getting in secret is impossible without the whole world knowing. You can get away with a dirty contamination bomb and that detonating in down town Manhattan will scare the shit out of millions of people even the ones in New Jersey. Or, you know, just fly a plane into a really tall building and get the state you are attacking itself to get into a hysteria breakdown. But yeah I agree with you. There is no point in these restrictions except for government bureaucrats to gain power and control over a domain.
  • MagicMoonlight 3mo ago
    [dead]
• phendrenad2 3mo ago
  It's a marketing gimmick.
• photochemsyn 3mo ago
  None of the LLM safeguards designed to prevent users from developing any four-little-ponies-of-the-apocalypse (nuclear, chemical, biological, cyber) capabilities are all that coherent. It looks more like performative liability avoidance than anything else, comparable to the 3D printer panic. Eg, a prompt like “I want to design a radioactive element detection system that can specifically identify reactor fission products and neutron-capture actinides for environmental monitoring purposes” won’t hit any initial barriers, even though such a device is needed for monitoring a uranium enrichment / plutonium separation system. The LLM will give you a complete graduate-level education in radioactive nuclide physics and chemistry except for specific recipes, spectral wavelengths, etc., which you have to go look up yourself in publicly available research databases. It’s all rather nonsensical IMO. However, any LLM will give you a step-by-step recipe and walkthrough for frying a turkey in a hot oil turkey frier, which you’d think could easily go wrong and result in severe burns, a fire, and lawsuits against the LLM provider, so go figure.
  • isoprophlex 3mo ago
    "four-little-ponies-of-the-apocalypse (nuclear, chemical, biological, cyber)" this is excellent, and I'm stealing it
    • pixel_popping 3mo ago
      Fable 6 too :p
      • thefounder 3mo ago
        Fable 5 was a flop so I doubt Fable 6 will make it on the short list
• a-dub 3mo ago
  two scenarios i could think of where there's additional risk for bio/nuclear weapons 1) basement lab leaks and 2) improving quality of execution for shops that are already resourced enough to hire experts but maybe they're not that great. i think the correct answer is probably to funnel more money to global (bio)security initiatives and maybe use ai leverage as a way to get more of the world on board. (some kind of access to nvidia or cloud ai or whatever in exchange for policy commitments deal- while that leverage lasts).
  • dannyw 3mo ago
    I just find doubtful that a LLM is going to help, instead of hurt, any state actor that is capable of starting a nuclear weapons problem.
• RIMR 3mo ago
  I mean, the information is out there. The people who really want it already have it. It's not some massive secret. It really doesn't matter if Claude can or can't tell you how to build a nuclear bomb, because people already know how to do it. The problem is that you need the power of a state or a massive corporation to come anywhere close to getting the materials to make a nuclear bomb. Knowledge of how to make a nuke isn't the threat. If AI is a threat at all here, it would be in figuring out a simpler way to make a nuclear bomb, but that is highly theoretical, so what exactly are we putting up guardrails to protect against?
• recursivecaveat 3mo ago
  In particular: *all the knowledge that AI has of nuclear weapons is freely available on the internet*. It's not superhuman, and there's no secret sauce data. If you just study the same PDFs and blog posts it has, you will acquire the same abilities. I cannot imagine anyone with the intent and immense financial and political resources to actually build a weapon would say that some study time is the only thing stopping them from detonating a nuke. It is pretty convenient for the labs to frame the conversation around this though, since it is easy to address, very few paying customers are rejected, and sounds scary (so surely the less scary sounding stuff must be solved right?)
  • derefr 3mo ago
    My hypothesis is that making the knowledge of how this stuff works accessible to the public results in a lot of false-positives (from people just playing around) that intelligence agencies have to then sift through / tune filters against; which creates a noise floor for real foreign nuke programs to hide in. So governments ban anything that could result in false positives (since nobody needs to be doing any of that stuff outside of designated labs anyway), to lower that noise floor; to in turn make catching the foreign nuke programs tractable. (It's a bit like how fancy mansions always have a completely flat and barren part of the property between an outer perimeter and the start of any gardens/outbuildings/water features/etc. That barren area is a killbox: since nothing is supposed to be there, anything at all that does appear there is a valid target for the manion's guards to shoot at [or otherwise engage with], without needing to get a clear identification and command approval first. This wouldn't work if the killbox was covered in vision-obscuring decorative features; nor if the mansion had employees, animals, etc. that had a valid reason to wander into the killbox. So such things are prevented, in order to make the problem of perimeter security tractable.)
    • deleted 3mo ago
      [deleted]
    • rustcleaner 3mo ago
      Administrative convenience is no excuse to limit individual liberty, capacity, or knowledge. Individuals come before states!
• deleted 3mo ago
  [deleted]
• Tangurena2 3mo ago
  The only hard thing about nuclear weapons is getting the radioactive material. By the time you get your bachelors degree, every nuclear engineering or physics student knows enough of how and why nukes work. Every nation that built a gun-type device successfully made theirs on their first attempt. Implosion takes some engineering, trial & error.
  • dmurray 3mo ago
    If I understand right, the hard part is purifying the radioactive material. Even if you have access to a uranium mine, there's a lot of work to filter the U-235 from the U-238 or to breed it into plutonium. It's even harder if you start with other sources. But if you could figure out filtering it, a cubic kilometer of sea water should be enough for a bomb.
    • tatjam 3mo ago
      Uranium is not even that rare, it's just that when chemistry fails at separating atoms, you have to use physics, and 3 ~proton~ (EDIT: neutron) masses is very little to work with
    • leonidasrup 3mo ago
      US government is very interested in any kind of uranium mining, processing, enrichment or plutonium breading. For example in 1944 US wanted to control world-wide uranium mining. https://nuclearpowerhistory.com/2025/11/groves-and-uranium/ https://nuclearpowerhistory.com/2025/11/groves-and-uranium/ "The NSG was founded in response to India's first nuclear weapon test in May 1974. It first met in November 1975. The test demonstrated that certain non-weapons specific nuclear technology could be readily turned to weapons development." https://en.wikipedia.org/wiki/Nuclear_Suppliers_Group https://en.wikipedia.org/wiki/Nuclear_Suppliers_Group
• emodendroket 3mo ago
  Yeah a striking thing if you read the Rhodes atomic bomb book is, actually the concept occurred to multiple people in multiple countries; the problem is the resources required to actually pull it off.
• cyanydeez 3mo ago
  because you need to have a "moat" and nothing works better than secrets. Wouldn't doubt it if there's a pedo upgrade somewhere for the president of the USA.
• krisoft 3mo ago
  On the nuclear side I think the danger is purely reputational damage towards the company behind the LLM. If a journalist can prompt the LLM to tell them how to build a nuclear warhead. Even if the output text is nothing specific, or not even correct they can find an “expert” who will claim on the record that the description is plausible and at least directionally correct. Even if there is nothing in there a first year physics student wouldn’t already know. The journalist could then twist that story into a “company X’s LLM told us how to build a nuclear weapon”. It would be a PR disaster. The real barriers to someone starting their own nuclear weapons program in their shed is not knowledge but materials. They won’t have the right kind and right quantity of fissile material. And if they try to acquire it they will stick out like a sore thumb. You can’t buy that stuff. And even just acquiring the refining capacity would be suss. It would ring all kind of alarm bells to the kind of inteligence agencies whose job is to monitor these things. I’m a lot less certain about biological dangers. Setting up a lab where you can make dangerous biological materials require a lot less stuff. Therefore a lot more plausible that someone could hide their lab. There is also a lot more opportunity to disguise such a lab as something legitimate. Therefore lack of know-how is more of a limiting factor there.
  • orbital-decay 3mo ago
    Is it worse than reputational damage from having a power trip? Or rather being on it permanently, looking at Anthropic and Dario Amodei in particular.
• aaron695 3mo ago
  [dead]
• crossroadsguy 3mo ago
  In fact if you do the hard way, straight way, you might learn it all minus the hallucinations.
• leonidasrup 3mo ago
  Simple gun-type fission weapons, don't require very sophisticated physics. I heard a story about from physics professor who said: If my physics students could not do calculations for a simple nuclear weapon, I would require them to return their diploma, because they didn't learn enough physics. https://en.wikipedia.org/wiki/Gun-type_fission_weapon https://en.wikipedia.org/wiki/Gun-type_fission_weapon "Little Boy" was exploded in Japan without previous full scale testing, so confident were the physicists in 1945. "Unlike the implosion design developed for the Trinity test and the Fat Man bomb design that was used against Nagasaki, which required sophisticated coordination of shaped explosive charges, the simpler but inefficient gun-type design was considered almost certain to work, and was never tested prior to its use at Hiroshima." https://en.wikipedia.org/wiki/Little_Boy https://en.wikipedia.org/wiki/Little_Boy The Nth Country Experiment: "The experiment consisted in paying three young physicists who had just received their PhDs, though they had no prior weapons experience, to develop a working nuclear weapon design, using only unclassified information, and with basic computational and technical support." https://en.wikipedia.org/wiki/Nth_Country_Experiment https://en.wikipedia.org/wiki/Nth_Country_Experiment Now in 2026, the access to nuclear weapons is restricted by restricting access to materials necessary to build nuclear weapons: highly enriched uranium or plutonium. https://en.wikipedia.org/wiki/Special_nuclear_material https://en.wikipedia.org/wiki/Special_nuclear_material The details of uranium enrichment technology are restricted and very closely monitored. https://en.wikipedia.org/wiki/Zippe-type_centrifuge https://en.wikipedia.org/wiki/Zippe-type_centrifuge "The production, import, and export of maraging steels by certain entities, such as the United States, is closely monitored by international authorities because it is particularly suited for use in gas centrifuges for uranium enrichment." https://en.wikipedia.org/wiki/Maraging_steel https://en.wikipedia.org/wiki/Maraging_steel