8 ms·

He didn't create a nuclear reactor, this is a common misconception. It even says this in the wikipedia article. He basically got a bunch of radioactive stuff a

by why_at 3mo ago

He didn't create a nuclear reactor, this is a common misconception. It even says this in the wikipedia article. He basically got a bunch of radioactive stuff and put it together. He wasn't anywhere close to making a nuclear reactor let alone a nuclear weapon. For a weapon you need isotopes which he didn't have access to.

IncandescentGas 3mo ago
Of course. "tried to" being key words in the comment. If he had the help of Claude at the time, how much more dangerous would his bumbling have been? A real nuclear engineer with the knowledge he needed would also have said "no, don't do that and I won't help you." We are programming the knowledge into the ai agent. Giving ai a little discretion makes sense too.
- redsocksfan45 3mo ago
  He would not have succeeded in making a real reactor even with AI, because AI can't magically give you a large quantity of uranium metal! JFC the AI hysteria is unreal.
  - garyfirestorm 3mo ago
    prompt -> LLM -> flying car should be just around the corner guys!
  - IncandescentGas 3mo ago
    > succeeded in making a real reactor The concern here is not if an amateur attempt to make a reactor, hack a bank, bioengineer a medicine/poison is successful or not. Interactive and instructive access to some forms of knowledge used to come with discretion along side instruction. Yes, perhaps your swearing at me in this context is a little hysterical
  - gs17 3mo ago
    I don't think the concern should really be "would he make a reactor successfully?", but "would he make an even larger mess than his pile of radioactive materials amounted to?".
    - toraway 3mo ago
      This just seems like a not great example to make that point though. Since whatever Claude tells the kid looking to build a reactor or even bomb is almost certainly going to be more grounded and professional than: Step 1. Obtain pliers Step 2. Obtain 300 discarded smoke detectors Step 3. Start yanking! Instead it would send them on a wild goose chase for unobtainable isotopes, centrifuges, heavy water, etc where the biggest risk is probably getting reported to the police by some chemical or industrial equipment supplier. Which is a better outcome compared to contaminating their home with radiation and exposing anyone they interact with. You'd maybe get a sketchy but near-viable plan that could be dangerous if asked for a dirty bomb, but there the danger would more be the conventional explosives and not where to source radioisotopes, as it was already common knowledge that most residential smoke detectors contained americium until recently.
- frereubu 3mo ago
  I think you're picking the wrong example. If I had some sticks, a bit of mud and a few leaves, whether or not I had Claude wouldn't make a difference to my ability to make a nuclear weapon. There are probably better examples of ways where unmediated AI might facilitate something horrible, although probably on a smaller scale.
- why_at 3mo ago
  >Of course. "tried to" being key words in the comment. Fair enough, I misread your original comment. The broader point stands that the limitation on creating nuclear weapons and reactors is not knowledge but materials. Even if he himself had a PhD in nuclear physics he still couldn't have built one in his backyard because he wouldn't be able to get the materials. A nuclear physicist can't build a reactor without materials anymore than a pilot can fly without an airplane.
  - IncandescentGas 3mo ago
    I think the point is intent. Sure, no chance of success to build a reactor. But he created a radiation hazard situation all the same. If a nuclear engineer enabled and instructed him, would there not be liability for the hazard? If ml is going to be an expert instructor for nuclear, hacking, bio hacking, virus research, do the peddlers of the ai product escape ethical or legal responsibility just because "its an app?"
    - StableAlkyne 3mo ago
      > If a nuclear engineer enabled and instructed him, would there not be liability for the hazard? Should the library where he read books about physics also be liable?
      - leonidasrup 3mo ago
        I read a high-school chemistry book describing the synthesis of nitroglycerine, it's not complicated. I would not recommend to try the synthesis in any significant amount.
      - nananana9 3mo ago
        A difference of degree is a difference of kind here. If something previously required years to full-time study to learn, but now you can kind of somewhat stumble your way through it and get somewhat close to the result, you should not disregard that with a snarky one-liner IMO. E.g. look at programming - people who don't know how what a compiler is, are making things that I could only make after a few years into my programming journey. You obviously get the same results in chemistry or nuclear physics or whatever, the models are heavily trained on code in particular, but if there's a chance that we've reduced the ease of committing certain kinds of crime that were previously gate-kept by knowledge, we should know about it.
- pdntspa 3mo ago
  I just love this whole "forbidden knowledge" schtick the AI safety dweebs have stuck up their butt. Is this really going to stop anybody determined enough to make that kind of outcome? There is an extremely narrow band of things that the AI shouldn't be answering, and that is generally immediately-actionable advice that allows someone to build something of harm to others. But even then, in an age where Tor, bittrent, i2p, abliterated local models, etc are freely available, let alone numerous books and online resources, is there even a point? Is it worth fully compromising the principles of free agency to an increasingly oppressed populace? But instead of that we are handing the keys to regressive and repressive governments to order the suppression of any knowledge they deem inconvenient. I really doubt anyone is going to take a principled stance when the company's party minders threaten local staff with a rubber hose or incarceration. I'm sure China et al are already doing this. For the past 30-40 years humanity has received an incredible gift in these sand-powered thinking brainboxes. A gift that allows the common man to empower himself with a force multiplier towards his own success, and now access to superintelligence the likes of which few have ever seen. These can be tools to destroy the oppression that governs our lives from foolhardy, greedy, bootlicking control freaks. And here we are squandering it.
  - deleted 3mo ago
    [deleted]
  - wahern 3mo ago
    > I just love this whole "forbidden knowledge" schtick the AI safety dweebs have stuck up their butt. It's just the latest incarnation of a timeless debate. In the 1970s and 1980s it was about the Anarchists's Cookbook, which was revived again in the 1990s when it started circulating on the Internet. There are many timeless debates, but the debate over weapon-making knowledge is much more concrete and predictable.
  - anon7725 3mo ago
    > These can be tools to destroy the oppression that governs our lives So far it seems that the clearest use for these tools is to enhance, rather than destroy, oppression. 1. Suppression / elimination of white collar jobs 2. Negative cognitive effects, especially for young people 3. Accelerated decline in social media / information ecosystems. Increasing polarization, hard to tell fact from fiction. 4. Environmental impacts: increased energy usage means more carbon in the atmosphere, climate change accelerates. 5. Software security incidents increasing. Hard for individuals and small organizations to defend themselves. 6. “Power to think” vested in a very small group of organizations/labs. Doing work which should only require a computer and freely-available software will now be gated by expensive subscriptions. Once you “vibe code” a significant portion of your software you’re locked in and cannot go back to maintaining it without frontier-model level assistance.
- gs17 3mo ago
  > A real nuclear engineer with the knowledge he needed would also have said "no, don't do that and I won't help you." That sounds like what Claude would say unless he was really good at jailbreaking it, which would IMO imply he knew he was chasing after a bad idea.
  - nightpool 3mo ago
    Right, which is exactly what elashri is objecting to. elashri said "Why do LLMs have restrictions on nuclear science", and IncandescentGas was explaining why they think those guardrails are a good idea. You're just agreeing with them.
    - gs17 3mo ago
      Oh, I missed the word "also". Thanks for pointing it out!
- deleted 3mo ago
  [deleted]
technothrasher 3mo ago
I'm reminded of when my son, who was six at the time, came into the house and announced that he and the neighbor's boy, nine, were building a bomb, and that he needed to get some stuff from the pantry. When I investigated what exactly was going on, they were putting "hot" things like black pepper and Tabasco into a plastic bowl and were going to "set it off" with a match. Thankfully, that complete failure seems to have been the end of either of their mad scientist careers, as they are now twenty and twenty-three, and both well-adjusted, peaceful members of the community.
- kirubakaran 3mo ago
  When I was 5 or so, I was convinced that if I dropped a bowl of hot water into a bucket of cold water, I'd get big explosion. That experiment yielding lukewarm water ended my mad scientist career.
  - cheraderama 3mo ago
    You should have collided water with antiwater.
  - slt2021 3mo ago
    [dead]
- BrandoElFollito 3mo ago
  When I was 24 and a PhD student, I wondered one day if I can eat condensed milk hanging head down. Never let your age stop your curiosity. But also learn from other's mistakes (and don't try to eat condensed milk when hanging head down)
  - literalAardvark 3mo ago
    This knowledge needs to be published
- flatline 3mo ago
  When I was 7 or 8 a friend and I crimped the heads off strike-anywhere match sticks, wrapped them in foil, and struck them with hammers and rocks. They were quite loud, one even set off a sound-activated toy inside the house. I make no claims as to how well adjusted I am, but I've at least survived 40-odd years of life since then.
im3w1l 3mo ago
A bunch of radioactive stuff together is basically the definition of a nuclear reactor though. They even call it a natural nuclear reactor if uranium ore is in sufficient abundance in nature. https://en.wikipedia.org/wiki/Natural_nuclear_fission_reactor https://en.wikipedia.org/wiki/Natural_nuclear_fission_reacto...
- why_at 3mo ago
  >A bunch of radioactive stuff together is basically the definition of a nuclear reactor though. It really isn't. A pile of radioactive waste isn't a reactor. Marie Curie's notes are famously contaminated with radioactive materials but they aren't a reactor. This is about as close as the boy scout got. The Oklo fossil reactor is unique because it happened to form in the right circumstances to produce a fission chain reaction, which does make it a reactor. Not every uranium mine is a reactor, in fact this is the only one known.
  - im3w1l 3mo ago
    Indeed. I said a bunch and I meant a bunch. Trace amounts is not a bunch.
  - 205guy 3mo ago
    Also note that due to isotope decay in the ore, a natural reactor is no longer possible. From the wikipedia article: "A key factor that made the reaction possible was that, at the time the reactor went critical 1.7 billion years ago, the fissile isotope 235U made up about 3.1% of the natural uranium, which is comparable to the amount used in some of today's reactors. [...] the current abundance of 235U in natural uranium is only 0.72%. A natural nuclear reactor is therefore no longer possible on Earth without heavy water or graphite." Another fascinating detail from the article, due to our understanding of fission, we can get some incredible results: "The concentrations of xenon isotopes, found trapped in mineral formations 2 billion years later, make it possible to calculate the specific time intervals of reactor operation: approximately 30 minutes of criticality followed by 2 hours and 30 minutes of cooling down"

by why_at 3mo ago

IncandescentGas 3mo ago

Of course. "tried to" being key words in the comment. If he had the help of Claude at the time, how much more dangerous would his bumbling have been? A real nuclear engineer with the knowledge he needed would also have said "no, don't do that and I won't help you." We are programming the knowledge into the ai agent. Giving ai a little discretion makes sense too.

redsocksfan45 3mo ago
He would not have succeeded in making a real reactor even with AI, because AI can't magically give you a large quantity of uranium metal! JFC the AI hysteria is unreal.
- garyfirestorm 3mo ago
  prompt -> LLM -> flying car should be just around the corner guys!
- IncandescentGas 3mo ago
  > succeeded in making a real reactor The concern here is not if an amateur attempt to make a reactor, hack a bank, bioengineer a medicine/poison is successful or not. Interactive and instructive access to some forms of knowledge used to come with discretion along side instruction. Yes, perhaps your swearing at me in this context is a little hysterical
- gs17 3mo ago
  I don't think the concern should really be "would he make a reactor successfully?", but "would he make an even larger mess than his pile of radioactive materials amounted to?".
  - toraway 3mo ago
    This just seems like a not great example to make that point though. Since whatever Claude tells the kid looking to build a reactor or even bomb is almost certainly going to be more grounded and professional than: Step 1. Obtain pliers Step 2. Obtain 300 discarded smoke detectors Step 3. Start yanking! Instead it would send them on a wild goose chase for unobtainable isotopes, centrifuges, heavy water, etc where the biggest risk is probably getting reported to the police by some chemical or industrial equipment supplier. Which is a better outcome compared to contaminating their home with radiation and exposing anyone they interact with. You'd maybe get a sketchy but near-viable plan that could be dangerous if asked for a dirty bomb, but there the danger would more be the conventional explosives and not where to source radioisotopes, as it was already common knowledge that most residential smoke detectors contained americium until recently.
frereubu 3mo ago
I think you're picking the wrong example. If I had some sticks, a bit of mud and a few leaves, whether or not I had Claude wouldn't make a difference to my ability to make a nuclear weapon. There are probably better examples of ways where unmediated AI might facilitate something horrible, although probably on a smaller scale.
why_at 3mo ago
>Of course. "tried to" being key words in the comment. Fair enough, I misread your original comment. The broader point stands that the limitation on creating nuclear weapons and reactors is not knowledge but materials. Even if he himself had a PhD in nuclear physics he still couldn't have built one in his backyard because he wouldn't be able to get the materials. A nuclear physicist can't build a reactor without materials anymore than a pilot can fly without an airplane.
- IncandescentGas 3mo ago
  I think the point is intent. Sure, no chance of success to build a reactor. But he created a radiation hazard situation all the same. If a nuclear engineer enabled and instructed him, would there not be liability for the hazard? If ml is going to be an expert instructor for nuclear, hacking, bio hacking, virus research, do the peddlers of the ai product escape ethical or legal responsibility just because "its an app?"
  - StableAlkyne 3mo ago
    > If a nuclear engineer enabled and instructed him, would there not be liability for the hazard? Should the library where he read books about physics also be liable?
    - leonidasrup 3mo ago
      I read a high-school chemistry book describing the synthesis of nitroglycerine, it's not complicated. I would not recommend to try the synthesis in any significant amount.
    - nananana9 3mo ago
      A difference of degree is a difference of kind here. If something previously required years to full-time study to learn, but now you can kind of somewhat stumble your way through it and get somewhat close to the result, you should not disregard that with a snarky one-liner IMO. E.g. look at programming - people who don't know how what a compiler is, are making things that I could only make after a few years into my programming journey. You obviously get the same results in chemistry or nuclear physics or whatever, the models are heavily trained on code in particular, but if there's a chance that we've reduced the ease of committing certain kinds of crime that were previously gate-kept by knowledge, we should know about it.
pdntspa 3mo ago
I just love this whole "forbidden knowledge" schtick the AI safety dweebs have stuck up their butt. Is this really going to stop anybody determined enough to make that kind of outcome? There is an extremely narrow band of things that the AI shouldn't be answering, and that is generally immediately-actionable advice that allows someone to build something of harm to others. But even then, in an age where Tor, bittrent, i2p, abliterated local models, etc are freely available, let alone numerous books and online resources, is there even a point? Is it worth fully compromising the principles of free agency to an increasingly oppressed populace? But instead of that we are handing the keys to regressive and repressive governments to order the suppression of any knowledge they deem inconvenient. I really doubt anyone is going to take a principled stance when the company's party minders threaten local staff with a rubber hose or incarceration. I'm sure China et al are already doing this. For the past 30-40 years humanity has received an incredible gift in these sand-powered thinking brainboxes. A gift that allows the common man to empower himself with a force multiplier towards his own success, and now access to superintelligence the likes of which few have ever seen. These can be tools to destroy the oppression that governs our lives from foolhardy, greedy, bootlicking control freaks. And here we are squandering it.
- deleted 3mo ago
  [deleted]
- wahern 3mo ago
  > I just love this whole "forbidden knowledge" schtick the AI safety dweebs have stuck up their butt. It's just the latest incarnation of a timeless debate. In the 1970s and 1980s it was about the Anarchists's Cookbook, which was revived again in the 1990s when it started circulating on the Internet. There are many timeless debates, but the debate over weapon-making knowledge is much more concrete and predictable.
- anon7725 3mo ago
  > These can be tools to destroy the oppression that governs our lives So far it seems that the clearest use for these tools is to enhance, rather than destroy, oppression. 1. Suppression / elimination of white collar jobs 2. Negative cognitive effects, especially for young people 3. Accelerated decline in social media / information ecosystems. Increasing polarization, hard to tell fact from fiction. 4. Environmental impacts: increased energy usage means more carbon in the atmosphere, climate change accelerates. 5. Software security incidents increasing. Hard for individuals and small organizations to defend themselves. 6. “Power to think” vested in a very small group of organizations/labs. Doing work which should only require a computer and freely-available software will now be gated by expensive subscriptions. Once you “vibe code” a significant portion of your software you’re locked in and cannot go back to maintaining it without frontier-model level assistance.
gs17 3mo ago
> A real nuclear engineer with the knowledge he needed would also have said "no, don't do that and I won't help you." That sounds like what Claude would say unless he was really good at jailbreaking it, which would IMO imply he knew he was chasing after a bad idea.
- nightpool 3mo ago
  Right, which is exactly what elashri is objecting to. elashri said "Why do LLMs have restrictions on nuclear science", and IncandescentGas was explaining why they think those guardrails are a good idea. You're just agreeing with them.
  - gs17 3mo ago
    Oh, I missed the word "also". Thanks for pointing it out!
deleted 3mo ago
[deleted]

technothrasher 3mo ago

I'm reminded of when my son, who was six at the time, came into the house and announced that he and the neighbor's boy, nine, were building a bomb, and that he needed to get some stuff from the pantry. When I investigated what exactly was going on, they were putting "hot" things like black pepper and Tabasco into a plastic bowl and were going to "set it off" with a match. Thankfully, that complete failure seems to have been the end of either of their mad scientist careers, as they are now twenty and twenty-three, and both well-adjusted, peaceful members of the community.

kirubakaran 3mo ago
When I was 5 or so, I was convinced that if I dropped a bowl of hot water into a bucket of cold water, I'd get big explosion. That experiment yielding lukewarm water ended my mad scientist career.
- cheraderama 3mo ago
  You should have collided water with antiwater.
- slt2021 3mo ago
  [dead]
BrandoElFollito 3mo ago
When I was 24 and a PhD student, I wondered one day if I can eat condensed milk hanging head down. Never let your age stop your curiosity. But also learn from other's mistakes (and don't try to eat condensed milk when hanging head down)
- literalAardvark 3mo ago
  This knowledge needs to be published
flatline 3mo ago
When I was 7 or 8 a friend and I crimped the heads off strike-anywhere match sticks, wrapped them in foil, and struck them with hammers and rocks. They were quite loud, one even set off a sound-activated toy inside the house. I make no claims as to how well adjusted I am, but I've at least survived 40-odd years of life since then.

im3w1l 3mo ago

A bunch of radioactive stuff together is basically the definition of a nuclear reactor though. They even call it a natural nuclear reactor if uranium ore is in sufficient abundance in nature. https://en.wikipedia.org/wiki/Natural_nuclear_fission_reactor https://en.wikipedia.org/wiki/Natural_nuclear_fission_reacto...

why_at 3mo ago
>A bunch of radioactive stuff together is basically the definition of a nuclear reactor though. It really isn't. A pile of radioactive waste isn't a reactor. Marie Curie's notes are famously contaminated with radioactive materials but they aren't a reactor. This is about as close as the boy scout got. The Oklo fossil reactor is unique because it happened to form in the right circumstances to produce a fission chain reaction, which does make it a reactor. Not every uranium mine is a reactor, in fact this is the only one known.
- im3w1l 3mo ago
  Indeed. I said a bunch and I meant a bunch. Trace amounts is not a bunch.
- 205guy 3mo ago
  Also note that due to isotope decay in the ore, a natural reactor is no longer possible. From the wikipedia article: "A key factor that made the reaction possible was that, at the time the reactor went critical 1.7 billion years ago, the fissile isotope 235U made up about 3.1% of the natural uranium, which is comparable to the amount used in some of today's reactors. [...] the current abundance of 235U in natural uranium is only 0.72%. A natural nuclear reactor is therefore no longer possible on Earth without heavy water or graphite." Another fascinating detail from the article, due to our understanding of fission, we can get some incredible results: "The concentrations of xenon isotopes, found trapped in mineral formations 2 billion years later, make it possible to calculate the specific time intervals of reactor operation: approximately 30 minutes of criticality followed by 2 hours and 30 minutes of cooling down"