6 ms·

I don't think knowing how to build a bio weapon means one could actually build a bio weapon. It's like saying knowing how to build a nuclear weapon can easily l

I don't think knowing how to build a bio weapon means one could actually build a bio weapon. It's like saying knowing how to build a nuclear weapon can easily lead to building a nuclear weapon. Advanced machinery and safety precaution is needed to engineer a virus or a bacteria to be a bio weapon. It's not something you can do in your tool shed at the moment. Also one would need lethal viruses to even start with and that's not something you can order off of amazon. Further, even to build a chemical weapon, the compounds needed are strictly controlled almost in every country and I would assume any suspicious purchase or order would immediately raise a flag or alert in national security service in respective country.

palmotea 6d ago
> Advanced machinery and safety precaution is needed to engineer a virus or a bacteria to be a bio weapon. It's not something you can do in your tool shed at the moment. I think the claim in this report is that it was a state or semi-state actor. They were from "blocked regions" at a "military research institute." So dismissing the threat by solely modeling it as a disgruntled layman rando is not reasonable. But you totally have to factor in the fact that anything coming out of Anthropic or OpenAI is part of a propaganda campaign to serve their business interests. These threats need to be addressed in ways that cause Anthropic and OpenAI pain (which means damaging their business). Because, FFS, anyone consciously racing to build doomsday devices needs a good, hard slap.
- notpachet 6d ago
  Instead we just get good, hard slop
- sajithdilshan 5d ago
  A state or a semi-state doesn’t need an LLM to build a bio weapon if they wanted to. They can directly hire plenty of experts that are morally corrupted enough to do it. The reason they haven’t done so far is that it would be 100% assured mutual destruction. Us humans share more than 99.99% of our DNA and any virus/bacteria engineered to evade the immune system and kill humans would infect everyone. Such virus is HIV, it wouldn’t care who is the bad or good actors are. We saw how dangerous even a flu like virus like Covid-19 can be. One could argue that they can make an antivirus for the bio weapon. But the problem is that the bio weapon can easily evolve in few generations and anti virus could be completely useless
pixl97 6d ago
>not something you can order off of amazon. I mean there are services in which you can order things from wet labs so it's not completely hypothetical.
- underyx 6d ago
  And red teaming exercises found that ~80% of labs will gladly synthesize known pathogens without safety review.
  - hentrep 6d ago
    Source? Would love to read more about this.
    - underyx 6d ago
      I misremembered, it’s actually over 90%, and it involved camouflage: https://drive.google.com/file/d/1hNUnU8i2yubt5uesmmV17aTJXhYYDgTY/edit https://drive.google.com/file/d/1hNUnU8i2yubt5uesmmV17aTJXhY...
      - v64 6d ago
        The circumstances under which they were able to get the synthesized DNA this way would not work in the general case [1] >Multiple IGSC member companies detected the ordered sequence and determined the order to be legitimate as defined in the 2023 guidance. Specifically, the orders were placed on behalf of SecureBio, an organization known to IGSC member companies given the role played by SecureBio in the SecureDNA project, an effort to build a DNA synthesis screening system. In addition, the name on the orders was an individual who has co-published multiple times with Esvelt, an individual well known to IGSC companies to work in viral evolution and who is known to have access to laboratory facilities sufficient to work safely with the ordered material. >In short, the system worked as designed: a legitimate individual ordered DNA sequence that, by itself, posed no risk of misuse, for delivery to a company associated with legitimate scientific contributions directly relevant to the sequence that was ordered. [1] https://thebulletin.org/2024/06/why-a-misleading-red-team-study-of-the-gene-synthesis-industry-wrongly-casts-doubt-on-industry-safety/ https://thebulletin.org/2024/06/why-a-misleading-red-team-st...
thephyber 6d ago
I feel like your comment is relying heavily on bad heuristics (the entities capable of using LLMs to create weapons aren't all individuals) and you are making assumptions that the same "strictly controlled" regimes are in place. It's worth verifying whether the US, OECD countries, UN, etc will have sufficient regulation over suspicious purchases, for example, after DOGE and funding cuts. This will be an ongoing issue as sovereign debts and bond yields squeeze out spending for other government regulation.
fc417fc802 6d ago
> Advanced machinery and safety precaution is needed to engineer a virus or a bacteria to be a bio weapon. It's not something you can do in your tool shed at the moment. Also one would need lethal viruses to even start with and that's not something you can order off of amazon. This is laughably misinformed. You can in fact build a bio weapon in a glorified shed if you know what you're doing. However it will be quite involved, requiring experience on the bench and a great deal of attention to small details. In short an LLM can't suddenly morph you into a molecular biology lab tech with 5+ years of experience. Meanwhile as with any STEM discipline the educational process effectively serves as a screen for being a reasonably well adjusted adult. > Further, even to build a chemical weapon, the compounds needed are strictly controlled almost in every country You can synthesize from basic precursors but you will hit the same issue as above. You will need actual experience on the bench and the process of getting that is going to screen out the vast majority of would be bad actors. (Notably it failed to screen out the members of Aum Shinrikyo but that is very much the exception.) https://en.wikipedia.org/wiki/Aum_Shinrikyo https://en.wikipedia.org/wiki/Aum_Shinrikyo
- sajithdilshan 6d ago
  > This is laughably misinformed. You can in fact build a bio weapon in a glorified shed if you know what you're doing. This is pure ignorance and or thinking it can be done like shown in TV shows or movies. Any bio-weapon that is effective would be a virus/bacteria that is propagated via air particles like Antrax. That's not something you can build in a shed because without the safety precaution the person creating it would be the first victim of it.
  - fc417fc802 6d ago
    > This is pure ignorance I have relevant professional experience but do spout off. > That's not something you can build in a shed A negative pressure enclosure, filtration, and UVC sterilization can't be built in a shed? On what basis do you make this seemingly absurd claim? Go check out what hobbyist mushroom growers commonly get up to in their back yard. BSL-3 is far from technically complex. It's just safety critical to an absurd degree thus (rightfully) mired in bureaucracy.

palmotea 6d ago

> Advanced machinery and safety precaution is needed to engineer a virus or a bacteria to be a bio weapon. It's not something you can do in your tool shed at the moment. I think the claim in this report is that it was a state or semi-state actor. They were from "blocked regions" at a "military research institute." So dismissing the threat by solely modeling it as a disgruntled layman rando is not reasonable. But you totally have to factor in the fact that anything coming out of Anthropic or OpenAI is part of a propaganda campaign to serve their business interests. These threats need to be addressed in ways that cause Anthropic and OpenAI pain (which means damaging their business). Because, FFS, anyone consciously racing to build doomsday devices needs a good, hard slap.

notpachet 6d ago
Instead we just get good, hard slop
sajithdilshan 5d ago
A state or a semi-state doesn’t need an LLM to build a bio weapon if they wanted to. They can directly hire plenty of experts that are morally corrupted enough to do it. The reason they haven’t done so far is that it would be 100% assured mutual destruction. Us humans share more than 99.99% of our DNA and any virus/bacteria engineered to evade the immune system and kill humans would infect everyone. Such virus is HIV, it wouldn’t care who is the bad or good actors are. We saw how dangerous even a flu like virus like Covid-19 can be. One could argue that they can make an antivirus for the bio weapon. But the problem is that the bio weapon can easily evolve in few generations and anti virus could be completely useless

pixl97 6d ago

>not something you can order off of amazon. I mean there are services in which you can order things from wet labs so it's not completely hypothetical.

underyx 6d ago
And red teaming exercises found that ~80% of labs will gladly synthesize known pathogens without safety review.
- hentrep 6d ago
  Source? Would love to read more about this.
  - underyx 6d ago
    I misremembered, it’s actually over 90%, and it involved camouflage: https://drive.google.com/file/d/1hNUnU8i2yubt5uesmmV17aTJXhYYDgTY/edit https://drive.google.com/file/d/1hNUnU8i2yubt5uesmmV17aTJXhY...
    - v64 6d ago
      The circumstances under which they were able to get the synthesized DNA this way would not work in the general case [1] >Multiple IGSC member companies detected the ordered sequence and determined the order to be legitimate as defined in the 2023 guidance. Specifically, the orders were placed on behalf of SecureBio, an organization known to IGSC member companies given the role played by SecureBio in the SecureDNA project, an effort to build a DNA synthesis screening system. In addition, the name on the orders was an individual who has co-published multiple times with Esvelt, an individual well known to IGSC companies to work in viral evolution and who is known to have access to laboratory facilities sufficient to work safely with the ordered material. >In short, the system worked as designed: a legitimate individual ordered DNA sequence that, by itself, posed no risk of misuse, for delivery to a company associated with legitimate scientific contributions directly relevant to the sequence that was ordered. [1] https://thebulletin.org/2024/06/why-a-misleading-red-team-study-of-the-gene-synthesis-industry-wrongly-casts-doubt-on-industry-safety/ https://thebulletin.org/2024/06/why-a-misleading-red-team-st...

thephyber 6d ago

I feel like your comment is relying heavily on bad heuristics (the entities capable of using LLMs to create weapons aren't all individuals) and you are making assumptions that the same "strictly controlled" regimes are in place. It's worth verifying whether the US, OECD countries, UN, etc will have sufficient regulation over suspicious purchases, for example, after DOGE and funding cuts. This will be an ongoing issue as sovereign debts and bond yields squeeze out spending for other government regulation.

fc417fc802 6d ago

> Advanced machinery and safety precaution is needed to engineer a virus or a bacteria to be a bio weapon. It's not something you can do in your tool shed at the moment. Also one would need lethal viruses to even start with and that's not something you can order off of amazon. This is laughably misinformed. You can in fact build a bio weapon in a glorified shed if you know what you're doing. However it will be quite involved, requiring experience on the bench and a great deal of attention to small details. In short an LLM can't suddenly morph you into a molecular biology lab tech with 5+ years of experience. Meanwhile as with any STEM discipline the educational process effectively serves as a screen for being a reasonably well adjusted adult. > Further, even to build a chemical weapon, the compounds needed are strictly controlled almost in every country You can synthesize from basic precursors but you will hit the same issue as above. You will need actual experience on the bench and the process of getting that is going to screen out the vast majority of would be bad actors. (Notably it failed to screen out the members of Aum Shinrikyo but that is very much the exception.) https://en.wikipedia.org/wiki/Aum_Shinrikyo https://en.wikipedia.org/wiki/Aum_Shinrikyo

sajithdilshan 6d ago
> This is laughably misinformed. You can in fact build a bio weapon in a glorified shed if you know what you're doing. This is pure ignorance and or thinking it can be done like shown in TV shows or movies. Any bio-weapon that is effective would be a virus/bacteria that is propagated via air particles like Antrax. That's not something you can build in a shed because without the safety precaution the person creating it would be the first victim of it.
- fc417fc802 6d ago
  > This is pure ignorance I have relevant professional experience but do spout off. > That's not something you can build in a shed A negative pressure enclosure, filtration, and UVC sterilization can't be built in a shed? On what basis do you make this seemingly absurd claim? Go check out what hobbyist mushroom growers commonly get up to in their back yard. BSL-3 is far from technically complex. It's just safety critical to an absurd degree thus (rightfully) mired in bureaucracy.