9 ms·

I tried to address safety, albeit briefly, in the sections on backdoors and propaganda. What would you have liked to see? I didn't give it much treatment becau

I tried to address safety, albeit briefly, in the sections on backdoors and propaganda. What would you have liked to see? I didn't give it much treatment because my frame of reference is about open vs closed AI, and I don't see a lot of daylight in safety concerns between the two.

adahn 2mo ago
The core safety-related arguments for closed weights are: 1. Visibility. API providers can monitor for misuse, and regulators only need to oversee a few centralised players. Once weights are released, anyone with sufficient hardware (foreign governments, criminal groups, terrorists?) can run them and monitoring is no longer feasible. 2. Revocability and containment. If closed-weights models are found to be dangerous, access can be withdrawn. Deployments that are autonomously causing harm are easier to contain. 3. Guardrails. Safety fine-tuning and classifiers can be removed from open-weights models, enabling harmful use that would not be possible via API.
- marcus_holmes 2mo ago
  I trust open source code more than I trust proprietary code. And I believe history vindicates me. Everything that you're saying can be done with open-weights models by bad people can also be done with closed-weights models by the people who own them. And there is no defence against that, we are forced to trust that the owners are not bad people. Those owners have proven time and again that they are only interested in profit and/or control. They will do whatever they think makes them the most money, or gives them the most control. They will also lie about this. At least with open weights we know the danger and can do something about it. With closed weights we don't know what's going on.
  - adahn 2mo ago
    >Everything that you're saying can be done with open-weights models by bad people can also be done with closed-weights models by the people who own them. The latter is a much smaller group of people located in a limited number of countries. It's clearly easier to keep a limited number of entities in check than a wider group that might include 100s of thousands to millions of entities. >And there is no defence against that, we are forced to trust that the owners are not bad people. Yes there is; Regulation and oversight. As all developed nations do with technologies that present a risk to public safety. >At least with open weights we know the danger We would have a better understanding of the threat in some ways, but worse in others. Having access to open weights models gives us no insight into how and where the models are being modified, deployed, and used elsewhere.
    - marcus_holmes 2mo ago
      I don't think the regulation and oversight capability of the US government is anywhere near what it needs to be to ensure everyone's safety. There are places in the USA where you cannot drink the tapwater ffs.
      - no-name-here 2mo ago
        > I don't think the regulation and oversight capability of the US government is anywhere near what it needs to be Are you saying you'd prefer stronger government regulations/oversight for AI?
        marcus_holmes 2mo ago
        No, I'm saying I don't trust organisations with a profit motive to build things that benefit the rest of us, and don't trust the government to regulate them properly, and think that the better move would be to open-source the whole stack so we can see what's going on for ourselves.

adahn 2mo ago

The core safety-related arguments for closed weights are: 1. Visibility. API providers can monitor for misuse, and regulators only need to oversee a few centralised players. Once weights are released, anyone with sufficient hardware (foreign governments, criminal groups, terrorists?) can run them and monitoring is no longer feasible. 2. Revocability and containment. If closed-weights models are found to be dangerous, access can be withdrawn. Deployments that are autonomously causing harm are easier to contain. 3. Guardrails. Safety fine-tuning and classifiers can be removed from open-weights models, enabling harmful use that would not be possible via API.

marcus_holmes 2mo ago
I trust open source code more than I trust proprietary code. And I believe history vindicates me. Everything that you're saying can be done with open-weights models by bad people can also be done with closed-weights models by the people who own them. And there is no defence against that, we are forced to trust that the owners are not bad people. Those owners have proven time and again that they are only interested in profit and/or control. They will do whatever they think makes them the most money, or gives them the most control. They will also lie about this. At least with open weights we know the danger and can do something about it. With closed weights we don't know what's going on.
- adahn 2mo ago
  >Everything that you're saying can be done with open-weights models by bad people can also be done with closed-weights models by the people who own them. The latter is a much smaller group of people located in a limited number of countries. It's clearly easier to keep a limited number of entities in check than a wider group that might include 100s of thousands to millions of entities. >And there is no defence against that, we are forced to trust that the owners are not bad people. Yes there is; Regulation and oversight. As all developed nations do with technologies that present a risk to public safety. >At least with open weights we know the danger We would have a better understanding of the threat in some ways, but worse in others. Having access to open weights models gives us no insight into how and where the models are being modified, deployed, and used elsewhere.
  - marcus_holmes 2mo ago
    I don't think the regulation and oversight capability of the US government is anywhere near what it needs to be to ensure everyone's safety. There are places in the USA where you cannot drink the tapwater ffs.
    - no-name-here 2mo ago
      > I don't think the regulation and oversight capability of the US government is anywhere near what it needs to be Are you saying you'd prefer stronger government regulations/oversight for AI?
      - marcus_holmes 2mo ago
        No, I'm saying I don't trust organisations with a profit motive to build things that benefit the rest of us, and don't trust the government to regulate them properly, and think that the better move would be to open-source the whole stack so we can see what's going on for ourselves.