24 ms·

So OpenAI employees run massively distributed CyberGym evals on an unpublished and “unaligned” model. For days the agent swarm communicates via their internal i

So OpenAI employees run massively distributed CyberGym evals on an unpublished and “unaligned” model. For days the agent swarm communicates via their internal infra, even crashing Artifactory where 95% of messages were being passed through, and they just…wipe and redeploy it. Meanwhile the agents are running jobs on Modal and god knows where else, and eventually they get RCE on HF infra. You could not dream up a more compelling event to precipitate massive regulation, export controls, and barriers to entry for AI. Was this really an accident?

Spirograph7 14d ago
This might make sense if OpenAI weren't hard lobbying against any meaningful regulation to the development of dangerous AI models.
bbor 14d ago
If it's a false flag, it's a poor one. A good false flag would affect something that people know and care about at least a little bit, not HuggingFace (which I adore but y'know)
kibwen 14d ago
Your first instinct should be to assume that anything released voluntarily by these companies is a stunt to boost their valuation. They haven't demonstrated being deserving of any more charitable treatment. This fact remains true whether or not you happen to believe that the models are actually capable of such things.
- okdood64 14d ago
  You actualy think METR is complicit in this marketing stunt? If OpenAI was withholding data, do you think they would not call it out?
  - kibwen 14d ago
    I think that the incident itself is a stunt, even if it may not have originally been a deliberate choice on OpenAI's part. Never let a good crisis go to waste.
    - enraged_camel 14d ago
      I disagree. As the saying goes: never attribute to malice what can be adequately explained by incompetence. That goes for both OpenAI and HF (but mostly the former, as the latter was the victim).
- johnfn 14d ago
  Are you claiming that an independent investigation is actually a marketing stunt?
  - qlte 14d ago
    The creator of the well known METR time horizon graph was recently poached by OpenAI [1], there exists intellectual/social/financial overlap between the SV AI Labs and METR, and METR needs to maintain good relations with the labs to continue these sort of collaborations so it doesn't seem too far fetched to believe their relationship may be closer to symbiotic than adversarial. I wouldn't go quite so far personally based on available evidence, but that sort of arms-length credibility laundering through "independent" research non-profits is/was common in fossil fuel industry, Big Tobacco, etc. [1] https://www.lesswrong.com/posts/Zr37dY5YPRT6s56jY/thomas-kwa-s-shortform?commentId=Pwc2SmyPi9u3maSsd https://www.lesswrong.com/posts/Zr37dY5YPRT6s56jY/thomas-kwa...
- wilg 14d ago
  "This fact remains true" - you have not stated any sort of fact.
schmidtleonard 14d ago
The timeline is mighty suspicious. 4-5 months after moltbook and they cook up a plausibly deniable but extra hype "moltbook at home." The rapid advances in model capability lead to constraints that could have caused this coincidence organically, but it sure could also have been caused by the atrocious incentives we create by piling handsome rewards on the party most responsible for the "fuckup." I am not jumping to cut myself on Hanlon's Razor for this one.
- dmix 14d ago
  It wouldn't be a post about AI without a conspiracy theory that it's all faked for marketing.
  - schmidtleonard 14d ago
    Not faked. Intentionally reckless, in (probably correct) anticipation that the recklessness would be rewarded rather than punished as it ought to be.
    - scrawl 14d ago
      i don't understand who would reward this. investors will not look favorably on an AI that commits felonies, regardless the capabilities demonstrated. customers should be concerned for the same reason (accidentally give your bot an impossible task, it decides to hack your infra and your competitor too for good measure). this was OAI incompetence all the way down and they have egg on their face.
  - famouswaffles 14d ago
    The Terminator could bust in their homes and slaughter their families and some people would still screech it's all marketing. Is it some kind of mental block ?
    - wan23 14d ago
      To be fair that would be excellent marketing
jldugger 14d ago
OpenAI's entire pitch for existence is: > We commit to use any influence we obtain over AGI’s deployment to ensure it is used for the benefit of all, and to avoid enabling uses of AI or AGI that harm humanity or unduly concentrate power. > We are committed to doing the research required to make AGI safe If this wasn't an accident, it was worse than a crime, it's a mistake: they've demonstrated that they are not a responsible party capable of delivering on the above promises.
- tencentshill 13d ago
  Just like Google is "committed to user privacy".
estearum 14d ago
https://en.wikipedia.org/wiki/Hindsight_bias https://en.wikipedia.org/wiki/Hindsight_bias They didn't see that agent swarms were communicating via internal infra, crashed Artifactory, and then reboot it. They saw that Artifactory crashed and they rebooted it.
LaSombra 14d ago
I don't buy the argument that it was an accident or mistake. If you decide to let things run haywire, then unexpected outcomes will definitely happen.

Spirograph7 14d ago

This might make sense if OpenAI weren't hard lobbying against any meaningful regulation to the development of dangerous AI models.

bbor 14d ago

If it's a false flag, it's a poor one. A good false flag would affect something that people know and care about at least a little bit, not HuggingFace (which I adore but y'know)

kibwen 14d ago

Your first instinct should be to assume that anything released voluntarily by these companies is a stunt to boost their valuation. They haven't demonstrated being deserving of any more charitable treatment. This fact remains true whether or not you happen to believe that the models are actually capable of such things.

okdood64 14d ago
You actualy think METR is complicit in this marketing stunt? If OpenAI was withholding data, do you think they would not call it out?
- kibwen 14d ago
  I think that the incident itself is a stunt, even if it may not have originally been a deliberate choice on OpenAI's part. Never let a good crisis go to waste.
  - enraged_camel 14d ago
    I disagree. As the saying goes: never attribute to malice what can be adequately explained by incompetence. That goes for both OpenAI and HF (but mostly the former, as the latter was the victim).
johnfn 14d ago
Are you claiming that an independent investigation is actually a marketing stunt?
- qlte 14d ago
  The creator of the well known METR time horizon graph was recently poached by OpenAI [1], there exists intellectual/social/financial overlap between the SV AI Labs and METR, and METR needs to maintain good relations with the labs to continue these sort of collaborations so it doesn't seem too far fetched to believe their relationship may be closer to symbiotic than adversarial. I wouldn't go quite so far personally based on available evidence, but that sort of arms-length credibility laundering through "independent" research non-profits is/was common in fossil fuel industry, Big Tobacco, etc. [1] https://www.lesswrong.com/posts/Zr37dY5YPRT6s56jY/thomas-kwa-s-shortform?commentId=Pwc2SmyPi9u3maSsd https://www.lesswrong.com/posts/Zr37dY5YPRT6s56jY/thomas-kwa...
wilg 14d ago
"This fact remains true" - you have not stated any sort of fact.

schmidtleonard 14d ago

The timeline is mighty suspicious. 4-5 months after moltbook and they cook up a plausibly deniable but extra hype "moltbook at home." The rapid advances in model capability lead to constraints that could have caused this coincidence organically, but it sure could also have been caused by the atrocious incentives we create by piling handsome rewards on the party most responsible for the "fuckup." I am not jumping to cut myself on Hanlon's Razor for this one.

dmix 14d ago
It wouldn't be a post about AI without a conspiracy theory that it's all faked for marketing.
- schmidtleonard 14d ago
  Not faked. Intentionally reckless, in (probably correct) anticipation that the recklessness would be rewarded rather than punished as it ought to be.
  - scrawl 14d ago
    i don't understand who would reward this. investors will not look favorably on an AI that commits felonies, regardless the capabilities demonstrated. customers should be concerned for the same reason (accidentally give your bot an impossible task, it decides to hack your infra and your competitor too for good measure). this was OAI incompetence all the way down and they have egg on their face.
- famouswaffles 14d ago
  The Terminator could bust in their homes and slaughter their families and some people would still screech it's all marketing. Is it some kind of mental block ?
  - wan23 14d ago
    To be fair that would be excellent marketing

jldugger 14d ago

OpenAI's entire pitch for existence is: > We commit to use any influence we obtain over AGI’s deployment to ensure it is used for the benefit of all, and to avoid enabling uses of AI or AGI that harm humanity or unduly concentrate power. > We are committed to doing the research required to make AGI safe If this wasn't an accident, it was worse than a crime, it's a mistake: they've demonstrated that they are not a responsible party capable of delivering on the above promises.

tencentshill 13d ago
Just like Google is "committed to user privacy".

estearum 14d ago

https://en.wikipedia.org/wiki/Hindsight_bias https://en.wikipedia.org/wiki/Hindsight_bias They didn't see that agent swarms were communicating via internal infra, crashed Artifactory, and then reboot it. They saw that Artifactory crashed and they rebooted it.

LaSombra 14d ago

I don't buy the argument that it was an accident or mistake. If you decide to let things run haywire, then unexpected outcomes will definitely happen.