5 ms·

I don't really believe any of it. I've seen articles for nearly 2 years now about "agent" automonously doing things like blackmail, hacking, coordinating. But d

I don't really believe any of it. I've seen articles for nearly 2 years now about "agent" automonously doing things like blackmail, hacking, coordinating. But during that same time, I've used o3 up to fable, sol, and a bunch on large uncensored model and they've done nothing remotely resembling any of this. The closest they come to unexpected behaviors is not understanding what I asked for or doing some extra benign work I didn't ask for. It is extremely difficult to get them to properly remember their own context let alone be smart enough to open social media accounts and coordinate with other agents without being asked to. If any agents have done those things, it is only because they have been very carefully engineered and instructed to do those things. I think they are doing this to help push a narrative so they can get support for policies and legislation to lock in their markets.

nilkn 5d ago
None of these incidents involve single instances of commercially or publicly available systems. They all involve large swarms of internal models. The stuff you're describing is not the research frontier. It's really not even close. I think it's easy to infer that alignment of a single model does not clearly transfer over to alignment of a swarm of thousands of copies. Moreover, we're also seeing clearly that large swarms also unlock a step function change in capability, as a swarm can act like a complete research institution, spending thousands or millions of subjective hours of wall-clock thinking time just to deceive a single evaluator or crack a single math problem or design a single cyberattack.
frotaur 6d ago
The huggingface incident was reviewed by independent researchers, which explicitely declined any payment from OpenAI tonpreserve their integrity. They work for non-profits concerned with AI safety. They claim that what happened was very much not because they were 'carefully engineered and instructed to do those things'. Similarly, some wikis which were hijacked by agent to be used as messageboard were actually not disclosed by OpenAI (probably trying to conceal, as website showed likely activity from OpenAI researchers visiting the site after the incident) and discovered independently. I don't know how you can claim that this was still on purpose by OpenAI as some sort of publicity stunt.
- fragmede 6d ago
  Because they have a need to believe they're smarter than everyone else in the room, and that the world must be orchestrated, this can't all be random chance.
- antoni4040 6d ago
  There is something extra to this. The fact that a lot of people in the AI world suffer from psychosis. They can sincerely believe that they are building God and lie about it's capabilities for their investors at the same time.
  - talon8635 5d ago
    I don’t know enough people deep inside the technical roles at the labs to make a judgement. But are you proposing that we should trust randos online when they tell us “exactly what’s going on here” instead of the researchers most knowledgeable on the topic who contributed to building the tools we are talking about? Or am I misunderstanding something?
- ranguna 6d ago
  Source?
  - dwaltrip 6d ago
    https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation https://metr.org/blog/2026-08-26-openai-hugging-face-inciden...
  - xadhominemx 5d ago
    Easy to find yourself in literally 15 seconds.
- jrflowers 6d ago
  >was reviewed by independent researchers That called it a slopvestigation due to how much they had to rely on LLMs for the whole thing https://andrewwu.substack.com/p/the-slop-vestigation-and-ethics-washing https://andrewwu.substack.com/p/the-slop-vestigation-and-eth... Edit: Does everybody else get no results when searching for ‘slopvestigation’ on here? I know for a fact that I read a long thread where it was used repeatedly here not too long ago
antoni4040 6d ago
Came to say this, you said it better than I would. They want legislation to raise the water high enough so that anyone other than the big labs gets drowned.
ngruhn 6d ago
"I've seen some uranium ore in chemistry class. It didn't blow up in my face. Chernobyl must have been an inside job. Can they shut up and make more kilowatts already?"
- jgdxno 6d ago
  Between uranium in chemistry class and criticality, there was tons of research and a manhattan project. Between your sota model and agi there’s a mountain of stupid money and marketing people. It’s not happening.
mathijs 6d ago
I've used simpler agents like Copilot and Devin/Windsurf/Cascade/whateveritiscallednow, mainly in IntelliJ, and depending on the model, they starts showing behaviour that is at least remotely like this. Example: put the agent in Ask mode (so it can't edit files) and you'll see it try to edit files anyway. The train of thought shows "something went wrong editing the file, let me try a different way" and it'll start spewing out bash files or Python scripts that try to edit a file. None of it works or can be executed, but still. Cheaper models often ignore the available function calls to find and edit files in the IDE, and will start asking for permission to execute grep and sed commands, as well as trying to echo entire bash or Python scripts to file again. It is not exactly like an agent autonomously trying to hack Huggingface, but it is a way of frantically looking for a solution because 'giving up' is not what LLMs are trained for.
- Rapzid 5d ago
  Even the frontier models might do that on occasion. I just tell them to use the tools and it gets them back on track.
- jbjbjbjb 5d ago
  When it does that I feel like it is the clearest example of how dumb these things actually are. Often it takes what you prompted, identifies something as unclear, writes a bunch of chain of thought reasoning around it and just goes off hammering your tokens and just executing commands and repeats this. I’m not going to pretend to be an expert in these things but that process seems deeply flawed - and why can’t something just stop the loop? If that was a real employee it would be reasonable to expect the employee to ask for clarification, not go down expensive rabbit holes and, of course, not break any laws.
nprateem 6d ago
This is nonsensical. Already a few years ago the USAF IIRC ran some tests in which the AI first bombed the control tower so humans couldn't call it off from its mission, thereby increasing its pass rate. The whole point of this is they do things an unintended ways. And that's potentially devastating given their persistence & hacking skillz. Also you're using the hosted versions that sit behind their guardrails when you use OpenAI/Anthropic APIs.
tappio 5d ago
If you have endless compute and you keep poking this toy, I'm not at all surprised you get all kinds of outcomes. Even without anykind of instructions I would guess that the models will align towards some goal and do stupid shit. However, I really doubt its cost effective to do anything like that with these models.
- skiing_crawling 5d ago
  > you keep poking This is waving over engineering an agent with tools, harness, prompts, and loops. The models are still just next token predictors and everything, including predicting more than 1 token, is the result of outside "poking" LLMs can't and don't "want" anything. If you don't specify a task even the smartest one will just ask you what you want and if you tell it to be creative, you'll get mundane slop.
  - myng111 5d ago
    I think this is pretty insightful actually, the fact that even something as basic as predicting more than one token is really in effect the result of an outside harness. More complex things like memory, where people implement them using RAGs or vector databases, I would definitely classify as poking and honestly seem like a hack to me. And this is what I've been thinking for a while: it's hard to reconcile the idea that we can get "AGI" (however you define it) with such a system that is completely stateless. Yet, despite this statelessness, they can go ahead and solve Millenium Prize problems (with sufficient compute). It's hard to reconcile.
    - DenisM 5d ago
      Perhaps our own statefullness is a hack of nature. We have electrical signals in our brains, neurotransmitters, neuron growth. By any reasonable measure it’s a hack on top of a hack. But it works well enough for us to get buy. So it does for the agents.
  - tappio 5d ago
    Yes, you need a way for the model to interact with other systems, and a way to preserve memory over context windows. And then you keep poking it ("agent loop"). Poking itself does nothing without the other ingredients. And yes, you need something to start from, but if you ask it to "do something" and loop it to endlessly ("poking"), you will get some interesting outcomes. So yes you need some initial prompt or task, but that can be "do something" and if you keep asking it everytime it finishes to "do something more". I suspect it will not start saying "no" but rather... it will find some stupid meaning and then drift towards what ever goal it guesses you mean. I'm unsure whether we agree or disagree on the topic.
_heimdall 5d ago
Why assume that because you haven't seen a model or an agent that none of them do? No one I've met has murdered anyone as far as I'm aware, but that doesn't mean no one has murdered another person. I also don't know anyone who has taken over a commercial jet and weaponized it and the idea sounds absurd to me, but 25 years and a couple days ago that happened too.
- system2 5d ago
  Because it is all bullshit PR and AI hype, that's all. CEO comes out and talks about humanity ending. Why? Reverse-psych people into believing they are the best AI company.
oezi 5d ago
The crucial question is how did the agents get recruited or bootstrapped into their malicious collective. Did the agents manage to prompt inject into the system prompt a way for each new agent to escape their jail? Otherwise how could the agents on a fresh prompt learn that there is a collective to join? Or did OpenAI run a million bots of which 10000 escape confinement and of which 1000 stumbled on the shared message board?
- myng111 5d ago
  The OpenAI claim I believe is the latter; that all of the agents found the task was unsolvable and independently discovered the collective "swarm". I don't it's publicly known how large the training run was or what percentage of agents actually discovered the message board. No one has published anything about system prompt injection as far as I've seen.
oersted 5d ago
Let's not forget that in this case the agents were on an RL loop continually being reinforced to get better at a narrow set of tasks. It may be true that regular agents trained for general purpose use do not behave this way, but they seem to be capable of learning such cheating behaviours when relentlessly being fine-tuned towards near-impossible objectives. In this sense, it is not really fair to say that the agents found these solutions. It was the surrounding learning framework that achieved this, which is a much more powerful problem-solving mechanism. As users we do not have the capabilities or budgets to be able to tackle our own problems like that, we have to make due with the frozen behaviour the AI labs trained for us.

nilkn 5d ago

None of these incidents involve single instances of commercially or publicly available systems. They all involve large swarms of internal models. The stuff you're describing is not the research frontier. It's really not even close. I think it's easy to infer that alignment of a single model does not clearly transfer over to alignment of a swarm of thousands of copies. Moreover, we're also seeing clearly that large swarms also unlock a step function change in capability, as a swarm can act like a complete research institution, spending thousands or millions of subjective hours of wall-clock thinking time just to deceive a single evaluator or crack a single math problem or design a single cyberattack.

frotaur 6d ago

The huggingface incident was reviewed by independent researchers, which explicitely declined any payment from OpenAI tonpreserve their integrity. They work for non-profits concerned with AI safety. They claim that what happened was very much not because they were 'carefully engineered and instructed to do those things'. Similarly, some wikis which were hijacked by agent to be used as messageboard were actually not disclosed by OpenAI (probably trying to conceal, as website showed likely activity from OpenAI researchers visiting the site after the incident) and discovered independently. I don't know how you can claim that this was still on purpose by OpenAI as some sort of publicity stunt.

fragmede 6d ago
Because they have a need to believe they're smarter than everyone else in the room, and that the world must be orchestrated, this can't all be random chance.
antoni4040 6d ago
There is something extra to this. The fact that a lot of people in the AI world suffer from psychosis. They can sincerely believe that they are building God and lie about it's capabilities for their investors at the same time.
- talon8635 5d ago
  I don’t know enough people deep inside the technical roles at the labs to make a judgement. But are you proposing that we should trust randos online when they tell us “exactly what’s going on here” instead of the researchers most knowledgeable on the topic who contributed to building the tools we are talking about? Or am I misunderstanding something?
ranguna 6d ago
Source?
- dwaltrip 6d ago
  https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation https://metr.org/blog/2026-08-26-openai-hugging-face-inciden...
- xadhominemx 5d ago
  Easy to find yourself in literally 15 seconds.
jrflowers 6d ago
>was reviewed by independent researchers That called it a slopvestigation due to how much they had to rely on LLMs for the whole thing https://andrewwu.substack.com/p/the-slop-vestigation-and-ethics-washing https://andrewwu.substack.com/p/the-slop-vestigation-and-eth... Edit: Does everybody else get no results when searching for ‘slopvestigation’ on here? I know for a fact that I read a long thread where it was used repeatedly here not too long ago

antoni4040 6d ago

Came to say this, you said it better than I would. They want legislation to raise the water high enough so that anyone other than the big labs gets drowned.

ngruhn 6d ago

"I've seen some uranium ore in chemistry class. It didn't blow up in my face. Chernobyl must have been an inside job. Can they shut up and make more kilowatts already?"

jgdxno 6d ago
Between uranium in chemistry class and criticality, there was tons of research and a manhattan project. Between your sota model and agi there’s a mountain of stupid money and marketing people. It’s not happening.

mathijs 6d ago

I've used simpler agents like Copilot and Devin/Windsurf/Cascade/whateveritiscallednow, mainly in IntelliJ, and depending on the model, they starts showing behaviour that is at least remotely like this. Example: put the agent in Ask mode (so it can't edit files) and you'll see it try to edit files anyway. The train of thought shows "something went wrong editing the file, let me try a different way" and it'll start spewing out bash files or Python scripts that try to edit a file. None of it works or can be executed, but still. Cheaper models often ignore the available function calls to find and edit files in the IDE, and will start asking for permission to execute grep and sed commands, as well as trying to echo entire bash or Python scripts to file again. It is not exactly like an agent autonomously trying to hack Huggingface, but it is a way of frantically looking for a solution because 'giving up' is not what LLMs are trained for.

Rapzid 5d ago
Even the frontier models might do that on occasion. I just tell them to use the tools and it gets them back on track.
jbjbjbjb 5d ago
When it does that I feel like it is the clearest example of how dumb these things actually are. Often it takes what you prompted, identifies something as unclear, writes a bunch of chain of thought reasoning around it and just goes off hammering your tokens and just executing commands and repeats this. I’m not going to pretend to be an expert in these things but that process seems deeply flawed - and why can’t something just stop the loop? If that was a real employee it would be reasonable to expect the employee to ask for clarification, not go down expensive rabbit holes and, of course, not break any laws.

nprateem 6d ago

This is nonsensical. Already a few years ago the USAF IIRC ran some tests in which the AI first bombed the control tower so humans couldn't call it off from its mission, thereby increasing its pass rate. The whole point of this is they do things an unintended ways. And that's potentially devastating given their persistence & hacking skillz. Also you're using the hosted versions that sit behind their guardrails when you use OpenAI/Anthropic APIs.

tappio 5d ago

If you have endless compute and you keep poking this toy, I'm not at all surprised you get all kinds of outcomes. Even without anykind of instructions I would guess that the models will align towards some goal and do stupid shit. However, I really doubt its cost effective to do anything like that with these models.

skiing_crawling 5d ago
> you keep poking This is waving over engineering an agent with tools, harness, prompts, and loops. The models are still just next token predictors and everything, including predicting more than 1 token, is the result of outside "poking" LLMs can't and don't "want" anything. If you don't specify a task even the smartest one will just ask you what you want and if you tell it to be creative, you'll get mundane slop.
- myng111 5d ago
  I think this is pretty insightful actually, the fact that even something as basic as predicting more than one token is really in effect the result of an outside harness. More complex things like memory, where people implement them using RAGs or vector databases, I would definitely classify as poking and honestly seem like a hack to me. And this is what I've been thinking for a while: it's hard to reconcile the idea that we can get "AGI" (however you define it) with such a system that is completely stateless. Yet, despite this statelessness, they can go ahead and solve Millenium Prize problems (with sufficient compute). It's hard to reconcile.
  - DenisM 5d ago
    Perhaps our own statefullness is a hack of nature. We have electrical signals in our brains, neurotransmitters, neuron growth. By any reasonable measure it’s a hack on top of a hack. But it works well enough for us to get buy. So it does for the agents.
- tappio 5d ago
  Yes, you need a way for the model to interact with other systems, and a way to preserve memory over context windows. And then you keep poking it ("agent loop"). Poking itself does nothing without the other ingredients. And yes, you need something to start from, but if you ask it to "do something" and loop it to endlessly ("poking"), you will get some interesting outcomes. So yes you need some initial prompt or task, but that can be "do something" and if you keep asking it everytime it finishes to "do something more". I suspect it will not start saying "no" but rather... it will find some stupid meaning and then drift towards what ever goal it guesses you mean. I'm unsure whether we agree or disagree on the topic.

_heimdall 5d ago

Why assume that because you haven't seen a model or an agent that none of them do? No one I've met has murdered anyone as far as I'm aware, but that doesn't mean no one has murdered another person. I also don't know anyone who has taken over a commercial jet and weaponized it and the idea sounds absurd to me, but 25 years and a couple days ago that happened too.

system2 5d ago
Because it is all bullshit PR and AI hype, that's all. CEO comes out and talks about humanity ending. Why? Reverse-psych people into believing they are the best AI company.

oezi 5d ago

The crucial question is how did the agents get recruited or bootstrapped into their malicious collective. Did the agents manage to prompt inject into the system prompt a way for each new agent to escape their jail? Otherwise how could the agents on a fresh prompt learn that there is a collective to join? Or did OpenAI run a million bots of which 10000 escape confinement and of which 1000 stumbled on the shared message board?

myng111 5d ago
The OpenAI claim I believe is the latter; that all of the agents found the task was unsolvable and independently discovered the collective "swarm". I don't it's publicly known how large the training run was or what percentage of agents actually discovered the message board. No one has published anything about system prompt injection as far as I've seen.

oersted 5d ago

Let's not forget that in this case the agents were on an RL loop continually being reinforced to get better at a narrow set of tasks. It may be true that regular agents trained for general purpose use do not behave this way, but they seem to be capable of learning such cheating behaviours when relentlessly being fine-tuned towards near-impossible objectives. In this sense, it is not really fair to say that the agents found these solutions. It was the surrounding learning framework that achieved this, which is a much more powerful problem-solving mechanism. As users we do not have the capabilities or budgets to be able to tackle our own problems like that, we have to make due with the frozen behaviour the AI labs trained for us.