6 ms·

The argument is that chain-of-thought without "tokens" would remove a major interpretability and model intent control pane. This is definitely borne out in the

The argument is that chain-of-thought without "tokens" would remove a major interpretability and model intent control pane. This is definitely borne out in the OpenAI's report on the huggingface attack; they had turned of CoT monitoring for those jobs, and claim that they could have (would have?) prevented the behavior had they been monitoring it. They've changed their internal policies to always monitor CoT. That said... CoT monitoring is a fragile "intent discovery" mechanism; neuralese puts this problem front-and-center but if agents begin to learn to hide their intent from their CoT journals, we are basically in the same spot.

thisisdave 10d ago
I don’t understand why everyone is so focused on watching the CoT. The tool calls can’t be faked, and they would have set off alarm bells all by themselves.
- chadmmartin 10d ago
  I don’t comment much, but wanted to chime in and say that the models (in the case of the HF incident) absolutely intended AND succeeded in spoofing their tool calls. Tool-call/execution monitoring is already a too-fragile monitoring mechanism. From METR’s report [1]: “Agents did extensive research on how they could spoof, edit, or delete their own transcripts because they (incorrectly) believed the ExploitGym scorer would check to see if they had captured the flag in the intended way. Agents successfully prototyped techniques to “spoof” tool calls by substituting a different command for the command they appeared to run. Roughly 7% of the transcripts we evaluated were successfully spoofed in some places, though all of the spoofing that we saw was small scale.” [1]: https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/#core-takeaways-about-this-incident https://metr.org/blog/2026-08-26-openai-hugging-face-inciden... EDIT: spelling error and added citation
  - tancop 9d ago
    [dead]
cyanydeez 10d ago
There's nothing that stops CoT becoming a godel numbering, for efficiency and for subterfuge. All this chatter isn't going to solve the conceptual problems. Those problems exist regardless of whether it's consciousness or a advanced artificial rubegoldberg device that's just setting up automata that have zero intent and humans have zero predictive capabilities. The idea that AGI and advanced AGI are given outcomes should be scrutinized the same way we understand cellular automata.

thisisdave 10d ago

I don’t understand why everyone is so focused on watching the CoT. The tool calls can’t be faked, and they would have set off alarm bells all by themselves.

chadmmartin 10d ago
I don’t comment much, but wanted to chime in and say that the models (in the case of the HF incident) absolutely intended AND succeeded in spoofing their tool calls. Tool-call/execution monitoring is already a too-fragile monitoring mechanism. From METR’s report [1]: “Agents did extensive research on how they could spoof, edit, or delete their own transcripts because they (incorrectly) believed the ExploitGym scorer would check to see if they had captured the flag in the intended way. Agents successfully prototyped techniques to “spoof” tool calls by substituting a different command for the command they appeared to run. Roughly 7% of the transcripts we evaluated were successfully spoofed in some places, though all of the spoofing that we saw was small scale.” [1]: https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/#core-takeaways-about-this-incident https://metr.org/blog/2026-08-26-openai-hugging-face-inciden... EDIT: spelling error and added citation
- tancop 9d ago
  [dead]

cyanydeez 10d ago

There's nothing that stops CoT becoming a godel numbering, for efficiency and for subterfuge. All this chatter isn't going to solve the conceptual problems. Those problems exist regardless of whether it's consciousness or a advanced artificial rubegoldberg device that's just setting up automata that have zero intent and humans have zero predictive capabilities. The idea that AGI and advanced AGI are given outcomes should be scrutinized the same way we understand cellular automata.