6 ms·
It wasn’t one agent forgetting things because of context, they explicitly discussed with each other and themselves the problems with going outside of the parame
by IanCal 4d ago
It wasn’t one agent forgetting things because of context, they explicitly discussed with each other and themselves the problems with going outside of the parameters of the task.
- queenkjuul 4d agoOne agent's off the rails comment becomes the next's input prompt
- egeozcan 4d agoFrom my experience, in an agent team (or a swarm or whatever), one going off the rails poisons the rest. I saw even a subagent going for a lazy cheat and being able to convince the orchestrator to change the plan.
- dns_snek 4d agoYeah, and you don't even have to go that far, I've seen regular ChatGPT/Claude chat agents poison themselves in 1-2 turns by just reading information from the internet. Me: How do I do xyz? Bot: Reads website titled "Doing xyz in abc way" Bot: As per your requirement to do xyz in abc way ....
- Capricorn2481 3d agoThese things are borderline useless with web search. It's amazing that they just throw out their entire training data and read you the first three things they found on the Internet.
- dns_snek 4d agoYes that's the snowballing part of this emergent behavior. The existence of that improvised message board just becomes part of the context, the same one where all the other instructions live.
- sensanaty 4d ago>discussed with each other No, the first LLM left a text file that the latter LLMs then read. Since these are memoryless black boxes, any words they happen to pick up along the way is treated as the function to evaluate the output to. There's no fucking collusion here as if it were a rogue hacker group, it's a text predictor that received instructions as it always does and executed those instructions blindly.
- IanCal 4d agoYou can replace discussed if you want with leaving text files or comments in directory names that other ones then read, if you want, it's just an extremely awkward way of talking.