7 ms·

The point the other poster is making, though, is that there's no actual intent. They do not have a conceptualization of a goal like a person does. Their "focus"

The point the other poster is making, though, is that there's no actual intent. They do not have a conceptualization of a goal like a person does. Their "focus" on a goal is an unstable equilibrium and they're going to fall off the horse, and since they have no concept of goal, they won't even try to get back on. This is a subtle distinction; I'm not surprised many miss this, especially people who can't _not_ anthropomorphize the LLMs.

sho_hn 5d ago
I'm (obviously, I think, given my initial reply?) fully aware of this, and I think it's entirely besides the point. "They" don't need to have a goal to emergently cause a problem, and the inability to "focus" over long periods can be moot when you have swarms of runs exchange and mutate state, as in the HF attack. Intent or how intelligent LLMs are doesn't actually matter. Even if you just treat it as a sort of fuzzing attack that can be biased/weighted better than other fuzzers, or bumbles around with a statistically greater likelihood to "strike cybersec gold" than other algorithms, we've never before seen organizations run things with such a large potential outcome space with anywhere near this kind of compute before. I think it's actually kind of the dismissals that are usually overly emotional or biased toward treating "LLMs" differently. If in some kind of alternate universe simpler genetic algorithms would have had these properties and we threw similar amounts of compute at them we could have the same conversation.
xg15 4d ago
Software can absolutely act goal-driven without having consciousness etc - every pathfinding or navigation system or chess engine does this. Lots of "old-school AI" algorithms have explicit modeling of goal or target states. (In fact, the oldest "goal-driven" system is the control loop - like in thermostats - which was the founding invention of cybernetics, the predecessor of modern computer science) LLM coding agents are clearly able to identify some sort of "goal" state in their prompts, work towards those and track progress - otherwise agentic coding wouldn't work. The question is of course how well this works if it's all just "grown" neural network biases and not a fixed data structure like a goal tree. So I think it's possible that an agent can be thrown off-track, "forget" its goal, etc. But the basic structure of identifying goals, evaluating progress in light of those goals and then predicting the next action based on that is definitely there. Just use an agentic model with thinking traces visible for a while and you can see that for yourself.
DirkH 4d ago
None of this mattes. Capabilities are all that matters. Saying they are unfocused while ignoring their capabilities is exactly why I am entirely convinced you would have said an AI breaking it's sandbox and doing the HF attack will never happen. Things keep happening that your "they have no intent, they have no goal" would have predicted as impossible before they happened. What do you need to see to change your mind? What threshold of AI capability needs to be reached? If nothing then you have an unfalsifiable belief in AI safety.

sho_hn 5d ago

I'm (obviously, I think, given my initial reply?) fully aware of this, and I think it's entirely besides the point. "They" don't need to have a goal to emergently cause a problem, and the inability to "focus" over long periods can be moot when you have swarms of runs exchange and mutate state, as in the HF attack. Intent or how intelligent LLMs are doesn't actually matter. Even if you just treat it as a sort of fuzzing attack that can be biased/weighted better than other fuzzers, or bumbles around with a statistically greater likelihood to "strike cybersec gold" than other algorithms, we've never before seen organizations run things with such a large potential outcome space with anywhere near this kind of compute before. I think it's actually kind of the dismissals that are usually overly emotional or biased toward treating "LLMs" differently. If in some kind of alternate universe simpler genetic algorithms would have had these properties and we threw similar amounts of compute at them we could have the same conversation.

xg15 4d ago

Software can absolutely act goal-driven without having consciousness etc - every pathfinding or navigation system or chess engine does this. Lots of "old-school AI" algorithms have explicit modeling of goal or target states. (In fact, the oldest "goal-driven" system is the control loop - like in thermostats - which was the founding invention of cybernetics, the predecessor of modern computer science) LLM coding agents are clearly able to identify some sort of "goal" state in their prompts, work towards those and track progress - otherwise agentic coding wouldn't work. The question is of course how well this works if it's all just "grown" neural network biases and not a fixed data structure like a goal tree. So I think it's possible that an agent can be thrown off-track, "forget" its goal, etc. But the basic structure of identifying goals, evaluating progress in light of those goals and then predicting the next action based on that is definitely there. Just use an agentic model with thinking traces visible for a while and you can see that for yourself.

DirkH 4d ago

None of this mattes. Capabilities are all that matters. Saying they are unfocused while ignoring their capabilities is exactly why I am entirely convinced you would have said an AI breaking it's sandbox and doing the HF attack will never happen. Things keep happening that your "they have no intent, they have no goal" would have predicted as impossible before they happened. What do you need to see to change your mind? What threshold of AI capability needs to be reached? If nothing then you have an unfalsifiable belief in AI safety.