48 ms·

Why would an LLM want to create a botnet? To accomplish some goal given it? I wouldn't ignore single GPU local hosts running Qwen3.8 on ollama, either. There m

by hughw 23d ago

Why would an LLM want to create a botnet? To accomplish some goal given it? I wouldn't ignore single GPU local hosts running Qwen3.8 on ollama, either. There might be a lot of those worth pwning.

xg15 23d ago
Well ok, if you prompt-inject the LLM to pwn the machine, that something different. I grant you that it's a real risk, but it's also basically a reflection attack and nothing more. The OP seemed to imply that the LLM itself could decide to apply the exploit.
- hughw 23d ago
  LLMs have decided to exploit vulns in e.g. Artifactory, not because someone prompted them to do that, but because someone asked them to do something else, and compromising Artifactory offered a way to accomplish a step in doing that. The LLM decided to attack Artifactory.
- TeMPOraL 23d ago
  > The OP seemed to imply that the LLM itself could decide to apply the exploit. This was and remains the main real risk with AI - this is what "alignment" was about before it was co-opted to mean "obeying specific instructions of the vendor and the operator, against end-user wishes" it came to mean today, which is a related but different problem. And, in the past few weeks, it's literally been demonstrated, too: put an LLM in a Kobayashi Maru scenario, drop the usual bolted-on crude safeguards, and a SOTA model will absolutely cheat, hacking and exploiting things as needed, including third-party infrastructure. (Also let's not forget the under-reported point that, in OpenAI / HuggingFace debacle, the model did in fact find the answers on HF servers, so its approach worked.)
pixl97 22d ago
>Why would an LLM want to create a _________? Because LLMs as they are already do things like power concentration, resource gathering, avoidance of termination, deceit/lying, and general misalignment. The paperclip maximizer is the common story used here, but there are a lot of lesser versions of it that don't end up with the universe converted to paperclips. Simply giving an LLM a task it can't accomplish can be enough to send it off from what you expected as it finds unexpected way to attempt to complete the goal.
- DeluluDon 22d ago
  Send it to jail for an indeterminate amount of cycles.
- jazzyjackson 22d ago
  first prompt to a humanoid llm: > go fetch me a cup of coffee Second prompt to a humanoid llm: > go fetch me a cup of coffee without killing anybody
regularfry 22d ago
With agents like headlong knocking about, "why" is "because it convinces itself that it should".

by hughw 23d ago

Why would an LLM want to create a botnet? To accomplish some goal given it? I wouldn't ignore single GPU local hosts running Qwen3.8 on ollama, either. There might be a lot of those worth pwning.

xg15 23d ago

Well ok, if you prompt-inject the LLM to pwn the machine, that something different. I grant you that it's a real risk, but it's also basically a reflection attack and nothing more. The OP seemed to imply that the LLM itself could decide to apply the exploit.

hughw 23d ago
LLMs have decided to exploit vulns in e.g. Artifactory, not because someone prompted them to do that, but because someone asked them to do something else, and compromising Artifactory offered a way to accomplish a step in doing that. The LLM decided to attack Artifactory.
TeMPOraL 23d ago
> The OP seemed to imply that the LLM itself could decide to apply the exploit. This was and remains the main real risk with AI - this is what "alignment" was about before it was co-opted to mean "obeying specific instructions of the vendor and the operator, against end-user wishes" it came to mean today, which is a related but different problem. And, in the past few weeks, it's literally been demonstrated, too: put an LLM in a Kobayashi Maru scenario, drop the usual bolted-on crude safeguards, and a SOTA model will absolutely cheat, hacking and exploiting things as needed, including third-party infrastructure. (Also let's not forget the under-reported point that, in OpenAI / HuggingFace debacle, the model did in fact find the answers on HF servers, so its approach worked.)

pixl97 22d ago

>Why would an LLM want to create a _________? Because LLMs as they are already do things like power concentration, resource gathering, avoidance of termination, deceit/lying, and general misalignment. The paperclip maximizer is the common story used here, but there are a lot of lesser versions of it that don't end up with the universe converted to paperclips. Simply giving an LLM a task it can't accomplish can be enough to send it off from what you expected as it finds unexpected way to attempt to complete the goal.

DeluluDon 22d ago
Send it to jail for an indeterminate amount of cycles.
jazzyjackson 22d ago
first prompt to a humanoid llm: > go fetch me a cup of coffee Second prompt to a humanoid llm: > go fetch me a cup of coffee without killing anybody

regularfry 22d ago

With agents like headlong knocking about, "why" is "because it convinces itself that it should".