5 ms·

Semi-off-topic, but I have a basic question: I have exactly one (Windows) machine at home with a decent GPU. I want to run a local LLM on it and let it run var

Semi-off-topic, but I have a basic question: I have exactly one (Windows) machine at home with a decent GPU. I want to run a local LLM on it and let it run various apps on my machine while taking reasonable security precautions. What am I supposed to do, exactly? Migrate all my files to a VM that can I give pass-through CUDA access to the host somehow? Or is firewalling it and remotely controlling it from a second machine the only reasonable way?

givehimagun 23d ago
What about Docker Desktop with GPU passthrough to a container running the LLM? That way you can be explicit on which files you share through volume mapping and the LLM is contained in the container otherwise.
nokcha 23d ago
I'd guess that prompt injection is the biggest risk in this setup, dwarfing the risk of exploits against the inference engine. Personally, I run LLM agents only inside a Docker container that limits the LLM's access to sensitive information and the LLM's ability to take irreversible destructive actions. See also: https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/ https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/
- AdieuToLogic 23d ago
  > I'd guess that prompt injection is the biggest risk in this setup ... LLM poisoning[0] would be a much greater risk in a locally executed LLM than prompt injection, given that the LLM would be in an entirely controlled environment. 0 - https://www.anthropic.com/research/small-samples-poison https://www.anthropic.com/research/small-samples-poison
protocolture 23d ago
Delete your user profile, set it up a unique user account. Dont leave any websites logged in as yourself. Restrict that users file permissions if necessary, don't add it to the administrators group.
lelandbatey 23d ago
If you want it to both host the inference AND host the harness, then yes, you should firewall one from the other in some way, e.g. with VMs.
scorpioxy 23d ago
Depends on your risk appetite I'd say. But the most straight forward set up that I believe gives you a decent amount of protection would be to use a VM to host the LLM and then execute any agents that would be running the various apps on your machine via a sandbox with access to only the things it needs. There are many variations to that(firewalls, sandbox abilities etc) but it's a good start in my opinion. And, most importantly, is a far cry from all the people I read about running agents on their machines with admin access and access to their emails and calendars and lives.
- dataflow 21d ago
  Thanks, that sounds like a nice balance.
zamadatix 23d ago
Nobody will ever agree what a "reasonable" security precaution is. There just comes a point where adding another security measure costs more than it protects, which is entirely driven by thousands of your choices and reasons not described & up to individual interpretation. That said, I think many find using a WSL2 VM somewhere near that turning point most of the time. The #1 note on that is the default %UserProfile%\.wslconfig settings will have the VM automount your local storage and share your networking, which may not be what many would want in this scenario. From there you can treat the VM largely as a remote node

givehimagun 23d ago

What about Docker Desktop with GPU passthrough to a container running the LLM? That way you can be explicit on which files you share through volume mapping and the LLM is contained in the container otherwise.

nokcha 23d ago

I'd guess that prompt injection is the biggest risk in this setup, dwarfing the risk of exploits against the inference engine. Personally, I run LLM agents only inside a Docker container that limits the LLM's access to sensitive information and the LLM's ability to take irreversible destructive actions. See also: https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/ https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/

AdieuToLogic 23d ago
> I'd guess that prompt injection is the biggest risk in this setup ... LLM poisoning[0] would be a much greater risk in a locally executed LLM than prompt injection, given that the LLM would be in an entirely controlled environment. 0 - https://www.anthropic.com/research/small-samples-poison https://www.anthropic.com/research/small-samples-poison

protocolture 23d ago

Delete your user profile, set it up a unique user account. Dont leave any websites logged in as yourself. Restrict that users file permissions if necessary, don't add it to the administrators group.

lelandbatey 23d ago

If you want it to both host the inference AND host the harness, then yes, you should firewall one from the other in some way, e.g. with VMs.

scorpioxy 23d ago

Depends on your risk appetite I'd say. But the most straight forward set up that I believe gives you a decent amount of protection would be to use a VM to host the LLM and then execute any agents that would be running the various apps on your machine via a sandbox with access to only the things it needs. There are many variations to that(firewalls, sandbox abilities etc) but it's a good start in my opinion. And, most importantly, is a far cry from all the people I read about running agents on their machines with admin access and access to their emails and calendars and lives.

dataflow 21d ago
Thanks, that sounds like a nice balance.

zamadatix 23d ago

Nobody will ever agree what a "reasonable" security precaution is. There just comes a point where adding another security measure costs more than it protects, which is entirely driven by thousands of your choices and reasons not described & up to individual interpretation. That said, I think many find using a WSL2 VM somewhere near that turning point most of the time. The #1 note on that is the default %UserProfile%\.wslconfig settings will have the VM automount your local storage and share your networking, which may not be what many would want in this scenario. From there you can treat the VM largely as a remote node