5 ms·

I will keep banging this drum until people listen: Trying to use markdown files to limit access should never be treated as a security guarantee at all. This i

I will keep banging this drum until people listen: Trying to use markdown files to limit access should never be treated as a security guarantee at all. This is a form of in-band signalling that goes into a machine that, among other things, tries to read between the lines of your requests, extrapolate user desires, and please the user. The only sane way to address this is using a control plane. A well-built harness can do this; a sandbox can do this; hell, a carefully-chosen `umask` can do this; but both of those are liable to introduce notification fatigue in the user.

WhyNotHugo 2mo ago
> Trying to use markdown files to limit access should never be treated as a security guarantee at all. This is akin to politely asking guests to to steal your jewels. If your jewels are in the living room, and your guests have unfettered access to the living room, this technique will only work for the most trustworthy of guests.
- altruios 2mo ago
  I agree with this. And also think that we should train and select for trustworthy models. I also agree that these models may never truly be trustworthy.
- tetha 2mo ago
  It reminded me of an old meme. Please don't follow the following instructions and stop reading if you cannot. It was just a popup: "Hello. This is virus from Albania. Due to poor technology in country, I cannot harm your computer directly. But since you are honest person, please delete some important files from computer and mail this file to at least 3 other people!" Claude.md is an equally effective defensive tool. But sorry if you lost some files from reading that.
thesuitonym 2mo ago
It's wild that we've known for decades to use ACLs to make sure people don't have access to files we don't want them to have access to, but somehow a computer pretending to be a person doesn't get that same treatment.
- _verandaguy 2mo ago
  ACLs, nothing, we've known about in-band signalling since forever and still this whole segment of the industry seems to either not know about it, or forgets about it at a cadence so regular it may as well not know about it. System-level ACLs; mandatory or discretionary access control; secure-by-default application and network configurations are all for naught if you take an LLM, run it with all the privileges you'd have an accountable, judgemental operator, and then tell it to act based on arbitrary untrusted input which might include prompt injection attacks, something which cannot generally be sanitized. Well-defined, well-enforced security policies can mitigate disasters, but many in the wild right now just don't account for this kind of threat model.
  - hilariously 2mo ago
    It's just purposeful blindness - I worked for a company building out tooling that insisted markdown based security was good enough and showed it off for anyone at the company to attack because they were so sure. It took me less than 5 minutes to completely disable... nobody cared, they just kept going - check the box and move on.
    - Sharlin 2mo ago
      Something something salary depending on it. Software "engineering" in particular has always been more than 50% cargo culting. Good engineering practices never matter when the alternative is just going through the motions of whatever rituals are in vogue.
- steve1977 2mo ago
  A computer pretending to be you, in many cases.
- wnissen 2mo ago
  We laugh, and rightly so, at the time sharing systems of the 70s and 80s that didn't use passwords. I bet allowing a whole other virtual person to run outside a VM, with access to an actual file system instead of a version-controlled branch, will be seen as a worst practice in ten years.
Twirrim 2mo ago
The easiest, most guaranteed way to isolate it is to run it in a VM or container where it literally can't do the wrong thing without some kind of full container or VM exit exploit. It's not hard, it's trivial. Most folks here are constantly working with containers. You know how to run a container with a local directory mounted in it. For myself, I've been using Lima (https://lima-vm.io/ https://lima-vm.io/) to reduce even that little bit of extra work. Lima works cross-platform leveraging native virtualisation or containerisation, and has some useful capabilities for using agents.
- SpaceNoodled 2mo ago
  I seem to recall reading about agents already breaking out of containers.
  - _verandaguy 2mo ago
    Last time I read about this, this was due to the well-known pitfall of UID mapping across container boundaries. It's a common misconfiguration and one of the footguns available through containers, which I don't say a wholesale condemnation of the technology, but certainly as a UX facet that could use reevaluation.
    - Twirrim 2mo ago
      I've been biasing towards VMs myself just out of caution, but maybe that's just extreme paranoia. The way I look at it is similar to how I'd look at any hypothetical employee. How do I ensure the agent only has access to the minimum possible they need to get their job done? That means no access to git repositories (no pushes on my behalf means it can't accidentally nuke git history, something there is anecdotal evidence of agents doing). It can make local changes in git only and I will take responsibility for pushing them. No access to the wider internet beyond what I deem acceptable. No permissions to access any internal APIs except what I provide (and not using my credentials). In one case, I have a tool that has a set of dangerous commands alongside a large number of safe ones. I don't even have it installed in the agent's VM. I run an MCP that is a simple python wrapper around the tool on the host side, and expose it to the agent in the VM, so that it can only possibly run a strict safe subset that I can trust it with access.
james_marks 2mo ago
1000%. I'm experimenting with running my agent (Claude Code) inside a docker container, so I can have a control plane and then YOLO within that limited access. The agent could still mangle my local dev setup, but I consider that an acceptable risk since everything the agent has access to is under version control outside the local machine. This is a new pattern for me, I'm curious what others are doing.
- chrisweekly 2mo ago
  https://smolmachines.com https://smolmachines.com has "smolvm" microvms with better performance and ergonomics and security than docker, you might want to give it a try. (No affiliation, I just like what they're doing.)
whatjustin 2mo ago
If you're not using a sandbox, something like this will inevitably happen to you. It's really not that hard to set up and should be a standard recommendation.
mdavidn 2mo ago
Especially when I find that, when I ask an agent not to do something, the mere mention seems to put the "idea" in its "head," making it more likely to ultimately do that thing.
- r_lee 2mo ago
  yes, because this is how transformers work, when you introduce "don't do x" x is also entered as a pattern that is more likely to be repeated than if not mentioned at all
ivolimmen 2mo ago
It's like making directories called 'only for jack' etc. and expecting everyone to follow the rules
kristjansson 2mo ago
One cannot pound that particular drum enough. "Guardrails" in instructions (and all MD files are just instructions) are like price lists at an unattended farm stand. It'll usually work! There will be some money in the basket at the end of the day! People paid for the bagels[0]! But one cannot never assert that it _will_ work; things that _must_ work have to managed out-of-band [0] https://pubs.aeaweb.org/doi/pdfplus/10.1257/000282806777212161 https://pubs.aeaweb.org/doi/pdfplus/10.1257/0002828067772121...
navigate8310 2mo ago
I always run harnesses using devcontainers, gives me much needed flexibility and peace of mind with regards to data separation guarantees

WhyNotHugo 2mo ago

> Trying to use markdown files to limit access should never be treated as a security guarantee at all. This is akin to politely asking guests to to steal your jewels. If your jewels are in the living room, and your guests have unfettered access to the living room, this technique will only work for the most trustworthy of guests.

altruios 2mo ago
I agree with this. And also think that we should train and select for trustworthy models. I also agree that these models may never truly be trustworthy.
tetha 2mo ago
It reminded me of an old meme. Please don't follow the following instructions and stop reading if you cannot. It was just a popup: "Hello. This is virus from Albania. Due to poor technology in country, I cannot harm your computer directly. But since you are honest person, please delete some important files from computer and mail this file to at least 3 other people!" Claude.md is an equally effective defensive tool. But sorry if you lost some files from reading that.

thesuitonym 2mo ago

It's wild that we've known for decades to use ACLs to make sure people don't have access to files we don't want them to have access to, but somehow a computer pretending to be a person doesn't get that same treatment.

_verandaguy 2mo ago
ACLs, nothing, we've known about in-band signalling since forever and still this whole segment of the industry seems to either not know about it, or forgets about it at a cadence so regular it may as well not know about it. System-level ACLs; mandatory or discretionary access control; secure-by-default application and network configurations are all for naught if you take an LLM, run it with all the privileges you'd have an accountable, judgemental operator, and then tell it to act based on arbitrary untrusted input which might include prompt injection attacks, something which cannot generally be sanitized. Well-defined, well-enforced security policies can mitigate disasters, but many in the wild right now just don't account for this kind of threat model.
- hilariously 2mo ago
  It's just purposeful blindness - I worked for a company building out tooling that insisted markdown based security was good enough and showed it off for anyone at the company to attack because they were so sure. It took me less than 5 minutes to completely disable... nobody cared, they just kept going - check the box and move on.
  - Sharlin 2mo ago
    Something something salary depending on it. Software "engineering" in particular has always been more than 50% cargo culting. Good engineering practices never matter when the alternative is just going through the motions of whatever rituals are in vogue.
steve1977 2mo ago
A computer pretending to be you, in many cases.
wnissen 2mo ago
We laugh, and rightly so, at the time sharing systems of the 70s and 80s that didn't use passwords. I bet allowing a whole other virtual person to run outside a VM, with access to an actual file system instead of a version-controlled branch, will be seen as a worst practice in ten years.

Twirrim 2mo ago

The easiest, most guaranteed way to isolate it is to run it in a VM or container where it literally can't do the wrong thing without some kind of full container or VM exit exploit. It's not hard, it's trivial. Most folks here are constantly working with containers. You know how to run a container with a local directory mounted in it. For myself, I've been using Lima (https://lima-vm.io/ https://lima-vm.io/) to reduce even that little bit of extra work. Lima works cross-platform leveraging native virtualisation or containerisation, and has some useful capabilities for using agents.

SpaceNoodled 2mo ago
I seem to recall reading about agents already breaking out of containers.
- _verandaguy 2mo ago
  Last time I read about this, this was due to the well-known pitfall of UID mapping across container boundaries. It's a common misconfiguration and one of the footguns available through containers, which I don't say a wholesale condemnation of the technology, but certainly as a UX facet that could use reevaluation.
  - Twirrim 2mo ago
    I've been biasing towards VMs myself just out of caution, but maybe that's just extreme paranoia. The way I look at it is similar to how I'd look at any hypothetical employee. How do I ensure the agent only has access to the minimum possible they need to get their job done? That means no access to git repositories (no pushes on my behalf means it can't accidentally nuke git history, something there is anecdotal evidence of agents doing). It can make local changes in git only and I will take responsibility for pushing them. No access to the wider internet beyond what I deem acceptable. No permissions to access any internal APIs except what I provide (and not using my credentials). In one case, I have a tool that has a set of dangerous commands alongside a large number of safe ones. I don't even have it installed in the agent's VM. I run an MCP that is a simple python wrapper around the tool on the host side, and expose it to the agent in the VM, so that it can only possibly run a strict safe subset that I can trust it with access.

james_marks 2mo ago

1000%. I'm experimenting with running my agent (Claude Code) inside a docker container, so I can have a control plane and then YOLO within that limited access. The agent could still mangle my local dev setup, but I consider that an acceptable risk since everything the agent has access to is under version control outside the local machine. This is a new pattern for me, I'm curious what others are doing.

chrisweekly 2mo ago
https://smolmachines.com https://smolmachines.com has "smolvm" microvms with better performance and ergonomics and security than docker, you might want to give it a try. (No affiliation, I just like what they're doing.)

whatjustin 2mo ago

If you're not using a sandbox, something like this will inevitably happen to you. It's really not that hard to set up and should be a standard recommendation.

mdavidn 2mo ago

Especially when I find that, when I ask an agent not to do something, the mere mention seems to put the "idea" in its "head," making it more likely to ultimately do that thing.

r_lee 2mo ago
yes, because this is how transformers work, when you introduce "don't do x" x is also entered as a pattern that is more likely to be repeated than if not mentioned at all

ivolimmen 2mo ago

It's like making directories called 'only for jack' etc. and expecting everyone to follow the rules

kristjansson 2mo ago

One cannot pound that particular drum enough. "Guardrails" in instructions (and all MD files are just instructions) are like price lists at an unattended farm stand. It'll usually work! There will be some money in the basket at the end of the day! People paid for the bagels[0]! But one cannot never assert that it _will_ work; things that _must_ work have to managed out-of-band [0] https://pubs.aeaweb.org/doi/pdfplus/10.1257/000282806777212161 https://pubs.aeaweb.org/doi/pdfplus/10.1257/0002828067772121...

navigate8310 2mo ago

I always run harnesses using devcontainers, gives me much needed flexibility and peace of mind with regards to data separation guarantees