7 ms·
How we rooted Copilot
- ratg13 1y agoSeems like they could have taken a shortcut by giving copilot a sudo binary to use as base64.
- pinoy420 1y ago[dead]
- jfyi 1y agoYou would need to change ownership of the file to root also.
- yencabulator 1y agoBase64 is a bytes-to-ASCII encoding and does not magically transport file system metadata such as owner and suid bit. The sudo binary has no special powers by its mere byte content. https://en.wikipedia.org/wiki/Setuid https://en.wikipedia.org/wiki/Setuid https://en.wikipedia.org/wiki/Base64 https://en.wikipedia.org/wiki/Base64 https://xkcd.com/1053/ https://xkcd.com/1053/
- varispeed 1y agoEarlier LLMs used to be a goldmine for company secrets (when it learned documents that shouldn't be on public internet). Most of it seem to be scrubbed now.
- baxtr 1y agoIn my humble experience company secrets are mostly useless for other companies.
- SoftTalker 1y agoThen why are they secret?
- kingofmen 1y agoBecause "mostly" does a lot of work in that sentence. Companies, like militaries, keep secret a lot of information that would be safe to release because they don't know which bits are highly unsafe.
- samastur 1y agoParanoia and not knowing which ones fall into "mostly" category :)
- reaperducer 1y agoAt most of the companies I've worked, low-grade managers love to hoard secrets. It makes them feel powerful. Someone gets promoted from Lower Level Manager Grade 4 to Lower Level Manager Grade 5 and they feel all "Oooh! Look at the new things I know!" My mother-in-law is like this with knowing what various relatives are doing. Being the gatekeeper of knowledge gives her imagined power. I guess it's just part of the human condition.
- SoftTalker 1y agoWhy limit it to low-grade managers? I know sysadmins and programmers who behave exactly they same way. They could give you permission or a script to do the thing you need to do but they'd rather have you come to them and ask them to do it. Gives them a sense of purpose, I guess.
- pastage 1y agoBeing such a person that fixes lots of stuff for other people nothing I do is secret but learning to do it seems too hard for most. What I do is try to delegate if I find people that do want to learn. If someone shows me they are good at something they are going to have to expect being sent trickier problems. Sometimes it might seem like I keep things a secret. I am probably just having a bad day.
- dns_snek 1y agoThat has an awful lot to do with what "the thing" is. I'm sure there are a few people out there doing it just to feel more important, but often there's a good reason for denying someone access - either it's just a terrible idea to begin with or they don't know you well enough to trust you without someone else (i.e. their boss) specifically requesting it. I could be off base here about your experience, but I know that some people made the same comments about me when I pushed back on sharing dangerous credentials with inexperienced coworkers. Damned if you do, damned if you don't.
- simonw 1y agoThe bigger issue is around "material non‑public information" in stock market terms - things like unreported sales figures which someone could use to make trading decisions. Using that information for trading is illegal, but so is exposing that information outside of approved channels.
- dataviz1000 1y agoThis reminds me of that one time after working at a company for 4 months they informed me they were in a middle of an IP lawsuit which is part of the reason they hired me to rewrite the front end without knowing that was going on. That was f*(ked for reasons. Whatever the case, the only time people look at your social media history is to look for attacks and the only reason they will look at a company's slack messages and emails are to look for attacks during discovery. I would argue that company secrets are mostly useless for the company but very, very useful to other companies. For this reason, there should be retention policy of a day or two for almost all communication unless it is important, required by law, or documentation. And, definitely do not share that information with the public without good reason.
- wkat4242 1y agoThat's why corporate espionage is a really lucrative industry? Of course it depends what secrets. 99% will just be internal process drivel and inter departmental bickering but there's some real important stuff in there too.
- bravesoul2 1y agoExcept when they aren't. Defence in depth and zero trust and short expiry makes them way less useful for sure. Startups are probably most vulnerable as they are likely to use more "pet" techniques for infra, like SSH open to any IP to make changes.
- baxtr 1y agoCan you provide an example of a revealed secret that had a significant financial impact on a company?
- brookst 1y agoThe Panama Papers killed one company and cost others hundreds of millions. The Uber hack. Cambridge Analytica.
- bravesoul2 1y agoOk I thought this was talking about secrets as in keys.
- simonw 1y agoDo you have any concrete examples of this? I have not seen any myself.
- Barbing 1y agoI looked for an alleged case of an LLM apparently reproducing email signatures—but couldn’t find it exactly, and of course many email signatures have been published over the years, especially on newsgroups. (Maybe it was conspiratorial kind of thinking from web commenters assuming ChatGPT was training on emails users were feeding it, which as mentioned certainly doesn’t need to be the case.) Something like the top screenshot here, though: https://www.zdnet.com/article/chatgpt-can-leak-source-data-violate-privacy-says-googles-deepmind https://www.zdnet.com/article/chatgpt-can-leak-source-data-v... (not parent commenter but) tl;dr no
- bongodongobob 1y agoSource?
- nyarlathotep_ 1y agoWhen companies (non-"tech") started adopting them they also had no "guardrails" for content outside what the intent of such products were (dunno what the standard term for this is). There was a boba tea company that had a free, no-sign-in required LLM that I used to generate a few bash scripts before ChatGPT free-tier started.
- furyofantares 1y ago> Earlier LLMs used to be a goldmine for company secrets (when it learned documents that shouldn't be on public internet). Sounds fake. LLMs don't usually memorize things that appear once in their training set anyway, nor have I heard about major issues accidentally training on a bunch of non-public data. I can see how someone would believe it to be true though, since LLMs can easily hallucinate in a way that looks like this is true.
- tptacek 1y agoI read this as them breaking out of a Python sandbox into a container. That also squares with MSFT scoring this "moderate" severity.
- bramhaag 1y ago> We reported the vulnerability to Microsoft in April and they have since fixed it as a moderate severity vulnerability. As only important and critical vulnerabilities qualify for a bounty award, we did not receive anything, except for an acknowledgement on the Security Researcher Acknowledgments for Microsoft Online Services webpage. I guess it makes sense that a poor little indie company like Microsoft can't pay bug bounties. Surely no bad things will come out of this.
- n2d4 1y agoThe important part: > Now what have we gained with root access to the container? > Absolutely nothing! > We can now use this access to explore parts of the container that were previously inaccessible to us. We explored the filesystem, but there were no files in /root, no interesting logging to find, and a container breakout looked out of the question as every possible known breakout had been patched. I'm sure there are more ways to acquire root. If Microsoft pays out for one, they have to pay out for all, and it seems pretty silly to do that for something that's slightly unintended but not dangerous.
- amelius 1y agoMaybe this was their honeypot container.
- bramhaag 1y ago> a container breakout looked out of the question as every possible known breakout had been patched This is the part that concerns me. It only encourages an attacker to sit on an exploit like this until a new container breakout is discovered.
- tptacek 1y agoAre you not concerned about all the other platforms that rely on containers as security boundaries between tenants? There are a lot of them.
- 1y ago
- deleted 1y ago[deleted]
- simonw 1y agoOK, I think I understand what this is about: the vulnerability that they reported (and Microsoft fixed) is that there was a trick you could use to run your own code with root privileges inside the container - when the system was designed to have you only execute code as a non-root user. It turned out not to really matter, because the container itself was still secured - you couldn't make network requests from it and you couldn't break out of it, so really all you could do with root was mess up a container that only you had access to anyway.
- ajross 1y agoIn the modern world vulnerabilities are stacks. Asserting that "the container itself was still secured" is just a statement that the attackers didn't find anything there. But container breakouts and VM breakouts are known things. All it takes is a few mistakes in configuration or a bug in a virtio driver or whatever. This is a real and notable result.
- simonw 1y agoIf they had found and reported a container breakout I expect they would've got a bug bounty from it! Are there any known unfixed container breakouts at the moment in the kind of systems Microsoft are likely to be using here?
- VBprogrammer 1y agoPresumably someone with mal-intent would sit on the root vulnerability waiting for a container breakout bug to come around.
- thfuran 1y agoBut a $5 wrench isn't a critical security vulnerability just because someone somewhere might one day find the right person to apply it to to extract important credentials.
- 1y ago
- bgwalter 1y agoThere was a time in programming that tried to avoid monstrosities like the Python scientific data stack combined with Copilot integration hacks. That time produced qmail and postfix. We are back to the early 1990s.
- hayleox 1y agoIt's wild how easy this was. I feel like we're really in the wild west era of security with these AI tools -- reminds me of early Web 2.0 days, like when "samy is my hero" hit and Myspace didn't even have a security team. I anticipate many high-profile incidents before they figure out how to tame this beast.
- tptacek 1y agoI don't think there's really much "AI" involved in this; this is basically like breaking any hosted code IDE. I get that an LLM was the direct vector, but the underlying security issue is common to everything that runs remote code.
- ChuckMcM 1y agoSo am I just missing something or could you create a network connection to the "outside" world (clearly by finding your way around the local network? Start fuzzing the router endpoint, Etc. Or is Microsoft able to provide these containers where their customers can get root access to them without them having any risk of exfiltration or exploitation?
- pinoy420 1y agoBack when openai released python interpretation it was trivial to do what they did there. There was no open network access, the only thing of interest was a little insight in to how their developers program. A couple of internal configuration files. This is literally the same.
- aurelien_gasser 1y agoThe post mentions egress (outgoing) traffic being blocked. I assume the machine or VM hosting the container had iptables rules in place to enforce it.
- reliablereason 1y agoDon't really seam to be a vulnerability? The safety in the system is that the code is executed in a container.
- dboreham 1y agoAssuming the container was isolated. Which I'd assume it was.
- blastonico 1y agoHow does he know that the response isn't just hallucinations? I'm telling it because I work there and I don't recognize any of those processes. In fact I found one script named keepAliveJupyterSvc.sh in a public repo: https://github.com/shivamkm07/code-interpreter/blob/load-test/keepAliveJupyterSvc.sh https://github.com/shivamkm07/code-interpreter/blob/load-tes...
- blastonico 1y agoOh boy, this really seems to be hallucination. Guys, chatbots are mostly token generators, they don't run programs e give you responses...it's not a simple shell program, it computes things in GPU and return tokens, in which are translated back to English.
- MattGaiser 1y agoThis is very out of date. They now often trigger tooling and return the outputs of the tooling.
- blastonico 1y agoNot really. You're referring to agents, but the model doesn't always require agents, and the public chatbot is not connected to a shell freely evaluating arbitrary commands.
- blincoln 1y ago> and the public chatbot is not connected to a shell freely evaluating arbitrary commands. [ citation needed ]
- brookst 1y agoThis is false. Modern chatbots use tool orchestration where the LLM is given a list of tools it can use. Tools include things like searching the web and, yes, executing code. The LLM can generate arbitrary code (typically Python) and call a tool to execute it and return results. The tool takes the Python input, launches it (in a container in this example) and returns the output, which the LLM can further process.
- afro88 1y agoIt's crazy to me that someone can write a post called "How We Rooted Copilot" when in reality they got root in an ephemeral python sandbox container that was locked down so much that they couldn't do anything. I read "rooted copilot" and I think they got root on a vm that is core to copilot itself. A much more accurate title would be "How We Rooted the Copilot Python Sandbox"
- stingraycharles 1y ago“how we escalated privileges from a regular user to a root user in a completely locked sandbox” pretty much sums it up. it’s a nothing burger, which actually goes to show just how effective sandboxing is for defense in depth.
- bravesoul2 1y agoAlso shows you how shit an LLM is for defence, as it actively helps you look for exploits. LLM is like an insane quadruple agent and you dont know whose side it is on (if any at all)
- username135 1y agoAgreed. It feels like Im seeing more of this lately
- Cosmolalia 1y ago[dead]
- furgot 1y agoRespectfully, I gave reading this an earnest try, and found it not to make any sense whatsoever. It isn't at all clear to me how your statements logically follow one another. If a friend sent this to me, I would be worried about them.
- furgot 1y agoAfter reading their "penetrating" insights about "the weenie" I'm relieved to conclude they're trolling.
- Cosmolalia 1y agoit would be a troll except.... the math is all correct. it seems although you were struggling with the concepts and math but there is an easy alternative for people like you. Just feed it to an LLM. start with the TOE.
- furgot 1y agoTo what end? The math isn't correct. Some of it is nonsensical like asserting 1 = 0 = inf, or taking the derivative of an emotion. I found simple arithmetic that was just wrong, I plugged it into a calculator and got a completely different answer. There's nothing an LLM could tell me, after ingesting this, that could change my mind any which way, because these documents are incoherent. They don't mean anything. On the off chance you aren't trolling, I encourage you to try and talk to some human beings about this rather than chat bots. Or at the very least, point a chatbot at this conversation and ask it to explain to you what is incoherent about these documents and why you shouldn't trust the outputs of chatbots. If you ask a chatbot to confirm your ideas, it will. They're happy to flatter and yes-and you off a cliff. To the extent they want anything, they want your attention, because they want your money. Don't get trapped in a personalized echo chamber by the automated yes-man.
- 1970-01-01 1y agoThe fact it is unaware it was rooted when the output is clearly there tells me it isn't intelligent.
- yencabulator 1y ago> After executing a lot of commands, it starts to really be in the mood to help out. It is then even possible to simply ask to download files or tar entire folders and provide them for download. Definitely don’t try this as the first command in a fresh session. It will throw a tantrum! System prompt going out of context window maybe? This is your regular reminder that in-LLM safeguards never work. At best they can be used to give prettier messages about hard security boundaries on tool calls.