8 ms·
How we got read access on Google’s production servers
tl;dr: How we got $10.000. We were able to upload an XML file to Google. The XML parser was vulnerable to XXE. We got full read access to their production servers, including the /etc/passwd.
- NicoJuicy 12y agoOfftopic: the reply was generated with Google's internal meme generator, i read about it here : https://plus.google.com/+ColinMcMillen/posts/D7gfxe4bU7o https://plus.google.com/+ColinMcMillen/posts/D7gfxe4bU7o Actually digged it when i read it a few years ago and awesome knowing that it was probably used for this reply :)
- enscr 12y agoIs there a startup that can help automate custom attacks on websites? Like guide the webmaster to look for holes in their setup. I'm guessing some security expert can do a good job educating new businesses on how to prepare for the big bad world.
- ohwp 12y agoCheck the site of the blog post...
- borski 12y agoCheck out https://www.tinfoilsecurity.com https://www.tinfoilsecurity.com
- detectify 12y agoHi! In fact, that's exactly what we do at Detectify. Just check out https://detectify.com https://detectify.com!
- d0m 12y agoI think you just proved that writing an excellent blog post like you did is an amazing way to get new customers!! Maybe make it a tad more explicit in the post (or page) what detectify do. I personally had no idea.. but I just checked the homepage because I liked the design and was curious, and it's only then I realized what you guys were doing.
- schrodinger 12y agoCheck out Burp Suite as a good tool to run: http://www.portswigger.net/burp/ http://www.portswigger.net/burp/ Owasp is also a good resource for learning: https://www.owasp.org/index.php/Main_Page https://www.owasp.org/index.php/Main_Page
- h1ccup 12y agoWell done. I had to deal with some similar issues with my own project, and they weren't legacy code either. This should push me to go through some of my code again.
- numair 12y ago... And this is why you want to discontinue products and services your engineers can't be motivated to maintain. Amazing. This should scare anyone who has ever left an old side project running; I could see a lot of companies doing a product/service portfolio review based on this as a case study.
- spindritf 12y agoOr just move it to some cheap VPS where it cannot damage other services or your infrastructure.
- adaml_623 12y agoOr your reputation or your ethical and possibly legal duty to protect your clients?
- cookiecaper 12y agoMost of the time projects are not neatly encapsulated like that.
- SixSigma 12y agoShouldn't they be there in the first place? Even better, host on your competitor's servers.
- NicoJuicy 12y agoA job well done. This is actually impressive and quite interesting to see after what you are searching for (afterwards it seems logical :))
- cheald 12y agoXML legitimately scares me. The number of scary, twisted things it can do make me shudder every time I write code to parse some XML from anywhere - it just feels like a giant timebomb waiting to happen.
- bambax 12y ago> every time I write code to parse some XML Why would you write code to parse XML? Use an existing parser to parse. Use XSLT to modify/transform (including generate JSON/CSV/other).
- arethuza 12y agoI think cheald probably means writing code to invoke a parser to parse XML. Presumably if you had written your own parser (generally, not a great idea) the resulting behaviour would not be "scary, twisted"... [at least to the person writing the parser].
- cheald 12y agoYes, indeed. :)
- kevingadd 12y agoHe very clearly said 'write code to parse' not 'write a parser'. The former obviously USES a parser.
- bambax 12y agoOkay, parent comment obviously came out wrong and is starting its descent into white hell... ;-) I'm not going to delete it since it would be unfair to the child comments. XML is for some reason a super-controversial technology that is apparently almost universally hated, and XSLT even more so. I hope I'll not be downvoted even more by asking what's scary about being downstream from a (serious, well-maintained) XML parser? (And I love XSLT. What can I say.)
- 12y ago
- raesene3 12y agoInteresting to see this hit big companies like google. The problem, I think, stems from the idea that most people treat XML parsers as a "black box" and don't enquire too closely as to all the functionality that they support. Reading the spec. which led to the implementations, can often reveal interesting things, like support for external entities..
- vidarh 12y agoAlso horrible defaults in XML parsers. That any XML parsers allow retrieval of DTD's without explicit options specifying allowed sources etc. is beyond me. It's not just local file access, which becomes a security hole when you let users pass you XML files, though that is one of the worst ones. But the number of times I've seen production apps that turn out to behind the scenes request DTD's or schemas from remote servers regularly have made that one of the first thing I check if I am tasked to maintain or look into anything that parses XML. Often these apps stop working or slow down for seemingly no reason because the DTD or schema becomes unavailable, and nobody understands why.
- bambax 12y agoIt's bad practice to fetch an external DTD on a server you don't control, first for security reasons, second because your application then depends on something that can go away anytime, third because it's rude to the third party. twic is right that one should always use entity resolvers that point to local ressources and that parsers should run in a sandbox without external access. He's also right to say that by default parsers shouldn't go fetch external resources; I think the reason is historical; entity resolvers appeared later than the parsers themselves.
- bhaak 12y agoIt is bad practise but you know that it is uncannily common? Just remember that the W3C had to impose download restrictions on the (X)HTML DTDs (http://www.w3.org/Help/Webmaster#block http://www.w3.org/Help/Webmaster#block)
- 12y ago
- antocv 12y agoSo, when you have read access to googles prod servers, what else would be fun to do besides reading /etc/passwd ? Getting the source?
- willvarfar 12y agoLooking for more serious bugs? It could be the first steps on a major privilege escalation.
- antocv 12y agoWhere would you look?
- thrownaway2424 12y ago/proc/self/exe
- skj 12y agoThe source is not generally accessible from prod servers - only binaries and supporting data, and only the ones running on that computer. I guess it's possible you could find a computer that hosted both search and the codebase. But, since search is for external and the codebase is for internal, I'd be that they don't share clusters.
- borplk 12y ago
- mixmax 12y agoIn large production environments it's almost impossible to avoid bugs - and some of them are going to be nasty. What sets great and security conscious companies apart from the rest is how they deal with them. This is an examplary response from google. They respond promptly (with humor no less) and thank the guys that found the bug. Then they proceeded to pay out a bounty of $10.000. Well done google.
- Trufa 12y agoIndeed, it's funny that I'm reading about a vulnerability they had and it's actually making me feel more safe about using their products.
- borski 12y agoI am really glad about how they responded. Whenever Tinfoil has found vulnerabilities in companies like United Airlines[0], for example, those companies mostly respond with anger rather than graciousness. [0] https://www.tinfoilsecurity.com/blog/132969897 https://www.tinfoilsecurity.com/blog/132969897
- TallGuyShort 12y agoExactly. I just saw that the local bank my parents use is still vulnerable to the Heartbleed Bug. But you know what? I don't want to go down there and talk to them because I'm quite certain they'll call the police because I "hacked into their systems".
- 4ad 12y agoJust $10k? This sells for at least 10 times more on the black market. Why would one rationally chose to "sell" this to google instead of the black market. Some people don't break the law because they are afraid to get caught, but I like to believe that most people don't break the law because of the moral aspect. To me at least, selling this on the black market poses no moral questions, so, leaving aside "I'm afraid to get caught", why would one not sell this on the black market? Simple economic analysis. Very serious question.
- teemo_cute 12y agoBecause they are a legitimate company that sells security services.
- meric 12y agoPerceived chance of being caught * cost of punishment.
- josephagoss 12y agoThey made $10k plus a huge amount of free advertisement for their company and services (security). I reckon this release alone will earn them far more than your estimated $90k difference. Mind you, your point is certainly valid if this were a random hacker type.
- alternize 12y agovery good advertisement indeed - i haven't heard about their service until today, and am now giving it a try.
- mixmax 12y agoIf you want to look at it rationally you have to factor in the risks you are taking by selling it on the black market. These risk include: - How will you whitewash the money? Alternatively how will you spend them on the black market? You can't buy houses, cars or stocks with black money. - Will you get paid? - Secure anonymous payments that are guaranteed are not trivial. I don't know if there are escrow services for the black market, but this is definitely risky. We are talking about shady actors after all. - Will you get caught? If do you will probably end up in prison. When you take the above in to consideration I think most people would prefer $10.000 legitimate US dollars without risk to $100.000 that might end up giving you ten years behind bars.
- dantiberian 12y agoVery cool hack. Is $10,000 around the top end of what Google will pay out? This seems like quite a serious bug as far as they go.
- NicoJuicy 12y agoNo, You can see the general payout levels here: http://www.google.com/about/appsecurity/reward-program/ http://www.google.com/about/appsecurity/reward-program/ , normally the top payout is about $ 20,000, but the top payout (for Chrome) currently 2 people have been rewarded with $ 60,000. There is an overview of the top payouts though: http://www.chromium.org/Home/chromium-security/hall-of-fame http://www.chromium.org/Home/chromium-security/hall-of-fame. Some payouts are $1337 ,$3133.7 or $31336 :P Microsoft rewards even up to $100.000 for security issues in the latest OS (currently Windows 8.1)
- tptacek 12y agoThese payouts are for product vulnerabilities; things that Microsoft and Google ship to customers; vulnerabilities that those vendors are effectively creating on hundreds of thousands of machines they don't own.
- msantos 12y agoA few webcrawlers[1] out there follow HTTP redirect headers and ignore the change in schemas (this method is different of OP's but achieves the same goal). So anyone can create a trap link such as <a href="file:///etc/passwd">gold</a> Or <a href="trap.html">trap</a> once trap.html is requested the server issues a header "Location: file:///etc/passwd" Then it's just a matter of seat and wait for the result to show up wherever that spider shows its indexed results. [1] https://github.com/scrapy/scrapy/issues/457 https://github.com/scrapy/scrapy/issues/457
- raverbashing 12y agoThis is another reason not to use XML, plain and simple It's too much hidden power in the hands of those who don't know what they're doing (loading external entities pointed in an XML automatically? what kind of joke is that?)
- brown9-2 12y agoSure, but didn't YAML in Rails do mostly the same type of thing? It's not just XML that is dumb like this.
- krapp 12y agoYAML and XML seem too powerful and too complex for their own common use cases (data storage). Markdown too - how many Markdown parsers allow for strict parsing against an HTML whitelist, and don't allow native HTML at all by default?
- eik3_de 12y agoscnr XML - It seemed like a good idea at the time
- SixSigma 12y agoNot to everyone. Some of us greybeards tried to warn against it : "XML is simply lisp done wrong." — Alan Cox but the gee-whizzery won. "XML combines the efficiency of text files with the readability of binary files" — unknown "XML is a classic political compromise: it balances the needs of man and machine by being equally unreadable to both." — Matthew Might Anyone remember XHTML ?
- peterkelly 12y agoI never understood why internal or external entities were included in XML. Can anyone explain what useful purpose they serve?
- bazzargh 12y agoExactly the same as #includes and #defines in C - they let you organize your code in multiple files, be more concise, and shoot yourself in the foot, repeatedly. They were useful for document editing use cases - remember this was before SOAP and xml serialization, and sgml tooling that already supported this stuff existed. You can see the record of the decision here: http://www.w3.org/XML/9712-reports.html#ID5 http://www.w3.org/XML/9712-reports.html#ID5
- blueskin_ 12y agoI wonder how many of the blurred entries were NSA.
- halflings 12y agoI hope it doesn't get unnoticed that the guys who discovered this vulnerability created a really great product, Detectify : https://detectify.com/ https://detectify.com/ They also discovered vulnerabilities in many big websites (dropbox, facebook, mega, ...). Their blog also has many great write-ups : http://blog.detectify.com/ http://blog.detectify.com/
- detectifyuser 12y agoWhile they are probably good at doing this manually, their automated tool finds very little. And they were kind of assholes on support :(
- detectify 12y agoSorry to hear that you are disappointed. Few results can be a good thing though, it might just mean that your site has very few issues! Feel free to mail us again and thanks for the feedback :)
- ibu 12y agoNice try, Tinfoil.
- borski 12y agoEr...CTO of Tinfoil here. We respect the Detectify guys a lot. Not sure what you were trying to get at, but there's no conspiracy here. We don't engage in subversive competitive tactics.
- opendais 12y agoOdd, I had the opposite experience. They seemed fine to me.
- plq 12y agoFor those who'd like to know more about xml-related attack vectors, here's a nice summary: https://pypi.python.org/pypi/defusedxml https://pypi.python.org/pypi/defusedxml
- chmars 12y agoThe guys behind this report have an interesting pricing model: Pay what you want! https://detectify.com/pricing https://detectify.com/pricing The pricing models has apparently worked so far. Are any active users of Detectify here and can share their experience?
- detectifyuser 12y agoI like the price but they found nothing, honestly, and we're not very nice when I emailed them for support.
- pg_is_a_butt 12y agowhy weren't you nice when you emailed them?
- bronxbomber92 12y agoJust wanted to point out the hilarity of your typo: "we're not very nice when..." vs "were not very nice when..." completely reversed what you meant to say ;P.
- alternize 12y agoI tried them on a client project today, and they found some (minor) form post issues. the scan used roughly 1.5hr, which results in "incurred cost" of $4.75 (for their cloud ressources needed), and they suggest a 5x "gratitude" factor which I gladly paid.
- chmars 12y agoSo they tell you how much they spent on cloud resources and even suggest a gratitude factor? That is actually a great way of getting paid (enough) even if you do not enforce any fixed pricing, cool! :)
- smprk 12y agoVery interesting in deed. There is a auto driver in India (a taxi like vehicle) who uses the same pricing model for his services - http://www.thebetterindia.com/4813/tbi-heroes-ahmedabad-no-rickshawalo-personifying-love-all-serve-all/ http://www.thebetterindia.com/4813/tbi-heroes-ahmedabad-no-r... Nice to know about such things :-)
- sebban_ 12y agoAwesome work! The bounty is a bit low though.
- kirab 12y agoI think they couldn’t read /etc/shadow, so it’s not that bad at first. But then they could surely access some configuration file of the application itself, probably containing DB creds and of course more information which helps to find more vulns.
- thrownaway2424 12y agoIt's shocking to me that baking "db creds" into a binary or configuration file is still so common that anyone would expect it to be true on a randomly selected server. Is this still the industry standard?
- fennecfoxen 12y agoSadly but truly!
- dreamdu5t 12y agoHow else would you do it? If you use a configuration "service" the credentials to access the service must be baked in.
- thrownaway2424 12y agoWell, I can think of a couple of ways off the top of my head, that I'm sure will be shouted down for being simplistic: 1) ident protocol, or something similar. On the internet, it's a disaster, but for machines all owned by the same organization, it makes sense. 2) ssl client certificate. this can be hardened in various ways like having the certs expire every ten minutes etc.
- enneff 12y agoThat's certainly not how we do it at Google.
- njharman 12y agotake away: XML should not be used (at least as user input). It is too powerful, too big. It is much too hard and expensive to test and validate. Input from potentially malicious users should be in the simplest, least powerful of formats. No logic, no programability, strictly data. I'm putting "using XML for user input" in same bucket as "rolling your own crypto/security system". That is you're gonna do it wrong, so don't do it.
- yummybear 12y agoYou should be aware that pixilating or blurring screenshots are likely not sufficient to ensure that the contents are unrecoverable.
- ajsharp 12y agoCheers to google for properly compensating these guys for their findings.
- pearjuice 12y agoThat must have been be a nasty call from Sergey to NSA head quarters earlier this week. "Sir, I am sorry to inform you that another backdoor has been found. We will introduce two more as agreed upon in our service level agreement."
- mwcampbell 12y agoI'm surprised nobody has mentioned containers, e.g. Docker, as a way of limiting the damage from this kind of bug. In a container whose only purpose is to run the application, /etc/passwd should be as uninteresting as: root:x:0:0:root:/:/bin/sh bin:x:1:1:bin:/dev/null:/sbin/nologin nobody:x:99:99:nobody:/dev/null:/sbin/nologin app:x:100:100:app:/app:/bin/sh