11 ms·
900 Sites, 125M accounts, 1 Vulnerability
- deleted 3y ago[deleted]
- Summiner 3y ago[flagged]
- bluelightning2k 3y ago[flagged]
- xyzeva 3y agoThank you! Means a lot, helps us keep going.
- T3RMINATED 3y ago[dead]
- abeisgreat 3y agoI worked at Firebase for many years and the concerns with security rules have always plagued the product. We tried a lot of approaches (self expiring default rules, more education, etc) but at the end of the day we still see a lot of insecure databases. I think the reasons for this are complex. First, security rules as implemented by Firebase are still a novel concept. A new dev joining a team adding data into an existing location probably won’t go back and fix rules to reflect that the privacy requirements of that data has changed. Second, without the security of obscurity created by random in-house implementations of backends, scanning en masse becomes easier. Finally, security rules are just hard. Especially for realtime database, they are hard to write and don’t scale well. This comes up a lot less than you’d think though, as any time automated scanning is used it’s just looking for open data, anything beyond “read write true” as we called it would have prevented this. Technically there is nothing wrong with the Firebase approach but because it is one of the only backends which use this model (one based around stored data and security rules), it opens itself up to misunderstanding, improper use, and issues like this.
- xyzeva 3y agoWe tried to contact google, via support to try to help or for them to help disclose the issues to the websites. We got no response other then a response telling us that they will be creating a feature request on our behalf if we wanted instead of helping us, which is fair as I think we'd have to escalate pretty far up in Firebase to get the attention of someone who could alert project owners.
- abeisgreat 3y agoOne of the things we fought for, for years after acquisition was to maintain a qualified staff of fulltime, highly paid support people who are capable of identifying and escalating issues like this with common sense. This is a battle we slowly lost. It started with all of support being the original team, then went to 3-4 fulltime staff plus some contracts, to entirely contractors (as far as I’m aware). This was a big sticking point for me. I told them I did not believe we should outsource support, but they did not believe we should have support for developer products at all, so I lost to that “compromise.” After that I volunteered myself to do the training of the support teams, which involved traveling to Manila, Japan and Mexico regularly. This did help but like support as whole, it was a losing battle and quality has declined over time. Your experience is definitely expected and perhaps even by design. Sadly this is true across Google, if you want help you’d best know a Googler.
- andybak 3y ago> they did not believe we should have support for developer products at all That explains a lot.
- xyzeva 3y agoWhat'd you expect, its google!
- GuB-42 3y agoI suspect it is going to end up being Google's downfall, or at least, be part of it. They simply don't know humans. Their repeated failures at building social networks is good enough evidence. They always try to have the human out of the loop, which, to be fair, worked for them in the early days, as their search engine was better than those that relied on human-made directories. But now it is becoming ridiculous. It is a company of bots, for bots. And when they need humans for some reason, they take away most of the value they can add with rigid frameworks, basically treating them like bots. They pay hundreds of thousands not for people who are competent and trustworthy to provide the best service, but instead, to people who write bots to provide mediocre service. I believe that at some point, a startup who understand humans will eat them up, bit by bit, by feeding on dissatisfied customers who don't want to deal with stupid bots.
- joshxyz 3y agodoing gods work here man, thank you for your work!
- MrBruh 3y agoNo worries, support like this really helps to make it feel worthwhile. :)
- pizzafeelsright 3y agoGo for the win. And we're still in the Wild West when it comes to internet business even after 20 years of "verified" domains.
- begueradj 3y agoThis reminds me of "How I pwned half of America’s fast food chains, simultaneously." https://mrbruh.com/chattr/ https://mrbruh.com/chattr/ HN: https://news.ycombinator.com/item?id=38933999 https://news.ycombinator.com/item?id=38933999
- MrBruh 3y agoThat's because I wrote both of those articles, and this is the sequel to that blog post. :P
- begueradj 3y agoIndeed :) Same authors ... referring to each others :)
- xyzeva 3y agoWe decided to make a shared blog because we will likely have other projects we will do together, so all of us posting on our personal blogs on the same topic would be counterproductive
- begueradj 3y agoThat's a good thing. There is nothing wrong with your approach. Glad you are sharing what you do because what you do interests many people.
- hk__2 3y agoThis is normal, the 6th word of the blog post is a link to it. > After the initial buzz of [pwning Chattr.ai] had settled down, […]
- balder1991 3y agoLaughed out loud with the “Customer support tried to flirt with me when attempting to report the issue”, “I want to be your gf, you very smart” lol the print looks like Kik Messenger?
- MrBruh 3y agoGlad to hear you enjoyed it, it was a messenger called "Line"
- kijin 3y agoThey must have trained their customer support AI with a database stolen from dating sites.
- MrBruh 3y agoCan't blame a company for trying to make their AI charismatic :D
- vishnugupta 3y agoAh nice! This brings back memories. I think this is a very popular messenger app in SEA region?
- phantomathkg 3y agoTaiwan, Japan and Hong Kong mostly. SEA used a mix of messaging app, where Whatsapp > Line.
- xyzeva 3y agoWe believe the gambling ring is based in Indonesia, which is uncommon to use Line, but they seem to be using it here for all of their customer support across all sites.
- robjan 3y agoNot really used in Hong Kong that much anymore apart from a small subset of people (WhatsApp is king). It's also heavily used in Thailand.
- minejerik 3y ago[flagged]
- rfl890 3y agoThe customer support gave me a good laugh. Thanks
- lawgimenez 3y agoCan you migrate data from Firebase database to PostgreSQL or similar?
- xyzeva 3y agoOn certain databases, yes We only scanned for firestore, which is a NoSQL database, conversion tools may still be possible, a good firebase alternative would be https://supabase.com https://supabase.com, but please set up RLS, its IMO much easier then Firebase.
- necovek 3y agoWouldn't it be easy to migrate a NoSQL database to Postgres without any adaptation? Postgres is an "object database", so you could use Array, JSON or JSONB fields wherever necessary, and you shouldn't introduce any foreign key relations or such.
- xyzeva 3y agoIt would, except if you have a f'ed up schema like most of these companies had.
- marcosdumay 3y agoThe point is that Postgres has no problem at all with a fucked-up schema. You just tell it not to check a thing, and it won't check a thing. But then, I may be underestimating how fucked-up things are.
- mooreds 3y agoDefinitely. They even let you export the password hashes (which you should do carefully). You can then import them into any identity provider that supports modified scrypt[0]. Your users will continue to be able to log in without a password reset. 0: https://firebase.google.com/docs/reference/admin/java/reference/com/google/firebase/auth/hash/Scrypt https://firebase.google.com/docs/reference/admin/java/refere...
- ddtaylor 3y agoGreat work and an awesome write up.
- johnnyAghands 3y ago900 Sites, 125 million accounts, 1 vulnerability, 0 Girlfriends.
- HaZeust 3y agoGreat comment, you really showed them!
- MrBruh 3y agoBro woke up on the wrong side of the bed
- throwaway984393 3y agoBro woke up with no gf :'(
- voidUpdate 3y agoApart from the customer service agent that tried to flirt with them :P
- rjbwork 3y agoCorrect me if I'm wrong, but 75% of sites with these vulns are still just hanging out there ready to be dumped, according to the end of this post? Insane. Some days I think one ought to be licensed to touch a computer.
- xyzeva 3y agoSadly, this is true, and theres probably much more. We did our best, sent customized emails to each of them, telling what was affected, how to fix it, and how to get in contact.
- MrBruh 3y agoCorrect, that's why we couldn't post a list of affected sites or malicious actors would immediately abuse it :/
- wavemode 3y agoIt seems reasonable to assume that the exposed information has already fallen into the wrong hands. Might as well post the list at this point (or at some point, at least) so that any users of those sites can become aware, no?
- tgv 3y agoIt'll now take them 2-3 weeks to get the details.
- prmoustache 3y agoShouldn't encrypting all databased records be the only sane, safe and legal solution with decryption key sent to local (to the website owner) law enforcement when site owners aren't responsive? Not saying you should do that given the current state of the laws.
- bsder 3y agoUntil PII compromise puts a company out of business, it's just a cost (to you) of doing business (for them).
- _tk_ 3y agoGood job! I’d be interested to know how you’re coming to the conclusion that the amount of affected users is likely higher. From the looks of it, I’d suspect that at least some of the sites you mention (gambling, lead carrot) to be littered with fake account data.
- MrBruh 3y agoWhen manually reviewing a lot of these sites it was not identifying PII that were in non-english since the automated scanner checks the variable name for known data types (e.g phone) but that would only work for English sites.
- xyzeva 3y agoWe confirmed that the gambling site is not fake data, I dont know about the lead one. Why we are saying its more is there is likely other services not in our scan list that could be vulnerable.
- zelon88 3y agobiggest threat on the web is Google. They lower the bar so low that people who have no business collecting user information are collecting user information, then they host it insecurely for you with no liability to the end user whatsoever. Not only that but they provide the same crappy services to schools and scummy gambling websites alike. It frustrates me watching people who believe they are professionals flock to these services. Honestly, if you can't roll your own you probably shouldn't let someone roll this for you. But Google won't say no and none of you cloud devs can help yourself. So we have this race to the bottom in cost and first to market and all the products are least common denominator shit that gets built in 6 hours by copy pasting as many GH repositories together as possible on rented infra.
- dazh 3y agoWhat would be the correct way to set up security to prevent this?
- xyzeva 3y agoSetting up firebase security rules: https://firebase.google.com/docs/rules/ https://firebase.google.com/docs/rules/
- suddenclarity 3y ago> Turns out that a Python program with ~500 threads will start to chew up memory over time. Anyone have more info about this issue? I've got a scraper myself in Python with a few hundred threads which seems to eat a lot of memory. Any workarounds or is the only solution to rewrite in another language?
- xyzeva 3y agoPython just isn't the language for this, really. Rewriting it is the only real solution, I don't know your exact problem.
- belorn 3y agoYou can do it in python but one has to dig into how python do reference counting and how that interacts with threads. Personally I prefer using processes rather than threads, with a worker pool and a message bus rather than shared memory. That solution has its own drawbacks (and a bit more overhead), but you don't need to worry so much about memory issues. Processes also seems a better match for crawlers since the number of processes will be fairly constant and the work the processes do is fairly independent.
- dist-epoch 3y agoYou really should be using ayncio. It's a perfect fit for your usecase.
- teddyh 3y agoimport multiprocessing as threading
- maipen 3y agoStuff like this, makes me thankful to have chosen password managers and virtual cards a long time ago... Still this makes the interent scarier. Most people don't have a clue how fragile the web is and how vunerable they are.
- xyzeva 3y agoYeah, funny how that works. Services as time goes on makes making websites easier, and abstracts more stuff, which makes devs oblivious to what they have to configure.
- user90131313 3y agoSomehow my assumption is it will only get worse from here, with AI agents looking for exploits etc with much more efficiently than bots? weird future is waiting
- tamimio 3y ago> chosen password managers It’s not enough; make sure to use a unique email for each service you sign up for. This limits the damage in case of an incident and protects your privacy, as no one can perform OSINT on you to cross reference other services. Additionally, I’ve found that sometimes you can detect a site breach before the owners do when you receive a malicious email sent to that unique address.
- maipen 3y ago> make sure to use a unique email for each service you sign up for. Unfortunatly that's a big hassle that I am not willing to go through. Apple's approach to pseudo emails was very nice and in my experience, works very well, but as mainly PC user I can't take advantage of this. Do you know or recommend a service for this thats easy and fast to use?
- tamimio 3y agoIt isn’t a hassle, you won’t create a separate email, but an aliases. > Do you know or recommend a service for this thats easy and fast to use? Honestly I don’t like to promote any commercial services, but there are few out there that automate it, simplelogin (I believe you can host it yourself too), anonaddy, or fastmail that has an integration with 1password, so your password manager generate a random pass and an email alias automatically for you. There are more and it’s better to research it yourself to find the best solution for you, again, my post sounds like shilling for these products but a good start.
- itqwertz 3y agoThis is the inevitable outcome of picking cheap-fast from the cheap-fast-good PM triangle. Unfortunately for some customers/users, their concerns were left out of the conversation and their PII is the cost. I’d be wary of any company listed here that made that decision and hasn’t changed leadership, as it has been proven time and time again that many companies simply don’t care enough about customers enough to protect them. History repeats itself.
- xyzeva 3y agoI agree for the most, but there was some good apples (even though very few) that were very thankful and fixed it fast.
- simonw 3y agoI have a very basic Firebase question: are most of the apps described in this post implemented entirely as statically hosted client-side JavaScript with no custom server-side code at all - the backend is 100% a hosted-by-Google Firebase configuration? If so, I hadn't realized how common that architecture had become for sites with millions of users.
- evantbyrne 3y agoYeah. Either entirely client-side or passing through a server naively. This is the inevitable result of having an "allow by default" security model in an API. Unfortunately, insecure defaults are a common theme with libraries targeted at JavaScript developers. GraphQL is another area I would expect to see these kinds of issues.
- hazelnut 3y ago> with no custom server-side code at all Could be a mix. Firebase also offers Firebase Functions which are callable functions in the cloud. That code is not public. However, Firestore or Firebase realtime database both require the user to setup security rules. Otherwise all data can be read by anybody.
- cryptonector 3y agoThat's a pretty crazy set-up, but it can work if appropriate authorization rules are coded into the SQL schema on the backend. Writing appropriate authz rules on the backend has to be made easy.
- hubraumhugo 3y agoHas Supabase learned from this and done a better job?
- tamimio 3y agoThat customer support looked like an automated AI response.. But I’m not surprised of the scale, years ago same thing happened with AWS cloud XY service, and you would find the token literally in plaintext in millions of smartphones apps.
- deleted 3y ago[deleted]
- cozzyd 3y agoSomeone should develop a browser plugin to warn you if a site is using firebase...
- andersa 3y agoI've never really understood how Firebase makes any sense. Why would you let the frontend access the database directly?
- bruce343434 3y agoBecause writing the boilerplate that hooks up the DB to an ORM Model to a ViewModel to a Router and then back via a Controller to the Model to the DB again sucks? A lot of the times it's equivalent to writing manual getters and setters except it's many many lines of code over many files... No wonder people are trying to cut corners!
- rglover 2y agoNever. Trust. The. Client.