6 ms·
Mailgun Security Incident and Important Customer Information
- devicenull 9y agoNo 2FA on staff accounts?
- deleted 9y ago[deleted]
- ppierald 9y ago> Finally, we’d like to assure our customers and partners that we take security at Mailgun very seriously. So very seriously that they don't even use https for their blog...
- StavrosK 9y agoWow, the certificate isn't even valid...
- menacingly 9y agoWell, it's for the wrong domain. Because they don't use SSL on their blog.
- deleted 9y ago[deleted]
- a_imho 9y agoFormer mailgun customer. Asked them to delete my personal data a couple of weeks ago (I was not able to do it myself... ) because I would rather they don't leak it in a security hiccup. They kindly refused to do so (as I don't believe any tech support can be that incompetent) and kept spamming my inbox instead. While the severity of this incident is not clear, never imagined curses can act on such a short notice.
- clhammer 9y agoHi there, This is Chris from the Mailgun team. I'm sorry that this happened, this shouldn't have been the case. I'd be happy to help rectify this issue, would you be able to send an email to help@mailgun.com with details so I can review?
- DarronWyke 9y agoThis is because Mailgun is in the practice of spam. The number of spam campaigns I've seen with Mailgun as the conduit is high, second only to Mailchimp.
- bwag 9y agoTo be fair, Mailgun is in the practice of sending email. It just happens to be that email is one of the main conduits of spam.
- DarronWyke 9y agoNo, that's just correlation. Email can be spam, but not all email is spam.
- trendia 9y agoCome on, Mailgun. Let's Encrypt is free and takes less than 5 minutes to set up (using certbot).
- jcadam 9y agoYea, I've been able to forget how painful getting SSL setup and configured used to be since letsencrypt + certbot came along. Automating that crap in ansible is almost too easy.
- r1ch 9y agoThis was used to steal bitcoin cash tips on Reddit by hijacking password reset emails (https://www.reddit.com/r/bugs/comments/7obxkb/mailgun_security_incident_an_update_on_the_state/ https://www.reddit.com/r/bugs/comments/7obxkb/mailgun_securi...) I find it amusing they still have a "trusted by Reddit" blurb on their homepage after this!
- ErikHuisman 9y agoOn 12/31, Reddit received several reports regarding password reset emails that were initiated and completed without the account owners’ requests. We have been working to investigate the issue and coordinating with Mailgun, a third-party vendor we’ve been using to send some of our account emails including password reset emails. A malicious actor targeted Mailgun and gained access to Reddit’s password reset emails. The nature of the exploit meant that an unauthorized person was able to access the contents of the reset email. This individual did not have access to either Reddit’s systems or to a redditor’s email account. As an immediate precautionary measure, we moved reset emails to an in-house mail server soon after we determined reset links were indeed being clicked without access to the user's email, and before Mailgun had confirmed to us that they were vulnerable. We know this is frustrating as a user, and we have put additional controls in place to help make sure it doesn’t happen again. We are continuing to work with Mailgun to make sure we have identified all impacted accounts. At this time, the overall number of confirmed impacted users is less than twenty. For those affected, we have resolved the issue and assisted in account recovery. Additional information about Mailgun’s security incident can be found on its blog here. We’re committed to keeping your Reddit account safe and will continue to monitor this situation carefully. u/sodypop, u/KeyserSosa, and I will be sitting around in the comments for any general questions.
- philipwhiuk 9y agoCarefully filed in an almost unread subreddit rather than in /r/announcements where it would be seen by everyone.
- simooooo 9y agoNever would have occurred to me that this could be used to intercept password reset emails. Very scary.
- Gys 9y ago> At this time, we believe less than 1% of our customer base was potentially affected. If you were not directly notified by Mailgun regarding this incident, then your account was not affected.
- rcMgD2BwE72F 9y agoDoes this only affect Mailgun's customers? If these customers hold data of third-party – let's call them "end-users" – in Mailgun accounts, Mailgun could/should communicate the total number of individuals affected. "1% of our customers/users" can affect millions of individuals.
- ram_rar 9y ago2FA, 2FA, 2FA!
- clon 9y agoWhy would employees need access to client API keys, as opposed to just client ID? Furthermore, this seems to indicate that the API keys are not hashed. I would expect some bits of the API key to work as an identifier and the rest of the bits treated as secret material (properly hashed). As a Mailgun customer, this is concerning..
- somedickhead 9y agoAs a former Rackspace employee, I had access to every customer secret IN PLAIN TEXT through multiple web-based systems with a click of a button (IE: business as usual).
- twunde 9y agoThanks for confirming this. I had my suspicions, especially after the last few years of using them and just seeing massive problems that seemed to be caused by the software at Rackspace.
- graystevens 9y agoAgreed. Super disappointed by this (cleartext details and the breach). Will be looking to move all services from Mailgun shortly.
- ad_hominem 9y agoWhen I get spam email, I usually check the headers and if it's coming from a reputable service (Postmark, Sendgrid, etc.) they usually have a web form or an abuse@ email to send the headers to so that they can shut down the account. Months ago I received spam from a Mailgun server and tried to use their web form[1] to report it, but it was broken. I reported both that bug and the spam email to their support, which acknowledged it. Weeks later I got another spam email from that same domain, popped open that report form and it was still broken (FWIW as of today it seems to be working again). So I followed up on my initial support request with that info but got no response. Just a few days ago I received another spam message from that domain. I personally consider all that a very bad sign in an email service provider and wouldn't use Mailgun myself. In contrast, I've been very happy with Postmark. [1]: https://www.mailgun.com/receiving-spam-from-mailgun https://www.mailgun.com/receiving-spam-from-mailgun
- aceoflala 9y agoPostmark costs money, Mailgun does not.
- ad_hominem 9y agoPostmark is free up to 100 mails / month. But mail deliverability issues are a hell that I'm happy to pay a small fee to avoid.
- eikenberry 9y agoDo they have an overage charge for the free 100/month, like the 1.25/1000 they list for their non-free use? So you could be free most of the time with the occasional 1.25 charge if you have a busy month?
- ad_hominem 9y agoUnfortunately I don't think so. :( > We offer a Free Trial plan for testing purposes only. The Trial is limited to 100 emails a month with no overages allowed. https://postmarkapp.com/support/article/1107-how-does-monthly-pricing-work https://postmarkapp.com/support/article/1107-how-does-monthl...
- OJFord 9y agoEr, can we expect more information to follow? 1. How was the employee's account accessed? No 2FA? 2. Do employees ordinarily have access to customer secrets (e.g. API keys) or was there some further exploit? 3. The advice in OP for affected customers is to roll keys and SMTP logins. Couldn't/shouldn't you do that for them? Surely security should trump up-time/deliverability?
- ufmace 9y agoFor 3. I'd say no way. There's no way for Mailgun to know what services are doing with those keys, how important those services are to their customers, how difficult it is for the service owners to rotate their keys, and how much bandwidth they have to do that right now. In an ideal world, every customer would have a good setup where they can rotate third-party supplier API keys painlessly and have plenty of bandwidth to handle security emergencies. Alas, there's a lot of bad setups out there, and some of them are critical to their customers' operations. Nothing I've personally worked with had a setup bad enough to make that painful, but I'd be very worried about how reckless a service is to rotate API keys that aren't being actively exploited to do something dangerous without getting a positive confirmation from the customer.
- somedickhead 9y agoAll Rackspace employees are issued hardware or software RSA tokens and a VPN client. I seriously suspect this was the job of an insider, not a compromised employee laptop.
- ralphm 9y agoMailGun has been spun out of Rackspace almost a year ago.
- gouggoug 9y agoIn those security disclosures, I often read what I see as contradictory language. For example, I'm confused by this kind of statement: > Mailgun has now completed its diagnostic of accounts that were affected and has notified each of the affected users. At this time, we believe less than 1% of our customer base was potentially affected. If you were not directly notified by Mailgun regarding this incident, then your account was not affected. If you believe that less than 1% of users were affected, it means you don't know for sure how many accounts were affected. From there, how can you state that "If you were not directly notified by Mailgun regarding this incident, then your account was not affected"? Doesn't this last statement mean you know for sure my account was not affected? Isn't it in direct contradiction with the previous statement?
- Goopplesoft 9y agoForemost, it was written by a human and unintended language contradictions are common. With that said, what you're suggesting isn't necessarily true -- the language can also indicate potential false positives, again because of the nuances of language.
- gouggoug 9y ago> unintended language contradictions are common Yes, definitely true. Although some contexts, like a security disclosure, might warrant a very carefully non-contradictory worded statement that leaves no doubts of interpretation. > the language can also indicate potential false positives, again because of the nuances of language. Yes, but in this context, false-positive aren't important to the audience of the disclosure. Nobody really cares if their account was "identified as affected, but in the end wasn't". If you announce that 1% of your user base was affected, and it turns out that 50% of this 1% were false-positive, great! You were still right in announcing that 1% of your user base was affected. You can always correct this later and announce that things panned out better and only 0.5% of your users were impacted.
- discreditable 9y agoThese sorts of articles always remind me of “We take security seriously”, otherwise known as “We didn’t take it seriously enough”: https://www.troyhunt.com/we-take-security-seriously-otherwise/ https://www.troyhunt.com/we-take-security-seriously-otherwis...
- deleted 9y ago[deleted]
- MechEStudent 9y agoOnly 1%? My eye. This has smell of Yahoo to it. I bet within 6 months, this goes up toward 10%. I bet they lost their entire data.
- deleted 9y ago[deleted]
- rajeemcariazo 9y agoI like Mailgun so much because of its simplicity but last November 2017 the default postmaster account of one of our domain in Mailgun was hacked. (I don't know where it was hacked but i suspect it was on the Mailgun server because I kept the secret key in my server very well). We moved to Sendgrid because my account in Mailgun got a very bad reputation. One of the hacked smtp credentials was used to send spam.