6 ms·

Security Hole in Sendgrid

181 points by ndaiger 12y ago

MasterScrat 12y ago
The problem is that it was technically possible, for a representative, to make this change without the proper verification. You just can't rely on humans for that.
- bostonvaulter2 12y ago
  > "... confirms your suspicion that these people convinced one of our representatives to change the email address on file." This is the part that scares me. Do they not have auditing in the system where the representatives are able to change the email address on file?
- dangrossman 12y ago
  SendGrid once went over my head and changed my account settings on behalf of a customer of mine, without even consulting me beforehand. I had a customer with a history of reporting mails as spam (activity reports he requested in a webapp). When you report a mail as spam, the address goes into a blacklist to avoid causing sender reputation issues. After the third or so time of having him ask to get the mails again, then mark one of them as spam, I told him I wouldn't be offering that feature to him any longer. He contacted SendGrid about it directly, and the SendGrid rep actually went into my account and added his address to a whitelist to bypass the blacklist, where I intended it to remain. I thought that was just the strangest thing, and it didn't sit well with me. Accessing a customer's account when they request service is one thing, but making changes on behalf of a stranger you know has no authority over that account? That was when I moved the last of my apps over to Mandrill.
- thrownaway2424 12y ago
  And people complain that Google has bad support! At least you can be certain that there's no way an attacker can get the Google support rep on the phone. There aren't any.
toomuchtodo 12y ago
Use Amazon SES to generate your outbound emails; ensure proper IAM policies, and that you're using 2 factor auth to login to your AWS account.
- balls187 12y ago
  Is AWS any less susceptible to Social Engineering attack of this type? Specifically--can AWS support staff grant access to AWS accounts, and if so: what are their criteria for doing so, and what are the policies in place to ensure those criteria are met, and how are those policies audited? As a TechStars alum, my company was granted $50k in AWS credits, which were tied to my AWS account[1]. When I left the Company, the CEO was able to get the credits moved to a different AWS account that was company owned, without my intervention at all, even though I was the only account owner. The fact that he could have credits moved out of the account without any kind of verification from me[2], should be cause for concern. [1] I should have created a new Amazon account for a group email [2] Obviously the credits belong to the company; they weren't mine to use, so I would have authorized the migration.
  - maxvu 12y ago
    It doesn't directly answer your question, but I don't think AWS has support staff -- just engineers that answer issue tickets.
  - driverdan 12y ago
    Amazon (non-AWS) is infamous for being vulnerable to social engineering attacks. I don't know if they've changed their polices more recently but they are (or were) often the first attack vector for social engineering. If you can get access to an Amazon account you can get the last 4 digits of the user's credit card number(s). You can then use that info to reset accounts over the phone with other companies.
cjbprime 12y ago
Looks like a deeply unsatisfactory response from SendGrid. They don't even know for sure ("it appears .. pretty much confirms") that their own support staff changed the email address?
- arjie 12y ago
  This shows why so-called PR-speak is necessary. Email like this should not be in a conversational style because it can easily be misinterpreted.
  - rnovak 12y ago
    are you serious? How was it misinterpreted? It point blank says that that's what happened? Please illustrate how exactly it was manipulated to say something else?
staunch 12y ago
Another title for this submission could have been: "Massive Security Hole in ChunkHost. Non-2FA accounts can be owned." Because it turns out anyone with a Sendgrid Support account also effectively had potential access to any account at ChunkHost not using two-factor authentication. Which is also true of thousands of other companies that are relaying their password reset emails through third party SMTP services. SendGrid seems lame, for allowing this and for their response promising to yell more loudly at their support people, but they're an SMTP relay service not an authentication service.
- Meekro 12y ago
  Are there any web hosting companies that don't rely on the "send a reset link to your email address on file" model of password resets? You're right, that model is deeply broken if anyone can intercept those emails (as happened in this case), but it seems unfair to single out ChunkHost for criticism.
  - deleted 12y ago
    [deleted]
  - adambenayoun 12y ago
    With that same logic in mind - we could say that anyone working at amazon AWS or Rackspace (or any other hosting companies) could gain access to your application. The thing is we trust these companies to have processes in place so that their representatives won't have the ability to potentially do something destructive and if they can because they are the highest ranked rep and they need that kind of access - then at least auditing and training should be in place to avoid that kind of behavior.
    - unclebucknasty 12y ago
      SoftLayer still asks for admin/root passwords to your boxes in pretty much any support scenario. Their ticketing system actually has a field for it in the submission form. If you omit it, the assigned tech will frequently ask for it. Always sends a little shiver down my spine.
hadoukenio 12y ago
So what's the answer? Here's two very legitimate scenarios: 1) You sign up, enable two-factor auth, then lock yourself out (lost password and your second-factor). How do you prove to the service provider that you are you? 2) You sign up, enable two-factor auth, then Mallory claims that they locked themselves out. How does the service provider prove that Mallory is not you?
- sanderjd 12y ago
  In this case, looping in the original email address on the SendGrid account before changing to a new one would have kept this from happening. SendGrid's support personnel should almost certainly not be able to change an email address without the change being signed off on through the old address first.
  - uptown 12y ago
    But what happens in those rare cases where that first account gets lost / locked permanently?
    - hadoukenio 12y ago
      Yes, this is part of what I'm trying to get answers to. Do you tell the user on signup to print an in-case-of-emergency-break-glass password which is only ever to be used to get into a locked account and other special circumstances? It may seem over the top but seeing as it's unique across service providers, I think it's a hell of a lot better than the overly abused "what is your mother's maiden name" type questions. I consider these questions to be in the same boat as sharing passwords between websites (since they are)!
      - vertex-four 12y ago
        The solution is to do what everyone who actually needs authentication from a company does; require a posted signed letter from a director, possibly along with an outbound (from SendGrid to the director) phone call to confirm. There's plenty of low-tech ways to confirm that a company really wants to do something.
callesgg 12y ago
Send your emails yourself. It's not like it is hard. At least not harder than integrating to a third party email sender.
- IgorPartola 12y ago
  If you use EC2 or the like your IP address or the entire address block could have easily ended up on a spam list so your email will be blocked. I wish SendGrid was only necessary for people who sends lots of mail but the reality is that no cloy provider can guarantee that email from their IP addresses will be delivered.
  - toomuchtodo 12y ago
    If you invoke sending email through Amazon SES, it's free for the first 2K per day, due EC2 blocks being on spam lists. http://aws.amazon.com/ses/pricing/ http://aws.amazon.com/ses/pricing/ "You can send 2,000 messages for free each day when you call Amazon SES from an Amazon EC2 instance directly or through AWS Elastic Beanstalk."
    - gtaylor 12y ago
      We just switched off of SES because of the lack of bounce/rejection diagnostics, and their internal blacklisting policies are really aggressive. If someone's email server goes down for a few hours, they're blacklisted for quite some time, even after it comes back up. After using SES for close to two years, I'd suggest looking elsewhere if you really care about deliverability or stats/metrics. We switched to Mandrill a few months ago and have been very impressed. It's still a tiny, microscopic percentage of our budget, but we get so much more (open/click reporting, sub-accounts, rendered email body history, much better blacklist/whitelist management).
      - toomuchtodo 12y ago
        Thanks for the recommendation! Will have to check them out.
      - 12y ago
IgorPartola 12y ago
So what they are saying is that SendGrid should have had two-factor auth and this would have never happened.
- Meekro 12y ago
  SendGrid's policy (as stated in their first email) is that the support people shouldn't be changing account emails in this fashion. Even if SendGrid had 2 factor auth, who's to say the support guy wouldn't have just disabled that?
  - jrochkind1 12y ago
    If their policy is that support staff should never be able to change an accounts email address... why does the system let them do it?
    - deleted 12y ago
      [deleted]
    - IgorPartola 12y ago
      At some point someone has direct DB access and can do this. Sure, normal support people don't but this just makes the social engineering aspect a bit more complex, not impossible.
  - IgorPartola 12y ago
    With TFA if the email got changed it would not make a difference. The attacker would need the second factor to rest the password and to log in. So the worst they could do is to lock out the account owner. The support staff being socially engineered is a different story, but yes this is a security hole in SendGrid's system and an easily patched one at that.
ogdenogly 12y ago
Sendgrid seems to be its own worst enemy (Adria Richards debacle, now this). I suggest we dub the act or state of being one's own worst enemy, being "sendgriddled."
jtchang 12y ago
Social engineering will almost always work. I don't really fault Sendgrid for this (though I could see this not working as well if you were using Amazon SES...no support to even talk to!). It sucks that they got caught with their pants down but I bet a good social engineering attempt on ChunkHost might have yielded similar results. The lesson here is to have multiple defenses. 2 factor auth is a great start and it worked in this case.
- mantrax3 12y ago
  In a nutshell, social engineering can be countered by proper software & processes engineering. So if social engineering is possible, blame the software architects. Maybe in extreme cases changing the email of an account might be needed, but there's no excuse a first level rep was able to do it. Least thing, he/she should've been forced by the system to escalate to someone above her, who has a much lesser chance to screw up.
  - danielweber 12y ago
    They didn't say it was first-level rep. Maybe the rep passed it up the chain. This was a pretty weak request, though. This one should have been pretty easy to at least make some attempt to verify. But as a human being I can verify how hard it is to not help the person crying on the other end of the phone because they need help RIGHT NOW. This didn't get to that level, but if you are designing a recovery process, you really need to think about how you handle that situation, and make sure the people making the call have the guts to say "I'm sorry but we need to do this one by the book."
- jyap 12y ago
  You should fault Sendgrid as they specifically have a policy NOT to perform this change of email request (from the article). Sendgrid can also change their systems so that phone support personnel can NOT perform this change or perform this change with approval from a supervisor. Sendgrid being in the business they are in should also know that they are susceptible to these types of attacks and what they can lead to (many, many systems which can have password requests sent to email addresses).
ef47d35620c1 12y ago
Email accounts are the weak link for many things... DNS registrations, web hosting, etc. Get the email account and you have it all.
- danielweber 12y ago
  I'm coming around to thinking you should get a secret email address for your sensitive accounts. Treat the string of that email address as almost as important as the string of your password. The problem is that you do have to tell it to people who probably don't share that philosophy to the same extreme.
steven2012 12y ago
We use Sendgrid and have hundreds of thousands of customers that might be phished by this social engineering trick. It's absolutely unacceptable that such a crucial piece of infrastructure is vulnerable to such a simple trick. I'm going to bring this up with our team and see if there's another vendor that can more reliably protect our customers.
- mantrax3 12y ago
  Your logic is a bit weird. Sendgrid just experienced this major embarrassment and are currently re-training their staff to avoid it again at all costs. And you're going to move away from them now?
  - steven2012 12y ago
    I'll talk it over with my group. Retraining doesn't guarantee anything. It could be that this problem is endemic to the company itself. For example, why should 1st level support have the ability to make major changes like this? It sounds like only 2nd level support, a smaller group of more highly trained support staff, should have the ability to do this. Does SendGrid have enough money/resources to split their team into 1st and 2nd level support? Would a larger company have those type of resources that would better protect my customers? These are questions I will talk over with my team.
  - mikeg8 12y ago
    The fact the Sendgrid even has to do re-training in the first place is the problem. They only need to do and handful of things well and keeping their users accounts secure is arguably the most important. If they are having issues like this this far along in their lifespan, it's a sign of more systemic issues in their company and does not instil much confidence to potential/current customers. Wanting to switch vendors doesn't seem weird at all to me.
  - rmrfrmrf 12y ago
    I think you're being lenient with SendGrid because it was a close call and not a full-blown catastrophe. First of all, SendGrid lied about its support staff's permissions. After the incident, SendGrid then sends an "oops! our bad!" e-mail where the employee in question will apparently be gently tapped on the wrist and maybe send a passive-aggressive reprimand. I mean, keep in mind that this is the same company that publicly crucified a female employee in order to stop a DDoS attack. Clearly there are some priorities out of whack there, and given the insecure nature of e-mail in the first place, I would never want to deal with a company that is so clearly unprofessional.
wojcikstefan 12y ago
It's an important lesson for all of us. I've seen a lot of privacy ignorance when it comes to support (e.g. folks handing over sensitive data after an anonymous request on Olark). We should all go an extra mile and verify the identity of the requestor. 1) If your support chat doesn't enforce authorization, always ask the requestor to send you an email. 2) Make sure the domain is correct (that's where Sendgrid screwed up). 3) Never agree on replying to a different email address than the one of the sender.
jusben1369 12y ago
What I found most interesting was they were targeted due to Bitcoin. Today most services would never store credit cards themselves but rather have them stored at a secure payment gateway. So they're unlikely to be attacked to get to those cards. T Trust me I know that CC's are getting sniffed in transit too often so I'm not saying they're 'safe'. I'm just wondering if there is something unique to Bitcoin that suddenly makes you a target as though you were known to be storing CC's data at rest onsite.
- driverdan 12y ago
  Bitcoin is like cash. It's far more valuable than credit card data.
davesque 12y ago
Alas, the weakest part of the system is...
foxylad 12y ago
My first thought was to whois chunkhost.info, which returns a clear name, email address and phone number. Or is it that easy to register a domain with a fictitious persona?
- johns 12y ago
  It's that easy. You can use anything you want.
- ceejayoz 12y ago
  > Or is it that easy to register a domain with a fictitious persona? Virtually every registrar allows instant update of whois records via a web interface. I can have Bill Gates be the technical contact on my personal domain in about a minute.
gmansoor 12y ago
BCC every message is evil, as it can be misused as in this case. SendGrid should never allow that, or at least should flag such behavior. At the minimum, they should notified account owners of this change.
- icebraining 12y ago
  The attacker got SG to change the email on file, so the notification would just be sent to him.
deleted 12y ago
[deleted]
platz 12y ago
http://twofactorauth.org/ http://twofactorauth.org/
hoodoof 12y ago
SMS verifications should be sent out to allow over the phone changes.
rootuid 12y ago
It's not a security hole, misleading title. Should read "social engineering resulted in security breach" and guess what, this happens all the time whether sendGrid or not.

181 points by ndaiger 12y ago

The problem is that it was technically possible, for a representative, to make this change without the proper verification. You just can't rely on humans for that.

bostonvaulter2 12y ago
> "... confirms your suspicion that these people convinced one of our representatives to change the email address on file." This is the part that scares me. Do they not have auditing in the system where the representatives are able to change the email address on file?
dangrossman 12y ago
SendGrid once went over my head and changed my account settings on behalf of a customer of mine, without even consulting me beforehand. I had a customer with a history of reporting mails as spam (activity reports he requested in a webapp). When you report a mail as spam, the address goes into a blacklist to avoid causing sender reputation issues. After the third or so time of having him ask to get the mails again, then mark one of them as spam, I told him I wouldn't be offering that feature to him any longer. He contacted SendGrid about it directly, and the SendGrid rep actually went into my account and added his address to a whitelist to bypass the blacklist, where I intended it to remain. I thought that was just the strangest thing, and it didn't sit well with me. Accessing a customer's account when they request service is one thing, but making changes on behalf of a stranger you know has no authority over that account? That was when I moved the last of my apps over to Mandrill.
thrownaway2424 12y ago
And people complain that Google has bad support! At least you can be certain that there's no way an attacker can get the Google support rep on the phone. There aren't any.

toomuchtodo 12y ago

Use Amazon SES to generate your outbound emails; ensure proper IAM policies, and that you're using 2 factor auth to login to your AWS account.

balls187 12y ago
Is AWS any less susceptible to Social Engineering attack of this type? Specifically--can AWS support staff grant access to AWS accounts, and if so: what are their criteria for doing so, and what are the policies in place to ensure those criteria are met, and how are those policies audited? As a TechStars alum, my company was granted $50k in AWS credits, which were tied to my AWS account[1]. When I left the Company, the CEO was able to get the credits moved to a different AWS account that was company owned, without my intervention at all, even though I was the only account owner. The fact that he could have credits moved out of the account without any kind of verification from me[2], should be cause for concern. [1] I should have created a new Amazon account for a group email [2] Obviously the credits belong to the company; they weren't mine to use, so I would have authorized the migration.
- maxvu 12y ago
  It doesn't directly answer your question, but I don't think AWS has support staff -- just engineers that answer issue tickets.
- driverdan 12y ago
  Amazon (non-AWS) is infamous for being vulnerable to social engineering attacks. I don't know if they've changed their polices more recently but they are (or were) often the first attack vector for social engineering. If you can get access to an Amazon account you can get the last 4 digits of the user's credit card number(s). You can then use that info to reset accounts over the phone with other companies.

cjbprime 12y ago

Looks like a deeply unsatisfactory response from SendGrid. They don't even know for sure ("it appears .. pretty much confirms") that their own support staff changed the email address?

arjie 12y ago
This shows why so-called PR-speak is necessary. Email like this should not be in a conversational style because it can easily be misinterpreted.
- rnovak 12y ago
  are you serious? How was it misinterpreted? It point blank says that that's what happened? Please illustrate how exactly it was manipulated to say something else?

staunch 12y ago

Another title for this submission could have been: "Massive Security Hole in ChunkHost. Non-2FA accounts can be owned." Because it turns out anyone with a Sendgrid Support account also effectively had potential access to any account at ChunkHost not using two-factor authentication. Which is also true of thousands of other companies that are relaying their password reset emails through third party SMTP services. SendGrid seems lame, for allowing this and for their response promising to yell more loudly at their support people, but they're an SMTP relay service not an authentication service.

Meekro 12y ago
Are there any web hosting companies that don't rely on the "send a reset link to your email address on file" model of password resets? You're right, that model is deeply broken if anyone can intercept those emails (as happened in this case), but it seems unfair to single out ChunkHost for criticism.
- deleted 12y ago
  [deleted]
- adambenayoun 12y ago
  With that same logic in mind - we could say that anyone working at amazon AWS or Rackspace (or any other hosting companies) could gain access to your application. The thing is we trust these companies to have processes in place so that their representatives won't have the ability to potentially do something destructive and if they can because they are the highest ranked rep and they need that kind of access - then at least auditing and training should be in place to avoid that kind of behavior.
  - unclebucknasty 12y ago
    SoftLayer still asks for admin/root passwords to your boxes in pretty much any support scenario. Their ticketing system actually has a field for it in the submission form. If you omit it, the assigned tech will frequently ask for it. Always sends a little shiver down my spine.

hadoukenio 12y ago

So what's the answer? Here's two very legitimate scenarios: 1) You sign up, enable two-factor auth, then lock yourself out (lost password and your second-factor). How do you prove to the service provider that you are you? 2) You sign up, enable two-factor auth, then Mallory claims that they locked themselves out. How does the service provider prove that Mallory is not you?

sanderjd 12y ago
In this case, looping in the original email address on the SendGrid account before changing to a new one would have kept this from happening. SendGrid's support personnel should almost certainly not be able to change an email address without the change being signed off on through the old address first.
- uptown 12y ago
  But what happens in those rare cases where that first account gets lost / locked permanently?
  - hadoukenio 12y ago
    Yes, this is part of what I'm trying to get answers to. Do you tell the user on signup to print an in-case-of-emergency-break-glass password which is only ever to be used to get into a locked account and other special circumstances? It may seem over the top but seeing as it's unique across service providers, I think it's a hell of a lot better than the overly abused "what is your mother's maiden name" type questions. I consider these questions to be in the same boat as sharing passwords between websites (since they are)!
    - vertex-four 12y ago
      The solution is to do what everyone who actually needs authentication from a company does; require a posted signed letter from a director, possibly along with an outbound (from SendGrid to the director) phone call to confirm. There's plenty of low-tech ways to confirm that a company really wants to do something.

callesgg 12y ago

Send your emails yourself. It's not like it is hard. At least not harder than integrating to a third party email sender.

IgorPartola 12y ago
If you use EC2 or the like your IP address or the entire address block could have easily ended up on a spam list so your email will be blocked. I wish SendGrid was only necessary for people who sends lots of mail but the reality is that no cloy provider can guarantee that email from their IP addresses will be delivered.
- toomuchtodo 12y ago
  If you invoke sending email through Amazon SES, it's free for the first 2K per day, due EC2 blocks being on spam lists. http://aws.amazon.com/ses/pricing/ http://aws.amazon.com/ses/pricing/ "You can send 2,000 messages for free each day when you call Amazon SES from an Amazon EC2 instance directly or through AWS Elastic Beanstalk."
  - gtaylor 12y ago
    We just switched off of SES because of the lack of bounce/rejection diagnostics, and their internal blacklisting policies are really aggressive. If someone's email server goes down for a few hours, they're blacklisted for quite some time, even after it comes back up. After using SES for close to two years, I'd suggest looking elsewhere if you really care about deliverability or stats/metrics. We switched to Mandrill a few months ago and have been very impressed. It's still a tiny, microscopic percentage of our budget, but we get so much more (open/click reporting, sub-accounts, rendered email body history, much better blacklist/whitelist management).
    - toomuchtodo 12y ago
      Thanks for the recommendation! Will have to check them out.
    - 12y ago

IgorPartola 12y ago

So what they are saying is that SendGrid should have had two-factor auth and this would have never happened.

Meekro 12y ago
SendGrid's policy (as stated in their first email) is that the support people shouldn't be changing account emails in this fashion. Even if SendGrid had 2 factor auth, who's to say the support guy wouldn't have just disabled that?
- jrochkind1 12y ago
  If their policy is that support staff should never be able to change an accounts email address... why does the system let them do it?
  - deleted 12y ago
    [deleted]
  - IgorPartola 12y ago
    At some point someone has direct DB access and can do this. Sure, normal support people don't but this just makes the social engineering aspect a bit more complex, not impossible.
- IgorPartola 12y ago
  With TFA if the email got changed it would not make a difference. The attacker would need the second factor to rest the password and to log in. So the worst they could do is to lock out the account owner. The support staff being socially engineered is a different story, but yes this is a security hole in SendGrid's system and an easily patched one at that.

ogdenogly 12y ago

Sendgrid seems to be its own worst enemy (Adria Richards debacle, now this). I suggest we dub the act or state of being one's own worst enemy, being "sendgriddled."

jtchang 12y ago

Social engineering will almost always work. I don't really fault Sendgrid for this (though I could see this not working as well if you were using Amazon SES...no support to even talk to!). It sucks that they got caught with their pants down but I bet a good social engineering attempt on ChunkHost might have yielded similar results. The lesson here is to have multiple defenses. 2 factor auth is a great start and it worked in this case.

mantrax3 12y ago
In a nutshell, social engineering can be countered by proper software & processes engineering. So if social engineering is possible, blame the software architects. Maybe in extreme cases changing the email of an account might be needed, but there's no excuse a first level rep was able to do it. Least thing, he/she should've been forced by the system to escalate to someone above her, who has a much lesser chance to screw up.
- danielweber 12y ago
  They didn't say it was first-level rep. Maybe the rep passed it up the chain. This was a pretty weak request, though. This one should have been pretty easy to at least make some attempt to verify. But as a human being I can verify how hard it is to not help the person crying on the other end of the phone because they need help RIGHT NOW. This didn't get to that level, but if you are designing a recovery process, you really need to think about how you handle that situation, and make sure the people making the call have the guts to say "I'm sorry but we need to do this one by the book."
jyap 12y ago
You should fault Sendgrid as they specifically have a policy NOT to perform this change of email request (from the article). Sendgrid can also change their systems so that phone support personnel can NOT perform this change or perform this change with approval from a supervisor. Sendgrid being in the business they are in should also know that they are susceptible to these types of attacks and what they can lead to (many, many systems which can have password requests sent to email addresses).

ef47d35620c1 12y ago

Email accounts are the weak link for many things... DNS registrations, web hosting, etc. Get the email account and you have it all.

danielweber 12y ago
I'm coming around to thinking you should get a secret email address for your sensitive accounts. Treat the string of that email address as almost as important as the string of your password. The problem is that you do have to tell it to people who probably don't share that philosophy to the same extreme.

steven2012 12y ago

We use Sendgrid and have hundreds of thousands of customers that might be phished by this social engineering trick. It's absolutely unacceptable that such a crucial piece of infrastructure is vulnerable to such a simple trick. I'm going to bring this up with our team and see if there's another vendor that can more reliably protect our customers.

mantrax3 12y ago
Your logic is a bit weird. Sendgrid just experienced this major embarrassment and are currently re-training their staff to avoid it again at all costs. And you're going to move away from them now?
- steven2012 12y ago
  I'll talk it over with my group. Retraining doesn't guarantee anything. It could be that this problem is endemic to the company itself. For example, why should 1st level support have the ability to make major changes like this? It sounds like only 2nd level support, a smaller group of more highly trained support staff, should have the ability to do this. Does SendGrid have enough money/resources to split their team into 1st and 2nd level support? Would a larger company have those type of resources that would better protect my customers? These are questions I will talk over with my team.
- mikeg8 12y ago
  The fact the Sendgrid even has to do re-training in the first place is the problem. They only need to do and handful of things well and keeping their users accounts secure is arguably the most important. If they are having issues like this this far along in their lifespan, it's a sign of more systemic issues in their company and does not instil much confidence to potential/current customers. Wanting to switch vendors doesn't seem weird at all to me.
- rmrfrmrf 12y ago
  I think you're being lenient with SendGrid because it was a close call and not a full-blown catastrophe. First of all, SendGrid lied about its support staff's permissions. After the incident, SendGrid then sends an "oops! our bad!" e-mail where the employee in question will apparently be gently tapped on the wrist and maybe send a passive-aggressive reprimand. I mean, keep in mind that this is the same company that publicly crucified a female employee in order to stop a DDoS attack. Clearly there are some priorities out of whack there, and given the insecure nature of e-mail in the first place, I would never want to deal with a company that is so clearly unprofessional.

wojcikstefan 12y ago

It's an important lesson for all of us. I've seen a lot of privacy ignorance when it comes to support (e.g. folks handing over sensitive data after an anonymous request on Olark). We should all go an extra mile and verify the identity of the requestor. 1) If your support chat doesn't enforce authorization, always ask the requestor to send you an email. 2) Make sure the domain is correct (that's where Sendgrid screwed up). 3) Never agree on replying to a different email address than the one of the sender.

jusben1369 12y ago

What I found most interesting was they were targeted due to Bitcoin. Today most services would never store credit cards themselves but rather have them stored at a secure payment gateway. So they're unlikely to be attacked to get to those cards. T Trust me I know that CC's are getting sniffed in transit too often so I'm not saying they're 'safe'. I'm just wondering if there is something unique to Bitcoin that suddenly makes you a target as though you were known to be storing CC's data at rest onsite.

driverdan 12y ago
Bitcoin is like cash. It's far more valuable than credit card data.

davesque 12y ago

Alas, the weakest part of the system is...

foxylad 12y ago

My first thought was to whois chunkhost.info, which returns a clear name, email address and phone number. Or is it that easy to register a domain with a fictitious persona?

johns 12y ago
It's that easy. You can use anything you want.
ceejayoz 12y ago
> Or is it that easy to register a domain with a fictitious persona? Virtually every registrar allows instant update of whois records via a web interface. I can have Bill Gates be the technical contact on my personal domain in about a minute.

gmansoor 12y ago

BCC every message is evil, as it can be misused as in this case. SendGrid should never allow that, or at least should flag such behavior. At the minimum, they should notified account owners of this change.

icebraining 12y ago
The attacker got SG to change the email on file, so the notification would just be sent to him.

deleted 12y ago

[deleted]

platz 12y ago

http://twofactorauth.org/ http://twofactorauth.org/

hoodoof 12y ago

SMS verifications should be sent out to allow over the phone changes.

rootuid 12y ago

It's not a security hole, misleading title. Should read "social engineering resulted in security breach" and guess what, this happens all the time whether sendGrid or not.