6 ms·

Bonsai Hosted ElasticSearch Is Down

53 points by sstarr 12y ago

dorfsmay 12y ago
Any word on how those accounts are getting compromised? Have they been complacent (easy password to guess, keys easy to be compromised (maybe in a public github repo)), or could there be some whole in the AWS secutiy model?
- count 12y ago
  If there were a hole in the AWS security model for this, I think it'd be pretty obvious pretty quickly, given what happens when US-East takes a dive... This happens constantly, and it's almost always through lack of best practices (as mentioned in higher up comment - IAM, MFA, etc.).
ihsw 12y ago
How were they 0wned? Lack of MFA, rogue API key, or something else? Are full-access accounts being handed out willy-nilly instead of IAM accounts? AWS Multi-Factor Authentication (MFA): http://aws.amazon.com/iam/details/mfa/ http://aws.amazon.com/iam/details/mfa/ AWS Identity Access and Management (IAM): http://docs.aws.amazon.com/IAM/latest/UserGuide/IAMBestPractices.html http://docs.aws.amazon.com/IAM/latest/UserGuide/IAMBestPract... Managing your AWS API Keys: http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSGettingStartedGuide/AWSCredentials.html http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSG... Go a step further with your AWS API keys and use AWS' API access logging (CloudTrail): http://aws.amazon.com/cloudtrail/ http://aws.amazon.com/cloudtrail/ Don't get burned. Check your stuff out.
- toomuchtodo 12y ago
  At this point, MFA for master AWS accounts should be mandatory.
  - earless1 12y ago
    MFA for all console accounts is the only right answer. If machines require credentials to do specific task or perform API calls then roles should be used.
    - toomuchtodo 12y ago
      > If machines require credentials to do specific task or perform API calls then roles should be used. Even then, if the data must be considered highly valuable/immutable, then versioning/delete protection should be enabled for the S3 bucket(s) in questions. This requires the MFA token to be in the API call for the delete to succeed.
- nzadrozny 12y ago
  Hey all, Bonsai cofounder here. It was an old API access key that got leaked, not our account credentials. We're still investigating how and where the key got leaked, but bottom line, it should have been revoked ages ago. 2FA is great, but it doesn't cover API keys. Rotate your API keys!
huntermeyer 12y ago
This brought my app down. http://jrdevjobs.com http://jrdevjobs.com. Our shards were all missing from Bonsai. We looped through each model and saved it. We're back up.
grandalf 12y ago
Not to focus on this when they are experiencing downtime, but Bonsai has been one of the least reliable service providers I've ever used.
- vegardx 12y ago
  Good that it's pretty easy to change elastic search provider with little downtime. I'd recommend checking out found.no. We've been pleased with performance and stability. Heck - I can't recall any downtime at all.
- donretag 12y ago
  qbox.io is another hosted Elasticsearch solution. I have never used them, but one of their developers is very knowledgeable and active on the mailing list.
jayzalowitz 12y ago
This sucks... I am happy we just put our search cluster on elasticbeanstalk atm, but I wish we had more services like this running.. good news is new security practices will hop up everywhere because of this.
bdcravens 12y ago
Full-blown AWS console compromise - this sounds similar to what happened to Code Spaces (https://news.ycombinator.com/item?id=7909791 https://news.ycombinator.com/item?id=7909791) Is there a new vulnerability?
- dorfsmay 12y ago
  My concern exactly.
- nzadrozny 12y ago
  It wasn't a management console compromise, it was an old API key that got leaked.
huslage 12y ago
AWS needs to improve usability of IAM so that it gets broader adoption. The learning curve is non-trivial.
- dorfsmay 12y ago
  So is creating policies.... They managed to make dealing with json worse than dealing with xml.
- earless1 12y ago
  What particular issues have you had with IAM? I find that the wizard is a good starting point for understanding the policies.
kitwalker12 12y ago
our site http://www.violetgrey.com http://www.violetgrey.com went down because of this. Luckily we were able to reindex pretty fast before their backups kicked in. Any ideas on how to have fallbacks in such cases?

53 points by sstarr 12y ago

Any word on how those accounts are getting compromised? Have they been complacent (easy password to guess, keys easy to be compromised (maybe in a public github repo)), or could there be some whole in the AWS secutiy model?

count 12y ago
If there were a hole in the AWS security model for this, I think it'd be pretty obvious pretty quickly, given what happens when US-East takes a dive... This happens constantly, and it's almost always through lack of best practices (as mentioned in higher up comment - IAM, MFA, etc.).

ihsw 12y ago

How were they 0wned? Lack of MFA, rogue API key, or something else? Are full-access accounts being handed out willy-nilly instead of IAM accounts? AWS Multi-Factor Authentication (MFA): http://aws.amazon.com/iam/details/mfa/ http://aws.amazon.com/iam/details/mfa/ AWS Identity Access and Management (IAM): http://docs.aws.amazon.com/IAM/latest/UserGuide/IAMBestPractices.html http://docs.aws.amazon.com/IAM/latest/UserGuide/IAMBestPract... Managing your AWS API Keys: http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSGettingStartedGuide/AWSCredentials.html http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSG... Go a step further with your AWS API keys and use AWS' API access logging (CloudTrail): http://aws.amazon.com/cloudtrail/ http://aws.amazon.com/cloudtrail/ Don't get burned. Check your stuff out.

toomuchtodo 12y ago
At this point, MFA for master AWS accounts should be mandatory.
- earless1 12y ago
  MFA for all console accounts is the only right answer. If machines require credentials to do specific task or perform API calls then roles should be used.
  - toomuchtodo 12y ago
    > If machines require credentials to do specific task or perform API calls then roles should be used. Even then, if the data must be considered highly valuable/immutable, then versioning/delete protection should be enabled for the S3 bucket(s) in questions. This requires the MFA token to be in the API call for the delete to succeed.
nzadrozny 12y ago
Hey all, Bonsai cofounder here. It was an old API access key that got leaked, not our account credentials. We're still investigating how and where the key got leaked, but bottom line, it should have been revoked ages ago. 2FA is great, but it doesn't cover API keys. Rotate your API keys!

huntermeyer 12y ago

This brought my app down. http://jrdevjobs.com http://jrdevjobs.com. Our shards were all missing from Bonsai. We looped through each model and saved it. We're back up.

grandalf 12y ago

Not to focus on this when they are experiencing downtime, but Bonsai has been one of the least reliable service providers I've ever used.

vegardx 12y ago
Good that it's pretty easy to change elastic search provider with little downtime. I'd recommend checking out found.no. We've been pleased with performance and stability. Heck - I can't recall any downtime at all.
donretag 12y ago
qbox.io is another hosted Elasticsearch solution. I have never used them, but one of their developers is very knowledgeable and active on the mailing list.

jayzalowitz 12y ago

This sucks... I am happy we just put our search cluster on elasticbeanstalk atm, but I wish we had more services like this running.. good news is new security practices will hop up everywhere because of this.

bdcravens 12y ago

Full-blown AWS console compromise - this sounds similar to what happened to Code Spaces (https://news.ycombinator.com/item?id=7909791 https://news.ycombinator.com/item?id=7909791) Is there a new vulnerability?

dorfsmay 12y ago
My concern exactly.
nzadrozny 12y ago
It wasn't a management console compromise, it was an old API key that got leaked.

huslage 12y ago

AWS needs to improve usability of IAM so that it gets broader adoption. The learning curve is non-trivial.

dorfsmay 12y ago
So is creating policies.... They managed to make dealing with json worse than dealing with xml.
earless1 12y ago
What particular issues have you had with IAM? I find that the wizard is a good starting point for understanding the policies.

kitwalker12 12y ago

our site http://www.violetgrey.com http://www.violetgrey.com went down because of this. Luckily we were able to reindex pretty fast before their backups kicked in. Any ideas on how to have fallbacks in such cases?