7 ms·

Cloudtrail events should be able to demonstrate WHAT created the EC2s. Off the top of my head I think it's the runinstance event.

sylens 11mo ago
RunInstances
ThreatSystems 11mo ago
I'm officially off of AWS so don't have any consoles to check against, but back on a laptop. Based on docs and some of the concerns about this happening to someone else, I would probably start with the following: 1. Check who/what created those EC2s[0] using the console to query: eventSource:ec2.amazonaws.com eventName:RunInstances 2. Based on the userIdentity field, query the following actions. 3. Check if someone manually logged into Console (identity dependent) [1]: eventSource:signin.amazonaws.com userIdentity.type:[Root/IAMUser/AssumedRole/FederatedUser/AWSLambda] eventName:ConsoleLogin 4. Check if someone authenticated against Security Token Service (STS) [2]: eventSource:sts.amazonaws.com eventName:GetSessionToken 5. Check if someone used a valid STS Session to AssumeRole: eventSource:sts.amazonaws.com eventName:AssumeRole userIdentity.arn (or other identifier) 6. Check for any new IAM Roles/Accounts made for persistence: eventSource:iam.amazonaws.com (eventName:CreateUser OR eventName:DeleteUser) 7. Check if any already vulnerable IAM Roles/Accounts modified to be more permissive [3]: eventSource:iam.amazonaws.com (eventName:CreateRole OR eventName:DeleteRole OR eventName:AttachRolePolicy OR eventName:DetachRolePolicy) 8. Check for any access keys made [4][5]: eventSource:iam.amazonaws.com (eventName:CreateAccessKey OR eventName:DeleteAccessKey) 9. Check if any production / persistent EC2s have had their IAMInstanceProfile changed, to allow for a backdoor using EC2 permissions from a webshell/backdoor they could have placed on your public facing infra. [6] etc. etc. But if you have had a compromise based on initial investigations, probably worth while getting professional support to do a thorough audit of your environment. [0] https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-log-file-examples.html#cloudtrail-log-file-examples-ec2 https://docs.aws.amazon.com/awscloudtrail/latest/userguide/c... [1] https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-event-reference-aws-console-sign-in-events.html https://docs.aws.amazon.com/awscloudtrail/latest/userguide/c... [2] https://docs.aws.amazon.com/IAM/latest/UserGuide/cloudtrail-integration.html https://docs.aws.amazon.com/IAM/latest/UserGuide/cloudtrail-... [3] https://docs.aws.amazon.com/awscloudtrail/latest/userguide/security_iam_id-based-policy-examples.html https://docs.aws.amazon.com/awscloudtrail/latest/userguide/s... [4] https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credenti... [5] https://research.splunk.com/sources/0460f7da-3254-4d90-b8c0-2ca657d0cea0/ https://research.splunk.com/sources/0460f7da-3254-4d90-b8c0-... [6] https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_ReplaceIamInstanceProfileAssociation.html https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_R...
jinen83 11mo ago
this is helpful. i will look for the logs. Also some more observations below: 1) some 20 organisations were created within our Root all with email id with same domain (co.jp) 2) attacker had created multiple fargate templates 3) they created resources in 16-17 AWS regions 4) they requested to raise SES,WS Fargate Resource Rate Quota Change was requested, sage maker Notebook maintenance - we have no need of using these instances (recd an email from aws for all of this) 5) in some of the emails i started seeing a new name added (random name @outlook.com)
- ThreatSystems 11mo ago
  It does sound like you've been compromised by an outfit that has got automation to run these types of activities across compromised accounts. A Reddit post[0] from 3 years ago seems to indicate similar activities. Do what you can to triage and see what's happened. But I would strongly recommend getting a professional outfit in ASAP to remediate (if you have insurance notify them of the incident as well - as often they'll be able to offer services to support in remediating), as well as, notify AWS that an incident has occurred. [0] https://www.reddit.com/r/aws/comments/119admy/300k_bill_after_aws_account_hacked/ https://www.reddit.com/r/aws/comments/119admy/300k_bill_afte...

sylens 11mo ago

RunInstances

ThreatSystems 11mo ago

I'm officially off of AWS so don't have any consoles to check against, but back on a laptop. Based on docs and some of the concerns about this happening to someone else, I would probably start with the following: 1. Check who/what created those EC2s[0] using the console to query: eventSource:ec2.amazonaws.com eventName:RunInstances 2. Based on the userIdentity field, query the following actions. 3. Check if someone manually logged into Console (identity dependent) [1]: eventSource:signin.amazonaws.com userIdentity.type:[Root/IAMUser/AssumedRole/FederatedUser/AWSLambda] eventName:ConsoleLogin 4. Check if someone authenticated against Security Token Service (STS) [2]: eventSource:sts.amazonaws.com eventName:GetSessionToken 5. Check if someone used a valid STS Session to AssumeRole: eventSource:sts.amazonaws.com eventName:AssumeRole userIdentity.arn (or other identifier) 6. Check for any new IAM Roles/Accounts made for persistence: eventSource:iam.amazonaws.com (eventName:CreateUser OR eventName:DeleteUser) 7. Check if any already vulnerable IAM Roles/Accounts modified to be more permissive [3]: eventSource:iam.amazonaws.com (eventName:CreateRole OR eventName:DeleteRole OR eventName:AttachRolePolicy OR eventName:DetachRolePolicy) 8. Check for any access keys made [4][5]: eventSource:iam.amazonaws.com (eventName:CreateAccessKey OR eventName:DeleteAccessKey) 9. Check if any production / persistent EC2s have had their IAMInstanceProfile changed, to allow for a backdoor using EC2 permissions from a webshell/backdoor they could have placed on your public facing infra. [6] etc. etc. But if you have had a compromise based on initial investigations, probably worth while getting professional support to do a thorough audit of your environment. [0] https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-log-file-examples.html#cloudtrail-log-file-examples-ec2 https://docs.aws.amazon.com/awscloudtrail/latest/userguide/c... [1] https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-event-reference-aws-console-sign-in-events.html https://docs.aws.amazon.com/awscloudtrail/latest/userguide/c... [2] https://docs.aws.amazon.com/IAM/latest/UserGuide/cloudtrail-integration.html https://docs.aws.amazon.com/IAM/latest/UserGuide/cloudtrail-... [3] https://docs.aws.amazon.com/awscloudtrail/latest/userguide/security_iam_id-based-policy-examples.html https://docs.aws.amazon.com/awscloudtrail/latest/userguide/s... [4] https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credenti... [5] https://research.splunk.com/sources/0460f7da-3254-4d90-b8c0-2ca657d0cea0/ https://research.splunk.com/sources/0460f7da-3254-4d90-b8c0-... [6] https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_ReplaceIamInstanceProfileAssociation.html https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_R...

jinen83 11mo ago

this is helpful. i will look for the logs. Also some more observations below: 1) some 20 organisations were created within our Root all with email id with same domain (co.jp) 2) attacker had created multiple fargate templates 3) they created resources in 16-17 AWS regions 4) they requested to raise SES,WS Fargate Resource Rate Quota Change was requested, sage maker Notebook maintenance - we have no need of using these instances (recd an email from aws for all of this) 5) in some of the emails i started seeing a new name added (random name @outlook.com)

ThreatSystems 11mo ago
It does sound like you've been compromised by an outfit that has got automation to run these types of activities across compromised accounts. A Reddit post[0] from 3 years ago seems to indicate similar activities. Do what you can to triage and see what's happened. But I would strongly recommend getting a professional outfit in ASAP to remediate (if you have insurance notify them of the incident as well - as often they'll be able to offer services to support in remediating), as well as, notify AWS that an incident has occurred. [0] https://www.reddit.com/r/aws/comments/119admy/300k_bill_after_aws_account_hacked/ https://www.reddit.com/r/aws/comments/119admy/300k_bill_afte...