6 ms·
I would normally say that "That must be a coincidence", but I had a client account compromise as well. And it was very strange: Client was a small org, and two
by timdev2 11mo ago
I would normally say that "That must be a coincidence", but I had a client account compromise as well. And it was very strange:
Client was a small org, and two very old IAM accounts had suddenly had recent (yesterday) console log ins and password changes.
I'm investigating the extent of the compromise, but so far it seems all they did was open a ticket to turn on SES production access and increase the daily email limit to 50k.
These were basically dormant IAM users from more than 5 years ago, and it's certainly odd timing that they'd suddenly pop on this particular day.
- tcdent 11mo agoSmells like a phishing attack to me. Receive an email that says AWS is experiencing an outage. Log into your console to view the status, authenticate through a malicious wrapper, and compromise your account security.
- SoftTalker 11mo agoGood point. Phishers would certainly take advantage of a widely reported outage to send emails related to "recovering your services." Even cautious people are more vulnerable to phishing when the message aligns with their expectations and they are under pressure because services are down. Always, always log in through bookmarked links or typing them manually. Never use a link in an email unless it's in direct response to something you initiated and even then examine it carefully.
- roblabla 11mo agoYou can also use phishing-resistant login/2FA like passkeys/FIDO keys, where it is available (and I'm pretty sure amazon supports it), to minimize the risk of accidentally login into a phishing website while under pressure.
- SoftTalker 11mo agoThey probably support it but how many accounts have not configured it? I'd bet it's a lot.
- akerl_ 11mo agoIf my memory is correct, AWS supports FIDO for web login but not for the API, so you either have to restrict access to FIDO and then use the web UI for everything done as that user, or have a separate non-FIDO MFA device (without FIDO's phishing resistance) for terminal/API interactions.
- jorvi 11mo agoYou can generate temporary AWS keys for privileged users: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_request.html https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credenti... Of course, as always, PEBKAC. You will have to strictly follow protocol, and not every team is willing to jump through annoying hoops every day.
- akerl_ 11mo agoCan you actually generate temporary AWS STS credentials via FIDO MFA? Again, last I looked, FIDO MFA credentials cannot be used for API calls, which you'd need to make for STS credential generation.
- jorvi 11mo agoYou don't put the temporary credentials behind FIDO because they're temporary anyway. You put FIDO on the main account that has the privilege to generate the temporary credentials. So in the off chance that you get a phishing mail, you generate temporary credentials to take whatever actions it wants, attempt to log in with those credentials, get phished, but they only have access to API for 900s (or whatever you put as the timeout, 900s is just the minimum). 900s won't stop them from running amok, but it caps the amok at 900s.
- akerl_ 11mo agoYou aren't grokking what I'm saying. AWS does not allow FIDO2 as an MFA method for API calls. So if your MFA device for your main account is a FIDO2 device, you either: 1. Don't require MFA to generate temporary credentials. Congrats, your MFA is now basically theater. 2. Do require MFA to generate temporary credentials. Congrats, the only way to generate temporary credentials is to instead use a non-FIDO MFA device on the main account. Nobody is getting a phishing email, going to the terminal, generating STS credentials, and then feeding those into the phish. The phish is punting them to a fake AWS webpage. Temporary credentials are a mitigation for session token theft, not for phishing.
- plaidfuji 11mo agoWhat if the outage and phishing attack were coordinated at a higher level? There’s a scary thought.
- BikiniPrince 11mo agoBezos will get to Mars at any cost!
- fragmede 11mo agoBezos is shooting for the Moon, Elons the one going for Mars. In fact, that's why Elon's going for Mars, to show up Bezos' plan.
- Scoundreller 11mo agoA phisher that did their homework would send out a tone deaf email with a subject line like this that aws sent me during their outage: > You could win $5,000 in AWS credits at Innovate
- Sebb767 11mo ago> Always, always log in through bookmarked links or typing them manually. Never use a link in an email unless it's in direct response to something you initiated and even then examine it carefully. If you still want to avoid the comfort of typing in stuff manually or navigating the webinterface, logging in on a new tab and then clicking on the link is also an option.
- morkalork 11mo agoMini-rant here but I hate how websites for SaaS products are so over-optimized for the sales funnel. It's like giant blue button to sign up, teeny tiny link to login, if there is even one at all on any of the main pages. Often your access is on an entirely different subdomain that barely ranks on Google. If it's something that "just works" and you only access every 6 months, it's pain to go hunting through your email to rediscover if it's clients.example.com, portal.example.com, or whatever the heck it is.
- rtkwe 11mo agoI hate how many sites do that, always a signup first then a small little "Already have an account" link below that. Feels almost hostile to your existing users.
- timdev2 11mo agoThese were accounts that shouldn't have had console access in the first place, and were never used by humans to log in AFAICT. I don't know exactly what they were originally for, but they were named like "foo-robots", were very old. At first I thought maybe some previous dev had set passwords for troubleshooting, saved those passwords in a password manager, and then got owned all these years later. But that's really, really, unlikely. And the timing is so curious.
- portaouflop 11mo agoWhy keep accounts like this around anyway? Sounds like a breach was just waiting to happen…
- Avicebron 11mo agoA cost center like security? Are you crazy..
- highfrequencyy 11mo agoI second this, pretty much immediately after my organization got hit with a wave of phishing emails.
- jbverschoor 11mo agoOr maybe it wasn't DNS, but they simply pulled the plug bc of some breach?
- LeonardoTolstoy 11mo agoAlmost this exact thing happened to me about a year ago. Very old account login, SES access with request to raise the email limit. We were only quickly tipped off because they had to open a ticket to get the limit raised. If you haven't check newly made Roles as well. We quashed the compromised users pretty quickly (including my own, the origin we figured out), but got a little lucky because I just started cruising the Roles and killing anything less than a month old or with admin access. To play devil's advocate a bit. In our case we are pretty sure my key actually did get compromised although we aren't precisely sure how (probably a combination of me being dumb and my org being dumb and some guy putting two and two together). But we did trace the initial users being created to nearly a month prior to the actual SES request. It is entirely possible whomever did your thing had you compromised for a bit, and then once AWS went down they decided that was the perfect time to attack, when you might not notice just-another-AWS-thing happening.
- timdev2 11mo agoThanks for sharing. After digging in, it appears that something very similar happened here, after all. It looks like an access key with admin role leaked some time ago. At first, they just ran a quiet GetCallerIdentity, then sat on it. Then, on outage day, they leveraged it. In our case, they just did the SES thing, and tried to persist access by setting up IAM Identity Center.
- orblivion 11mo agoI wonder if a few cases of compromise right after the outage can also be a coincidence. If we have a lot of reports of the same, then it gets interesting. (The particulars of your case being strange is a separate question though.)