10 ms·
Kurt Got Got
- deleted 1y ago[deleted]
- vednig 11mo agoIrony would be if we found out the hackers ran their website on fly.io that would be a swell
- 0xdeadbeefbabe 11mo agoHuh, so I'm stupid I guess, but how is MFA phish proof? Why did Kurt's commit access get revoked?
- tptacek 11mo agoThe commit access thing is a joke. I think it's a joke. It's mostly a joke. MFA is not in general phish-resistant. But Passkeys, U2F, and FIDO2 generally are, because they mutually authenticate; they're not just "one time passwords" you type into a field, but rather a cryptographic protocol running between you and the site.
- 0xdeadbeefbabe 11mo agoWell he must be punished somehow!
- bigyabai 1y ago[dead]
- jryio 1y agoI'm always glad to see when companies, developers and CEOs make a heartfelt and humanistic mae culpa. We would like to think that we're the smart ones and above such low level types of exploits, but the reality is that they can catch us at any moment on a good or bad day. Good write up
- 000ooo000 1y ago>heartfelt and humanistic mae culpa They literally admit they pay a Zoomer to make memes for Twitter. I think you are falling for the PR.
- deleted 1y ago[deleted]
- herval 1y agoGreat writeup, but also gotta say that’s some excellent phishing
- tptacek 1y agoThis exact phish has been going around lately and we're not the only ones who got bit. But we didn't know that before it happened.
- ChrisMarshallNY 1y agoI enjoyed the self-deprecating humor behind it. I have been almost got, a couple of times. I'm not sure, but I may have realized that I got got, about 0.5 seconds after clicking[0], and was able to lock down, before they were able to grab it. [0] https://imgur.com/EfQrdWY https://imgur.com/EfQrdWY
- tptacek 1y agoI want to say again that the key thing in this post is that anything "serious" at Fly.io couldn't have gotten phished: your SSO login won't work if you don't have mandatory phish-resistant 2FA set up for it. What went wrong here is that Twitter wasn't behind that perimeter, because, well, we have trouble taking Twitter seriously. We shouldn't have, and we do take it seriously now.
- latchkey 1y ago[deleted]
- tptacek 1y agoTwitter isn't an operational dependency of ours and we don't attest to it at all. It also doesn't require we do that: what SOC2 actually demands of vendor security practices is much more complicated (and performative) than that. If Twitter were a real vendor dependency of ours, most of what we'd need would be a SOC2 attestation from them.
- davidw 1y agoGiven the CEO's responsibility for starving children to death through his political activities, there's an argument for not having any dependency at all on Twitter. https://apnews.com/article/myanmar-usaid-thailand-trump-rubio-aid-7f6919a1863ceea2ddf6708e47bb88f0 https://apnews.com/article/myanmar-usaid-thailand-trump-rubi...
- breakingcups 1y agoI will say that a "Critical Security Vulnerability in flyctl, update now: https://bad-link/to/update.zip" tweet will have very serious consequences for a portion of your userbase, despite not directly compromising your own infra.
- tptacek 11mo agoYou could do that yourself today by getting a blue-checked @realFlyDotIo. But there's a paragraph in the article about this, and we know what we would have done had there been any signs of direct attacks on our users.
- stavros 1y agoEver since I almost got phished (wasn't looking closely enough at the domain to notice a little stress mark over the "s" in the domain name, thankfully I was using a hardware wallet that prevented the attack entirely), I realized that anyone can get phished. They just rely on you being busy, or out, or tired, and just not checking closely enough. Use passkeys for everything, like Thomas says.
- Y_Y 1y agoCounterpoint: don't use passkeys, they're a confused mess and add limitations while not giving any benefits over a good long password in a password manager.
- bigyabai 1y agoYep. A technical half-baked solution to a problem that has been solved since it's inception. Really just feels like FAANG exists to invent new ways to charge rent...
- akerl_ 1y agoWhat’s the solution for preventing this kind of phishing attack?
- NoGravitas 11mo agoTLS client certificates. Unfortunately, the browser UI for them ranges from godawful to removed-because-nobody-used-them.
- akerl_ 11mo agoSo the solution for this isn’t actually usable?
- dewey 1y agoThey prevent you from being one of these, and copy pasting the password from password manager into the wrong input field. Something that still happens often with many websites not properly auto-filling from password managers. > They just rely on you being busy, or out, or tired, and just not checking closely enough
- x0x0 1y ago... could we get webauthn / yubikeys prioritized for fly? afaik (don't want to disable 2fa to find out), it only supports totp. For everyone reading though, you should try fly. Unaffiliated except for being a happy customer. 50 lines of toml is so so much better than 1k+ lines of cloudformation.
- tptacek 1y agoWe don't like TOTP, at all, for reasons even more obvious now, but our standard answer for advanced MFA has been OIDC, which is what most people should do rather than setting up bespoke U2F/FIDO2/Passkeys. We will get to this though. https://fly.io/blog/tokenized-tokens/ https://fly.io/blog/tokenized-tokens/
- parliament32 11mo agoThat would be great, but > Fly.io supports Google and GitHub as Identity Providers[1] How about you just support SAML like a real enterprise vendor, so IdP-specific support isn't your problem anymore? I get it, SAML is hard, but it's really the One True Path when it comes to this stuff. [1] https://fly.io/docs/security/sso/ https://fly.io/docs/security/sso/
- tptacek 11mo agoSAML is awful, maybe the worst cryptographic protocol ever devised, and we won't implement it unless we absolutely have to. OIDC is the future. I'm not exaggerating; you can use the search bar and find longer comments from me on SAML and XMLDSIG. You might just as well ask when we're going to implement DNSSEC.
- parliament32 11mo agoI certainly see you whining a lot about SAML in your history. This lines up with my "SAML is hard" comment above -- SAML is filled with footguns and various perils, but that doesn't necessarily make it bad. OIDC is certainly better in a few aspects (note trading XML parsing for JSON parsing is not one of them), but the killer SAML feature that you (and by you, I mean fly.io, to be clear) is missing is being IdP-agnostic. You cannot reasonably expect that those two vendors will cover even half of your potential enterprise user base; and yes, for anyone working in an even remotely regulated industry, not being compatible with our SSO ensures you get dropped even before the evaluation phase. My favourite slop-generator summarizes this as "While SAML is significantly more complex to implement than OIDC, its design for robust enterprise federation and its maturity have resulted in vendors converging on a more uniform interpretation of its detailed specification, reducing the relative frequency of non-standard implementation quirks when dealing with core B2B SSO scenarios." That being said, if your org is more B2C, maybe it makes sense you haven't prioritized this yet. You'll get there one day :)
- tgsovlerkhgsel 1y agoThis is why properly working password managers are important, and why as a web site operator you should make sure to not break them. My password not auto-filling on a web site is a sufficient red flag to immediately become very watchful. Code-based 2FA, on the other hand, is completely useless against phishing. If I'm logging in, I'm logging in, and you're getting my 2FA code (regardless of whether it's coming from an SMS or an app).
- akerl_ 1y agoHow does this square with the fact that the tech savvy person in the post was phished despite using a password manager.
- otterley 1y agoPrecisely. 1Password's browser integration would have noticed a domain mismatch and refused to autofill the password -- but in a panic, Kurt apparently opened 1Password and then copied/pasted the credentials manually.
- akerl_ 1y agoWhich is why a properly working password manager is not a strong defense against phishing.
- otterley 1y agoCorrect. The moral of the story is that hardware MFA and/or passkeys are a necessity in today's world. An infinitely complex password and 2FA are no match for attacks that leverage human psychology.
- onionisafruit 1y agoIt's a strong defense that this guy decided not to use
- 1y ago
- bradgessler 1y agoWhen we did annual pen testing audits for my last company, the security audit company always offered to do phishing or social engineering attacks, but advised against it because they said it worked every single time. One of the most memorable things they shared is they'd throw USB sticks in the parking lot of the company they were pentesting and somebody would always put the thing into a workstation to see what as on it and get p0wned. Phishing isn't really that different. Great reminder to setup Passkeys: https://help.x.com/en/managing-your-account/how-to-use-passkey https://help.x.com/en/managing-your-account/how-to-use-passk...
- KingOfCoders 1y agoOnce a head of security worked for me (CTO), and she was great great great. She did the same, putting USB sticks on the printers for example and see who would plug one into their computer.
- dilyevsky 1y agoThe stray USB stick is how Stuxnet allegedly got deployed. Tbh I doubt that works in this day and age.
- stavros 1y agoHah, watch me.
- roblabla 1y agoA USB can pretend to be just about any type of device to get the appropriate driver installed and loaded. They can then send malformed packets to that driver to trigger some vulnerability and take over the system. There are a _lot_ of drivers for devices on a default windows install. There are a _lot more_ if you allow for Windows Update to install drivers for devices (which it does by default). I would not trust all of them to be secure against a malicious device. I know this is not how stuxxnet worked (instead using a vulnerability in how LNK files were shown in explorer.exe as the exploit), but that just goes to show how much surface there is to attack using this kind of USB stick. And yeah, people still routinely plug random USBs in their computers. The average person is simultaneously curious and oblivious to this kind of threat (and I don't blame them - this kind of threat is hard to explain to a lay person).
- silexia 1y agoCEO here, I also almost got taken by a fake legal notice about a Facebook post. My password manager would not auto enter my password so I tried manually entering it like a dummy. Fortunately, it was the wrong one.
- latchkey 1y agoThis is exactly why I turned off auto enter.
- akerl_ 1y agoIsn’t turning off auto enter exacerbating the problem? The avenue for catching this is that the password manager’s autofill won’t work on the phishing site, and the user could notice that and catch that it’s a malicious domain
- tptacek 1y agoYes. This is the problem with the "just use a password manager" answer to phishing-resistance. They can be a line of defense, situationally, but you have to have them configured just right, and if you're using phishing-resistant authentication you don't need that line of defense in the first place.
- rtpg 1y agoIsn't this backwards? If the autocomplete doesn't show up that's a flag that the password is going somewhere it doesn't belong. If you're always copy-pasting from a password manager then you're not getting that check "for free". Obviously SSO-y stuff is _better_, but autofill seems important for helping to prevent this kind of scam. Doesn't prevent everything of course!
- tptacek 1y agoNone of this password manager configuration stuff matters; we've just got Passkeys set up for the account now, which is what we should have done, but didn't, because we spent the last 2 years with one foot out the door on Twitter altogether. Since this attack happened despite Kurt using 1Password, I'm really not all that receptive to the idea that 1Password is a good answer to this problem.
- dyauspitr 1y agoWhen did fly.io create their own crypto?
- ezfe 1y agoThey didn't...?
- dyauspitr 1y agoThe first line of this article is talking about the $FLY token
- vintagedave 1y agoI think it's a security token: https://fly.io/blog/macaroons-escalated-quickly/ https://fly.io/blog/macaroons-escalated-quickly/ And calling it $FLY like a crypto thing is part of the joke.
- __jonas 1y agoThat's some impressive work on the attackers part having that whole fake landing page ready to go, and a pretty convincing phishing email. I'm don't know much about crypto so I'm not sure what makes them call the scam 'not very plausible' and say it 'probably generated $0 for the attackers', is that something that can be verified by checking the wallet used in that fake landing page?
- paxys 1y ago> This is, in fact, how all of our infrastructure is secured at Fly.io; specifically, we get everything behind an IdP (in our case: Google’s) and have it require phishing-proof MFA. Every system is only as secure as its weakest link. If the company's CEO is idiotic enough to pull credentials from 1Password and manually copy-past them on a random website whose domain does not match the service that issued it, what is to say they won't do the same for an MFA token?
- tptacek 1y agoThe. whole. point. of. phishing-resistant. MFA. is. that. you. can't. do. the. same. thing.
- deleted 1y ago[deleted]
- akerl_ 1y agoFIDO2 won’t send an authentication to a fake site, no matter what the human does. That’s what makes it phishing-resistant.
- roblabla 1y agoThey literally explain in the article they're using FIDO MFA that is phishing proof as the key authenticates the website (it's not your run-of-the-mill sms 2FA, it's using WebAuthn to talk to your MFA). With this setup, you can't fuck up.
- parliament32 11mo agoPasskeys are called "phishing-resistant" because (when properly implemented) it's impossible for users to fuck up. They literally cannot be phished into giving an adversary their credentials, no matter what they click or what they do.
- nofriend 1y ago> But if we’d actually done an ICO, you’d have lost all your money anyways. tru tru
- theturtle 1y ago[flagged]
- tomhow 1y agoWe've banned this account.
- roughly 1y agoI was reading this and wondering why it was posted so high (I didn’t recognize the company name), and then I got to the name at the bottom. I think the lesson here is “if it could happen to Kurt, it could happen to anyone.” Yeah, the consequences here were pretty limited, but everyone’s got Some vulnerability, and it’s usually in the junk pile in the corner that you’re ignoring. If the attacker were genuinely trying to do damage (as opposed to just running a two-bit crypto scam), assuming the company’s official account is a fine start to leverage for some social engineering.
- akerl_ 1y agoI think you mean Kurt.
- stavros 1y agoIt would help if they mentioned his name anywhere in the post, title, or subtitle.
- roughly 1y agoYeah, that was definitely a pebkac on my part.
- roughly 1y agoYou’re right - I flagged on Thomas’s name in the signature and because I’ve seen him around here, well, forever, but Kurt is also extremely savvy.
- pants2 1y agoThis "content violation on your X post" phishing email is so common, we get about a dozen of those a week, and had to change the filters many times to catch them (because it's not easy to just detect the letter X and they keep changing the wording). We also ended up dropping our email security provider because they consistently missed these. We evaluated/trialed almost a dozen different providers and finally found one that did detect every X phishing email! (Check Point fyi, not affiliated) It was actually embarrassing for most of those security companies because the signs of phishing are very obvious if you look.
- pixl97 11mo agoIt easy to block all phishing email. Just block all email. It's much much harder to block emails that aren't actually phishing but have components that would flag them anyway.
- rtpg 1y agoFly has consistently surprised me at how late they have been to doing the "standard company" stuff. Their sort of lack of support engineering teams for a while affected me way more though. You gotta take the Legos away from the CEO! Being CEO means you stop doing the other stuff! Sorry! And yes they have their silly disclaimer on their blog, but this is Yet Another "oh lol we made a whoopsie" tone that they've taken in the past several times for "real" issues. My favorite being "we did a thing, you should have read the forums where we posted about it, but clearly some of you didn't". You have my e-mail address! Please.... please... get real comms. I'm tired of the "oh lol we're just doing shit" vibes from the only place I can _barely_ recommend as an alternative to Heroku. I don't need the cuteness. And 60% of that is because one of your main competitors has a totally unsearchable name. Still using fly, just annoyed.
- tptacek 1y agoWe've had an unusually large security team for the size of our company since 2021. I'm sorry if you don't like the way I communicate about it but I have no plans to change that. We take security extremely seriously. We just didn't take Twitter that seriously. The "CEO" thing is just a running joke. Kurt's an engineer. Any of us could have been taken by this. I joke about this because I assume everybody gets the subtext, which is that anything you don't have behind phishing-resistant authentication is going to get phished. You apparently took it on the surface level, and believe I'm actually dunking on Kurt. No.
- rtpg 1y agoI'm not talking security, which I generally feel like is probably being done correctly. I was thinking about, IIRC, back in 2023[0], where you all were suffering a lot of issues. And I _believe_ I saw some chatter about Fly building out a team of support/devops-y/SRE engineers around that time. And I had just assumed up until there that, as a company about operations, that you would already have a team that is about reliability. I am not a major user of you (You're only selling me like 40 bucks a month of compute/storage/etc), but I had relatively often been hitting weird stuff. Some of it was me, some of it was your side. But... well... I was using Heroku for this stuff before and it seemed to run swimmingly for very long. So I was definitely a bit like "oh OK so you just didn't care about reliability until then?" I mean this lightly, but I started basically anti-recommending you after the combo of the issues and the statements your team was making (both on this kind of operations and also communications after the fact). I think you all generally do this better now though, so maybe I'm just bringing up old grudges. > You apparently took it on the surface level, and believe I'm actually dunking on Kurt. No, I took it in the same tone I take a lot of your company's writing. > The "CEO" thing is just a running joke. Kurt's an engineer. I think if you are the CEO of a company above a certain (very low!) headcount you put down the Legos. There are enough "running a company" things to do. Maybe your dynamics are different, since your team is indeed quite small according to the teams page. Every startup engineer has had to deal with "The CEO is the one with admin rights on this account and he's not doing the thing because somehow we haven't pried the credentials from him so that people doing the work does it". And then the dual of this, "The CEO fixes the thing at 2AM but does it the wrong way and now thing is weird". A way you avoid this is by yanking all credentials from the CEO. I'm being glib here, because obviously y'all have your success, the Twitter thing "doesn't matter", etc. I just want to be able to recommend you fully, and the issues I hit + the amateur hour comms in response (EDIT: in the past) gets on my nerves and prevents me from doing it! Anyways, I want you all to succeed. [0]: https://community.fly.io/t/reliability-its-not-great/11253 https://community.fly.io/t/reliability-its-not-great/11253
- deepfriedrice 1y agoI don't know the gullibility of the average tech CEO but this doesn't strike me as a very convincing phishing attempt. * "We've received reports about the latest content" - weird copy * "which doesn't meet X Terms of Service" - bad grammar lol * "Important:Simply ..." - no spacing lol * "Simply removing the content from your page doesn't help your case" - weird tone * "We've opened a support portal for you " - weird copy There should so many red flags here if you're a native english speaker. There are some UX red flags as well, but I admit those are much less noticeable. * Weird and inconsistent font size/weight * Massive border radius on the twitter card image (lol) * Gap sizes are weird/small * Weird CTA
- akerl_ 1y agoI think you'll be led astray thinking this is CEO-specific. The whole theory of phishing, and especially targeted phishing, is to present a scenario that tricks the user into ignoring the red flags. Usually, this is an urgent call to action that something negative will happen, coupled with a tie-in to something that seems legit. In this case, it was referencing a real post that the company had made. A parallel example is when parents get phone calls saying "hey it's your kid, I took a surprise trip to a tiny island nation and I've been kidnapped, I need you to wire $1000 immediately or they're going to kill me". That interaction is full of red flags, but the psychological hit is massive and people pay out all the time.
- deepfriedrice 1y agoI razz CEOs in jest, but my point is: This is an example of a good phishing attempt? ChatGPT could surely find and fix most of the red flags I called out. Perhaps the red flags ensure they don't phish more people than they can productively exploit.
- akerl_ 1y agoThere are certainly phishing attempts that are pixel perfect, but I'd say way more energy tends to go into making phishing websites perfect. The goal of the email is to flip people into action as quickly as possible with as little validation.
- grinich 1y agoI got hit with the same kind of phishing attack a couple months ago It's pretty incredible the level of UI engineering that went into it. Some screenshots I took: https://x.com/grinich/status/1963744947053703309 https://x.com/grinich/status/1963744947053703309
- giarc 1y agoI'm sorry but the imagecontent-x.com url should throw red flags for anyone.
- tptacek 1y agoThis is exactly how not to defend against phishing. The meaningful defense is to foreclose on it entirely, not to just get super good at spotting fakes.
- classified 1y ago> The meaningful defense is to foreclose on it entirely Sounds easy enough in theory. How do you do that in practice?
- 9dev 1y agoUse passkeys. Bully services that don’t offer them or lock them behind enterprise plans into implementing them. That’s it. The single working Defense against credential theft.
- everybodyknows 1y agoSo, in that case the browser (correctly) did not autofill? Is that a common occurrence for legit traffic from X? And no complaint about the website's identity from the browser -- the expected "lock" icon left of the URL?
- 9dev 1y agoAs long as people are used to companies just buying new domains for the hell of it, yes. Just look at the amount of domains Microsoft uses for signing in! My password manager currently holds 8 of them. Eight! Who can be blamed for thinking it’s the password managers fault?
- chews 1y agoif anyone @ x.com infosec is here, my buddy got her account phished / there is someone in CS selling creds. Then it was used to pump a crypto scam and she has been trying for months to get it sorted. She's had the account for 16 plus years, it's surprising it's this hard to fix. It's x.com/leighleighsf, we've tried every channel but for filing a small claims lawsuit in Texas to get her account back.
- foxglacier 1y agoLike with occupational safety, we should worry about near misses as well as actual hacks. If you realize you just logged into X from a link in an email, you should berate yourself for could-have-been-hacked. Never enter credentials into links from emails!
- reassess_blind 1y agoIs there an anti-phishing extension that detects whether the domain is close to, but not exactly the popular legitimate domain? Would probably need to use a local LLM for the detection. If not I might look into making one.
- haruka_ff 1y agoMetaMask (the crypto wallet) has one that shows warning pages to all domains that are remotely similar to crypto-related domains, and it is very prone to false positives and annoying. They have to maintain a list to skip the detection for real domains, and it's really inefficient. Feels like this kind of detection is hard to balance, and calling legit websites possible phishing might be problematic...
- reassess_blind 1y agoSeems like the kind of problem LLMs would be perfect for. ChatGPT does a great job at giving a score of whether domains are attempting to appear legitimate, but of course no one wants their browsing history being sent off to OpenAI. Unfortunately from my testing local 4B-7B LLMs aren't up to the task.
- typpilol 1y agoEdge has some basic typo squat protection
- deleted 11mo ago[deleted]
- kwar13 1y agoThat was beautiful to read. lol.
- IG_Semmelweiss 1y agoDank post mortem ?
- lawik 1y agoFunny! Now that Kurt doesn't have commit access, who do I ask to get internal Fly Slack bot fizz off of my behind. I was in a devrel channel for a short while and ever since it has asked me to write updates in a channel I don't have access to. Frequently.
- KingOfCoders 1y ago[Deleted]
- deleted 1y ago[deleted]
- deleted 1y ago[deleted]
- KingOfCoders 1y agoPhishing training does not work. "Understanding the Efficacy of Phishing Training in Practice" https://arianamirian.com/docs/ieee-25.pdf https://arianamirian.com/docs/ieee-25.pdf
- man8alexd 1y agoThe same paper is linked in the original article.
- KingOfCoders 1y agoThanks. Stopped reading after I found out the article was not by the CEO.
- loloquwowndueo 1y agoWhy?
- classified 1y ago> We are reliably informed by our zoomer children that we are too cringe to be trusted on these matters. Bullseye. At least they take it with good humor.
- karel-3d 1y ago"Don't put your password into the website that you shouldn't and put it only to website that you should" is a circular advice. It's like those 2FA SMS that say "don't tell this token to anyone!" while you literally share it with the website that you login to. I am always so frustrated when I receive those
- ctennis1 11mo agoMaybe not - but I work in a regulated industry, we had an employee get phished a few years ago, and the regulatory bodies wanted detailed records of all phishing testing and training conducted for the previous 5 years. So for some of us it's a necessary evil.
- classified 1y agoX Terms of Service error: Meme not dank enough.
- baphomet88f 1y ago[flagged]
- deleted 1y ago[deleted]
- siskiyou 1y agoThe part I found surprising: 'Twitter fell outside the “things we take seriously” boundary' Sure Twitter is rubbish, but it's still a huge platform, still tied to your brand, you're still using it, so it can still hurt you. Either take it seriously or stop using it.
- loloquwowndueo 1y agoYou mean X, right? Sounds like neither them nor you take it seriously :)
- siskiyou 11mo ago(Turning up my hearing aids) that's right mate
- tptacek 11mo agoBefore the Twitter Change of Control, we were actively using it. After, it fell into a kind of limbo. There was a solid 6 months or so when we thought maybe we were just going to do everything via our Hachyderm account. Shit's complicated. And if we'd stopped using it altogether, we'd still be in the same boat!
- siskiyou 11mo agoThanks for your answer. I get it, you're people, it's a mistake, it's not the most horrible thing ever, but a simple "oops my bad" would have been a shorter blog post. Would you really have been in the same boat if the tweet shown mentioned in the phishing effort had never been posted, or did the phisher fabricate that entirely?
- black_puppydog 1y agoKudos to Thomas and whoever else contributed here, the writing is great! <3
- lijok 11mo agoThis makes fly.io seem like an unserious business. I was under the impression they were trying to build something of substance.
- zahlman 11mo ago> Had this been an impactful attack, we would not be this flippant about it. For this, though, any other tone on our part would be false. > ... > If you were inclined to take us up on an “airdrop” to “claim a share” of the “token” powering Fly.io, the site is still up. You can connect your wallet it [sic] it! You’ll lose all your money. But if we’d actually done an ICO, you’d have lost all your money anyways. > Somebody involved in pulling this attack off had to come up with “own a piece of the sky!”, and I think that’s punishment enough for them. I was amused by all of this, but I still feel like they should care more about how impactful this was for anyone who got crypto-scammed at the link. I mean, yes, those are people who would believe the story and also click a link like that. But what if fly.io were found to share liability?