7 ms·
Fastmail, Runbox, and Posteo under DDoS extortion attack
- dubcanada 5y agoOn another note, the amount they are asking for seems really reasonable, like how are they making money? A DDoS attack must cost more then like 0.06BTC (like $3500 USD) to run all weekend?
- rgrmrts 5y agoI’d assume they’re not actually paying for these attacks and have access to a large botnet, which would imply no cost (outside of hours spent constructing the attack) to the adversary, right?
- jacobobryant 5y agoThey might be renting the botnet.
- stilley2 5y agoIf they're using a botnet of compromised servers then I suspect the marginal cost is negligible.
- bombcar 5y agoA DDoS can be rented for less than $100/mo depending on how much bandwidth you want to flood. Remember, they're compromised machines; you're not paying for egress bandwidth.
- jorgesborges 5y agoAre most machines involved in attacks like this compromised? Are we talking generic consumer desktop machines, like my grandma’s old XP desktop running in her basement?
- IceWreck 5y ago> Are most machines involved in attacks like this compromised? Yes > Are we talking generic consumer desktop machines, like my grandma’s old XP desktop running in her basement? Those combined with hundreds of thousands of compromised VPSes, etc.
- bombcar 5y agoAnd don’t forget the millions of IoT devices.
- harikb 5y agoIt could also be seed investment to see who is likely to pay up. ROI comes from second attack.
- rgrmrts 5y agoIs the attack on its webmail/website or on their smtp servers? I’ve been wondering about this, but how does one protect smtp servers from distributed attacks? Let’s assume smaller attackers, do you just need good firewalls in front of your servers to prevent congestion to the smtp servers? Are there off the shelf tools that can be configured to help here (pf maybe)? Do tools like fail2ban help? For context, I’ve picked up self-hosting again after many years and though I don’t anticipate being a target for large attacks I have been curious what tools individuals have at their disposal or if it’s a fools errand to even try.
- lxchase 5y agoI’ve been a server operator and was subject to DDoS attacks. I’ve also tried self hosting since I am familiar with the challenges. Short answer is 80% of the time you will be able to block a <10gbps attack (assuming that’s your uplink speed) at the node level. Most boosters all use similar attack methods, and it just requires you to drop packets as upstream as possible (preferably on the NIC as a hardware upload). In most cases you can block common SOURCE ports of packets. Note, there may be legitimate uses for some protocols, but I doubt most customers use them. If the attack is >10gbps you must likely need a BGP scrubber, that can divert your traffic and “tank” any bandwidth before it comes to your line lease. If your uplink is saturated it doesn’t matter how much your hardware can filter, good traffic won’t be able to get through. There are theoretical cheap ways where you can assist your customer subject to attacks to setup a GRE tunnel to act as that scrubbing however it’s a bit annoying. Get a VM on a host that has that capability already, and route your packets through them first. Con is you have one more failure mode and increased latency.
- lxchase 5y agoTo add to what I said, the general options and availability on the stack is something like: 1. Network level: - BGP Filtering (meaning in order to reach your IP range, packets must first route through another company that has sufficient bandwidth to receive AND filter out bad packets), then clean traffic will go through to your data center, and to your server. - GRE Tunnels, Similar to above, but it will not be transparent in that you will likely use their IPs. 2. Datacenter level: - Colocate or find a dedicated server that sits behind a dedicated appliance that solely exists to act as a filter. You will also need to ask what their upstream link speeds are (i.e. 40gbps to that appliance). You still might encounter leaks, and rely on the fact that they have configured correctly or are willing to apply custom filtering if you have an advanced attack. 3. Node/Server level: - By now, you can only filter whatever your line rate is (i.e. 1gbps or 10gbps most likely). There are various methods, but all of them require to create custom filters and be active in patching up leaks. You'll want to do it as far up in the server stack as possible. Best option is SmartNICs or NICs that support hardware offloads. Second best option is in front of IPtables. Most tutorials online talk about IPtables. That assumes you have Linux first, and second not the optimal way. Use tc (traffic control) instead, it's further up on the network stack.
- stilley2 5y agoJust to add a data point, I had trouble accessing fastmail early this morning (around 7 EDT). I haven't had any issues since then.
- NabiDev 5y agoOwn the whole email provider, use protonmail? Ironic.
- jlund-molfese 5y agoIt doesn't look like that email actually belongs to Fastmail staff, but if it did, having a backup on another email service is a good idea in case there's a DNS issue with Fastmail or something. It's the same idea behind a VPS provider serving their status page on another company's infrastructure.
- spinax 5y agoThe name on that email is not one of the Runbox folks listed on their About page, so one can only guess who that actually is or how the email was sent; it could have been a BCC for all we know. https://runbox.com/about/runbox-team/ https://runbox.com/about/runbox-team/
- lrae 5y agoI don't think they'd publish some random e-mail address of a random person. Are we sure that isn't the address the extortion mail came from? Would make more sense?
- eloeffler 5y agoThe corresponding posteo blog post is archived here: https://archive.md/ZmKAA https://archive.md/ZmKAA
- TacticalCoder 5y agoWe still hear about DDoS attacks like this once in a while but it seems it's not anywhere near as common as it used to be. What happened? It looks like the bad guys are really having more and more trouble mounting succesful DDoS: how comes? It also looks like, in despair, they're targetting smaller fishes. Why? Smaller botnets? Cloudflare and OVH and the likes just being too good at absorbing everything and anything you can throw at them? Simple firewall rules getting rid of 99% of the crap? What's the reason it's not as prevalent as it used to be?
- xyzzy123 5y agoCloudFlare, AWS, GCP.
- wongarsu 5y agoCloudflare I agree, but "give us $10000 or we increase your AWS bill by $20000 per day" sounds like a viable extortion scheme to me.
- __s 5y agohttps://aws.amazon.com/shield/pricing https://aws.amazon.com/shield/pricing Might want to check how much it costs to increase someone's billing 20k a day (granted, a botnet makes it cheaper, but measure opportunity cost of what else that botnet could be doing), see also https://www.reddit.com/r/aws/comments/7z6uc3/comment/dutgw6u/?utm_source=share&utm_medium=web2x&context=3 https://www.reddit.com/r/aws/comments/7z6uc3/comment/dutgw6u... Full disclosure: I work for Azure
- posnet 5y agoAWS Shield (while expensive $36000 a year) does have `DDoS cost protection` as one of it's features. i.e. if you have to 10x your server fleet to outscale + outlast the ddos attack, then AWS will forgive the additional cost.
- Kye 5y agoMore protection at the OS and ISP level. ISPs can isolate nodes that become part of botnets, and operating systems increasingly remove the avenues malicious actors use to cause trouble. Microsoft's push for hardware security is justly controversial, but the move to TPM by default in Windows 11 is the latest in a long line of changes that's made it harder to take over an ordinary person's computer. Android has had an equivalent since 8, and I'm pretty sure iOS has it. Put that with the change to more people being mobile-only and there's fewer ways to create the botnets behind these.
- ryantgtg 5y agoWonder if it’s the same folks who hit voip.ms last week. That one lasted a while.
- 29083011397778 5y agoReading through r/VOIP, I could have sworn Voip.ms was 2 weeks ago - then other providers began to be hit with DDoS attacks. Frankly, it doesn't seem like such a bad strategy. With the amount of panic the attack on Voip.ms caused, a number of people were discussing switching voip providers. And switching targets to other providers just as they get a massive influx of new customers seems like a force multiplier IMO.
- psdmm 5y agoتتحرك جسيمات المادة الصلبة حركة اهتزازية باستمرار - https://mesael.com/25/%D8%AA%D8%AA%D8%AD%D8%B1%D9%83-%D8%AC%D8%B3%D9%8A%D9%85%D8%A7%D8%AA-%D8%A7%D9%84%D9%85%D8%A7%D8%AF%D8%A9-%D8%A7%D9%84%D8%B5%D9%84%D8%A8%D8%A9-%D8%AD%D8%B1%D9%83%D8%A9-%D8%A7%D9%87%D8%AA%D8%B2%D8%A7%D8%B2%D9%8A%D8%A9-%D8%A8%D8%A7%D8%B3%D8%AA%D9%85%D8%B1%D8%A7%D8%B1 https://mesael.com/25/%D8%AA%D8%AA%D8%AD%D8%B1%D9%83-%D8%AC%...
- SavantIdiot 5y agoI've been with RunBox since 2012 because of Norway's internet privacy policies. I support them not giving into to extortion. If I don't have email for a few days, or a month, big deal. I remember how to use a phone to pay bills.
- LeoPanthera 5y agoThis is a surprisingly insensitive comment. Email is as much a a part of modern life as phone calls and postal mail. Asking people to "just go without" for a few days may be possible for you, but is certainly not possible for a huge fraction of people.
- Aachen 5y agoMy employer uses runbox. I'm happy you're so unattached that you can go a month without any email (can't sign up anywhere, for example) but we'd miss incoming invoices, can't send invoices to our customers, can't deliver reports to customers (we do security audits so that's kinda important to be able to deliver using e.g. pgp), would miss requests for new assignments... I'm fully in favor of not paying into the extortion and not going for a global tap system like cloudflare, but we do need some solution in at most a few days. In the Netherlands there is for example the Nawas (it's also a pun, referring to a laundry cleaning thing) that scrubs malicious traffic for ISPs. I don't know the details of how they're being attacked, but with typical reflector services it's easy to remove that traffic based on a source port. Any large Norwegian hosting company (they just need an uplink bigger than the DDoS, or work with whoever the traffic comes from at their peering points) could provide that service as well. Not saying this is trivial but there are options other than waiting.
- trulyrandom 5y agoCoincidentally, the app password that I've used for Fastmail's CalDAV service for years suddenly started causing 403's today. I wonder if that's related (but can't think of how it could be)
- mmcclimon 5y ago(I work for Fastmail.) One of our attempts at doing some mitigation of the attack caused this; we fixed it about 16:00 US Eastern this afternoon. Sorry about that! Without getting into too much detail: we were limiting some HTTP methods, but unintentionally blocked REPORT, which DAV clients use to see what’s changed.
- rswail 5y agoThis is going to sound weird, but my father can't send email with attachments currently. I'm a long time fastmail user/customer and I'm also a long time IT guy so can't see what's going on. Basically SMTP login and everything works fine, client sends the message and then it just times out.
- trulyrandom 5y agoAll good now. Thanks!
- muttantt 5y agoYet another reason to host your email at your own domain which you can move at will.
- qwertox 5y agoIs it really worth the hassle to move to another provider/self-hosted server for such a temporary problem? Moving to another provider would mean to set up a new account, and usually the free tiers won't allow to use your own domain. So you would be locked to a new provider where you could face the same problems. Temporarily moving to a self-hosted server would be an option, but probably just to receive emails during this time. You'd have to set up your certificate, and optionally DKIM, DMARC, SPF or whatever is required to ensure that your sent emails arrive properly. I can't imagine self-hosting email being something which won't give you a hard time every now and then. Generally yes, I am in favor of owning your email-domain, but then using it with a professional provider like mailbox.org unless you really know enough about the topic. But in this case, where the issue is a DDoS attack, I wouldn't do anything, since all undelivered email will be re-sent at a later time.
- kiwijamo 5y agoOur Bank "A" suffered a major DDoS attack recently. My non-techy partner, upset, declared were moving to Bank "B". I pointed out that Bank B had also been target of DDoS attacks so we would be moving banks only to face the exact same issues. Point being you may move to a different service provider (email, banking, whatever) to find the new one has the same problems anyway.
- Aachen 5y agoOr less hassle than completely moving over to a competitor (terrorists win in that case, to use counter strike terminology): add a backup MX record, perhaps to a small vps that just forwards mail to the real server with no retry timeout.
- throwaway984393 5y agoIt sounds like it could work. Are there any mail admins in the comments who could propose an smtp daemon & configs that would store all mail for a domain and wait indefinitely to forward it on? I think there's a lot of users/businesses who would like to implement this.
- julianlam 5y agoThe sooner these kids realize that DDoS extortion attacks do nothing except waste resources (and monetary resource to CloudFlare, basically), the sooner they'll give up. They hit a number of VoIP providers recently, too.
- muttantt 5y agoThe latest attacks on VoIP providers resulted in a SIP/VoIP specific solution from Cloudflare. I expect we will see Cloudflare mail protection soon enough... Makes you wonder.
- yarcob 5y agoI first learned that someone cracked my app from a company that offered to help me fight cracked downloads of my app. I was immediately suspicious.
- a1369209993 5y agoThis isn't new; it's called a protection racket and it works especially well if you actually do have a mechanism to protect against the thing (though of course that's not a requirement if you're generating most of the problem yourself).
- piperswe 5y agoCloudflare Spectrum can already protect arbitrary TCP services, such as SMTP or IMAP
- prdonahue 5y agoStay tuned .. Today we can help quite a bit by proxying TCP traffic using Cloudflare Spectrum, e.g., spoofed traffic will never reach origin as it can't complete a handshake, we can use things like SYN cookies to challenge source, etc. In the future, there's a lot more we plan to do here.
- pan69 5y agoAnyone knows what the cost of performing a DDoS is these days? As a target, if you can't do anything else, is it best to hang in there and wait it out due to mounting costs for the attacker?
- exegete 5y agomailbox.org was also attacked https://twitter.com/mailbox_org/status/1451237052855595012?s=21 https://twitter.com/mailbox_org/status/1451237052855595012?s... I kept having errors in sending emails and logging into the web interface and the mailbox.org status page indicated nothing was wrong. Why have the page if that’s not where I’m supposed to find out something like this?
- omnimus 5y agoThe server was probably running it was just flooded. Probably requires different type of checks to catch this.
- dang 5y agoUrl changed from https://blog.runbox.com/2021/10/runbox-is-under-attack-by-extortionists/ https://blog.runbox.com/2021/10/runbox-is-under-attack-by-ex..., which points to this.
- Aachen 5y agoDDoS is said to be very cheap. What if we used that to boot the culprits? It's obviously illegal, so just as a thought experiment: if those amplifiers (dns resolvers or whatever is popular at the moment) started experiencing issues due to their servers being a nuisance to others...? It still seems much easier to just blackhole IPs that are causing problems, like collectively (at the edge of your AS) block IPs that long-term host a service that is actively involved in facilitating DDoS attacks, but for some reason nobody is doing that. This could be a more direct way: see where DDoS traffic is coming from and... poof
- prdonahue 5y agoNot really practical or effective to implement. The attacks more often than not come from botnets comprised of compromised consumer devices. You can’t just outright drop traffic from residential ISPs. We didn’t disclose it at the time but this 17.2M rps attack came from (home) Mikrotik devices that were running proxy services: https://blog.cloudflare.com/cloudflare-thwarts-17-2m-rps-ddos-attack-the-largest-ever-reported/ https://blog.cloudflare.com/cloudflare-thwarts-17-2m-rps-ddo....
- Aachen 5y agoAh right, I was figuring most of those misconfigured udp services (dns, ntp, ...) were running on servers rather than regular home IPs. That does make it a little different. Still, if an ISP has had multiple abuse reports for the same subscriber and they're not doing anything, after some time it starts to become reasonable to block this IP, and in a further escalation, this ISP's ranges altogether until they clean their act up. I remember getting the Internet connection blocked as a teenager on an XS4ALL connection for being an ass on the Internet (I tried to DoS a domain squatter that tried to sell a domain I wanted for a thousand times the price with no added value). The abuse desk which I had to contact to unblock it took my promise to not do it again seriously (as did I), not sure how other ISPs handle this.
- sysadm1n 5y ago> Since these DDoS attacks started we have worked with our system administrators and Internet Service Provider to mitigate the attacks How do you mitigate such an attack though? I know Cloudflare can stop this, but how do you create your own bespoke 'DDOS mitigation' tool, and what does that look like?
- ahmedalsudani 5y agoAdding capacity, dropping suspect traffic.
- 0des 5y agoYou simply need bigger pipes to ingest more traffic than the attack can provide. It is presumed these days that packet analysis in some cases can require too much power costwise rather than scaling up the connection to swallow it.
- bombcar 5y agoAnd the "further up the chain" you can move the mitigation, the easier that is. Mitigating on your box requires a huge pipe to your box, but if your provider can mitigate at their border router, well those are bigger and already have huge traffic to and through them. And it's in their interest to talk to their providers and mitigate even further back if possible. And some mitigations are relatively easy (block all DNS traffic to this subnet, etc).
- capableweb 5y agoOne way is to build your software on top of distributed/content-addressed P2P software (not Blockchain, but pure P2P). The angle of attack disappears completely then. Otherwise, with centralized infrastructure, you identify the bad traffic and send it elsewhere, an instance that keeps connections open but delivers nothing and have a small amount of resource usage. Problem is that you still need to be able to handle the traffic, but at least you avoid hitting your main infrastructure.
- baybal2 5y ago
- Wronnay 5y agoI don't wonder that Runbox and Posteo are vulnerable, but I would have thought that Fastmail is too big for such an attack...
- meribold 5y agoFastmail is working just fine for me. At least the web interface was briefly offline yesterday, though.
- throwawaysea 5y agoIf you didn’t want to depend on a big provider like AWS or Cloudflare, what is the approach to fending off a DDoS attack? What type of hardware would you need to acquire? What type of software? Are there guides on this type of thing?
- neom 5y agoYou could try to do it yourself with firewall rules, reverse proxies, things like that, but to fight large scale ddos you really need to be moving the traffic around using BGP, and you'd want to dump the traffic somewhere, so you'd need bandwidth to dump the traffic into, that's why companies like cloudflare exist, they're able to work with bgp and they have a lot of bandwidth to absorb the traffic on behalf of the customer.
- bluGill 5y agoYou need a massive amount of bandwidth and a few redundant servers. There are counties with less bandwidth than you need to handle. It isn't impossible, but cloudflare isn't evil (that I know of?) and so it is best to support them as your backup
- deleted 5y ago[deleted]
- deleted 5y ago[deleted]
- swozey 5y agoWell that explains it, I literally switched from gmail to fastmail 2 days ago and it was going amazingly until last night when I couldn't load it. "Oh great, an unreliable service I just paid a year for" -- this makes me want to support them even more. Their customer support was fantastic. I didn't inquire about the outage, instead a dns issue I'd created. If you're a heavy gmail user you should read this story of my last year (out of 10+) with them and IMO migrate off as soon as you can. I wish I hadn't used an @gmail as long as I did. For some unknown reason, they won't tell me, G marked my google pay account as "possible fraud, unable to verify identity" at some point this year. No idea when, I realized it when CC expired and my 3 month old Google Fi account wouldn't pay its bill and it wouldn't let me enter a credit card to pay for any G service. My G services started to drop like flies once my card expired. They locked me out of Google Pay and won't let me enter a new CC to pay for subs/gdrive. I had to clear out the entire 50gb of my gdrive so that I could get EMAIL again to my now "free" gmail account which is hovering around its 15gb limit, that's how old it is. The real kicker; I can't contact support anymore because I'm not a "paying customer" who gets their "world class support." The fun part of this was I was completely without a cell phone for a week until I got onto ATT. Support (live chat) was completely useless and just sent me to a page to send them a copy of my license and a utility bill. I did that about 4 times now and I still can't change CCs on my Google account to pay for things. And they still won't contact me, won't tell me what the deal is or anything. It's been about 6 months. I was joking to a friend before I switched to Fi "watch this be a terrible idea, it breaks and I'm without a phone for a month and have nobody to fix it." Welp. Anyway, I tried out Protonmail months ago and was completely unimpressed with its search, interface, etc. I had low expectations of fastmail (due to proton exp) but it's great, I'm really impressed. It threads conversations like you'd expect and I could actually find things I searched for, I sold a house in the middle of my Proton use and it was miserable keeping track of all of the emails/docs/etc I was getting/sending for some reason. I remember searching for important attachments and it finding 10s or hundreds of attachments that were my realtors signature picture of herself that showed up in every email. Sorry for the tangent, be very careful of how much you rely on G services because this cliche horror story you randomly see on twitter/reddit completely happened to me. I'll never be able to stop using this @gmail account because I have so, so many things tied to it but I'm going to try my best to undo most of that. My biggest fear now is that my G account gets completely locked and all of the things I use G to authenticate to will be lost. Undoing all that is a nightmare. Use a custom domain for your email..
- deleted 5y ago[deleted]
- oars 5y agoFantastic discussion about DDoS in these comments, thank you.
- deleted 5y ago[deleted]