5 ms·
Multiple Digital Ocean services down
- jsheard 10mo agoThey don't name names but it's probably due to the ongoing Cloudflare explosion. I know the DigitalOcean Spaces CDN is just Cloudflare under the hood.
- matt-p 10mo agoJust spaces CDN, not spaces - you'd think they'd just turn the CDN off for a bit.
- potato3732842 10mo agoYou can't just "turn off CDN" on the modern internet. You'd instantly DDOS your customers' origins. They're not provisioned to handle it, and even if they were the size of the pipe going to them isn't. The modern internet is built around the expectation that everything is distributed via CDN. Some more "traditional" websites would probably be fine.
- matt-p 10mo agoIf you are DO you could, you just decided not to bother. They control the origins it's spaces (s3), so they could absolutely spin up further gateways or a cache layer and then turn the CDN off.
- graemep 10mo agoEither you are wrong and they do not have the capacity to do that, or they have decided it is acceptable to be down because a major provider is down I imagine a cache layer cannot be that easy to spin up - otherwise why would they outsource it?
- matt-p 10mo agoYou outsource it because clouflare have more locations than you so offer lower latency and can offer it at a cost that's cheaper or the same price as doing it yourself.
- graemep 10mo agoWhich suggests its expensive enough for it to be unlikely they just have the capacity lying around to spin up.
- oasisbob 10mo agoTo the contrary, CDN pricing will usually beat cloud provider egress fees. Common example: you can absolutely serve static content from an S3 bucket worldwide without using a CDN. It will usually scale OK under load. However, you're going to pay more for egress and give your customers a worse experience. Capacity isn't the problem you're engineering around. For a site serving content at scale, a CDN is purpose-built to get content around the world efficiently. This is usually cheaper and faster than trying to do it yourself.
- graemep 10mo agoThat is not what I said. I said DO will not have the spare capacity because its too expensive. Can you please tell me who DO pay egress fees to?
- matt-p 10mo agoThey will be doing a mix of peering both across free PNIs and very low cost IXP ports, with the reminder going down transit like Colt or cogent. Probably average cost of the order of about $1 per 20TB of egress in Europe and NA markets. The thing is with edge capacity is that you massively overbuild on the basis that; It's generally a long ISH lead time to scale capacity (days not minutes or hours). Transit contracts are usually 12-60 months Traffic is variable, not all ports cover all destinations Redundancy So if you are doing say 100Gbps 95%ile out of say London then you will probably have at least 6+ 100Gb ports, so you do have quite a bit of latent capacity if you just need it for a few hours.
- oasisbob 10mo agoMight be just me, but I can think of many origins under my control which could live without a (non-functional) CDN for a while. CDN is great for peak-load, latency reductions, and cost - but not all sites depend on it for scale 24/7
- tgma 10mo agonit: that's more DoS (from a handful of DO LBs) than DDoS.
- TechRemarker 10mo agoYes all sites showing the CloudFlare error due to the massive outage. Seems their outages are getting more frequent and taking down the internet in new ways each time.
- archerx 10mo agoI have two projects on DO using droplets and they are still running fine.
- hshdhdhj4444 10mo agoIt seems mostly a CludFlare related issue. My DOs are working fine as well.
- sgc 10mo agoAre you using their "reserved IPs"? I was thinking of starting to use them, but now I wonder if it is part of their load balancing stack under the hood.
- soheilpro 10mo agoDroplets are fine. > This incident affects: API, App Platform (Global), Load Balancers (Global), and Spaces (Global).
- giancarlostoro 10mo agoSo yesterday Azure got hit hard, today CF and DO are down, bad week or something else?
- zx8080 10mo agoYear-end promotion cycle is the worst time for end-users and the best one for engineers greedy for promotions.
- Lammy 10mo agoDon't blame individual engineers who want to do what will be rewarded instead of company performance policies that reward this type of behavior.
- fridder 10mo agoshoot, there are also end of year layoffs and reorgs to pump up those year end numbers
- red-iron-pine 10mo agowhat engineers, mate? they AI now and they're doing just spectacular
- BubbleRings 10mo agoI would also like to know people’s opinion on this.
- matt-p 10mo agoDDOS, but I don't really understand why in particular.
- giancarlostoro 10mo agoHaving known people like this, its either flexing about who has the more powerful botnet or advertising who can do what.
- showerst 10mo agoI use DO's load balancers in a couple of projects, and they don't list Cloudflare as an upstream dependency anywhere that I've seen. It's so frustrating to think you're clear of a service then find out that you're actually in their blast radius too through no fault of your own.
- wesammikhail 10mo agoslight off topic: I used DO LBs for a little while but found myself moving away from that toward a small droplet with haproxy or nginx setup. Worked much better for me personally!
- showerst 10mo agoThe point of an LB for these projects is to get away from a single point of failure, and I find configuring HA and setting up the networking and everything to be a pain point. These are all low-traffic projects so it's more cost effective to just throw on the smallest LB than spend the time setting it up myself.
- grayhatter 10mo agoIf they are small projects, why are they behind a load balancer to begin with?
- nickmonad 10mo agoUsually because of SSL termination. It's generally "easier" to just let DO manage getting the cert installed. Of course, there are tradeoffs.
- showerst 10mo agoI use the LB's for high availability rather than needing load balancing. The LB + 2 web back-ends + Managed DB means a project is resilient to a single server failing, for relatively low devops effort and around $75/mo.
- drob518 10mo agoDominos falling into dominos falling into dominos…
- aforty 10mo agoCloudflare outage.
- igtztorrero 10mo agoI knew it, DigitalOcean CDN is using Cloudflare behind the scenes. Why DO ?
- mrkramer 10mo agoWho is next?
- red-iron-pine 10mo agomy guesses would be look at who has a FedRAMP capable service first. maybe also GCP, hetzner, akamai
- foxyv 10mo agoMan, it really seems like the cloud providers are having some tough times lately. Azure, AWS, and Cloudflare! Is everything just secretly AWS?