8 ms·

Pretty much everything is down (checking from the Netherlands). The Cloudflare dashboard itself is experiencing an outage as well. Not-so-funny thing is that t

by farhadhf 10mo ago

Pretty much everything is down (checking from the Netherlands). The Cloudflare dashboard itself is experiencing an outage as well. Not-so-funny thing is that the Betterstack dashboard is down but our status page hosted by Betterstack is up, and we can't access the dashboard to create an incident and let our customers know what's going on. Edit: wording.

biinjo 10mo ago
Same here. We’re using OhDear. The status page is available but I can’t post an incident because their service is also behind Cloudflare.
- Mojah 10mo ago
  Co-founder here, we'll be working on better ways to handle this over the coming days. Update: our app is available again without Cloudflare, you'll be able to post updates to status pages smoothly again.
ablation 10mo ago
This is a big one.
csomar 10mo ago
Seems like workers are less affected and maybe betterstack has decided to bypass cloudflare "stuff" for the status pages? (maybe to cut down costs). My site is still up though some GitHub runners did show it failed at certain points.
- tyingq 10mo ago
  I have a workers + kv app that seems fine right now.
  - csomar 10mo ago
    Pretty sure they went down for a while because I have 4xx errors they returned but apparently it was short-lived. I wonder if their workers infra. failed for a moment and that let to a total collapse of all of their products?
pell 10mo ago
It’s that time of the year again where we all realize that relying on AWS and Cloudflare to this degree is pretty dangerous but then again it’s difficult to switch at this point. If there is a slight positive note to all this, then it is that these outages are so large that customers usually seem to be quite understanding.
- tommica 10mo ago
  > It’s that time of the year again It's monthly by now
- fusl 10mo ago
  Happy to hear anyone's suggestions about where else to go or what else to do in regards to protecting from large-scale volumetric DDoS attacks. Pretty much every CDN provider nowadays has stacked up enough capacity to tank these kind of attacks, good luck trying to combat these yourself these days?
  - isodev 10mo ago
    Anubis and/or Bunny are good alternatives/combination depending on your exact needs - https://anubis.techaro.lol/ https://anubis.techaro.lol/ - https://bunny.net/ https://bunny.net/
    - fusl 10mo ago
      Unfortunately Anubis doesn't help where my pipe to the internet isn't fat enough to just eat up all the bandwidth that the attacker has available. Renting tens of terabits of capacity isn't cheap and DDoS attacks nowadays are in the scale of that. BunnyCDN's DDoS protection is unfortunately too basic to filter out anything that's ever so slightly more sophisticated. Cloudflare's flexibility in terms of custom rulesets and their global pre-trained rulesets (based on attacks they've seen in the past) is imo just unbeatable at this time.
      - isodev 10mo ago
esskay 10mo ago
When its back up, do yourself a favour and rent a $5/mo vps in another country from a provider like OVH or Hetzner and stick your status page on that. "Yes but what if they go down" - it doesnt matter, having it hosted by someone who can be down for the same reason as your main product/service is a recipe for disaster.
- fodi 10mo ago
  Definitely. Tangentially, I encountered 504 Gateway Timeout errors on cloudflarestatus.com about an hour ago. The error page also disclosed the fact that it's powered by CloudFront (Amazon's CDN).
- hcaz 10mo ago
  https://cachethq.io/ https://cachethq.io/ is great for this
  - fusl 10mo ago
    Been using Cachet for quite a while before inevitably migrating to Atlassian's Statuspage.io. I'm a huge fan of self-hosting and self-managing every single thing in existence but Cachet was just such a PITA to maintain and there was just no other good alternative to Cachet that was also open source.
  - jwr 10mo ago
    Amusingly enough, it is down right now because of Cloudflare :-)
- jwr 10mo ago
  Or use a service like https://updown.io/ https://updown.io/ (I host my status page there).
postalcoder 10mo ago
Cloudflare dashboard is down-ish, not totally down. If you're persistent you can turn off the turnstile and proxy. It took a few minutes but I got https://hcker.news https://hcker.news off of it.
- trollbridge 10mo ago
  I can't sign in since Turnstile is down so I can't complete the captcha to log in. I also can't log in via Google SSO since Cloudflare's SSO service is down.
  - deleted 10mo ago
    [deleted]
- farhadhf 10mo ago
  I'm already logged in on the cloudflare dashboard and trying to disable the CF proxy, but getting "404 | Either this page does not exist, or you do not have permission to access it" when trying to access the DNS configuration page.
  - deleted 10mo ago
    [deleted]
- skywhopper 10mo ago
  Not saying not to do this to get through, but just as an observation, it’s also the sort of thing that can make these issues a nightmare to remediate, since the outage can actually draw more traffic just as things are warming up, from customers desperate to get through. But then, that’s what Cloudflare signed up to be.
celltalk 10mo ago
I think there is a big business opportunity here. Make a site that let companies put their status update on local vps for $100.
- colinbartlett 10mo ago
  Even the Cloudflare status page, hosted by Atlassian Statuspage, is suffering. Probably due to the traffic crush.
- ramon156 10mo ago
  on-demand status balancing!
- nrhrjrjrjtntbt 10mo ago
  Status pigeons.
- alt227 10mo ago
  Atlassian has this business model sewn up https://www.atlassian.com/software/statuspage https://www.atlassian.com/software/statuspage
  - lc64 10mo ago
    It's worth noting that cloudflare's status page is hosted there. Pretty good proof that it works
    - dmit 10mo ago
      And I got a 504 error (served by CloudFront) on that status page earlier. The error message suggested there may have been a great increase in traffic that caused it.
- codethief 10mo ago
  Maybe that's precisely what Cloudflare did and now their status page is down because it's receiving an unusual amount of traffic that the VPS can't handle.
chrisandchris 10mo ago
I don't get why you need such a service for a status page with 99.whatever% uptime. I mean, your status page only has to be up if everything else is down, so maybe 1% uptime is fine. /s
huijzer 10mo ago
Yep that's also my experience. Except HN because it does not use *** Cloudflare because it knows it is not necessary. I just wrote a blog titled "Do Not Put Your Site Behind Cloudflare if You Don't Need To" [1]. [1]: https://huijzer.xyz/posts/123/ https://huijzer.xyz/posts/123/
- MinimalAction 10mo ago
  Yes, I never understand this obsession for centralized services like Cloudflare. To be fair though, if our tiny blogs anyway had a hundred or so visitors monthly, does it matter if it had an outage for a day?
  - ThunderSizzle 10mo ago
    I think partially is not having to worry about certs is a nice reason to hide behind the proxy. Also, to help hide your IP address, I guess. Of course, on the other hand, I know that relying on Cloudflare cert's is basically inviting a MITM attack.
    - huijzer 10mo ago
      > I think partially is not having to worry about certs is a nice reason to hide behind the proxy. Use Caddy. I never worry about certs.
      - ThunderSizzle 10mo ago
        Interesting. I've done a lot of manual work to set up a whole nginx layer to properly route stuff through one domain to various self-hosted services, with way to many hard lessons when I started this journey (from trying to do manual setup without docker, to moving onto repeatable setups via docker, etc.). The setup appears very simple in Caddy - amazingly simple, honestly. I'm going to give it a good try.
      - immibis 10mo ago
        Or certbot-plugin-nginx if you prefer a bit less magic.
davedx 10mo ago
All my stuff is working. Things on GCP. Things on Fly.io. Tooling I use. "Only" 10% of the internet is behind Cloudflare so far ;)
- grabcadder 10mo ago
  Happy for you :) I am curious about these two things: 1- Does GCP also have any outages recently similar to AWS, Azure or CF? If a similar size (14 TB?) DDoS were to hit GCP, would it stand or would it fail? 2- If this DDoS was targeting Fly.io, would it stand? :)
  - progbits 10mo ago
    For GCP network that would be a rounding error. Of course GCP sometimes has outages too, all providers do.
  - davedx 10mo ago
    I actually spoke too soon, and accept I have egg on my face! Apparently prisma's `npm exec prisma generate` command tries to download "engine binaries" from https://binaries.prisma.sh https://binaries.prisma.sh, which is behind... guess what... So now my CI/CD is broken, while my production env is down, and I can't fix it. Amazing lol
    - grabcadder 10mo ago
      I spoke too soon as well :) Apparently this wasn't a DDoS attack (or more correctly, they kind of DDoS themselves)
compumike 10mo ago
Could always just use a status page that updates itself. For my side project Total Real Returns [1], if you scroll down and look at the page footer, I have a live status/uptime widget [2] (just an <img> tag, no JS) which links to an externally-hosted status page [3]. Obviously not critical for a side project, but kind of neat, and was fun to build. :) [1] https://totalrealreturns.com/ https://totalrealreturns.com/ [2] https://status.heyoncall.com/svg/uptime/zCFGfCmjJN6XBX0pACYY/uptime_bars_and_percent_400x200 https://status.heyoncall.com/svg/uptime/zCFGfCmjJN6XBX0pACYY... [3] https://status.heyoncall.com/o/zCFGfCmjJN6XBX0pACYY https://status.heyoncall.com/o/zCFGfCmjJN6XBX0pACYY
- jcfrei 10mo ago
  This is unrelated to the cloudflare incident but thanks a lot for making that page. I keep checking it from time to time and it's basically the main data source for my long term investing.
  - compumike 10mo ago
    I appreciate that, thank you! :)
touristtam 10mo ago
Thankfully the usual social media are still up ... oh wait https://www.bbc.co.uk/news/articles/c629pny4gl7o https://www.bbc.co.uk/news/articles/c629pny4gl7o
talonx 10mo ago
BetterStack did report issues with some of their services, but they were not very informative.

biinjo 10mo ago

Same here. We’re using OhDear. The status page is available but I can’t post an incident because their service is also behind Cloudflare.

Mojah 10mo ago
Co-founder here, we'll be working on better ways to handle this over the coming days. Update: our app is available again without Cloudflare, you'll be able to post updates to status pages smoothly again.

ablation 10mo ago

This is a big one.

csomar 10mo ago

Seems like workers are less affected and maybe betterstack has decided to bypass cloudflare "stuff" for the status pages? (maybe to cut down costs). My site is still up though some GitHub runners did show it failed at certain points.

tyingq 10mo ago
I have a workers + kv app that seems fine right now.
- csomar 10mo ago
  Pretty sure they went down for a while because I have 4xx errors they returned but apparently it was short-lived. I wonder if their workers infra. failed for a moment and that let to a total collapse of all of their products?

pell 10mo ago

It’s that time of the year again where we all realize that relying on AWS and Cloudflare to this degree is pretty dangerous but then again it’s difficult to switch at this point. If there is a slight positive note to all this, then it is that these outages are so large that customers usually seem to be quite understanding.

tommica 10mo ago
> It’s that time of the year again It's monthly by now
fusl 10mo ago
Happy to hear anyone's suggestions about where else to go or what else to do in regards to protecting from large-scale volumetric DDoS attacks. Pretty much every CDN provider nowadays has stacked up enough capacity to tank these kind of attacks, good luck trying to combat these yourself these days?
- isodev 10mo ago
  Anubis and/or Bunny are good alternatives/combination depending on your exact needs - https://anubis.techaro.lol/ https://anubis.techaro.lol/ - https://bunny.net/ https://bunny.net/
  - fusl 10mo ago
    Unfortunately Anubis doesn't help where my pipe to the internet isn't fat enough to just eat up all the bandwidth that the attacker has available. Renting tens of terabits of capacity isn't cheap and DDoS attacks nowadays are in the scale of that. BunnyCDN's DDoS protection is unfortunately too basic to filter out anything that's ever so slightly more sophisticated. Cloudflare's flexibility in terms of custom rulesets and their global pre-trained rulesets (based on attacks they've seen in the past) is imo just unbeatable at this time.
    - isodev 10mo ago

esskay 10mo ago

When its back up, do yourself a favour and rent a $5/mo vps in another country from a provider like OVH or Hetzner and stick your status page on that. "Yes but what if they go down" - it doesnt matter, having it hosted by someone who can be down for the same reason as your main product/service is a recipe for disaster.

fodi 10mo ago
Definitely. Tangentially, I encountered 504 Gateway Timeout errors on cloudflarestatus.com about an hour ago. The error page also disclosed the fact that it's powered by CloudFront (Amazon's CDN).
hcaz 10mo ago
https://cachethq.io/ https://cachethq.io/ is great for this
- fusl 10mo ago
  Been using Cachet for quite a while before inevitably migrating to Atlassian's Statuspage.io. I'm a huge fan of self-hosting and self-managing every single thing in existence but Cachet was just such a PITA to maintain and there was just no other good alternative to Cachet that was also open source.
- jwr 10mo ago
  Amusingly enough, it is down right now because of Cloudflare :-)
jwr 10mo ago
Or use a service like https://updown.io/ https://updown.io/ (I host my status page there).

postalcoder 10mo ago

Cloudflare dashboard is down-ish, not totally down. If you're persistent you can turn off the turnstile and proxy. It took a few minutes but I got https://hcker.news https://hcker.news off of it.

trollbridge 10mo ago
I can't sign in since Turnstile is down so I can't complete the captcha to log in. I also can't log in via Google SSO since Cloudflare's SSO service is down.
- deleted 10mo ago
  [deleted]
farhadhf 10mo ago
I'm already logged in on the cloudflare dashboard and trying to disable the CF proxy, but getting "404 | Either this page does not exist, or you do not have permission to access it" when trying to access the DNS configuration page.
- deleted 10mo ago
  [deleted]
skywhopper 10mo ago
Not saying not to do this to get through, but just as an observation, it’s also the sort of thing that can make these issues a nightmare to remediate, since the outage can actually draw more traffic just as things are warming up, from customers desperate to get through. But then, that’s what Cloudflare signed up to be.

celltalk 10mo ago

I think there is a big business opportunity here. Make a site that let companies put their status update on local vps for $100.

colinbartlett 10mo ago
Even the Cloudflare status page, hosted by Atlassian Statuspage, is suffering. Probably due to the traffic crush.
ramon156 10mo ago
on-demand status balancing!
nrhrjrjrjtntbt 10mo ago
Status pigeons.
alt227 10mo ago
Atlassian has this business model sewn up https://www.atlassian.com/software/statuspage https://www.atlassian.com/software/statuspage
- lc64 10mo ago
  It's worth noting that cloudflare's status page is hosted there. Pretty good proof that it works
  - dmit 10mo ago
    And I got a 504 error (served by CloudFront) on that status page earlier. The error message suggested there may have been a great increase in traffic that caused it.
codethief 10mo ago
Maybe that's precisely what Cloudflare did and now their status page is down because it's receiving an unusual amount of traffic that the VPS can't handle.

chrisandchris 10mo ago

I don't get why you need such a service for a status page with 99.whatever% uptime. I mean, your status page only has to be up if everything else is down, so maybe 1% uptime is fine. /s

huijzer 10mo ago

Yep that's also my experience. Except HN because it does not use *** Cloudflare because it knows it is not necessary. I just wrote a blog titled "Do Not Put Your Site Behind Cloudflare if You Don't Need To" [1]. [1]: https://huijzer.xyz/posts/123/ https://huijzer.xyz/posts/123/

MinimalAction 10mo ago
Yes, I never understand this obsession for centralized services like Cloudflare. To be fair though, if our tiny blogs anyway had a hundred or so visitors monthly, does it matter if it had an outage for a day?
- ThunderSizzle 10mo ago
  I think partially is not having to worry about certs is a nice reason to hide behind the proxy. Also, to help hide your IP address, I guess. Of course, on the other hand, I know that relying on Cloudflare cert's is basically inviting a MITM attack.
  - huijzer 10mo ago
    > I think partially is not having to worry about certs is a nice reason to hide behind the proxy. Use Caddy. I never worry about certs.
    - ThunderSizzle 10mo ago
      Interesting. I've done a lot of manual work to set up a whole nginx layer to properly route stuff through one domain to various self-hosted services, with way to many hard lessons when I started this journey (from trying to do manual setup without docker, to moving onto repeatable setups via docker, etc.). The setup appears very simple in Caddy - amazingly simple, honestly. I'm going to give it a good try.
    - immibis 10mo ago
      Or certbot-plugin-nginx if you prefer a bit less magic.

davedx 10mo ago

All my stuff is working. Things on GCP. Things on Fly.io. Tooling I use. "Only" 10% of the internet is behind Cloudflare so far ;)

grabcadder 10mo ago
Happy for you :) I am curious about these two things: 1- Does GCP also have any outages recently similar to AWS, Azure or CF? If a similar size (14 TB?) DDoS were to hit GCP, would it stand or would it fail? 2- If this DDoS was targeting Fly.io, would it stand? :)
- progbits 10mo ago
  For GCP network that would be a rounding error. Of course GCP sometimes has outages too, all providers do.
- davedx 10mo ago
  I actually spoke too soon, and accept I have egg on my face! Apparently prisma's `npm exec prisma generate` command tries to download "engine binaries" from https://binaries.prisma.sh https://binaries.prisma.sh, which is behind... guess what... So now my CI/CD is broken, while my production env is down, and I can't fix it. Amazing lol
  - grabcadder 10mo ago
    I spoke too soon as well :) Apparently this wasn't a DDoS attack (or more correctly, they kind of DDoS themselves)

compumike 10mo ago

Could always just use a status page that updates itself. For my side project Total Real Returns [1], if you scroll down and look at the page footer, I have a live status/uptime widget [2] (just an <img> tag, no JS) which links to an externally-hosted status page [3]. Obviously not critical for a side project, but kind of neat, and was fun to build. :) [1] https://totalrealreturns.com/ https://totalrealreturns.com/ [2] https://status.heyoncall.com/svg/uptime/zCFGfCmjJN6XBX0pACYY/uptime_bars_and_percent_400x200 https://status.heyoncall.com/svg/uptime/zCFGfCmjJN6XBX0pACYY... [3] https://status.heyoncall.com/o/zCFGfCmjJN6XBX0pACYY https://status.heyoncall.com/o/zCFGfCmjJN6XBX0pACYY

jcfrei 10mo ago
This is unrelated to the cloudflare incident but thanks a lot for making that page. I keep checking it from time to time and it's basically the main data source for my long term investing.
- compumike 10mo ago
  I appreciate that, thank you! :)

touristtam 10mo ago

Thankfully the usual social media are still up ... oh wait https://www.bbc.co.uk/news/articles/c629pny4gl7o https://www.bbc.co.uk/news/articles/c629pny4gl7o

talonx 10mo ago

BetterStack did report issues with some of their services, but they were not very informative.