22 ms·

Linode / Akamai US-EAST is down

30 points by mherrmann 1y ago

dreeves 1y ago
Is this hard downtime for anyone else? We're not sure what they mean by "emerging service issue" and they haven't replied to our support ticket yet.
- upseo 1y ago
  For us too, nothing works at all
- mathrawka 1y ago
  Yes, it has gotten worse as time progressed. Some k8s services started to fail, which is how I noticed something was wrong. Then the k8s control plane was up and down. Then k8s control plan completely done. Now I can't even connect to any of my non k8s servers over ssh.
- dreeves 1y ago
  Some signs of life from some of our servers now. Still no word from Linode.
- basilgohar 1y ago
  Yes, my servers and all associate services are down: DNS, email, websites. It's a major outage at the whole Newark datacenter, which is their main one, no less.
  - basilgohar 1y ago
    Thankfully all my nodes are back up. But the DNS server as down so long that Namecheap registered it as a personal DNS server, so I am working with them to get it back up. These are secondary effects of outage, not Linode directly, but caused by the outage itself.
- cmg 1y ago
  I woke up to a few hundred messages from Icinga - thankfully my phone is on do-not-disturb overnight. Some of my servers in Newark are up and responding, some are not. Happy Sunday! Cleaning up the automatically-created maintenance/alert tickets generated by this is going to be a fun time.
dreeves 1y ago
More assurance that it's not just us: https://downdetector.com/status/linode/ https://downdetector.com/status/linode/ "Mal de muchos, consuela de tontos."
dreeves 1y ago
Encouraging update on the status page: "The issue is related to heating/cooling complications in the data center due to a power outage . The power outage has been fixed and we are working quickly to bring our services back online."
skbohra123 1y ago
We are also impacted. Services still down.
morpheous77 1y ago
Hey guys I also fall into this issue, have you ever heard about some compensation it's been about 8 hours and problem persists. Good day all
PierreJasmin 1y ago
It affects the company we use as online license server - lots of worried customers today here...
bobadob 1y ago
They said it started with a power outage, but I thought Linode data centers ran on UPS and didn't have power outages? And, given the length of this outage, the problem seems more serious.
- basilgohar 1y ago
  The power outage affected their HVAC system, so it seems to me like the entire DC had to be shutdown, so bringing everything back up from a black start seems to be the situation.
pupppet 1y ago
16 hours downtime now, yeesh.
- morpheous77 1y ago
  yup, I have one missing for booting...the most important
hecanjog 1y ago
Our instance just came back up.
ajdude 1y ago
I'm using a linode VPS as my mail server, this is hurting me badly.
- basilgohar 1y ago
  I have a Mail-in-a-Box instance that does DNS as well for all my sites. Still down....
chriskkim 1y ago
One of the three nodes came back online at 5PM EST, still waiting for the other two at 1AM EST ...
lgeek 1y ago
Over 22 hours of downtime for the one VPS I have in that region. My infrastructure is redundant and spread out among hosting providers and DCs so there's no real impact, but I'm pretty sure this is the longest outage I've ever had with any provider. And the communication level has been so dissapointing. 4 hrs to say it's a power / HVAC issue? Updates that basically just say we're still working on it since then.
- ajdude 1y ago
  We are approaching 24 hours of downtime, I'm still one of those also affected and I'm starting to wonder if the situation is worse than they are letting on.
- lgeek 1y ago
  Update: it's finally up again after around 25.5 hrs of downtime
  - ajdude 1y ago
    Here's to hoping mine comes up again soon
haik90 1y ago
It is still ongoing and impacting multiple services and regions. Looking into the status page history, it seems they're facinga different issue on Kubernetes for the last couple of days
cmg 1y ago
Of 60-something Linodes in Newark across a few accounts (we don't use LKE, Node Balancers, etc) - Many came back up yesterday. Most of the rest came back up this morning. - All but two are back online. One of those is "Powered off" but can't be turned on because "Linode busy". The other is online but unreachable, same behavior as most of them during the outage. - Three required me to put them in Rescue Mode and run fsck.ext4 -F /dev/sda to get them back online.
basilgohar 1y ago
Thankfully all my nodes are back online and accessible via SSH but there are currently secondary effects due to the length of the outage that I am dealing with, such as my DNS server being deregistered with Namecheap from serving as a personal DNS server (their name for that feature). The consequence of this is also that my email is not currently work, since that affects the hostname the email server operates under serving multiple other domains. I am still grateful that it seems I suffered no data loss, thank God! Also, I've taken it upon myself to PROPERLY implement a better, redundant backup strategy (since I was mainly relying on Linode's service, but now I feel I should go beyond that). I am using restic backing-up to a Backblaze bucket via the S3 interface. Nice thing is I can put all hosts into the same bucket and restic will organize by host but still full deduplication support. Not sure how much that'll net me but it's nice to have.
- ajdude 1y ago
  I'm in the exact same boat- mail server hosted on linode with namecheap as my registrar. I couldn't figure out why my DNS stopped working despite my linode finally coming back up, and... now I know where to look.
- mnordhoff 1y ago
  Wait, Namecheap deactivates your domain name if you have a temporary DNS outage?
  - basilgohar 1y ago
    It seems it did not, but my glue records are no longer propagating. Still debugging until now.

30 points by mherrmann 1y ago

Is this hard downtime for anyone else? We're not sure what they mean by "emerging service issue" and they haven't replied to our support ticket yet.

upseo 1y ago
For us too, nothing works at all
mathrawka 1y ago
Yes, it has gotten worse as time progressed. Some k8s services started to fail, which is how I noticed something was wrong. Then the k8s control plane was up and down. Then k8s control plan completely done. Now I can't even connect to any of my non k8s servers over ssh.
dreeves 1y ago
Some signs of life from some of our servers now. Still no word from Linode.
basilgohar 1y ago
Yes, my servers and all associate services are down: DNS, email, websites. It's a major outage at the whole Newark datacenter, which is their main one, no less.
- basilgohar 1y ago
  Thankfully all my nodes are back up. But the DNS server as down so long that Namecheap registered it as a personal DNS server, so I am working with them to get it back up. These are secondary effects of outage, not Linode directly, but caused by the outage itself.
cmg 1y ago
I woke up to a few hundred messages from Icinga - thankfully my phone is on do-not-disturb overnight. Some of my servers in Newark are up and responding, some are not. Happy Sunday! Cleaning up the automatically-created maintenance/alert tickets generated by this is going to be a fun time.

dreeves 1y ago

More assurance that it's not just us: https://downdetector.com/status/linode/ https://downdetector.com/status/linode/ "Mal de muchos, consuela de tontos."

dreeves 1y ago

Encouraging update on the status page: "The issue is related to heating/cooling complications in the data center due to a power outage . The power outage has been fixed and we are working quickly to bring our services back online."

skbohra123 1y ago

We are also impacted. Services still down.

morpheous77 1y ago

Hey guys I also fall into this issue, have you ever heard about some compensation it's been about 8 hours and problem persists. Good day all

PierreJasmin 1y ago

It affects the company we use as online license server - lots of worried customers today here...

bobadob 1y ago

They said it started with a power outage, but I thought Linode data centers ran on UPS and didn't have power outages? And, given the length of this outage, the problem seems more serious.

basilgohar 1y ago
The power outage affected their HVAC system, so it seems to me like the entire DC had to be shutdown, so bringing everything back up from a black start seems to be the situation.

pupppet 1y ago

16 hours downtime now, yeesh.

morpheous77 1y ago
yup, I have one missing for booting...the most important

hecanjog 1y ago

Our instance just came back up.

ajdude 1y ago

I'm using a linode VPS as my mail server, this is hurting me badly.

basilgohar 1y ago
I have a Mail-in-a-Box instance that does DNS as well for all my sites. Still down....

chriskkim 1y ago

One of the three nodes came back online at 5PM EST, still waiting for the other two at 1AM EST ...

lgeek 1y ago

Over 22 hours of downtime for the one VPS I have in that region. My infrastructure is redundant and spread out among hosting providers and DCs so there's no real impact, but I'm pretty sure this is the longest outage I've ever had with any provider. And the communication level has been so dissapointing. 4 hrs to say it's a power / HVAC issue? Updates that basically just say we're still working on it since then.

ajdude 1y ago
We are approaching 24 hours of downtime, I'm still one of those also affected and I'm starting to wonder if the situation is worse than they are letting on.
lgeek 1y ago
Update: it's finally up again after around 25.5 hrs of downtime
- ajdude 1y ago
  Here's to hoping mine comes up again soon

haik90 1y ago

It is still ongoing and impacting multiple services and regions. Looking into the status page history, it seems they're facinga different issue on Kubernetes for the last couple of days

cmg 1y ago

Of 60-something Linodes in Newark across a few accounts (we don't use LKE, Node Balancers, etc) - Many came back up yesterday. Most of the rest came back up this morning. - All but two are back online. One of those is "Powered off" but can't be turned on because "Linode busy". The other is online but unreachable, same behavior as most of them during the outage. - Three required me to put them in Rescue Mode and run fsck.ext4 -F /dev/sda to get them back online.

basilgohar 1y ago

Thankfully all my nodes are back online and accessible via SSH but there are currently secondary effects due to the length of the outage that I am dealing with, such as my DNS server being deregistered with Namecheap from serving as a personal DNS server (their name for that feature). The consequence of this is also that my email is not currently work, since that affects the hostname the email server operates under serving multiple other domains. I am still grateful that it seems I suffered no data loss, thank God! Also, I've taken it upon myself to PROPERLY implement a better, redundant backup strategy (since I was mainly relying on Linode's service, but now I feel I should go beyond that). I am using restic backing-up to a Backblaze bucket via the S3 interface. Nice thing is I can put all hosts into the same bucket and restic will organize by host but still full deduplication support. Not sure how much that'll net me but it's nice to have.

ajdude 1y ago
I'm in the exact same boat- mail server hosted on linode with namecheap as my registrar. I couldn't figure out why my DNS stopped working despite my linode finally coming back up, and... now I know where to look.
mnordhoff 1y ago
Wait, Namecheap deactivates your domain name if you have a temporary DNS outage?
- basilgohar 1y ago
  It seems it did not, but my glue records are no longer propagating. Still debugging until now.