5 ms·

Digital Ocean's network is down?

30 points by komrade 13y ago

My sites are down and I cannot sign in to their site.

mikecane 13y ago
I know WordPress dotcom uses Amazon S3 storage for images, so my blog isn't showing any due to this outage. Amazon apparently also uses its own S3 for images because right now at Amazon none of the product images are showing.
ddorian43 13y ago
It took a while(5seconds) but it opened for me. Next time check: http://www.digitaloceanstatus.com/ http://www.digitaloceanstatus.com/
- entropyneur 13y ago
  > Next time check: http://www.digitaloceanstatus.com/ http://www.digitaloceanstatus.com/ Huh... Not the best source really. It took >30 minutes between I got an alert from the monitoring service and an update appearing on the status site.
  - cobrabyte 13y ago
    Same here. Pingdom let me know one of my sites was down roughly 20 minutes before DO added their notification to the status page. Unfortunately, it appears all of my DO instances are in the NY1 facility. Bad on me.
zarvox 13y ago
As of this posting, https://www.digitalocean.com/login https://www.digitalocean.com/login is returning CloudFlare's "Always Online" cache. I just got woken up by two of our boxes being unreachable, but one of my other boxes is up, so maybe it's just affecting one of their datacenters?
mikecane 13y ago
There seems to be an issue with the entire Net this morning. Drudge Report unreachable. Same for Twitter. And WordPress dotcom's link to their Amazon S3 image storage is broken too.
- xfax 13y ago
  Damn. Looking forward to reading CloudFlare's post-mortem blog post on this. Seems to be widespread.
acesubido 13y ago
tumblr seems to be down on my side
- easy_rider 13y ago
  Confirmed, down in NL
- th0br0 13y ago
  can confirm that from germany
  - keithpeter 13y ago
    ...and UK.
    - petercooper 13y ago
      Loading okay here from the UK. I'm getting a feeling from these threads here that there's some serious routing trouble going down rather than service outages as such.
      - keithpeter 13y ago
        Was able to post to tumblr blog from Blackberry but not able to see page (on Orange UK broadband) until just now. Supports routing hypothesis I suppose.
- mikecane 13y ago
  I can't reach Tumblr either, or Toogles (YouTube search site).
  - easy_rider 13y ago
    Contrary to Tumblr, Toogles works from Amsterdam, NL.
- deleted 13y ago
  [deleted]
petercooper 13y ago
My DO based services, which are monitored by Pingdom, are all still showing as up. Internet Traffic Report is showing several routers as down in North America however: http://www.internettrafficreport.com/namerica.htm http://www.internettrafficreport.com/namerica.htm
- lgierth 13y ago
  A couple of routers in Europe are down as well.
- mikecane 13y ago
  Wow, if I'm counting correctly, eleven routers down. WTF happened? Is this a cyber attack?
  - elliottcarlson 13y ago
    That's insignificant - there are always routers down, and the 24 hour traffic index is stable.
njpatel 13y ago
Can ssh into some of the instances (and from there contact all the instances internally), but http/https ports definitely don't work. I think it's only NY1. I realised, it would be great for cloudflare to optionally allow us to show a better 'down' page for non-static sites like dashboards etc. Right now it's showing the same page with a bar on top (and obviously login etc won't work), ideally it would show a static page where we could explain something is wrong, without login boxes etc.
zarvox 13y ago
Update from https://status.digitalocean.com/ https://status.digitalocean.com/ : Networking Issue in NY1 Facility At this time, we are currently investigating an issue in our NY1 facility. We are working to determine the exact cause/resolution of the issue and will provide details as soon as they become available.
JohnTHaller 13y ago
One of the backbones is having issues. I think over in Level 3 land, again. I'm on Time Warner and unable to reach anything at Rackspace, Facebook, Hotmail, etc.
- elliottcarlson 13y ago
  Level3 has been having a lot of intermittent outages recently..
tehwalrus 13y ago
all the things people are reporting down are working fine for me from the UK. EDIT: except, I don't have a Digital Ocean NY instance I can try. Those might be down, but their website's fine.
tgeek 13y ago
everyone saying they can't hit certain sites(images on Wordpress/amazon/etc) is actually seeing the affect of their ISP's networks and the CDNs those sites use. Lotta Level3 issues right now. CDNs are often peered as close to the major ISP hubs as possible for speed reasons, but it can unfortunately result in these kind of chained outages. Since a lot of companies pump full site traffic through CDNs, it can appear as them being down.
JohnTHaller 13y ago
It's an issue with Level 3. Details here: https://news.ycombinator.com/item?id=6576399 https://news.ycombinator.com/item?id=6576399
ollydbg 13y ago
Looks like DO Europe nodes are up.

30 points by komrade 13y ago

My sites are down and I cannot sign in to their site.

I know WordPress dotcom uses Amazon S3 storage for images, so my blog isn't showing any due to this outage. Amazon apparently also uses its own S3 for images because right now at Amazon none of the product images are showing.

ddorian43 13y ago

It took a while(5seconds) but it opened for me. Next time check: http://www.digitaloceanstatus.com/ http://www.digitaloceanstatus.com/

entropyneur 13y ago
> Next time check: http://www.digitaloceanstatus.com/ http://www.digitaloceanstatus.com/ Huh... Not the best source really. It took >30 minutes between I got an alert from the monitoring service and an update appearing on the status site.
- cobrabyte 13y ago
  Same here. Pingdom let me know one of my sites was down roughly 20 minutes before DO added their notification to the status page. Unfortunately, it appears all of my DO instances are in the NY1 facility. Bad on me.

zarvox 13y ago

As of this posting, https://www.digitalocean.com/login https://www.digitalocean.com/login is returning CloudFlare's "Always Online" cache. I just got woken up by two of our boxes being unreachable, but one of my other boxes is up, so maybe it's just affecting one of their datacenters?

mikecane 13y ago

There seems to be an issue with the entire Net this morning. Drudge Report unreachable. Same for Twitter. And WordPress dotcom's link to their Amazon S3 image storage is broken too.

xfax 13y ago
Damn. Looking forward to reading CloudFlare's post-mortem blog post on this. Seems to be widespread.

acesubido 13y ago

tumblr seems to be down on my side

easy_rider 13y ago
Confirmed, down in NL
th0br0 13y ago
can confirm that from germany
- keithpeter 13y ago
  ...and UK.
  - petercooper 13y ago
    Loading okay here from the UK. I'm getting a feeling from these threads here that there's some serious routing trouble going down rather than service outages as such.
    - keithpeter 13y ago
      Was able to post to tumblr blog from Blackberry but not able to see page (on Orange UK broadband) until just now. Supports routing hypothesis I suppose.
mikecane 13y ago
I can't reach Tumblr either, or Toogles (YouTube search site).
- easy_rider 13y ago
  Contrary to Tumblr, Toogles works from Amsterdam, NL.
deleted 13y ago
[deleted]

petercooper 13y ago

My DO based services, which are monitored by Pingdom, are all still showing as up. Internet Traffic Report is showing several routers as down in North America however: http://www.internettrafficreport.com/namerica.htm http://www.internettrafficreport.com/namerica.htm

lgierth 13y ago
A couple of routers in Europe are down as well.
mikecane 13y ago
Wow, if I'm counting correctly, eleven routers down. WTF happened? Is this a cyber attack?
- elliottcarlson 13y ago
  That's insignificant - there are always routers down, and the 24 hour traffic index is stable.

njpatel 13y ago

Can ssh into some of the instances (and from there contact all the instances internally), but http/https ports definitely don't work. I think it's only NY1. I realised, it would be great for cloudflare to optionally allow us to show a better 'down' page for non-static sites like dashboards etc. Right now it's showing the same page with a bar on top (and obviously login etc won't work), ideally it would show a static page where we could explain something is wrong, without login boxes etc.

zarvox 13y ago

Update from https://status.digitalocean.com/ https://status.digitalocean.com/ : Networking Issue in NY1 Facility At this time, we are currently investigating an issue in our NY1 facility. We are working to determine the exact cause/resolution of the issue and will provide details as soon as they become available.

JohnTHaller 13y ago

One of the backbones is having issues. I think over in Level 3 land, again. I'm on Time Warner and unable to reach anything at Rackspace, Facebook, Hotmail, etc.

elliottcarlson 13y ago
Level3 has been having a lot of intermittent outages recently..

tehwalrus 13y ago

all the things people are reporting down are working fine for me from the UK. EDIT: except, I don't have a Digital Ocean NY instance I can try. Those might be down, but their website's fine.

tgeek 13y ago

everyone saying they can't hit certain sites(images on Wordpress/amazon/etc) is actually seeing the affect of their ISP's networks and the CDNs those sites use. Lotta Level3 issues right now. CDNs are often peered as close to the major ISP hubs as possible for speed reasons, but it can unfortunately result in these kind of chained outages. Since a lot of companies pump full site traffic through CDNs, it can appear as them being down.

JohnTHaller 13y ago

It's an issue with Level 3. Details here: https://news.ycombinator.com/item?id=6576399 https://news.ycombinator.com/item?id=6576399

ollydbg 13y ago

Looks like DO Europe nodes are up.