5 ms·

Fastmail is down

49 points by _jcwu 8y ago

zkms 8y ago
Looks like an issue with their upstream, nyi.net: traceroute to mail.fastmail.com (66.111.4.148), 30 hops max, 60 byte packets 1 192.168.1.1 (192.168.1.1) 2.582 ms 4.760 ms 4.748 ms 2 142.254.XXX.XXX (142.254.XXX.XXX) 14.748 ms 41.677 ms 43.837 ms 3 agg60.vnnzca2401h.socal.rr.com (76.167.27.73) 44.784 ms 47.498 ms 47.498 ms 4 agg11.vnnycajz01r.socal.rr.com (72.129.14.96) 47.489 ms 47.479 ms 47.499 ms 5 agg29.lsancarc01r.socal.rr.com (72.129.13.0) 47.475 ms 47.466 ms 47.452 ms 6 agg10.tustcaft01r.socal.rr.com (66.75.161.48) 47.458 ms 20.444 ms 19.780 ms 7 agg10.lsancarc01r.socal.rr.com (66.75.161.49) 24.220 ms 25.283 ms 25.654 ms 8 agg10.tustcaft01r.socal.rr.com (66.75.161.48) 27.438 ms 29.922 ms 29.924 ms 9 agg10.lsancarc01r.socal.rr.com (66.75.161.49) 32.877 ms 33.666 ms 34.208 ms 10 agg10.tustcaft01r.socal.rr.com (66.75.161.48) 35.399 ms 35.901 ms 36.878 ms 11 agg10.lsancarc01r.socal.rr.com (66.75.161.49) 34.177 ms 34.492 ms 34.462 ms 12 agg10.tustcaft01r.socal.rr.com (66.75.161.48) 31.085 ms 28.907 ms 30.610 ms 13 agg10.lsancarc01r.socal.rr.com (66.75.161.49) 27.499 ms 35.354 ms 36.167 ms 14 agg10.tustcaft01r.socal.rr.com (66.75.161.48) 38.251 ms 39.675 ms 39.303 ms 15 agg10.lsancarc01r.socal.rr.com (66.75.161.49) 36.657 ms 36.652 ms 38.186 ms 16 agg10.tustcaft01r.socal.rr.com (66.75.161.48) 39.633 ms 46.706 ms 45.702 ms 17 agg10.lsancarc01r.socal.rr.com (66.75.161.49) 43.710 ms 34.548 ms 42.508 ms 18 agg10.tustcaft01r.socal.rr.com (66.75.161.48) 42.399 ms 38.360 ms 39.058 ms 19 agg10.lsancarc01r.socal.rr.com (66.75.161.49) 41.678 ms 43.000 ms 44.069 ms 20 agg10.tustcaft01r.socal.rr.com (66.75.161.48) 44.913 ms 45.806 ms 46.209 ms 21 agg10.lsancarc01r.socal.rr.com (66.75.161.49) 51.609 ms 50.484 ms 51.050 ms 22 agg10.tustcaft01r.socal.rr.com (66.75.161.48) 53.087 ms 55.045 ms 39.316 ms 23 agg10.lsancarc01r.socal.rr.com (66.75.161.49) 52.467 ms 52.470 ms 52.450 ms 24 agg10.tustcaft01r.socal.rr.com (66.75.161.48) 53.596 ms 43.254 ms 46.939 ms 25 agg10.lsancarc01r.socal.rr.com (66.75.161.49) 51.514 ms 53.067 ms 54.051 ms 26 agg10.tustcaft01r.socal.rr.com (66.75.161.48) 61.721 ms 61.724 ms 61.715 ms 27 agg10.lsancarc01r.socal.rr.com (66.75.161.49) 54.020 ms 54.011 ms 54.003 ms 28 agg10.tustcaft01r.socal.rr.com (66.75.161.48) 51.628 ms 53.483 ms 53.482 ms 29 agg10.lsancarc01r.socal.rr.com (66.75.161.49) 56.560 ms 57.956 ms 58.392 ms 30 agg10.tustcaft01r.socal.rr.com (66.75.161.48) 57.149 ms 58.921 ms 57.447 ms
- graystevens 8y ago
  Matches up with their current Twitter thread: We're experiencing a network outage. Stay tuned at https://www.fastmailstatus.com https://www.fastmailstatus.com . We're in close communication with our provider trying to restore service. 1:34 AM - 1 Aug 2018 https://twitter.com/FastMail/status/1024573967393157121 https://twitter.com/FastMail/status/1024573967393157121
  - chrisper 8y ago
    I find it a bit concerning that they don't have a backup provider.
    - zkms 8y ago
      Yeah I'd have expected them to have multiple upstreams and an ASN to announce their IPs but apparently they don't. Pretty odd.
      - regecks 8y ago
        They do have AS redundancy, but only for their MX.
      - toomuchtodo 8y ago
        Diminishing returns. This is the first time I’ve ever heard of a FastMail outage, so it seems their architecture is commercially reasonable as it relates to outages.
chimeracoder 8y ago
Their status page says > Weâre experiencing network issues with several providers. Get updates on fastmailstatus.com as the situation progresses. (including the quirks characters, copied directly). That implies the outage isn't universal, and some ISPs/locations may not be affected. Is anybody able to access through a particular VPN (or country)? If so, posting that here could help others get access to their inboxes faster.
- regecks 8y ago
  Yeah, Linode Newark is an unaffected vantage point. Same with Vultr, but Digital Ocean is no go. Since the MX are spread between NYI/AS11403 and AS201628, I think that all mail delivery should still be fine. I'm receiving occasional push notifications on my iOS FastMail app, can't actually read them though!
- kornakiewicz 8y ago
  I randomly checked few servers using NordVPN and was able to connect via Russia.
  - deleted 8y ago
    [deleted]
- yamrzou 8y ago
  I was able to connect via Melbourne, Asutralia through Mullvad VPN.
kaptain 8y ago
Thanks for posting this. I live in China and thought Fastmail was finally being blocked by the GFW. Over the past year I've had intermittent issues with connectivity and I've wondered if FM's time in China was up. Are there any HN readers in China that have experienced similar issues with FM?
brongondwana 8y ago
Core router issue at NYI - apparently it's not just us, though we're one of their major non-US customer-base customers, so it affects a lot of our users. I spoke to their techs briefly to confirm details, but otherwise am leaving them alone to do their jobs! It's particularly annoying that it's a partial outage and our main 66.111.4.x range is the most affected (I suspect because it runs through DDoS protection that our more hidden ranges don't) so most our monitoring is running just fine, but customers can't get in. And it would take an hour (thanks "sensible" DNS timeouts) to move the traffic with the knobs we have access to.
graystevens 8y ago
Top work Fastmail team! Good news! Mail access has been restored for users on some major networks. We apologize to those customers who still cannot reach their Inboxes - we're working to restore full access as fast as we can. No mail has been lost during this downtime. https://www.fastmailstatus.com https://www.fastmailstatus.com 3:09 AM - 1 Aug 2018 https://twitter.com/FastMail/status/1024598033793601536 https://twitter.com/FastMail/status/1024598033793601536
brongondwana 8y ago
And... we appear to be back :) Far out. Stay tuned for a blog post about this later. No data lost, and the servers themselves have been happily humming along the entire time, so we don't even have filesystem checks to worry about.
- zkms 8y ago
  > And... we appear to be back :) Far out. Stay tuned for a blog post about this later. Congrats and thanks, good luck with the search for another upstream or two :)

49 points by _jcwu 8y ago

zkms 8y ago

Looks like an issue with their upstream, nyi.net: traceroute to mail.fastmail.com (66.111.4.148), 30 hops max, 60 byte packets 1 192.168.1.1 (192.168.1.1) 2.582 ms 4.760 ms 4.748 ms 2 142.254.XXX.XXX (142.254.XXX.XXX) 14.748 ms 41.677 ms 43.837 ms 3 agg60.vnnzca2401h.socal.rr.com (76.167.27.73) 44.784 ms 47.498 ms 47.498 ms 4 agg11.vnnycajz01r.socal.rr.com (72.129.14.96) 47.489 ms 47.479 ms 47.499 ms 5 agg29.lsancarc01r.socal.rr.com (72.129.13.0) 47.475 ms 47.466 ms 47.452 ms 6 agg10.tustcaft01r.socal.rr.com (66.75.161.48) 47.458 ms 20.444 ms 19.780 ms 7 agg10.lsancarc01r.socal.rr.com (66.75.161.49) 24.220 ms 25.283 ms 25.654 ms 8 agg10.tustcaft01r.socal.rr.com (66.75.161.48) 27.438 ms 29.922 ms 29.924 ms 9 agg10.lsancarc01r.socal.rr.com (66.75.161.49) 32.877 ms 33.666 ms 34.208 ms 10 agg10.tustcaft01r.socal.rr.com (66.75.161.48) 35.399 ms 35.901 ms 36.878 ms 11 agg10.lsancarc01r.socal.rr.com (66.75.161.49) 34.177 ms 34.492 ms 34.462 ms 12 agg10.tustcaft01r.socal.rr.com (66.75.161.48) 31.085 ms 28.907 ms 30.610 ms 13 agg10.lsancarc01r.socal.rr.com (66.75.161.49) 27.499 ms 35.354 ms 36.167 ms 14 agg10.tustcaft01r.socal.rr.com (66.75.161.48) 38.251 ms 39.675 ms 39.303 ms 15 agg10.lsancarc01r.socal.rr.com (66.75.161.49) 36.657 ms 36.652 ms 38.186 ms 16 agg10.tustcaft01r.socal.rr.com (66.75.161.48) 39.633 ms 46.706 ms 45.702 ms 17 agg10.lsancarc01r.socal.rr.com (66.75.161.49) 43.710 ms 34.548 ms 42.508 ms 18 agg10.tustcaft01r.socal.rr.com (66.75.161.48) 42.399 ms 38.360 ms 39.058 ms 19 agg10.lsancarc01r.socal.rr.com (66.75.161.49) 41.678 ms 43.000 ms 44.069 ms 20 agg10.tustcaft01r.socal.rr.com (66.75.161.48) 44.913 ms 45.806 ms 46.209 ms 21 agg10.lsancarc01r.socal.rr.com (66.75.161.49) 51.609 ms 50.484 ms 51.050 ms 22 agg10.tustcaft01r.socal.rr.com (66.75.161.48) 53.087 ms 55.045 ms 39.316 ms 23 agg10.lsancarc01r.socal.rr.com (66.75.161.49) 52.467 ms 52.470 ms 52.450 ms 24 agg10.tustcaft01r.socal.rr.com (66.75.161.48) 53.596 ms 43.254 ms 46.939 ms 25 agg10.lsancarc01r.socal.rr.com (66.75.161.49) 51.514 ms 53.067 ms 54.051 ms 26 agg10.tustcaft01r.socal.rr.com (66.75.161.48) 61.721 ms 61.724 ms 61.715 ms 27 agg10.lsancarc01r.socal.rr.com (66.75.161.49) 54.020 ms 54.011 ms 54.003 ms 28 agg10.tustcaft01r.socal.rr.com (66.75.161.48) 51.628 ms 53.483 ms 53.482 ms 29 agg10.lsancarc01r.socal.rr.com (66.75.161.49) 56.560 ms 57.956 ms 58.392 ms 30 agg10.tustcaft01r.socal.rr.com (66.75.161.48) 57.149 ms 58.921 ms 57.447 ms

graystevens 8y ago
Matches up with their current Twitter thread: We're experiencing a network outage. Stay tuned at https://www.fastmailstatus.com https://www.fastmailstatus.com . We're in close communication with our provider trying to restore service. 1:34 AM - 1 Aug 2018 https://twitter.com/FastMail/status/1024573967393157121 https://twitter.com/FastMail/status/1024573967393157121
- chrisper 8y ago
  I find it a bit concerning that they don't have a backup provider.
  - zkms 8y ago
    Yeah I'd have expected them to have multiple upstreams and an ASN to announce their IPs but apparently they don't. Pretty odd.
    - regecks 8y ago
      They do have AS redundancy, but only for their MX.
    - toomuchtodo 8y ago
      Diminishing returns. This is the first time I’ve ever heard of a FastMail outage, so it seems their architecture is commercially reasonable as it relates to outages.

chimeracoder 8y ago

Their status page says > Weâre experiencing network issues with several providers. Get updates on fastmailstatus.com as the situation progresses. (including the quirks characters, copied directly). That implies the outage isn't universal, and some ISPs/locations may not be affected. Is anybody able to access through a particular VPN (or country)? If so, posting that here could help others get access to their inboxes faster.

regecks 8y ago
Yeah, Linode Newark is an unaffected vantage point. Same with Vultr, but Digital Ocean is no go. Since the MX are spread between NYI/AS11403 and AS201628, I think that all mail delivery should still be fine. I'm receiving occasional push notifications on my iOS FastMail app, can't actually read them though!
kornakiewicz 8y ago
I randomly checked few servers using NordVPN and was able to connect via Russia.
- deleted 8y ago
  [deleted]
yamrzou 8y ago
I was able to connect via Melbourne, Asutralia through Mullvad VPN.

kaptain 8y ago

Thanks for posting this. I live in China and thought Fastmail was finally being blocked by the GFW. Over the past year I've had intermittent issues with connectivity and I've wondered if FM's time in China was up. Are there any HN readers in China that have experienced similar issues with FM?

brongondwana 8y ago

Core router issue at NYI - apparently it's not just us, though we're one of their major non-US customer-base customers, so it affects a lot of our users. I spoke to their techs briefly to confirm details, but otherwise am leaving them alone to do their jobs! It's particularly annoying that it's a partial outage and our main 66.111.4.x range is the most affected (I suspect because it runs through DDoS protection that our more hidden ranges don't) so most our monitoring is running just fine, but customers can't get in. And it would take an hour (thanks "sensible" DNS timeouts) to move the traffic with the knobs we have access to.

graystevens 8y ago

Top work Fastmail team! Good news! Mail access has been restored for users on some major networks. We apologize to those customers who still cannot reach their Inboxes - we're working to restore full access as fast as we can. No mail has been lost during this downtime. https://www.fastmailstatus.com https://www.fastmailstatus.com 3:09 AM - 1 Aug 2018 https://twitter.com/FastMail/status/1024598033793601536 https://twitter.com/FastMail/status/1024598033793601536

brongondwana 8y ago

And... we appear to be back :) Far out. Stay tuned for a blog post about this later. No data lost, and the servers themselves have been happily humming along the entire time, so we don't even have filesystem checks to worry about.

zkms 8y ago
> And... we appear to be back :) Far out. Stay tuned for a blog post about this later. Congrats and thanks, good luck with the search for another upstream or two :)