7 ms·

Google Apps Services Were Down

174 points by laumac 13y ago

brokentone 13y ago
Still getting errors sending transactional messages to users on gmail Apr 17 10:23:30 postfix/smtp: connect to gmail-smtp-in.l.google.com[2607:f8b0:4002:c02::1a]:25: Network is unreachable
minimax 13y ago
This is the message: "Sorry, there seems to be a problem. The service you're looking for is temporarily unavailable. We're working hard to restore your access as soon as possible. Please try again in a few hours. Thanks for your patience." 0900 CDT edit: works for me now.
- viame 13y ago
  Getting the same message. However, this is happening to some of my logins only. Same with e-mails, some of them are working fine and some of them are timing out or asking for passwords. For a second I thought someone hacked my e-mail. :>
  - g-garron 13y ago
    Me too, I was really scared. Now I know is Google's problem I am breading again.
    - kaolinite 13y ago
      Same.. I was the only person in the company unable to access emails. Was scared I was about to be called into a boardroom ;-)
    - laumars 13y ago
      > Me too, I was really scared. Now I know is Google's problem I am breading again. I hate it when Google get in the way of my sandwich making ;-)
      - deleted 13y ago
        [deleted]
      - Samuel_Michon 13y ago
        Given the naming scheme of Android releases, it seems like Google would let you eat cake instead.
- saurik 13y ago
  That error message seems to just be Google's default 500 error message page; even when it is an API request that fails, even if the API is specified to return errors as some kind of JSON or XML, you get that HTML page in response. (I know this as Google's OpenID+OAuth pipeline has been failing occassionally for the last month, something I've been trying to get them to fix on the Google Federated Login API mailing list.)
- infoman 13y ago
  so many developers hacking on google glass apps?
terryjsmith 13y ago
I'm getting IMAP logins rejected as well.
eggnet 13y ago
down for me too
runemadsen 13y ago
I'm also getting errors when trying to retrieve Gmail messages in IOS Mail. "Wrong username or password", but it works in a browser.
appleflaxen 13y ago
Since you always worry you are going crazy when a high-uptime site is down: mine too FWIW: I was able to log in by going to www.gmail.com and logging in suing myusername@mydomain.com as the username.
- bkanber 13y ago
  Tried this -- didn't work for me unfortunately.
deleted 13y ago
[deleted]
deleted 13y ago
[deleted]
bdg 13y ago
This is happening for gmail for some accounts, but mine seem to be running. Seems the outage is per user account.
js2 13y ago
It's intermittent for me. I first noticed trouble when the Android gmail app said it couldn't connect for one of my accounts. I cannot login to my apps domain control panel right now, but I can access its email via both IMAP and the web interface. However, another account on that same domain cannot access via IMAP.
Kristories 13y ago
Already discussed here https://news.ycombinator.com/item?id=5563949 https://news.ycombinator.com/item?id=5563949 Service disruption http://www.google.com/appsstatus#hl=en&v=issue&ts=1366217999000&iid=bb886daf880ab16f9a8b5f8bbe2f34b0 http://www.google.com/appsstatus#hl=en&v=issue&ts=13...
josephb 13y ago
Nearly 30 minutes since it started and the status page is it's customary blank screen. http://imgur.com/OLyGf0O http://imgur.com/OLyGf0O Almost as bad as AWS at getting notices up.
- deleted 13y ago
  [deleted]
- WimLeers 13y ago
  Odd, "Admin control panel/API" is marked as disrupted for me.
  - josephb 13y ago
    Yeah it is now, only took 30 minutes :-) The outage is affecting IMAP/POP/HTTP for email plus Gchat, the admin control panel/API is probably the least impact.
- guilhermetk 13y ago
  Now it is showing Service disruption for Admin control panel / API http://www.google.com/appsstatus#hl=en&v=status&ts=1366201987843 http://www.google.com/appsstatus#hl=en&v=status&ts=1...
theg2 13y ago
Was having problems with play developers console yesterday, suddenly it was a 404. Seems like they're planning something in my opinion.
jschuur 13y ago
Google Drive is down for me on Apps too.
delinka 13y ago
I also can't authorize an app to have access to a Gmail account. Google's auth page presents an error.
- traeregan 13y ago
  Same here. Intermittent though. I'm like 2 for 5.
glennos 13y ago
The dashboard hasn't caught up. Admin Control Panel is one thing, but GMail/Apps has a huge impact!
mattvot 13y ago
I did wonder about this. My phone said my username or password was wrong for my email, got a bit worried, so I tried with their webmail and it worked there.
frankydp 13y ago
Webmaster tools also down with auth failure.
bkanber 13y ago
Down here. Additionally, my iCal won't sync with goocal.
yaskyj 13y ago
It's interesting that I kind find news for this on google itself, but Hacker News already has two posts.
JosephRedfern 13y ago
Getting IMAP rejections, and Google Docs doesn't seem to be working: http://cl.ly/image/3W2e0X1O1i0o http://cl.ly/image/3W2e0X1O1i0o
- wodow 13y ago
  smtp.googlemail.com also failing (edit: intermittently)
- fish2000 13y ago
  Crazy – my IMAP client (Apple Mail) has no problem until it tries to send via SMTP… when it chokes. Also, no working gmail for all my accounts except my @gmail.com address – at least fail consistently, you Googlers. Blech!
varun1308 13y ago
Yeah.. my pop mail stopped working too
deleted 13y ago
[deleted]
ac132 13y ago
At least Google Reader still works :-)
arcosdev 13y ago
Faaack. I thought I was just having a local connection issue. Same problem here.
silasb 13y ago
Appears Groups is down also.
kyrra 13y ago
Oddly enough, gmail itself works on my Google Apps domain. But accessing my account page does not work. As well, I can't log into Gchat using one of the generated passwords (says password rejected). I'm guessing this is the same reason IMAP is failing for others.
TranceMan 13y ago
Google Apps status page: http://www.google.com/appsstatus http://www.google.com/appsstatus
- deleted 13y ago
  [deleted]
X-Istence 13y ago
That explains why I can't get my email =).
calleskonto 13y ago
The provision api is still fine doh.
9k9 13y ago
They probably don't know what to do
freehunter 13y ago
I'm wondering if this is related to the new update they did to the admin control panel. http://arstechnica.com/information-technology/2013/04/make-google-chrome-do-your-bidding-with-new-google-apps-control-panel/ http://arstechnica.com/information-technology/2013/04/make-g...
- corresation 13y ago
  There is a massively linked Boston bomber Google Docs spreadsheet making the rounds -- the last time a viral Google doc saw that sort of exposure, the service went down.
ndespres 13y ago
I'm a Google Apps reseller and thought, oh no, I should call all my clients and let them know about this outage before they call me! Where can I get a full customer list.. the admin panel.. oh.
- lewispb 13y ago
  The phone calls from my clients have started to come in already...!
- stedaniels 13y ago
  Your only copy of your customers, is in a third party reseller control panel?
  - ndespres 13y ago
    Of course not. Just the fastest way to get them! Fortunately, Confluence, Autotask, and my memory are all still intact!
    - mathattack 13y ago
      "and my memory" When all else fails, this is the true old school solution. :-)
  - deleted 13y ago
    [deleted]
- jqueryin 13y ago
  There are API endpoints for that. Not sure if they're affected.
paulmolluzzo 13y ago
It's back.
misnome 13y ago
Well, their email services are still working, because I got a message about our trial ending today and that we need to start paying. I hope this comes up before we get automatically cut off..
digitalpbk 13y ago
digitalpbk$ nslookup smtp.google.com Server: 8.8.8.8 Address: 8.8.8.8#53 server can't find smtp.google.com: NXDOMAIN SMTP servers are down for me. Anyone else?
- brokentone 13y ago
  NXDOMAIN wouldn't be server down, that's no DNS record. Still won't work, but different (much bigger) issue.
  - tekacs 13y ago
    I'm pretty sure the 'bigger' issue here is that it's smtp.gmail.com - smtp.google.com is, I believe, always down on account of not existing. :P
    - pyre 13y ago
      Or: smtp.googlemail.com
    - deno 13y ago
      Works fine for me: > dig +short @8.8.8.8 smtp.gmail.com gmail-smtp-msa.l.google.com. 74.125.136.108 74.125.136.109 http://homepage.ntlworld.com/jonathan.deboynepollard/FGA/nslookup-flaws.html http://homepage.ntlworld.com/jonathan.deboynepollard/FGA/nsl...
- duncans 13y ago
  try `nslookup -querytype=MX google.com`
level09 13y ago
not only that, seems like pop3 is also facing issues ..
jlengrand 13y ago
I got a 500 when I try to access our Drive too. It's been an hour now. Using the business google services.
abhi_21 13y ago
Now its working..
EvanAnderson 13y ago
SAML logons are failing, too.
Leszek 13y ago
This explains the SMTP problems I was having, I was worried someone had hacked my account or something...
meerita 13y ago
I cannot access my email from Sparrow app too.
dotbill 13y ago
As I said in the other google apps thread on hackernews.. I thought I'd give them a call to find out whats going on, however to call them I need my support/admin pin - which is only accessible from the admin portal and without it they won't answer the phone. Nice.
AndrewDucker 13y ago
The Register reports that GMail went down exactly a year ago: http://www.theregister.co.uk/2012/04/18/google_gmail_outage/ http://www.theregister.co.uk/2012/04/18/google_gmail_outage/
- saurik 13y ago
  Huffington Post reports that Gmail also went offline on April 29th of 2010. http://www.huffingtonpost.com/2010/04/29/gmail-down-check-gmail-ou_n_557351.html http://www.huffingtonpost.com/2010/04/29/gmail-down-check-gm... Google reports that Gmail also went offline on April 24th of 2009. http://googleblog.blogspot.com/2009/02/current-gmail-outage.html http://googleblog.blogspot.com/2009/02/current-gmail-outage.... (Sadly, the partial Gmail outage of 2011 seemed to be February 27th, or this would be an epic pattern of "don't trust Gmail during the end of April" ;P.) http://www.eweek.com/c/a/Messaging-and-Collaboration/Google-Suffers-First-Gmail-Outage-of-2011-850632/ http://www.eweek.com/c/a/Messaging-and-Collaboration/Google-...
Selfcommit 13y ago
And We're up.
logn 13y ago
I don't get why apps and all sorts of other google sites go down a lot, but search never has in my experience. Why is that? Is search just simpler? Does it fail in non-obvious ways? Enormous redundancy/fail-over on search?
- therealarmen 13y ago
  Contributing the vast majority of revenue for the company has something to do with it.
- bdonlan 13y ago
  Search is (mostly) stateless. They can copy the search database to hundreds of DCs around the world, and each can operate independently - so if one of them has a problem, it's not likely to affect the others (unless it's bad data, but that's why you shouldn't deploy index updates to all DCs simultaneously). There's some personalization, sure, but you can just fall back to non-personalized search if that breaks. Things like gmail, on the other hand, are inherently stateful. When you log into gmail, you have to eventually connect to one system that maintains your mailbox. Sure, there might be replication - but the replicas are all talking to each other. It's surprisingly easy to have a cascade failure in a system like this, where one of the replicas going down triggers (directly or indirectly) all the others failing as well. Or you can have some bad data that gets replicated out, and then proceeds to confuse everything that's looking at it - unlike search, you have to replicate that data immediately, and don't get to enjoy the benefits of a staged deployment. This also explains why not all users were affected - I'd guess that their system is divided into some number of shards, and users are assigned to a particular shard. That 0.07% of users affected probably represents a single unhealthy shard.
ckdarby 13y ago
Why do people report this kind of stuff on hacker news. I am sure everyone knows how to google for a site's status page.

174 points by laumac 13y ago

Still getting errors sending transactional messages to users on gmail Apr 17 10:23:30 postfix/smtp: connect to gmail-smtp-in.l.google.com[2607:f8b0:4002:c02::1a]:25: Network is unreachable

minimax 13y ago

This is the message: "Sorry, there seems to be a problem. The service you're looking for is temporarily unavailable. We're working hard to restore your access as soon as possible. Please try again in a few hours. Thanks for your patience." 0900 CDT edit: works for me now.

viame 13y ago
Getting the same message. However, this is happening to some of my logins only. Same with e-mails, some of them are working fine and some of them are timing out or asking for passwords. For a second I thought someone hacked my e-mail. :>
- g-garron 13y ago
  Me too, I was really scared. Now I know is Google's problem I am breading again.
  - kaolinite 13y ago
    Same.. I was the only person in the company unable to access emails. Was scared I was about to be called into a boardroom ;-)
  - laumars 13y ago
    > Me too, I was really scared. Now I know is Google's problem I am breading again. I hate it when Google get in the way of my sandwich making ;-)
    - deleted 13y ago
      [deleted]
    - Samuel_Michon 13y ago
      Given the naming scheme of Android releases, it seems like Google would let you eat cake instead.
saurik 13y ago
That error message seems to just be Google's default 500 error message page; even when it is an API request that fails, even if the API is specified to return errors as some kind of JSON or XML, you get that HTML page in response. (I know this as Google's OpenID+OAuth pipeline has been failing occassionally for the last month, something I've been trying to get them to fix on the Google Federated Login API mailing list.)
infoman 13y ago
so many developers hacking on google glass apps?

terryjsmith 13y ago

I'm getting IMAP logins rejected as well.

eggnet 13y ago

down for me too

runemadsen 13y ago

I'm also getting errors when trying to retrieve Gmail messages in IOS Mail. "Wrong username or password", but it works in a browser.

appleflaxen 13y ago

Since you always worry you are going crazy when a high-uptime site is down: mine too FWIW: I was able to log in by going to www.gmail.com and logging in suing myusername@mydomain.com as the username.

bkanber 13y ago
Tried this -- didn't work for me unfortunately.

deleted 13y ago

[deleted]

deleted 13y ago

[deleted]

bdg 13y ago

This is happening for gmail for some accounts, but mine seem to be running. Seems the outage is per user account.

js2 13y ago

It's intermittent for me. I first noticed trouble when the Android gmail app said it couldn't connect for one of my accounts. I cannot login to my apps domain control panel right now, but I can access its email via both IMAP and the web interface. However, another account on that same domain cannot access via IMAP.

Kristories 13y ago

Already discussed here https://news.ycombinator.com/item?id=5563949 https://news.ycombinator.com/item?id=5563949 Service disruption http://www.google.com/appsstatus#hl=en&v=issue&ts=1366217999000&iid=bb886daf880ab16f9a8b5f8bbe2f34b0 http://www.google.com/appsstatus#hl=en&v=issue&ts=13...

josephb 13y ago

Nearly 30 minutes since it started and the status page is it's customary blank screen. http://imgur.com/OLyGf0O http://imgur.com/OLyGf0O Almost as bad as AWS at getting notices up.

deleted 13y ago
[deleted]
WimLeers 13y ago
Odd, "Admin control panel/API" is marked as disrupted for me.
- josephb 13y ago
  Yeah it is now, only took 30 minutes :-) The outage is affecting IMAP/POP/HTTP for email plus Gchat, the admin control panel/API is probably the least impact.
guilhermetk 13y ago
Now it is showing Service disruption for Admin control panel / API http://www.google.com/appsstatus#hl=en&v=status&ts=1366201987843 http://www.google.com/appsstatus#hl=en&v=status&ts=1...

theg2 13y ago

Was having problems with play developers console yesterday, suddenly it was a 404. Seems like they're planning something in my opinion.

jschuur 13y ago

Google Drive is down for me on Apps too.

delinka 13y ago

I also can't authorize an app to have access to a Gmail account. Google's auth page presents an error.

traeregan 13y ago
Same here. Intermittent though. I'm like 2 for 5.

glennos 13y ago

The dashboard hasn't caught up. Admin Control Panel is one thing, but GMail/Apps has a huge impact!

mattvot 13y ago

I did wonder about this. My phone said my username or password was wrong for my email, got a bit worried, so I tried with their webmail and it worked there.

frankydp 13y ago

Webmaster tools also down with auth failure.

bkanber 13y ago

Down here. Additionally, my iCal won't sync with goocal.

yaskyj 13y ago

It's interesting that I kind find news for this on google itself, but Hacker News already has two posts.

JosephRedfern 13y ago

Getting IMAP rejections, and Google Docs doesn't seem to be working: http://cl.ly/image/3W2e0X1O1i0o http://cl.ly/image/3W2e0X1O1i0o

wodow 13y ago
smtp.googlemail.com also failing (edit: intermittently)
fish2000 13y ago
Crazy – my IMAP client (Apple Mail) has no problem until it tries to send via SMTP… when it chokes. Also, no working gmail for all my accounts except my @gmail.com address – at least fail consistently, you Googlers. Blech!

varun1308 13y ago

Yeah.. my pop mail stopped working too

deleted 13y ago

[deleted]

ac132 13y ago

At least Google Reader still works :-)

arcosdev 13y ago

Faaack. I thought I was just having a local connection issue. Same problem here.

silasb 13y ago

Appears Groups is down also.

kyrra 13y ago

Oddly enough, gmail itself works on my Google Apps domain. But accessing my account page does not work. As well, I can't log into Gchat using one of the generated passwords (says password rejected). I'm guessing this is the same reason IMAP is failing for others.

TranceMan 13y ago

Google Apps status page: http://www.google.com/appsstatus http://www.google.com/appsstatus

deleted 13y ago
[deleted]

X-Istence 13y ago

That explains why I can't get my email =).

calleskonto 13y ago

The provision api is still fine doh.

9k9 13y ago

They probably don't know what to do

freehunter 13y ago

I'm wondering if this is related to the new update they did to the admin control panel. http://arstechnica.com/information-technology/2013/04/make-google-chrome-do-your-bidding-with-new-google-apps-control-panel/ http://arstechnica.com/information-technology/2013/04/make-g...

corresation 13y ago
There is a massively linked Boston bomber Google Docs spreadsheet making the rounds -- the last time a viral Google doc saw that sort of exposure, the service went down.

ndespres 13y ago

I'm a Google Apps reseller and thought, oh no, I should call all my clients and let them know about this outage before they call me! Where can I get a full customer list.. the admin panel.. oh.

lewispb 13y ago
The phone calls from my clients have started to come in already...!
stedaniels 13y ago
Your only copy of your customers, is in a third party reseller control panel?
- ndespres 13y ago
  Of course not. Just the fastest way to get them! Fortunately, Confluence, Autotask, and my memory are all still intact!
  - mathattack 13y ago
    "and my memory" When all else fails, this is the true old school solution. :-)
- deleted 13y ago
  [deleted]
jqueryin 13y ago
There are API endpoints for that. Not sure if they're affected.

paulmolluzzo 13y ago

It's back.

misnome 13y ago

Well, their email services are still working, because I got a message about our trial ending today and that we need to start paying. I hope this comes up before we get automatically cut off..

digitalpbk 13y ago

digitalpbk$ nslookup smtp.google.com Server: 8.8.8.8 Address: 8.8.8.8#53 server can't find smtp.google.com: NXDOMAIN SMTP servers are down for me. Anyone else?

brokentone 13y ago
NXDOMAIN wouldn't be server down, that's no DNS record. Still won't work, but different (much bigger) issue.
- tekacs 13y ago
  I'm pretty sure the 'bigger' issue here is that it's smtp.gmail.com - smtp.google.com is, I believe, always down on account of not existing. :P
  - pyre 13y ago
    Or: smtp.googlemail.com
  - deno 13y ago
    Works fine for me: > dig +short @8.8.8.8 smtp.gmail.com gmail-smtp-msa.l.google.com. 74.125.136.108 74.125.136.109 http://homepage.ntlworld.com/jonathan.deboynepollard/FGA/nslookup-flaws.html http://homepage.ntlworld.com/jonathan.deboynepollard/FGA/nsl...
duncans 13y ago
try `nslookup -querytype=MX google.com`

level09 13y ago

not only that, seems like pop3 is also facing issues ..

jlengrand 13y ago

I got a 500 when I try to access our Drive too. It's been an hour now. Using the business google services.

abhi_21 13y ago

Now its working..

EvanAnderson 13y ago

SAML logons are failing, too.

Leszek 13y ago

This explains the SMTP problems I was having, I was worried someone had hacked my account or something...

meerita 13y ago

I cannot access my email from Sparrow app too.

dotbill 13y ago

As I said in the other google apps thread on hackernews.. I thought I'd give them a call to find out whats going on, however to call them I need my support/admin pin - which is only accessible from the admin portal and without it they won't answer the phone. Nice.

AndrewDucker 13y ago

The Register reports that GMail went down exactly a year ago: http://www.theregister.co.uk/2012/04/18/google_gmail_outage/ http://www.theregister.co.uk/2012/04/18/google_gmail_outage/

saurik 13y ago
Huffington Post reports that Gmail also went offline on April 29th of 2010. http://www.huffingtonpost.com/2010/04/29/gmail-down-check-gmail-ou_n_557351.html http://www.huffingtonpost.com/2010/04/29/gmail-down-check-gm... Google reports that Gmail also went offline on April 24th of 2009. http://googleblog.blogspot.com/2009/02/current-gmail-outage.html http://googleblog.blogspot.com/2009/02/current-gmail-outage.... (Sadly, the partial Gmail outage of 2011 seemed to be February 27th, or this would be an epic pattern of "don't trust Gmail during the end of April" ;P.) http://www.eweek.com/c/a/Messaging-and-Collaboration/Google-Suffers-First-Gmail-Outage-of-2011-850632/ http://www.eweek.com/c/a/Messaging-and-Collaboration/Google-...

Selfcommit 13y ago

And We're up.

logn 13y ago

I don't get why apps and all sorts of other google sites go down a lot, but search never has in my experience. Why is that? Is search just simpler? Does it fail in non-obvious ways? Enormous redundancy/fail-over on search?

therealarmen 13y ago
Contributing the vast majority of revenue for the company has something to do with it.
bdonlan 13y ago
Search is (mostly) stateless. They can copy the search database to hundreds of DCs around the world, and each can operate independently - so if one of them has a problem, it's not likely to affect the others (unless it's bad data, but that's why you shouldn't deploy index updates to all DCs simultaneously). There's some personalization, sure, but you can just fall back to non-personalized search if that breaks. Things like gmail, on the other hand, are inherently stateful. When you log into gmail, you have to eventually connect to one system that maintains your mailbox. Sure, there might be replication - but the replicas are all talking to each other. It's surprisingly easy to have a cascade failure in a system like this, where one of the replicas going down triggers (directly or indirectly) all the others failing as well. Or you can have some bad data that gets replicated out, and then proceeds to confuse everything that's looking at it - unlike search, you have to replicate that data immediately, and don't get to enjoy the benefits of a staged deployment. This also explains why not all users were affected - I'd guess that their system is divided into some number of shards, and users are assigned to a particular shard. That 0.07% of users affected probably represents a single unhealthy shard.

ckdarby 13y ago

Why do people report this kind of stuff on hacker news. I am sure everyone knows how to google for a site's status page.