6 ms·

Gmail service disruption

101 points by jpadilla_ 13y ago

aroch 13y ago
So, at the 2012 I/O conference Google announced there were 425million gmail users. I'll tack an extra 75million to make things an even 500mil for growth over the last year. That's 12million affected users.
- anamexis 13y ago
  500 million * 0.00024 = 120,000
  - deleted 13y ago
    [deleted]
  - m0th87 13y ago
    It's been updated to say "less than 50% of Gmail users" now. We're seeing it across most/all of our accounts.
    - benjohnson 13y ago
      Based on my experience this morning, I'd say that 15% of our 300 or so Gmail users over about 25 domains were effected. In my opinion, their initial estimate was way too low and seemed dismissive.
chuckp 13y ago
I can confirm our service is having issues right now with delayed inbound emails. We are Google Apps for Business users, non edu/gov. Not all users are effected just specific domains within the account.
- jryedinak 13y ago
  Us too. Seems that when you attach a file, it's even less likely to go through.
  - chuckp 13y ago
    Not sure about you, but I find these status updates a little insulting: http://www.google.com/appsstatus#hl=en&v=issue&ts=1379995199000&sid=1&iid=043f082bc7cd18e15458318035d9bc7a http://www.google.com/appsstatus#hl=en&v=issue&ts=1379995199... At least with my cloud servers if there is an issue I at least get an idea of what is going on or an expectation of resolution. Google... what is really going on that an issue remains unresolved for 3-4hrs impacting paying users.
Terretta 13y ago
We've been reporting delayed inbound emails to Gmail (Google Apps) since Friday 13th. Glad to see they're now acknowledging it, though the percentage affected listed here seems remarkably low, considering an informal survey of employees and clients suggests from 1 in 50 increasing to 1 in 20 affected over the past week, with delays up to 24 hours.
donohoe 13y ago
TIMEZONE! I wish they (and others) would include timezone information. I always have to second-guess if its PST, EST, GMT or something else.
- bdcravens 13y ago
  Does list timezone info at bottom of page. It appears that it might display time based on browser's timezone.
- onestone 13y ago
  Bottom of the page: "All times are shown in your local timezone unless otherwise noted."
chuckp 13y ago
"The email delays are affecting less than 50% of Gmail users." That is a HUGE change from 0.024% this has been changed from a "Service disruption" to a "Service outage". Also, not sure if it is dependent on your Google login, but mine are all in EST.
- benjohnson 13y ago
  PST is broken too - all my clients are PST and most are effected. About half are paid, others are grandfathered free accounts and I see no appreciable difference.
  - _vn5r 13y ago
    I'm in Nicaragua and having issues with free accounts and apps accounts.
zippergz 13y ago
This is really killing my productivity today. I don't like being so reliant on email, but I work remotely, so it's a necessary evil. Having worked in operations for prominent companies, I sympathize with the gmail team right now, but my patience is wearing thin with the problem approaching the 5 hour mark.
ltnately 13y ago
I've had issues getting steadily worse all day across domain since ~9 am est or so. I just received this from Google as part of an auto reply to support ticket. "Thanks for your patience. Please read below for updates on the status of your issue: Updates since last message: This issue has been partially resolved for some users. We expect it to be fully resolved within the next hour. The issue caused severe delays sending emails, especially emails with large attachments. It also caused delays in loading Drive files with large images. The delays were caused because of a problem with an undersea cable. All parties involved are actively working to: 1) Redirect traffic and add additional capacity to resolve the immediate issue. 2) Fix the problem with the faulty undersea cable. "
- Terretta 13y ago
  What kind of mechanism for USA-based email service causes email delays for 50% of Gmail users due to an undersea cable? Not disputing, just curious about cause and effect here, and Gmail architecture.
  - thrownaway2424 13y ago
    http://www.google.com/about/datacenters/inside/locations/index.html http://www.google.com/about/datacenters/inside/locations/ind...
  - throwit1979 13y ago
    Wild guess: google routes non-latency-critical US traffic like SMTP outside the US so the NSA's beam splitters can more easily claim intercepted domestic data as "international".
  - sp332 13y ago
    If one route is disrupted, traffic might be re-routed and overwhelm another route?
guilhermetk 13y ago
9/23/13 4:00 PM Gmail service has already been restored for some users, and we expect a resolution for all users within the next 1 hours. Please note this time frame is an estimate and may change.
bluedino 13y ago
I was wondering why my emails on my phone weren't going through of a server I took pictures of this morning, and then I was getting 'rejected by server' emails after that. Here I was blaming iOS 7.
busterc 13y ago
Worse than an outage, I got this dreaded email from Google a few hours ago. I hope there isn't some sort of attack going on: Someone recently tried to use an application to sign in to your Google Account - xxx@gmail.com. We prevented the sign-in attempt in case this was a hijacker trying to access your account.
- busterc 13y ago
  ... We prevented the sign-in attempt in case this was a hijacker trying to access your account.
- mr_spothawk 13y ago
  I had this today as well. Where was your attempt coming from? Mine was in HK. (To be fair, I think it was my own machine attempting to connect via VPN)
  - cheapsteak 13y ago
    Got one yesterday from Anhui China. I'm definitely not using any VPN services there
- Windrider 13y ago
  Messages like this are only sent if the password was correct, AFAIK. (Google sends the message in part as a warning and in part to explain that no, your password wasn't wrong, they just don't think it was actually you.) This was presumably not you, so change your password immediately.
fjabre 13y ago
So does this mean we've lost some emails forever or does this mean we'll get the emails eventually just delayed? EDIT: I have an email currently lost in limbo. 3 hours delayed and counting. Not sure if it will ever get delivered.
- ChikkaChiChi 13y ago
  It should be fine.
- alternize 13y ago
  emails started to pour in here... some with roughly 4hrs delay.
- blhack 13y ago
  I think this is a bit more complicated than normal. You can read the RFCs if you want to, but normally what happens if an email cannot be delivered is that it will sit in the queue on the SMTP server trying to relay it with status: deferred. What's different this time around is that gmail is actually reporting that emails ARE being delivered, even though they may not be. Earlier this morning, I did a manual SMTP transaction[1] with aspmx.l.google.com and got an OK status when I sent the email. This means that a server relaying mail to gmail would NOT leave a message in its own queue (to retry delivery later), because the server would "think" that it had been delivered. The problem here is that my "delivered" message did not show up in my inbox until quite a bit later. SO! Instead of the SMTP system being able to take care of its own outages, the responsibility is upon google to have properly queued messages for delivery internally, and then to deliver them when the system goes back to normal. It sounds like what is happening is that google is accepting the messages, but then they aren't getting delivered to a mailbox. They're being queued somewhere along the way (at google). -- Your emails will probably be fine, but unfortunately you probably can't call your mail administrator and ask what is up (unless you work at google). Normally, you'd be able to look in your smtp logs and see a message status (delivered, deferred, etc.)
ChikkaChiChi 13y ago
Did anyone else just realize that there is NO WAY for us to be notified when an issue is affecting your specfic account? /appstatus isn't good enough...
ltcoleman 13y ago
That announcement at least tells me something is going on, but i am not seeing a disruption. I have a total outage. I cannot even get to the login page.
blhack 13y ago
This is really frustrating for me. I argued up and DOWN with people about us switching to google apps for our email. Up until recently, we were hosting our own mail on a server in our switch room. "It's more reliable!" I would insist. Well...it hasn't been more reliable. There has been more than one longish-term downtime that I can remember since we switched over. When I was hosting it myself, I could give people status updates, an estimate for when it would be fixed, some kind of explanation. Now, all I can give them is "it's broken. I don't know why, and google isn't telling anybody why." Very frustrating, especially considering how much we pay every month for it.
- kmfrk 13y ago
  What's customer service like in a situation like that, given that you're paying for it?
  - blhack 13y ago
    Because you asked, I tracked down a support number and called it. It said that: "we are experiencing a problem with gmail. Check google.com/appstatus for updates. If your issue is NOT about gmail, please stay on the line." So...nothing.
    - kmfrk 13y ago
      Jesus. I know that was the answer I expected, but it still amazes me.
      - cenhyperion 13y ago
        It's a mess, but on the flip side what could support do for you? They knew about it and I'm guessing every engineer on the team was working on getting service back online. All they really can do is tell you "It's down. We're aware of the issue. We're actively working to resolve it."
- chuckp 13y ago
  This was pretty much my reaction. We switched about 3 years ago from a self hosted Exchange system and up until now it has been pretty good. Only issues were isolated accounts offline for under a hour. Back then I could say we are about 75% back and give a full description of the issue in laymans terms which usually curbed the anxieties. Today I basically pointed them to the status page saying that this is all I have to work with. In my experience Google's support has been pretty bad but that is because most of the time their products work well, but yea I feel in the dark when something like this occurs. Still their uptime has been better than having that "oh shit" moment because a DC is degraded. In the end, everyone could still access the email and nothing was lost.
PaulHoule 13y ago
Google services have lately seemed like something Ralph Nader warned you about. Lately I've found that many Google services (like Google Groups) don't work entirely right with IE or Firefox and they only work right with chrome. Today I logged into my Google Analytics for the first time, and Google wanted to verify my phone by sending me an SMS. Well, my house is outside the GSM service footprint except when there's a strong temperature inversion, so I get in my car and drive 2 miles, park at the school, don't get the message, I reboot my phone, the message finally comes, then I drive home, and by that time I guess my session timed out because after I typed in my verification code into my computer, it asked me for my password and wanted me to verify again. I canceled and then Google Analytics in years came up (after about three minutes loading) and locked my browser (Google Chrome) up, until I discovered there was some dialog box that wound up off my main screen that was locking it up. And then they came out with the new gmail which seems to be yet another scheme to shove Google+ down my throat. Try doing a search for a programming related query and instead of getting the relevant documentation you get something in broken English that tells you what a loser you are because you are trying to do what you're trying to do and gives you ten incoherent reasons why you shouldn't do what you're doing. Meanwhile, Matt Butts is making Youtube videos about how you'd better make sure that anybody who links to your site had better make it a nofollow link because if anybody makes you a link that's not a nofollow link you must surely be a spammer. Therefore, you should check all your links in Google Webmaster Tools and write a polite letter to anybody who links to your web site that they'd better remove the links or if they can't do that, make the links nofollow. Speaking of Youtube, if you're in the 80% of the land area of the USA where all you can get is bad DSL and bad satellite service, it's just unwatchable. Amazon Prime, Vudu, Netflix and everybody else seem to be able to cope, but I don't find anything entertaining at all on Youtube when I have to wait 15 minutes to see a 3 minute video,. Oh yeah, but if you get 50,000 people to plus you, maybe your site rankings will move from the third page to the second page, or maybe not. The only organizations who can get away with being linked to today are Wikipedia and spam factories that used to be search engines, like about.com, because if Google bans them (as they should) the DoJ will be on them with an anti-trust lawsuit. What's worst about it is that if some small competitor comes up with something better, Google buys them up, shuts them down, and then you're left with Google's offerings which just get worse every week. I remember when I was proud to be a Google AdSense publisher because it didn't have the "one rule to a flat tummy" ads and you know what, they bought the sleazy ad networks and now AdSense is as sleazy as anything. Oh, but if you have one erotic painting from 1649 on your site out of 1,000,000 images they'll kick you out of AdSense without warning the day you are flying back from a conference in San Francisco after one of their agents watches you give a talk at a conference about your web site. I will say I'm impressed with android, both as an end user and a developer -- it's amazing. But my understanding is that their "scalable" platform for distributed applications is getting long in the tooth and that everything takes four times longer for Googlers to do it than anybody else. Their recruiters never stop calling, never get the hint that you don't want to go work at their beehive where whatever they pay you won't go very far because unless you commute to Fresno you'll spend it all on a mortgage for a house that will be worthless in 25 years and, of all places, you know you'll be just a number to them, if you don't fit their mathematical model of what a coder or a product manager or whatever it is, you just suck. That won't stop the recruiters from calling though, because the whole point isn't that they want your labor but they just want to kneecap their competitors by taking anybody who can pass an IQ test and wants to live around San Jose away. Google used to be about a culture of excellence, but today Google is the new Microsoft. Because they never release versions, the press never catches on to the fact that Google's services are like Windows 8.
pfortuny 13y ago
I know this is nitpicking but this datum is at least confusing: "The average (median) delay was just 2.6 seconds, but some mail was more severely delayed. However, this issue did not affect users' access to the Gmail page or other functionality." What does "some" mean in Google's order of magnitude? A median time without the maximum and minimum is not quite useful, as a mean time without more data.

101 points by jpadilla_ 13y ago

aroch 13y ago

So, at the 2012 I/O conference Google announced there were 425million gmail users. I'll tack an extra 75million to make things an even 500mil for growth over the last year. That's 12million affected users.

anamexis 13y ago
500 million * 0.00024 = 120,000
- deleted 13y ago
  [deleted]
- m0th87 13y ago
  It's been updated to say "less than 50% of Gmail users" now. We're seeing it across most/all of our accounts.
  - benjohnson 13y ago
    Based on my experience this morning, I'd say that 15% of our 300 or so Gmail users over about 25 domains were effected. In my opinion, their initial estimate was way too low and seemed dismissive.

chuckp 13y ago

I can confirm our service is having issues right now with delayed inbound emails. We are Google Apps for Business users, non edu/gov. Not all users are effected just specific domains within the account.

jryedinak 13y ago
Us too. Seems that when you attach a file, it's even less likely to go through.
- chuckp 13y ago
  Not sure about you, but I find these status updates a little insulting: http://www.google.com/appsstatus#hl=en&v=issue&ts=1379995199000&sid=1&iid=043f082bc7cd18e15458318035d9bc7a http://www.google.com/appsstatus#hl=en&v=issue&ts=1379995199... At least with my cloud servers if there is an issue I at least get an idea of what is going on or an expectation of resolution. Google... what is really going on that an issue remains unresolved for 3-4hrs impacting paying users.

Terretta 13y ago

We've been reporting delayed inbound emails to Gmail (Google Apps) since Friday 13th. Glad to see they're now acknowledging it, though the percentage affected listed here seems remarkably low, considering an informal survey of employees and clients suggests from 1 in 50 increasing to 1 in 20 affected over the past week, with delays up to 24 hours.

donohoe 13y ago

TIMEZONE! I wish they (and others) would include timezone information. I always have to second-guess if its PST, EST, GMT or something else.

bdcravens 13y ago
Does list timezone info at bottom of page. It appears that it might display time based on browser's timezone.
onestone 13y ago
Bottom of the page: "All times are shown in your local timezone unless otherwise noted."

chuckp 13y ago

"The email delays are affecting less than 50% of Gmail users." That is a HUGE change from 0.024% this has been changed from a "Service disruption" to a "Service outage". Also, not sure if it is dependent on your Google login, but mine are all in EST.

benjohnson 13y ago
PST is broken too - all my clients are PST and most are effected. About half are paid, others are grandfathered free accounts and I see no appreciable difference.
- _vn5r 13y ago
  I'm in Nicaragua and having issues with free accounts and apps accounts.

zippergz 13y ago

This is really killing my productivity today. I don't like being so reliant on email, but I work remotely, so it's a necessary evil. Having worked in operations for prominent companies, I sympathize with the gmail team right now, but my patience is wearing thin with the problem approaching the 5 hour mark.

ltnately 13y ago

I've had issues getting steadily worse all day across domain since ~9 am est or so. I just received this from Google as part of an auto reply to support ticket. "Thanks for your patience. Please read below for updates on the status of your issue: Updates since last message: This issue has been partially resolved for some users. We expect it to be fully resolved within the next hour. The issue caused severe delays sending emails, especially emails with large attachments. It also caused delays in loading Drive files with large images. The delays were caused because of a problem with an undersea cable. All parties involved are actively working to: 1) Redirect traffic and add additional capacity to resolve the immediate issue. 2) Fix the problem with the faulty undersea cable. "

Terretta 13y ago
What kind of mechanism for USA-based email service causes email delays for 50% of Gmail users due to an undersea cable? Not disputing, just curious about cause and effect here, and Gmail architecture.
- thrownaway2424 13y ago
  http://www.google.com/about/datacenters/inside/locations/index.html http://www.google.com/about/datacenters/inside/locations/ind...
- throwit1979 13y ago
  Wild guess: google routes non-latency-critical US traffic like SMTP outside the US so the NSA's beam splitters can more easily claim intercepted domestic data as "international".
- sp332 13y ago
  If one route is disrupted, traffic might be re-routed and overwhelm another route?

guilhermetk 13y ago

9/23/13 4:00 PM Gmail service has already been restored for some users, and we expect a resolution for all users within the next 1 hours. Please note this time frame is an estimate and may change.

bluedino 13y ago

I was wondering why my emails on my phone weren't going through of a server I took pictures of this morning, and then I was getting 'rejected by server' emails after that. Here I was blaming iOS 7.

busterc 13y ago

Worse than an outage, I got this dreaded email from Google a few hours ago. I hope there isn't some sort of attack going on: Someone recently tried to use an application to sign in to your Google Account - xxx@gmail.com. We prevented the sign-in attempt in case this was a hijacker trying to access your account.

busterc 13y ago
... We prevented the sign-in attempt in case this was a hijacker trying to access your account.
mr_spothawk 13y ago
I had this today as well. Where was your attempt coming from? Mine was in HK. (To be fair, I think it was my own machine attempting to connect via VPN)
- cheapsteak 13y ago
  Got one yesterday from Anhui China. I'm definitely not using any VPN services there
Windrider 13y ago
Messages like this are only sent if the password was correct, AFAIK. (Google sends the message in part as a warning and in part to explain that no, your password wasn't wrong, they just don't think it was actually you.) This was presumably not you, so change your password immediately.

fjabre 13y ago

So does this mean we've lost some emails forever or does this mean we'll get the emails eventually just delayed? EDIT: I have an email currently lost in limbo. 3 hours delayed and counting. Not sure if it will ever get delivered.

ChikkaChiChi 13y ago
It should be fine.
alternize 13y ago
emails started to pour in here... some with roughly 4hrs delay.
blhack 13y ago
I think this is a bit more complicated than normal. You can read the RFCs if you want to, but normally what happens if an email cannot be delivered is that it will sit in the queue on the SMTP server trying to relay it with status: deferred. What's different this time around is that gmail is actually reporting that emails ARE being delivered, even though they may not be. Earlier this morning, I did a manual SMTP transaction[1] with aspmx.l.google.com and got an OK status when I sent the email. This means that a server relaying mail to gmail would NOT leave a message in its own queue (to retry delivery later), because the server would "think" that it had been delivered. The problem here is that my "delivered" message did not show up in my inbox until quite a bit later. SO! Instead of the SMTP system being able to take care of its own outages, the responsibility is upon google to have properly queued messages for delivery internally, and then to deliver them when the system goes back to normal. It sounds like what is happening is that google is accepting the messages, but then they aren't getting delivered to a mailbox. They're being queued somewhere along the way (at google). -- Your emails will probably be fine, but unfortunately you probably can't call your mail administrator and ask what is up (unless you work at google). Normally, you'd be able to look in your smtp logs and see a message status (delivered, deferred, etc.)

ChikkaChiChi 13y ago

Did anyone else just realize that there is NO WAY for us to be notified when an issue is affecting your specfic account? /appstatus isn't good enough...

ltcoleman 13y ago

That announcement at least tells me something is going on, but i am not seeing a disruption. I have a total outage. I cannot even get to the login page.

blhack 13y ago

This is really frustrating for me. I argued up and DOWN with people about us switching to google apps for our email. Up until recently, we were hosting our own mail on a server in our switch room. "It's more reliable!" I would insist. Well...it hasn't been more reliable. There has been more than one longish-term downtime that I can remember since we switched over. When I was hosting it myself, I could give people status updates, an estimate for when it would be fixed, some kind of explanation. Now, all I can give them is "it's broken. I don't know why, and google isn't telling anybody why." Very frustrating, especially considering how much we pay every month for it.

kmfrk 13y ago
What's customer service like in a situation like that, given that you're paying for it?
- blhack 13y ago
  Because you asked, I tracked down a support number and called it. It said that: "we are experiencing a problem with gmail. Check google.com/appstatus for updates. If your issue is NOT about gmail, please stay on the line." So...nothing.
  - kmfrk 13y ago
    Jesus. I know that was the answer I expected, but it still amazes me.
    - cenhyperion 13y ago
      It's a mess, but on the flip side what could support do for you? They knew about it and I'm guessing every engineer on the team was working on getting service back online. All they really can do is tell you "It's down. We're aware of the issue. We're actively working to resolve it."
chuckp 13y ago
This was pretty much my reaction. We switched about 3 years ago from a self hosted Exchange system and up until now it has been pretty good. Only issues were isolated accounts offline for under a hour. Back then I could say we are about 75% back and give a full description of the issue in laymans terms which usually curbed the anxieties. Today I basically pointed them to the status page saying that this is all I have to work with. In my experience Google's support has been pretty bad but that is because most of the time their products work well, but yea I feel in the dark when something like this occurs. Still their uptime has been better than having that "oh shit" moment because a DC is degraded. In the end, everyone could still access the email and nothing was lost.

PaulHoule 13y ago

Google services have lately seemed like something Ralph Nader warned you about. Lately I've found that many Google services (like Google Groups) don't work entirely right with IE or Firefox and they only work right with chrome. Today I logged into my Google Analytics for the first time, and Google wanted to verify my phone by sending me an SMS. Well, my house is outside the GSM service footprint except when there's a strong temperature inversion, so I get in my car and drive 2 miles, park at the school, don't get the message, I reboot my phone, the message finally comes, then I drive home, and by that time I guess my session timed out because after I typed in my verification code into my computer, it asked me for my password and wanted me to verify again. I canceled and then Google Analytics in years came up (after about three minutes loading) and locked my browser (Google Chrome) up, until I discovered there was some dialog box that wound up off my main screen that was locking it up. And then they came out with the new gmail which seems to be yet another scheme to shove Google+ down my throat. Try doing a search for a programming related query and instead of getting the relevant documentation you get something in broken English that tells you what a loser you are because you are trying to do what you're trying to do and gives you ten incoherent reasons why you shouldn't do what you're doing. Meanwhile, Matt Butts is making Youtube videos about how you'd better make sure that anybody who links to your site had better make it a nofollow link because if anybody makes you a link that's not a nofollow link you must surely be a spammer. Therefore, you should check all your links in Google Webmaster Tools and write a polite letter to anybody who links to your web site that they'd better remove the links or if they can't do that, make the links nofollow. Speaking of Youtube, if you're in the 80% of the land area of the USA where all you can get is bad DSL and bad satellite service, it's just unwatchable. Amazon Prime, Vudu, Netflix and everybody else seem to be able to cope, but I don't find anything entertaining at all on Youtube when I have to wait 15 minutes to see a 3 minute video,. Oh yeah, but if you get 50,000 people to plus you, maybe your site rankings will move from the third page to the second page, or maybe not. The only organizations who can get away with being linked to today are Wikipedia and spam factories that used to be search engines, like about.com, because if Google bans them (as they should) the DoJ will be on them with an anti-trust lawsuit. What's worst about it is that if some small competitor comes up with something better, Google buys them up, shuts them down, and then you're left with Google's offerings which just get worse every week. I remember when I was proud to be a Google AdSense publisher because it didn't have the "one rule to a flat tummy" ads and you know what, they bought the sleazy ad networks and now AdSense is as sleazy as anything. Oh, but if you have one erotic painting from 1649 on your site out of 1,000,000 images they'll kick you out of AdSense without warning the day you are flying back from a conference in San Francisco after one of their agents watches you give a talk at a conference about your web site. I will say I'm impressed with android, both as an end user and a developer -- it's amazing. But my understanding is that their "scalable" platform for distributed applications is getting long in the tooth and that everything takes four times longer for Googlers to do it than anybody else. Their recruiters never stop calling, never get the hint that you don't want to go work at their beehive where whatever they pay you won't go very far because unless you commute to Fresno you'll spend it all on a mortgage for a house that will be worthless in 25 years and, of all places, you know you'll be just a number to them, if you don't fit their mathematical model of what a coder or a product manager or whatever it is, you just suck. That won't stop the recruiters from calling though, because the whole point isn't that they want your labor but they just want to kneecap their competitors by taking anybody who can pass an IQ test and wants to live around San Jose away. Google used to be about a culture of excellence, but today Google is the new Microsoft. Because they never release versions, the press never catches on to the fact that Google's services are like Windows 8.

pfortuny 13y ago

I know this is nitpicking but this datum is at least confusing: "The average (median) delay was just 2.6 seconds, but some mail was more severely delayed. However, this issue did not affect users' access to the Gmail page or other functionality." What does "some" mean in Google's order of magnitude? A median time without the maximum and minimum is not quite useful, as a mean time without more data.