I think the duration of this outage was basically about waiting for the GFS team to roll in. They knew at 6:44am that they had a crashing GFS Master on their h

I think the duration of this outage was basically about waiting for the GFS team to roll in. They knew at 6:44am that they had a crashing GFS Master on their hands, but they didn't decide to try upgrading until 8am, and only escalated to the GFS team after the upgrade failed to help, at 9am. That's the story of this outage, to me: 6:44 AM --- A GFS Site Reliability Engineer (SRE) reports that the GFS Master in App Engine's primary data center is failing and continuously restarting. 8:00 AM --- The cause of the GFS Master failures has not yet been identified. [...] the GFS SRE decides to commence the upgrade immediately in an attempt to alleviate the problem. 9:00 AM --- The GFS upgrade [...] finishes, but the Master is still failing. The GFS SRE escalates directly to the GFS engineering team, and [they begin] live debugging of the failing software[...]. 10:00 AM --- GFS SRE advises that the GFS engineering team has identified the cause [...] GFS Master is no longer failing and GFS Chunkservers, which hold the actual needed data, are starting to come back up by 10:30 AM.