7 ms·

Hi - I work at Google on GKE - sorry about the problems you're experiencing. There's a lot of people inside Google looking into this right now! It looks like

Hi - I work at Google on GKE - sorry about the problems you're experiencing. There's a lot of people inside Google looking into this right now! It looks like the UI issue was actually fixed, and that we just didn't update the status dashboard correctly. But we're double checking that and looking into some of the additional things you all have reported here.

marcinzm 8y ago
I appreciate all the effort you're putting in and I understand such situations can be stressful but user's having to depend on someone responding on hacker news for status updates seems really amateur for an organization the size of google.
- ben_jones 8y ago
  As much as I love bashing big corps I see HN as a supplementary communication channel for products like GCP - its a luxury we get to access alongside normal customer support channels in the GCP console, twitter, etc.
  - thwy12321 8y ago
    Critical service is failing, minimal information about why, but we should be so happy someone says a few sentences on here? For all of the engineering elitism coming out of google, Amazon is way more on their game across a number of products.
  - yeukhon 8y ago
    Let me put it this way. HackerNews, or in fact, any news outlets are not official. Customers should be getting emails from Google and be informed on its official webpage to explain what's going on. You don't want your neighbor to tell you you owe taxes. You want the government to send a notice to you.
- NicoJuicy 8y ago
  The default is : https://status.cloud.google.com/incident/container-engine/18005 https://status.cloud.google.com/incident/container-engine/18... People who respond here could be employees of Google, caring about it and respond here because they know it. What he can mention ( a lot of people are working on it) is what you can suspect when something is going down. All other cloud providers do the same.
  - marcinzm 8y ago
    The default you linked to has not been updated in 2 days... which is my whole point regarding having to rely on hacker news for any status updates. edit: The default is also only about the UI issue and there's no issue tracker for the broader non-UI disruptions going on since Friday.
    - Waterluvian 8y ago
      Even an update of "no change" is tremendously valuable.
- trhway 8y ago
  >really amateur for an organization the size of google. There is a reason while Google have been having hard time making inroads in the enterprise cloud. Kind of impedance mismatch between enterprise and the Google style. That 2 stories like high "We heart API" sign on the Google Enterprise building facing 237 just screams about it :)
- rdtsc 8y ago
  Strangely and sadly with gmail account blocking and other such issues HN and Twitter is often better way to get Google's support than to contact support.
rlancer 8y ago
Creating clusters via the UI is still not working for me.
- rlancer 8y ago
  UPDATE: Created a Cluster successfully in Australia... Still not able to do so in the US.
- zachberger 8y ago
  Have you tried via the gcloud command?
antpls 8y ago
The status dashboard is inaccurate and/or a lie. It only tells about the GKE incident, while in fact the problem also impacts Google Compute Engine users. I was unable to create any google compute instance today, not even a basic 1vcpu, on NA and Europe-west. As another comment pointed out, what's the point of having so many zones and redundancy around the globe if such global failure can still happen? I thought the "cloud" was supposed to make this kind of failure impossible
- carbocation 8y ago
  > I was unable to create any google compute instance today, not even a basic 1vcpu, on NA and Europe-west. I've been creating GCP instances in us-central1-a and us-central1-c today without issue. Which zone were you using in NA? I have been noticing unusual restarts, but I haven't been able to pin down the cause yet (may be my software and not GCP itself).
  - antpls 8y ago
    Tried on us-east, us-north, europe-west, also tried asia, with different instance sizes and with both UI and CLI. None worked for me.
    - pfd1986 8y ago
      Same here.
  - aviv 8y ago
    Have not seen any restarts this weekend, and we have several hundred instances on GCE.
    - carbocation 8y ago
      Thanks! I'm running Skylake 96 core instances but I haven't given up to try the 64 core instances for comparison yet. If I get another restart, I'll do a 96 vs 64 to try to narrow down the cause. Most likely, of course, this is a software issue on my end, not Google's.
tomcam 8y ago
Thanks for jumping in here on your own time. The following question is not meant to be hostile, it is merely curiosity. Isn’t this supposed to be the kind of thing that monitoring and diagnostics software should find automatically? Serious question, not meant to embarrass you.
dilyevsky 8y ago
So, given that i filed this months ago via official support and it’s still not fixed, can you look into misleading container memory reporting ui bug. It reports memory_total but should be working_set
fizzledbits 8y ago
As of this morning, I am still unable to reliably start my docker+machine autoscaling instances. In all cases the error is "Error: The zone <my project> does not have enough resources available to fulfill the request" An instance in us-central1-a has refused to start since last Thursday or Friday. I created a new instance in us-west2-c, which worked briefly but began to fail midday Friday, and kept failing through the weekend. On Saturday I created yet another clone in northamerica-northeast1-b. That worked Saturday and Sunday, but this morning, it is failing to start. Fortunately my us-west2-c instance has begun to work again, but I'm having doubts about continuing to use GCE as we scale up. And yet, the status page says all services are available.

marcinzm 8y ago

I appreciate all the effort you're putting in and I understand such situations can be stressful but user's having to depend on someone responding on hacker news for status updates seems really amateur for an organization the size of google.

ben_jones 8y ago
As much as I love bashing big corps I see HN as a supplementary communication channel for products like GCP - its a luxury we get to access alongside normal customer support channels in the GCP console, twitter, etc.
- thwy12321 8y ago
  Critical service is failing, minimal information about why, but we should be so happy someone says a few sentences on here? For all of the engineering elitism coming out of google, Amazon is way more on their game across a number of products.
- yeukhon 8y ago
  Let me put it this way. HackerNews, or in fact, any news outlets are not official. Customers should be getting emails from Google and be informed on its official webpage to explain what's going on. You don't want your neighbor to tell you you owe taxes. You want the government to send a notice to you.
NicoJuicy 8y ago
The default is : https://status.cloud.google.com/incident/container-engine/18005 https://status.cloud.google.com/incident/container-engine/18... People who respond here could be employees of Google, caring about it and respond here because they know it. What he can mention ( a lot of people are working on it) is what you can suspect when something is going down. All other cloud providers do the same.
- marcinzm 8y ago
  The default you linked to has not been updated in 2 days... which is my whole point regarding having to rely on hacker news for any status updates. edit: The default is also only about the UI issue and there's no issue tracker for the broader non-UI disruptions going on since Friday.
  - Waterluvian 8y ago
    Even an update of "no change" is tremendously valuable.
trhway 8y ago
>really amateur for an organization the size of google. There is a reason while Google have been having hard time making inroads in the enterprise cloud. Kind of impedance mismatch between enterprise and the Google style. That 2 stories like high "We heart API" sign on the Google Enterprise building facing 237 just screams about it :)
rdtsc 8y ago
Strangely and sadly with gmail account blocking and other such issues HN and Twitter is often better way to get Google's support than to contact support.

rlancer 8y ago

Creating clusters via the UI is still not working for me.

rlancer 8y ago
UPDATE: Created a Cluster successfully in Australia... Still not able to do so in the US.
zachberger 8y ago
Have you tried via the gcloud command?

antpls 8y ago

The status dashboard is inaccurate and/or a lie. It only tells about the GKE incident, while in fact the problem also impacts Google Compute Engine users. I was unable to create any google compute instance today, not even a basic 1vcpu, on NA and Europe-west. As another comment pointed out, what's the point of having so many zones and redundancy around the globe if such global failure can still happen? I thought the "cloud" was supposed to make this kind of failure impossible

carbocation 8y ago
> I was unable to create any google compute instance today, not even a basic 1vcpu, on NA and Europe-west. I've been creating GCP instances in us-central1-a and us-central1-c today without issue. Which zone were you using in NA? I have been noticing unusual restarts, but I haven't been able to pin down the cause yet (may be my software and not GCP itself).
- antpls 8y ago
  Tried on us-east, us-north, europe-west, also tried asia, with different instance sizes and with both UI and CLI. None worked for me.
  - pfd1986 8y ago
    Same here.
- aviv 8y ago
  Have not seen any restarts this weekend, and we have several hundred instances on GCE.
  - carbocation 8y ago
    Thanks! I'm running Skylake 96 core instances but I haven't given up to try the 64 core instances for comparison yet. If I get another restart, I'll do a 96 vs 64 to try to narrow down the cause. Most likely, of course, this is a software issue on my end, not Google's.

tomcam 8y ago

Thanks for jumping in here on your own time. The following question is not meant to be hostile, it is merely curiosity. Isn’t this supposed to be the kind of thing that monitoring and diagnostics software should find automatically? Serious question, not meant to embarrass you.

dilyevsky 8y ago

So, given that i filed this months ago via official support and it’s still not fixed, can you look into misleading container memory reporting ui bug. It reports memory_total but should be working_set

fizzledbits 8y ago

As of this morning, I am still unable to reliably start my docker+machine autoscaling instances. In all cases the error is "Error: The zone <my project> does not have enough resources available to fulfill the request" An instance in us-central1-a has refused to start since last Thursday or Friday. I created a new instance in us-west2-c, which worked briefly but began to fail midday Friday, and kept failing through the weekend. On Saturday I created yet another clone in northamerica-northeast1-b. That worked Saturday and Sunday, but this morning, it is failing to start. Fortunately my us-west2-c instance has begun to work again, but I'm having doubts about continuing to use GCE as we scale up. And yet, the status page says all services are available.