5 ms·

While there are a series of vulnerabilities here, none of them would be exploitable in this way if the metadata server was accessed via an IP instead of the hos

While there are a series of vulnerabilities here, none of them would be exploitable in this way if the metadata server was accessed via an IP instead of the hostname metadata.google.internal. The metadata server is documented to be at 169.254.169.25, always[1]. But Google software (agents and libraries on VMs) resolves it by looking up metadata.google.internal. If metadata.google.internal isn't in /etc/hosts, as can be the case in containers, this can result in actual DNS lookups over the network to get an address that should be known. AWS uses the same address for their metadata server, but accesses via the IP address and not some hostname[2]. I've seen Google managed DNS servers (in GKE clusters) fall over under the load of Google libraries querying for the metadata address[3]. I'm guessing Google wants to maintain some flexibility, which is why they are using a hostname, but there are tradeoffs. [1] https://cloud.google.com/compute/docs/internal-dns https://cloud.google.com/compute/docs/internal-dns [2] https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance... [3] This is easily solvable with Kubernetes HostAliases that write /etc/hosts in the containers.

tryauuum 5y ago
I think even without metadata server replacement this attack would still be painful. The ability to reconfigure network on a victim sounds painful
- antoncohen 5y ago
  That is true, I was thinking specifically about the metadata and SSH keys. But DHCP can also set DNS servers, NTP servers, and other things that can either cause disruptions or be used to facilitate a different attack. There might be a persistence issue, it seems like part of this attack was that the IP was persisted to /etc/hosts even after the real DHCP server took over again. But even just writing to /etc/hosts could open the door redirecting traffic to an attacker controlled server.
aenis 5y ago
It does not even take a lot. I run a production service on cloud run, the typical load is around 500qps, and the dns queries to resolve metadata server do fail frequently enough for this to be noticeable.
bradfitz 5y ago
Even Google's Go client for the GCE metadata uses an IP address: > Using a fixed IP makes it very difficult to spoof the metadata https://github.com/googleapis/google-cloud-go/commit/ae568912e42f2dd25f2c7cc8f34bc7ae2087eb15 https://github.com/googleapis/google-cloud-go/commit/ae56891...
- skj 5y ago
  Hmm cloud build spoofs it :) if the customer build accessed the underlying VM's metadata it would be very confusing (though not a security issue). It was not straightforward. I learned a lot about iptables and docker networking.

tryauuum 5y ago

I think even without metadata server replacement this attack would still be painful. The ability to reconfigure network on a victim sounds painful

antoncohen 5y ago
That is true, I was thinking specifically about the metadata and SSH keys. But DHCP can also set DNS servers, NTP servers, and other things that can either cause disruptions or be used to facilitate a different attack. There might be a persistence issue, it seems like part of this attack was that the IP was persisted to /etc/hosts even after the real DHCP server took over again. But even just writing to /etc/hosts could open the door redirecting traffic to an attacker controlled server.

aenis 5y ago

It does not even take a lot. I run a production service on cloud run, the typical load is around 500qps, and the dns queries to resolve metadata server do fail frequently enough for this to be noticeable.

bradfitz 5y ago

Even Google's Go client for the GCE metadata uses an IP address: > Using a fixed IP makes it very difficult to spoof the metadata https://github.com/googleapis/google-cloud-go/commit/ae568912e42f2dd25f2c7cc8f34bc7ae2087eb15 https://github.com/googleapis/google-cloud-go/commit/ae56891...

skj 5y ago
Hmm cloud build spoofs it :) if the customer build accessed the underlying VM's metadata it would be very confusing (though not a security issue). It was not straightforward. I learned a lot about iptables and docker networking.