16 ms·

Google reveals its servers all contain custom security silicon

363 points by chris-at 10y ago

Duobix 10y ago
I barely dug into the article when It came to me: Google just did a lockout chip, 1980s Nintendo style.
- na85 10y ago
  I suppose if you're looking for a sound bite, yes. But whereas Nintendo's chip was DRM, this Google chip appears to be more about determinism in boots and server provisioning, allowing them to immediately cut out a server that appears malicious or that has been compromised. I.e. pry open case to insert an implant, chip notices bios has been altered, sends the "don't trust me" message to the network.
  - digi_owl 10y ago
    Makes me think of Intel's IME. It has legitimate uses on corporate desktops and servers. But when it makes its way to consumer desktops it runs face first into a massive conflict of interest.
    - kabdib 10y ago
      IME's huge problem is its shroud of secrecy. The CPU can do just about anything on the bus, it has access to external ports, and the code it runs is encrypted. From the viewpoint of a government agency, that's a tremendous surveillance enabler. It's really hard to imagine it's not been compromised.
woliveirajr 10y ago
> Disks get the following treatment: > “We enable hardware encryption support in our hard drives and SSDs and meticulously track each drive through its lifecycle. Before a decommissioned encrypted storage device can physically leave our custody, it is cleaned using a multi-step process that includes two independent verifications. Devices that do not pass this wiping procedure are physically destroyed (e.g. shredded) on-premise.” Interesting. There were discussions on the past on how to clean HDD, if multiple-passes were really necessary or not. Then SDD become the problem, since there is a interface between what you see (from the OS) and where the data really is (inside those chips). Now Google not only encrypts data before saving (that should be enough, no?) but also tries to wipe using multiple passes and 2 verifications. Wonder how many companies do that.
- nom 10y ago
  'multi-step process' doesn't imply they are wiping more than once, only that their procedure consists of multiple steps (e.g. wipe + verification)
  - sphix 10y ago
    The article specifies two independent verifications.
- problems 10y ago
  If you use on-board crypto on most SSDs, there's a dedicated place for key storage and using the SSD's onboard wipe feature just changes the key and TRIMs the whole drive. Most of these drives use cryptographic keys even if you don't use a password on the device. Think about it as an SSD manufacturer - what's the easiest way to wipe a drive? To actually go and zero out every cell on the disk or to overwrite a very small cryptographic key with a new one - effectively destroying the data without the need for any other write cycles to occur. Pretty easy to verify - if you have an SSD with support for this, which most do now.
  - throwawayish 10y ago
    > Think about it as an SSD manufacturer - what's the easiest way to wipe a drive? That's not the reason why encryption is always on. Flash endurance is; encrypting the data before FEC means that it will have a random distribution, which avoids pathological worst cases with certain workloads. You could also use a different (cheaper) scrambler than AES (like CPUs do [1]), but since encryption is a marketable feature... [1] Which are also switching to using AES and offering memory encryption in current mainstream architectures.
    - problems 10y ago
      Ah, interesting. That's really cool. I guess it makes things easier for them and better for their customers in several ways at once.
- 10y ago
sergiosgc 10y ago
This is another signal of an interesting development on the hardware front. What used to be decoupled, with some companies offering hardware, and different companies buying hardware, is now coupled and hidden within these mega-companies (Google, Amazon, FB). Google is big enough to develop a trusted hardware solution for internal use only, it has no financial need to sell it. Worse, due to competitiveness in the cloud segment, it is dis-incentivized from selling the solution. Amazon Glacier is another one. It's an interesting long-term storage solution, whose hardware implementation is unavailable to the market, since AMZN can better explore it as a service under AWS. We are heading onto a more closed ecosystem than we are used to up until here. The cloud, which gave us the immense positive benefit of moving all capex to opex, is birthing this immense negative side effect of closing off hardware implementations in favour of exploring the added value in the form of services.
- GFK_of_xmaspast 10y ago
  So basically what was assumed to be a linear trend is turning out to be a dialectic.
- nom 10y ago
  "We are heading onto a more closed ecosystem than we are used to up until here. " It wouldn't be more open even if they didn't spin their own hardware. You won't ever see it nor have access to it on a low level - it's 'the cloud'. The only thing it tells us is that they have reached a scale where custom hardware makes things cheaper, more reliable and more manageable for them.
  - deelowe 10y ago
    Google joined open compute last year.
  - sergiosgc 10y ago
    > It wouldn't be more open even if they didn't spin their own hardware. Of course. If the alternative is non-development, we are better off. However, my text states that the alternative is de-coupling: One company develops and markets the innovative hardware solution, other companies build services on top of that solution. This would be more open, it would present a more liquid market.
NelsonMinar 10y ago
This is what security looks like when your threat model is well funded government agencies.
- dx034 10y ago
  Exactly, and I think it's worth noting that they likely only apply this level of security because of state actors. Which then shows that they are trying to prevent eavesdropping by NSA & Co., they probably just realized too late how far advanced they were.
  - DannyB2 10y ago
    . . . how far advanced, and the lengths they would go to.
  - nickik 10y ago
    Eric Grosse from Google says as much here, ... Edit, it seems I have copied the wrong video, please us the link in the child comment
    - prattmic 10y ago
      This talk is also discusses the adversaries: https://www.youtube.com/watch?v=0knR6vXba7g https://www.youtube.com/watch?v=0knR6vXba7g
  - a_imho 10y ago
    Which then shows that they are trying to prevent eavesdropping by NSA & Co. Why would the NSA eavesdrop on Google, they are in bed with them, aren't they?
    - kakarot 10y ago
      Got links?
      - a_imho 10y ago
amelius 10y ago
But if they don't own an IC fab, how do they know it is secure?
- swegg 10y ago
  Basically splitting the trusted circuit and testing the parts separately. This requires a trusted master circuit, but it can be arbitrarily small. See https://perso.uclouvain.be/fstandae/PUBLIS/177.pdf https://perso.uclouvain.be/fstandae/PUBLIS/177.pdf
  - amelius 10y ago
    But what if the malicious code is time activated? (just an example)
    - swegg 10y ago
      This is actually addressed in the paper. Basically you can use testing to detect the timebomb, up to a negligible probability. This paper is approachable, it's understandable without too much background if you're interested in the topic.
- nickpsecurity 10y ago
  They don't. Too many attack vectors. I illustrate here: https://news.ycombinator.com/item?id=10468624 https://news.ycombinator.com/item?id=10468624 My solution was to print the TCB on a process node that was verifiable by eye. Then, verify a random sample of each batch. Possibly speed it up with image processing algorithms if producing the same component or components. Note: Deviations from intended circuitry in deep sub-micron can have measurable differences at analog or RF level. DARPA is funding research to do such things. A monitor at visible node could then be combined with CPU's on cutting-edge node. Common practice in commercial sector is obfuscations, though.
foobiekr 10y ago
The actual document - https://cloud.google.com/security/security-design/ https://cloud.google.com/security/security-design/ - was linked previously. It is interesting that they are doing some variant of trusted computing mostly because their homogeneity would allow Google to build a robust containment architecture with much more rigorous whitelisting and a robust SW distribution rules that go beyond what a measuring host and local SW bundle verification can do. So defense in depth. We (skyport systems) do the same thing as a service for enterprises (we sell and operate cloud-managed trusted systems as a service) and I will say it's pretty hard to get people to think about depth and trustworthiness when the entire security industry has trained CIOs to believe that all they need to do is install some random agent on their VMs. Good for Google.
gerbilly 10y ago
Still it doesn't make me want to use their services. They may indeed be really good at securing their data but 'their data' ironically is derived from my emails and browsing history and that of my friends.
tlb 10y ago
"Before a decommissioned encrypted storage device can physically leave our custody, it is cleaned using a multi-step process that includes two independent verifications. Devices that do not pass this wiping procedure are physically destroyed (e.g. shredded) on-premise" Why not just shred all decommissioned disks? Someone must be buying them for enough money that Google created a multi-step process for cleaning and verifying them. Presumably Google keeps disks in commission until they're no longer economic in their own operation. So, does anyone know about the operation that makes profitable use of disks that are no longer economic for Google?
- hodgesrm 10y ago
  Just speculation but they could be used elsewhere in Google for test/dev or pre-prod environments.
- wang_li 10y ago
  I'd imagine it's easier to recycle an intact HDD than a jumbled mix of silicon, ceramic, steel, aluminum and what not.
- deleted 10y ago
  [deleted]
- phire 10y ago
  Probably because of the verification. You can't really verify a shredding, the pile of shredded remains no-longer has a serial number. I assume the cleaning process cryptographically verifies the identify of the disk both before and after the wipe, making it impossible to sneak a drive out. For those drives which fail the cleaning process, they probably have a complex process with multiple witnesses to ensure it actually gets shredded.
- willk 10y ago
  My guess to the decommissioned disks is that HDD manufactures will sometimes give a hefty discount on disks if they can have them back at the end to run diagnostics on them. My company has a no disks leave the company policy and there has been talk about modifying this for the discount on disks.
bogomipz 10y ago
I was curious about this: >"There's plenty more in the document, like news that Google's public cloud runs virtual machines in a custom version of the KVM hypervisor." Does anyone know if this "container inside kvm" is true of their internal infrastructure as well or its just an extra layer of security for their public facing cloud?
- wmf 10y ago
  Internal Google stuff does not use KVM and that's one reason it took them a while to offer VMs — they had little experience with it.
  - bogomipz 10y ago
    Do you or anyone else know if there is another reason for doing this besides security?
    - wmf 10y ago
      I can't speak for Google, but there are several reasons. Docker and k8s are not multitenant, so if you want to build a public k8s cloud you need a tenant layer under it. That layer could also be containers (e.g. LXD), but then you're talking about secure nested containers which was not really available in November 2014.
      - bogomipz 10y ago
        Oh good insight. That makes a lot of sense. Thanks.
DanielDent 10y ago
A lot of stuff from this made it's way into the chromebook. There's a verified boot process, hardware assisted key management, rollback protection, ... And it's all open source and nicely documented for anyone who cares to look. With a bit of work you can actually create your own chain of trust and run your own verified boot process. It's very cool.
- knowaveragejoe 10y ago
  What is this called? I'd like to look into it, but cursory searching is giving me more vague results.

363 points by chris-at 10y ago

Duobix 10y ago

I barely dug into the article when It came to me: Google just did a lockout chip, 1980s Nintendo style.

na85 10y ago
I suppose if you're looking for a sound bite, yes. But whereas Nintendo's chip was DRM, this Google chip appears to be more about determinism in boots and server provisioning, allowing them to immediately cut out a server that appears malicious or that has been compromised. I.e. pry open case to insert an implant, chip notices bios has been altered, sends the "don't trust me" message to the network.
- digi_owl 10y ago
  Makes me think of Intel's IME. It has legitimate uses on corporate desktops and servers. But when it makes its way to consumer desktops it runs face first into a massive conflict of interest.
  - kabdib 10y ago
    IME's huge problem is its shroud of secrecy. The CPU can do just about anything on the bus, it has access to external ports, and the code it runs is encrypted. From the viewpoint of a government agency, that's a tremendous surveillance enabler. It's really hard to imagine it's not been compromised.

woliveirajr 10y ago

> Disks get the following treatment: > “We enable hardware encryption support in our hard drives and SSDs and meticulously track each drive through its lifecycle. Before a decommissioned encrypted storage device can physically leave our custody, it is cleaned using a multi-step process that includes two independent verifications. Devices that do not pass this wiping procedure are physically destroyed (e.g. shredded) on-premise.” Interesting. There were discussions on the past on how to clean HDD, if multiple-passes were really necessary or not. Then SDD become the problem, since there is a interface between what you see (from the OS) and where the data really is (inside those chips). Now Google not only encrypts data before saving (that should be enough, no?) but also tries to wipe using multiple passes and 2 verifications. Wonder how many companies do that.

nom 10y ago
'multi-step process' doesn't imply they are wiping more than once, only that their procedure consists of multiple steps (e.g. wipe + verification)
- sphix 10y ago
  The article specifies two independent verifications.
problems 10y ago
If you use on-board crypto on most SSDs, there's a dedicated place for key storage and using the SSD's onboard wipe feature just changes the key and TRIMs the whole drive. Most of these drives use cryptographic keys even if you don't use a password on the device. Think about it as an SSD manufacturer - what's the easiest way to wipe a drive? To actually go and zero out every cell on the disk or to overwrite a very small cryptographic key with a new one - effectively destroying the data without the need for any other write cycles to occur. Pretty easy to verify - if you have an SSD with support for this, which most do now.
- throwawayish 10y ago
  > Think about it as an SSD manufacturer - what's the easiest way to wipe a drive? That's not the reason why encryption is always on. Flash endurance is; encrypting the data before FEC means that it will have a random distribution, which avoids pathological worst cases with certain workloads. You could also use a different (cheaper) scrambler than AES (like CPUs do [1]), but since encryption is a marketable feature... [1] Which are also switching to using AES and offering memory encryption in current mainstream architectures.
  - problems 10y ago
    Ah, interesting. That's really cool. I guess it makes things easier for them and better for their customers in several ways at once.
10y ago

sergiosgc 10y ago

This is another signal of an interesting development on the hardware front. What used to be decoupled, with some companies offering hardware, and different companies buying hardware, is now coupled and hidden within these mega-companies (Google, Amazon, FB). Google is big enough to develop a trusted hardware solution for internal use only, it has no financial need to sell it. Worse, due to competitiveness in the cloud segment, it is dis-incentivized from selling the solution. Amazon Glacier is another one. It's an interesting long-term storage solution, whose hardware implementation is unavailable to the market, since AMZN can better explore it as a service under AWS. We are heading onto a more closed ecosystem than we are used to up until here. The cloud, which gave us the immense positive benefit of moving all capex to opex, is birthing this immense negative side effect of closing off hardware implementations in favour of exploring the added value in the form of services.

GFK_of_xmaspast 10y ago
So basically what was assumed to be a linear trend is turning out to be a dialectic.
nom 10y ago
"We are heading onto a more closed ecosystem than we are used to up until here. " It wouldn't be more open even if they didn't spin their own hardware. You won't ever see it nor have access to it on a low level - it's 'the cloud'. The only thing it tells us is that they have reached a scale where custom hardware makes things cheaper, more reliable and more manageable for them.
- deelowe 10y ago
  Google joined open compute last year.
- sergiosgc 10y ago
  > It wouldn't be more open even if they didn't spin their own hardware. Of course. If the alternative is non-development, we are better off. However, my text states that the alternative is de-coupling: One company develops and markets the innovative hardware solution, other companies build services on top of that solution. This would be more open, it would present a more liquid market.

NelsonMinar 10y ago

This is what security looks like when your threat model is well funded government agencies.

dx034 10y ago
Exactly, and I think it's worth noting that they likely only apply this level of security because of state actors. Which then shows that they are trying to prevent eavesdropping by NSA & Co., they probably just realized too late how far advanced they were.
- DannyB2 10y ago
  . . . how far advanced, and the lengths they would go to.
- nickik 10y ago
  Eric Grosse from Google says as much here, ... Edit, it seems I have copied the wrong video, please us the link in the child comment
  - prattmic 10y ago
    This talk is also discusses the adversaries: https://www.youtube.com/watch?v=0knR6vXba7g https://www.youtube.com/watch?v=0knR6vXba7g
- a_imho 10y ago
  Which then shows that they are trying to prevent eavesdropping by NSA & Co. Why would the NSA eavesdrop on Google, they are in bed with them, aren't they?
  - kakarot 10y ago
    Got links?
    - a_imho 10y ago

amelius 10y ago

But if they don't own an IC fab, how do they know it is secure?

swegg 10y ago
Basically splitting the trusted circuit and testing the parts separately. This requires a trusted master circuit, but it can be arbitrarily small. See https://perso.uclouvain.be/fstandae/PUBLIS/177.pdf https://perso.uclouvain.be/fstandae/PUBLIS/177.pdf
- amelius 10y ago
  But what if the malicious code is time activated? (just an example)
  - swegg 10y ago
    This is actually addressed in the paper. Basically you can use testing to detect the timebomb, up to a negligible probability. This paper is approachable, it's understandable without too much background if you're interested in the topic.
nickpsecurity 10y ago
They don't. Too many attack vectors. I illustrate here: https://news.ycombinator.com/item?id=10468624 https://news.ycombinator.com/item?id=10468624 My solution was to print the TCB on a process node that was verifiable by eye. Then, verify a random sample of each batch. Possibly speed it up with image processing algorithms if producing the same component or components. Note: Deviations from intended circuitry in deep sub-micron can have measurable differences at analog or RF level. DARPA is funding research to do such things. A monitor at visible node could then be combined with CPU's on cutting-edge node. Common practice in commercial sector is obfuscations, though.

foobiekr 10y ago

The actual document - https://cloud.google.com/security/security-design/ https://cloud.google.com/security/security-design/ - was linked previously. It is interesting that they are doing some variant of trusted computing mostly because their homogeneity would allow Google to build a robust containment architecture with much more rigorous whitelisting and a robust SW distribution rules that go beyond what a measuring host and local SW bundle verification can do. So defense in depth. We (skyport systems) do the same thing as a service for enterprises (we sell and operate cloud-managed trusted systems as a service) and I will say it's pretty hard to get people to think about depth and trustworthiness when the entire security industry has trained CIOs to believe that all they need to do is install some random agent on their VMs. Good for Google.

gerbilly 10y ago

Still it doesn't make me want to use their services. They may indeed be really good at securing their data but 'their data' ironically is derived from my emails and browsing history and that of my friends.

tlb 10y ago

"Before a decommissioned encrypted storage device can physically leave our custody, it is cleaned using a multi-step process that includes two independent verifications. Devices that do not pass this wiping procedure are physically destroyed (e.g. shredded) on-premise" Why not just shred all decommissioned disks? Someone must be buying them for enough money that Google created a multi-step process for cleaning and verifying them. Presumably Google keeps disks in commission until they're no longer economic in their own operation. So, does anyone know about the operation that makes profitable use of disks that are no longer economic for Google?

hodgesrm 10y ago
Just speculation but they could be used elsewhere in Google for test/dev or pre-prod environments.
wang_li 10y ago
I'd imagine it's easier to recycle an intact HDD than a jumbled mix of silicon, ceramic, steel, aluminum and what not.
deleted 10y ago
[deleted]
phire 10y ago
Probably because of the verification. You can't really verify a shredding, the pile of shredded remains no-longer has a serial number. I assume the cleaning process cryptographically verifies the identify of the disk both before and after the wipe, making it impossible to sneak a drive out. For those drives which fail the cleaning process, they probably have a complex process with multiple witnesses to ensure it actually gets shredded.
willk 10y ago
My guess to the decommissioned disks is that HDD manufactures will sometimes give a hefty discount on disks if they can have them back at the end to run diagnostics on them. My company has a no disks leave the company policy and there has been talk about modifying this for the discount on disks.

bogomipz 10y ago

I was curious about this: >"There's plenty more in the document, like news that Google's public cloud runs virtual machines in a custom version of the KVM hypervisor." Does anyone know if this "container inside kvm" is true of their internal infrastructure as well or its just an extra layer of security for their public facing cloud?

wmf 10y ago
Internal Google stuff does not use KVM and that's one reason it took them a while to offer VMs — they had little experience with it.
- bogomipz 10y ago
  Do you or anyone else know if there is another reason for doing this besides security?
  - wmf 10y ago
    I can't speak for Google, but there are several reasons. Docker and k8s are not multitenant, so if you want to build a public k8s cloud you need a tenant layer under it. That layer could also be containers (e.g. LXD), but then you're talking about secure nested containers which was not really available in November 2014.
    - bogomipz 10y ago
      Oh good insight. That makes a lot of sense. Thanks.

DanielDent 10y ago

A lot of stuff from this made it's way into the chromebook. There's a verified boot process, hardware assisted key management, rollback protection, ... And it's all open source and nicely documented for anyone who cares to look. With a bit of work you can actually create your own chain of trust and run your own verified boot process. It's very cool.

knowaveragejoe 10y ago
What is this called? I'd like to look into it, but cursory searching is giving me more vague results.