20 ms·

TL;DR: In the DigitalOcean web panel you can check the "scrub data" checkbox when destroying a VM. When using the API this option is not ticked. This can lead t

by xSwag 13y ago

TL;DR: In the DigitalOcean web panel you can check the "scrub data" checkbox when destroying a VM. When using the API this option is not ticked. This can lead to other customers being able to retrieve your data. The author thinks that this is a security issue because this option should be enabled by default. However, (I assume) it's not in Digital Oceans interest to do full disk scrub because it reduces the lifespan of their SSD. If a user forgets to log out of Facebook on a public computer, is it Facebook's responsibility? Similarly, if a user does not correctly delete data on a budget host, is it the hosts fault?

sneak 13y ago
I think it's less in Digital Ocean's interest to silently give customer data to a subsequent user of that block device?
- jlawer 13y ago
  Why? If customers don't care enough about the data why should DO? DO is a budget provider. Their main hook was cheap SSDs, and is still cheap costs. They billable time it takes to wipe a VM adds to the cost. If you want the data zeroed... then pay for it, otherwise it must not be important enough.
  - coolj 13y ago
    DO is essentially saying they're using insecure defaults by design, which is a Really Bad Idea, even if it prima facie saves time/money. It's relatively easy for them to post a point in a FAQ about why you're billed for x minutes after an instance is destroyed, and include the request params/headers to avoid it. On the other hand, it's effectively impossible to unleak an inadvertently leaked apple developer signing key because an unwitting customer didn't rtfm.
Stealth- 13y ago
The term "disc scrub" makes little to no sense in terms of SSDs, which makes this issue even more complicated. People who work in the disc recovery field have been dealing with this since SSDs became popular. An SSD is limited by its number of writes. To compensate for this, the SSD has very complicated on board logic that abstracts the actual SSD away from what it tells the OS system. This allows it to do certain tricks to save writes. However, when you are "scrubbing" an SSD, internally the SSD might be writing somewhere else entirely. Scrubbing is not considered an effective way of wiping SSDs, from what I believe.
- eurleif 13y ago
  Ok, so the data isn't necessarily destroyed immediately after a scrub. But how does that play out at the level of VM users? Is there a normal usage scenario where the portion of the SSD containing my deleted data is made readable to someone else's VM, or will it be inaccessible to normal VM users until it's overwritten?
  - Stealth- 13y ago
    I'm not an expert on this, but I believe that at the VM user level they would see wiped data because of the internal mapping. I think physical analysis of the drive would be required.
- mikeash 13y ago
  There is a vast difference between writing out zeroes to the SSD but still having some of the original data potentially persist on the SSD but unreachable without special techniques, and not zeroing out the SSD and giving the device to a new VM and letting it trivially access everything that was previously there. If I can provision a new VM and cat /dev/vda and see data from the VM that previously occupied that spot, then you are doing it horribly, horribly, horribly wrong. That zeroing out the data leaves open a different and vastly more difficult attack path doesn't make that any less true.
nwh 13y ago
If it was a lifespan issue they could just encrypt the partitions (minimal overhead with AES) and then destroy the key with the droplet. Due to the way that SSDs wear level, even if every block returns 0, you don't know if that's the same block or if the real data is squirrelled away on out-of-life blocks. From what I understand the SSD has a lot more capacity than advertised and moves the blocks forward as sections of the chip wear out. Assuming you could reprogram the controller (or use a different one), you could go back and read the old blocks and recover data in the clear.
- pilsetnieks 13y ago
  Not several times more capacity, it's generally about 7% (that's why you often see 120 GB or 240 GB drives that are in reality 128 GB or 256 GB.)
- spullara 13y ago
  You're right. Encryption is the right answer. What does Amazon do with their SSDs? Anyone know?
  - jlawer 13y ago
    I would expect any sane large provider to zero the disks simply to make it easier to overcommit on the storage layer.
- rythie 13y ago
  The problem with encryption at that level is that trim no longer works as there are no longer any zeroed blocks - so writes are slower. Also there will be more write applification as every write will cause a block to be erased.
  - quotemstr 13y ago
    Of course it works. The filesystem knows which blocks the underlying storage device can forget and tell it using TRIM. That the content of the block was encrypted is irrelevant. TRIM is for putting blocks in some undefined state when no longer needed, not for zeroing them. See http://worldsmostsecret.blogspot.com/2012/04/how-to-activate-trim-on-luks-encrypted.html http://worldsmostsecret.blogspot.com/2012/04/how-to-activate...
sneak 13y ago
> Similarly, if a user does not correctly delete data on a budget host, is it the hosts fault? What does the verb "destroy" mean? Screenshot taken today (30DEC13) from a fresh blank Digital Ocean server I just provisioned: http://i.imgur.com/fJOxRN9.png http://i.imgur.com/fJOxRN9.png
- deleted 13y ago
  [deleted]
- EdwardDiego 13y ago
  What are we looking at in your screenshot? I've been Where's-Waldoing it for the word 'destroy' but can't find it.
  - sneak 13y ago
    That's someone else's data (three someone elses' if you count the iPhone end users) read from the root block device on a minutes-old brand new fresh Digital Ocean VM that I got from them for a $5 PayPal payment. It had been mkfs'd but not zeroed. Command was: apt-get -y install binutils ; dd if=/dev/vda bs=1M | strings -n 100 | grep 2013-12 The destroy api call docs are here: https://developers.digitalocean.com https://developers.digitalocean.com (It's the /droplets/[droplet_id]/destroy one.)
  - coolj 13y ago
    I think he's showing that the data is not "destroyed" in the sense it still exists. Destroy in a virt context doesn't necessarily mean "destroy all the resources associated with a VM"--I don't know about DO's product offering, but at the hypervisor level, at least with Xen and libvirt, you often want to "destroy" the instance (forcibly terminate/undefine from the hypervisor) and leave the resources (storage pools/devices, IP pools/addresses, network flows/filters etc). I think focusing on the word "destroy" is a bit of a canard; the real problem is insecure defaults wrt block device scrubbing when you issue an API "destroy" (which wouldn't be any better if it was called "delete" or "undefine").
- icmp 13y ago
  nice post - also #fmc says hi ;)
rgbrenner 13y ago
Wait a second. Why does this option exist at all? Are you saying there are people who DO NOT WANT their data destroyed when they destroy a VM? Who are these people?
- deleted 13y ago
  [deleted]
- jd007 13y ago
  It's probably there for DO's sake. SSDs have limited rewrite cycles, and each secure wipe will definitely shorten the lifespan of the drive much more than a simple quick wipe. This translates into cost savings.
  - thirsteh 13y ago
    Worst cost-saving measure ever.
  - switch007 13y ago
    Yes. This is a shitty default peddled as a benefit for the customer, when in fact it benefits DO.
    - sneak 13y ago
      No other provider charges you extra to erase your data when tearing down an instance. It's called a dark pattern.
      - dmxt 13y ago
        Most other provider use hard drives and they're different from SSD.
      - bluefinity 13y ago
        Okay, the misinformation is getting a little out of hand here. DigitalOcean certainly aren't charging you to erase your data, all you have to do is check the "Scrub Data" checkbox. Yes, you have to pay for the time your VPS is on while erasing, but that's perfectly reasonable.
tombrossman 13y ago
I'm surprised I didn't see anyone raise the issue of Data Protection given DO's European presence. This comment below [1] appears to confirm that customer data is mishandled in violation of EU Data Protection law. When you ask 'is it the hosts fault?' I think the answer is a most definite YES. Surely some lawyer out there took a look at this, so maybe I have missed something, but this looks like a big problem to me. From my own experience using DO I can say I'm a happy customer and I plan to keep using them. I tick that box when I've used anything remotely sensitive in a VM when destroying it, and leave it empty other times (like when I've made a mess experimenting with something and want to quickly trash & re-create a droplet). [1] https://news.ycombinator.com/item?id=6983260 https://news.ycombinator.com/item?id=6983260
- AlexHamilton 13y ago
  DO is a data processor under EU data protection law, while the customer would be the data controller. EU data protection law currently (it will change with the new regulations) only imposes legal duties on the data controller. As such, it is the customer's legal problem if it (or its data processor) has failed to handle personal data correctly.
nhaehnle 13y ago
They don't have to do a full disk scrub on the physical SSD to fix this security problem. All they have to do is what every sane sparse disk image implementation in the history of mankind has done: catch reads from unwritten blocks in software and return a block of all zeros.
drzaiusapelord 13y ago
The pro-DO bias of HN is showing again. If this was Amazaon or Linode, there would be endless hand-wringing and calls for lawsuits. When its DO doing something incredibly stupid and short-sighted, its 'blame the customer.' This is penny pinching at the cost of security, which is something we often complain about when it comes to other companies. Why does DO get a free pass? "Oh, there's an option for that" to stop default dangerous behavior should not be excusable. This is further proof on why DO isn't enterprise ready and still a toy for barely stable dotcoms. Thank god for EU privacy laws. If these guys have any EU presence, they'll be forced to clean up their act via regulation. Clearly, the invisible hand of the market and "there's a option for that" is the fail its always been in regards to security.
- warrenm 13y ago
  They have a datacenter in Amsterdam
- crohrc2 13y ago
  Glad somebody pointed this out. It's also hilarious to me that we were all screaming for heads with the GD (mt) acquisition a few months back, and this is just a "pat on the back" type of thing. Their growth and offerings are great, but they were going to run into problems sooner or later. As we've seen with many other hosts in the past, this is probably just the beginning.
- ahallock 13y ago
  I was with you until you started talking about the "invisible hand of the free market" failing and how instead we should be using State force to solve these problems. Isn't your comment a free market action, raising awareness about this problem? Won't this thread cause DO to take action? Say it doesn't. There are plenty of other hosting companies to switch to. And also there are times when I might be willing to trade security for convenience, price, practicality, etc. As a consumer, I want this choice. I think the free market is working fine. You can't bully your way to good security.

by xSwag 13y ago

sneak 13y ago

I think it's less in Digital Ocean's interest to silently give customer data to a subsequent user of that block device?

jlawer 13y ago
Why? If customers don't care enough about the data why should DO? DO is a budget provider. Their main hook was cheap SSDs, and is still cheap costs. They billable time it takes to wipe a VM adds to the cost. If you want the data zeroed... then pay for it, otherwise it must not be important enough.
- coolj 13y ago
  DO is essentially saying they're using insecure defaults by design, which is a Really Bad Idea, even if it prima facie saves time/money. It's relatively easy for them to post a point in a FAQ about why you're billed for x minutes after an instance is destroyed, and include the request params/headers to avoid it. On the other hand, it's effectively impossible to unleak an inadvertently leaked apple developer signing key because an unwitting customer didn't rtfm.

Stealth- 13y ago

The term "disc scrub" makes little to no sense in terms of SSDs, which makes this issue even more complicated. People who work in the disc recovery field have been dealing with this since SSDs became popular. An SSD is limited by its number of writes. To compensate for this, the SSD has very complicated on board logic that abstracts the actual SSD away from what it tells the OS system. This allows it to do certain tricks to save writes. However, when you are "scrubbing" an SSD, internally the SSD might be writing somewhere else entirely. Scrubbing is not considered an effective way of wiping SSDs, from what I believe.

eurleif 13y ago
Ok, so the data isn't necessarily destroyed immediately after a scrub. But how does that play out at the level of VM users? Is there a normal usage scenario where the portion of the SSD containing my deleted data is made readable to someone else's VM, or will it be inaccessible to normal VM users until it's overwritten?
- Stealth- 13y ago
  I'm not an expert on this, but I believe that at the VM user level they would see wiped data because of the internal mapping. I think physical analysis of the drive would be required.
mikeash 13y ago
There is a vast difference between writing out zeroes to the SSD but still having some of the original data potentially persist on the SSD but unreachable without special techniques, and not zeroing out the SSD and giving the device to a new VM and letting it trivially access everything that was previously there. If I can provision a new VM and cat /dev/vda and see data from the VM that previously occupied that spot, then you are doing it horribly, horribly, horribly wrong. That zeroing out the data leaves open a different and vastly more difficult attack path doesn't make that any less true.

nwh 13y ago

If it was a lifespan issue they could just encrypt the partitions (minimal overhead with AES) and then destroy the key with the droplet. Due to the way that SSDs wear level, even if every block returns 0, you don't know if that's the same block or if the real data is squirrelled away on out-of-life blocks. From what I understand the SSD has a lot more capacity than advertised and moves the blocks forward as sections of the chip wear out. Assuming you could reprogram the controller (or use a different one), you could go back and read the old blocks and recover data in the clear.

pilsetnieks 13y ago
Not several times more capacity, it's generally about 7% (that's why you often see 120 GB or 240 GB drives that are in reality 128 GB or 256 GB.)
spullara 13y ago
You're right. Encryption is the right answer. What does Amazon do with their SSDs? Anyone know?
- jlawer 13y ago
  I would expect any sane large provider to zero the disks simply to make it easier to overcommit on the storage layer.
rythie 13y ago
The problem with encryption at that level is that trim no longer works as there are no longer any zeroed blocks - so writes are slower. Also there will be more write applification as every write will cause a block to be erased.
- quotemstr 13y ago
  Of course it works. The filesystem knows which blocks the underlying storage device can forget and tell it using TRIM. That the content of the block was encrypted is irrelevant. TRIM is for putting blocks in some undefined state when no longer needed, not for zeroing them. See http://worldsmostsecret.blogspot.com/2012/04/how-to-activate-trim-on-luks-encrypted.html http://worldsmostsecret.blogspot.com/2012/04/how-to-activate...

sneak 13y ago

> Similarly, if a user does not correctly delete data on a budget host, is it the hosts fault? What does the verb "destroy" mean? Screenshot taken today (30DEC13) from a fresh blank Digital Ocean server I just provisioned: http://i.imgur.com/fJOxRN9.png http://i.imgur.com/fJOxRN9.png

deleted 13y ago
[deleted]
EdwardDiego 13y ago
What are we looking at in your screenshot? I've been Where's-Waldoing it for the word 'destroy' but can't find it.
- sneak 13y ago
  That's someone else's data (three someone elses' if you count the iPhone end users) read from the root block device on a minutes-old brand new fresh Digital Ocean VM that I got from them for a $5 PayPal payment. It had been mkfs'd but not zeroed. Command was: apt-get -y install binutils ; dd if=/dev/vda bs=1M | strings -n 100 | grep 2013-12 The destroy api call docs are here: https://developers.digitalocean.com https://developers.digitalocean.com (It's the /droplets/[droplet_id]/destroy one.)
- coolj 13y ago
  I think he's showing that the data is not "destroyed" in the sense it still exists. Destroy in a virt context doesn't necessarily mean "destroy all the resources associated with a VM"--I don't know about DO's product offering, but at the hypervisor level, at least with Xen and libvirt, you often want to "destroy" the instance (forcibly terminate/undefine from the hypervisor) and leave the resources (storage pools/devices, IP pools/addresses, network flows/filters etc). I think focusing on the word "destroy" is a bit of a canard; the real problem is insecure defaults wrt block device scrubbing when you issue an API "destroy" (which wouldn't be any better if it was called "delete" or "undefine").
icmp 13y ago
nice post - also #fmc says hi ;)

rgbrenner 13y ago

Wait a second. Why does this option exist at all? Are you saying there are people who DO NOT WANT their data destroyed when they destroy a VM? Who are these people?

deleted 13y ago
[deleted]
jd007 13y ago
It's probably there for DO's sake. SSDs have limited rewrite cycles, and each secure wipe will definitely shorten the lifespan of the drive much more than a simple quick wipe. This translates into cost savings.
- thirsteh 13y ago
  Worst cost-saving measure ever.
- switch007 13y ago
  Yes. This is a shitty default peddled as a benefit for the customer, when in fact it benefits DO.
  - sneak 13y ago
    No other provider charges you extra to erase your data when tearing down an instance. It's called a dark pattern.
    - dmxt 13y ago
      Most other provider use hard drives and they're different from SSD.
    - bluefinity 13y ago
      Okay, the misinformation is getting a little out of hand here. DigitalOcean certainly aren't charging you to erase your data, all you have to do is check the "Scrub Data" checkbox. Yes, you have to pay for the time your VPS is on while erasing, but that's perfectly reasonable.

tombrossman 13y ago

I'm surprised I didn't see anyone raise the issue of Data Protection given DO's European presence. This comment below [1] appears to confirm that customer data is mishandled in violation of EU Data Protection law. When you ask 'is it the hosts fault?' I think the answer is a most definite YES. Surely some lawyer out there took a look at this, so maybe I have missed something, but this looks like a big problem to me. From my own experience using DO I can say I'm a happy customer and I plan to keep using them. I tick that box when I've used anything remotely sensitive in a VM when destroying it, and leave it empty other times (like when I've made a mess experimenting with something and want to quickly trash & re-create a droplet). [1] https://news.ycombinator.com/item?id=6983260 https://news.ycombinator.com/item?id=6983260

AlexHamilton 13y ago
DO is a data processor under EU data protection law, while the customer would be the data controller. EU data protection law currently (it will change with the new regulations) only imposes legal duties on the data controller. As such, it is the customer's legal problem if it (or its data processor) has failed to handle personal data correctly.

nhaehnle 13y ago

They don't have to do a full disk scrub on the physical SSD to fix this security problem. All they have to do is what every sane sparse disk image implementation in the history of mankind has done: catch reads from unwritten blocks in software and return a block of all zeros.

drzaiusapelord 13y ago

The pro-DO bias of HN is showing again. If this was Amazaon or Linode, there would be endless hand-wringing and calls for lawsuits. When its DO doing something incredibly stupid and short-sighted, its 'blame the customer.' This is penny pinching at the cost of security, which is something we often complain about when it comes to other companies. Why does DO get a free pass? "Oh, there's an option for that" to stop default dangerous behavior should not be excusable. This is further proof on why DO isn't enterprise ready and still a toy for barely stable dotcoms. Thank god for EU privacy laws. If these guys have any EU presence, they'll be forced to clean up their act via regulation. Clearly, the invisible hand of the market and "there's a option for that" is the fail its always been in regards to security.

warrenm 13y ago
They have a datacenter in Amsterdam
crohrc2 13y ago
Glad somebody pointed this out. It's also hilarious to me that we were all screaming for heads with the GD (mt) acquisition a few months back, and this is just a "pat on the back" type of thing. Their growth and offerings are great, but they were going to run into problems sooner or later. As we've seen with many other hosts in the past, this is probably just the beginning.
ahallock 13y ago
I was with you until you started talking about the "invisible hand of the free market" failing and how instead we should be using State force to solve these problems. Isn't your comment a free market action, raising awareness about this problem? Won't this thread cause DO to take action? Say it doesn't. There are plenty of other hosting companies to switch to. And also there are times when I might be willing to trade security for convenience, price, practicality, etc. As a consumer, I want this choice. I think the free market is working fine. You can't bully your way to good security.