5 ms·

I'm pretty sure they use some sort of event store for their backend in which case all versions/changes/updates/deletes are stored as a separate revision alongsi

I'm pretty sure they use some sort of event store for their backend in which case all versions/changes/updates/deletes are stored as a separate revision alongside the original content.

echelon 7y ago
I suspect this is the case as well. All the big companies are doing immutable, append-only event logging and probably have no mechanism to expunge this data. All because storage is cheap and they need to hold on to everything for testing or whatever future need that might arise. What a pitiful dystopia we're living in.
- kerng 7y ago
  GDPR required companies to add features to append only systems to allow for true deletion (usually with something like a 30 day delay). Not sure how many actually did that, but to be compliant append only does not fly. I was on a product that did exactly that for data lakes.
  - wybiral 7y ago
    I wonder how that works with Facebook's Blu-Ray cold storage [1]. Are optical disks treated like paper documents or is it still considered electronic storage? [1] http://www.digitalpreservation.gov/meetings/documents/storage14/Kestutis_Patiejunas_Facebook_FreezingExabytesOfDataFacebooksColdStorage.pdf http://www.digitalpreservation.gov/meetings/documents/storag...
    - kerng 7y ago
      Ah, that's interesting. If they store customer data on BlueRay disks I assume Facebook took the necessary steps to delete/destroy records according to GDPR requirements when customers request deletion....
- jefftk 7y ago
  > All the big companies are doing immutable, append-only event logging and probably have no mechanism to expunge this data. All because storage is cheap and they need to hold on to everything for testing or whatever future need that might arise. This is not true at all. Here's a recent HN discussion that talks about how Google handles this: https://news.ycombinator.com/item?id=19809259 https://news.ycombinator.com/item?id=19809259 (Disclosure: I work for Google, but I'm speaking for myself)
  - VectorLock 7y ago
    So according to you Google will propagate deletes, but you don't know how/if its deleted from cold/offline/long term storage?
    - jefftk 7y ago
      I'm very confident that the data is fully removed, because properly deleting data within NN days is treated as very serious internally. But I don't know the details of how it's done for cold storage. (I would love to see someone subpoena something deleted, say, 1y ago and write up whether it was produced.)

I'm pretty sure they use some sort of event store for their backend in which case all versions/changes/updates/deletes are stored as a separate revision alongside the original content.

echelon 7y ago

I suspect this is the case as well. All the big companies are doing immutable, append-only event logging and probably have no mechanism to expunge this data. All because storage is cheap and they need to hold on to everything for testing or whatever future need that might arise. What a pitiful dystopia we're living in.

kerng 7y ago
GDPR required companies to add features to append only systems to allow for true deletion (usually with something like a 30 day delay). Not sure how many actually did that, but to be compliant append only does not fly. I was on a product that did exactly that for data lakes.
- wybiral 7y ago
  I wonder how that works with Facebook's Blu-Ray cold storage [1]. Are optical disks treated like paper documents or is it still considered electronic storage? [1] http://www.digitalpreservation.gov/meetings/documents/storage14/Kestutis_Patiejunas_Facebook_FreezingExabytesOfDataFacebooksColdStorage.pdf http://www.digitalpreservation.gov/meetings/documents/storag...
  - kerng 7y ago
    Ah, that's interesting. If they store customer data on BlueRay disks I assume Facebook took the necessary steps to delete/destroy records according to GDPR requirements when customers request deletion....
jefftk 7y ago
> All the big companies are doing immutable, append-only event logging and probably have no mechanism to expunge this data. All because storage is cheap and they need to hold on to everything for testing or whatever future need that might arise. This is not true at all. Here's a recent HN discussion that talks about how Google handles this: https://news.ycombinator.com/item?id=19809259 https://news.ycombinator.com/item?id=19809259 (Disclosure: I work for Google, but I'm speaking for myself)
- VectorLock 7y ago
  So according to you Google will propagate deletes, but you don't know how/if its deleted from cold/offline/long term storage?
  - jefftk 7y ago
    I'm very confident that the data is fully removed, because properly deleting data within NN days is treated as very serious internally. But I don't know the details of how it's done for cold storage. (I would love to see someone subpoena something deleted, say, 1y ago and write up whether it was produced.)