4 ms·

Soft deletion is certainly very situationally worth it. I've found the most value when 1. it is well supported at the ORM layer and 2. business requirements dic

Soft deletion is certainly very situationally worth it. I've found the most value when 1. it is well supported at the ORM layer and 2. business requirements dictate strong auditability of data. While I have undeleted items on occasion, I've used soft deletes more frequently to debug and build a timeline of events around the data. For context, I've worked in fintech where I often needed to review backoffice approvals, transactions, offers, etc.

krstf13 4y ago
Wouldn’t storing the deleted data in an immutable storage, with time stamp, be much better for auditability ? I mean how could you audit deleted, restored and deleted again data with that setup? Also, while I know it’s not really accurate, I tend to understand relations as sets, it makes me uncomfortable to have soft deleted data that are neither member or not member of the set.
chomp 4y ago
Yep, we have an abstraction layer on top of the ORM to provide common queries. "Give me all X" will always return stuff not soft deleted. Data people also like to go diving through old data, and without getting into data warehousing and stuff like that, it's not too complex to support a single flag to enable us to keep old stuff.
- delusional 4y ago
  They might like to, but you should definitely consider if saving data no longer required for your business violates privacy regulation/ethical considerations.
  - chomp 4y ago
    Definitely. When a user "deletes" their account, we null out all identifying fields to "DELETED_PII_$user_id". We have running metrics we compute that would go off the rails if we dropped the row completely.
hinkley 4y ago
In my limited experience, soft deletion also has better prospects where partial indexes are involved, since it reduces the size of the index and reduces search and insert time a little bit. If soft deletes are rare, you aren't going to see much of a payback for your investment in code complexity. And since you can never really be sure what you'll need 2 years from now, I imagine there are a lot of anecdotes out there of people who implemented it thinking it would be used a lot, and turned out to be wrong.

krstf13 4y ago

Wouldn’t storing the deleted data in an immutable storage, with time stamp, be much better for auditability ? I mean how could you audit deleted, restored and deleted again data with that setup? Also, while I know it’s not really accurate, I tend to understand relations as sets, it makes me uncomfortable to have soft deleted data that are neither member or not member of the set.

chomp 4y ago

Yep, we have an abstraction layer on top of the ORM to provide common queries. "Give me all X" will always return stuff not soft deleted. Data people also like to go diving through old data, and without getting into data warehousing and stuff like that, it's not too complex to support a single flag to enable us to keep old stuff.

delusional 4y ago
They might like to, but you should definitely consider if saving data no longer required for your business violates privacy regulation/ethical considerations.
- chomp 4y ago
  Definitely. When a user "deletes" their account, we null out all identifying fields to "DELETED_PII_$user_id". We have running metrics we compute that would go off the rails if we dropped the row completely.

hinkley 4y ago

In my limited experience, soft deletion also has better prospects where partial indexes are involved, since it reduces the size of the index and reduces search and insert time a little bit. If soft deletes are rare, you aren't going to see much of a payback for your investment in code complexity. And since you can never really be sure what you'll need 2 years from now, I imagine there are a lot of anecdotes out there of people who implemented it thinking it would be used a lot, and turned out to be wrong.