7 ms·
Organizational and as a part of that access control. One of the current critical issues I see in my own work is that we have a significant amount of downstream
by Grimm1 5y ago
Organizational and as a part of that access control. One of the current critical issues I see in my own work is that we have a significant amount of downstream consumers of our data and each one needs access at a different layer.
This means someone may need access to raw data, data after it's been cleaned and put in the data lake, data in an intermediate state at some point in the transformation chain, pristine fully joined master records for some data in the warehouse. Some teams shouldn't have access to certain types of data, or data earlier and later in the stack.
There's not a central way to do access control at a fine grained (row in some cases, files in others etc) level across layers and multiple services/dbs in the data stack. At least no plug and play way from what I can see. I'll be in the midst of building that layer for an entire enterprise organization soon and I expect it to be enlightening.
I'm at the same time putting in some leg work to explore how common this problem is across other companies right now for obvious reasons.