6 ms·
So how do they determine whether a user has viewed a post already? I would think that unique counting is accomplished using the hyperloglog counter, but the art
by alzaeem 9y ago
So how do they determine whether a user has viewed a post already? I would think that unique counting is accomplished using the hyperloglog counter, but the article says that this decision is made by the Nazar system, which doesn't use the hyperloglog counter in Redis.
- hrshtr 9y agoThats true, I am thinking that Nazar is more like spam filter and monitors the user behavior.
- kchandra 9y agoPretty much, yeah.
- lucasschm 9y agoBloom Filters? It has false positives but no false negatives
- jimmaswell 9y agoWhy can't they just associate a list of viewed posts with each user, or list of users that viewed a post with each post, and check that? I don't get why this needs any consideration.
- eropple 9y agoHave you stopped to think how many users that is and how many posts? Viewing a single thread could require five hundred associations.
- jimmaswell 9y agoAnd it already requires reading five hundred comments.
- sethammons 9y agoThey addressed your second point in the article. On a popular post, you would be storing several megabytes of data to capture/relate each unique user that visited. That gets expensive at scale. HLL takes then down to a few kilobytes, less than 1% of the original size. For your first suggestion, you would have to do a very expensive look up. You couldn't cache it effectively due to the requirement of near real time stats. You could improve look up time using columnar storage, but the performance and memory usage will be nowhere near as nice as with HLL. Problems are harder at scale.
- eropple 9y agoI've had a "phases of computing" article percolating for a while to this end. Problems aren't just harder at scale, but they actively change their observable properties because of the stressors involved and where they crop up.