6 ms·
user specific topics isn't the right path, conventional wisdom would be that you want < 1000s of topics. It looks like that limit has been changed more recentl
by rubyissimo 3y ago
user specific topics isn't the right path, conventional wisdom would be that you want < 1000s of topics.
It looks like that limit has been changed more recently: https://stackoverflow.com/questions/32950503/can-i-have-100s-of-thousands-of-topics-in-a-kafka-cluster https://stackoverflow.com/questions/32950503/can-i-have-100s... but I'm still skeptical it's what you want. You'd need a consumer for each of them, you're tracking offsets for all of them.
Much more normal would be a single topic here, partitioned on user.
You can have 2 consumers then. One that pulls things off and notifies. Another one that pulls things off and stores to the DB, which is indexed by user ID.
Then when they read a notification, you flip a bit in the DB for the notifications.
- BWStearns 3y agoPulsar is similar (and I think API compatible? but I might be remembering wrong) but has effectively no limit on topics. Maybe appropriate for their usecase depending on number of users?