6 ms·
I'm also wondering why this kind of intervention is necessary at all. The NoSQL solution we use at work has load based automatic splitting, and I'd have thought
by houseabsolute 16y ago
I'm also wondering why this kind of intervention is necessary at all. The NoSQL solution we use at work has load based automatic splitting, and I'd have thought (though I haven't confirmed) that this would be an obvious feature to include.
- icey 16y agoWhich NoSQL store do you use?
- houseabsolute 16y agohttp://labs.google.com/papers/bigtable.html http://labs.google.com/papers/bigtable.html
- staunch 16y agoThat looks potentially useful in some cases. Can you send me the code/docs for that?
- ddlatham 16y agohttp://hbase.apache.org/ http://hbase.apache.org/
- ifesdjeen 16y agoDoes it work that well for production for you? Really, that's extremely interesting.
- rb2k_ 16y agoAlso look at dynamo based systems like cassandra and riak (riak seems to have a better load balancing at the moment, cassandra is a bit more "bumpy")
- dmytton 16y agoLoad based splitting is on the MongoDB roadmap, but doesn't exist yet.
- fizx 16y agoJust because the feature exists, doesn't mean it works.
- mcfunley 16y agoI would speculate that it's a poorly-chosen shard key. MongoDB's built-in sharding uses range-based indexing. If you choose user_id as your shard key, and those are autoincrementing integers, then you're screwed if newer users tend to be more active on average than older ones.
- jrockway 16y agoWait, people shard on a key other than something approximately random, like an sha1 hash!?
- mcfunley 16y agoWhere I work (Etsy) we keep an index server that maps each user to a shard on an individual basis. There are a number of advantages to it. For example, if one user generated a ton of activity they could in theory be moved to their own server. Approximately random works for the initial assignment. Flickr works the same way (not by coincidence, since we have several former Flickr engineers on staff).
- jrockway 16y agoSounds like a good system. I've noticed that people tend to do things like shard based on even/odd, and then they realize that they need three databases. I've never had either problem though... but if I ever need to shard I plan on doing it based on object ID. Then one request can be handled by multiple databases, "for free", increasing both throughput and response time.
- fizx 16y agoEven/odd isn't the end of the world, but you would then be best jumping to mod 4.
- moe 16y agoActually for anonymous sharding (without a central index) a consistent hash is about the closest you can get to ideal distribution and flexibility. I haven't looked but I presume that's what mongo uses under the hood for their auto-sharding, too.