5 ms·
It's not about splitting the task into smaller pieces. It's about factoring out the parts of the task that need synchronization among all machines into one spec
by alextp 14y ago
It's not about splitting the task into smaller pieces. It's about factoring out the parts of the task that need synchronization among all machines into one specific subroutine (groupBy) which makes mapreduce so powerful.
If you speak with people experienced in multithreaded and distributed programming you will see that synchronization with fault-tolerance is _hard_, and mapreduce provides a widely-applicable set of sufficient conditions for an algorithm to be executable with implicit fault-tolerance and implicit synchronization.
Without mapreduce-like abstractions eveyr piece of software has to be responsible for its own (1) checkpointing (to recover from errors), (2) checksumming (to ensure that no errors happened), and (3) distributed communication (to make sure the global state becomes global and the local state becomes local).
- fa_il 14y agoBecause you consider a problem to be "hard" and because someone else has proferred a solution that previously was not obvious to you does not lead me to the conclusion that map-reduce is something new (nor the result of ground-breaking research). It leads me to the conclusion that it is helpful to you. Those are to me two different things.