6 ms·
On modern hardware with many cpu cores you can use a similar process of fork and join to maximise throughput of large datasets.
by markuskobler 13y ago
On modern hardware with many cpu cores you can use a similar process of fork and join to maximise throughput of large datasets.
- collyw 13y agoThat sounds more like parallelisation rather than a use case for NoSQL. Why is that any better than all of your data on one database server, and each cluster node querying for part of the data to process it? Obviously there will be a bottleneck if all nodes try to access the database at the same time, but I see no benefit otherwise, and depending on the data organisation, I don't even see NoSQL solving that problem (you are going to have to separate the data to different servers for the NoSQL solution, why is that any better than cached query from a central server?).
- Roboprog 13y agoForks/threads on (e.g.) 12 core CPUs works up to a point. But that point probably does solve many problems without further complication :-)
- virtuabhi 13y agoSingle hardware with many cores does not give the same performance as multiple machines. For example, consider disk throughput. If the data is striped across multiple nodes then the read request can be executed in parallel, resulting in linear speed up! In a single machine you have issues of cache misses, inefficient scatter-gather operations in main memory, etc. And it is much more easier to let the MapReduce framework handle parallelism than writing error prone code with locks/threads/mpi/architecture-dependent parallelism etc.