8 ms·
1 billion is small for hadoop?
by jalino23 3y ago
1 billion is small for hadoop?
- brokensegue 3y ago~14GB file? it's on the small side for hadoop
- rapsey 3y agoIf it fits on one computer it's not a hadoop problem.
- quickthrower2 3y agoIt fits on a dusty ten year old USB stick
- badgersnake 3y agoSounds like an awk problem tbh.
- 8organicbits 3y agoA Hadoop submission may help people realize that. But since you only have one machine to work with it should be obvious that you're not going to get any speed-up via divide and conquer.
- chmod775 3y agoAnything that fits in RAM on one machine is easily too small for Hadoop. In those cases, the overhead of Hadoop is going to make it get destroyed by a single beefy machine. The only times where this might not be the case is when you're doing a crazy amount of computation relative to the data you have. Note that you can easily reach 1TB of RAM on (enterprise) commodity hardware now, and SSDs are pretty fast too. Old but gold post from 2014: https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html https://adamdrake.com/command-line-tools-can-be-235x-faster-...
- makapuf 3y agoAlso from 2013: https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html