5 ms·

Does anyone on HN have any first-hand experience configuring and using Hadoop? If so, how was the process - was it straightfoward or complex, did it require lot

Does anyone on HN have any first-hand experience configuring and using Hadoop? If so, how was the process - was it straightfoward or complex, did it require lots of peeking into the source, any major pitfalls, etc - and, are you satisfied with how it's currently working and with your ability to add new boxes easily?

strlen 18y ago
Hi, I've setup a hadoop cluster for a start-up I am working in. Initial reason? The amount of incoming log data was getting too slow to process on a single machine and having had a single graduate course in distributed systems I knew a) what is required to go from a single script to a system distributed across nodes b) that I am not in the capacity (alone) to create this system in a time window that will meet business needs. As a result, I saw hadoop as offering a solution to: implement a distribute process where by records are processes in parallel by multiple machines without having to worry about implementing locking and message passing; distribute disk seek times across a cluster of nodes, pool together disk space with ability to add disk space by simply adding new commodity machines to a cluster. If you have these specific needs: a) need to rapidly and robustly process more data than a single machine can handle (think of it this way -- if you do data processing with a Perl script on a single machine, will the data be simply too old by the time the Perl script is done?) b) ability to add storage for new data without powering machines down c) the data is non-relational (i.e. you aren't exporting data from MySQL and then re-importing it again - if your data can be stored effectively inside a MySQL cluster and queried in a relational manner, you don't need to use hadoop). Now, if you still think you need hadoop (in most of the cases, answer is "you don't"): To get hadoop going minimally to be useful right away is actually almost trivial. However, as your dataset grows, the amount of jobs and the size of the cluster grow as well as the expectation of hadoop to also be a reliable data store grow, you will run into plateaus that you'll need to overcome. To answer your questions directly: - Getting it going is easy, trivial. Do it first on your desktop (if you run Linux directly, if not under VMWare). Process is straight forward. - Peeking into the source is not required to get it going - Pitfalls: * Use enough memory, JVM is a memory hog. Always make sure you have sufficient swap (4x your memory) * Find the optimal size of input data for your task (if you are using compressed files for input). (Yes, I will write these learnings up eventually).
- strlen 18y ago
  Posted some thoughts on what problems are appropriate for Hadoop here: http://weblog.strlen.net/?p=8 http://weblog.strlen.net/?p=8 (Another post on parameter settings and machine configurations for Hadoop cluster in the making, once that's done I'll post to news.y.c)
- bporterfield 18y ago
  Great reply. My current thinking for the project I'm working on is that I don't need Hadoop...yet. That said, if it's trivial to implement on a single machine it might be wise to just begin the project in Hadoop so that I don't have to make the switch later. Always balance between 'premature optimization is the root of all evil' and 'choosing the right tools for the right job'. I'll give it a shot on my desktop to start - thanks for the tips!

strlen 18y ago

Hi, I've setup a hadoop cluster for a start-up I am working in. Initial reason? The amount of incoming log data was getting too slow to process on a single machine and having had a single graduate course in distributed systems I knew a) what is required to go from a single script to a system distributed across nodes b) that I am not in the capacity (alone) to create this system in a time window that will meet business needs. As a result, I saw hadoop as offering a solution to: implement a distribute process where by records are processes in parallel by multiple machines without having to worry about implementing locking and message passing; distribute disk seek times across a cluster of nodes, pool together disk space with ability to add disk space by simply adding new commodity machines to a cluster. If you have these specific needs: a) need to rapidly and robustly process more data than a single machine can handle (think of it this way -- if you do data processing with a Perl script on a single machine, will the data be simply too old by the time the Perl script is done?) b) ability to add storage for new data without powering machines down c) the data is non-relational (i.e. you aren't exporting data from MySQL and then re-importing it again - if your data can be stored effectively inside a MySQL cluster and queried in a relational manner, you don't need to use hadoop). Now, if you still think you need hadoop (in most of the cases, answer is "you don't"): To get hadoop going minimally to be useful right away is actually almost trivial. However, as your dataset grows, the amount of jobs and the size of the cluster grow as well as the expectation of hadoop to also be a reliable data store grow, you will run into plateaus that you'll need to overcome. To answer your questions directly: - Getting it going is easy, trivial. Do it first on your desktop (if you run Linux directly, if not under VMWare). Process is straight forward. - Peeking into the source is not required to get it going - Pitfalls: * Use enough memory, JVM is a memory hog. Always make sure you have sufficient swap (4x your memory) * Find the optimal size of input data for your task (if you are using compressed files for input). (Yes, I will write these learnings up eventually).

strlen 18y ago
Posted some thoughts on what problems are appropriate for Hadoop here: http://weblog.strlen.net/?p=8 http://weblog.strlen.net/?p=8 (Another post on parameter settings and machine configurations for Hadoop cluster in the making, once that's done I'll post to news.y.c)
bporterfield 18y ago
Great reply. My current thinking for the project I'm working on is that I don't need Hadoop...yet. That said, if it's trivial to implement on a single machine it might be wise to just begin the project in Hadoop so that I don't have to make the switch later. Always balance between 'premature optimization is the root of all evil' and 'choosing the right tools for the right job'. I'll give it a shot on my desktop to start - thanks for the tips!