5 ms·

…yes - processing 3.2G of data will be quicker on a single machine. This is not the scale of Hadoop or any other distributed compute platform. The reason we us

…yes - processing 3.2G of data will be quicker on a single machine. This is not the scale of Hadoop or any other distributed compute platform. The reason we use these is for when we have a data set _larger_ than what can be done on a single machine.

ralph84 3y ago
Most people who wasted $millions setting up Hadoop didn’t have data sets larger than could fit on a single machine.
- hobos_delight 3y ago
  I completely agree. I love the tech and have spent a lot of time in it - but come on people, let’s use the right tool for the right job!
- hiAndrewQuinn 3y ago
  Moore's law and its analogues makes this harder to back-predict than one might think, though. A decade ago computers had only had about an eighth (rough upper bound) of the resources modern machines tend to have at similar price points.
- saberience 3y ago
  Do you have any examples of companies building Hadoop clusters for amounts of data that fit on a single machine? I’ve heard this anecdote on HN before but without ever seeing actual evidence it happened, it reads like an old wives tale and I’m not sure I believe it. I’ve worked on a Hadoop cluster and setting it up and running it takes quite serious technical skills and experience and those same technical skills and experience would mean the team wouldn’t be doing it unless they needed it. Can you really imagine some senior data and infrastructure engineers setting up 100 nodes knowing it was for 60GB of data? Does that make any sense at all?
  - geraldwhen 3y ago
    Developers are taught that you must scale horizontally. They become seniors and managers and ruin everything they touch. I have to teach developers that yes, we can have a 500MB data cache in ram, and that’s actually not a lot at all.
  - oblio 3y ago
    I used to work for a pretty famous 2nd tier US company (smaller and less cool than FAANG). They had a team working on a Hadoop based solution and their biggest internal implementations was about what you're describing, in practice. It makes sense because internal politics.
    - jerven 3y ago
      In 2014 I was at Oracle Open World. A 3rd party hardware vendor was saying (and having customers) for Hadoop "clusters" that had 8 cpu cores. Basically their pitch was that Oracle Hardware (ex sun) started at a dense full rack of about a 1 million USD or so, but with the 3d party you could have a hadoop "cluster" in 2U and for 20K. The oracle thing was actually quite price competitive at the time, if you needed hadoop. The 3rd party thing was overpriced for what it was. Yet, I am sure that 3rd party hardware vendor made out like bandits.
  - dijit 3y ago
    I did some data processing at Ubisoft. each node in our hadoop cluster had 64GiB of ram (which is the max amount you should have for a single node java application, where 32G is allocated for heap FWIW), we had I think 6 of these nodes for a total of 384GiB memory. Our storage was something like 18TiB across all nodes. It would be a big machine, but our entire cluster could easily fit. Largest machine on the market right now is something like 128CPU's and 20TiB of Memory. 384GiB was available in a single 1U rackmount server at least as early as 2014. Storage is basically unlimited with direct-attached-storage controllers and rackmount units.
    - genewitch 3y ago
      I had an HP from 2010 that supported 1.5TB of ram with 40 cores, but it was 4U. I'm not sure what the height has to do with memory other than a 1U doesn't have the luxury of the backplane(s) being vertical or otherwise above the motherboard, so maybe it's limited space?
- faet 3y ago
  I've worked places where it would be 1000x harder getting a spare laptop from the IT closet to run some processing than it would be to spend $50k-100k at Azure.
MrBuddyCasino 3y ago
This will not stop BigCorp to spend weeks to setup a big ass data analytics pipeline to process a few hundred MB from their „Data Lake“ via Spark. And this isn’t even wrong, bc what they need is a long-term maintainable method that scales up IF needed (rarely), is documented and survives loss of institutional knowledge three layoffs down the line.
- hobos_delight 3y ago
  Scaling _if_ needed has been the death knell of many companies. Every engineer wants to assume that they will need to scale to millions of QPS, most of the time this is incorrect, and when it is not then the requirement have changed and it needs to be rebuilt anyway.
  - MrBuddyCasino 3y ago
    This is true for startups an small companies, Big Corps IT is so far away from operating efficiently that this doesn't really matter.
    - hobos_delight 3y ago
      I think it completely matters - yes these orgs are a lot more wasteful, but there is still an opportunity to save money here, especially is this economy, if not for the internal politics wins. I’ve spent time in some of the largest distributed computing deployments and cost was always a constant factor we had to account for. The easiest promos were always “I saved X hundred million” because it was hard to argue against saving money. And these happened way more than you would guess.
      - MrBuddyCasino 3y ago
        > I’ve spent time in some of the largest distributed computing deployments Yeah obviously if you run hundreds or thousands of severs then efficiency matters a lot, but then there isn't really the option to use a single machine with a lot of RAM instead, is there? I'm talking about the typical BigCorp whose core business is something else than IT, like insurance, construction, mining, retail, whatever. Saving a single AKS cluster just doesn't move the needle.
OskarS 3y ago
This is exactly the point of the article. From the conclusion: > Hopefully this has illustrated some points about using and abusing tools like Hadoop for data processing tasks that can better be accomplished on a single machine with simple shell commands and tools.
the8472 3y ago
What can be done on a single machine grows with time though. You can have terabytes of ram and petabytes of flash in a single machine now.

ralph84 3y ago

Most people who wasted $millions setting up Hadoop didn’t have data sets larger than could fit on a single machine.

hobos_delight 3y ago
I completely agree. I love the tech and have spent a lot of time in it - but come on people, let’s use the right tool for the right job!
hiAndrewQuinn 3y ago
Moore's law and its analogues makes this harder to back-predict than one might think, though. A decade ago computers had only had about an eighth (rough upper bound) of the resources modern machines tend to have at similar price points.
saberience 3y ago
Do you have any examples of companies building Hadoop clusters for amounts of data that fit on a single machine? I’ve heard this anecdote on HN before but without ever seeing actual evidence it happened, it reads like an old wives tale and I’m not sure I believe it. I’ve worked on a Hadoop cluster and setting it up and running it takes quite serious technical skills and experience and those same technical skills and experience would mean the team wouldn’t be doing it unless they needed it. Can you really imagine some senior data and infrastructure engineers setting up 100 nodes knowing it was for 60GB of data? Does that make any sense at all?
- geraldwhen 3y ago
  Developers are taught that you must scale horizontally. They become seniors and managers and ruin everything they touch. I have to teach developers that yes, we can have a 500MB data cache in ram, and that’s actually not a lot at all.
- oblio 3y ago
  I used to work for a pretty famous 2nd tier US company (smaller and less cool than FAANG). They had a team working on a Hadoop based solution and their biggest internal implementations was about what you're describing, in practice. It makes sense because internal politics.
  - jerven 3y ago
    In 2014 I was at Oracle Open World. A 3rd party hardware vendor was saying (and having customers) for Hadoop "clusters" that had 8 cpu cores. Basically their pitch was that Oracle Hardware (ex sun) started at a dense full rack of about a 1 million USD or so, but with the 3d party you could have a hadoop "cluster" in 2U and for 20K. The oracle thing was actually quite price competitive at the time, if you needed hadoop. The 3rd party thing was overpriced for what it was. Yet, I am sure that 3rd party hardware vendor made out like bandits.
- dijit 3y ago
  I did some data processing at Ubisoft. each node in our hadoop cluster had 64GiB of ram (which is the max amount you should have for a single node java application, where 32G is allocated for heap FWIW), we had I think 6 of these nodes for a total of 384GiB memory. Our storage was something like 18TiB across all nodes. It would be a big machine, but our entire cluster could easily fit. Largest machine on the market right now is something like 128CPU's and 20TiB of Memory. 384GiB was available in a single 1U rackmount server at least as early as 2014. Storage is basically unlimited with direct-attached-storage controllers and rackmount units.
  - genewitch 3y ago
    I had an HP from 2010 that supported 1.5TB of ram with 40 cores, but it was 4U. I'm not sure what the height has to do with memory other than a 1U doesn't have the luxury of the backplane(s) being vertical or otherwise above the motherboard, so maybe it's limited space?
faet 3y ago
I've worked places where it would be 1000x harder getting a spare laptop from the IT closet to run some processing than it would be to spend $50k-100k at Azure.

MrBuddyCasino 3y ago

This will not stop BigCorp to spend weeks to setup a big ass data analytics pipeline to process a few hundred MB from their „Data Lake“ via Spark. And this isn’t even wrong, bc what they need is a long-term maintainable method that scales up IF needed (rarely), is documented and survives loss of institutional knowledge three layoffs down the line.

hobos_delight 3y ago
Scaling _if_ needed has been the death knell of many companies. Every engineer wants to assume that they will need to scale to millions of QPS, most of the time this is incorrect, and when it is not then the requirement have changed and it needs to be rebuilt anyway.
- MrBuddyCasino 3y ago
  This is true for startups an small companies, Big Corps IT is so far away from operating efficiently that this doesn't really matter.
  - hobos_delight 3y ago
    I think it completely matters - yes these orgs are a lot more wasteful, but there is still an opportunity to save money here, especially is this economy, if not for the internal politics wins. I’ve spent time in some of the largest distributed computing deployments and cost was always a constant factor we had to account for. The easiest promos were always “I saved X hundred million” because it was hard to argue against saving money. And these happened way more than you would guess.
    - MrBuddyCasino 3y ago
      > I’ve spent time in some of the largest distributed computing deployments Yeah obviously if you run hundreds or thousands of severs then efficiency matters a lot, but then there isn't really the option to use a single machine with a lot of RAM instead, is there? I'm talking about the typical BigCorp whose core business is something else than IT, like insurance, construction, mining, retail, whatever. Saving a single AKS cluster just doesn't move the needle.

OskarS 3y ago

This is exactly the point of the article. From the conclusion: > Hopefully this has illustrated some points about using and abusing tools like Hadoop for data processing tasks that can better be accomplished on a single machine with simple shell commands and tools.

the8472 3y ago

What can be done on a single machine grows with time though. You can have terabytes of ram and petabytes of flash in a single machine now.