SQL is familiar, but it is not simple. The vocabulary is large, and inconsistent between implementations. It is hard to predict the performance of a complex que

SQL is familiar, but it is not simple. The vocabulary is large, and inconsistent between implementations. It is hard to predict the performance of a complex query, without resorting to rules of thumb. Understanding EXPLAIN ... PLAN requires a fairly deep comp sci background, and familiarity with a variety of data structures that are rarely used directly by programmers. Contrast that with a system of map-filter-reduce pipelines over an append-only data set, like a classic CouchDB. A reasonable pipeline can be composed by a junior dev, just by repeatedly asking "What do I want this report to summarize? What information do I need to collect or reject, for that summary? How can I transform the shape of the information that is currently in front of me, into the input that I wanted when I planned the high-level end result?" And, if they need help with that last part, then at least they are asking for help with a small subset of the problem, instead of "Something is wrong in this forest of queries, can you take a look at it with me?" Or, "I need to add a column, may I ALTER TABLE?" They can even prototype the whole thing on an array in Javascript, if they are more comfortable there. SQL can be a beautiful language that feels very natural, once you have had a few years to build up fluency in it. It might make for an excellent shell language. But, having spent time prototyping systems in CouchDB (which were admired for their elegance, but rejected due to the relative obscurity of Couch, grrr!), I have to say, that my previous bias for querying over transforming, was ultimately holding me back, bogging me down in leaky abstractions. We should have started with MR, and then learned SQL only when presented with something that doesn't fit the MR paradigm, or even the graph processing paradigm, which IMO is also simpler than SQL. As for the original subject, yes, Hadoop is a pig, ideally suited to enterprisey make-work projects. All the way through the book, I kept thinking, "there has got to be a simpler way to set this up."