6 ms·

(I'm the author of the article) I'm 28 and I’m currently a Big Data developer (I use Hadoop, HBase, Hive …) and I don’t understand the buzz surrounding Big Data

by mgachka 11y ago

(I'm the author of the article) I'm 28 and I’m currently a Big Data developer (I use Hadoop, HBase, Hive …) and I don’t understand the buzz surrounding Big Data and NoSQL. With a relational database the complexity is hidden (more or less…) whereas with Big Data and NoSQL the developer needs to deal with this complexity himself/herself. As a result, most of the Big Data applications I’ve seen don’t work well. A really like Big Data because it’s more complex but to be honest, most of the time my work does not required the “Big Data scale”.

jchrisa 11y ago
At Couchbase we did a survey of developers (this was ages ago) and the biggest motivator for NoSQL was schema flexibility. Not having to coordinate migrations is seen as a productivity boost. [1] The other thing document databases can offer that relational databases struggle with is taking subsets (which we use for offline sync.) [2] [1] http://blog.couchbase.com/nosql-adoption-survey-surprises http://blog.couchbase.com/nosql-adoption-survey-surprises [2] http://developer.couchbase.com/mobile/ http://developer.couchbase.com/mobile/
- somedudethere 11y ago
  I'm in the opposite camp. I don't like NoSQL because of the flexible schema. I have to build tools to make sure my data is consistent or have error handling. Migrations ensure that whenever I pull down a version of the code, the database is in the right state. I don't see too many use cases where having being schemaless is a good thing outside of infrastructure ease of use. If you want to store arbitrary data in a table, BJSON in postgres is very efficient and flexible in that regard, while allowing you to have a schema for the rest of the data (e.g. if you are collecting data you probably want a schema for things like name, email, timestamp, etc and then have a BJSON field to jam in whatever you need.)
  - nstart 11y ago
    Having worked in ERP, Ecommerce, Financial tech, and general SASS based tech stuff, I agree with you on the "not too many use cases where having schemaless is a good thing". In most cases it's a shortcut and an unnecessary tradeoff made by people to avoid the few technical issues like DB migrations (also, a solved problem in many ways as long as you don't attempt to reinvent the wheel). The only time I saw a good use for a NoSQL db was to store products in Ecommerce. Managing taxonomy and attributes was always a nightmare and everyone was constantly afraid of performance issues (being on Magento and battling the EAV system didn't help). It would have been great to have only the products being stored on a NoSQL instance and the rest of the data being on the traditional relational data store.
- mkehrt 11y ago
  As someone who spent several years studying programming languages, the thing that drives me crazy about traditional relational databases is the assumption that all data is tuple-structured. Much data is structured as unions of alternates or more complex things like maps. Shoehorning your data model into a tuple-based system is always possible, but often unnatural. The place NoSQL shines is the acknowledgement that most data is complex. Of course it often also does away with ACIDity, which is a huge disaster (EDIT: the doing-away-with is a disaster, I mean).
  - rhizome 11y ago
    Where do you get that "most data is complex" in a way that leans away from relational DBs?
  - duaneb 11y ago
    I don't think this is correct. You can have a relational, columnar-stored key-value map that stores any values you want. Bonus: it's super easy to make these kinds of updates ACID. Of course, if you're maxing out the storage space, you're gonna have a rough time with indexes unless you take the EXACT SAME approach as you would with NoSQL. I don't think there are any "inherent" problems to relational or NoSQL databases, but there are many tradeoffs. The tradeoff of NoSQL databases is that complexity gets very, very difficult to pull off in a distributed fashion. So throw 99% of the indices out the window, dumb your queries down, and cache any joins or scans as much as possible. The upside, I guess, is that the "schema" is pretty irrelevant if it's not your primary key (or secondary, in some databases). But, you lose joins, schemas, subqueries, orderings, many types of transactions, etc, etc, and a lot of "free" stuff that is really only "free" for small numbers of rows per table or strong assumptions about the data. EDIT: Clarification, spelling.
  - parasubvert 11y ago
    The relational model is extremely general, its' been argued (fairly successfully) by Date and Codd to be THE most general model (Graph being a close second). It's a rigorous approach to managing data with integrity. I used to be a programming language oriented person, was big into data structures and objects, but then I read Date and my mind was blown at how beautiful and expressive the relational model is -- for its intended purpose (managing data for logical integrity and ad hoc queriability). The main issues are 1. is that many implementations don't include some features such as unions. 2. Certain things (tree traversal) have also been hard to express in older versions of SQL or older versions of Tutorial D (Chris Date's language that's closer to the model). 3. Sometimes you don't care about long term data management (i.e. ad hoc queriability and integrity), you just want programmatic data persistence with pre-baked access paths that are FAST. 4. Relational integrity features are often crude implementations that slow things down too much or require custom triggers. 5. Queriability in reality requires decent knowledge of the physical layout and indexing if you're going to make it performant 6. Most relational databases have not been built in a cloud native era where we assume distribution across ephemeral disks and compute So... great mathematical model, great way to think through and organize your data for no ambiguity, but the practical implementations leave a lot to be desired. The problem is that "my data is too complex for the relational model" often means "I haven't thought through my data". Things like maps, unions, ordered sets, N-ary relationships, graphs and trees, are actually quite straight forward to represent in relations. The challenge is many of the lessons and arguments for this are trapped in books from the 70s-90s, not on the Web.
kodablah 11y ago
"With a relational database the complexity is hidden" That is my main issue. I use Cassandra over relational firstly for its linear scalability and multi-master-esque HA. But even ignoring those, I understand exactly what is being scanned and what is not, I don't have to fight with an optimizer at runtime based on several parameters.
- mgachka 11y ago
  I understand your point (and it’s a good one) and here is mine: Unless you're working in a team with a lot of good IT guys, you're likely to end up with worse performances and problems. For example, when I started in Big Data, in less than 3 weeks I was able to optimize some batches just because I read the documentation of the framework used (PIG in this case) and read a small part of the source code to dig deeper. And it was not some touchy optimizations: I used in-memory joins and reduced the number relations in the scripts to reduce the generation of Hadoop jobs (which led to batchs 4 times faster). There are often problems with our HBase database because it’s often overloaded (I’m not an IT operator so I can’t give more details) and no one really masters this database whereas it’s in production since 2014. I do understand that in some cases a NoSQL database is mandatory and like you I like to understand what I’m doing. But: - I’m not working in Silicon Valley - Most of my co-worker are not geeks (and I respect that) - It's VERY hard to find guys with real Big Data or NoSQL skills (this comes from a French technical recruiter) So, if the geek part of me loves Big Data and NoSQL, the rational part prefers using well known technologies. If NoSQL and Big Data becomes mainstream and more known then the rational part will love them too.
  - kodablah 11y ago
    While I agree with a couple of your premises, I don't think they all apply to Cassandra as much and is too broad of a brush to use. I don't believe NoSQL means no validation. In fact, I've found things like Cassandra CQL actively prevent me from running expensive queries unless I opt in (e.g. ALLOW FILTERING). Validation is DB specific, but I don't think it's fair to say it's a footgun in NoSQL any more than in SQL databases. As for choosing what is known by the employee market, I personally don't choose technologies that way (but I do choose based on maturity of course). I rarely look for skills as much as the ability to learn new ones, but I understand it can be a pipe dream when in the market for juniors.
threeseed 11y ago
Not really sure what you are talking about. Teradata, Oracle, PostgreSQL for example are reasonably complex databases to cluster and manage yourself. Just as easy/hard as setting up HDFS and installing Hive. In all cases people who are at big data scale are buying OTS solutions e.g. Cloudera appliance. They aren't rolling their own. And if you are using Hive then I can understand why you are not feeling the buzz. But play around for Spark for a while and it's easy to see the future. Being able to write Scala/Python/SQL/R against a data set that can be anywhere from 100MB to 100PB without any changes is pretty compelling.
- angrybits 11y ago
  You are either being dishonest or just patently out of your mind if you think that you can query 100MB and 100PB in the same way. That's not even reasonable by HN standards of hyperbole. Do you have any idea how many orders of magnitude that is?
  - MichaelGG 11y ago
    He's right. Pretend you have 100PB. Write code for that. It'll work for 100MB but have terrible overheads.
    - parasubvert 11y ago
      He's talking about writing it for Apache Spark. Which really isn't intended for 100 MB (I bet I could write a unix pipe & filter script that's faster than Spark), but is intended for 10 TB through several PB.
  - parasubvert 11y ago
    He's talking about Apache Spark. While it can handle 100 MB easily there probably are faster ways to handle that small amount of data. But yes, Spark can handle many PB and doesn't require a ton of changes in the code as you scale up from say 10 TB to 100 PB. The underlying cluster would change, and the performance profile would change a lot (10 TB can be done in-memory ... many PB, not so much)
duaneb 11y ago
The buzz around NoSQL is you don't have to worry about scaling the database. There are many, many more options now for e.g. multi-master, sharding, no-downtime copy-on-write migrations, etc., but just the idea of being able to run a tiny subset of queries or writes without having to worry about running out of resource capacity is a HUGE plus.
- plonh 11y ago
  But having data corruption baked into the system design is shuge minus. Even the big shots at Google and Amazon are constantly firefighting data corruption in their NoSQL systems.
  - duaneb 11y ago
    I'm not quite sure what you're referring to. But at certain scales of data (petabytes, probably?) data corruption is inevitable. I do not know if they use ECC.

by mgachka 11y ago

jchrisa 11y ago

At Couchbase we did a survey of developers (this was ages ago) and the biggest motivator for NoSQL was schema flexibility. Not having to coordinate migrations is seen as a productivity boost. [1] The other thing document databases can offer that relational databases struggle with is taking subsets (which we use for offline sync.) [2] [1] http://blog.couchbase.com/nosql-adoption-survey-surprises http://blog.couchbase.com/nosql-adoption-survey-surprises [2] http://developer.couchbase.com/mobile/ http://developer.couchbase.com/mobile/

somedudethere 11y ago
I'm in the opposite camp. I don't like NoSQL because of the flexible schema. I have to build tools to make sure my data is consistent or have error handling. Migrations ensure that whenever I pull down a version of the code, the database is in the right state. I don't see too many use cases where having being schemaless is a good thing outside of infrastructure ease of use. If you want to store arbitrary data in a table, BJSON in postgres is very efficient and flexible in that regard, while allowing you to have a schema for the rest of the data (e.g. if you are collecting data you probably want a schema for things like name, email, timestamp, etc and then have a BJSON field to jam in whatever you need.)
- nstart 11y ago
  Having worked in ERP, Ecommerce, Financial tech, and general SASS based tech stuff, I agree with you on the "not too many use cases where having schemaless is a good thing". In most cases it's a shortcut and an unnecessary tradeoff made by people to avoid the few technical issues like DB migrations (also, a solved problem in many ways as long as you don't attempt to reinvent the wheel). The only time I saw a good use for a NoSQL db was to store products in Ecommerce. Managing taxonomy and attributes was always a nightmare and everyone was constantly afraid of performance issues (being on Magento and battling the EAV system didn't help). It would have been great to have only the products being stored on a NoSQL instance and the rest of the data being on the traditional relational data store.
mkehrt 11y ago
As someone who spent several years studying programming languages, the thing that drives me crazy about traditional relational databases is the assumption that all data is tuple-structured. Much data is structured as unions of alternates or more complex things like maps. Shoehorning your data model into a tuple-based system is always possible, but often unnatural. The place NoSQL shines is the acknowledgement that most data is complex. Of course it often also does away with ACIDity, which is a huge disaster (EDIT: the doing-away-with is a disaster, I mean).
- rhizome 11y ago
  Where do you get that "most data is complex" in a way that leans away from relational DBs?
- duaneb 11y ago
  I don't think this is correct. You can have a relational, columnar-stored key-value map that stores any values you want. Bonus: it's super easy to make these kinds of updates ACID. Of course, if you're maxing out the storage space, you're gonna have a rough time with indexes unless you take the EXACT SAME approach as you would with NoSQL. I don't think there are any "inherent" problems to relational or NoSQL databases, but there are many tradeoffs. The tradeoff of NoSQL databases is that complexity gets very, very difficult to pull off in a distributed fashion. So throw 99% of the indices out the window, dumb your queries down, and cache any joins or scans as much as possible. The upside, I guess, is that the "schema" is pretty irrelevant if it's not your primary key (or secondary, in some databases). But, you lose joins, schemas, subqueries, orderings, many types of transactions, etc, etc, and a lot of "free" stuff that is really only "free" for small numbers of rows per table or strong assumptions about the data. EDIT: Clarification, spelling.
- parasubvert 11y ago
  The relational model is extremely general, its' been argued (fairly successfully) by Date and Codd to be THE most general model (Graph being a close second). It's a rigorous approach to managing data with integrity. I used to be a programming language oriented person, was big into data structures and objects, but then I read Date and my mind was blown at how beautiful and expressive the relational model is -- for its intended purpose (managing data for logical integrity and ad hoc queriability). The main issues are 1. is that many implementations don't include some features such as unions. 2. Certain things (tree traversal) have also been hard to express in older versions of SQL or older versions of Tutorial D (Chris Date's language that's closer to the model). 3. Sometimes you don't care about long term data management (i.e. ad hoc queriability and integrity), you just want programmatic data persistence with pre-baked access paths that are FAST. 4. Relational integrity features are often crude implementations that slow things down too much or require custom triggers. 5. Queriability in reality requires decent knowledge of the physical layout and indexing if you're going to make it performant 6. Most relational databases have not been built in a cloud native era where we assume distribution across ephemeral disks and compute So... great mathematical model, great way to think through and organize your data for no ambiguity, but the practical implementations leave a lot to be desired. The problem is that "my data is too complex for the relational model" often means "I haven't thought through my data". Things like maps, unions, ordered sets, N-ary relationships, graphs and trees, are actually quite straight forward to represent in relations. The challenge is many of the lessons and arguments for this are trapped in books from the 70s-90s, not on the Web.

kodablah 11y ago

"With a relational database the complexity is hidden" That is my main issue. I use Cassandra over relational firstly for its linear scalability and multi-master-esque HA. But even ignoring those, I understand exactly what is being scanned and what is not, I don't have to fight with an optimizer at runtime based on several parameters.

mgachka 11y ago
I understand your point (and it’s a good one) and here is mine: Unless you're working in a team with a lot of good IT guys, you're likely to end up with worse performances and problems. For example, when I started in Big Data, in less than 3 weeks I was able to optimize some batches just because I read the documentation of the framework used (PIG in this case) and read a small part of the source code to dig deeper. And it was not some touchy optimizations: I used in-memory joins and reduced the number relations in the scripts to reduce the generation of Hadoop jobs (which led to batchs 4 times faster). There are often problems with our HBase database because it’s often overloaded (I’m not an IT operator so I can’t give more details) and no one really masters this database whereas it’s in production since 2014. I do understand that in some cases a NoSQL database is mandatory and like you I like to understand what I’m doing. But: - I’m not working in Silicon Valley - Most of my co-worker are not geeks (and I respect that) - It's VERY hard to find guys with real Big Data or NoSQL skills (this comes from a French technical recruiter) So, if the geek part of me loves Big Data and NoSQL, the rational part prefers using well known technologies. If NoSQL and Big Data becomes mainstream and more known then the rational part will love them too.
- kodablah 11y ago
  While I agree with a couple of your premises, I don't think they all apply to Cassandra as much and is too broad of a brush to use. I don't believe NoSQL means no validation. In fact, I've found things like Cassandra CQL actively prevent me from running expensive queries unless I opt in (e.g. ALLOW FILTERING). Validation is DB specific, but I don't think it's fair to say it's a footgun in NoSQL any more than in SQL databases. As for choosing what is known by the employee market, I personally don't choose technologies that way (but I do choose based on maturity of course). I rarely look for skills as much as the ability to learn new ones, but I understand it can be a pipe dream when in the market for juniors.

threeseed 11y ago

Not really sure what you are talking about. Teradata, Oracle, PostgreSQL for example are reasonably complex databases to cluster and manage yourself. Just as easy/hard as setting up HDFS and installing Hive. In all cases people who are at big data scale are buying OTS solutions e.g. Cloudera appliance. They aren't rolling their own. And if you are using Hive then I can understand why you are not feeling the buzz. But play around for Spark for a while and it's easy to see the future. Being able to write Scala/Python/SQL/R against a data set that can be anywhere from 100MB to 100PB without any changes is pretty compelling.

angrybits 11y ago
You are either being dishonest or just patently out of your mind if you think that you can query 100MB and 100PB in the same way. That's not even reasonable by HN standards of hyperbole. Do you have any idea how many orders of magnitude that is?
- MichaelGG 11y ago
  He's right. Pretend you have 100PB. Write code for that. It'll work for 100MB but have terrible overheads.
  - parasubvert 11y ago
    He's talking about writing it for Apache Spark. Which really isn't intended for 100 MB (I bet I could write a unix pipe & filter script that's faster than Spark), but is intended for 10 TB through several PB.
- parasubvert 11y ago
  He's talking about Apache Spark. While it can handle 100 MB easily there probably are faster ways to handle that small amount of data. But yes, Spark can handle many PB and doesn't require a ton of changes in the code as you scale up from say 10 TB to 100 PB. The underlying cluster would change, and the performance profile would change a lot (10 TB can be done in-memory ... many PB, not so much)

duaneb 11y ago

The buzz around NoSQL is you don't have to worry about scaling the database. There are many, many more options now for e.g. multi-master, sharding, no-downtime copy-on-write migrations, etc., but just the idea of being able to run a tiny subset of queries or writes without having to worry about running out of resource capacity is a HUGE plus.

plonh 11y ago
But having data corruption baked into the system design is shuge minus. Even the big shots at Google and Amazon are constantly firefighting data corruption in their NoSQL systems.
- duaneb 11y ago
  I'm not quite sure what you're referring to. But at certain scales of data (petabytes, probably?) data corruption is inevitable. I do not know if they use ECC.