9 ms·
Rich Hickey: Deconstructing the Database
- brlewis 14y agoAnyone have a summary for those of us who don't want to watch an hour-long video?
- bct 14y agoI haven't watched the video yet either, but I'm guessing he's talking about Datomic's design, which would make this a good place to start: http://docs.datomic.com/architecture.html http://docs.datomic.com/architecture.html
- breckinloggins 14y agoI'm only a few minutes into it, but so far the tl;dr is: "There are lots of things we do with respect to databases simply because of the limitations of our databases. Here are some things that could be made far simpler if you just had a better database." Yes it's partly a sales presentation for Datomic, but you could do worse than be sold by Rich Hickey.
- breckinloggins 14y agoMore notes on the video: - Rich's whole view on the world is pretty consistent with respect to this talk. If you know his view on immutability, values vs identity, transactions, and so forth, then you already have a pretty good idea about what kind of database Rich Hickey would build if Rich Hickey built a database (which, of course, he did!) - The talk extends his "The Value of Values" keynote [1] with specific applicability to databases - Further, there is an over-arching theme of "decomplecting" a database so that problems are simpler. This follows from his famous "Simple made easy" talk [2] - His data product, Datomic, is what you get when you apply the philosophies of Clojure to a database I've talked about this before, but I still think Datomic has a marketing problem. Whenever I think of it, I think "cool shit, big iron". Why don't I think about Datomic the same way I think about, say, "Mongodb". As in, "Hey, let me just download this real quick and play around with it!" I really think the folks at Datomic need to steal some marketing tricks from the NoSQL guys so we get more people writing hipster blog posts about it ;-) [1] http://www.infoq.com/presentations/Value-Values http://www.infoq.com/presentations/Value-Values [2] http://www.infoq.com/presentations/Simple-Made-Easy http://www.infoq.com/presentations/Simple-Made-Easy
- lhnz 14y agoThere are two people that I will stop what I'm doing and watch every new lecture they make: Rich Hickey and Bret Victor. Both are visionaries.
- chasingtheflow 14y agoEverytime I watch "Inventing on Principle" I learn something new. Are there any other great Bret Victor talks available that the community would suggest? Or Rich (Value of Values, Simple Made Easy ...) for that matter.
- scottjad 14y agoFor Rich, two of my favorites are: Are We There Yet http://www.infoq.com/presentations/Are-We-There-Yet-Rich-Hickey http://www.infoq.com/presentations/Are-We-There-Yet-Rich-Hic... Clojure Concurrency http://blip.tv/clojure/clojure-concurrency-819147 http://blip.tv/clojure/clojure-concurrency-819147 These early videos of Rich demoing his little project have such a cool feel. There are several more on the blip.tv account. And in addition to the two you mention and these are good: Hammock Driven Development http://blip.tv/clojure/hammock-driven-development-4475586 http://blip.tv/clojure/hammock-driven-development-4475586 Clojurescript http://blip.tv/clojure/rich-hickey-unveils-clojurescript-5399498 http://blip.tv/clojure/rich-hickey-unveils-clojurescript-539...
- puredanger 14y agoAnd both will be at Strange Loop this year! http://thestrangeloop.com/sessions http://thestrangeloop.com/sessions
- Sandman 14y agoStrange Loop is such a great conference. I wish there was something like that here in Europe.
- puredanger 14y agoWhat about Devs Love Bacon? http://devslovebacon.com/ http://devslovebacon.com/
- duck 14y agoI'm getting "This video is currently unavailable"?
- anatoly 14y agoI don't know - are you?
- arscan 14y agoI recall Datomic making a bit of a splash on HN when it was announced 6+ months ago, but basically crickets since then. Anybody build something cool that took advantage of Datomic's unique design?
- puredanger 14y agoWe're building some stuff with it atm but I can't go into details. I've run into several others using it for a variety of things as well and the support group is active. http://groups.google.com/group/datomic http://groups.google.com/group/datomic
- Tuna-Fish 14y agoPeople tend to be more conservative about data than they are about the other parts of their stack -- probably for a good reason. I don't think datomic (or it's kin) will have that huge of an influence until years from now.
- kinleyd 14y agoI think Datomic is potentially disruptive and represents some great thinking on the part of an individual. Whether it will be disruptive will hinge on how well that thinking has subsumed the years of industry experience and practicalities, not to forget the conservative approach to data. I'd be interested to see how it pans out.
- nickik 14y agoI think if will also be imortend what other databases come out of this. There is space for other prioritys in terms of CAP in a world were perseption and process are seperated. Or just other implmentations of the same ideas, opensource maybe.
- it 14y agoIf you want to be conservative about your data, it looks like Datomic is an excellent choice. It preserves the entire history of your data instead of allowing it to be modified in place.
- erikpukinskis 14y agoFascinating stuff. Some things that came up for me while watching this and the other videos on their site[1]: It's not Open Source, for anyone who cares about that. It's interesting how strange it feels to me for infrastructure code to be anything other then Open Source. I'm sort of shocked that the query language is still passing strings, when Hickey made a big deal of how the old database do it that way. I guess for me a query is a data structure that we build programmatically, so why force the developer to collapse it into a string? Maybe because they want to support languages that aren't expressive enough to do that concisely? [1] http://www.datomic.com/videos.html http://www.datomic.com/videos.html
- bsaul 14y agoAnyone understands how this system would deal with CAP theorem, in the case of a regular "add 100$ then remove 50$ to the bank account, in that order and in one go" type of transaction ? The transactor is supposed to "send" novelty to peers, so that they update their live index. That's one point where i would see trouble (suppose it lags, one "add" request goes to one peer, the "read" goes to the second, you don't find what you just add...) Another place i see where it could mess things up is the "Data store" tier, which uses the same traditional technics as of today to replicate data between different servers (one peer requests facts from a "part" of the data store that's not yet synchronized with the one a second peer requests). It seems like all those issues are addressed on his "a fact can also be a function" slide, but he skips it very quickly, so if anyone here could tell me more...
- ilaksh 14y agoWell he did put "atomic" in the name.
- plaeremans 14y agoWell there is one transactor, the transactor handles each transaction sequential, so the transactor can abort a transaction when the world has changed since it got queued to the transactor. There is (virtually) infinite read scalability. Each datum has a time associated with it, so you might not see the latest information (yet), you know the state of the world at a certain point in time. I think it 's a really well designed system.
- olivergeorge 14y agoThis might only be vaguely related. It's an example of managing bank account balances using datomic taking advantage of transaction functions. https://gist.github.com/3134849 https://gist.github.com/3134849
- azolotko 14y agoIn Datomic you can setup a function the transactor can call within a transaction. This function takes the current value of the database and other supplied arguments (e.g. $100 and -$50 from your example) and according to its logic produces and returns a list of "changes" the transactor should apply to the database. The function is pure in sense that it doesn't have any side effects, it just "expands" into new data. And of course this "expantion" and application of its results happens in the same transaction.
- hobbyist 14y agoI often wonder, is Phd in computer science really required to do awesome work?
- leibniz 14y agoWhen I heard the first couple of videos of Rich, I also asked myself this question. Here's Matt Welsh's take on "Do you need a PhD"?: http://matt-welsh.blogspot.co.at/2012/03/do-you-need-phd.html http://matt-welsh.blogspot.co.at/2012/03/do-you-need-phd.htm...
- tensor 14y agoOf course not. You can certainly learn all you need to know on your own. However, that doesn't make the process of learning any easier. If you can get into a PhD program, it is a wonderful way to get access to information of various sorts. If not, then the new free CS courses that are now being offered by Stanford and others provide extra help beyond reading books and papers. I'd highly recommend trying some!
- swannodette 14y agoRich Hickey never went to school for Computer Science as far as I know. He studied music composition.
- wooby 14y agoRich has a CS master's degree.
- swannodette 14y agoDid not know that!
- Jach 14y agoI would say no, but a good chunk of knowledge is (though probably less is required than imagined) and if the academic methods of acquiring knowledge suit you (for many hackers they don't) then a Ph.D is a fine way to get a good chunk of knowledge. Here's Rich Hickey's recommended reading for Clojure specifically: http://www.amazon.com/Clojure-Bookshelf/lm/R3LG3ZBZS4GCTH/ref=cm_lm_byauthor_title_full http://www.amazon.com/Clojure-Bookshelf/lm/R3LG3ZBZS4GCTH/re... It's not exhaustive for even functional programming and design let alone the entirety of computer science, but it's certainly a good chunk of knowledge enough to do awesome work from.
- sbmassey 14y agoHow would you idiomatically fix invalid data in Datomic? for example, if you needed to update a badly entered value in a record, but keep the record's timestamp the same so as not to screw up historical queries?
- bmurphy1976 14y agoYou wouldn't. You can't change history either. You would insert a compensating transaction and the data would be fixed from that time forward when you inserted the compensating transaction.
- sbmassey 14y agoFigures. So I guess that, in the case where you both need to worry about fixing invalid data, and also need to do historical queries, you would have to add your own timestamp to the data to represent the actual time of the event, because the built-in timestamp is just giving you the time of the state of the database. Hopefully that wouldn't get too hairy.
- puredanger 14y agoYou can retract facts, just like you can assert facts.
- danecjensen 14y agothis reminds me a lot of "How to beat the CAP theorem" http://nathanmarz.com/blog/how-to-beat-the-cap-theorem.html http://nathanmarz.com/blog/how-to-beat-the-cap-theorem.html
- sriram_malhar 14y agoI'm always puzzled when the Datomic folks speak of reads not being covered under a transaction. This is dangerous. Here's the scenario, that in a conventional update-oriented store, is termed as a "lost update". "A" reads object.v1, "B" reads the same version, "B" adds a fact to the object making it v2, then "A" comes along and writes obj.v3 based on its own _stale_ knowledge of the object. In effect, it has clobbered what "B" wrote, because A's write came later and has become the latest version of the object. The fact that DAtomic's transactor serialized writes is meaningless because it doesn't take into account read dependency. In other words, DAtomic gives you an equivalent of Read-committed or snapshot isolation, but not true serializability. I wouldn't use it for a banking transaction for sure. To fix it, DAtomic would need to add a test-and-set primitive to implement optimistic concurrency, so that a client can say, "process this write only if this condition is still true". Otherwise, two clients are only going to be talking past each other.
- Tuna-Fish 14y agoYou are incorrect -- datomic transactions can depend on the previous state of the DB, and prevent it from being modified from under it. To do this, you do the transaction in the transactor process.
- tensor 14y agoI think "transaction functions" are intended to provide a solution to that, although I do wonder how it would fair performance wise under very cpu intensive transaction functions.
- fogus 14y ago...would need to add a test-and-set primitive Datomic provides CAS via its `:db.fn/cas` database function. I'm not sure that it's documented at the moment.
- richhickey 14y agoAddressed here: http://news.ycombinator.com/item?id=4448351 http://news.ycombinator.com/item?id=4448351
- saurik 14y agoWatching this talk has so far (I'm halfway through, and now giving up) been very disappointing, primarily because many of the features and implementation details ascribed to "traditional databases" are not true of the common modern SQL databases, and almost none of them are true of PostgreSQL. As an initial trivial example, many database systems allow you to store arrays. In the case of PostgreSQL, you can have quite complex data types, from dictionaries and trees to JSON, or even whatever else you want to come up with, as it is a runtime extensible system. However, it really gets much deeper than these kinds of surface details. As a much more bothersome example that is quite fundamental to the point he seems to be taking with this talk, at about 15:30 he seriously says "in general, that is an update-in-place model", and then has multiple slides about the problems of this data storage model. Yet, modern databases don't do this. Even MySQL doesn't do this (anymore). Instead, modern databases use MVCC, which involves storing all historical versions of the data for at least some time; in PostgreSQL, this could be a very long time (when a manual VACUUM occurs; if you want to store things forever, this can be arranged ;P). http://en.wikipedia.org/wiki/Multiversion_concurrency_control http://en.wikipedia.org/wiki/Multiversion_concurrency_contro... This MVCC model thereby directly solves one of the key problems he spends quite a bit of time at the beginning of his talk attempting to motivate: that multiple round-trips to the server are unable to get cohesive state; in actuality, you can easily get consistent state from these multiple queries, as within a single transaction (which, for the record, is very cheap under MVCC if you are just reading things) almost all modern databases (Oracle, PostgreSQL, MySQL...) will give you an immutable snapshot of what the database looked like when you started your transaction. The situation is actually only getting better and more efficient (I recommend looking at PostgreSQL 9.2's serializable snapshot isolation). At ~20:00, he then describes the storage model he is proposing, and keys in on how important storing time is in a database; the point is also made that storing a timestamp isn't enough: that the goal should be to store a transaction identifier... but again, this is how PostgreSQL already stores its data: every version (as again: it doesn't delete data the way Rich believes it does) stores the transaction range that it is valid for. The only difference between existing SQL solutions and Rich's ideal is that it happens per row instead of per individual field (which could easily be modeled, and is simply less efficient). Now, the point he makes at ~24:00 actually has some merit: that you can't easily look up this information using the presented interfaces of databases. However, if I wanted to hack that feature into PostgreSQL, it would be quite simple, as the fundamental data model is already what he wants: so much so that the indexes are still indexing the dead data, so I could not only provide a hacked up feature to query the past but I could actually do so efficiently. Talking about transactions is even already simple: you can get the identifier of a transaction using txid_current() (and look up other running transactions if you must using info tables; the aforementioned per-row transaction visibility range is even already accessible as magic xmin and xmax columns on every table).