6 ms·

I agree with you on the landing page. The introductory paragraph seems rather "fluffy". That combined with the fact that it uses a whitepaper immediately gave m

I agree with you on the landing page. The introductory paragraph seems rather "fluffy". That combined with the fact that it uses a whitepaper immediately gave me the feeling that it's not really meant as something for regular programmers to check out and hack with. It's surprising, since that's how many Clojure programmers get their start. On the other hand, it's very new so maybe they'll add more developer-friendly pages soon. Or maybe it's only meant for "enterprise" environments? Time will tell.

nickik 15y ago
Maybe its so new and unique that its hard or impossible to explain it in a pragraph.
- fogus 15y ago
  Datomic is a UFO filled with advanced alien technology that has landed right on the National Mall.
  - nickik 15y ago
    Probebly an UFO from the Land of Lisp.
  - alkby 15y ago
    Any access to source? Looks nice but I have a bunch of questions about transactor.
- rjn945 15y ago
  I think this has a lot to do with it. After an hour of reading, watching and thinking, I can't come up with any way to put it into one paragraph. Here's the shortest what and why I could come up with: Questioning Assumptions Many relational databases today operate based on assumptions that were true in the 1970s but are no longer true. Newer solutions such as key-value stores ("NoSQL") make unnecessary compromises in the ability to perform queries or make consistency guarantees. Datomic reconsiders the database in light of current computer set-ups: millions of times larger and faster disks and RAM, and distributed architectures connected over the internet. Data Model Instead of using table-based storage with explicit schemas, Datomic uses a simpler model wherein the database is made up of a large collection of "datoms" or facts. Each datom has 4 parts: an entity, an attribute, a value, and a time (denoted by the transaction number that added it the database). Example: John, :street, "23 Swift St.", T27 This simple data model has two main benefits. It makes your data less rigid and hence more agile and easier to change. Additionally, it makes it easy to handle data in non-traditional structures, such as hierarchies, sets or sparse tables. It also enables Datomic's time model... Time Like Clojure, Datomic incorporates an explicit model of time. All data is associated with a time and new data does not replace old data, but is added to it. Returning to our previous example, if John later changes his address, a new datom would be added to the database, e.g. John, :street, "17 Maple St.", T43 This mirrors the real world where the fact that John has moved does not erase the fact that John once lived on Swift St. This has multiple benefits: the ability to view the database at a point in time other than the present; no data is lost; the immutability of each datom allows for easy and pervasive caching. Move Data and Data Processing to Peers Traditionally databases use a client-server model where clients send queries and commands to a central database. This database holds all the data, performs all data processing, and manages the data storage and synchronization. Clients may only to access the data through the interface the server provides - typically SQL strings which may include a (relatively small) set of functions provided by the database. Datomic breaks this system apart. The only centralized component is data storage. Peers access the data storage through a new distributed component called a transactor. Finally, the most important part, data processing, now happens in the clients, which, considering their importance, have been renamed "peers". Queries are made in a declarative language called Datalog which is similar to but better than SQL. It's better because it more closely matches the model of the data itself (rather than thinking in terms of the implementation of tables in a database). Additionally, it's not restricted like SQL. It allows you to use your full programming language. You can write reusable rules that can then be composed in queries. Additionally, you can call any of your own functions. This is a big step up in power and it's made practical because of the distribution. If ran your query on central server, you'd have to be concerned about tying up a scare resource with a long-running query. When processing locally, that's not a concern. When a query is performed that data is loaded from central storage and placed into RAM (if it will fit). Later queries can use this locally cached data for fast queries. ---- That's definitely not all it does or all the benefits, but hopefully that's a good start.
  - anamax 15y ago
    > a time (denoted by the transaction number that added it the database). Do transaction numbers have total order or just partial order? Total order is serializing. (And no, using real time as the transaction number doesn't help because it's impossible to keep an interesting number of servers time-synched.) Partial order is "interesting".
    - programnature 15y ago
      It is totally ordered. The transactor is a single point of failure. However, since its only job is doing the transactions, the idea is it can be faster than a database server that does both the transactions and the queries.
      - snprbob86 15y ago
        Hmm... presumably an application can act in read-only mode in the absence of a transactor. That's an interesting thought :-)
        weavejester 15y ago
        The transactor is only used for writes, so if the transactor went down, you could still run queries.
      - anamax 15y ago
        > However, since [the transactor's] only job is doing the transactions Huh? How is that consistent with: > access the data storage through a new distributed component called a transactor. If "doing the transactions" consists of more than passing out incrementing transaction tokens, won't the transactor be a bottleneck?
        rjn945 15y ago
        Yeah, it looks like I got that part wrong. (I intentionally skimmed over the transactor, because I was avoiding "how" issues and because my understanding of it wasn't that clear.) The transactor is involved in just writes, not reads. (So that helps.) It's not distributed and cannot be distributed, in this system, because it ensures consistency, so yes, it is potentially a bottleneck. In blog comments by Rich Hickey[1], he states: "Writes can’t be distributed, and that is one of the many tradeoffs that preclude the possibility of any universal data solution. The idea is that, by stripping out all the other work normally done by the server (queries, reads, locking, disk sync), many workloads will be supported by this configuration. We don’t target the highest write volumes, as those workloads require different tradeoffs." Presumably, 1) the creators of Datomic think that performance can be good enough to be useful, 2) this is a new model that probably requires testing to prove is practical. [1] Multiple people have linked to it, but for convenience: http://blog.fogus.me/2012/03/05/datomic/comment-page-1/#comment-48817 http://blog.fogus.me/2012/03/05/datomic/comment-page-1/#comm...

nickik 15y ago

Maybe its so new and unique that its hard or impossible to explain it in a pragraph.

fogus 15y ago
Datomic is a UFO filled with advanced alien technology that has landed right on the National Mall.
- nickik 15y ago
  Probebly an UFO from the Land of Lisp.
- alkby 15y ago
  Any access to source? Looks nice but I have a bunch of questions about transactor.
rjn945 15y ago
I think this has a lot to do with it. After an hour of reading, watching and thinking, I can't come up with any way to put it into one paragraph. Here's the shortest what and why I could come up with: Questioning Assumptions Many relational databases today operate based on assumptions that were true in the 1970s but are no longer true. Newer solutions such as key-value stores ("NoSQL") make unnecessary compromises in the ability to perform queries or make consistency guarantees. Datomic reconsiders the database in light of current computer set-ups: millions of times larger and faster disks and RAM, and distributed architectures connected over the internet. Data Model Instead of using table-based storage with explicit schemas, Datomic uses a simpler model wherein the database is made up of a large collection of "datoms" or facts. Each datom has 4 parts: an entity, an attribute, a value, and a time (denoted by the transaction number that added it the database). Example: John, :street, "23 Swift St.", T27 This simple data model has two main benefits. It makes your data less rigid and hence more agile and easier to change. Additionally, it makes it easy to handle data in non-traditional structures, such as hierarchies, sets or sparse tables. It also enables Datomic's time model... Time Like Clojure, Datomic incorporates an explicit model of time. All data is associated with a time and new data does not replace old data, but is added to it. Returning to our previous example, if John later changes his address, a new datom would be added to the database, e.g. John, :street, "17 Maple St.", T43 This mirrors the real world where the fact that John has moved does not erase the fact that John once lived on Swift St. This has multiple benefits: the ability to view the database at a point in time other than the present; no data is lost; the immutability of each datom allows for easy and pervasive caching. Move Data and Data Processing to Peers Traditionally databases use a client-server model where clients send queries and commands to a central database. This database holds all the data, performs all data processing, and manages the data storage and synchronization. Clients may only to access the data through the interface the server provides - typically SQL strings which may include a (relatively small) set of functions provided by the database. Datomic breaks this system apart. The only centralized component is data storage. Peers access the data storage through a new distributed component called a transactor. Finally, the most important part, data processing, now happens in the clients, which, considering their importance, have been renamed "peers". Queries are made in a declarative language called Datalog which is similar to but better than SQL. It's better because it more closely matches the model of the data itself (rather than thinking in terms of the implementation of tables in a database). Additionally, it's not restricted like SQL. It allows you to use your full programming language. You can write reusable rules that can then be composed in queries. Additionally, you can call any of your own functions. This is a big step up in power and it's made practical because of the distribution. If ran your query on central server, you'd have to be concerned about tying up a scare resource with a long-running query. When processing locally, that's not a concern. When a query is performed that data is loaded from central storage and placed into RAM (if it will fit). Later queries can use this locally cached data for fast queries. ---- That's definitely not all it does or all the benefits, but hopefully that's a good start.
- anamax 15y ago
  > a time (denoted by the transaction number that added it the database). Do transaction numbers have total order or just partial order? Total order is serializing. (And no, using real time as the transaction number doesn't help because it's impossible to keep an interesting number of servers time-synched.) Partial order is "interesting".
  - programnature 15y ago
    It is totally ordered. The transactor is a single point of failure. However, since its only job is doing the transactions, the idea is it can be faster than a database server that does both the transactions and the queries.
    - snprbob86 15y ago
      Hmm... presumably an application can act in read-only mode in the absence of a transactor. That's an interesting thought :-)
      - weavejester 15y ago
        The transactor is only used for writes, so if the transactor went down, you could still run queries.
    - anamax 15y ago
      > However, since [the transactor's] only job is doing the transactions Huh? How is that consistent with: > access the data storage through a new distributed component called a transactor. If "doing the transactions" consists of more than passing out incrementing transaction tokens, won't the transactor be a bottleneck?
      - rjn945 15y ago
        Yeah, it looks like I got that part wrong. (I intentionally skimmed over the transactor, because I was avoiding "how" issues and because my understanding of it wasn't that clear.) The transactor is involved in just writes, not reads. (So that helps.) It's not distributed and cannot be distributed, in this system, because it ensures consistency, so yes, it is potentially a bottleneck. In blog comments by Rich Hickey[1], he states: "Writes can’t be distributed, and that is one of the many tradeoffs that preclude the possibility of any universal data solution. The idea is that, by stripping out all the other work normally done by the server (queries, reads, locking, disk sync), many workloads will be supported by this configuration. We don’t target the highest write volumes, as those workloads require different tradeoffs." Presumably, 1) the creators of Datomic think that performance can be good enough to be useful, 2) this is a new model that probably requires testing to prove is practical. [1] Multiple people have linked to it, but for convenience: http://blog.fogus.me/2012/03/05/datomic/comment-page-1/#comment-48817 http://blog.fogus.me/2012/03/05/datomic/comment-page-1/#comm...