5 ms·

An Opinionated Clojure CQRS/ES Using Onyx, Datomic, DynamoDB, Kafka

111 points by CmdrDats 12y ago

chadly 12y ago
The current relational database is too limiting. We're dropping all sorts of interesting data on the ground because we don't have suitable pigeonholes to put it into. That is the best description of the problem event sourcing tries to solve that I have seen. Short and to the point.
- mrits 12y ago
  I disagree. I mean sure, I put humans and radiation in the same pingeonhole, but thats more of a devops problem to work out.
- solipsism 12y ago
  Maybe to someone who's already familiar with other, longer descriptions of the problem. I don't know what problem this is solving and that description doesn't do much for me. Can someone either explain, or point to a fuller explanation, of the purported problem?
  - Terr_ 12y ago
    There are some overlapping architectural ideas here, I find they build upwards like this: CQRS: You have one "write" data-model which is specialized for transactions and applying business rules... and other "read" models that take care of things like dashboards, detailed information pages, searching, generating reports, etc. Chronological events: A really handy way to propagate changes from the write-model into the various read-models. Event-sourcing: This is "dogfooding" your own events. Instead of just emitting optional messages to other systems, you make them essential, and use them as the authoritative record for your own system's state. This adds certain kinds of flexibility that you may (or may not) need.
    - dj-wonk 12y ago
      If one describes CQRS as being mostly about a write (e.g. master data) + read (e.g. derived data) combination, how far could one get with a relational system with views and triggers?
      - Terr_ 12y ago
        Pretty far, I imagine, especially with things like materialized views... I'm just biased against putting more logic into the DB than strictly necessary, given the difficulty of versioning that logic and how the DB is often the bottleneck among multiple webservers.
  - jamii 12y ago
    This is an excellent explanation - http://blog.confluent.io/2015/01/29/making-sense-of-stream-processing/ http://blog.confluent.io/2015/01/29/making-sense-of-stream-p...
    - CmdrDats 12y ago
      Thanks jamii, that's an excellent read - I forgot to add it to the Further Reading section! Will do that now :)
    - reubano 12y ago
      Very nice!Here's a good follow up on the inner workings of onyx. https://github.com/MichaelDrogalis/onyx/blob/0.5.x/doc/user-guide/internal-design.md https://github.com/MichaelDrogalis/onyx/blob/0.5.x/doc/user-...
- Terr_ 12y ago
  I'd add: "All data-models have strengths and weaknesses. The event-log is weak on its own, but its strength is that it nurtures an ecosystem of other models for specific use-cases."
- deleted 12y ago
  [deleted]
- dragonwriter 12y ago
  > That is the best description of the problem event sourcing tries to solve that I have seen. Huh. I always thought that the problem event sourcing was trying to solve was more that modeling complex interactive systems through their static state rather than modeling the interactions directly as the principal domain objects is invariably futile. While "we're dropping all sorts of interesting data on the ground" may be one manifestation of that, I think the more common manifestation is "we bear too high of an expense when adapting our information system to changes in business process".
  - solipsism 12y ago
    modeling complex interactive systems through their static state rather than modeling the interactions directly as the principal domain objects is invariably futile Now that actually makes sense to someone not already familiar with the context.
    - vosper 12y ago
      I agree - that was the best explanation of CQRS I've seen.
      - socceroos 12y ago
        CQRS != Event Sourcing. ;)
jared314 12y ago
I was under the impression that Datomic alone could handle everything involved with an eventstore, just by adding the domain event as an attribute on the "transaction" (because transactions themselves are also entities). I don't understand what this adds.
- CmdrDats 12y ago
  Technically it could - but I felt Datomic is particularly strong as an aggregate view. Adding transactions that just have events attached to them isn't a particularly useful aggregate to query. Having the eventstore separate means that you can keep Datomic focussed at what it's really good at. When you find good aggregate views for your events, you can scan through old events and populate Datomic. This separation also means that you can make the events as rich and plentiful as you want without burdening Datomic with stuff that you don't have a good way to query anyhow.
  - jared314 12y ago
    > Adding transactions that just have events attached to them isn't a particularly useful aggregate to query. I still don't understand. The transaction log, in datomic, can be queried as easily as you would query the rest of the data. The view exists on the client. Changes are already queued by the transactor. And, changes are pushed to the clients from the transactor.
    - CmdrDats 12y ago
      So to have a concrete example: in our preliminary test of Datomic, we threw all our page view events directly at Datomic. Every single attribute in the event gets indexed at least 3 times and it grew out of a manageable scale really quickly. We had to find a solution where we can still maintain that data and aggregate it into Datomic when and where we can make useful sense out of it. In certain cases you can definitely put the events directly into the transactions. In our case, it's just not the right fit and I suspect that many others will find the same constraints apply.
mandeepj 12y ago
There is a guy by name Udi Dahan. He is a self proclaimed CQRS guru. He also sells his trainings for at least few grands. I have listened to his podcasts and read his blogs but I never got any clue what is CQRS and under what scenarios it is best to go with this pattern. Here, I just read the first page and I got it what it is and when I can use it.
_pmf_ 12y ago
I like this trend of indicating articles as being opinionated. It acknowledges the fact that there's no be-all end-all solution and prevents having to include useless fuzzy apologies and hand waving in the article.
crumley 12y ago
I really like the ideas behind CQRS and Event Sourcing. So much so we are building a platform using these patterns. Want to live in Austin, TX and build a data platform from the ground up? We're hiring: https://careers.stackoverflow.com/jobs/80205/senior-developer-data-platform-nuve https://careers.stackoverflow.com/jobs/80205/senior-develope...

111 points by CmdrDats 12y ago

The current relational database is too limiting. We're dropping all sorts of interesting data on the ground because we don't have suitable pigeonholes to put it into. That is the best description of the problem event sourcing tries to solve that I have seen. Short and to the point.

mrits 12y ago
I disagree. I mean sure, I put humans and radiation in the same pingeonhole, but thats more of a devops problem to work out.
solipsism 12y ago
Maybe to someone who's already familiar with other, longer descriptions of the problem. I don't know what problem this is solving and that description doesn't do much for me. Can someone either explain, or point to a fuller explanation, of the purported problem?
- Terr_ 12y ago
  There are some overlapping architectural ideas here, I find they build upwards like this: CQRS: You have one "write" data-model which is specialized for transactions and applying business rules... and other "read" models that take care of things like dashboards, detailed information pages, searching, generating reports, etc. Chronological events: A really handy way to propagate changes from the write-model into the various read-models. Event-sourcing: This is "dogfooding" your own events. Instead of just emitting optional messages to other systems, you make them essential, and use them as the authoritative record for your own system's state. This adds certain kinds of flexibility that you may (or may not) need.
  - dj-wonk 12y ago
    If one describes CQRS as being mostly about a write (e.g. master data) + read (e.g. derived data) combination, how far could one get with a relational system with views and triggers?
    - Terr_ 12y ago
      Pretty far, I imagine, especially with things like materialized views... I'm just biased against putting more logic into the DB than strictly necessary, given the difficulty of versioning that logic and how the DB is often the bottleneck among multiple webservers.
- jamii 12y ago
  This is an excellent explanation - http://blog.confluent.io/2015/01/29/making-sense-of-stream-processing/ http://blog.confluent.io/2015/01/29/making-sense-of-stream-p...
  - CmdrDats 12y ago
    Thanks jamii, that's an excellent read - I forgot to add it to the Further Reading section! Will do that now :)
  - reubano 12y ago
    Very nice!Here's a good follow up on the inner workings of onyx. https://github.com/MichaelDrogalis/onyx/blob/0.5.x/doc/user-guide/internal-design.md https://github.com/MichaelDrogalis/onyx/blob/0.5.x/doc/user-...
Terr_ 12y ago
I'd add: "All data-models have strengths and weaknesses. The event-log is weak on its own, but its strength is that it nurtures an ecosystem of other models for specific use-cases."
deleted 12y ago
[deleted]
dragonwriter 12y ago
> That is the best description of the problem event sourcing tries to solve that I have seen. Huh. I always thought that the problem event sourcing was trying to solve was more that modeling complex interactive systems through their static state rather than modeling the interactions directly as the principal domain objects is invariably futile. While "we're dropping all sorts of interesting data on the ground" may be one manifestation of that, I think the more common manifestation is "we bear too high of an expense when adapting our information system to changes in business process".
- solipsism 12y ago
  modeling complex interactive systems through their static state rather than modeling the interactions directly as the principal domain objects is invariably futile Now that actually makes sense to someone not already familiar with the context.
  - vosper 12y ago
    I agree - that was the best explanation of CQRS I've seen.
    - socceroos 12y ago
      CQRS != Event Sourcing. ;)

jared314 12y ago

I was under the impression that Datomic alone could handle everything involved with an eventstore, just by adding the domain event as an attribute on the "transaction" (because transactions themselves are also entities). I don't understand what this adds.

CmdrDats 12y ago
Technically it could - but I felt Datomic is particularly strong as an aggregate view. Adding transactions that just have events attached to them isn't a particularly useful aggregate to query. Having the eventstore separate means that you can keep Datomic focussed at what it's really good at. When you find good aggregate views for your events, you can scan through old events and populate Datomic. This separation also means that you can make the events as rich and plentiful as you want without burdening Datomic with stuff that you don't have a good way to query anyhow.
- jared314 12y ago
  > Adding transactions that just have events attached to them isn't a particularly useful aggregate to query. I still don't understand. The transaction log, in datomic, can be queried as easily as you would query the rest of the data. The view exists on the client. Changes are already queued by the transactor. And, changes are pushed to the clients from the transactor.
  - CmdrDats 12y ago
    So to have a concrete example: in our preliminary test of Datomic, we threw all our page view events directly at Datomic. Every single attribute in the event gets indexed at least 3 times and it grew out of a manageable scale really quickly. We had to find a solution where we can still maintain that data and aggregate it into Datomic when and where we can make useful sense out of it. In certain cases you can definitely put the events directly into the transactions. In our case, it's just not the right fit and I suspect that many others will find the same constraints apply.

mandeepj 12y ago

There is a guy by name Udi Dahan. He is a self proclaimed CQRS guru. He also sells his trainings for at least few grands. I have listened to his podcasts and read his blogs but I never got any clue what is CQRS and under what scenarios it is best to go with this pattern. Here, I just read the first page and I got it what it is and when I can use it.

_pmf_ 12y ago

I like this trend of indicating articles as being opinionated. It acknowledges the fact that there's no be-all end-all solution and prevents having to include useless fuzzy apologies and hand waving in the article.

crumley 12y ago

I really like the ideas behind CQRS and Event Sourcing. So much so we are building a platform using these patterns. Want to live in Austin, TX and build a data platform from the ground up? We're hiring: https://careers.stackoverflow.com/jobs/80205/senior-developer-data-platform-nuve https://careers.stackoverflow.com/jobs/80205/senior-develope...