8 ms·

Turning the database inside-out (2015)

192 points by andection 2y ago

jackeyzhang 2y ago
Awesome，very useful
esafak 2y ago
(2015)
swyx 2y ago
relatedly: https://restate.dev/blog/every-system-is-a-log-avoiding-coordination-in-distributed-applications/ https://restate.dev/blog/every-system-is-a-log-avoiding-coor... https://news.ycombinator.com/item?id=42813049 https://news.ycombinator.com/item?id=42813049
dagss 2y ago
We did this style on top of plain MSSQL. Each event would have a SQL table which is the primary storage. Then we have workers that listens to new data in tables and updates projections we needed. (Sometimes DB triggers but mostly async workers.) The main issue is "listening to new data in a SQL table". I wrote this code to achieve it in MSSQL (perhaps it is somehow built into postgres?): https://github.com/vippsas/mssql-changefeed https://github.com/vippsas/mssql-changefeed In my experience this approach is beautiful; as Martins says, our backend code is mostly stateless and functional these days, why have mutable objects in the DB? And the approach is extremely useful for dry-running business logic etc But we didn't like the prospect of adopting Kafka wholesale. Having all the data in a SQL DB is extremely convenient for debugging, and since we already used SQL it was a smaller change that was done first where it made most sense and then spread out. It would be great with more DB features targeting this style. Explicit partition event tables (kafka-in-SQL), and writing a projection simply as a SQL query which is inverted into an async trigger by the DB would be awesome. (MSSQL has indexed views, but it cannot be done online...) Materialize is the DB I know about in this territory.
- hans_castorp 2y ago
  > perhaps it is somehow built into postgres? Postgres has a built-in listen/notify mechanism. The problem with that is, that it doesn't guarantee delivery and if no process is listening, notifications will be lost. Most solutions that need something like that use "logical decoding" these days. That's the built-in change data capture exposed as a public API as part of the logical replication.
  - dagss 2y ago
    Yes, listen/notify is something very different. We would often write new projections that consumes events from years back and until today. You want sequence numbers that indicate the event's position in a partitioned log. Something like "int identity" except that the int is assigned during commit, so that you have guarantee that if you see IDs 5 and 7, then 6 will never show up, so that each consumer can store a cursor of its progress of consuming the table which is safe against inserts. I was hoping to do it using CDC, but Microsoft SQL has a minimum 1 minute delay on CDC which destroys any live data usecase. Perhaps postgres allows listening to the replication log with lower latency?
    - andyferris 2y ago
      > Perhaps postgres allows listening to the replication log with lower latency? Yes, I think that's what the "logical decoding" referred to. Postgres can emit a "logical" version of the WAL (something with a stable spec writtten down so that other services can stream and decode it). My understanding was that "logical replication" was designed for low latency situations like creating read replicas. I haven't heard of the logical log being preserved for "years back" but that's an interesting case...
      - dagss 2y ago
        That is OK, guess I would write a job to listen to the logical WAL and use it to do an update that writes an event sequence number.
    - 2y ago
joshlemer 2y ago
The thing I always get stuck on with these techniques is, how do you handle transactions which perform validations/enforce invariants on data when you’re just writing writes to a log and computing materialized views down the line? How can you do essentially, an “add item to shopping cart” if for example, users can only have max 10 items and so you need to validate that there aren’t already 10 items in the cart?
- swiftcoder 2y ago
  I assume that shopping cart limit is a made-up example, but I'm curious what preconditions are you actually enforcing in the real world via DB transaction rollback?
- cwalv 2y ago
  You write the 'add item' event regardless, and when building the 'cart' view you handle the limit.
  - philbo 2y ago
    Alternatively "invalid cart" could itself become an event.
  - RedShift1 2y ago
    Sounds like an easy way to run out of storage space
  - eyads 2y ago
    add_item is not an event, rather a command/ request that is yet to be validated. item_added is the event = a fact that was 'allowed to happen' by the system. Keeping commands in a persistent store is a matter of choice but not necessary. I've seen people doing command sourcing and calling it event sourcing.
  - dagss 2y ago
    Well, assume the non-overdraftable bank account example instead then, what do you do then?
- 2y ago
Joker_vD 2y ago
> Databases are global, shared, mutable state. [...] However, most self-respecting developers have got rid of mutable global variables in their code long ago. So why do we tolerate databases as they are? Because the world itself is a global, shared, mutable state (which, incidentally, is also a single source of truth) and databases were invented to mirror it (well, relevant parts of it) 1-to-1, or close to it. This style of "we use the database as a proxy of the physical world itself" is still pretty common, see e.g. the example with the shopping cart somewhere else in these comments.
- reubenmorais 2y ago
  Seeing the world as mutable is a matter of perspective, if you explicitly model time as a dimension it can instead be seen as a sequence of transitions from immutable state to immutable state, an accumulation of events over time, which fits the log abstraction perfectly.
  - hahn-kev 2y ago
    Except lots of applications need to be able to forget things, similar to how things can be destroyed in the real world
    - fnordsensei 2y ago
      Forgetting what you had for lunch last Friday is not necessarily the same as changing what you had for lunch last Friday. You could, for example, throw away the encryption key for that fact. What you had for lunch is now inaccessible, but the fact remains unchanged.
  - Swizec 2y ago
    > if you explicitly model time as a dimension it can instead be seen as a sequence of transitions from immutable state to immutable state, an accumulation of events over time, which fits the log abstraction perfectly I worked with a feature that used this approach once. It even made sense for the feature (an immutable history log of patient chart data). It was absolute hell to work with. Querying current state, which was 99% of the usecases, was cumbersome and extremely slow. Turns out doctors rarely care about any of that immutable history. They just wanna know what’s up right now. In their ideal world, you’d re-answer all the same questions 30 seconds before walking in the door. Turns out a combination mutable table of current state + derived immutable log/snapshot/audit table works much better for most things.
fungiblecog 2y ago
Datomic
- agumonkey 2y ago
  My first thought. And then I realized that the talk is from 2015, so I wonder if there was cross pollination between people at the time (datomic was released in 2012 according to wikipedia, so it's plausible it predates Martin's ideas but I don't know) https://news.ycombinator.com/item?id=20937215 https://news.ycombinator.com/item?id=20937215
- amelius 2y ago
  The concept is even older than that.
  - guskel 2y ago
    Yeah, like 15th century old.
    - blacklion 2y ago
      More like 5th or 6th century BCE ;-)
mgaunard 2y ago
Stopped reading at the word "Kafka".
- chikere232 2y ago
  fair
chikere232 2y ago
Isn't this essentially how a modern transactional database works anyway? All mutations end up in the Write Ahead Log (WAL) and you can replicate or back up that to be able to recover the state at a point in time?
- isbvhodnvemrwvn 2y ago
  One difficult to replicate thing is visibility rules and rollbacks, with postgres you can abort and your changes are hidden, no such luxuries worth this architecture unless you make it very complex with partial states, drafts or something similar.
- mrkeen 2y ago
  Two points: * Technically the data is probably there, but I really don't think you want back-up ops invoked by your REST call to /getUserHistory/. Is it even possible to mix old data and new data within the same SQL expression? * The DB is still a god object at the centre of your system. It doesn't give you consistency across partner systems and end users. If a partner sends the event CustomerBanned(2025-02-04, 1234) and you try to translate it into CRUD with 'UPDATE Customer SET Banned=True WHERE id=1234' it could fail (or worse - be rejected by an invariant for "data integrity" reasons) and then it's gone. If you just blindly write the event, then you always know that fact about customer 1234 in any future query.
  - zbentley 2y ago
    > If you just blindly write the event, then you always know that fact about customer 1234 in any future query. Unless the write times out or the DB is down for maintenance when the event arrives. Sure, you could block acknowledgement of the event until the event log receives it, but can your DB handle synchronous write volume from however many people are out there sending events? If your RPC servers listening for events are unavailable, do you trust event senders to retry when they're back? Down that road lies "let's put every event in a fast message bus with higher insert volume and availability than the database, and feed that into the DB asynchronously", hence Kafka and friends.
Xenoamorphous 2y ago
Does any one have some resources where a real, practical example is implemented? Because I can only find fairly theoretical resources but not real world examples. Say, like how some simple CMS would work with a datastore like this. What does the event to update the headline of an article look like? How are integrity constraints enforced, e.g. an article can't reference an author that doesn't exist? Things like that.
- lamp_book 2y ago
  Similar enough is double entry bookkeeping and generally I think event sourcing is more common for fraud detection, log analysis for security, etc. - use cases where how the application got in its state is as important as the state.
sriku 2y ago
https://materialize.com/ https://materialize.com/ provides another approach, based on "timely dataflow" (https://timelydataflow.github.io/timely-dataflow/ https://timelydataflow.github.io/timely-dataflow/) - originated at MS.
belter 2y ago
This is Aurora... https://pages.cs.wisc.edu/~yxy/cs764-f20/papers/aurora-sigmod-17.pdf https://pages.cs.wisc.edu/~yxy/cs764-f20/papers/aurora-sigmo...
- mrkeen 2y ago
  This is the technique that developers used to build Aurora, not Aurora the end-product. Customers writing code against Aurora are still doing plain ol' destructive CRUD mutations "now". Event-sourcing is write-ahead-logging is CQRS is journaling-file-systems is Git-reflog is persistent-data-structures is copy-on-write. It's all good stuff and is decades old.
  - belter 2y ago
    > This is the technique that developers used to build Aurora, not Aurora the end-product. It's what I meant. But I am behaving like an LLM and economizing on tokens... :-)
anacrolix 2y ago
Isn't this what you get with Datomic?
crabbone 2y ago
> So why do we tolerate databases as they are? Because they reflect the way we understand the world? We understand that things are made of smaller things, and that sometimes the smaller things making up larger things may change, while the larger thing stays the same? The idea that as soon as one component changes the whole thing needs to be discarded and rebuilt from ground up is insane and creates a lot of problems. It's absolutely not worth it to try to redefine the way we deal with the world to get the benefits of stateless code. Making database stateless is making it worthless. The world has a state, and if you want a useful program, it needs to accept this "unfortunate" aspect of the world. The alternative is the world where as soon as you finish drinking your coffee, your cup, your table, your kitchen, your credit card history, your grandparents and all planets in the solar system disappear, and have to be built fresh. But you wouldn't know about it, because your memory of how the world used to be would disappear too.
deleted 2y ago
[deleted]
agentultra 2y ago
This is also known as event sourcing [0] and is a common pattern used inside of databases, in git, in lots of popular software. I don't generally recommend it for every application as the tooling is not as well integrated as it is in an RDBMS and the data model doesn't fit every use-case. However, if you have a system that needs to know "when" something happened in an on-going process, it can be a very handy architecture... although with data-retention laws it can get tricky quickly (among other reasons). [0] https://martinfowler.com/eaaDev/EventSourcing.html https://martinfowler.com/eaaDev/EventSourcing.html
- arialdomartini 2y ago
  I saw other times Git being related to event sourcing, but the argument is wrong. Most of the VCSs before Git (RCS, CVS, SVN) used to store deltas and to rebuild the state reapplying them. The very reason why Git took them by the storm is exactly because, on the contrary, Git does not store deltas but snapshots. Each commit is not there collection of the occurred chances but a complete snapshot of the whole project. Git is very efficient in reusing the blob objects to save space, but it’s still a whole snapshot. The occurred changes are not stored, and they are calculated on demand. The very opposite of event sourcing, where it’s the state to be calculated and the occurred changes / events to be stored. Git is really the demonstration that for code versioning state sourcing is way more efficient than event sourcing.
  - jdkoeck 2y ago
    Git is still event sourced, it’s just there is only one kind of event (a commit), and its payload is the whole state ¯\_(ツ)_/¯
    - arialdomartini 2y ago
      Eh eh, this is an interesting point of view, but it’s really not like this. Take the case of the event of “deleting a file”. There has been an interesting discussion between Linus and the orher developers, when Git was being d initially esigned: some of them wanted to capture and track this event. Linus firmly rejected the whole idea of track events, providing very solid arguments http://web.archive.org/web/20200117061404/http://www.gelato.unsw.edu.au:80/archives/git/0504/0598.html http://web.archive.org/web/20200117061404/http://www.gelato....
kragen 2y ago
It seems like event sourcing keeps gaining mindshare.
- alecco 2y ago
  Event sourcing is a PITA, from experience. Something like Datomic makes a lot more sense: https://vvvvalvalval.github.io/posts/2018-11-12-datomic-event-sourcing-without-the-hassle.html https://vvvvalvalval.github.io/posts/2018-11-12-datomic-even...
  - kragen 2y ago
    Thanks! Does the article accurately describe the PITA in your experience? Because it seems to say that it's separable from the core architectural principle of event sourcing.
    - alecco 2y ago
      In my experience most event-sourcing was implemented as storing versions of objects (it came from the OOP camp). All the consistency checks had to be done manually in imperative code across countless classes. Many large investment banks use it. And then all the actual DB stuff has to be exported to SQL or other actual database engines to be processed properly.
      - kragen 2y ago
        No true Scotsman! No true Scotsman! (Which is to say, that sure isn't what I thought "event sourcing" was.)
dang 2y ago
Related: Turning the database inside out (2014) [video] - https://news.ycombinator.com/item?id=41664271 https://news.ycombinator.com/item?id=41664271 - Sept 2024 (1 comment) Turning the database inside-out with Apache Samza (2015) - https://news.ycombinator.com/item?id=13581096 https://news.ycombinator.com/item?id=13581096 - Feb 2017 (30 comments) Turning the database inside-out with Apache Samza - https://news.ycombinator.com/item?id=9145197 https://news.ycombinator.com/item?id=9145197 - March 2015 (64 comments)

192 points by andection 2y ago

jackeyzhang 2y ago

Awesome，very useful

esafak 2y ago

(2015)

swyx 2y ago

relatedly: https://restate.dev/blog/every-system-is-a-log-avoiding-coordination-in-distributed-applications/ https://restate.dev/blog/every-system-is-a-log-avoiding-coor... https://news.ycombinator.com/item?id=42813049 https://news.ycombinator.com/item?id=42813049

dagss 2y ago

We did this style on top of plain MSSQL. Each event would have a SQL table which is the primary storage. Then we have workers that listens to new data in tables and updates projections we needed. (Sometimes DB triggers but mostly async workers.) The main issue is "listening to new data in a SQL table". I wrote this code to achieve it in MSSQL (perhaps it is somehow built into postgres?): https://github.com/vippsas/mssql-changefeed https://github.com/vippsas/mssql-changefeed In my experience this approach is beautiful; as Martins says, our backend code is mostly stateless and functional these days, why have mutable objects in the DB? And the approach is extremely useful for dry-running business logic etc But we didn't like the prospect of adopting Kafka wholesale. Having all the data in a SQL DB is extremely convenient for debugging, and since we already used SQL it was a smaller change that was done first where it made most sense and then spread out. It would be great with more DB features targeting this style. Explicit partition event tables (kafka-in-SQL), and writing a projection simply as a SQL query which is inverted into an async trigger by the DB would be awesome. (MSSQL has indexed views, but it cannot be done online...) Materialize is the DB I know about in this territory.

hans_castorp 2y ago
> perhaps it is somehow built into postgres? Postgres has a built-in listen/notify mechanism. The problem with that is, that it doesn't guarantee delivery and if no process is listening, notifications will be lost. Most solutions that need something like that use "logical decoding" these days. That's the built-in change data capture exposed as a public API as part of the logical replication.
- dagss 2y ago
  Yes, listen/notify is something very different. We would often write new projections that consumes events from years back and until today. You want sequence numbers that indicate the event's position in a partitioned log. Something like "int identity" except that the int is assigned during commit, so that you have guarantee that if you see IDs 5 and 7, then 6 will never show up, so that each consumer can store a cursor of its progress of consuming the table which is safe against inserts. I was hoping to do it using CDC, but Microsoft SQL has a minimum 1 minute delay on CDC which destroys any live data usecase. Perhaps postgres allows listening to the replication log with lower latency?
  - andyferris 2y ago
    > Perhaps postgres allows listening to the replication log with lower latency? Yes, I think that's what the "logical decoding" referred to. Postgres can emit a "logical" version of the WAL (something with a stable spec writtten down so that other services can stream and decode it). My understanding was that "logical replication" was designed for low latency situations like creating read replicas. I haven't heard of the logical log being preserved for "years back" but that's an interesting case...
    - dagss 2y ago
      That is OK, guess I would write a job to listen to the logical WAL and use it to do an update that writes an event sequence number.
  - 2y ago

joshlemer 2y ago

The thing I always get stuck on with these techniques is, how do you handle transactions which perform validations/enforce invariants on data when you’re just writing writes to a log and computing materialized views down the line? How can you do essentially, an “add item to shopping cart” if for example, users can only have max 10 items and so you need to validate that there aren’t already 10 items in the cart?

swiftcoder 2y ago
I assume that shopping cart limit is a made-up example, but I'm curious what preconditions are you actually enforcing in the real world via DB transaction rollback?
cwalv 2y ago
You write the 'add item' event regardless, and when building the 'cart' view you handle the limit.
- philbo 2y ago
  Alternatively "invalid cart" could itself become an event.
- RedShift1 2y ago
  Sounds like an easy way to run out of storage space
- eyads 2y ago
  add_item is not an event, rather a command/ request that is yet to be validated. item_added is the event = a fact that was 'allowed to happen' by the system. Keeping commands in a persistent store is a matter of choice but not necessary. I've seen people doing command sourcing and calling it event sourcing.
- dagss 2y ago
  Well, assume the non-overdraftable bank account example instead then, what do you do then?
2y ago

Joker_vD 2y ago

> Databases are global, shared, mutable state. [...] However, most self-respecting developers have got rid of mutable global variables in their code long ago. So why do we tolerate databases as they are? Because the world itself is a global, shared, mutable state (which, incidentally, is also a single source of truth) and databases were invented to mirror it (well, relevant parts of it) 1-to-1, or close to it. This style of "we use the database as a proxy of the physical world itself" is still pretty common, see e.g. the example with the shopping cart somewhere else in these comments.

reubenmorais 2y ago
Seeing the world as mutable is a matter of perspective, if you explicitly model time as a dimension it can instead be seen as a sequence of transitions from immutable state to immutable state, an accumulation of events over time, which fits the log abstraction perfectly.
- hahn-kev 2y ago
  Except lots of applications need to be able to forget things, similar to how things can be destroyed in the real world
  - fnordsensei 2y ago
    Forgetting what you had for lunch last Friday is not necessarily the same as changing what you had for lunch last Friday. You could, for example, throw away the encryption key for that fact. What you had for lunch is now inaccessible, but the fact remains unchanged.
- Swizec 2y ago
  > if you explicitly model time as a dimension it can instead be seen as a sequence of transitions from immutable state to immutable state, an accumulation of events over time, which fits the log abstraction perfectly I worked with a feature that used this approach once. It even made sense for the feature (an immutable history log of patient chart data). It was absolute hell to work with. Querying current state, which was 99% of the usecases, was cumbersome and extremely slow. Turns out doctors rarely care about any of that immutable history. They just wanna know what’s up right now. In their ideal world, you’d re-answer all the same questions 30 seconds before walking in the door. Turns out a combination mutable table of current state + derived immutable log/snapshot/audit table works much better for most things.

fungiblecog 2y ago

Datomic

agumonkey 2y ago
My first thought. And then I realized that the talk is from 2015, so I wonder if there was cross pollination between people at the time (datomic was released in 2012 according to wikipedia, so it's plausible it predates Martin's ideas but I don't know) https://news.ycombinator.com/item?id=20937215 https://news.ycombinator.com/item?id=20937215
amelius 2y ago
The concept is even older than that.
- guskel 2y ago
  Yeah, like 15th century old.
  - blacklion 2y ago
    More like 5th or 6th century BCE ;-)

mgaunard 2y ago

Stopped reading at the word "Kafka".

chikere232 2y ago
fair

chikere232 2y ago

Isn't this essentially how a modern transactional database works anyway? All mutations end up in the Write Ahead Log (WAL) and you can replicate or back up that to be able to recover the state at a point in time?

isbvhodnvemrwvn 2y ago
One difficult to replicate thing is visibility rules and rollbacks, with postgres you can abort and your changes are hidden, no such luxuries worth this architecture unless you make it very complex with partial states, drafts or something similar.
mrkeen 2y ago
Two points: * Technically the data is probably there, but I really don't think you want back-up ops invoked by your REST call to /getUserHistory/. Is it even possible to mix old data and new data within the same SQL expression? * The DB is still a god object at the centre of your system. It doesn't give you consistency across partner systems and end users. If a partner sends the event CustomerBanned(2025-02-04, 1234) and you try to translate it into CRUD with 'UPDATE Customer SET Banned=True WHERE id=1234' it could fail (or worse - be rejected by an invariant for "data integrity" reasons) and then it's gone. If you just blindly write the event, then you always know that fact about customer 1234 in any future query.
- zbentley 2y ago
  > If you just blindly write the event, then you always know that fact about customer 1234 in any future query. Unless the write times out or the DB is down for maintenance when the event arrives. Sure, you could block acknowledgement of the event until the event log receives it, but can your DB handle synchronous write volume from however many people are out there sending events? If your RPC servers listening for events are unavailable, do you trust event senders to retry when they're back? Down that road lies "let's put every event in a fast message bus with higher insert volume and availability than the database, and feed that into the DB asynchronously", hence Kafka and friends.

Xenoamorphous 2y ago

Does any one have some resources where a real, practical example is implemented? Because I can only find fairly theoretical resources but not real world examples. Say, like how some simple CMS would work with a datastore like this. What does the event to update the headline of an article look like? How are integrity constraints enforced, e.g. an article can't reference an author that doesn't exist? Things like that.

lamp_book 2y ago
Similar enough is double entry bookkeeping and generally I think event sourcing is more common for fraud detection, log analysis for security, etc. - use cases where how the application got in its state is as important as the state.

sriku 2y ago

https://materialize.com/ https://materialize.com/ provides another approach, based on "timely dataflow" (https://timelydataflow.github.io/timely-dataflow/ https://timelydataflow.github.io/timely-dataflow/) - originated at MS.

belter 2y ago

This is Aurora... https://pages.cs.wisc.edu/~yxy/cs764-f20/papers/aurora-sigmod-17.pdf https://pages.cs.wisc.edu/~yxy/cs764-f20/papers/aurora-sigmo...

mrkeen 2y ago
This is the technique that developers used to build Aurora, not Aurora the end-product. Customers writing code against Aurora are still doing plain ol' destructive CRUD mutations "now". Event-sourcing is write-ahead-logging is CQRS is journaling-file-systems is Git-reflog is persistent-data-structures is copy-on-write. It's all good stuff and is decades old.
- belter 2y ago
  > This is the technique that developers used to build Aurora, not Aurora the end-product. It's what I meant. But I am behaving like an LLM and economizing on tokens... :-)

anacrolix 2y ago

Isn't this what you get with Datomic?

crabbone 2y ago

> So why do we tolerate databases as they are? Because they reflect the way we understand the world? We understand that things are made of smaller things, and that sometimes the smaller things making up larger things may change, while the larger thing stays the same? The idea that as soon as one component changes the whole thing needs to be discarded and rebuilt from ground up is insane and creates a lot of problems. It's absolutely not worth it to try to redefine the way we deal with the world to get the benefits of stateless code. Making database stateless is making it worthless. The world has a state, and if you want a useful program, it needs to accept this "unfortunate" aspect of the world. The alternative is the world where as soon as you finish drinking your coffee, your cup, your table, your kitchen, your credit card history, your grandparents and all planets in the solar system disappear, and have to be built fresh. But you wouldn't know about it, because your memory of how the world used to be would disappear too.

deleted 2y ago

[deleted]

agentultra 2y ago

This is also known as event sourcing [0] and is a common pattern used inside of databases, in git, in lots of popular software. I don't generally recommend it for every application as the tooling is not as well integrated as it is in an RDBMS and the data model doesn't fit every use-case. However, if you have a system that needs to know "when" something happened in an on-going process, it can be a very handy architecture... although with data-retention laws it can get tricky quickly (among other reasons). [0] https://martinfowler.com/eaaDev/EventSourcing.html https://martinfowler.com/eaaDev/EventSourcing.html

arialdomartini 2y ago
I saw other times Git being related to event sourcing, but the argument is wrong. Most of the VCSs before Git (RCS, CVS, SVN) used to store deltas and to rebuild the state reapplying them. The very reason why Git took them by the storm is exactly because, on the contrary, Git does not store deltas but snapshots. Each commit is not there collection of the occurred chances but a complete snapshot of the whole project. Git is very efficient in reusing the blob objects to save space, but it’s still a whole snapshot. The occurred changes are not stored, and they are calculated on demand. The very opposite of event sourcing, where it’s the state to be calculated and the occurred changes / events to be stored. Git is really the demonstration that for code versioning state sourcing is way more efficient than event sourcing.
- jdkoeck 2y ago
  Git is still event sourced, it’s just there is only one kind of event (a commit), and its payload is the whole state ¯\_(ツ)_/¯
  - arialdomartini 2y ago
    Eh eh, this is an interesting point of view, but it’s really not like this. Take the case of the event of “deleting a file”. There has been an interesting discussion between Linus and the orher developers, when Git was being d initially esigned: some of them wanted to capture and track this event. Linus firmly rejected the whole idea of track events, providing very solid arguments http://web.archive.org/web/20200117061404/http://www.gelato.unsw.edu.au:80/archives/git/0504/0598.html http://web.archive.org/web/20200117061404/http://www.gelato....

kragen 2y ago

It seems like event sourcing keeps gaining mindshare.

alecco 2y ago
Event sourcing is a PITA, from experience. Something like Datomic makes a lot more sense: https://vvvvalvalval.github.io/posts/2018-11-12-datomic-event-sourcing-without-the-hassle.html https://vvvvalvalval.github.io/posts/2018-11-12-datomic-even...
- kragen 2y ago
  Thanks! Does the article accurately describe the PITA in your experience? Because it seems to say that it's separable from the core architectural principle of event sourcing.
  - alecco 2y ago
    In my experience most event-sourcing was implemented as storing versions of objects (it came from the OOP camp). All the consistency checks had to be done manually in imperative code across countless classes. Many large investment banks use it. And then all the actual DB stuff has to be exported to SQL or other actual database engines to be processed properly.
    - kragen 2y ago
      No true Scotsman! No true Scotsman! (Which is to say, that sure isn't what I thought "event sourcing" was.)

dang 2y ago

Related: Turning the database inside out (2014) [video] - https://news.ycombinator.com/item?id=41664271 https://news.ycombinator.com/item?id=41664271 - Sept 2024 (1 comment) Turning the database inside-out with Apache Samza (2015) - https://news.ycombinator.com/item?id=13581096 https://news.ycombinator.com/item?id=13581096 - Feb 2017 (30 comments) Turning the database inside-out with Apache Samza - https://news.ycombinator.com/item?id=9145197 https://news.ycombinator.com/item?id=9145197 - March 2015 (64 comments)