5 ms·

SierraDB: A distributed event store built in Rust

74 points by tqwewe 11mo ago

jayy-lmao 11mo ago
This looks really cool! Have always wanted to dip my toe in EventStoreDB/Kurrent but this looks super intuitive and nice to use. Especially like the js projections, can imagine it being really handy in prototyping or ad-hoc reporting.
jauntywundrkind 11mo ago
In memory partitions, yeah? It's persisted to S3 storage, but SlateDB feels like it might sort of have some fit, maybe, as a scale-out distributed LSM-tree. https://slatedb.io https://slatedb.io https://news.ycombinator.com/item?id=41714858 https://news.ycombinator.com/item?id=41714858 There's an old 404 post too that looks like a reasonably on target introduction: Why SlateDB is the right choice for Stream Processing. https://web.archive.org/web/20241102212325/https://www.responsive.dev/blog/why-slatedb-for-kafka-streams https://web.archive.org/web/20241102212325/https://www.respo...
- curtistyr 11mo ago
  Interesting point about SlateDB - I've been thinking about how different architectures handle event sourcing and stream processing. SierraDB's append-only model with fixed partitions is really compelling for event sourcing, but I'm curious how it compares to something like SlateDB when you need more general-purpose streaming capabilities. Do you think the trade-offs between these approaches are starting to converge, or are they solving fundamentally different problems? Also, SierraDB's use of RESP3 is smart - anything that reduces client complexity is a win in my book.
conceptme 11mo ago
Does it also have snapshot capabilities, as mostly over time it becomes very difficult to replay events due to the shear amount of them.
- tqwewe 11mo ago
  I haven't put any effort into any kind of snapshotting capabilities yet, since I won't want the scope to be too large and there's often ways of designing your system in ways where replaying isn't a big issue (the db scans are fast, designing aggregates to have a smaller scope/less overall events, etc). But with increased resources, I can see some solutions being considered around snapshotting. For now the goal is heavily unix philosophy inspired: a really fast and purpose built event sourcing database
12_throw_away 11mo ago
Very excited to see how this progresses! Honestly, it's always a little surprising to me that event store architectures aren't more widely used. The article is extremely correct about why: > there's absolutely no clear cut way of approaching it for new projects That's definitely my experience - there's no open source "batteries-included" event store where you can just `docker compose up` and start sourcing your events right away. (Maybe KurrentDB née EventStoreDB might offer something like this? But unfortunately, it has a weird license and feels heavily pivoted towards SaaS). And if you want to implement it yourself, a lot of the writing about event stores comes from the the Enterprise Design Patterns™ world.
- tqwewe 11mo ago
  Yes exactly, I heard from an existing KurrentDB customer that the weird licensing change was actually a deal breaker causing them to move away from KurrentDB despite the migration pains. I think a community, open source built project in Rust has been a missing piece I can hopefully help to solve.
_mocha 11mo ago
I've wanted to do this for the last 7 years and never got around to it. Couldn't be more proud that you put in the grit to bring this to fruition!
Havoc 11mo ago
Was a bit surprised to see this end in a docker instruction rather than k8s given the emphasis on multi node, replication etc.
- CaptainOfCoit 11mo ago
  Makes sense to me, lowest common denominator across people who deal with containers is probably basic docker usage. So providing instructions with docker lets people using Nomad pick it up as quickly as people who are using Kubernetes, or any other runtime.
rufusthedogwoof 11mo ago
Hi. I feel like I've been using XTDB as an event store? https://docs.xtdb.com/intro/installation-via-docker.html https://docs.xtdb.com/intro/installation-via-docker.html
- refset 11mo ago
  SierraDB looks closer to Rama than XTDB https://blog.redplanetlabs.com/2024/01/09/everything-wrong-with-databases-and-why-their-complexity-is-now-unnecessary/ https://blog.redplanetlabs.com/2024/01/09/everything-wrong-w... XTDB doesn't currently solve the problems of user-defined projections (via stored procedures, triggers, Incremental View Maintenance etc.) or multi-partition scaling.
int3trap 11mo ago
> Here's where SierraDB diverges from traditional distributed databases: reads don't require quorum. Would we say this is divergent? Cassandra, DynamoDB, and many others allow you to specify the consistency of reads at the request level. > Here's where SierraDB diverges from traditional distributed databases: reads don't require quorum. Instead, each event stores a confirmation count in its metadata. When a write achieves quorum, a background process broadcasts this confirmation to all replicas, updating their local confirmation counts. This means any single node can serve consistent reads without network round-trips - a massive performance win. I have no context outside of this blog post, but this seems actually divergent from the typical definition of consistency given its not linearizable. What systems benefit most from this low latency stale-but-ordered consistency guarantee?

74 points by tqwewe 11mo ago

This looks really cool! Have always wanted to dip my toe in EventStoreDB/Kurrent but this looks super intuitive and nice to use. Especially like the js projections, can imagine it being really handy in prototyping or ad-hoc reporting.

jauntywundrkind 11mo ago

In memory partitions, yeah? It's persisted to S3 storage, but SlateDB feels like it might sort of have some fit, maybe, as a scale-out distributed LSM-tree. https://slatedb.io https://slatedb.io https://news.ycombinator.com/item?id=41714858 https://news.ycombinator.com/item?id=41714858 There's an old 404 post too that looks like a reasonably on target introduction: Why SlateDB is the right choice for Stream Processing. https://web.archive.org/web/20241102212325/https://www.responsive.dev/blog/why-slatedb-for-kafka-streams https://web.archive.org/web/20241102212325/https://www.respo...

curtistyr 11mo ago
Interesting point about SlateDB - I've been thinking about how different architectures handle event sourcing and stream processing. SierraDB's append-only model with fixed partitions is really compelling for event sourcing, but I'm curious how it compares to something like SlateDB when you need more general-purpose streaming capabilities. Do you think the trade-offs between these approaches are starting to converge, or are they solving fundamentally different problems? Also, SierraDB's use of RESP3 is smart - anything that reduces client complexity is a win in my book.

conceptme 11mo ago

Does it also have snapshot capabilities, as mostly over time it becomes very difficult to replay events due to the shear amount of them.

tqwewe 11mo ago
I haven't put any effort into any kind of snapshotting capabilities yet, since I won't want the scope to be too large and there's often ways of designing your system in ways where replaying isn't a big issue (the db scans are fast, designing aggregates to have a smaller scope/less overall events, etc). But with increased resources, I can see some solutions being considered around snapshotting. For now the goal is heavily unix philosophy inspired: a really fast and purpose built event sourcing database

12_throw_away 11mo ago

Very excited to see how this progresses! Honestly, it's always a little surprising to me that event store architectures aren't more widely used. The article is extremely correct about why: > there's absolutely no clear cut way of approaching it for new projects That's definitely my experience - there's no open source "batteries-included" event store where you can just `docker compose up` and start sourcing your events right away. (Maybe KurrentDB née EventStoreDB might offer something like this? But unfortunately, it has a weird license and feels heavily pivoted towards SaaS). And if you want to implement it yourself, a lot of the writing about event stores comes from the the Enterprise Design Patterns™ world.

tqwewe 11mo ago
Yes exactly, I heard from an existing KurrentDB customer that the weird licensing change was actually a deal breaker causing them to move away from KurrentDB despite the migration pains. I think a community, open source built project in Rust has been a missing piece I can hopefully help to solve.

_mocha 11mo ago

I've wanted to do this for the last 7 years and never got around to it. Couldn't be more proud that you put in the grit to bring this to fruition!

Havoc 11mo ago

Was a bit surprised to see this end in a docker instruction rather than k8s given the emphasis on multi node, replication etc.

CaptainOfCoit 11mo ago
Makes sense to me, lowest common denominator across people who deal with containers is probably basic docker usage. So providing instructions with docker lets people using Nomad pick it up as quickly as people who are using Kubernetes, or any other runtime.

rufusthedogwoof 11mo ago

Hi. I feel like I've been using XTDB as an event store? https://docs.xtdb.com/intro/installation-via-docker.html https://docs.xtdb.com/intro/installation-via-docker.html

refset 11mo ago
SierraDB looks closer to Rama than XTDB https://blog.redplanetlabs.com/2024/01/09/everything-wrong-with-databases-and-why-their-complexity-is-now-unnecessary/ https://blog.redplanetlabs.com/2024/01/09/everything-wrong-w... XTDB doesn't currently solve the problems of user-defined projections (via stored procedures, triggers, Incremental View Maintenance etc.) or multi-partition scaling.

int3trap 11mo ago

> Here's where SierraDB diverges from traditional distributed databases: reads don't require quorum. Would we say this is divergent? Cassandra, DynamoDB, and many others allow you to specify the consistency of reads at the request level. > Here's where SierraDB diverges from traditional distributed databases: reads don't require quorum. Instead, each event stores a confirmation count in its metadata. When a write achieves quorum, a background process broadcasts this confirmation to all replicas, updating their local confirmation counts. This means any single node can serve consistent reads without network round-trips - a massive performance win. I have no context outside of this blog post, but this seems actually divergent from the typical definition of consistency given its not linearizable. What systems benefit most from this low latency stale-but-ordered consistency guarantee?