10 ms·

Every System is a Log: Avoiding coordination in distributed applications

229 points by sewen 2y ago

bfair 2y ago
Isn't this just trading consistency for availability? From what I understand the single log is single node. What happens when throughput is not enough at scale? "We will distribute the log." you say. Well, then we are back to square one.
sewen 2y ago
A short summary: Complex distributed coordination and orchestration is at the root of what makes many apps brittle and prone to inconsistencies. But we can mitigate much of complexity with a neat trick, building on the fact that every system (database, queue, state machine) is effectively a log underneath the hood. By implementing interaction with those systems as (conditional) events on a shared log, we can build amazingly robust apps. If you have come across “Turning the Database Inside Out” (https://martin.kleppmann.com/2015/11/05/database-inside-out-at-oredev.html https://martin.kleppmann.com/2015/11/05/database-inside-out-...), you can think of this a bit like “Turning the Microservice Inside Out” The post also looks at how this can be used in practice, given that our DBs and queues aren't built like this, and how to strike a sweet-spot balance between this model with its great consistency, and maintaining healthy decoupling and separation of concerns.
- teddyh 2y ago
  Is this summary AI generated?
  - sewen 2y ago
    Haha, no, but maybe all the AI-generated contents out there is starting to train me to write in a similar style...
    - p10jkle 2y ago
      I feel this happening to me too... depressing
      - TeMPOraL 2y ago
        Or an improvement, given that SOTA models write better than most people...
whoiskatrin 2y ago
whats your take on handling log compaction to prevent unbounded growth, especially in systems with high write throughput?
- sewen 2y ago
  Nice question! Restate is not a log that retains the raw events for a long time - conceptually just until they where processed by the handlers, DB, locking, etc. When you build stateful handlers, the state per key is in the internal DB, and that get's you a similar effect to log compaction, i.e., retain one value per key.
- trollbridge 2y ago
  I have a “summarise” log entry: the current log’s contents that will be relevant to the future are summarised. For example, if it’s FY2023’s financial transactions, we compute the final balances at the end of the year. We then close the log, and write an entry to it of “no more log entries after this are valid”. We then copy the summary transactions to a new log, and compress and archive the old log. You can identify high throughput and low throughput types of log entries and segregate them into different log streams. For example, the “new customer/change customer info” stream probably gets way less traffic than the “customer has logged in” stream. The former is also harder to summarise. Put the hard to summarise but low volume stuff in its own log.
EGreg 2y ago
Since we’re on the subject of logs and embarassingly parallel distributed systems, I know someone who’s also in NYC who’s been building a project exactly along these lines. It’s called gossiplog and it uses Prolly trees to make some interesting results. https://www.npmjs.com/package/@canvas-js/gossiplog https://www.npmjs.com/package/@canvas-js/gossiplog Joel Gustafson started this stuff at MIT and used to work at Protocol Labs. It’s very straightforward. By any chance sewen do you know him? I first became aware of this guy’s work when he posted “merklizing the key value store for fun and profit” or something like that. Afterwards I looked at log protocols, including SLEEP protocol for Dat/Hypercore/ pear and time-travel DBs that track diffs, including including Dolt and even Quadrable. https://news.ycombinator.com/item?id=36265429 https://news.ycombinator.com/item?id=36265429 Gossiplog’s README says exactly what this article says— everything is a log underneath and if you can sync that (using prolly tree techniques) people can just focus on business logic and get sync for free!
- hem777 2y ago
  There’s also OrbitDB https://github.com/orbitdb/orbitdb https://github.com/orbitdb/orbitdb which to my understanding has been a pioneer for p2p logs, databases and CRDTs.
- sewen 2y ago
  Never encountered it before, but it looks cool. I think they are trying to solve a related problem. "We can consolidate the work by making a generic log that has networking and syncing built-in. This can be used by developers to make automatically-decentralized apps without writing a single line of networking code." At a first glance, I would say that Gossiplog is a bit more low level, targeting developers of databases and queues, to save them from re-building a log every time. But then there are elements of sharing the log between components. Worth a deeper look, but seems a bit lower level abstraction.
  - EGreg 2y ago
    It’s part of his higher-level framework called Canvas. Check this out: https://joelgustafson.com/posts/2024-09-30/introduction-to-causal-logs https://joelgustafson.com/posts/2024-09-30/introduction-to-c... And this: https://github.com/canvasxyz/canvas https://github.com/canvasxyz/canvas
davexunit 2y ago
My takeaway from this article is that the proposed solution for distributed app coordination is a shared, centralized log. What did I miss?
- azmy 2y ago
  IMHO the article is not mainly about the implementation of the Log, but rather leveraging on the idea of the log to build reliable and fault tolerant applications. The implementation of the log itself can be either centralised or decentralised.
- sewen 2y ago
  That gist is correct - I would add that the log needs a few specific properties and conceptually be the shared log for state, communication, execution scheduling. The next step is the, how do you make this usable in practice...
trollbridge 2y ago
I’ve been doing a similar thing, although I called it “append only transaction ledgers”. Same idea as a log. A few principles: - The order of log entries does not matter. - Users of the log are peers. No client / server distinction. - When appending a log entry, you can send a copy of the append to all your peers. - You can ask your peers to refresh the latest log entries. - When creating a new entry, it is a very good idea to have a nonce field. (I use nano IDs for this purpose along with a timestamp, which is probabilistically unique.) - If you want to do database style queries of the data, load all the log entries into an in memory database and query away. - You can append a log entry containing a summary of all log entries you have so far. For example: you’ve been given 10 new customer entries. You can create a log entry of “We have 10 customers as of this date.” - When creating new entries, prepare the entry or list of entries in memory, allow the user to edit/revise them as a draft, then when they click “Save”, they are in the permanent record. - To fix a mistake in an entry, create a new entry that “negates” that entry. A lot of parallelism / concurrency problems just go away with this design.
- XorNot 2y ago
  How do you know summary entries are valid if order doesn't matter? I.e. "we have 10 customers as of this date" can become immediately invalid if a new entry is appended afterwards with a date before that summary entry (i.e. because it was on a peer which hadn't yet sent it)
  - clayg 2y ago
    IME you have to be willing to recalculate the summaries up to some kind of consistency window. Yes you may be changing history and you may have a business reason not to address that revision immediately (you've already billed them?) - but the system can still learn it made a mistake and fix it (add activity from Jan 30 evening that comes in late to the Feb bill?)
  - withinboredom 2y ago
    Realistically, you never store summaries in the log. Instead, you store what it took to calculate them. So you won't store "we have 10 customers on this date with this range" but instead store "we found these 10 customers on this date with this range". This assumes you can store infinite sized lists in your log, but realistically, this is never a concern if you can keep your time windows small enough. Then, you periodically do a reconciliation and log corrections (look for entries not summarized -- easily done via a bloom filter which can tell you what entries are definitely NOT in your set) over a longer period. For example, we had a 28-day reconciliation period at one company I worked at (and handled over 120 million events per day). If you appended an event earlier than 28 days prior, it was simply ignored. This very rarely happened, but allowed us to fix bugs with events for up to 28 days.
jaseemabid 2y ago
A notable example of a large-scale app built with a very similar architecture is ATproto/Bluesky[1]. "ATProto for Distributed Systems Engineers" describes how updates from the users end up in their own small databases (called PDS) and then a replicated log. What we traditionally think of as an API server (called a view server in ATProto) is simply one among the many materializations of this log. I personally find this model of thinking about dataflow in large-scale apps pretty neat and easy to understand. The parallels are unsurprising since both the Restate blog and ATProto docs link to the same blog post by Martin Kleppmann. This arch seems to be working really well for Bluesky, as they clearly aced through multiple 10x events very recently. [1]: https://atproto.com/articles/atproto-for-distsys-engineers https://atproto.com/articles/atproto-for-distsys-engineers
- sewen 2y ago
  That blog post is a great read as well. Truely, the log abstraction [1] and "Turning the DB inside out" [2] have been hugely influential. In a way this article here suggests to extend that (1) from a log that represents data (upserts, cdc, etc.) to a log of coordination commands (update this, acquire that log, journal that steo) (2) have a way to link the events related to a broader operation (handler execution) together (3) make the log aware of handler execution (better yet, put it in charge), so you can automatically fence outdated executions [1] https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying https://engineering.linkedin.com/distributed-systems/log-wha...
  - sewen 2y ago
    [2] https://martin.kleppmann.com/2015/11/05/database-inside-out-at-oredev.html https://martin.kleppmann.com/2015/11/05/database-inside-out-...
- grahamj 2y ago
  Table/log duality goes back further than Kleppmann though. An earlier article that really influenced me was https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying https://engineering.linkedin.com/distributed-systems/log-wha...
dboreham 2y ago
This is basically CSP no?
- sewen 2y ago
  I assume CSP is communicating sequential processes? Interesting analogy - in a way it is doing something CSP-like in a distributed app/service architecture with the all the different processes and components that are there. The shared log (or a partition of that) being a way to establish a sequential order.
pjc50 2y ago
> Having a single place (the one log) that forces a linear history of events as the ground truth and owns the decision of who can add to that ground truth, means we don’t have to coordinate much any more. Well, yes, but then you've backed into CAP again because you only have one log.
- clayg 2y ago
  But can't any log be implemented as a CRDT? Was that not implied in the post? I didn't read it that close...
  - ismailmaj 2y ago
    CRDT is really only useful when inconsistencies could be acceptable in some situations and so it depends on the application. For something that is trying to solve the general problem of consistent single ground-truth log, you can't really do much better than Spanner.
  - logsr 2y ago
    the "conflict-free" in crdt is like miller high life being the "champagne of beers." it really means conflicts ignored and conflicting items discarded by established rules, which works well for some use cases, but for many does not.
- mrkeen 2y ago
  If I've understood it, it's like using Kafka with 1 topic and 1 partition. But it shouldn't rule out multiple brokers with a >1 replication factor, giving you CP.
- sewen 2y ago
  Yes, we are assuming a log that picks linearizability at the cost of availability under partitions. Like most logs do, including Kafka, Pulsar, RedPanda, etc. The application state is defined by the log here, and the log drives retries/recovery, so it doesn't much matter if the process that executes the app code splits off. The log would hydrate another one. Also the one log is at the granularity of a single key or handler execution. More of a logical log, than a physical log or even partition. In Restate, we implement a logical log-per-key, backed by a partitioned physical log.
qudat 2y ago
Great post! At pico we've been spending a lot of time thinking about logs and a distributed system that can read and respond to events from logs. This is being driven in part by building out global services and a need for centralized logs for monitoring. The end result is https://pipe.pico.sh https://pipe.pico.sh which is an authenticated, networked *nix pipes over SSH. Since it relies on stdin/stdout via SSH it's one of the easiest pubsub systems we've used and we keep finding its ergonomics powerful. We have a centralized log-drain, metric-drain, and cache-clearing-drain all using `pipe`.
bruce343434 2y ago
> If everything’s in one log, there’s nothing to coordinate # On the contrary. Everything becomes coordinated. The entire "log" becomes a giant ass mutex lock. Good luck scaling it.
- sewen 2y ago
  There is nothing to coordinate for the application, because, yes, the log coordinates everything. But not globally, on the level of a single event handler execution, or a single key. That has been proven to scale well - the way we implement that in Restate is classical shared nothing physical partitioning, with indexing on a key granularity. So nothing like a shared mutex unless you want to access the same key, which otherwise your database synchronizes, if you want any reasonable level of consistency.
- mrkeen 2y ago
  I think the author is motte-and-baileying between: Literally one log - which does indeed reduce your coordination headache, but is susceptible to your "giant ass mutex" comment, and One log per ... - which brings the coordination problems right back into existence.
  - sewen 2y ago
    I can see where some of that could be written more clearly. To elaborate: - We mean using one log across different concerns like state a, communication with b, lock c. Often that is in the scope of a single entity (payment, user, session, etc.) and thus the scope for the one log is still small, and it reduces coordination headache for coordinating between the systems. You would have a lot of independent logs still, for separate payments. - It does _not_ mean that one should share the same log (and partition) for all the entities in your app, like necessarily funneling all users, payments, etc. through the same log. What would be needed if you try and do some multi-key-distributed transaction processing. That goes actually beyond the proposal here, and has some benefits of its own, but have a hard time scaling.
    - mrkeen 2y ago
mrkeen 2y ago
Using one-log-only for an entire system does have its upsides, but it will kill performance. It would be like building a CRUD system with a single mutex for everyone to share.
- hcarvalhoalves 2y ago
  I believe you want "one log" in the logical sense. In theory, you could have "one log" per user or group of users, or whatever sharding technique makes sense for multi tenancy model. It can also be "one log" per bounded context - e.g. the entire payment pipeline in one log.
  - sewen 2y ago
    Yes, exactly right. One log per logical entity, here "payment ID". The way our open source project implements that is with a partitioned log and indexes at key-granularity, so it is like virtually a log per key.
    - ianburrell 2y ago
      How do you deal with one side of transaction getting lost? Traditional double entry accounting puts entry for each side of the transaction. It will cause problems if one side is lost or delayed. How do you handle transactions in general? It is pretty common to have multiple changes that need to be applied together. In a single log, can write the changes and then write a record that says the changes are committed. This is how databases work.
      - hcarvalhoalves 2y ago
        You can simplify things significantly if you can maintain a 1:1 mapping of log entry to transaction (in the business sense). This way you advance the state of your system in discrete steps by advancing in the log, and can make rewind easier too.
  - mrkeen 2y ago
    That's what the rest of us are doing in event-sourcing land, but TFA is arguing for something much stronger: If everything’s in one log, there’s nothing to coordinate The rest of us have to coordinate the logical log of user creation/deletion and the logical log of user payments, etc. Separate logical logs with no need for coordination can only work if they are truly independent systems - no causality between them.
xnorswap 2y ago
It sounds like they have just re-discovered Distributed Transactions with a Distributed Transaction Coordinator. But DTs have a huge problem: What happens if the owner of the lock netsplits? Either the DTC waits (potentially forever?) for the owner of the lock to get back in touch and release the lock, or a timeout is applied and now the owner of the lock (who may be unaware of the netsplit) will be out of sync with the system.
- logsr 2y ago
  they point out that they are adding conditional append to a log, which gives the universality of a log data structure and a mechanism for implementing lock-free/wait-free concurrency algorithms. it is a complete toolkit for building distributed systems because you can build anything else on top of those primitives.
hiAndrewQuinn 2y ago
I am a huge fan of append-only logs as a fundamental architectural principle. The Log [1] should be required reading for any CS undergraduate. [1]: https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying https://engineering.linkedin.com/distributed-systems/log-wha...
- kasey_junk 2y ago
  I love them so much that I’ve noodled with building a programming language optimized for using them. Things like types that encode what events are legal in a log, first class support for data versions, fast file read and writes, etc
  - daxfohl 2y ago
    How do you do GDPR takedowns?
    - jaseemabid 2y ago
      Immutable append only persistent log doesn't imply store everything _forever_. If you want to remove something you could add a tombstone record (like Cassandra) and eventually remove the original entry during routine maintenance operations like repacking into a more efficient format, archival into cold storage, TTL handling etc.
    - UltraSane 2y ago
      Best method is to encrypt data with a per user key so you if you delete the encryption key that customer's data cannot be read. You can create a new customer key at a fixed time interval to make deletion more granular.
daxfohl 2y ago
Haven't formed thoughts on the content yet, but happy to see a company launching something non-AI for a change.
- gjtorikian 2y ago
  My startup, Yetto (http://www.yetto.app http://www.yetto.app) is building a better way for support professionals to do their job. (Shameless plug but we always gotta hustle.) We, too, are weighed down by how much space AI-focused companies are taking.
  - hansonkd 2y ago
    TBH looking at helpdesk software in 2025, I would expect new ones to be built AI first. It would be hard for me to consider one without at least some sort of LLMs helping with triage or at classifications of tickets, etc.
    - gk1 2y ago
      See https://www.fixify.com/ https://www.fixify.com/, although they are too modest about mentioning the AI they're using under the hood.
    - gjtorikian 2y ago
      I never said we didn’t have AI :) Just that the vast majority of apps out there push AI agents, when a human will do just fine. We should be making the harder problems easier, not the easy problems faster. We support parsing your entire help docs into a DAG, so support professionals can have easy access to the most relevant existing docs to answer an end user’s problem. We also provide summarization of incoming tickets so that support teams can understand quickly what the problem is. I’ve toyed with doing sentiment analysis, but our experience with customers in the support space is that they want the AI to give them information and then get out of the way.
sewen 2y ago
Some clarification on what "one log" means here: - It means using one log across different concerns like state a, communication with b, lock c. Often that is in the scope of a single entity (payment, user, session, etc.) and thus the scope for the one log is still small. You would have a lot of independent logs still, for separate payments. - It does _not_ mean that one should share the same log (and partition) for all the entities in your app, like necessarily funneling all users, payments, etc. through the same log. That goes actually beyond the proposal here - has some benefits of its own, but have a hard time scaling.
- magicalhippo 2y ago
  Interesting read, not my area but I think I got the gist of it. In your Restate example of the "processPayment" function, how do you handle errors of the "accountService" call? Like, what if it times out or returns a server error? Do you store the error result and the caller of "processPayment" has to re-trigger the payment, in order to generate a new log?
  - stsffap 2y ago
    By default, failing ctx.run() calls (like the accountService call) will be retried indefinitely until they succeed unless you have configured a retry policy for them. In the case of a configured retry policy where you have exhausted the number of retry attempts, Restate will mark this call as terminally failed and record it in its log as such and return it to the caller.
    - sewen 2y ago
      You can catch these errors and handle them in a common try/catch manner, and because the results of `ctx.run` are recorded in the log, this is deterministic and reliable
shikhar 2y ago
This post makes a great case for how universal logs are in data systems. It was strange to me that there was no log-as-service with the qualities that make it suitable for building higher-level systems like durable execution: conditional appends (as called out by the post!), support very large numbers of logs, allow pushing high throughputs with strict ordering, and just generally provide a simple serverless experience like object storage. This led to https://s2.dev/ https://s2.dev/ which is now available in preview. It was interesting to learn how Restate links events for a key, with key-level logical logs multiplexed over partitioned physical logs. I imagine this is implemented with a leader per physical log, so you can consistently maintain an index. A log service supporting conditional appends allows such a leader to act like the log is local to it, despite offering replicated durability. Leadership can be an important optimization for most systems, but shared logs also allow for multi-writer systems pretty easily. We blogged about this pattern https://s2.dev/blog/kv-store https://s2.dev/blog/kv-store
- thesz 2y ago
  > It was strange to me that there was no log-as-service... Actually, there are plenty of them. Most heard of is Ethereum 2.0 - it is a distributed log of distributed logs. Any blockchain that is built upon PBFT derivative is such a system.
- logsr 2y ago
  > log as a service very exciting. this is the future. i am working on a very similar concept. every database is a log at its core, so the log, which is the highest performance part of the system, is buried behind many layers of much lower performing cruft. edge persistence with log-per-user application patterns opens up so many possibilities.
- gavindean90 2y ago
  What about journalctl?
  - shikhar 2y ago
    This is why we didn't actually call it logs as a service, but streams :P I meant to refer to the log abstraction this post talks about, see links therein. Observability events are but one kind of data you may want as a stream of durable records.
jamamp 2y ago
I wonder how this compares, conceptually, to Temporal? While Temporal doesn't talk about a single centralized log, I feel the output is the same: your event handlers become durable and can be retried without re-executing certain actions with outside systems. Both Restate and Temporal feel, as a developer coding these event handlers, like a framework where they handle a lot of the "has this action been performed yet?" and such for you. Though to be fair I've only read Temporal docs, and this Restate blog post, without much experience in either. Temporal may not have as much on the distributed locking (or concept of) side of things that Restate does, in this post.
- sewen 2y ago
  Temporal is related, but I would say it is a subset of this. If you only consider appending results of steps of a handler, then you have something like Temporal. This here uses the log also for RPC between services, for state that outlives an individual handler execution (state that outlives a workflow, in Temporal's terms).
  - jamamp 2y ago
    That makes a lot of sense, thank you! Extending out to other operations and not just event handlers/workflows would be neat.
erikerikson 2y ago
It's there a hosted offering? (Or plans to offer one?)
- sewen 2y ago
  Yes, there is one, have a look at https://restate.dev/cloud/ https://restate.dev/cloud/
  - erikerikson 2y ago
    Thank you and sorry I missed it.
zellyn 2y ago
sewen (et al) This is lovely and I'm itching to try it. One question: We have a use case where a location gets cut off completely from the internet at large. In that case, it makes sense for the local hardware (typically Android and/or iOS tablets or equivalent) to take over as a log owner: even though you're cut off, if you're willing to swallow the risk (and hence cost) of offline payments, you should be able to create orders, fulfill them, pay for them, close them out, send tickets to the kitchen to cook the food or to the warehouse to fetch the tractor, etc. Does restate include something that covers that use-case? In the noodling/daydreaming a colleague and I have done, we ended up with something very close to restate (I imagined just using Kafka), except that additionally many operations would have a CRDT nature: eg. you should _always_ be allowed to add a payment to an order, because presumably a real-life payment happened. I've also noodled with the idea of logs whose canonical ownership can be transferred. That covers cases where you start offline and then reconnect, but doesn't work so well for transactions that start out connected (and thus owned in the datacenter) and need to continue offline. One could also imagine ensuring that > n/2 consensus members are always located inside the restaurant/hardware store/etc., so if you go offline, you can still proceed. It might even be possible to recognize disconnection and then take one full vote to further subdivide that pool of consensus members so if one dies it doesn't halt progress. This feels like it would be getting very tricksy…
- withinboredom 2y ago
  I'm actually working on a database implementation for this exact use-case... It's a distributed edge database and still quite a long ways to go -- https://github.com/bottledcode/atlas-db https://github.com/bottledcode/atlas-db if you want to give it a star. It's mostly based on wpaxos (wide-area consensus), spaxos, fpaxos and pretty neat. The repo above is a productionization of several proof of concepts to get there. > One could also imagine ensuring that > n/2 consensus members are always located inside the restaurant/hardware store/etc., so if you go offline, you can still proceed. This is what annoys me to no end about RAFT. It's a great protocol, don't get me wrong, but its too simple for these types of problems. RAFT fails when it doesn't have consensus and because the consensus is non-deterministic, it must have an odd number of nodes. PAXOS, while far more complex than RAFT in terms of "grok", is deterministic so you don't need an odd number of nodes. If you throw in some flexible quorums, you can do some really neat stuff, like how Atlas handles a "region" (ie, areas connected via the internet instead of the same network) becoming disconnected; but I'm not ready yet. There's still a long way to go!
amirjak 2y ago
How would you compare this to the actor model or to temporal?
- sewen 2y ago
  Great question: The Virtual Objects in Restate are much like actors. They are somewhat inspired by Orleans [1], and you could call them virtual stateful actors. They blend with the durable execution for processing messages with multiple durable steps. Regarding temporal, check also this question: https://news.ycombinator.com/item?id=42815318 https://news.ycombinator.com/item?id=42815318
TuringTest 2y ago
Excuse me for sounding rough, but - isn't this reinventing comp-sci, one step at a time? I learned about distributed incrementally -monotonic logs back at the late 90s, with many other ways to do guaranteed transactional database actions. And I'm quite certain these must have been invented in the 50s or 60s, as these are the problems that early business computer users had: banking software. These are the techniques that were buried in legacy COBOL routines, and needed to be slowly replaced by robust Java core services. I'm sure the Restate designers will have learned terribly useful insights in how to translate these basic principles into a working system with the complexities of today's hardware/software ecosystem. Yet it makes me wonder if young programmers are only being taught the "build fast-break things" mentality and there are no longer SW engineers able to insert these guarantees into their systems from the beginning, by standing on the shoulders of the ancients that invented our discipline, so that their lore is actually used in practice? Or am I just missing something new in the article that describes some novel twist?
- mrkeen 2y ago
  I think your points are pretty spot on - most things have already been invented, and there's too much of a move-fast-and-break-things mentality. Here's a follow-up thought: to what extent did the grey-beards let us juniors down by steering us down a different path? A few instances: DB creators knew about replicated logs, but we got given DBs, not replicated log products. The Java creators knew about immutability: "I would use an immutable whenever I can." [James Gosling, 1] but it was years later when someone else provided us with pcollections/javaslang/vavr. And they're still far from widespread, and nowhere near the standard library. Brendan Eich supposedly wanted to put Scheme into browsers, but his superiors had him make JS instead. What other stuff have we been missing out on? [1] https://www.artima.com/articles/james-gosling-on-java-may-2001 https://www.artima.com/articles/james-gosling-on-java-may-20...
  - BrendanEich 2y ago
    James (my source was an insider in the Java team at Sun, pre-Marimba) wrote java.util.Date, which I had my one assistant (Ken Smith of Netscape) translate from Java to C for JS's first Date object, regrets all around but it "made it look like Java". I wish James had been in favor of immutability in designing java.util.Date!
Thaxll 2y ago
This is exactly this example from Temporal: https://github.com/temporal-sa/temporal-order-fulfill-demo https://github.com/temporal-sa/temporal-order-fulfill-demo
paulsutter 2y ago
This is very compelling, nice work. I'm going to spend some quality time on this.
Animats 2y ago
This is a basic concept in accounting. The general ledger is an immutable log of transactions. Other accounting documents are constructed from the general ledger, and can, if necessary, be rebuilt from it. This is the accepted way to do money-related things. Synchronization is called "reconcilation" in accounting terminology. The computer concept is that we have a current state, and changes to it come in. The database with the current state is authoritative. This is not suitable for handling money. The real question is, do you really care what happened last month? Last year? If yes, a log-based approach is appropriate.
- LeanOnSheena 2y ago
  You're correct on all points. Some additional refining points regarding accounting concepts: - General legers are formed by way of transactions recorded as journal entries. Journal entries are where two or more accounts from the general ledger are debited & credited such that total debits equals total credits. For example, a sale will involve a journal entry which debits cash or accounts receivable, and credits revenue. - The concept of the debits always needing to equal credits is the most important and fundamental control in accounting. It's is the core idea around which all of double entry bookkeeping is built. - temporally ordered Journal entries are what form a log from which a general ledger can be derived. That log of journal entries is append-only and immutable. If you make an mistake with a journal entry, you typically don't delete it, you just make another adjusting (i.e. correcting) entry. Having a traditional background in accounting as a CPA, as a programmer I have written systems that are built around a log of temporally ordered transactions that can be used to construct state across time. To my colleagues that didn't have that background they found it interesting but very strange as an idea (led to a lot of really interesting discussions!). It was totally strange to me that they found it odd because it was the most comfortable & natural way for me to think about many problems.
- inopinatus 2y ago
  I’ve always concurred with the Helland/Kleppman observation mentioned viz. that the transaction log of a typical RDBMS is the canonical form and all the rows & tables merely projections. It’s curious that over those projections, we then build event stores for CQRS/ES systems, ledgers etc, with their own projections mediated by application code. But look underneath too. The journaled filesystem on which the database resides also has a log representation, and under that, a modern SSD is using an adaptive log structure to balance block writes. It’s been a long time since we wrote an application event stream linearly straight to media, and although I appreciate the separate concerns that each of these layers addresses, I’d probably struggle to justify them all from first principles to even a slightly more Socratic version of myself.
random3 2y ago
looks like a new generation is ready to discover Paxos, Zab, Raft
yftsui 2y ago
The diagrams are remarkably neat, has a feeling of both excalidraw and draw.io, anyone know what tool was used create those?
- wlonkly 2y ago
  Not sure what made the diagrams, but I think you've correctly identified the Excalidraw handwriting font. There's two versions, Virgil[1] and Excalifont[2], both under the MIT license. I think the monospaced font is Comic Mono.[3] [1] https://plus.excalidraw.com/virgil https://plus.excalidraw.com/virgil [2] https://plus.excalidraw.com/excalifont https://plus.excalidraw.com/excalifont [3] https://dtinth.github.io/comic-mono-font/ https://dtinth.github.io/comic-mono-font/
- gvdongen 2y ago
  We've used purely excalidraw. Nice to hear you like them!
baq 2y ago
Or, every database is bitemporal, some just don’t know it yet
ris 2y ago
> Restate is open source and you can download it at... https://github.com/restatedev/restate/blob/main/LICENSE#L1 https://github.com/restatedev/restate/blob/main/LICENSE#L1 > Business Source License 1.1 https://spdx.org/licenses/BUSL-1.1.html https://spdx.org/licenses/BUSL-1.1.html > The Business Source License (this document, or the “License”) is not an Open Source license. Suggest exploring e.g. https://github.com/dbos-inc/dbos-transact-py https://github.com/dbos-inc/dbos-transact-py
- drewbug01 2y ago
  I definitely don't consider the BSL to be a true "open source" license. However, I think that if the Additional Use Grant is written clearly and unambiguously, it may well be a reasonable compromise. That said - the Additional Use Grant for Restate is fairly ambiguous: > Additional Use Grant: You may make use of the Licensed Work, provided that you may not use the Licensed Work for an Application Platform Service. > An “Application Platform Service” is a commercial offering that allows third parties (other than your employees and contractors) to access the functionality of the Licensed Work by registering new services (directly or indirectly) with the Licensed Work for the purpose of invoking such services through the Licensed Work. What does it mean to "access the functionality"? What does it mean to "register [a] new service" ? How do you determine the "purpose" of invoking such services? I understand that the intent is to prevent someone from launching a competing "Restate-as-a-Service" business; but broadly written clauses like this scare me off. If I want to build a totally different kind of business that heavily _relies_ on Restate, I would worry that these ambiguous, broad definitions could easily be used against me. I really wish that companies adopting the BSL would put much more clarity into the license text. I know there's likely very good business reasons not to - you may constrain yourself in the future by doing so - but in my mind it would be the "right" thing to do. You'd get people actually using and contributing back to your software, much like they would under a more permissive license.

229 points by sewen 2y ago

Isn't this just trading consistency for availability? From what I understand the single log is single node. What happens when throughput is not enough at scale? "We will distribute the log." you say. Well, then we are back to square one.

sewen 2y ago

A short summary: Complex distributed coordination and orchestration is at the root of what makes many apps brittle and prone to inconsistencies. But we can mitigate much of complexity with a neat trick, building on the fact that every system (database, queue, state machine) is effectively a log underneath the hood. By implementing interaction with those systems as (conditional) events on a shared log, we can build amazingly robust apps. If you have come across “Turning the Database Inside Out” (https://martin.kleppmann.com/2015/11/05/database-inside-out-at-oredev.html https://martin.kleppmann.com/2015/11/05/database-inside-out-...), you can think of this a bit like “Turning the Microservice Inside Out” The post also looks at how this can be used in practice, given that our DBs and queues aren't built like this, and how to strike a sweet-spot balance between this model with its great consistency, and maintaining healthy decoupling and separation of concerns.

teddyh 2y ago
Is this summary AI generated?
- sewen 2y ago
  Haha, no, but maybe all the AI-generated contents out there is starting to train me to write in a similar style...
  - p10jkle 2y ago
    I feel this happening to me too... depressing
    - TeMPOraL 2y ago
      Or an improvement, given that SOTA models write better than most people...

whoiskatrin 2y ago

whats your take on handling log compaction to prevent unbounded growth, especially in systems with high write throughput?

sewen 2y ago
Nice question! Restate is not a log that retains the raw events for a long time - conceptually just until they where processed by the handlers, DB, locking, etc. When you build stateful handlers, the state per key is in the internal DB, and that get's you a similar effect to log compaction, i.e., retain one value per key.
trollbridge 2y ago
I have a “summarise” log entry: the current log’s contents that will be relevant to the future are summarised. For example, if it’s FY2023’s financial transactions, we compute the final balances at the end of the year. We then close the log, and write an entry to it of “no more log entries after this are valid”. We then copy the summary transactions to a new log, and compress and archive the old log. You can identify high throughput and low throughput types of log entries and segregate them into different log streams. For example, the “new customer/change customer info” stream probably gets way less traffic than the “customer has logged in” stream. The former is also harder to summarise. Put the hard to summarise but low volume stuff in its own log.

EGreg 2y ago

Since we’re on the subject of logs and embarassingly parallel distributed systems, I know someone who’s also in NYC who’s been building a project exactly along these lines. It’s called gossiplog and it uses Prolly trees to make some interesting results. https://www.npmjs.com/package/@canvas-js/gossiplog https://www.npmjs.com/package/@canvas-js/gossiplog Joel Gustafson started this stuff at MIT and used to work at Protocol Labs. It’s very straightforward. By any chance sewen do you know him? I first became aware of this guy’s work when he posted “merklizing the key value store for fun and profit” or something like that. Afterwards I looked at log protocols, including SLEEP protocol for Dat/Hypercore/ pear and time-travel DBs that track diffs, including including Dolt and even Quadrable. https://news.ycombinator.com/item?id=36265429 https://news.ycombinator.com/item?id=36265429 Gossiplog’s README says exactly what this article says— everything is a log underneath and if you can sync that (using prolly tree techniques) people can just focus on business logic and get sync for free!

hem777 2y ago
There’s also OrbitDB https://github.com/orbitdb/orbitdb https://github.com/orbitdb/orbitdb which to my understanding has been a pioneer for p2p logs, databases and CRDTs.
sewen 2y ago
Never encountered it before, but it looks cool. I think they are trying to solve a related problem. "We can consolidate the work by making a generic log that has networking and syncing built-in. This can be used by developers to make automatically-decentralized apps without writing a single line of networking code." At a first glance, I would say that Gossiplog is a bit more low level, targeting developers of databases and queues, to save them from re-building a log every time. But then there are elements of sharing the log between components. Worth a deeper look, but seems a bit lower level abstraction.
- EGreg 2y ago
  It’s part of his higher-level framework called Canvas. Check this out: https://joelgustafson.com/posts/2024-09-30/introduction-to-causal-logs https://joelgustafson.com/posts/2024-09-30/introduction-to-c... And this: https://github.com/canvasxyz/canvas https://github.com/canvasxyz/canvas

davexunit 2y ago

My takeaway from this article is that the proposed solution for distributed app coordination is a shared, centralized log. What did I miss?

azmy 2y ago
IMHO the article is not mainly about the implementation of the Log, but rather leveraging on the idea of the log to build reliable and fault tolerant applications. The implementation of the log itself can be either centralised or decentralised.
sewen 2y ago
That gist is correct - I would add that the log needs a few specific properties and conceptually be the shared log for state, communication, execution scheduling. The next step is the, how do you make this usable in practice...

trollbridge 2y ago

I’ve been doing a similar thing, although I called it “append only transaction ledgers”. Same idea as a log. A few principles: - The order of log entries does not matter. - Users of the log are peers. No client / server distinction. - When appending a log entry, you can send a copy of the append to all your peers. - You can ask your peers to refresh the latest log entries. - When creating a new entry, it is a very good idea to have a nonce field. (I use nano IDs for this purpose along with a timestamp, which is probabilistically unique.) - If you want to do database style queries of the data, load all the log entries into an in memory database and query away. - You can append a log entry containing a summary of all log entries you have so far. For example: you’ve been given 10 new customer entries. You can create a log entry of “We have 10 customers as of this date.” - When creating new entries, prepare the entry or list of entries in memory, allow the user to edit/revise them as a draft, then when they click “Save”, they are in the permanent record. - To fix a mistake in an entry, create a new entry that “negates” that entry. A lot of parallelism / concurrency problems just go away with this design.

XorNot 2y ago
How do you know summary entries are valid if order doesn't matter? I.e. "we have 10 customers as of this date" can become immediately invalid if a new entry is appended afterwards with a date before that summary entry (i.e. because it was on a peer which hadn't yet sent it)
- clayg 2y ago
  IME you have to be willing to recalculate the summaries up to some kind of consistency window. Yes you may be changing history and you may have a business reason not to address that revision immediately (you've already billed them?) - but the system can still learn it made a mistake and fix it (add activity from Jan 30 evening that comes in late to the Feb bill?)
- withinboredom 2y ago
  Realistically, you never store summaries in the log. Instead, you store what it took to calculate them. So you won't store "we have 10 customers on this date with this range" but instead store "we found these 10 customers on this date with this range". This assumes you can store infinite sized lists in your log, but realistically, this is never a concern if you can keep your time windows small enough. Then, you periodically do a reconciliation and log corrections (look for entries not summarized -- easily done via a bloom filter which can tell you what entries are definitely NOT in your set) over a longer period. For example, we had a 28-day reconciliation period at one company I worked at (and handled over 120 million events per day). If you appended an event earlier than 28 days prior, it was simply ignored. This very rarely happened, but allowed us to fix bugs with events for up to 28 days.

jaseemabid 2y ago

A notable example of a large-scale app built with a very similar architecture is ATproto/Bluesky[1]. "ATProto for Distributed Systems Engineers" describes how updates from the users end up in their own small databases (called PDS) and then a replicated log. What we traditionally think of as an API server (called a view server in ATProto) is simply one among the many materializations of this log. I personally find this model of thinking about dataflow in large-scale apps pretty neat and easy to understand. The parallels are unsurprising since both the Restate blog and ATProto docs link to the same blog post by Martin Kleppmann. This arch seems to be working really well for Bluesky, as they clearly aced through multiple 10x events very recently. [1]: https://atproto.com/articles/atproto-for-distsys-engineers https://atproto.com/articles/atproto-for-distsys-engineers

sewen 2y ago
That blog post is a great read as well. Truely, the log abstraction [1] and "Turning the DB inside out" [2] have been hugely influential. In a way this article here suggests to extend that (1) from a log that represents data (upserts, cdc, etc.) to a log of coordination commands (update this, acquire that log, journal that steo) (2) have a way to link the events related to a broader operation (handler execution) together (3) make the log aware of handler execution (better yet, put it in charge), so you can automatically fence outdated executions [1] https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying https://engineering.linkedin.com/distributed-systems/log-wha...
- sewen 2y ago
  [2] https://martin.kleppmann.com/2015/11/05/database-inside-out-at-oredev.html https://martin.kleppmann.com/2015/11/05/database-inside-out-...
grahamj 2y ago
Table/log duality goes back further than Kleppmann though. An earlier article that really influenced me was https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying https://engineering.linkedin.com/distributed-systems/log-wha...

dboreham 2y ago

This is basically CSP no?

sewen 2y ago
I assume CSP is communicating sequential processes? Interesting analogy - in a way it is doing something CSP-like in a distributed app/service architecture with the all the different processes and components that are there. The shared log (or a partition of that) being a way to establish a sequential order.

pjc50 2y ago

> Having a single place (the one log) that forces a linear history of events as the ground truth and owns the decision of who can add to that ground truth, means we don’t have to coordinate much any more. Well, yes, but then you've backed into CAP again because you only have one log.

clayg 2y ago
But can't any log be implemented as a CRDT? Was that not implied in the post? I didn't read it that close...
- ismailmaj 2y ago
  CRDT is really only useful when inconsistencies could be acceptable in some situations and so it depends on the application. For something that is trying to solve the general problem of consistent single ground-truth log, you can't really do much better than Spanner.
- logsr 2y ago
  the "conflict-free" in crdt is like miller high life being the "champagne of beers." it really means conflicts ignored and conflicting items discarded by established rules, which works well for some use cases, but for many does not.
mrkeen 2y ago
If I've understood it, it's like using Kafka with 1 topic and 1 partition. But it shouldn't rule out multiple brokers with a >1 replication factor, giving you CP.
sewen 2y ago
Yes, we are assuming a log that picks linearizability at the cost of availability under partitions. Like most logs do, including Kafka, Pulsar, RedPanda, etc. The application state is defined by the log here, and the log drives retries/recovery, so it doesn't much matter if the process that executes the app code splits off. The log would hydrate another one. Also the one log is at the granularity of a single key or handler execution. More of a logical log, than a physical log or even partition. In Restate, we implement a logical log-per-key, backed by a partitioned physical log.

qudat 2y ago

Great post! At pico we've been spending a lot of time thinking about logs and a distributed system that can read and respond to events from logs. This is being driven in part by building out global services and a need for centralized logs for monitoring. The end result is https://pipe.pico.sh https://pipe.pico.sh which is an authenticated, networked *nix pipes over SSH. Since it relies on stdin/stdout via SSH it's one of the easiest pubsub systems we've used and we keep finding its ergonomics powerful. We have a centralized log-drain, metric-drain, and cache-clearing-drain all using `pipe`.

bruce343434 2y ago

> If everything’s in one log, there’s nothing to coordinate # On the contrary. Everything becomes coordinated. The entire "log" becomes a giant ass mutex lock. Good luck scaling it.

sewen 2y ago
There is nothing to coordinate for the application, because, yes, the log coordinates everything. But not globally, on the level of a single event handler execution, or a single key. That has been proven to scale well - the way we implement that in Restate is classical shared nothing physical partitioning, with indexing on a key granularity. So nothing like a shared mutex unless you want to access the same key, which otherwise your database synchronizes, if you want any reasonable level of consistency.
mrkeen 2y ago
I think the author is motte-and-baileying between: Literally one log - which does indeed reduce your coordination headache, but is susceptible to your "giant ass mutex" comment, and One log per ... - which brings the coordination problems right back into existence.
- sewen 2y ago
  I can see where some of that could be written more clearly. To elaborate: - We mean using one log across different concerns like state a, communication with b, lock c. Often that is in the scope of a single entity (payment, user, session, etc.) and thus the scope for the one log is still small, and it reduces coordination headache for coordinating between the systems. You would have a lot of independent logs still, for separate payments. - It does _not_ mean that one should share the same log (and partition) for all the entities in your app, like necessarily funneling all users, payments, etc. through the same log. What would be needed if you try and do some multi-key-distributed transaction processing. That goes actually beyond the proposal here, and has some benefits of its own, but have a hard time scaling.
  - mrkeen 2y ago

mrkeen 2y ago

Using one-log-only for an entire system does have its upsides, but it will kill performance. It would be like building a CRUD system with a single mutex for everyone to share.

hcarvalhoalves 2y ago
I believe you want "one log" in the logical sense. In theory, you could have "one log" per user or group of users, or whatever sharding technique makes sense for multi tenancy model. It can also be "one log" per bounded context - e.g. the entire payment pipeline in one log.
- sewen 2y ago
  Yes, exactly right. One log per logical entity, here "payment ID". The way our open source project implements that is with a partitioned log and indexes at key-granularity, so it is like virtually a log per key.
  - ianburrell 2y ago
    How do you deal with one side of transaction getting lost? Traditional double entry accounting puts entry for each side of the transaction. It will cause problems if one side is lost or delayed. How do you handle transactions in general? It is pretty common to have multiple changes that need to be applied together. In a single log, can write the changes and then write a record that says the changes are committed. This is how databases work.
    - hcarvalhoalves 2y ago
      You can simplify things significantly if you can maintain a 1:1 mapping of log entry to transaction (in the business sense). This way you advance the state of your system in discrete steps by advancing in the log, and can make rewind easier too.
- mrkeen 2y ago
  That's what the rest of us are doing in event-sourcing land, but TFA is arguing for something much stronger: If everything’s in one log, there’s nothing to coordinate The rest of us have to coordinate the logical log of user creation/deletion and the logical log of user payments, etc. Separate logical logs with no need for coordination can only work if they are truly independent systems - no causality between them.

xnorswap 2y ago

It sounds like they have just re-discovered Distributed Transactions with a Distributed Transaction Coordinator. But DTs have a huge problem: What happens if the owner of the lock netsplits? Either the DTC waits (potentially forever?) for the owner of the lock to get back in touch and release the lock, or a timeout is applied and now the owner of the lock (who may be unaware of the netsplit) will be out of sync with the system.

logsr 2y ago
they point out that they are adding conditional append to a log, which gives the universality of a log data structure and a mechanism for implementing lock-free/wait-free concurrency algorithms. it is a complete toolkit for building distributed systems because you can build anything else on top of those primitives.

hiAndrewQuinn 2y ago

I am a huge fan of append-only logs as a fundamental architectural principle. The Log [1] should be required reading for any CS undergraduate. [1]: https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying https://engineering.linkedin.com/distributed-systems/log-wha...

kasey_junk 2y ago
I love them so much that I’ve noodled with building a programming language optimized for using them. Things like types that encode what events are legal in a log, first class support for data versions, fast file read and writes, etc
- daxfohl 2y ago
  How do you do GDPR takedowns?
  - jaseemabid 2y ago
    Immutable append only persistent log doesn't imply store everything _forever_. If you want to remove something you could add a tombstone record (like Cassandra) and eventually remove the original entry during routine maintenance operations like repacking into a more efficient format, archival into cold storage, TTL handling etc.
  - UltraSane 2y ago
    Best method is to encrypt data with a per user key so you if you delete the encryption key that customer's data cannot be read. You can create a new customer key at a fixed time interval to make deletion more granular.

daxfohl 2y ago

Haven't formed thoughts on the content yet, but happy to see a company launching something non-AI for a change.

gjtorikian 2y ago
My startup, Yetto (http://www.yetto.app http://www.yetto.app) is building a better way for support professionals to do their job. (Shameless plug but we always gotta hustle.) We, too, are weighed down by how much space AI-focused companies are taking.
- hansonkd 2y ago
  TBH looking at helpdesk software in 2025, I would expect new ones to be built AI first. It would be hard for me to consider one without at least some sort of LLMs helping with triage or at classifications of tickets, etc.
  - gk1 2y ago
    See https://www.fixify.com/ https://www.fixify.com/, although they are too modest about mentioning the AI they're using under the hood.
  - gjtorikian 2y ago
    I never said we didn’t have AI :) Just that the vast majority of apps out there push AI agents, when a human will do just fine. We should be making the harder problems easier, not the easy problems faster. We support parsing your entire help docs into a DAG, so support professionals can have easy access to the most relevant existing docs to answer an end user’s problem. We also provide summarization of incoming tickets so that support teams can understand quickly what the problem is. I’ve toyed with doing sentiment analysis, but our experience with customers in the support space is that they want the AI to give them information and then get out of the way.

sewen 2y ago

Some clarification on what "one log" means here: - It means using one log across different concerns like state a, communication with b, lock c. Often that is in the scope of a single entity (payment, user, session, etc.) and thus the scope for the one log is still small. You would have a lot of independent logs still, for separate payments. - It does _not_ mean that one should share the same log (and partition) for all the entities in your app, like necessarily funneling all users, payments, etc. through the same log. That goes actually beyond the proposal here - has some benefits of its own, but have a hard time scaling.

magicalhippo 2y ago
Interesting read, not my area but I think I got the gist of it. In your Restate example of the "processPayment" function, how do you handle errors of the "accountService" call? Like, what if it times out or returns a server error? Do you store the error result and the caller of "processPayment" has to re-trigger the payment, in order to generate a new log?
- stsffap 2y ago
  By default, failing ctx.run() calls (like the accountService call) will be retried indefinitely until they succeed unless you have configured a retry policy for them. In the case of a configured retry policy where you have exhausted the number of retry attempts, Restate will mark this call as terminally failed and record it in its log as such and return it to the caller.
  - sewen 2y ago
    You can catch these errors and handle them in a common try/catch manner, and because the results of `ctx.run` are recorded in the log, this is deterministic and reliable

shikhar 2y ago

This post makes a great case for how universal logs are in data systems. It was strange to me that there was no log-as-service with the qualities that make it suitable for building higher-level systems like durable execution: conditional appends (as called out by the post!), support very large numbers of logs, allow pushing high throughputs with strict ordering, and just generally provide a simple serverless experience like object storage. This led to https://s2.dev/ https://s2.dev/ which is now available in preview. It was interesting to learn how Restate links events for a key, with key-level logical logs multiplexed over partitioned physical logs. I imagine this is implemented with a leader per physical log, so you can consistently maintain an index. A log service supporting conditional appends allows such a leader to act like the log is local to it, despite offering replicated durability. Leadership can be an important optimization for most systems, but shared logs also allow for multi-writer systems pretty easily. We blogged about this pattern https://s2.dev/blog/kv-store https://s2.dev/blog/kv-store

thesz 2y ago
> It was strange to me that there was no log-as-service... Actually, there are plenty of them. Most heard of is Ethereum 2.0 - it is a distributed log of distributed logs. Any blockchain that is built upon PBFT derivative is such a system.
logsr 2y ago
> log as a service very exciting. this is the future. i am working on a very similar concept. every database is a log at its core, so the log, which is the highest performance part of the system, is buried behind many layers of much lower performing cruft. edge persistence with log-per-user application patterns opens up so many possibilities.
gavindean90 2y ago
What about journalctl?
- shikhar 2y ago
  This is why we didn't actually call it logs as a service, but streams :P I meant to refer to the log abstraction this post talks about, see links therein. Observability events are but one kind of data you may want as a stream of durable records.

jamamp 2y ago

I wonder how this compares, conceptually, to Temporal? While Temporal doesn't talk about a single centralized log, I feel the output is the same: your event handlers become durable and can be retried without re-executing certain actions with outside systems. Both Restate and Temporal feel, as a developer coding these event handlers, like a framework where they handle a lot of the "has this action been performed yet?" and such for you. Though to be fair I've only read Temporal docs, and this Restate blog post, without much experience in either. Temporal may not have as much on the distributed locking (or concept of) side of things that Restate does, in this post.

sewen 2y ago
Temporal is related, but I would say it is a subset of this. If you only consider appending results of steps of a handler, then you have something like Temporal. This here uses the log also for RPC between services, for state that outlives an individual handler execution (state that outlives a workflow, in Temporal's terms).
- jamamp 2y ago
  That makes a lot of sense, thank you! Extending out to other operations and not just event handlers/workflows would be neat.

erikerikson 2y ago

It's there a hosted offering? (Or plans to offer one?)

sewen 2y ago
Yes, there is one, have a look at https://restate.dev/cloud/ https://restate.dev/cloud/
- erikerikson 2y ago
  Thank you and sorry I missed it.

zellyn 2y ago

sewen (et al) This is lovely and I'm itching to try it. One question: We have a use case where a location gets cut off completely from the internet at large. In that case, it makes sense for the local hardware (typically Android and/or iOS tablets or equivalent) to take over as a log owner: even though you're cut off, if you're willing to swallow the risk (and hence cost) of offline payments, you should be able to create orders, fulfill them, pay for them, close them out, send tickets to the kitchen to cook the food or to the warehouse to fetch the tractor, etc. Does restate include something that covers that use-case? In the noodling/daydreaming a colleague and I have done, we ended up with something very close to restate (I imagined just using Kafka), except that additionally many operations would have a CRDT nature: eg. you should _always_ be allowed to add a payment to an order, because presumably a real-life payment happened. I've also noodled with the idea of logs whose canonical ownership can be transferred. That covers cases where you start offline and then reconnect, but doesn't work so well for transactions that start out connected (and thus owned in the datacenter) and need to continue offline. One could also imagine ensuring that > n/2 consensus members are always located inside the restaurant/hardware store/etc., so if you go offline, you can still proceed. It might even be possible to recognize disconnection and then take one full vote to further subdivide that pool of consensus members so if one dies it doesn't halt progress. This feels like it would be getting very tricksy…

withinboredom 2y ago
I'm actually working on a database implementation for this exact use-case... It's a distributed edge database and still quite a long ways to go -- https://github.com/bottledcode/atlas-db https://github.com/bottledcode/atlas-db if you want to give it a star. It's mostly based on wpaxos (wide-area consensus), spaxos, fpaxos and pretty neat. The repo above is a productionization of several proof of concepts to get there. > One could also imagine ensuring that > n/2 consensus members are always located inside the restaurant/hardware store/etc., so if you go offline, you can still proceed. This is what annoys me to no end about RAFT. It's a great protocol, don't get me wrong, but its too simple for these types of problems. RAFT fails when it doesn't have consensus and because the consensus is non-deterministic, it must have an odd number of nodes. PAXOS, while far more complex than RAFT in terms of "grok", is deterministic so you don't need an odd number of nodes. If you throw in some flexible quorums, you can do some really neat stuff, like how Atlas handles a "region" (ie, areas connected via the internet instead of the same network) becoming disconnected; but I'm not ready yet. There's still a long way to go!

amirjak 2y ago

How would you compare this to the actor model or to temporal?

sewen 2y ago
Great question: The Virtual Objects in Restate are much like actors. They are somewhat inspired by Orleans [1], and you could call them virtual stateful actors. They blend with the durable execution for processing messages with multiple durable steps. Regarding temporal, check also this question: https://news.ycombinator.com/item?id=42815318 https://news.ycombinator.com/item?id=42815318

TuringTest 2y ago

Excuse me for sounding rough, but - isn't this reinventing comp-sci, one step at a time? I learned about distributed incrementally -monotonic logs back at the late 90s, with many other ways to do guaranteed transactional database actions. And I'm quite certain these must have been invented in the 50s or 60s, as these are the problems that early business computer users had: banking software. These are the techniques that were buried in legacy COBOL routines, and needed to be slowly replaced by robust Java core services. I'm sure the Restate designers will have learned terribly useful insights in how to translate these basic principles into a working system with the complexities of today's hardware/software ecosystem. Yet it makes me wonder if young programmers are only being taught the "build fast-break things" mentality and there are no longer SW engineers able to insert these guarantees into their systems from the beginning, by standing on the shoulders of the ancients that invented our discipline, so that their lore is actually used in practice? Or am I just missing something new in the article that describes some novel twist?

mrkeen 2y ago
I think your points are pretty spot on - most things have already been invented, and there's too much of a move-fast-and-break-things mentality. Here's a follow-up thought: to what extent did the grey-beards let us juniors down by steering us down a different path? A few instances: DB creators knew about replicated logs, but we got given DBs, not replicated log products. The Java creators knew about immutability: "I would use an immutable whenever I can." [James Gosling, 1] but it was years later when someone else provided us with pcollections/javaslang/vavr. And they're still far from widespread, and nowhere near the standard library. Brendan Eich supposedly wanted to put Scheme into browsers, but his superiors had him make JS instead. What other stuff have we been missing out on? [1] https://www.artima.com/articles/james-gosling-on-java-may-2001 https://www.artima.com/articles/james-gosling-on-java-may-20...
- BrendanEich 2y ago
  James (my source was an insider in the Java team at Sun, pre-Marimba) wrote java.util.Date, which I had my one assistant (Ken Smith of Netscape) translate from Java to C for JS's first Date object, regrets all around but it "made it look like Java". I wish James had been in favor of immutability in designing java.util.Date!

Thaxll 2y ago

This is exactly this example from Temporal: https://github.com/temporal-sa/temporal-order-fulfill-demo https://github.com/temporal-sa/temporal-order-fulfill-demo

paulsutter 2y ago

This is very compelling, nice work. I'm going to spend some quality time on this.

Animats 2y ago

This is a basic concept in accounting. The general ledger is an immutable log of transactions. Other accounting documents are constructed from the general ledger, and can, if necessary, be rebuilt from it. This is the accepted way to do money-related things. Synchronization is called "reconcilation" in accounting terminology. The computer concept is that we have a current state, and changes to it come in. The database with the current state is authoritative. This is not suitable for handling money. The real question is, do you really care what happened last month? Last year? If yes, a log-based approach is appropriate.

LeanOnSheena 2y ago
You're correct on all points. Some additional refining points regarding accounting concepts: - General legers are formed by way of transactions recorded as journal entries. Journal entries are where two or more accounts from the general ledger are debited & credited such that total debits equals total credits. For example, a sale will involve a journal entry which debits cash or accounts receivable, and credits revenue. - The concept of the debits always needing to equal credits is the most important and fundamental control in accounting. It's is the core idea around which all of double entry bookkeeping is built. - temporally ordered Journal entries are what form a log from which a general ledger can be derived. That log of journal entries is append-only and immutable. If you make an mistake with a journal entry, you typically don't delete it, you just make another adjusting (i.e. correcting) entry. Having a traditional background in accounting as a CPA, as a programmer I have written systems that are built around a log of temporally ordered transactions that can be used to construct state across time. To my colleagues that didn't have that background they found it interesting but very strange as an idea (led to a lot of really interesting discussions!). It was totally strange to me that they found it odd because it was the most comfortable & natural way for me to think about many problems.
inopinatus 2y ago
I’ve always concurred with the Helland/Kleppman observation mentioned viz. that the transaction log of a typical RDBMS is the canonical form and all the rows & tables merely projections. It’s curious that over those projections, we then build event stores for CQRS/ES systems, ledgers etc, with their own projections mediated by application code. But look underneath too. The journaled filesystem on which the database resides also has a log representation, and under that, a modern SSD is using an adaptive log structure to balance block writes. It’s been a long time since we wrote an application event stream linearly straight to media, and although I appreciate the separate concerns that each of these layers addresses, I’d probably struggle to justify them all from first principles to even a slightly more Socratic version of myself.

random3 2y ago

looks like a new generation is ready to discover Paxos, Zab, Raft

yftsui 2y ago

The diagrams are remarkably neat, has a feeling of both excalidraw and draw.io, anyone know what tool was used create those?

wlonkly 2y ago
Not sure what made the diagrams, but I think you've correctly identified the Excalidraw handwriting font. There's two versions, Virgil[1] and Excalifont[2], both under the MIT license. I think the monospaced font is Comic Mono.[3] [1] https://plus.excalidraw.com/virgil https://plus.excalidraw.com/virgil [2] https://plus.excalidraw.com/excalifont https://plus.excalidraw.com/excalifont [3] https://dtinth.github.io/comic-mono-font/ https://dtinth.github.io/comic-mono-font/
gvdongen 2y ago
We've used purely excalidraw. Nice to hear you like them!

baq 2y ago

Or, every database is bitemporal, some just don’t know it yet

ris 2y ago

> Restate is open source and you can download it at... https://github.com/restatedev/restate/blob/main/LICENSE#L1 https://github.com/restatedev/restate/blob/main/LICENSE#L1 > Business Source License 1.1 https://spdx.org/licenses/BUSL-1.1.html https://spdx.org/licenses/BUSL-1.1.html > The Business Source License (this document, or the “License”) is not an Open Source license. Suggest exploring e.g. https://github.com/dbos-inc/dbos-transact-py https://github.com/dbos-inc/dbos-transact-py

drewbug01 2y ago
I definitely don't consider the BSL to be a true "open source" license. However, I think that if the Additional Use Grant is written clearly and unambiguously, it may well be a reasonable compromise. That said - the Additional Use Grant for Restate is fairly ambiguous: > Additional Use Grant: You may make use of the Licensed Work, provided that you may not use the Licensed Work for an Application Platform Service. > An “Application Platform Service” is a commercial offering that allows third parties (other than your employees and contractors) to access the functionality of the Licensed Work by registering new services (directly or indirectly) with the Licensed Work for the purpose of invoking such services through the Licensed Work. What does it mean to "access the functionality"? What does it mean to "register [a] new service" ? How do you determine the "purpose" of invoking such services? I understand that the intent is to prevent someone from launching a competing "Restate-as-a-Service" business; but broadly written clauses like this scare me off. If I want to build a totally different kind of business that heavily _relies_ on Restate, I would worry that these ambiguous, broad definitions could easily be used against me. I really wish that companies adopting the BSL would put much more clarity into the license text. I know there's likely very good business reasons not to - you may constrain yourself in the future by doing so - but in my mind it would be the "right" thing to do. You'd get people actually using and contributing back to your software, much like they would under a more permissive license.