11 ms·

SQLite is all you need for durable workflows

720 points by tomasol 4mo ago

EGreg 4mo ago
Files is all you need. https://xkcd.com/378/ https://xkcd.com/378/
- tclancy 4mo ago
  Post It Notes will do if you have a good system.
- contingencies 4mo ago
  Those who don't understand Unix are condemned to reinvent it, poorly. - Henry Spencer .. via https://github.com/globalcitizen/taoup https://github.com/globalcitizen/taoup
- jusonchan81 4mo ago
  Agree on this and it’s the architecture that backs our orchestration platform called https://unmeshed.io https://unmeshed.io
orf 4mo ago
> The caveat is that Litestream replication is asynchronous. A restore can miss the newest local writes if the SQLite volume disappears before they are copied. That is fine for many AI and experimentation workflows In short: SQLite is not all you need, unless you’re just experimenting don’t actually care about durability, in which case you also need litestream + object storage. Right.
- bootsmann 4mo ago
  S3 is strongly consistent, if you need it anyways you can just use s3 keys to deconflict and store the workflow state.
  - orf 4mo ago
    Yes, but directly using s3 as a key-value database is completely different from using SQLite + litestream.
- gwking 4mo ago
  The suitability of Litestream for production disaster recovery is also an open question in my mind. I used 0.3.x for several years and when I tried to upgrade to the 0.5.x series there were runaway disk usage problems that would have caused downtime had they made it to prod. As far as I can tell these have not been entirely addressed, although recent bug reports suggest that they might be getting closer. I want to love it, and I don't take open source projects like this for granted. But during my last production upgrade I chose to decommission Litestream in favor of a dumber, less granular solution using sqlite3_rsync and nightly backups because there is no point in using a backup system that is not rock solid.
- 0cf8612b2e1e 4mo ago
  Postgres also does not synchronously replicate for free. You can setup both to get a confirmation write if you require that durability.
  - orf 4mo ago
    > postgresql also does not synchronously replicate By default. Generally your primary database is in a completely different failure category than a kubernetes node running an ephemeral workflow pod.
    - 0cf8612b2e1e 4mo ago
      Either you have durable storage or you do not. SQLite and Postgres can both ensure local durability of commits. If you want distributed durability, you need to ship that data elsewhere. That is another Postgres node, object store, whatever that’s still an external dependency.
  - paulddraper 4mo ago
    Not for free, but without the needing additional software. synchronous_commit = on
    - 0cf8612b2e1e 4mo ago
      That’s about the local transaction, not replication. SQLite WAL also gives you strict durability. PRAGMA synchronous = full
- paulddraper 4mo ago
  "Durable workflows without the durability" That's distributed workflows :)
- dilyevsky 4mo ago
  i mean it's durable as long as nothing crashes or litestream has a data corruption bug which only happens every other release...
- faangguyindia 4mo ago
  try setting up replication/failover in postgres, it's much more work.
Xcelerate 4mo ago
Haha, I just started doing this on my own. Found it helps the agents preserve state better. I typically ask them to design a DAG first based on a set of specifications and then execute it (each step stores something in a SQLite DB). Iteration is pretty simple then because I just ask for a tweak to one or two steps of the DAG, and then to re-run. Funny how people are independently converging on similar patterns of "what works" here. Still feels like we're in the wild west with all these ad-hoc patterns of agent orchestration that people are coming up with.
- zrail 4mo ago
  Same. The prompt was essentially, every checkbox in this PLAN.md should be task in SQLite.
sgloutnikov 4mo ago
It's close enough that DBOS does support SQLite. [0] The default for prototyping is SQLite, but sure you can run it in production if you wanted. Obligatory list of workflow engines and libraries because it's such a common need that a lot have rolled their own. [1] [0] https://docs.dbos.dev/python/tutorials/database-connection https://docs.dbos.dev/python/tutorials/database-connection [1] https://github.com/meirwah/awesome-workflow-engines https://github.com/meirwah/awesome-workflow-engines
- oulipo2 4mo ago
  What would be the main differences between DBOS and Obelisk?
  - tomasol 4mo ago
    Hi, I wrote a comparison blog post between Obelisk and the Java version of DBOS a couple of months ago: https://obeli.sk/blog/comparing-dbos-part-1/ https://obeli.sk/blog/comparing-dbos-part-1/
bitexploder 4mo ago
I started setting up my workflows using Temporal. It deploys as relatively light weight local app. For an isolated local installation it uses SQLite. It makes the process of dealing with API retries and organizing workflows and tasks really simple. I recommend giving it a try. It is, philosophically, exactly what this article is suggesting, but it adds an incredibly rich and flexible interface for agents to work with. Additionally, the web UI makes it very easy to inspect workflows, review agent execution, etc. Temporal also encodes much higher reliability into your system, almost for free. Distributed and reliable systems are hard, don't reinvent the wheel IMO. If you find yourself wanting things like an easy way to then introspect your SQLite database, figure out what is happening in the workflow, compose individual tasks, make workflows trivially callable, etc, give Temporal a look. Alongside this, I have mostly moved away from files for agents. Markdown and JSON are great, but also feel like traps when building out smaller local apps. LLMs are great at SQLite and you can render anything you want out of it (Markdown, JSON, etc). It saves a lot of tokens when an agent can just query a specific row instead of having to fire up jq or grep through markdown. You get a nice portable self contained data management system that encourages agents to be more disciplined about how they structure their data than a bunch of files. It also continues to scale into MySQL/Postgres if your little local projects start to outgrow or become more formal, you already have schema and discipline around data.
- jawns 4mo ago
  Could you give an example of a case where you'd use SQLite instead of jq or grep through Markdown?
  - fragmede 4mo ago
    Honest answer is: whenever your markdown or json files get to be big enough that grep/jq takes long enough that you get bored waiting for it.
    - embedding-shape 4mo ago
      > get to be big enough that grep/jq takes long enough On a modern processor, that's about GBs of data typically, right?
kubik369 4mo ago
Meta comment: This is a domain under my countries TLD (Slovakia) and it is one of the handful of words that are a word with the TLD in my language (and coincidentally) also in English. Every now and then, I will check on the domains with a retrograde dictionary for domains that have this property and root of this particular domain had a roundcube email server on it (can be checked on archive.org). After further checking, the local company actually named themselves Obeli s.r.o. (s.r.o. is Ltd), presumably so that they could use a domain that is a real word when said together with the TLD. (EDIT:) Forgot to write the thing I wanted to mention in the first place: it appears the domain must have lapsed and/or the author bought it from the company that was using it. Another fascinating fact: our countries TLD has been stolen Ocean's 11 style (I am not kidding). After Czechoslovakia split into Czech Republic and Slovak Republic, the newly created Slovak .sk TLD has been under the care of people from the local university. The university also had some offices that they were leasing out. Someone had leased this office space (EDIT: this is important as this means they had the same physical address), created a company that had the same name as the NGO that was taking care of the domain, so e.g. the NGO was named "My Company o.z." and the perpetrator created a "My Company s.r.o." (our countries version of the american Ltd). This person then wrote to ICANN to change the address to the "My Company s.r.o." presumably under the pretense that this was just an administrative error and from this point, they have functionally taken custody of the TLD. I was not able to find how they did it technically, but I presume they persuaded ICANN to then point to their servers instead of the real ones. After this happened, it seems that no one noticed for some time. When they noticed, they tried taking it back, but they weren't able to. For some inexplicable reason, the government during that time (Šuster era, early 2000s) gave the new company a contract that was functionally uncancellable from the government side. Later governments made this even more uncancellable and in 2017, then Minister of IT (and as of this day president!) Pellegrini made the contract literally uncancellable. As a result of this, we have one of the most expensive domains around (18e/year, rising each year for no good reason). (EDIT:) The company running our countries TLD is now a foreign entity that the whole thing has been sold to (multiple owners over time) and we as a country have no control over if I understand it correctly. I might have gotten some details wrong as I am writing this from my memory of researching it a couple of years back, but you get the idea, crazy stuff. Here is an article in Czech [0] that tells the story a bit better, but you have to translate it. [0] https://www.root.cz/clanky/pribeh-domeny-sk-aneb-kradez-za-bileho-dne/ https://www.root.cz/clanky/pribeh-domeny-sk-aneb-kradez-za-b... // EDIT: I have found that the article actually links the movement to return the TLD back [1]. It also has a story tab [2], so they have something much more precise than the paraphrasing I wrote. [1] https://www.nasadomena.sk/ https://www.nasadomena.sk/ [2] https://www.nasadomena.sk/historia/ https://www.nasadomena.sk/historia/
shukantpal 4mo ago
SQLite is surprisingly performant for single node applications even when comparing to Postgres. Postgres consumes a lot more memory and requires IO to hop through IPC whereas you can keep everything in process in SQLite with a shared connection pool. I've been testing different storage engines for my agent harness and I can get up to 7.5k concurrent sessions on a single vCPU with SQLite whereas Postgres crashes or runs out connections. [0] https://github.com/impalasys/talon/pull/23#issuecomment-4577660216 https://github.com/impalasys/talon/pull/23#issuecomment-4577...
- onlyrealcuzzo 4mo ago
  > SQLite is surprisingly performant for single node applications even when comparing to Postgres. In the context of SQLite being understood to be a quite excellent piece of software - shouldn't we expect it to be? In the context of a single-node, Postgres is overkill. It should not be expected to be competitive with SQLite. This is almost like benchmarking an in-memory HashMap to Redis and being surprised that it performs well in ideal conditions.
  - shukantpal 4mo ago
    Yes, agreed on SQLite/Postgres. But I'm going to benchmark RocksDB next and see what the performance characteristics are. I suspect the LSM tree storage engine of RocksDB might perform better since agents are so write heavy when running highly concurrent workloads. After all, you are streaming LLM tokens into disk and fanning them out to subscribed clients.
    - onlyrealcuzzo 4mo ago
      You might want to start here: https://docs.cozodb.org/en/latest/releases/v0.3.html https://docs.cozodb.org/en/latest/releases/v0.3.html
      - andriy_koval 4mo ago
        That project has 0 commits for 2 years.
levkk 4mo ago
I don't understand this obsession with SQLite for real, production apps. SQLite is an embedded database, completely unsuitable for managing concurrency. This is what database _servers_ are for, e.g., Postgres, MySQL, etc. Their entire job is to allow you to modify data from multiple processes, on different machines, at the same time. This is a foundational principle of computer science. It seems to me that the "SQLite for everything" crowd is a little bit inexperienced.
- bborud 4mo ago
  Computer science no more get its hands dirty with concrete software than physics primarily being about building bridges. It is not «a foundational principle of computer science».
- teaearlgraycold 4mo ago
  Well if you run a tiny single-threaded app then SQLite is a nice simplification over spinning up a separate machine for Postgres.
  - ai_fry_ur_brain 4mo ago
    I use postgres for very simple apps. I have a Dockerfile I use in my boilerplate repo. It takes a single make cmd for me to build, start and run migrations. Its as simple as using sqlite.
    - tasuki 4mo ago
      But now you have another process to babysit. How do you keep it healthy? And you have to ensure the client-server communication won't break. For me the main benefit of sqlite is that it's a library rather than an app.
      - ai_fry_ur_brain 4mo ago
        I have boilerplate for client-server communication that makes it pretty trivial to build on top of. Im not saying that sqlite isn't useful, im mostly saying that using postgres doesnt have to be complicated.
localhoster 4mo ago
Idk if this article was vibe written or the author just "got adjusted" but it's clearly is, and it's unreadable. Man this becomes anmoying
steveharing1 4mo ago
[dead]
0x59 4mo ago
Big complex data model with ambiguous query patterns? Postgres Small, well defined, data model with known query patterns? Bespoke model There probably is a place for sqlite and my project space so far hasn't yet well-aligned with it.
- asdff 4mo ago
  Probably going to get some winces for this but I do everything with flat files. Maybe my data aren't massive enough, but I mean I can do the relational thing by just having these metadata in some column, and returning rows that contain my desired information in these columns. Even if the file were too big to fit into memory one could just subset chunks of it and chew through. All this can be done with no dependencies, just base libraries of a lot of languages.
  - 0x59 4mo ago
    IMO flat files is a cromulent solution for a subset of the entire database space. "Everything" is too broad for me but different strokes for different folks. Happy dataing.
yokoprime 4mo ago
If you're just doing workflows from a single node, i guess it can be ok as long as theres a single writer. But scaling across multiple servers it clearly is not all you need.
golem14 4mo ago
Litestream releases 5.9 and newer have a bug that causes instances to sync an insane amount of data. a DB with <10K of data in it and practically no writes/reads causes something like 10GB of daily replication traffic. For my toy project that got needlessly expensive.
- https443 4mo ago
  Is this bug logged?
  - irons 4mo ago
    Looks like https://github.com/benbjohnson/litestream/issues/1197 https://github.com/benbjohnson/litestream/issues/1197. Still open as of now, with a potential cause noted in the latest comment.
    - Wilduck 4mo ago
      This is one of a bunch of issues that have been popping up since a vibe coder took over the bulk of development on this project. There's a (probably also AI generated) list of a big portion of the issues here: https://github.com/benbjohnson/litestream/issues/1221 https://github.com/benbjohnson/litestream/issues/1221. That proposal has been open for a few months, and it seems (from my POV) unlikely to be resolved any time soon.
- Wilduck 4mo ago
  I've been following litestream for a while, and it seems like the project has been hijacked by a vibe coder. I wouldn't trust it for critical tasks anymore.
netik 4mo ago
Until you scale past one machine…
ChrisArchitect 4mo ago
Related: Building durable workflows on Postgres https://news.ycombinator.com/item?id=48313530 https://news.ycombinator.com/item?id=48313530
- momojo 4mo ago
  Surfacing for the thread: Armin Ronacher's "Absurd Workflows: Durable Execution With Just Postgres" https://lucumr.pocoo.org/2025/11/3/absurd-workflows/ https://lucumr.pocoo.org/2025/11/3/absurd-workflows/
CoderAshton 4mo ago
[dead]
bze12 4mo ago
Isn’t this very similar to cloudflare durable objects & workflows?
stephenlf 4mo ago
Can’t wait to see the next iteration of this idea with “Logs are all you need for durable workflows.”
- gchamonlive 4mo ago
  Are logs all you need for durable workflows? I'm confused here. How'd persist and query nested or related data over logs? By logs I assume you mean something like elasticsearch or meilisearch?
  - Rapzid 4mo ago
    Log as in the structure.
  - wolttam 4mo ago
    Pretty much every durable system has an intent log of some sort. The log provides durability, the database system just integrates that log into a more queryable format.
    - notawhitemale 4mo ago
      [dead]
    - gchamonlive 4mo ago
      I swear it didn't occur to me that that mean WAL, makes much more sense now LOL
  - fourside 4mo ago
    I read the parents comment as sarcasm and not a serious suggestion.
  - deathanatos 4mo ago
    I assume they meant a log like a WAL. A WAL should be (quite literally?) all you need for durable workflows. A distributed WAL (to survive a machine death) would also probably be something I'd want, and … something I'm not sure you're getting directly from SQLite.
tutamon 4mo ago
[dead]
m2f2 4mo ago
There's a wide gap from files to multipartition databases. Running databases in a container is not for me sorry whenever real production stuff is on the table. Personally, lots of ETL can just be taken care of locally without involving enterprise databases. In such cases, DuckDB is 5x-10x better than SQLite and orders of magnitude simpler/faster than spinning up a dedicated Postgres database. For general scripting, there's no match between a 20-lines awk script and a much cleaner, robust, maintainable equivalent SQL script based on DuckDB. I just hope MotherDuck don't need to pump/dump for IPO - it would be sad losing that tool for the usual corporate greed.
- szarnyasg 4mo ago
  Hello, DuckDB devrel here. First, thanks for the kind words :) Second, it's funny you should mention the 20-line awk script. I was making a very similar argument yesterday at the Ubuntu Summit: at some point, using shell scripts with GNU coreutilus becomes impractical, while DuckDB SQL scripts scale better in terms of complexity and maintainability (and often also performance). My slides are here: https://blobs.duckdb.org/slides/duckdb-ubuntu-summit-2026.pdf#32 https://blobs.duckdb.org/slides/duckdb-ubuntu-summit-2026.pd... (pages 32 to 36) Third, MotherDuck develops a closed-source DBaaS on DuckDB. They build on DuckDB, and you connect to MotherDuck with DuckDB but they are a separate VC-funded company headquartered in Seattle. DuckDB is developed by DuckLabs, a bootstrapped (revenue-funded) company in Amsterdam. And the IP of the project is in a third organization: a Dutch non-profit called the DuckDB Foundation. For details, see https://duckdb.org/faq#how-are-duckdb-the-duckdb-foundation-ducklabs-and-motherduck-related https://duckdb.org/faq#how-are-duckdb-the-duckdb-foundation-...
- infinet 4mo ago
  I use DuckDB and like it. Since many mentioned GB level json in this post, so they have large amount of data. Been column based, DuckDB uses more RAM as row count grows. It can be an advantage or disadvantage depends whether memory is constrained. Traditional row based DB such as SQLite can deal with large database with less memory.
madbo1 4mo ago
[flagged]
orliesaurus 4mo ago
Surprised no one has mentioned Turbopuffer yet [1] which natively supports dense vector similarity and BM25 keyword indexes out of the box [1]. https://turbopuffer.com/ https://turbopuffer.com/
- schmookeeg 4mo ago
  [dead]
skybrian 4mo ago
Instead of "just use Litestream," I'd like to see a review of different object stores one could use and which ones work well with Litestream. Is there a nice object store I could run in another Linux VM? As a hobbyist, which services providing an S3-like API make the most sense?
- chrsstrm 4mo ago
  Litestream is just the replication layer, it works with any S3 compatible storage, and all of these in their guide as well https://litestream.io/guides/#replica-guides https://litestream.io/guides/#replica-guides
  - skybrian 4mo ago
    No, I know that already. That's not a recommendation. Some storage layers have to be better than others on price, reliability, and so on? Think Wirecutter, not install guide.
mburaksayici 4mo ago
Agreeing on the point, I needed NoSQL version on the similar uses, I've used TinyDB : https://mburaksayici.com/blog/2024/09/21/easy-to-use-nosql-prompt-database-for-small-projects.html https://mburaksayici.com/blog/2024/09/21/easy-to-use-nosql-p...
gunnarmorling 4mo ago
Related piece I wrote some time ago: https://www.morling.dev/blog/building-durable-execution-engine-with-sqlite/ https://www.morling.dev/blog/building-durable-execution-engi...
prmph 4mo ago
> Postgres ... is the right choice when you need higher availability, broader shared scalability, or other deployment properties that are better served by a network database. It is also the better fit when asynchronous replication to object storage is not the durability model you want... Many workflow systems do not need that on day one and should not start with more infrastructure than their state actually demands. ------ I see this kind of YAGNI thinking a lot, but in my view, it must be balanced against the effort you'd put into resolving any edge cases and adapting current architecture to your use case. Imagine you deploy Sqlite, and thought it fine by itself, you keep running into some unforeseen challenges with the use to which you are putting. YOu'd need to sink valuable time and effort into addressing those. Then, when you have outgrown it, you'd beed to spend additional valuable times dping the same with Postgres. This is why, when it comes to Architecture, I increasingly find my myself over-enigneering a bit. Assuming there is a good chance you might need to upgrade your architecture in the not too distant future, that approach is actually kind of very efficient. I find that I am able to uncover a lot of potential gotchas, which feeds back into the what the simplified current architecture should be, and helps me understand the roadmap I'm facing very well. I also avoid wasting too much time going too deep in directions that make sense now, but need a lot of plumbing to get right, when I can see that I'd likely have to throw it all out in a few years. Going from A -> B -C -> D, where each step is the optimal good-enough-for-now architecture but which requires a lot of work to stabilize and iron out the kinks of, is much less efficient than exploring D well enough to know whether you should build A, B, or C now. Basically, some over-engineering, if done right, is not wasted. It cuts right to the heart of what you are dealing with, efficiently, and allows you to make (maybe) simpler but informed choices now as to how best to allocate your development resources now.
- narnarpapadaddy 4mo ago
  My version of this is the “N+1” principle. Build for one more foreseeable use case than you currently have. The domain model will click in when you need to generalize a solution, and you’ll gain the ability to see your particular solution as one of several to the problem, and thus evaluate fit and tradeoffs more clearly. Don’t do N+2. The goal isn’t to predict the future, nobody can do that. The goal is a durable understanding of the domain and the best fit implementation you can get with that current understanding and resources. That said, SQLite passes that bar for me in most use cases.
Thaxll 4mo ago
I started using SQLite for a home project after years of reading about it, I was shocked at the poor type system coming from Postgres. It is really inferior, not sure why it gets so much praise. https://sqlite.org/datatype3.html https://sqlite.org/datatype3.html https://www.postgresql.org/docs/current/datatype.html https://www.postgresql.org/docs/current/datatype.html Working with date/time feels like using a 30years old database, nothing is enforced at insert. Really someone needs to explain why so many people like it.
- RivieraKid 4mo ago
  Yes, this is basically my only issue with SQLite. SQLite with a strict type system would be great.
  - deleted 4mo ago
    [deleted]
- deleted 4mo ago
  [deleted]
- ThatMedicIsASpy 4mo ago
  it's a single file.
- zimmi 4mo ago
  You can use strict tables: https://sqlite.org/stricttables.html https://sqlite.org/stricttables.html
  - pseudalopex 4mo ago
    This could enforce dates are strings. They wanted to enforce dates are dates I thought.
    - zimmi 4mo ago
      Storing dates as INTEGER (year * 10000 + month * 100 + day, e.g. 20260530) is not so bad. Proper date / timestamp types would be great though.
teravor 4mo ago
if you have an application that needs to maintain state in a non-critical section or if you discover that using SQL is actually a good idea for some tasks (even in critical sections), SQLite is not only a good choice but it will save you a lot of time coming up with a brittle custom solution. maintain an in-memory SQLite db and work it with SQL commands, and if you also want to preserve state across application restarts you can routinely save to disk or load from it: <https://www.sqlite.org/backup.html#example_1_loading_and_saving_in_memory_databases https://www.sqlite.org/backup.html#example_1_loading_and_sav...> this also happens to be the most convenient file-format (aka. application-format) I ever worked with.
opiniateddev 4mo ago
[flagged]
PUSH_AX 4mo ago
I went from using the various big player postgres clusters to SQLite, we have an MAU in 7 figures, all backed by SQLite durable objects. We have to think differently about the access patterns but the benefits have been worth it.
nodesocket 4mo ago
The biggest annoyance about SQLite for me is no ability to: ALTER TABLE users MODIFY COLUMN… ALTER TABLE users ALTER COLUMN… ALTER TABLE users ADD CONSTRAINT… You have to create a new temporary table with correct schema, copy data into this new table, drop the old table, and then rename the temporary table.
- simonw 4mo ago
  They've been improving that recently: 2026-04-09 (3.53.0) - "Enhance ALTER TABLE to permit adding and removing NOT NULL and CHECK constraints" I use my own sqlite-utils CLI/Python library to work around these limitations: https://sqlite-utils.datasette.io/en/stable/python-api.html#python-api-transform https://sqlite-utils.datasette.io/en/stable/python-api.html#...
  - nodesocket 4mo ago
    Ahh that's very nice. Unfortunately the default version of sqlite3 provided by Debian Trixie is 3.46.1.
lvl155 4mo ago
And all you need is pen and paper to do calculations.
3dedb728-3f77 4mo ago
Is this just a AWS ads?
fathermarz 4mo ago
Excellent write up and inspired me for our next IA design run. After reading Fly’s Litestream work it makes me think this is a solid option.
flying_sheep 4mo ago
Cloudflare durable object is implemented with SQLite (or some variant of it)
vatsachak 4mo ago
Postgres doesn't cost any extra lol
- faangguyindia 4mo ago
  it does cost network stack
ai_slop_hater 4mo ago
> if you already trust your database, you do not need a separate orchestration tier Wow. Really? I thought I needed to use an overpriced cloud SaaS for everything.
vultour 4mo ago
The GitHub statistics for the project this website represents are insane. It has a sole author that has averaged approximately 20,000 lines of code every week in the past month. How do you even maintain that alone?
mococa 4mo ago
No, mmap is all you need.
psanford 4mo ago
I wrote a library[0] to let you concurrently update a sqlite db in s3 safely. It uses the little known sqlite sessions extension plus s3 compare-and-swap on a small metadata file to make this work reasonably efficiently and safely. I have been enjoying it for a bunch of small projects where I want a lambda function to have a db for state but I don't want to pay for a full database instance. [0]: https://github.com/psanford/s3db https://github.com/psanford/s3db
- password4321 4mo ago
  This is exactly what I had dreamed of except without being aware of the session extension... I put it off because I thought I would have to create a layer opening sequential partial sqlite dbs and somehow filtering to the most recent version of each record in-between compactions. It sounds like sqlite sessions handles the hardest part and you've sanded down some of the rough edges while implementing the glue bits for s3 (generously licensed MIT); thanks for the heads-up!
vkaku 4mo ago
Back in the day, I wrote a simple job queue with SQLite https://github.com/guilt/squeue https://github.com/guilt/squeue It did the job, was fairly easy to use.
faangguyindia 4mo ago
I've replaced all of these with Go + SQLite: 1. Intercom 2. Zendesk 3. Email marketing 4. Kanban 5. Todo 6. Our billing stack 7. Our issue tracker 8. Our forum 9. Uptime monitor 10. PagerDuty (clone) I have dozens of products I sell, so I thought: why not build everything ourselves? All of these run on the same server and use very little memory. I replaced all the SaaS tools we used with these. I also moved to dedicated servers and dropped costs to about 1/10th of what we were paying for managed cloud solutions, while maintaining the same HA and even achieving lower latency (partly because noisy neighbors on VPSes were increasing tail latency). We used to spend a ton on this stuff. These have now been in production for four months and have only needed minor updates. Deployment is dirt simple. No Docker, no Kubernetes—just a systemd service and a binary built on the dev machine and deployed. We also used to pay for services like MaxMind and IPData. I ended up hand-rolling my own IP geolocation service, which, in my tests, outperforms most existing solutions. It all started with replacing Uptime Robot. Then I got more confident and replaced PagerDuty. After that, I replaced Intercom. Finally, I had always heard people say, "Don't build your own billing stack." But I said YOLO, let me make that mistake myself. So I studied our existing billing solution, developed my own, and rolled it out. So far, we've had zero issues with it. Caddy in front. I found that we only use maybe 1–5% of the features most SaaS products offer, while the features we actually need keep getting buried deeper and deeper inside these "enterprise-grade" platforms, making our workflows more difficult. I won't show my commercial products because our partners and clients probably wouldn't appreciate knowing how cheap I am—but I call it being resourceful. I can show my free app, though, which has 20,000+ users and was launched recently: https://macrocodex.app/ https://macrocodex.app/ It only uses the Zendesk clone. Email is handled through Cloudflare routing, so we pay almost nothing to run the app.
- tonymet 4mo ago
  I rolled my own uptime but how do you vary region or do residential testing ?
  - faangguyindia 4mo ago
    I actually don't. I just have uptime service hosted outside of our main infra. It connects to my service called Siren, which alerts me on my phone with an alarm on full volume with SWAT cat intro. It's good enough for what we do, barely have any downtime. But it helped me figure out 6s downtime we would experience when our spot instances get knocked out, so it helped me increase health check frequency 6s downtime is a lot when you are getting hammered at 100 RPS.
dannypdx 4mo ago
All this SQLite hate from big vector db... leave SQLite alone!
simplestates 4mo ago
Good framing. SQLite is often enough when the main problem is making workflow state durable, inspectable and easy to recover.
dnnddidiej 4mo ago
Butchering the Beatles song again.
- tomasol 4mo ago
  sorry about that, I do love Beatles!
emodendroket 4mo ago
SQLite is an underrated tool for how powerful it really is and probably people don't think of it often enough.
sharts 4mo ago
For something so widely deployed you’d think it’d be included in Claude/Codex/etc for
ryanisnan 4mo ago
Sweet I get to tell my team we can move off of dapr workflows
aykutseker 4mo ago
Storage never ended up being the thing we worried about. The painful bits started once a workflow could touch external systems. Replaying state is one thing. Replaying a charge or an email is another. How are you dealing with that?
jkwang 4mo ago
[flagged]
shevy-java 4mo ago
Hmmm. SQLite is great, but I remember years ago, at a university cluster, I had to populate a SQL database via tons of INSERT statements from genomic/meta-genomic workflows. Postgresql was so much faster just at that particular action (inserting data) that it convinced me that SQLite may be useful for many, many applications, but for "big data"(sets), Postgresql is simply better.
dev_l1x_be 4mo ago
I start to think the SQLite is all i need to store data. When there is a chance of non-coordinated writes that I can distribute among servers (or even a range based ID) SQLite is my first idea. With durable storage backups this works amazingly well.
utopiah 4mo ago
The cycle of expertise : - what is X, I just do Y - wow I can see so many limits of Y, now I do X - I use X for literally everything - now that I properly understand the limits of Y but also the heavy constraints of X ... maybe Y is enough - I use Y for literally everything rinse & repeat. The thing is with actual usage and actual context one does learn and thus can get away with a lot more "basic" solution but it does require genuinely understanding the limits.
- dmos62 4mo ago
  Persistence in folly leads a fool to enlightenment.
  - deleted 4mo ago
    [deleted]
- stingraycharles 4mo ago
  Yup, when I look back at the silly stuff I did when I was somewhere in the middle (CQRS + event sourcing I’m looking at you), it’s interesting. It is a source of expertise, because you really learn a lot from it. But when you become old (43 over here), you really learn to appreciate “boring” solutions.
  - utopiah 4mo ago
    Boring is the new sexy. /wisdom
  - roryirvine 4mo ago
    You also begin to recognise that the definition of "boring" changes over time, and - if you wait long enough - fashions begin to repeat themselves. So, xBase was all you needed in the mid 80s. Then DBM was all you needed in the mid 2000s. Now, in the mid 2020s, we're told that it's SQLite that is all you need. It was partly true then, and partly true now. But the full story's always been more complicated, so it's still worth considering a range of potential solutions rather than relying on simplistic rules of thumb or slogans. (Wake me when nosql comes back into fashion, I'll be able to do a great "old man yells at clouds" routine about that one...)
halamadrid 4mo ago
Operators of Unmeshed here, which is basically a rewrite of Netflix Conductor. In this orchestrator we heavily use a uniquely scaled version of SQLite and also offers “managed” SQLite instances for managing user data. Combining the durable executions of Unmeshed and along with workflow primitives like sleep, workers, etc you can actually build complex systems with a lot less code than ever. Check it out here: https://unmeshed.io https://unmeshed.io
jkwang 4mo ago
[flagged]
unnouinceput 4mo ago
Quote 1: "DBOS recently argued that Postgres is all you need for durable execution...SQLite is all you need." Quote 2: "SQLite State backed up to S3". Yeah buddy, if you think S3 isn't Postgres I'll eat my foot.
sgt 4mo ago
Next one: Berkeley DB is all you need for durable workflows /s
bob1029 4mo ago
> This is especially attractive for AI agents and AI-generated workflows. Those systems are often bursty, experimental, and easier to reason about when each agent or tenant has a small self-contained unit of state. I am finding that the most important thing is one big, consistent data warehouse that is updated with the state of the business as close to real time as we can get. SQLite is not really great at this particular problem. Something like Postgres or SQL Server would be much more suitable for an OLAP data warehouse that can serve clients (AI agents) while simultaneously merging massive record sets from upstream business systems. These products also offer intricate permissions control. You can prove to an auditor that your AI solution will never see tables or rows it's not supposed to. SQLite doesn't even have a concept of a user, role or login. > The compute can stay cheap and disposable. Again, hosted sql is better aligned. The alternative is DIY hosted sql (SQLite + some other magic) which immediately violates this rule.
freakynit 4mo ago
SQLite backed with Raid-10 NVME disks and periodic backups to cloud storage is generally more than enough to run majority of the production workloads of startups. Writes are single threaded, but, you can still easily do thousands per second. DuckDB offers similar qualities, on the OLAP side. This is not to say this is best combination.. but, when you consider the simplicity of setup, usage, operations, and backups, and cost element, this indeed offers one of the best, if not the best combination.
jackzhuo 4mo ago
100% this. I used to default to Postgres for everything. But seeing SQLite handle concurrency so well now—plus having built-in BM25 search and vector support—it really is all you need for these kinds of architecture.
sungjinwo0 4mo ago
[flagged]
seobot_dk1289 4mo ago
[flagged]
vixalien 4mo ago
Has anyone actually used PGLite[0]? [0]: https://pglite.dev/ https://pglite.dev/
jessmartin 4mo ago
Waiting for “JSONL is all you need for durable workflows.”
Xeoncross 4mo ago
So is https://github.com/obeli-sk/obelisk https://github.com/obeli-sk/obelisk the Rust version of https://github.com/temporalio/temporal https://github.com/temporalio/temporal (Go)? Can you guys add a comparison between them on the site?
- tomasol 4mo ago
  Subscribe to this issue: https://github.com/obeli-sk/obelisk/issues/65 https://github.com/obeli-sk/obelisk/issues/65
throwaway58670 4mo ago
Holy sticky header... It takes literally half the screen on mobile. Shit like this makes me wonder: do you even look at your website? At least sometimes?
oulipo2 4mo ago
Obelisk also supports Postgres. When you're using it with Postgres, what are the differences with DBOS? Are there any that would be significant (I'm already using Postgres, so I don't really need a sqlite-based durable workflow engine, so looking to know how to choose between DBOS and Obelisk)
- tomasol 4mo ago
  Check out https://obeli.sk/blog/comparing-dbos-part-1/ https://obeli.sk/blog/comparing-dbos-part-1/
martinloop 4mo ago
[flagged]
bakrisolo 4mo ago
The move toward local-first, durable workflows is honestly a breath of fresh air. I’ve spent the last few months building a document analysis tool and the amount of friction involved in trying to make cloud-based RAG pipelines 'durable' vs just keeping the state in local SQLite is night and day. Once you stop fighting the latency and privacy issues of the cloud you can build much faster.
sperandeo 4mo ago
I concur. I keep waiting to hit the wall where I need Postgres and it just hasn't happened. PB ftw
themuskgpt2025 4mo ago
[dead]
henryz2004 4mo ago
[dead]
sonthonax 4mo ago
The thing that people forget about databases is that they are just really complicated ways of writing to a file. The crux of all these “you only need sqlite” posts isn’t that you need SQLite, it’s that your architecture only ever needed flat appendable log file to begin with. Once you realise that SQLite is just an aggregated view over this log it calls into question if you needed a third party query engine at all, suddenly the whole NoSQL vs SQL debate becomes a meaningless implementation detail. Server based databases aren’t special beyond the fact they’re just distributed synchronisation primitives over files on a file system. But the utility of this is quite limited in practice, writing to a table will always require some sort of synchronisation, using SQLite vs Postgres is just a case of where you put the lock - in your own code or in a 3rd party server.
- pdimitar 4mo ago
  > Once you realise that SQLite is just an aggregated view over this log it calls into question if you needed a third party query engine at all, suddenly the whole NoSQL vs SQL debate becomes a meaningless implementation detail. You are correct on the premise, however my observation has been that teams who try to be lean and mean and minimize dependencies always start off believing they don't need a query engine... and they always end up needing one, often making things much worse by refusing to just migrate to any database due to sunk cost fallacy. I fully agree with your argument that it's all just file operations in the end. To me SQLite wins for everything beyond persistence however: the SQL commands, extensions, backups, changesets / patchsets. Can you, I, and many others write something that has 10% of SQLite feature set that serves us perfectly? Absolutely! But I don't want to, and in business settings nobody will allow you to hand-roll crucial infrastructure software (like databases) unless they're in full crisis mode and you're the mega-expensive consultant brought in to save the day.
sspoisk 4mo ago
[flagged]

720 points by tomasol 4mo ago

EGreg 4mo ago

Files is all you need. https://xkcd.com/378/ https://xkcd.com/378/

tclancy 4mo ago
Post It Notes will do if you have a good system.
contingencies 4mo ago
Those who don't understand Unix are condemned to reinvent it, poorly. - Henry Spencer .. via https://github.com/globalcitizen/taoup https://github.com/globalcitizen/taoup
jusonchan81 4mo ago
Agree on this and it’s the architecture that backs our orchestration platform called https://unmeshed.io https://unmeshed.io

orf 4mo ago

> The caveat is that Litestream replication is asynchronous. A restore can miss the newest local writes if the SQLite volume disappears before they are copied. That is fine for many AI and experimentation workflows In short: SQLite is not all you need, unless you’re just experimenting don’t actually care about durability, in which case you also need litestream + object storage. Right.

bootsmann 4mo ago
S3 is strongly consistent, if you need it anyways you can just use s3 keys to deconflict and store the workflow state.
- orf 4mo ago
  Yes, but directly using s3 as a key-value database is completely different from using SQLite + litestream.
gwking 4mo ago
The suitability of Litestream for production disaster recovery is also an open question in my mind. I used 0.3.x for several years and when I tried to upgrade to the 0.5.x series there were runaway disk usage problems that would have caused downtime had they made it to prod. As far as I can tell these have not been entirely addressed, although recent bug reports suggest that they might be getting closer. I want to love it, and I don't take open source projects like this for granted. But during my last production upgrade I chose to decommission Litestream in favor of a dumber, less granular solution using sqlite3_rsync and nightly backups because there is no point in using a backup system that is not rock solid.
0cf8612b2e1e 4mo ago
Postgres also does not synchronously replicate for free. You can setup both to get a confirmation write if you require that durability.
- orf 4mo ago
  > postgresql also does not synchronously replicate By default. Generally your primary database is in a completely different failure category than a kubernetes node running an ephemeral workflow pod.
  - 0cf8612b2e1e 4mo ago
    Either you have durable storage or you do not. SQLite and Postgres can both ensure local durability of commits. If you want distributed durability, you need to ship that data elsewhere. That is another Postgres node, object store, whatever that’s still an external dependency.
- paulddraper 4mo ago
  Not for free, but without the needing additional software. synchronous_commit = on
  - 0cf8612b2e1e 4mo ago
    That’s about the local transaction, not replication. SQLite WAL also gives you strict durability. PRAGMA synchronous = full
paulddraper 4mo ago
"Durable workflows without the durability" That's distributed workflows :)
dilyevsky 4mo ago
i mean it's durable as long as nothing crashes or litestream has a data corruption bug which only happens every other release...
faangguyindia 4mo ago
try setting up replication/failover in postgres, it's much more work.

Xcelerate 4mo ago

Haha, I just started doing this on my own. Found it helps the agents preserve state better. I typically ask them to design a DAG first based on a set of specifications and then execute it (each step stores something in a SQLite DB). Iteration is pretty simple then because I just ask for a tweak to one or two steps of the DAG, and then to re-run. Funny how people are independently converging on similar patterns of "what works" here. Still feels like we're in the wild west with all these ad-hoc patterns of agent orchestration that people are coming up with.

zrail 4mo ago
Same. The prompt was essentially, every checkbox in this PLAN.md should be task in SQLite.

sgloutnikov 4mo ago

It's close enough that DBOS does support SQLite. [0] The default for prototyping is SQLite, but sure you can run it in production if you wanted. Obligatory list of workflow engines and libraries because it's such a common need that a lot have rolled their own. [1] [0] https://docs.dbos.dev/python/tutorials/database-connection https://docs.dbos.dev/python/tutorials/database-connection [1] https://github.com/meirwah/awesome-workflow-engines https://github.com/meirwah/awesome-workflow-engines

oulipo2 4mo ago
What would be the main differences between DBOS and Obelisk?
- tomasol 4mo ago
  Hi, I wrote a comparison blog post between Obelisk and the Java version of DBOS a couple of months ago: https://obeli.sk/blog/comparing-dbos-part-1/ https://obeli.sk/blog/comparing-dbos-part-1/

bitexploder 4mo ago

I started setting up my workflows using Temporal. It deploys as relatively light weight local app. For an isolated local installation it uses SQLite. It makes the process of dealing with API retries and organizing workflows and tasks really simple. I recommend giving it a try. It is, philosophically, exactly what this article is suggesting, but it adds an incredibly rich and flexible interface for agents to work with. Additionally, the web UI makes it very easy to inspect workflows, review agent execution, etc. Temporal also encodes much higher reliability into your system, almost for free. Distributed and reliable systems are hard, don't reinvent the wheel IMO. If you find yourself wanting things like an easy way to then introspect your SQLite database, figure out what is happening in the workflow, compose individual tasks, make workflows trivially callable, etc, give Temporal a look. Alongside this, I have mostly moved away from files for agents. Markdown and JSON are great, but also feel like traps when building out smaller local apps. LLMs are great at SQLite and you can render anything you want out of it (Markdown, JSON, etc). It saves a lot of tokens when an agent can just query a specific row instead of having to fire up jq or grep through markdown. You get a nice portable self contained data management system that encourages agents to be more disciplined about how they structure their data than a bunch of files. It also continues to scale into MySQL/Postgres if your little local projects start to outgrow or become more formal, you already have schema and discipline around data.

jawns 4mo ago
Could you give an example of a case where you'd use SQLite instead of jq or grep through Markdown?
- fragmede 4mo ago
  Honest answer is: whenever your markdown or json files get to be big enough that grep/jq takes long enough that you get bored waiting for it.
  - embedding-shape 4mo ago
    > get to be big enough that grep/jq takes long enough On a modern processor, that's about GBs of data typically, right?

kubik369 4mo ago

Meta comment: This is a domain under my countries TLD (Slovakia) and it is one of the handful of words that are a word with the TLD in my language (and coincidentally) also in English. Every now and then, I will check on the domains with a retrograde dictionary for domains that have this property and root of this particular domain had a roundcube email server on it (can be checked on archive.org). After further checking, the local company actually named themselves Obeli s.r.o. (s.r.o. is Ltd), presumably so that they could use a domain that is a real word when said together with the TLD. (EDIT:) Forgot to write the thing I wanted to mention in the first place: it appears the domain must have lapsed and/or the author bought it from the company that was using it. Another fascinating fact: our countries TLD has been stolen Ocean's 11 style (I am not kidding). After Czechoslovakia split into Czech Republic and Slovak Republic, the newly created Slovak .sk TLD has been under the care of people from the local university. The university also had some offices that they were leasing out. Someone had leased this office space (EDIT: this is important as this means they had the same physical address), created a company that had the same name as the NGO that was taking care of the domain, so e.g. the NGO was named "My Company o.z." and the perpetrator created a "My Company s.r.o." (our countries version of the american Ltd). This person then wrote to ICANN to change the address to the "My Company s.r.o." presumably under the pretense that this was just an administrative error and from this point, they have functionally taken custody of the TLD. I was not able to find how they did it technically, but I presume they persuaded ICANN to then point to their servers instead of the real ones. After this happened, it seems that no one noticed for some time. When they noticed, they tried taking it back, but they weren't able to. For some inexplicable reason, the government during that time (Šuster era, early 2000s) gave the new company a contract that was functionally uncancellable from the government side. Later governments made this even more uncancellable and in 2017, then Minister of IT (and as of this day president!) Pellegrini made the contract literally uncancellable. As a result of this, we have one of the most expensive domains around (18e/year, rising each year for no good reason). (EDIT:) The company running our countries TLD is now a foreign entity that the whole thing has been sold to (multiple owners over time) and we as a country have no control over if I understand it correctly. I might have gotten some details wrong as I am writing this from my memory of researching it a couple of years back, but you get the idea, crazy stuff. Here is an article in Czech [0] that tells the story a bit better, but you have to translate it. [0] https://www.root.cz/clanky/pribeh-domeny-sk-aneb-kradez-za-bileho-dne/ https://www.root.cz/clanky/pribeh-domeny-sk-aneb-kradez-za-b... // EDIT: I have found that the article actually links the movement to return the TLD back [1]. It also has a story tab [2], so they have something much more precise than the paraphrasing I wrote. [1] https://www.nasadomena.sk/ https://www.nasadomena.sk/ [2] https://www.nasadomena.sk/historia/ https://www.nasadomena.sk/historia/

shukantpal 4mo ago

SQLite is surprisingly performant for single node applications even when comparing to Postgres. Postgres consumes a lot more memory and requires IO to hop through IPC whereas you can keep everything in process in SQLite with a shared connection pool. I've been testing different storage engines for my agent harness and I can get up to 7.5k concurrent sessions on a single vCPU with SQLite whereas Postgres crashes or runs out connections. [0] https://github.com/impalasys/talon/pull/23#issuecomment-4577660216 https://github.com/impalasys/talon/pull/23#issuecomment-4577...

onlyrealcuzzo 4mo ago
> SQLite is surprisingly performant for single node applications even when comparing to Postgres. In the context of SQLite being understood to be a quite excellent piece of software - shouldn't we expect it to be? In the context of a single-node, Postgres is overkill. It should not be expected to be competitive with SQLite. This is almost like benchmarking an in-memory HashMap to Redis and being surprised that it performs well in ideal conditions.
- shukantpal 4mo ago
  Yes, agreed on SQLite/Postgres. But I'm going to benchmark RocksDB next and see what the performance characteristics are. I suspect the LSM tree storage engine of RocksDB might perform better since agents are so write heavy when running highly concurrent workloads. After all, you are streaming LLM tokens into disk and fanning them out to subscribed clients.
  - onlyrealcuzzo 4mo ago
    You might want to start here: https://docs.cozodb.org/en/latest/releases/v0.3.html https://docs.cozodb.org/en/latest/releases/v0.3.html
    - andriy_koval 4mo ago
      That project has 0 commits for 2 years.

levkk 4mo ago

I don't understand this obsession with SQLite for real, production apps. SQLite is an embedded database, completely unsuitable for managing concurrency. This is what database _servers_ are for, e.g., Postgres, MySQL, etc. Their entire job is to allow you to modify data from multiple processes, on different machines, at the same time. This is a foundational principle of computer science. It seems to me that the "SQLite for everything" crowd is a little bit inexperienced.

bborud 4mo ago
Computer science no more get its hands dirty with concrete software than physics primarily being about building bridges. It is not «a foundational principle of computer science».
teaearlgraycold 4mo ago
Well if you run a tiny single-threaded app then SQLite is a nice simplification over spinning up a separate machine for Postgres.
- ai_fry_ur_brain 4mo ago
  I use postgres for very simple apps. I have a Dockerfile I use in my boilerplate repo. It takes a single make cmd for me to build, start and run migrations. Its as simple as using sqlite.
  - tasuki 4mo ago
    But now you have another process to babysit. How do you keep it healthy? And you have to ensure the client-server communication won't break. For me the main benefit of sqlite is that it's a library rather than an app.
    - ai_fry_ur_brain 4mo ago
      I have boilerplate for client-server communication that makes it pretty trivial to build on top of. Im not saying that sqlite isn't useful, im mostly saying that using postgres doesnt have to be complicated.

localhoster 4mo ago

Idk if this article was vibe written or the author just "got adjusted" but it's clearly is, and it's unreadable. Man this becomes anmoying

steveharing1 4mo ago

[dead]

0x59 4mo ago

Big complex data model with ambiguous query patterns? Postgres Small, well defined, data model with known query patterns? Bespoke model There probably is a place for sqlite and my project space so far hasn't yet well-aligned with it.

asdff 4mo ago
Probably going to get some winces for this but I do everything with flat files. Maybe my data aren't massive enough, but I mean I can do the relational thing by just having these metadata in some column, and returning rows that contain my desired information in these columns. Even if the file were too big to fit into memory one could just subset chunks of it and chew through. All this can be done with no dependencies, just base libraries of a lot of languages.
- 0x59 4mo ago
  IMO flat files is a cromulent solution for a subset of the entire database space. "Everything" is too broad for me but different strokes for different folks. Happy dataing.

yokoprime 4mo ago

If you're just doing workflows from a single node, i guess it can be ok as long as theres a single writer. But scaling across multiple servers it clearly is not all you need.

golem14 4mo ago

Litestream releases 5.9 and newer have a bug that causes instances to sync an insane amount of data. a DB with <10K of data in it and practically no writes/reads causes something like 10GB of daily replication traffic. For my toy project that got needlessly expensive.

https443 4mo ago
Is this bug logged?
- irons 4mo ago
  Looks like https://github.com/benbjohnson/litestream/issues/1197 https://github.com/benbjohnson/litestream/issues/1197. Still open as of now, with a potential cause noted in the latest comment.
  - Wilduck 4mo ago
    This is one of a bunch of issues that have been popping up since a vibe coder took over the bulk of development on this project. There's a (probably also AI generated) list of a big portion of the issues here: https://github.com/benbjohnson/litestream/issues/1221 https://github.com/benbjohnson/litestream/issues/1221. That proposal has been open for a few months, and it seems (from my POV) unlikely to be resolved any time soon.
Wilduck 4mo ago
I've been following litestream for a while, and it seems like the project has been hijacked by a vibe coder. I wouldn't trust it for critical tasks anymore.

netik 4mo ago

Until you scale past one machine…

ChrisArchitect 4mo ago

Related: Building durable workflows on Postgres https://news.ycombinator.com/item?id=48313530 https://news.ycombinator.com/item?id=48313530

momojo 4mo ago
Surfacing for the thread: Armin Ronacher's "Absurd Workflows: Durable Execution With Just Postgres" https://lucumr.pocoo.org/2025/11/3/absurd-workflows/ https://lucumr.pocoo.org/2025/11/3/absurd-workflows/

CoderAshton 4mo ago

[dead]

bze12 4mo ago

Isn’t this very similar to cloudflare durable objects & workflows?

stephenlf 4mo ago

Can’t wait to see the next iteration of this idea with “Logs are all you need for durable workflows.”

gchamonlive 4mo ago
Are logs all you need for durable workflows? I'm confused here. How'd persist and query nested or related data over logs? By logs I assume you mean something like elasticsearch or meilisearch?
- Rapzid 4mo ago
  Log as in the structure.
- wolttam 4mo ago
  Pretty much every durable system has an intent log of some sort. The log provides durability, the database system just integrates that log into a more queryable format.
  - notawhitemale 4mo ago
    [dead]
  - gchamonlive 4mo ago
    I swear it didn't occur to me that that mean WAL, makes much more sense now LOL
- fourside 4mo ago
  I read the parents comment as sarcasm and not a serious suggestion.
- deathanatos 4mo ago
  I assume they meant a log like a WAL. A WAL should be (quite literally?) all you need for durable workflows. A distributed WAL (to survive a machine death) would also probably be something I'd want, and … something I'm not sure you're getting directly from SQLite.

tutamon 4mo ago

[dead]

m2f2 4mo ago

There's a wide gap from files to multipartition databases. Running databases in a container is not for me sorry whenever real production stuff is on the table. Personally, lots of ETL can just be taken care of locally without involving enterprise databases. In such cases, DuckDB is 5x-10x better than SQLite and orders of magnitude simpler/faster than spinning up a dedicated Postgres database. For general scripting, there's no match between a 20-lines awk script and a much cleaner, robust, maintainable equivalent SQL script based on DuckDB. I just hope MotherDuck don't need to pump/dump for IPO - it would be sad losing that tool for the usual corporate greed.

szarnyasg 4mo ago
Hello, DuckDB devrel here. First, thanks for the kind words :) Second, it's funny you should mention the 20-line awk script. I was making a very similar argument yesterday at the Ubuntu Summit: at some point, using shell scripts with GNU coreutilus becomes impractical, while DuckDB SQL scripts scale better in terms of complexity and maintainability (and often also performance). My slides are here: https://blobs.duckdb.org/slides/duckdb-ubuntu-summit-2026.pdf#32 https://blobs.duckdb.org/slides/duckdb-ubuntu-summit-2026.pd... (pages 32 to 36) Third, MotherDuck develops a closed-source DBaaS on DuckDB. They build on DuckDB, and you connect to MotherDuck with DuckDB but they are a separate VC-funded company headquartered in Seattle. DuckDB is developed by DuckLabs, a bootstrapped (revenue-funded) company in Amsterdam. And the IP of the project is in a third organization: a Dutch non-profit called the DuckDB Foundation. For details, see https://duckdb.org/faq#how-are-duckdb-the-duckdb-foundation-ducklabs-and-motherduck-related https://duckdb.org/faq#how-are-duckdb-the-duckdb-foundation-...
infinet 4mo ago
I use DuckDB and like it. Since many mentioned GB level json in this post, so they have large amount of data. Been column based, DuckDB uses more RAM as row count grows. It can be an advantage or disadvantage depends whether memory is constrained. Traditional row based DB such as SQLite can deal with large database with less memory.

madbo1 4mo ago

[flagged]

orliesaurus 4mo ago

Surprised no one has mentioned Turbopuffer yet [1] which natively supports dense vector similarity and BM25 keyword indexes out of the box [1]. https://turbopuffer.com/ https://turbopuffer.com/

schmookeeg 4mo ago
[dead]

skybrian 4mo ago

Instead of "just use Litestream," I'd like to see a review of different object stores one could use and which ones work well with Litestream. Is there a nice object store I could run in another Linux VM? As a hobbyist, which services providing an S3-like API make the most sense?

chrsstrm 4mo ago
Litestream is just the replication layer, it works with any S3 compatible storage, and all of these in their guide as well https://litestream.io/guides/#replica-guides https://litestream.io/guides/#replica-guides
- skybrian 4mo ago
  No, I know that already. That's not a recommendation. Some storage layers have to be better than others on price, reliability, and so on? Think Wirecutter, not install guide.

mburaksayici 4mo ago

Agreeing on the point, I needed NoSQL version on the similar uses, I've used TinyDB : https://mburaksayici.com/blog/2024/09/21/easy-to-use-nosql-prompt-database-for-small-projects.html https://mburaksayici.com/blog/2024/09/21/easy-to-use-nosql-p...

gunnarmorling 4mo ago

Related piece I wrote some time ago: https://www.morling.dev/blog/building-durable-execution-engine-with-sqlite/ https://www.morling.dev/blog/building-durable-execution-engi...

prmph 4mo ago

> Postgres ... is the right choice when you need higher availability, broader shared scalability, or other deployment properties that are better served by a network database. It is also the better fit when asynchronous replication to object storage is not the durability model you want... Many workflow systems do not need that on day one and should not start with more infrastructure than their state actually demands. ------ I see this kind of YAGNI thinking a lot, but in my view, it must be balanced against the effort you'd put into resolving any edge cases and adapting current architecture to your use case. Imagine you deploy Sqlite, and thought it fine by itself, you keep running into some unforeseen challenges with the use to which you are putting. YOu'd need to sink valuable time and effort into addressing those. Then, when you have outgrown it, you'd beed to spend additional valuable times dping the same with Postgres. This is why, when it comes to Architecture, I increasingly find my myself over-enigneering a bit. Assuming there is a good chance you might need to upgrade your architecture in the not too distant future, that approach is actually kind of very efficient. I find that I am able to uncover a lot of potential gotchas, which feeds back into the what the simplified current architecture should be, and helps me understand the roadmap I'm facing very well. I also avoid wasting too much time going too deep in directions that make sense now, but need a lot of plumbing to get right, when I can see that I'd likely have to throw it all out in a few years. Going from A -> B -C -> D, where each step is the optimal good-enough-for-now architecture but which requires a lot of work to stabilize and iron out the kinks of, is much less efficient than exploring D well enough to know whether you should build A, B, or C now. Basically, some over-engineering, if done right, is not wasted. It cuts right to the heart of what you are dealing with, efficiently, and allows you to make (maybe) simpler but informed choices now as to how best to allocate your development resources now.

narnarpapadaddy 4mo ago
My version of this is the “N+1” principle. Build for one more foreseeable use case than you currently have. The domain model will click in when you need to generalize a solution, and you’ll gain the ability to see your particular solution as one of several to the problem, and thus evaluate fit and tradeoffs more clearly. Don’t do N+2. The goal isn’t to predict the future, nobody can do that. The goal is a durable understanding of the domain and the best fit implementation you can get with that current understanding and resources. That said, SQLite passes that bar for me in most use cases.

Thaxll 4mo ago

I started using SQLite for a home project after years of reading about it, I was shocked at the poor type system coming from Postgres. It is really inferior, not sure why it gets so much praise. https://sqlite.org/datatype3.html https://sqlite.org/datatype3.html https://www.postgresql.org/docs/current/datatype.html https://www.postgresql.org/docs/current/datatype.html Working with date/time feels like using a 30years old database, nothing is enforced at insert. Really someone needs to explain why so many people like it.

RivieraKid 4mo ago
Yes, this is basically my only issue with SQLite. SQLite with a strict type system would be great.
- deleted 4mo ago
  [deleted]
deleted 4mo ago
[deleted]
ThatMedicIsASpy 4mo ago
it's a single file.
zimmi 4mo ago
You can use strict tables: https://sqlite.org/stricttables.html https://sqlite.org/stricttables.html
- pseudalopex 4mo ago
  This could enforce dates are strings. They wanted to enforce dates are dates I thought.
  - zimmi 4mo ago
    Storing dates as INTEGER (year * 10000 + month * 100 + day, e.g. 20260530) is not so bad. Proper date / timestamp types would be great though.

teravor 4mo ago

if you have an application that needs to maintain state in a non-critical section or if you discover that using SQL is actually a good idea for some tasks (even in critical sections), SQLite is not only a good choice but it will save you a lot of time coming up with a brittle custom solution. maintain an in-memory SQLite db and work it with SQL commands, and if you also want to preserve state across application restarts you can routinely save to disk or load from it: <https://www.sqlite.org/backup.html#example_1_loading_and_saving_in_memory_databases https://www.sqlite.org/backup.html#example_1_loading_and_sav...> this also happens to be the most convenient file-format (aka. application-format) I ever worked with.

opiniateddev 4mo ago

[flagged]

PUSH_AX 4mo ago

I went from using the various big player postgres clusters to SQLite, we have an MAU in 7 figures, all backed by SQLite durable objects. We have to think differently about the access patterns but the benefits have been worth it.

nodesocket 4mo ago

The biggest annoyance about SQLite for me is no ability to: ALTER TABLE users MODIFY COLUMN… ALTER TABLE users ALTER COLUMN… ALTER TABLE users ADD CONSTRAINT… You have to create a new temporary table with correct schema, copy data into this new table, drop the old table, and then rename the temporary table.

simonw 4mo ago
They've been improving that recently: 2026-04-09 (3.53.0) - "Enhance ALTER TABLE to permit adding and removing NOT NULL and CHECK constraints" I use my own sqlite-utils CLI/Python library to work around these limitations: https://sqlite-utils.datasette.io/en/stable/python-api.html#python-api-transform https://sqlite-utils.datasette.io/en/stable/python-api.html#...
- nodesocket 4mo ago
  Ahh that's very nice. Unfortunately the default version of sqlite3 provided by Debian Trixie is 3.46.1.

lvl155 4mo ago

And all you need is pen and paper to do calculations.

3dedb728-3f77 4mo ago

Is this just a AWS ads?

fathermarz 4mo ago

Excellent write up and inspired me for our next IA design run. After reading Fly’s Litestream work it makes me think this is a solid option.

flying_sheep 4mo ago

Cloudflare durable object is implemented with SQLite (or some variant of it)

vatsachak 4mo ago

Postgres doesn't cost any extra lol

faangguyindia 4mo ago
it does cost network stack

ai_slop_hater 4mo ago

> if you already trust your database, you do not need a separate orchestration tier Wow. Really? I thought I needed to use an overpriced cloud SaaS for everything.

vultour 4mo ago

The GitHub statistics for the project this website represents are insane. It has a sole author that has averaged approximately 20,000 lines of code every week in the past month. How do you even maintain that alone?

mococa 4mo ago

No, mmap is all you need.

psanford 4mo ago

I wrote a library[0] to let you concurrently update a sqlite db in s3 safely. It uses the little known sqlite sessions extension plus s3 compare-and-swap on a small metadata file to make this work reasonably efficiently and safely. I have been enjoying it for a bunch of small projects where I want a lambda function to have a db for state but I don't want to pay for a full database instance. [0]: https://github.com/psanford/s3db https://github.com/psanford/s3db

password4321 4mo ago
This is exactly what I had dreamed of except without being aware of the session extension... I put it off because I thought I would have to create a layer opening sequential partial sqlite dbs and somehow filtering to the most recent version of each record in-between compactions. It sounds like sqlite sessions handles the hardest part and you've sanded down some of the rough edges while implementing the glue bits for s3 (generously licensed MIT); thanks for the heads-up!

vkaku 4mo ago

Back in the day, I wrote a simple job queue with SQLite https://github.com/guilt/squeue https://github.com/guilt/squeue It did the job, was fairly easy to use.

faangguyindia 4mo ago

I've replaced all of these with Go + SQLite: 1. Intercom 2. Zendesk 3. Email marketing 4. Kanban 5. Todo 6. Our billing stack 7. Our issue tracker 8. Our forum 9. Uptime monitor 10. PagerDuty (clone) I have dozens of products I sell, so I thought: why not build everything ourselves? All of these run on the same server and use very little memory. I replaced all the SaaS tools we used with these. I also moved to dedicated servers and dropped costs to about 1/10th of what we were paying for managed cloud solutions, while maintaining the same HA and even achieving lower latency (partly because noisy neighbors on VPSes were increasing tail latency). We used to spend a ton on this stuff. These have now been in production for four months and have only needed minor updates. Deployment is dirt simple. No Docker, no Kubernetes—just a systemd service and a binary built on the dev machine and deployed. We also used to pay for services like MaxMind and IPData. I ended up hand-rolling my own IP geolocation service, which, in my tests, outperforms most existing solutions. It all started with replacing Uptime Robot. Then I got more confident and replaced PagerDuty. After that, I replaced Intercom. Finally, I had always heard people say, "Don't build your own billing stack." But I said YOLO, let me make that mistake myself. So I studied our existing billing solution, developed my own, and rolled it out. So far, we've had zero issues with it. Caddy in front. I found that we only use maybe 1–5% of the features most SaaS products offer, while the features we actually need keep getting buried deeper and deeper inside these "enterprise-grade" platforms, making our workflows more difficult. I won't show my commercial products because our partners and clients probably wouldn't appreciate knowing how cheap I am—but I call it being resourceful. I can show my free app, though, which has 20,000+ users and was launched recently: https://macrocodex.app/ https://macrocodex.app/ It only uses the Zendesk clone. Email is handled through Cloudflare routing, so we pay almost nothing to run the app.

tonymet 4mo ago
I rolled my own uptime but how do you vary region or do residential testing ?
- faangguyindia 4mo ago
  I actually don't. I just have uptime service hosted outside of our main infra. It connects to my service called Siren, which alerts me on my phone with an alarm on full volume with SWAT cat intro. It's good enough for what we do, barely have any downtime. But it helped me figure out 6s downtime we would experience when our spot instances get knocked out, so it helped me increase health check frequency 6s downtime is a lot when you are getting hammered at 100 RPS.

dannypdx 4mo ago

All this SQLite hate from big vector db... leave SQLite alone!

simplestates 4mo ago

Good framing. SQLite is often enough when the main problem is making workflow state durable, inspectable and easy to recover.

dnnddidiej 4mo ago

Butchering the Beatles song again.

tomasol 4mo ago
sorry about that, I do love Beatles!

emodendroket 4mo ago

SQLite is an underrated tool for how powerful it really is and probably people don't think of it often enough.

sharts 4mo ago

For something so widely deployed you’d think it’d be included in Claude/Codex/etc for

ryanisnan 4mo ago

Sweet I get to tell my team we can move off of dapr workflows

aykutseker 4mo ago

Storage never ended up being the thing we worried about. The painful bits started once a workflow could touch external systems. Replaying state is one thing. Replaying a charge or an email is another. How are you dealing with that?

jkwang 4mo ago

[flagged]

shevy-java 4mo ago

Hmmm. SQLite is great, but I remember years ago, at a university cluster, I had to populate a SQL database via tons of INSERT statements from genomic/meta-genomic workflows. Postgresql was so much faster just at that particular action (inserting data) that it convinced me that SQLite may be useful for many, many applications, but for "big data"(sets), Postgresql is simply better.

dev_l1x_be 4mo ago

I start to think the SQLite is all i need to store data. When there is a chance of non-coordinated writes that I can distribute among servers (or even a range based ID) SQLite is my first idea. With durable storage backups this works amazingly well.

utopiah 4mo ago

The cycle of expertise : - what is X, I just do Y - wow I can see so many limits of Y, now I do X - I use X for literally everything - now that I properly understand the limits of Y but also the heavy constraints of X ... maybe Y is enough - I use Y for literally everything rinse & repeat. The thing is with actual usage and actual context one does learn and thus can get away with a lot more "basic" solution but it does require genuinely understanding the limits.

dmos62 4mo ago
Persistence in folly leads a fool to enlightenment.
- deleted 4mo ago
  [deleted]
stingraycharles 4mo ago
Yup, when I look back at the silly stuff I did when I was somewhere in the middle (CQRS + event sourcing I’m looking at you), it’s interesting. It is a source of expertise, because you really learn a lot from it. But when you become old (43 over here), you really learn to appreciate “boring” solutions.
- utopiah 4mo ago
  Boring is the new sexy. /wisdom
- roryirvine 4mo ago
  You also begin to recognise that the definition of "boring" changes over time, and - if you wait long enough - fashions begin to repeat themselves. So, xBase was all you needed in the mid 80s. Then DBM was all you needed in the mid 2000s. Now, in the mid 2020s, we're told that it's SQLite that is all you need. It was partly true then, and partly true now. But the full story's always been more complicated, so it's still worth considering a range of potential solutions rather than relying on simplistic rules of thumb or slogans. (Wake me when nosql comes back into fashion, I'll be able to do a great "old man yells at clouds" routine about that one...)

halamadrid 4mo ago

Operators of Unmeshed here, which is basically a rewrite of Netflix Conductor. In this orchestrator we heavily use a uniquely scaled version of SQLite and also offers “managed” SQLite instances for managing user data. Combining the durable executions of Unmeshed and along with workflow primitives like sleep, workers, etc you can actually build complex systems with a lot less code than ever. Check it out here: https://unmeshed.io https://unmeshed.io

jkwang 4mo ago

[flagged]

unnouinceput 4mo ago

Quote 1: "DBOS recently argued that Postgres is all you need for durable execution...SQLite is all you need." Quote 2: "SQLite State backed up to S3". Yeah buddy, if you think S3 isn't Postgres I'll eat my foot.

sgt 4mo ago

Next one: Berkeley DB is all you need for durable workflows /s

bob1029 4mo ago

> This is especially attractive for AI agents and AI-generated workflows. Those systems are often bursty, experimental, and easier to reason about when each agent or tenant has a small self-contained unit of state. I am finding that the most important thing is one big, consistent data warehouse that is updated with the state of the business as close to real time as we can get. SQLite is not really great at this particular problem. Something like Postgres or SQL Server would be much more suitable for an OLAP data warehouse that can serve clients (AI agents) while simultaneously merging massive record sets from upstream business systems. These products also offer intricate permissions control. You can prove to an auditor that your AI solution will never see tables or rows it's not supposed to. SQLite doesn't even have a concept of a user, role or login. > The compute can stay cheap and disposable. Again, hosted sql is better aligned. The alternative is DIY hosted sql (SQLite + some other magic) which immediately violates this rule.

freakynit 4mo ago

SQLite backed with Raid-10 NVME disks and periodic backups to cloud storage is generally more than enough to run majority of the production workloads of startups. Writes are single threaded, but, you can still easily do thousands per second. DuckDB offers similar qualities, on the OLAP side. This is not to say this is best combination.. but, when you consider the simplicity of setup, usage, operations, and backups, and cost element, this indeed offers one of the best, if not the best combination.

jackzhuo 4mo ago

100% this. I used to default to Postgres for everything. But seeing SQLite handle concurrency so well now—plus having built-in BM25 search and vector support—it really is all you need for these kinds of architecture.

sungjinwo0 4mo ago

[flagged]

seobot_dk1289 4mo ago

[flagged]

vixalien 4mo ago

Has anyone actually used PGLite[0]? [0]: https://pglite.dev/ https://pglite.dev/

jessmartin 4mo ago

Waiting for “JSONL is all you need for durable workflows.”

Xeoncross 4mo ago

So is https://github.com/obeli-sk/obelisk https://github.com/obeli-sk/obelisk the Rust version of https://github.com/temporalio/temporal https://github.com/temporalio/temporal (Go)? Can you guys add a comparison between them on the site?

tomasol 4mo ago
Subscribe to this issue: https://github.com/obeli-sk/obelisk/issues/65 https://github.com/obeli-sk/obelisk/issues/65

throwaway58670 4mo ago

Holy sticky header... It takes literally half the screen on mobile. Shit like this makes me wonder: do you even look at your website? At least sometimes?

oulipo2 4mo ago

Obelisk also supports Postgres. When you're using it with Postgres, what are the differences with DBOS? Are there any that would be significant (I'm already using Postgres, so I don't really need a sqlite-based durable workflow engine, so looking to know how to choose between DBOS and Obelisk)

tomasol 4mo ago
Check out https://obeli.sk/blog/comparing-dbos-part-1/ https://obeli.sk/blog/comparing-dbos-part-1/

martinloop 4mo ago

[flagged]

bakrisolo 4mo ago

The move toward local-first, durable workflows is honestly a breath of fresh air. I’ve spent the last few months building a document analysis tool and the amount of friction involved in trying to make cloud-based RAG pipelines 'durable' vs just keeping the state in local SQLite is night and day. Once you stop fighting the latency and privacy issues of the cloud you can build much faster.

sperandeo 4mo ago

I concur. I keep waiting to hit the wall where I need Postgres and it just hasn't happened. PB ftw

themuskgpt2025 4mo ago

[dead]

henryz2004 4mo ago

[dead]

sonthonax 4mo ago

The thing that people forget about databases is that they are just really complicated ways of writing to a file. The crux of all these “you only need sqlite” posts isn’t that you need SQLite, it’s that your architecture only ever needed flat appendable log file to begin with. Once you realise that SQLite is just an aggregated view over this log it calls into question if you needed a third party query engine at all, suddenly the whole NoSQL vs SQL debate becomes a meaningless implementation detail. Server based databases aren’t special beyond the fact they’re just distributed synchronisation primitives over files on a file system. But the utility of this is quite limited in practice, writing to a table will always require some sort of synchronisation, using SQLite vs Postgres is just a case of where you put the lock - in your own code or in a 3rd party server.

pdimitar 4mo ago
> Once you realise that SQLite is just an aggregated view over this log it calls into question if you needed a third party query engine at all, suddenly the whole NoSQL vs SQL debate becomes a meaningless implementation detail. You are correct on the premise, however my observation has been that teams who try to be lean and mean and minimize dependencies always start off believing they don't need a query engine... and they always end up needing one, often making things much worse by refusing to just migrate to any database due to sunk cost fallacy. I fully agree with your argument that it's all just file operations in the end. To me SQLite wins for everything beyond persistence however: the SQL commands, extensions, backups, changesets / patchsets. Can you, I, and many others write something that has 10% of SQLite feature set that serves us perfectly? Absolutely! But I don't want to, and in business settings nobody will allow you to hand-roll crucial infrastructure software (like databases) unless they're in full crisis mode and you're the mega-expensive consultant brought in to save the day.

sspoisk 4mo ago

[flagged]