5 ms·

Can anyone explain why I would want to use this over an orchestration tool that lives outside the DB? Read through the Readme and some of the examples, I still

• jpalomaki 3mo ago
  It’s sometimes convenient if database is the only ”stateful” component in architecture. Also if all the "state" is in one database, then you have better chance of getting consistent backups.
• thibaut_barrere 3mo ago
  You can have well-integrated applicative workflows (eg: progress report on a permalink in your front end app), app-restart-proof resumable workflows, and it avoids adding an extra piece of infrastructure. We use Postgres for that on https://transport.data.gouv.fr https://transport.data.gouv.fr (Elixir app which does a fair bit of processing), and it helps. Not familiar yet with pg_durable though, but I have used or implemented similar solutions and can relate.
• rswail 3mo ago
  Snapshot PITR of your database means everything restores including the durable jobs at the PIT. Don't need to synchronize the backups with anything else that is part of the same data store, good for ETL pipelines and other state machine type jobs. If your ETL is mostly SQL anyway, then having the actual job being run on the same server helps as well.
  • regularfry 3mo ago
    Yes, but that doesn't have to imply that the compute part of the durable jobs framework also needs to be part of the database snapshot. You almost certainly want that defined in code anyway, if only to have a sane versioning story. So then by having it also be part of the snapshot, you've now got the problem that there are apparently two sources of truth for that bit of the code.
• gdecandia 3mo ago
  Contributor here. At Microsoft, our Postgres customers seem to split pretty evenly into 2 camps, those that want to do as much as they can in the database, and those that agree with your take - want to keep apps and compute outside the DB.
  • guhidalg 3mo ago
    I bet this is correlated with how much they like/know Postgres already. When people don’t understand their database’s features, they want it to behave like something else they do understand (code). They’re leaving a lot of performance on the table by not leveraging everything their database can do.
    • sgarland 3mo ago
      Yup. Anymore, “we’ll handle that in code” reads to me as “I don’t understand my tools, and don’t want to learn.” Or hubris. The sheer number of times I’ve seen data integrity errors because someone didn’t think they needed a database-level constraint is too damn high. The other one (also related) is normalization. They’ll have hundreds of millions of rows of duplicated, low-cardinality strings, because “joins are expensive,” while somehow missing the fact that the increased I/O from reduced page-packing also has a performance cost.
      • Kaliboy 3mo ago
        Yeah but the increased I/O is cheaper. It's easier to add another webserver as opposed to upgrading your db server. And I don't think it's as simple as you make it. So where I work we use amongst other things Rails. There are places in our codebase where using joins just isn't feasable cause the database would use too much memory and let's just say N is large. But since we use Rails we can have it query the tables apart and join the tables through the defined model. We literally save like 20 seconds in some cases because 1 HUGE query becomes 8 straightforward ones with maximum index usage. And because we have this capability in Rails we would never use something like this, cause that would neccesate us holding two mental models and have a clear "what do we run where" directive which honestly is a PTA.
• keynha 3mo ago
  [dead]
• hmaxdml 3mo ago
  Because you likely already have a database and likely don't need to bring on an entire new distributed system to orchestrate your workflows.