6 ms·

I run a large on-prem temporal setup - throwaway acct as they will likely out me. Temporal is, in my opinion having run it in prod for over a year - poorly des

I run a large on-prem temporal setup - throwaway acct as they will likely out me. Temporal is, in my opinion having run it in prod for over a year - poorly designed, slow and ridicliously heavy infra wise. If you're doing anything non-trivial (say, 200+ events/workflow) and you need to run only a couple hundred of them concurrently all day, you're going to spend millions on infra, and it's still going to absolutely suck. Try running their own benchmarks, the numbers are pathetic. Their sales team is also absolutely appalling and desperate. From a Developer standpoint, the SDK is quite nice though. Don't get trapped into nexus, and if the sales team call you make sure legal is in the room.

temporal_thr123 4mo ago
Since I'm in a ranting mode -- here's a good example: you're limited to _ONE_ IO per shard in the history service: https://github.com/temporalio/temporal/blob/e22e6304b3c4a409b3f5c66ab0f3385d9e610270/service/history/shard/context_impl.go#L2105 https://github.com/temporalio/temporal/blob/e22e6304b3c4a409... https://github.com/temporalio/temporal/blob/e22e6304b3c4a409b3f5c66ab0f3385d9e610270/common/dynamicconfig/constants.go#L1802 https://github.com/temporalio/temporal/blob/e22e6304b3c4a409... Temporal does a crazy amount of database operations and all of these are behind that mutex. Oh, and you can't change the shard count on existing clusters. Great stuff.
dakiol 4mo ago
Agree. Have worked in a codebase using Temporal, and is pretty much a nightmare. I don't know about the infra side, but from the developer side, all the abstractions they bring to the table are poorly designed. Wouldn't recommend
- jiggunjer 4mo ago
  Biggest design bug imo is the workers need to register for the workflows they support, but will happily pull tasks from unrelated workflows if they're on the same queue. No way to put failed tasks back into the queue again either.
Linell 4mo ago
[dead]
lll-o-lll 4mo ago
> If you're doing anything non-trivial (say, 200+ events/workflow) and you need to run only a couple hundred of them concurrently all day, you're going to spend millions on infra, and it's still going to absolutely suck. Where are the “millions” on infra going? It’s a handful of services and a Postgres? > Their sales team is also absolutely appalling and desperate. You said “on-prem”. It’s open source; why are you dealing with their sales team? > If you're doing anything non-trivial (say, 200+ events/workflow) and you need to run only a couple hundred of them concurrently all day… If “millions” were required to obtain such tiny scale, I’d agree there’d be a massive problem. No one would use Temporal; it would be a complete waste of resource. If this were true.
- turtlebits 4mo ago
  The same with any "open-source" enterprise ($$$) software. It sucks to run yourself. Docs on running/errors are non-existent. Their helm charts are broken. Instead of degraded performance, it just fails.
  - lll-o-lll 4mo ago
    With all due respect – if that’s the attitude, you have no business running anything on-prem. And that’s fine, there’s a reason the various cloud providers are the go-to for many businesses.
    - turtlebits 4mo ago
      It's not an attitude, it's an opinion that comes from experience. Operational burden/overhead is a real thing. Just like knowing that German cars will cost $$$ in maintenance. It doesn't mean I shouldn't drive.
  - cyberpunk 4mo ago
    Yeah, they've had so much VC cash pumped in lately they really need to pump the SAAS side of the business.
- cyberpunk 4mo ago
quacker 4mo ago
Honest question: Can you use Temporal Cloud? Have you evaluated Temporal Cloud pricing? Ballparking: 200 events/workflow, 200 workflows/per day and assuming 1 event = 1 cloud action[1], that is 1.2M or so actions per month. The $100/month plan includes 1M actions each month, and even the pay-as-you pricing when you exceed that is $50 per 1M actions[2]. Temporal Cloud seems extremely cheap for your use case, even if I'm off by a factor of 10. Is there a catch? You still need infra to run your Temporal workers, and I assume there are storage and other costs, but I assume action usage is the majority of it. 1. Not sure exactly what constitutes an "Action". At a glance, seems like most events have a corresponding action(?) and a subset of those actions are actually billable(?) 2. https://docs.temporal.io/cloud/pricing#payg-action-pricing https://docs.temporal.io/cloud/pricing#payg-action-pricing
- temporal_thr321 4mo ago
  I was not clear; I did not mean not 200 a day, it's 10s of thousands of concurrently running workflows, sometimes into the hundreds of thousands, each with 200 events. We run many hundreds of thousands of these a day. Temporal was a bad fit for us, and we regret it deeply.
  - mikert89 4mo ago
    what would you use instead?
  - quacker 4mo ago
    Ah. So multiple billion actions per month, and probably multiple million dollars per year on their cloud, if they can even support that load (plus, the vendor lock in and etc). Makes sense.
    - gaurdro 3mo ago
      I'm reasonably sure their cloud product can handle that. you might be in the scale where they want to hold your hand in the transition though. Given the open source nature of the product, I'm not sure what vendor lock-in you mean. Migrating off cloud to your own infra again is doable. https://daylight.ai/blog/how-we-migrated-off-temporal-cloud-without-downtime https://daylight.ai/blog/how-we-migrated-off-temporal-cloud-...
llmslave 4mo ago
I think critical parts of openai run on temporal
potamic 4mo ago
> if the sales team call you make sure legal is in the room. What's the deal? It couldn't harm just listening to sales, could it? I presume legal would it be involved before anything is signed in any case?

temporal_thr123 4mo ago

Since I'm in a ranting mode -- here's a good example: you're limited to _ONE_ IO per shard in the history service: https://github.com/temporalio/temporal/blob/e22e6304b3c4a409b3f5c66ab0f3385d9e610270/service/history/shard/context_impl.go#L2105 https://github.com/temporalio/temporal/blob/e22e6304b3c4a409... https://github.com/temporalio/temporal/blob/e22e6304b3c4a409b3f5c66ab0f3385d9e610270/common/dynamicconfig/constants.go#L1802 https://github.com/temporalio/temporal/blob/e22e6304b3c4a409... Temporal does a crazy amount of database operations and all of these are behind that mutex. Oh, and you can't change the shard count on existing clusters. Great stuff.

dakiol 4mo ago

Agree. Have worked in a codebase using Temporal, and is pretty much a nightmare. I don't know about the infra side, but from the developer side, all the abstractions they bring to the table are poorly designed. Wouldn't recommend

jiggunjer 4mo ago
Biggest design bug imo is the workers need to register for the workflows they support, but will happily pull tasks from unrelated workflows if they're on the same queue. No way to put failed tasks back into the queue again either.

Linell 4mo ago

[dead]

lll-o-lll 4mo ago

> If you're doing anything non-trivial (say, 200+ events/workflow) and you need to run only a couple hundred of them concurrently all day, you're going to spend millions on infra, and it's still going to absolutely suck. Where are the “millions” on infra going? It’s a handful of services and a Postgres? > Their sales team is also absolutely appalling and desperate. You said “on-prem”. It’s open source; why are you dealing with their sales team? > If you're doing anything non-trivial (say, 200+ events/workflow) and you need to run only a couple hundred of them concurrently all day… If “millions” were required to obtain such tiny scale, I’d agree there’d be a massive problem. No one would use Temporal; it would be a complete waste of resource. If this were true.

turtlebits 4mo ago
The same with any "open-source" enterprise ($$$) software. It sucks to run yourself. Docs on running/errors are non-existent. Their helm charts are broken. Instead of degraded performance, it just fails.
- lll-o-lll 4mo ago
  With all due respect – if that’s the attitude, you have no business running anything on-prem. And that’s fine, there’s a reason the various cloud providers are the go-to for many businesses.
  - turtlebits 4mo ago
    It's not an attitude, it's an opinion that comes from experience. Operational burden/overhead is a real thing. Just like knowing that German cars will cost $$$ in maintenance. It doesn't mean I shouldn't drive.
- cyberpunk 4mo ago
  Yeah, they've had so much VC cash pumped in lately they really need to pump the SAAS side of the business.
cyberpunk 4mo ago

quacker 4mo ago

Honest question: Can you use Temporal Cloud? Have you evaluated Temporal Cloud pricing? Ballparking: 200 events/workflow, 200 workflows/per day and assuming 1 event = 1 cloud action[1], that is 1.2M or so actions per month. The $100/month plan includes 1M actions each month, and even the pay-as-you pricing when you exceed that is $50 per 1M actions[2]. Temporal Cloud seems extremely cheap for your use case, even if I'm off by a factor of 10. Is there a catch? You still need infra to run your Temporal workers, and I assume there are storage and other costs, but I assume action usage is the majority of it. 1. Not sure exactly what constitutes an "Action". At a glance, seems like most events have a corresponding action(?) and a subset of those actions are actually billable(?) 2. https://docs.temporal.io/cloud/pricing#payg-action-pricing https://docs.temporal.io/cloud/pricing#payg-action-pricing

temporal_thr321 4mo ago
I was not clear; I did not mean not 200 a day, it's 10s of thousands of concurrently running workflows, sometimes into the hundreds of thousands, each with 200 events. We run many hundreds of thousands of these a day. Temporal was a bad fit for us, and we regret it deeply.
- mikert89 4mo ago
  what would you use instead?
- quacker 4mo ago
  Ah. So multiple billion actions per month, and probably multiple million dollars per year on their cloud, if they can even support that load (plus, the vendor lock in and etc). Makes sense.
  - gaurdro 3mo ago
    I'm reasonably sure their cloud product can handle that. you might be in the scale where they want to hold your hand in the transition though. Given the open source nature of the product, I'm not sure what vendor lock-in you mean. Migrating off cloud to your own infra again is doable. https://daylight.ai/blog/how-we-migrated-off-temporal-cloud-without-downtime https://daylight.ai/blog/how-we-migrated-off-temporal-cloud-...

llmslave 4mo ago

I think critical parts of openai run on temporal

potamic 4mo ago

> if the sales team call you make sure legal is in the room. What's the deal? It couldn't harm just listening to sales, could it? I presume legal would it be involved before anything is signed in any case?