7 ms·

One recurring pattern I have seen at multiple customer sites is that scaling makes the engineers lazy to optimize. One production performance calamity requires

by zevv 2y ago

One recurring pattern I have seen at multiple customer sites is that scaling makes the engineers lazy to optimize. One production performance calamity requires the team to add CPU as a quick fix, and from that time on the baseline for the product's requirement has been set to the new number of CPUs. "Back in the olden days", if your product was slow but the number of CPUs was fixed (or could not be increased instantly), the solution was to go and fix your code. Basic system level skills are now no longer taught or practiced at the appropriate levels, so teams end up without engineers who actually know how to profile and optimize. The cloud providers are the big winners here.

mgkimsal 2y ago
"scaling makes the engineers lazy to optimize" I've lost track of the times I've heard "compute is cheap! engineers are expensive!" Except... that compute cost will live forever. The time it takes someone to debug a bad loop or poor query is at worst a one time cost. Longer term, it may even make other stuff faster in the future.
- Ekaros 2y ago
  Looking at some numbers on cloud, yeah I don't believe in that statement anymore. End device compute might be cheap, but cloud services certainly are not.
  - mgkimsal 2y ago
    We've also seen a rise (or maybe I just notice it more) of stories of "I changed these 3 lines in 5 minutes and saved my company $40k/month!" And then you'll get responses like "pfft.... that's hardly the cost of one part time FAANG person who makes $680k/year - what's the point?" And around we go...
jazz9k 2y ago
Because it's cheaper now to throw hardware at a problem than actually try to fix the code or root of the issue. This wasn't the case a couple of decades ago. It also means less engineers are needed for most companies.
- jjav 2y ago
  > Because it's cheaper now to throw hardware at a problem than actually try to fix the code or root of the issue. It's not cheaper, it's just more opaque. Back when your service was deployed on that 2 CPU box and it was too slow for obvious reasons, you optimized it and then it was good. Today you just shrug and increase that kubernetes cluster from 16 to 48 nodes and forget about it. Costs a lot but the bill shows up somewhere else, in most groups the engineer doesn't even know what it is.
duxup 2y ago
"scaling makes the engineers lazy to optimize" More often I think that's more of an overall engineering department time budgeting / culture issue.
bluGill 2y ago
It isn't engineers that are lazy, it is management that tells the engineers to work on features that make money not performance that will save money. Sometimes management is correct in that decision, sometimes it is "penny wise, pound foolish".
deleted 2y ago
[deleted]

by zevv 2y ago

mgkimsal 2y ago

"scaling makes the engineers lazy to optimize" I've lost track of the times I've heard "compute is cheap! engineers are expensive!" Except... that compute cost will live forever. The time it takes someone to debug a bad loop or poor query is at worst a one time cost. Longer term, it may even make other stuff faster in the future.

Ekaros 2y ago
Looking at some numbers on cloud, yeah I don't believe in that statement anymore. End device compute might be cheap, but cloud services certainly are not.
- mgkimsal 2y ago
  We've also seen a rise (or maybe I just notice it more) of stories of "I changed these 3 lines in 5 minutes and saved my company $40k/month!" And then you'll get responses like "pfft.... that's hardly the cost of one part time FAANG person who makes $680k/year - what's the point?" And around we go...

jazz9k 2y ago

Because it's cheaper now to throw hardware at a problem than actually try to fix the code or root of the issue. This wasn't the case a couple of decades ago. It also means less engineers are needed for most companies.

jjav 2y ago
> Because it's cheaper now to throw hardware at a problem than actually try to fix the code or root of the issue. It's not cheaper, it's just more opaque. Back when your service was deployed on that 2 CPU box and it was too slow for obvious reasons, you optimized it and then it was good. Today you just shrug and increase that kubernetes cluster from 16 to 48 nodes and forget about it. Costs a lot but the bill shows up somewhere else, in most groups the engineer doesn't even know what it is.

duxup 2y ago

"scaling makes the engineers lazy to optimize" More often I think that's more of an overall engineering department time budgeting / culture issue.

bluGill 2y ago

It isn't engineers that are lazy, it is management that tells the engineers to work on features that make money not performance that will save money. Sometimes management is correct in that decision, sometimes it is "penny wise, pound foolish".

deleted 2y ago

[deleted]