6 ms·

Saving months of compute time with a single Grafana query

47 points by serverlessmom 2y ago

fs111 2y ago
[flagged]
- serverlessmom 2y ago
  Slightly longer: having many versions of the same SDK required added to startup time.
mrits 2y ago
Without proper telemetry and performance metrics you will get to do this in a few more months again
BobbyTables2 2y ago
I do not understand how cloud proponents talk about the he costs of self hosting but then get into situations like this. Spending serious engineering time to wrangle with the complexities of cloud orchestration is not something that should be taken lightly. Cloud services should be required to have a black-box Surgeon’s General warning.
- rjmunro 2y ago
  $5k/month was 25% of his pods, so the total was ≈$20k/month. It's entirely possible that self hosting would cost much more than that, particularly as they wouldn't be able to save costs by scaling down.
  - serverlessmom 2y ago
    Yeah. My dad had told me more than one horror story of early tech startups buying truckloads of hardware to scale way beyond demand growth. And I remember when getting on Slashdot meant your service would inevitably go down. Of course, given stable demand and known requirements, bare metal can be a great option. But it’s not strictly better than public cloud hosting. I think it’s just been long enough that people have forgotten the limitations of bare metal engineering.
    - latchkey 2y ago
      > I think it’s just been long enough that people have forgotten the limitations of bare metal engineering. Not just engineering, but deploying it too. I'm building a whole business around deploying it based at least partially on the fact that it has been forgotten. I've just been doing it long enough that I remember how to do it and it isn't getting any easier as the need for compute grows into even more complex and powerful hardware.
  - tnolet 2y ago
    Moreover, we need to have presence in 20+ regions around the world. This multiplies the hassle of self hosting / bare metal / colo. Disclaimer: Checkly founder.
- bravetraveler 2y ago
  'accept vendor lock in, it'll save you the cost of engineers' Routinely: oops, our API usage slipped and we mistakenly paid more than the staff to avoid this would cost Keep fucking up, tech industry. My job role depends on it (SRE)
  - belter 2y ago
    The Tech Industry is well know for shunning the Price of Education, while rejecting the Cost of Ignorance.
    - JackMorgan 2y ago
      Maybe even shunning the reasonable Price of Education while paying the significant Costs of Ignorance.
      - belter 2y ago
        AWS has a three day course on the technicals of efficient cost management. Believe it or not...I heard is one of the least requested classes....
        wisemang 2y ago
        Maybe by the time you need it you can’t afford to pay for it…
  - bluGill 2y ago
    Even in Germany you won't get an engineer for only $5k/month. You can get bad engineers in places like India for that, but good engineers cost more than that.
    - bravetraveler 2y ago
      German companies for all their precision probably make more than one spending mistake, like we do over here. There's asymmetry you aren't considering: one engineer can make/solve many problems. Not even all on either side. To be clear: I'm not taking a dig at this specific case. Larger patterns. It's more nuanced than X of these or Y of those. I'm not arguing in your vacuum/hypothetical There need-not be an increase in staff in a lot of cases. Just better, or different, staff. If I wanted to earnestly dissect/solve this I wouldn't have opened with snide jokes. The engineers hired to stave expense tend to go on to create it The positive spin is: 'get the engineers anyway, self-hosted or not'. They'll help you optimize $solution. My point is things are rarely this positive. Things can be optimized to the point of being non-optimal.
- tamiral 2y ago
  It’s not a set of up and leave it! You have to continuously monitor and improve! Yes using some cloud service will save XYZ time but doesn’t mean it’s a set it and forget it feature. I’ll add this is a really good write up ! Love this comment : “There is no harm in using boring/simple methods if the results are right.”
- hibikir 2y ago
  The best advantage of cloud was never price: It was not having to argue with your data center organization, which often lead to taking months to provision anything, even a very boring VM. If those companies were good at managing data centers, and could hire people actually interested in helping the company run, they'd have had little need for the cloud in predictable compute loads. Until you get quite big, all necessary interactions with the cloud provider are just bills. It's just much easier, even though it is often expensive
  - vb-8448 2y ago
    > which often lead to taking months to provision anything, even a very boring VM It's still true? In my experience it used to be, nowadays most of the organizations have an internal serf-provisioning portal.
    - dartos 2y ago
      Who do you think builds and maintains those portals? Internal data center organizations that need to be argued with to acquire quota. They usually don’t care for internal or dev stuff, but as soon as something needs prod levels of quota or new licenses, prepare to wait.
    - michpoch 2y ago
      > organizations have an internal serf-provisioning portal Which will create things in cloud…
  - silverquiet 2y ago
    Cloud was absolutely sold as saving money once upon a time, but like any other industry, they wanted to push the margins higher by getting away from providing commodity services (cloud primitives in their case), and thus the marketing started saying it was never about saving money or how you're not supposed to use the primitives; you need to use the higher-level, managed services (that also happen to have tons of vendor lock-in). I've spent the last decade or so wondering if the emperor was wearing clothes and not really getting what everyone else has been talking about. Which isn't to say that cloud is useless, but it's not the universal panacea that it was often sold as, and it seems that others are waking up to that.
- candiddevmike 2y ago
  > Spending serious engineering time to wrangle with the complexities of cloud orchestration is not something that should be taken lightly. Bare metal and datacenter orchestration is leaps and bounds more complex. You're paying for the abstraction.
  - latch 2y ago
    If your scale is crazy, or your product doesn't allow you to use battle-tested pieces, then orchestration is complex in both cases. In most cases, managing software on bare metal is more complex in exactly one case: when engineers only know cloud abstractions.
    - rbranson 2y ago
      Yeah calling the EC2 API is definitely more complex than leasing datacenter space, purchasing racks of hardware, deploying a fault tolerant and secure network, capturing and managing offsite backups, dealing with hardware component failures, etc.
      - latch 2y ago
        Serious question, are you familiar with dedicated hosting?
      - trollbridge 2y ago
        If you’re at that scale, there are plenty of other platforms that don’t have vendor lock in and overly complex, proprietary APIs by design.
    - skrtskrt 2y ago
      after working at a cloud provider, I would say: 1. managing compute clusters has gotten a lot easier but managing storage clusters and running good storage products for block & object storage on them is very very very far from a solved problem and quite frankly it sucks and is not fun. 2. planning out buying, installing, upgrading, patching, and retiring hardware, server/hypervisor OSs, takes waaaaaaaaay more engineering management skill and experience than 99.9% of companies have. Plus you probably have to fight for every dollar of investment against a board / investors. Even at the cloud provider we were constantly getting kneecapped by upper management not wanting to spend money on hardware.
- helsinkiandrew 2y ago
  The problem wasn't between the cloud and self hosting - the problem was they had stateful code that didn't scale to thousands of requests for different clients. So they are bringing up new instances every invocation. The same 3s runtime startup cost (and need for more hardware) would happen if they were running their own servers.
  - mgkimsal 2y ago
    Would the actual costs have been less, the same, or more, running on their own hardware? Processes can take longer running on your own hardware, but still have a lower TCO.
    - helsinkiandrew 2y ago
      Possibly - the advantage of cloud is you don't need to provision for your peak load, it scales with usage from a few requests a day, to millions a second for an hour a day. If your usage is lumpy or growing fast then paying 2-3 times more than the cost of your own hardware can be cheaper.
- deleted 2y ago
  [deleted]
roboben 2y ago
Sadly Grafana (cloud) comes at a cost too. Anyone struggles with this horrible active metrics based pricing too? Not only Grafana Cloud but others do it like that too. We moved shitloads to self hosted Thanos. While this comes with its own drawbacks obv, I think it was worth it.
- skrtskrt 2y ago
  you can self host all the Grafana solutions too if you liked them but didn't like the pricing
Ekrekr 2y ago
I really enjoyed this read! One thing that wasn't clear to me, is that if running NPM to install dependencies on pod startup is slow, why not pre build an image with dependencies already installed, and deploy that instead?
- lmz 2y ago
  Surely they weren't running npm at start. It's just that nodejs allows multiple versions of the same module to coexist and all the different version clients have different version dependencies which could be collapsed to one common version.
- mavidser 2y ago
  > if running NPM to install dependencies on pod startup is slow Loading the AWS SDK via `require` was slow, not installing. As sibling comment says - collapsing different SDKs into one helped reduce loading times of the many SDKs.
zug_zug 2y ago
I'm really surprised that 300ms at startup would result in 25% fewer pods.... What % reduction in the total startup time is that? Is it possible the prior measurement happened during a high traffic period and the post measurement happened in a low traffic period?
- serverlessmom 2y ago
  It’s a 50% reduction in startup time, and each “run” for a pod is fairly quick.
throwthrow5643 2y ago
The 'one weird trick' could've been spotted in a graphical bundle analyser. But are they not caching npm packages somewhere, seems like an awful waste downloading from the npm registry over and over? I would think it would be parsing four different versions of the AWS sdk that was so slow.
- candiddevmike 2y ago
  > seems like an awful waste downloading from the npm registry over and over Pondering this question across every organization in the world and the countless opportunities for caching leads to dark places. Would be interesting to see CDN usage for Linux distributions pre and post docker builds becoming popular.
sebstefan 2y ago
I really don't understand spinning up a whole pod just for a request Wouldn't it be cheaper to just keep a pod up with a service running? If scaleability is an issue just plop a load balancer in front of it and scale them up with load but surely you can't need a whole pod for every single one of those millions of requests right? > Checkly is a synthetic monitoring tool that lets teams monitor their API’s and sites continually, and find problems faster. >With some users sending *millions of request a day*, that 300ms added up to massive overall compute savings No shit, right?
- crummy 2y ago
  The article said they had to do a bunch of cleanup between requests when it was handled by one service. Which surprised me but these requests must be doing more than just HTTP requests I guess.
  - c0brac0bra 2y ago
    Yea they do E2E checks with playwright as well, among other things. A bunch of stuff could get cached from those checks I suppose, especially if it's user-written code.
deleted 2y ago
[deleted]
dxbydt 2y ago
many of the tricks we learned in the late 90s - 2000s can no longer be pulled off. We used to download jar files over the net. Running a major prop trading platform meant 1000s of dependencies. You’d have swing and friends for front end tables, sax xml parsers, various numerical libraries, logging modules- all of this shit downloaded in the jar when the customer impatiently waited to trade some 100MM worth of fx. We learned how to cut down on dependencies. Built tools to massively compress class files. Tradeoff 1 jar with lots of little jars that downloaded on demand. Better yet, cache most of these jars so they wouldn’t need to download every single time. It became a fine art at one point - the difference between a rookie and a professional was that the latter could not just write a spiffy java frontend, but actually deploy it in prod so customers wouldn’t even know there was a startup time - it would just start like instantly. then that whole industry just vanished overnight- poof! now i write ml code and deploy it on a docker in gcp and the same issues all over again. you import pandas gbq and pretty much the entire google bq set of libraries is part of the build. throw in a few stadard ml libs and soon you are looking at upwards of 2 seconds in Cloud Run startup time. You pay premium for autoscaling, for keeping one instance warm at all times, for your monitoring and metrics, on and on. i am yet to see startup times below 500ms. you can slice the cake any which way, you still pay the startup cost penalty. quite sad.

47 points by serverlessmom 2y ago

fs111 2y ago

[flagged]

serverlessmom 2y ago
Slightly longer: having many versions of the same SDK required added to startup time.

mrits 2y ago

Without proper telemetry and performance metrics you will get to do this in a few more months again

BobbyTables2 2y ago

I do not understand how cloud proponents talk about the he costs of self hosting but then get into situations like this. Spending serious engineering time to wrangle with the complexities of cloud orchestration is not something that should be taken lightly. Cloud services should be required to have a black-box Surgeon’s General warning.

rjmunro 2y ago
$5k/month was 25% of his pods, so the total was ≈$20k/month. It's entirely possible that self hosting would cost much more than that, particularly as they wouldn't be able to save costs by scaling down.
- serverlessmom 2y ago
  Yeah. My dad had told me more than one horror story of early tech startups buying truckloads of hardware to scale way beyond demand growth. And I remember when getting on Slashdot meant your service would inevitably go down. Of course, given stable demand and known requirements, bare metal can be a great option. But it’s not strictly better than public cloud hosting. I think it’s just been long enough that people have forgotten the limitations of bare metal engineering.
  - latchkey 2y ago
    > I think it’s just been long enough that people have forgotten the limitations of bare metal engineering. Not just engineering, but deploying it too. I'm building a whole business around deploying it based at least partially on the fact that it has been forgotten. I've just been doing it long enough that I remember how to do it and it isn't getting any easier as the need for compute grows into even more complex and powerful hardware.
- tnolet 2y ago
  Moreover, we need to have presence in 20+ regions around the world. This multiplies the hassle of self hosting / bare metal / colo. Disclaimer: Checkly founder.
bravetraveler 2y ago
'accept vendor lock in, it'll save you the cost of engineers' Routinely: oops, our API usage slipped and we mistakenly paid more than the staff to avoid this would cost Keep fucking up, tech industry. My job role depends on it (SRE)
- belter 2y ago
  The Tech Industry is well know for shunning the Price of Education, while rejecting the Cost of Ignorance.
  - JackMorgan 2y ago
    Maybe even shunning the reasonable Price of Education while paying the significant Costs of Ignorance.
    - belter 2y ago
      AWS has a three day course on the technicals of efficient cost management. Believe it or not...I heard is one of the least requested classes....
      - wisemang 2y ago
        Maybe by the time you need it you can’t afford to pay for it…
- bluGill 2y ago
  Even in Germany you won't get an engineer for only $5k/month. You can get bad engineers in places like India for that, but good engineers cost more than that.
  - bravetraveler 2y ago
    German companies for all their precision probably make more than one spending mistake, like we do over here. There's asymmetry you aren't considering: one engineer can make/solve many problems. Not even all on either side. To be clear: I'm not taking a dig at this specific case. Larger patterns. It's more nuanced than X of these or Y of those. I'm not arguing in your vacuum/hypothetical There need-not be an increase in staff in a lot of cases. Just better, or different, staff. If I wanted to earnestly dissect/solve this I wouldn't have opened with snide jokes. The engineers hired to stave expense tend to go on to create it The positive spin is: 'get the engineers anyway, self-hosted or not'. They'll help you optimize $solution. My point is things are rarely this positive. Things can be optimized to the point of being non-optimal.
tamiral 2y ago
It’s not a set of up and leave it! You have to continuously monitor and improve! Yes using some cloud service will save XYZ time but doesn’t mean it’s a set it and forget it feature. I’ll add this is a really good write up ! Love this comment : “There is no harm in using boring/simple methods if the results are right.”
hibikir 2y ago
The best advantage of cloud was never price: It was not having to argue with your data center organization, which often lead to taking months to provision anything, even a very boring VM. If those companies were good at managing data centers, and could hire people actually interested in helping the company run, they'd have had little need for the cloud in predictable compute loads. Until you get quite big, all necessary interactions with the cloud provider are just bills. It's just much easier, even though it is often expensive
- vb-8448 2y ago
  > which often lead to taking months to provision anything, even a very boring VM It's still true? In my experience it used to be, nowadays most of the organizations have an internal serf-provisioning portal.
  - dartos 2y ago
    Who do you think builds and maintains those portals? Internal data center organizations that need to be argued with to acquire quota. They usually don’t care for internal or dev stuff, but as soon as something needs prod levels of quota or new licenses, prepare to wait.
  - michpoch 2y ago
    > organizations have an internal serf-provisioning portal Which will create things in cloud…
- silverquiet 2y ago
  Cloud was absolutely sold as saving money once upon a time, but like any other industry, they wanted to push the margins higher by getting away from providing commodity services (cloud primitives in their case), and thus the marketing started saying it was never about saving money or how you're not supposed to use the primitives; you need to use the higher-level, managed services (that also happen to have tons of vendor lock-in). I've spent the last decade or so wondering if the emperor was wearing clothes and not really getting what everyone else has been talking about. Which isn't to say that cloud is useless, but it's not the universal panacea that it was often sold as, and it seems that others are waking up to that.
candiddevmike 2y ago
> Spending serious engineering time to wrangle with the complexities of cloud orchestration is not something that should be taken lightly. Bare metal and datacenter orchestration is leaps and bounds more complex. You're paying for the abstraction.
- latch 2y ago
  If your scale is crazy, or your product doesn't allow you to use battle-tested pieces, then orchestration is complex in both cases. In most cases, managing software on bare metal is more complex in exactly one case: when engineers only know cloud abstractions.
  - rbranson 2y ago
    Yeah calling the EC2 API is definitely more complex than leasing datacenter space, purchasing racks of hardware, deploying a fault tolerant and secure network, capturing and managing offsite backups, dealing with hardware component failures, etc.
    - latch 2y ago
      Serious question, are you familiar with dedicated hosting?
    - trollbridge 2y ago
      If you’re at that scale, there are plenty of other platforms that don’t have vendor lock in and overly complex, proprietary APIs by design.
  - skrtskrt 2y ago
    after working at a cloud provider, I would say: 1. managing compute clusters has gotten a lot easier but managing storage clusters and running good storage products for block & object storage on them is very very very far from a solved problem and quite frankly it sucks and is not fun. 2. planning out buying, installing, upgrading, patching, and retiring hardware, server/hypervisor OSs, takes waaaaaaaaay more engineering management skill and experience than 99.9% of companies have. Plus you probably have to fight for every dollar of investment against a board / investors. Even at the cloud provider we were constantly getting kneecapped by upper management not wanting to spend money on hardware.
helsinkiandrew 2y ago
The problem wasn't between the cloud and self hosting - the problem was they had stateful code that didn't scale to thousands of requests for different clients. So they are bringing up new instances every invocation. The same 3s runtime startup cost (and need for more hardware) would happen if they were running their own servers.
- mgkimsal 2y ago
  Would the actual costs have been less, the same, or more, running on their own hardware? Processes can take longer running on your own hardware, but still have a lower TCO.
  - helsinkiandrew 2y ago
    Possibly - the advantage of cloud is you don't need to provision for your peak load, it scales with usage from a few requests a day, to millions a second for an hour a day. If your usage is lumpy or growing fast then paying 2-3 times more than the cost of your own hardware can be cheaper.
deleted 2y ago
[deleted]

roboben 2y ago

Sadly Grafana (cloud) comes at a cost too. Anyone struggles with this horrible active metrics based pricing too? Not only Grafana Cloud but others do it like that too. We moved shitloads to self hosted Thanos. While this comes with its own drawbacks obv, I think it was worth it.

skrtskrt 2y ago
you can self host all the Grafana solutions too if you liked them but didn't like the pricing

Ekrekr 2y ago

I really enjoyed this read! One thing that wasn't clear to me, is that if running NPM to install dependencies on pod startup is slow, why not pre build an image with dependencies already installed, and deploy that instead?

lmz 2y ago
Surely they weren't running npm at start. It's just that nodejs allows multiple versions of the same module to coexist and all the different version clients have different version dependencies which could be collapsed to one common version.
mavidser 2y ago
> if running NPM to install dependencies on pod startup is slow Loading the AWS SDK via `require` was slow, not installing. As sibling comment says - collapsing different SDKs into one helped reduce loading times of the many SDKs.

zug_zug 2y ago

I'm really surprised that 300ms at startup would result in 25% fewer pods.... What % reduction in the total startup time is that? Is it possible the prior measurement happened during a high traffic period and the post measurement happened in a low traffic period?

serverlessmom 2y ago
It’s a 50% reduction in startup time, and each “run” for a pod is fairly quick.

throwthrow5643 2y ago

The 'one weird trick' could've been spotted in a graphical bundle analyser. But are they not caching npm packages somewhere, seems like an awful waste downloading from the npm registry over and over? I would think it would be parsing four different versions of the AWS sdk that was so slow.

candiddevmike 2y ago
> seems like an awful waste downloading from the npm registry over and over Pondering this question across every organization in the world and the countless opportunities for caching leads to dark places. Would be interesting to see CDN usage for Linux distributions pre and post docker builds becoming popular.

sebstefan 2y ago

I really don't understand spinning up a whole pod just for a request Wouldn't it be cheaper to just keep a pod up with a service running? If scaleability is an issue just plop a load balancer in front of it and scale them up with load but surely you can't need a whole pod for every single one of those millions of requests right? > Checkly is a synthetic monitoring tool that lets teams monitor their API’s and sites continually, and find problems faster. >With some users sending *millions of request a day*, that 300ms added up to massive overall compute savings No shit, right?

crummy 2y ago
The article said they had to do a bunch of cleanup between requests when it was handled by one service. Which surprised me but these requests must be doing more than just HTTP requests I guess.
- c0brac0bra 2y ago
  Yea they do E2E checks with playwright as well, among other things. A bunch of stuff could get cached from those checks I suppose, especially if it's user-written code.

deleted 2y ago

[deleted]

dxbydt 2y ago

many of the tricks we learned in the late 90s - 2000s can no longer be pulled off. We used to download jar files over the net. Running a major prop trading platform meant 1000s of dependencies. You’d have swing and friends for front end tables, sax xml parsers, various numerical libraries, logging modules- all of this shit downloaded in the jar when the customer impatiently waited to trade some 100MM worth of fx. We learned how to cut down on dependencies. Built tools to massively compress class files. Tradeoff 1 jar with lots of little jars that downloaded on demand. Better yet, cache most of these jars so they wouldn’t need to download every single time. It became a fine art at one point - the difference between a rookie and a professional was that the latter could not just write a spiffy java frontend, but actually deploy it in prod so customers wouldn’t even know there was a startup time - it would just start like instantly. then that whole industry just vanished overnight- poof! now i write ml code and deploy it on a docker in gcp and the same issues all over again. you import pandas gbq and pretty much the entire google bq set of libraries is part of the build. throw in a few stadard ml libs and soon you are looking at upwards of 2 seconds in Cloud Run startup time. You pay premium for autoscaling, for keeping one instance warm at all times, for your monitoring and metrics, on and on. i am yet to see startup times below 500ms. you can slice the cake any which way, you still pay the startup cost penalty. quite sad.