9 ms·

Cloud has always been more expensive. I remember being quoted 250k/month for bandwidth when I was paying 15k with rackspace 10+ years ago. You’re paying for con

Cloud has always been more expensive. I remember being quoted 250k/month for bandwidth when I was paying 15k with rackspace 10+ years ago. You’re paying for convenience and speed. The Math stops working when you grow to a certain point. You can mitigate this to some extent by making some key architecture + vendor decisions upfront when first building… or just consider that some day you’ll need to do things like this. It’s not a novel problem.

jstummbillig 1y ago
I find it intuitively absolutely bizarre that Cloud does not outright win at any scale. In my mind everything about it seems more optimizable with more scale. Obviously I am missing something, but all Cloud pricing looks so significantly more expensive than I feel it should in a healthy and mature market.
vidarh 1y ago
It's horrifyingly hard to convince people of this, though, even you can present them with actual numbers. A lot of people have convinced themselves that cloud is cheap, to the point that they don't even do a cursory investigation. A lot of those even don't do the bare minimum to reduce hosting costs within the cloud they choose, or choose one of the cheaper clouds (AWS is absolutely extortionate for anything that requires significant amount of outbound bandwith), or put caching/CDN's in front (you can trivially slash your AWS egress costs dramatically). Most of my consultancy work is on driving cost efficiencies for cloud, and I can usually safely guarantee the fee will pay for itself within months because people don't fix even the most low hanging fruit.
- ksec 1y ago
  Partly because AWS give out a lot of free credit for start ups, and basically allow them to grow without planning any infrastructure. VCs who are invested into Amazon also wants to push the cloud narrative. Starts up who dont want to deal with servers, want massive scale when they think the website and later an app went viral. That was in the late 00s and early 10s. PHP, Python, Ruby and even Java were slow. Every single language and framework has had massive performance improvements in the past 15 to 20 years. Anywhere from Java 2x to Ruby 3 - 10x. When a server max out at 6 - 8 with Xeon core, compare to today at 192 Core. Every Core is at leats 2 - 3x faster per clock, with higher clock speed we are talking about 100x difference. Especially when IO used to be on HDD, SSD is easily 1000x faster. What used to wait for I/O is no longer an issue, the aggregate difference when all things added together including software could be 300x to 500x. What you would need 500 2U server in 2010, you could now do it in one. Modern web developers are so abstracted with hardware I dont think many realise what sort of difference in hardware improvements. I remember someone posted before 2016 Basecamp had dozens of Racks before moving to cloud. Now they have grown a lot bigger with Hey and they are only doing it with 8 racks and room to spare. AWS on the other hand is trying to move more workload to ARM Graviton where they have a cost advantage. Given Amazon's stock price are now dependent on AWS, I dont think they will lower their price by much in the future. And we desperately need some competition in that area.
- mbreese 1y ago
  I think one business argument for cloud is capital expenses vs operational expenses. If you’re (over) paying for cloud resources vs an in house option (or colo), those are numbers that are a straight expense. When you own hardware, those are on your books until they depreciate off. For some businesses, that can make sense. Now, a good accountant probably wouldn’t care one way or the other. Debits and credits balance either way. And spending more still means less profit in the long term, no matter how it looks on the books. But, in addition to the flexibility, that was what I always thought of as the main cloud benefit. It’s the same with leasing vs buying cars/computers/etc…
  - vidarh 1y ago
    But that too is based on people not knowing the alternatives, as renting managed servers can be close to a wash vs. leasing hardware for a colo (often to the point that relatively cost of land near your preferred managed hosting providers vs. colos that work with access to staff etc. might be what makes one or the other cheaper). Buying outright can be cheaper but isn't necessary. None of the colo'd setups I've worked on bar one used purchased servers - it's all been leased. But the majority of non-cloud workloads I've worked on have not even been leased, but rented.
  - tialaramex 1y ago
    Cloud made sense for the startup I worked for previously. If you are a startup then a $1M per year expense makes much more sense than a $5M up front purchase with 5-10 years of life - in five years you might be billionaires or you might be bankrupt and until then the Cloud was better.
    - vidarh 1y ago
      Or you could have rented or leased-to-own. There's hardly ever any need to actually purchase outright to get prices far below equivalent capacity in clouds. In fact, in 25 years, only one of the colo'd server setups I've worked on had any hardware purchased up-front in it.
    - dilyevsky 1y ago
      Like sibling is saying it's not $5M upfront but on the order of 1 year cloud spend for large enough accounts. There are also such things as leases and loans. One justifiable excuse is you simply don't know how much hardware you will need to buy if you're hitting hockeystick growth. That until you realize you can also go hybrid...
- tialaramex 1y ago
  Periodically management says we shouldn't have a DC, just put everything in the cloud. OK says HPC, here's the quote for replacing one of the (currently three) supercomputers with a cloud service. Oh dear, that's bigger than your entire IT budget isn't it? So I guess we do need the DC for housing the supercomputers. If we'd done that once I'd feel like well management weren't to know, but it recurs with about a 3-5 year periodicity. The perception seems to be "Cloud exists, therefore it must be cheaper, because if it wasn't cheaper why would it exist?" which reminds me of how people persuade themselves the $50 "genuine Apple" part must be better because if it wasn't better than this $15 part why would Apple charge $50 for it? Because you are a sucker is why.
  - vidarh 1y ago
    Yeah, I used to be asked to price out a move to AWS every year at one position. After several years Hetzner finally got cheaper than operating our own colo's, but only basically because we were in London and London real-estate is expensive, and so colo space is accordingly expensive, while Hetzner's DC space is dirt cheap. AWS, however, remained 2x-3x as expensive, with the devops time factored in. > The perception seems to be "Cloud exists, therefore it must be cheaper, because if it wasn't cheaper why would it exist? People are also blithely unaware that large customers get significant discounts, and so I regularly has to explain that BigCo X being hosted in AWS means at most that it is cost-effective for them because their spend means they're getting a significant discount over the already highest volume published pricing, and my clients usually are nowhere close to spend enough to be able to get those discounts.
  - koliber 1y ago
    Regarding apple parts, I recently replaced a broken screen on a MacBook pro with an OEM part. I can’t get the color to look right. Not to mention the one vertical row where pixels look off (not dead, but not normal either). The guy at the shop said I would not notice. I am now kicking myself for not going with the real thing.
  - paulcole 1y ago
    [flagged]
  - 1y ago
- diggan 1y ago
  > A lot of people have convinced themselves that cloud is cheap I've noticed this too, freelancing/consulting around in companies. I'm not sure where this idea even comes from, because when cloud first started making the news, the reasoning went something like "We're OK paying more since it's flexible, so we can scale up/down quickly", and that made sense. But somehow today a bunch of people (even engineers) are under the belief that cloud somehow is cheaper than the alternatives. That never made sense to me, even when you take into account hiring people specifically for running the infrastructure, unless you're a one-person team or have to aggressively scale up/down during a normal day.
  - vidarh 1y ago
    I think it's because people think their workloads are extremely spiky, and so assume they will spin up/down loads enough to save money, and that has translated into cloud being perceived as cheap. But devs rarely pay attention to metrics. I've had clients with expensive Datadog setups where it was blatantly obvious that nobody had ever dug into the performance data, because if they did they'd have noticed that key metrics were simply not fed to it. If they did pay attention, most of them would realise that their autoscaling rarely kicks in all that much, if at all. Often because it's poorly tuned, but also because most businesses see small enough daily cycles. Factor in that the cost difference between instances vs. managed servers is quite significant, and you need to have significant spikes much shorter in duration than most businesses day/night variation to save money. It can make sense to be able to spin up more capacity quickly, but then people need to consider that 1) a lot of managed hosting providers has hardware standing by and can automatically provision it for you rapidly too - unless you insist on only using your own purchased servers in a colo, you can get additional capacity quickly, 2) a lot of managed hosting providers also have cloud instances so you can mix and match, 3) worst case you can spin up cloud instances elsewhere and tie it into your network via a VPN. Some offer the full range from colo via managed servers to cloud instances in the same datacentres. Once you prep for a hybrid setup, incidentally, cloud becomes even less competitive, because suddenly you can risk pushing the load factor on your own/managed servers much closer to the wire, knowing you can spin up cloud instances as a fallback. As a result, the cost per request for managed servers drops significantly. I also blame a lot of this on business often shielding engineering from seeing budgets and costs. I've been in quite senior positions in a number of companies where the CEO or CFO were flabbergasted when I asked for basics costing of staff and infra, because I saw it as essential in planning out architecture. Engineers who aren't used to seeing cost as part of their domain will never have a good picture of costs.
- graemep 1y ago
  For smaller businesses it seems to be its the safe option because its what everyone does. I have even had it suggested that it might make selling a business or attracting investors harder if you used your own servers (not at the scale of having your own datacentre, just rented servers - smaller businesses still). Another thing that comes up is that it might be more expensive but its a small fraction of operational expenses so no one really cares.
  - whstl 1y ago
    For smaller businesses it's often "the only thing Joe knew when he was building it".
- jinjin2 1y ago
  Yes. We saved ridiculous amounts of money (and made it a lot faster) by moving our analytics workloads from Snowflake to a few bare-metal nodes running Exasol. But it took months to convince management even though we had clear numbers showing the sheer magnitude of the cost reduction. They had drunk the cloud kool-aid, and were adamant that it would be cheaper, numbers be damned.
- j45 1y ago
  You have a great point about finding cost efficiencies - there was a time cloud was cheaper. Maybe it's an understanding that doesn't change because the decision makers were non-techincal people (when finance oversees IT despite not understanding it) Virtualizing and then sharing a dedicated server as a VPS was a big step forward. Only, hardware kept getting cheaper and faster, as well as internet.
  - vidarh 1y ago
    > when finance oversees IT despite not understanding it ... and when IT often do not even get to see the spend, and/or isn't expected to. I've had clients where only finance had permissions to get at the billing reports, and engineering only ever saw the billing data when finance were sufficiently shocked by a bill to ask them to dig into it - at which point they cared for long enough to get finance off their backs, and then stopped caring again.
    - j45 1y ago
      Unsure how I missed this. Great points. Overseeing things they don’t understand and wanting to manage and direct it feels unnatural for finance to do to other departments. Maybe this was more common for businesses with stable business processes that aren’t evolving. Covid and now AI will ensure change is constant and where a practice is outdated so will the organizations become.
peeters 1y ago
The reality is when you get to another certain point (larger than the point you describe) you start negotiating directly with those cloud providers and bypass their standard pricing models entirely. It's the time in between that's the most awkward. When the potential savings are there that hiring an engineering team to internalize infrastructure will give a good return (were current pricing to stay), but you're not so big that just threatening to leave will cause the provider to offer you low margin pricing. All I'd say is don't assume you're getting the best price you can get. Engineers are often terrible negotiators, we'd rather spend months solving a problem than have an awkward conversation. Before you commit to leaving, take that leverage into a conversation with your cloud sales rep.
- selfhoster 1y ago
  > It's the time in between that's the most awkward. That's an odd way to describe hemorrhaging money.
- jonatron 1y ago
  At what sort of scale can you do that? $1M, $10M, $100M, $1B?
  - sokoloff 1y ago
    I don’t remember if our first enterprise agreement was at $1M or $2M, but it was low and in that neighborhood [but also 10 years ago, well before cloud was the default and had growth baked into it]. Cloud providers are looking for multi-year term, commitment to growth as much as/more than exact spend level now.
  - peeters 1y ago
    So obviously this is an extreme, but I worked for a company that had long dismissed third party cloud providers as too expensive (customers would be routing all of their network traffic through our data centers, so obviously the bandwidth costs would just be too dang high). Then that company got purchased by a certain mega corporation who then negotiated an exclusive deal with GCP, and the math flipped. It was now far too expensive to run our own set of datacenters. Google was willing to take such a low margin on bandwidth that it made no sense not to. So in this case, hundreds of billions. But the principle stands at lower company sizes, just with different numbers and amounts of leverage.
dangus 1y ago
It’s more than mere “convenience.” You’re also paying to avoid hiring a bunch of employees to physically visit data centers around the globe. And if you’re not doing that you are hiring a bare metal servers provider that is still taking a portion of the money you’d be paying AWS. Even if you don’t need to physically visit data centers thanks to your server management tools, the difference in the level of control you have between cloud and bare metal servers is large. You’re paying to enable workflows that have better automation and virtual networking capabilities. I recently stood up an entire infrastructure in multiple global locations at once and the only reason I was able to do it in days instead of weeks or months was because of the APIs that Amazon provides that I can leverage with infrastructure automation tooling. Once you are buying AWS reservations and avoiding their most expensive specialized managed products the price difference isn’t really worth trying to recover for many types of businesses. It’s probably worth it for Hey since they are providing a basic email service to consumers who aren’t paying a whole lot. But they still need something that’s “set it and forget it” which is why they are buying a storage solution that already comes with an S3 compatible API. So then I have to ask why they don’t save even more money and just buy Supermicro servers and install their own software? We all know why: because Amazon’s APIs are where the value is. There is a lot of profit margin in software and usually your business is best spending their effort working on their core product rather than keeping the lights on, even for large companies. Plus, large companies get the largest discounts from cloud providers which makes data centers even less appealing. “Convenience” isn’t just convenience, it’s also the flexibility to tear it all down and instantly stop spend. If I launch a product and it fails I just turn it off and it’s gone. Not so if I have my own data center and now I’ve got excess capacity.
- luckylion 1y ago
  I agree, but I don't think you're in the majority. I don't think most cloud-customers are utilizing all of those additional things that a big cloud provider offers. How many are actually multi-region? How many actually do massive up/down-scaling on short notice? How many actually use many of those dozens to hundreds of services? How many actually use those complex permissions? My experience tells me there are some, but there are more who treat AWS/GPC/Azure like a VPS-hoster that's 5-10x more expensive than other hosters. They are not multi-region, they don't do scaling, they go down entirely whenever the AZ has some issues etc. The most they do is maybe use RDS instead of installing mysql/pgsql themselves.
binarymax 1y ago
Even without your own rack or colo, The math with AWS stops working as soon as you no longer fit in the free tier, since providers like Hetzner are 40% cheaper.
- gizmo 1y ago
  S3 is designed for 99.999999999% durability. Hetzner's Volume storage is just replication between 3 different physical servers. In terms of durability that's a universe apart.
  - SteveNuts 1y ago
    S3 is beyond impressive, but how many workloads truly need that? I’ve never had a single instance of data loss on a NetApp or Pure array.
    - jessekv 1y ago
      Truly need, I don't know. But customers will request (and pay for) the 9's.
      - SteveNuts 1y ago
        I suppose it’s like how someone who’s already made up their mind to buy a Lamborghini never questions whether they really need a 800HP engine.
  - sebazzz 1y ago
    On the other hand you have transient failures in the cloud (at least on Azure - this behavior is even documented) so does that count towards the 99.99999%?
    - jwiz 1y ago
      That sounds like accessibility, not durability.
djha-skin 1y ago
It's not a novel problem but it _is_ a relatively novel (bad) economic environment. We've been in "let the good times roll" mode longer than ten years. In comparison to 2009-2011, it was different. Many ops professionals are younger than that and have gone their entire careers without doing anything on premise. I remember trying to convince some very talented but newly minted ops professionals -- my colleagues -- to go on prem for cost. This was last year. They were scared. They didn't know how that would work or what safety guarantees there would be. They had a point, because the org I was at then didn't have any on prem presence, since they were such a young organization that they started in the cloud during "the good times". They always hired younger engineers for cost, so nearly no one in the org even knew how to do on prem infra. Switching then would have been a mistake for that org, even though cloud costs (even with large commit agreements) were north of six figures a month.
- tomrod 1y ago
  What do you mean "for cost" in your comment? For cost savings / frugal purposes? Or using something like a sweetheart deal with a PEO?
t0mas88 1y ago
Around that certain point you can also talk to AWS or GCP and get very significant discounts. I'm surprised 37signals and AWS didn't find a number that worked for both. I've seen a few of these deals with other vendors up close, the difference with public pricing is huge if you spend millions per year.
- bigfatkitten 1y ago
  DHH has said previously that they already have a very good deal when compared with list price. But AWS still couldn't come close to on prem costs.
- dilyevsky 1y ago
  I worked/consulted for several companies who had multimillion per year cloud commits, sometimes with different clouds, and those discounts are not competitive with onprem like at all
- j45 1y ago
  If it takes talking to them to get discounts, might as well look at all the options and get the real discount of not being on the cloud.
hodgesrm 1y ago
> The Math stops working when you grow to a certain point. That point is different for every business. For most of them it depends on how big cloud is in your COGS (cost of goods sold) which affects gross margins, which in turn is one of the most meaningful measures of company financial health. Depending on the nature of your business and the amount of revenue you collect in sales, many companies will never reach the point where there's measurable payback from repatriating. Others may reach that point, but it's a lower priority than other things like opening up new markets. Many commenters seem to hold very doctrinaire opinions on this topic, when it's mostly basic P&L math.

jstummbillig 1y ago

I find it intuitively absolutely bizarre that Cloud does not outright win at any scale. In my mind everything about it seems more optimizable with more scale. Obviously I am missing something, but all Cloud pricing looks so significantly more expensive than I feel it should in a healthy and mature market.

vidarh 1y ago

It's horrifyingly hard to convince people of this, though, even you can present them with actual numbers. A lot of people have convinced themselves that cloud is cheap, to the point that they don't even do a cursory investigation. A lot of those even don't do the bare minimum to reduce hosting costs within the cloud they choose, or choose one of the cheaper clouds (AWS is absolutely extortionate for anything that requires significant amount of outbound bandwith), or put caching/CDN's in front (you can trivially slash your AWS egress costs dramatically). Most of my consultancy work is on driving cost efficiencies for cloud, and I can usually safely guarantee the fee will pay for itself within months because people don't fix even the most low hanging fruit.

ksec 1y ago
Partly because AWS give out a lot of free credit for start ups, and basically allow them to grow without planning any infrastructure. VCs who are invested into Amazon also wants to push the cloud narrative. Starts up who dont want to deal with servers, want massive scale when they think the website and later an app went viral. That was in the late 00s and early 10s. PHP, Python, Ruby and even Java were slow. Every single language and framework has had massive performance improvements in the past 15 to 20 years. Anywhere from Java 2x to Ruby 3 - 10x. When a server max out at 6 - 8 with Xeon core, compare to today at 192 Core. Every Core is at leats 2 - 3x faster per clock, with higher clock speed we are talking about 100x difference. Especially when IO used to be on HDD, SSD is easily 1000x faster. What used to wait for I/O is no longer an issue, the aggregate difference when all things added together including software could be 300x to 500x. What you would need 500 2U server in 2010, you could now do it in one. Modern web developers are so abstracted with hardware I dont think many realise what sort of difference in hardware improvements. I remember someone posted before 2016 Basecamp had dozens of Racks before moving to cloud. Now they have grown a lot bigger with Hey and they are only doing it with 8 racks and room to spare. AWS on the other hand is trying to move more workload to ARM Graviton where they have a cost advantage. Given Amazon's stock price are now dependent on AWS, I dont think they will lower their price by much in the future. And we desperately need some competition in that area.
mbreese 1y ago
I think one business argument for cloud is capital expenses vs operational expenses. If you’re (over) paying for cloud resources vs an in house option (or colo), those are numbers that are a straight expense. When you own hardware, those are on your books until they depreciate off. For some businesses, that can make sense. Now, a good accountant probably wouldn’t care one way or the other. Debits and credits balance either way. And spending more still means less profit in the long term, no matter how it looks on the books. But, in addition to the flexibility, that was what I always thought of as the main cloud benefit. It’s the same with leasing vs buying cars/computers/etc…
- vidarh 1y ago
  But that too is based on people not knowing the alternatives, as renting managed servers can be close to a wash vs. leasing hardware for a colo (often to the point that relatively cost of land near your preferred managed hosting providers vs. colos that work with access to staff etc. might be what makes one or the other cheaper). Buying outright can be cheaper but isn't necessary. None of the colo'd setups I've worked on bar one used purchased servers - it's all been leased. But the majority of non-cloud workloads I've worked on have not even been leased, but rented.
- tialaramex 1y ago
  Cloud made sense for the startup I worked for previously. If you are a startup then a $1M per year expense makes much more sense than a $5M up front purchase with 5-10 years of life - in five years you might be billionaires or you might be bankrupt and until then the Cloud was better.
  - vidarh 1y ago
    Or you could have rented or leased-to-own. There's hardly ever any need to actually purchase outright to get prices far below equivalent capacity in clouds. In fact, in 25 years, only one of the colo'd server setups I've worked on had any hardware purchased up-front in it.
  - dilyevsky 1y ago
    Like sibling is saying it's not $5M upfront but on the order of 1 year cloud spend for large enough accounts. There are also such things as leases and loans. One justifiable excuse is you simply don't know how much hardware you will need to buy if you're hitting hockeystick growth. That until you realize you can also go hybrid...
tialaramex 1y ago
Periodically management says we shouldn't have a DC, just put everything in the cloud. OK says HPC, here's the quote for replacing one of the (currently three) supercomputers with a cloud service. Oh dear, that's bigger than your entire IT budget isn't it? So I guess we do need the DC for housing the supercomputers. If we'd done that once I'd feel like well management weren't to know, but it recurs with about a 3-5 year periodicity. The perception seems to be "Cloud exists, therefore it must be cheaper, because if it wasn't cheaper why would it exist?" which reminds me of how people persuade themselves the $50 "genuine Apple" part must be better because if it wasn't better than this $15 part why would Apple charge $50 for it? Because you are a sucker is why.
- vidarh 1y ago
  Yeah, I used to be asked to price out a move to AWS every year at one position. After several years Hetzner finally got cheaper than operating our own colo's, but only basically because we were in London and London real-estate is expensive, and so colo space is accordingly expensive, while Hetzner's DC space is dirt cheap. AWS, however, remained 2x-3x as expensive, with the devops time factored in. > The perception seems to be "Cloud exists, therefore it must be cheaper, because if it wasn't cheaper why would it exist? People are also blithely unaware that large customers get significant discounts, and so I regularly has to explain that BigCo X being hosted in AWS means at most that it is cost-effective for them because their spend means they're getting a significant discount over the already highest volume published pricing, and my clients usually are nowhere close to spend enough to be able to get those discounts.
- koliber 1y ago
  Regarding apple parts, I recently replaced a broken screen on a MacBook pro with an OEM part. I can’t get the color to look right. Not to mention the one vertical row where pixels look off (not dead, but not normal either). The guy at the shop said I would not notice. I am now kicking myself for not going with the real thing.
- paulcole 1y ago
  [flagged]
- 1y ago
diggan 1y ago
> A lot of people have convinced themselves that cloud is cheap I've noticed this too, freelancing/consulting around in companies. I'm not sure where this idea even comes from, because when cloud first started making the news, the reasoning went something like "We're OK paying more since it's flexible, so we can scale up/down quickly", and that made sense. But somehow today a bunch of people (even engineers) are under the belief that cloud somehow is cheaper than the alternatives. That never made sense to me, even when you take into account hiring people specifically for running the infrastructure, unless you're a one-person team or have to aggressively scale up/down during a normal day.
- vidarh 1y ago
  I think it's because people think their workloads are extremely spiky, and so assume they will spin up/down loads enough to save money, and that has translated into cloud being perceived as cheap. But devs rarely pay attention to metrics. I've had clients with expensive Datadog setups where it was blatantly obvious that nobody had ever dug into the performance data, because if they did they'd have noticed that key metrics were simply not fed to it. If they did pay attention, most of them would realise that their autoscaling rarely kicks in all that much, if at all. Often because it's poorly tuned, but also because most businesses see small enough daily cycles. Factor in that the cost difference between instances vs. managed servers is quite significant, and you need to have significant spikes much shorter in duration than most businesses day/night variation to save money. It can make sense to be able to spin up more capacity quickly, but then people need to consider that 1) a lot of managed hosting providers has hardware standing by and can automatically provision it for you rapidly too - unless you insist on only using your own purchased servers in a colo, you can get additional capacity quickly, 2) a lot of managed hosting providers also have cloud instances so you can mix and match, 3) worst case you can spin up cloud instances elsewhere and tie it into your network via a VPN. Some offer the full range from colo via managed servers to cloud instances in the same datacentres. Once you prep for a hybrid setup, incidentally, cloud becomes even less competitive, because suddenly you can risk pushing the load factor on your own/managed servers much closer to the wire, knowing you can spin up cloud instances as a fallback. As a result, the cost per request for managed servers drops significantly. I also blame a lot of this on business often shielding engineering from seeing budgets and costs. I've been in quite senior positions in a number of companies where the CEO or CFO were flabbergasted when I asked for basics costing of staff and infra, because I saw it as essential in planning out architecture. Engineers who aren't used to seeing cost as part of their domain will never have a good picture of costs.
graemep 1y ago
For smaller businesses it seems to be its the safe option because its what everyone does. I have even had it suggested that it might make selling a business or attracting investors harder if you used your own servers (not at the scale of having your own datacentre, just rented servers - smaller businesses still). Another thing that comes up is that it might be more expensive but its a small fraction of operational expenses so no one really cares.
- whstl 1y ago
  For smaller businesses it's often "the only thing Joe knew when he was building it".
jinjin2 1y ago
Yes. We saved ridiculous amounts of money (and made it a lot faster) by moving our analytics workloads from Snowflake to a few bare-metal nodes running Exasol. But it took months to convince management even though we had clear numbers showing the sheer magnitude of the cost reduction. They had drunk the cloud kool-aid, and were adamant that it would be cheaper, numbers be damned.
j45 1y ago
You have a great point about finding cost efficiencies - there was a time cloud was cheaper. Maybe it's an understanding that doesn't change because the decision makers were non-techincal people (when finance oversees IT despite not understanding it) Virtualizing and then sharing a dedicated server as a VPS was a big step forward. Only, hardware kept getting cheaper and faster, as well as internet.
- vidarh 1y ago
  > when finance oversees IT despite not understanding it ... and when IT often do not even get to see the spend, and/or isn't expected to. I've had clients where only finance had permissions to get at the billing reports, and engineering only ever saw the billing data when finance were sufficiently shocked by a bill to ask them to dig into it - at which point they cared for long enough to get finance off their backs, and then stopped caring again.
  - j45 1y ago
    Unsure how I missed this. Great points. Overseeing things they don’t understand and wanting to manage and direct it feels unnatural for finance to do to other departments. Maybe this was more common for businesses with stable business processes that aren’t evolving. Covid and now AI will ensure change is constant and where a practice is outdated so will the organizations become.

peeters 1y ago

The reality is when you get to another certain point (larger than the point you describe) you start negotiating directly with those cloud providers and bypass their standard pricing models entirely. It's the time in between that's the most awkward. When the potential savings are there that hiring an engineering team to internalize infrastructure will give a good return (were current pricing to stay), but you're not so big that just threatening to leave will cause the provider to offer you low margin pricing. All I'd say is don't assume you're getting the best price you can get. Engineers are often terrible negotiators, we'd rather spend months solving a problem than have an awkward conversation. Before you commit to leaving, take that leverage into a conversation with your cloud sales rep.

selfhoster 1y ago
> It's the time in between that's the most awkward. That's an odd way to describe hemorrhaging money.
jonatron 1y ago
At what sort of scale can you do that? $1M, $10M, $100M, $1B?
- sokoloff 1y ago
  I don’t remember if our first enterprise agreement was at $1M or $2M, but it was low and in that neighborhood [but also 10 years ago, well before cloud was the default and had growth baked into it]. Cloud providers are looking for multi-year term, commitment to growth as much as/more than exact spend level now.
- peeters 1y ago
  So obviously this is an extreme, but I worked for a company that had long dismissed third party cloud providers as too expensive (customers would be routing all of their network traffic through our data centers, so obviously the bandwidth costs would just be too dang high). Then that company got purchased by a certain mega corporation who then negotiated an exclusive deal with GCP, and the math flipped. It was now far too expensive to run our own set of datacenters. Google was willing to take such a low margin on bandwidth that it made no sense not to. So in this case, hundreds of billions. But the principle stands at lower company sizes, just with different numbers and amounts of leverage.

dangus 1y ago

It’s more than mere “convenience.” You’re also paying to avoid hiring a bunch of employees to physically visit data centers around the globe. And if you’re not doing that you are hiring a bare metal servers provider that is still taking a portion of the money you’d be paying AWS. Even if you don’t need to physically visit data centers thanks to your server management tools, the difference in the level of control you have between cloud and bare metal servers is large. You’re paying to enable workflows that have better automation and virtual networking capabilities. I recently stood up an entire infrastructure in multiple global locations at once and the only reason I was able to do it in days instead of weeks or months was because of the APIs that Amazon provides that I can leverage with infrastructure automation tooling. Once you are buying AWS reservations and avoiding their most expensive specialized managed products the price difference isn’t really worth trying to recover for many types of businesses. It’s probably worth it for Hey since they are providing a basic email service to consumers who aren’t paying a whole lot. But they still need something that’s “set it and forget it” which is why they are buying a storage solution that already comes with an S3 compatible API. So then I have to ask why they don’t save even more money and just buy Supermicro servers and install their own software? We all know why: because Amazon’s APIs are where the value is. There is a lot of profit margin in software and usually your business is best spending their effort working on their core product rather than keeping the lights on, even for large companies. Plus, large companies get the largest discounts from cloud providers which makes data centers even less appealing. “Convenience” isn’t just convenience, it’s also the flexibility to tear it all down and instantly stop spend. If I launch a product and it fails I just turn it off and it’s gone. Not so if I have my own data center and now I’ve got excess capacity.

luckylion 1y ago
I agree, but I don't think you're in the majority. I don't think most cloud-customers are utilizing all of those additional things that a big cloud provider offers. How many are actually multi-region? How many actually do massive up/down-scaling on short notice? How many actually use many of those dozens to hundreds of services? How many actually use those complex permissions? My experience tells me there are some, but there are more who treat AWS/GPC/Azure like a VPS-hoster that's 5-10x more expensive than other hosters. They are not multi-region, they don't do scaling, they go down entirely whenever the AZ has some issues etc. The most they do is maybe use RDS instead of installing mysql/pgsql themselves.

binarymax 1y ago

Even without your own rack or colo, The math with AWS stops working as soon as you no longer fit in the free tier, since providers like Hetzner are 40% cheaper.

gizmo 1y ago
S3 is designed for 99.999999999% durability. Hetzner's Volume storage is just replication between 3 different physical servers. In terms of durability that's a universe apart.
- SteveNuts 1y ago
  S3 is beyond impressive, but how many workloads truly need that? I’ve never had a single instance of data loss on a NetApp or Pure array.
  - jessekv 1y ago
    Truly need, I don't know. But customers will request (and pay for) the 9's.
    - SteveNuts 1y ago
      I suppose it’s like how someone who’s already made up their mind to buy a Lamborghini never questions whether they really need a 800HP engine.
- sebazzz 1y ago
  On the other hand you have transient failures in the cloud (at least on Azure - this behavior is even documented) so does that count towards the 99.99999%?
  - jwiz 1y ago
    That sounds like accessibility, not durability.

djha-skin 1y ago

It's not a novel problem but it _is_ a relatively novel (bad) economic environment. We've been in "let the good times roll" mode longer than ten years. In comparison to 2009-2011, it was different. Many ops professionals are younger than that and have gone their entire careers without doing anything on premise. I remember trying to convince some very talented but newly minted ops professionals -- my colleagues -- to go on prem for cost. This was last year. They were scared. They didn't know how that would work or what safety guarantees there would be. They had a point, because the org I was at then didn't have any on prem presence, since they were such a young organization that they started in the cloud during "the good times". They always hired younger engineers for cost, so nearly no one in the org even knew how to do on prem infra. Switching then would have been a mistake for that org, even though cloud costs (even with large commit agreements) were north of six figures a month.

tomrod 1y ago
What do you mean "for cost" in your comment? For cost savings / frugal purposes? Or using something like a sweetheart deal with a PEO?

t0mas88 1y ago

Around that certain point you can also talk to AWS or GCP and get very significant discounts. I'm surprised 37signals and AWS didn't find a number that worked for both. I've seen a few of these deals with other vendors up close, the difference with public pricing is huge if you spend millions per year.

bigfatkitten 1y ago
DHH has said previously that they already have a very good deal when compared with list price. But AWS still couldn't come close to on prem costs.
dilyevsky 1y ago
I worked/consulted for several companies who had multimillion per year cloud commits, sometimes with different clouds, and those discounts are not competitive with onprem like at all
j45 1y ago
If it takes talking to them to get discounts, might as well look at all the options and get the real discount of not being on the cloud.

hodgesrm 1y ago

> The Math stops working when you grow to a certain point. That point is different for every business. For most of them it depends on how big cloud is in your COGS (cost of goods sold) which affects gross margins, which in turn is one of the most meaningful measures of company financial health. Depending on the nature of your business and the amount of revenue you collect in sales, many companies will never reach the point where there's measurable payback from repatriating. Others may reach that point, but it's a lower priority than other things like opening up new markets. Many commenters seem to hold very doctrinaire opinions on this topic, when it's mostly basic P&L math.