9 ms·

I really don't understand why the industry seems to have lost sight of this. It's really common to see super complicated, incredibly expensive, but highly scala

I really don't understand why the industry seems to have lost sight of this. It's really common to see super complicated, incredibly expensive, but highly scalable cloud deployments for problems that can be trivially solved with one or two dedicated servers. Even the mere suggestion of renting a dedicated server provokes scorn from devops teams. The overall difference in cost when taking into account all of the complexity, feature-lag and general ceremony must be at least 10x and maybe even closer to 100x. It's madness.

sinenomine 4y ago
> Even the mere suggestion of renting a dedicated server provokes scorn from devops teams. There is a method to the madness, here it is called "job-security-driven development".
dpweb 4y ago
Its big companies - their priorities are different. Saving a couple million dollars isn't worth the tradeoff of not being industry standrd (AWS, Google, MS). They hire tech consulting companies who sell their product, large deployments.
adrianmsmith 4y ago
I think it might be due to the incentives - the costs are borne by the company (thus irrelevant to the employee), however putting new cool technologies on their CV is very relevant to the employee.
- bell-cot 4y ago
  > ...company...employee... Yes - but stupid yearnings to "do what all the cool kids are doing now" are at least as strong in those who would normally be referred to as "managers", vs. "employees".
  - adrianmsmith 4y ago
    Good point - and in the case of managers there's also the "nobody got fired for buying IBM" effect: outcome A) huge microservices/cloud spend goes wrong, well at least you were following best practices, these things happen, what can you do. outcome B) you went with a Hetzner server and something went wrong, well you are a fool and should have gone with microservices, enjoy looking for a new job. Thus encouraging managers to choose microservices/cloud. It might not be the right decision for the company, but it's the right decision for the manager, and it's the manager making the decision.
- rr888 4y ago
  Agreed. I love our physical in house servers, but not knowing AWS in depth really hurt my last job search.
  - adrianmsmith 4y ago
    Yeah that's the problem. It's not just a "devs are idiots, they should think of their company and not be obsessed with their CV" - not having worked with the latest trendy tech has a very real cost for developers. (As does being a consultant wanting an extension and writing software that works, as I found out the hard way.)
onlyrealcuzzo 4y ago
> I really don't understand why the industry seems to have lost sight of this. There's a disconnect between founders and everyone else. Founders believe they're going to >10x every year. Reality is that they're 90% likely to fail. 90% of the time - you're fine failing on whatever. Some % of the 10% of the time you succeed you're still fine without the cloud - at least for several years of success, plenty of time to switch if ever necessary.
- wongarsu 4y ago
  Even if you 10x every year, going from year 1 to year 2 you switch from renting two 6-core servers to renting two 64 core servers, in year 3 you rent 20 servers, in year 4 you are renting a small room of racks in a colo space and add a server every other day (or you continue renting servers, if that's your style). Out of all the problems that arise from 10x-ing your organisation every year, this is a really minor one.
  - ipaddr 4y ago
    The fear.. I'll need 20 servers in 3 years.. the reality: A 4 dollar vps is powerful enough until you hit millions and millions of customers.
    - l5870uoo9y 4y ago
      would prefer to spend 20-30-50 USD and get a managed deployment infrastructure with rollback, blue/green deployment, github integration and so. this setup can last a long time.
      - justsomehnguy 4y ago
        And all that to host what could be served by Hugo static site.
      - mring33621 4y ago
        I'd pay good money if you could truly teach me how to do that for recurring costs within a factor of 2 from cheap rental vps/box.
criley2 4y ago
I'll be the counter opinion for this. Let's say you're Sprocket Masters, and you have to run two front ends (a Sprocket Sales site and a Sprocket Repair site) and a backend connecting them. But ultimately your sprockets are made in your factory on-site, and the majority of your staff is related to the manufacture of sprockets. You're not a software company, fundamentally you make and sell Sprockets. The opinions here would be hire a big eng/IT staff to "buy and maintain servers" (PaaS is bad) and then likely "write a bunch of code yourself" (SaaS is bad) or whatever is currently popular here (last thread here it was "Pushing PHP over SFTP without Git, and you're silly if you need more" lol) But I believe businesses should do One Thing Well and avoid trying to compete (doing it manually) for things outside of their core competency. In this case, I would definitely think Sprocket Masters should not attempt to manage their own hardware and should rely on a provider to handle scaling, security, uptime, compliance, and all the little details. I also think their software should be bog-standard with as little in-house as possible. They're not a software shop and should be writing as little code as possible. Realistically Widget Masters could run these sites with a rather small staff unless they decided to do it all manually, in which case they'd probably need a lot larger staff.
- lukeramsden 4y ago
  You still have to do all that with cloud though - instead of firewalls and IP tables it’s VPCs, IAM, security groups, auto scaling groups, etc etc. The reason I think it’s still probably better for Sprocket Masters to go cloud is purely because those skills are easier to hire for.
- arp242 4y ago
  The core business for Sprocket Masters isn't really tech stuff: it's making sprockets. For those kind of businesses it makes a lot of sense; the tech stuff is required, but only in the sane sense that a phone line or electricity is. However, what I also see – and what I think the previous poster was talking about – are businesses where tech is at the core of the business, and there it often makes less sense, and instead of saving time it seems to cost time. There's a reason there are AWS experts: it's not trivial. "Real" servers also aren't trivial, but also not necessarily harder than cloud services.
esskay 4y ago
A lot of its the buzz. 'Cloud Hosting' is something everyone thinks they need, despite not grasping why, or that the chances are they arent getting cloud hosting at all, just a VPS. A solid dedicated server is 99% of the time far more useful than a crippled VPS on shared hardware but it obviously comes at an increased cost if you dont need all the resources they provide.
xnx 4y ago
devops people want things to be expensive and complicated to justify their salaries
- k8sToGo 4y ago
  Nice generalization. So far I haven’t noticed that if I spend more for the company that I also get paid more.
- DrBenCarson 4y ago
  Remember how Twitter lost 80% of its staff and still kept running?
  - baq 4y ago
    And Twitter ceo turned his account private because they can’t figure out what the algorithm does
  - wkat4242 4y ago
    For now... There's a lot of inertia obviously.
- dsr_ 4y ago
  Anybody with experience on the ops side of that is absolutely in favor of simple and easy to understand -- and that often gets you cheap as a side effect. Source: almost thirty years of ops.
- sshine 4y ago
  Having talked to AWS evangelists, it seems that they truly believe it cuts costs. In their reality this is true, because an alternative with the same guarantees and risk costs more. The implicit disagreement is that you often don’t need all those guarantees.
  - marcosdumay 4y ago
    > because an alternative with the same guarantees and risk costs more I really doubt this. The one theme you see on almost every AWS proponent is some high amount of delusion about what guarantees AWS actually provides you.
cameronh90 4y ago
If you're ever trying to get big enterprise contracts, the kind of scalability, compliance and disaster recovery that providers like AWS/Azure enable are now just table stakes. Sure, you can get something working on Hetzner but be prepared to answer a lot more questions.
- axelthegerman 4y ago
  Yes fair point though I doubt some of these super distributed, scalable apps are able to properly execute a disaster recovery... And just because all the serverless components scale "infinitely" doesn't mean the patchwork stitching them together is. Agreed on your last point of enterprise asking for this which again is just sad that these business "requirements" dictate how to architect and host your software when another way might be the much better one
- jwr 4y ago
  I run a self-funded SaaS business, so I only deal with real problems, not imaginary ones. Disaster recovery is a very real problem. Which is why I test it regularly. In a scorched-earth scenario, I can be up and running on a secondary (staging) system or a terraformed cloud system in less than an hour. For my business, that is enough.
alex7734 4y ago
In addition to what has already been said in the other comments, devs have an incentive to push cloud tech against the interest of their employers so they can put the experience in their resume, since cloud tech is seen as more specialized and in higher regard and as such a dev experienced in it commands a higher salary. Plus who doesn't want to play with the newest, coolest toy on another's dime?
- corford 4y ago
  In my experience it's often the other way around. CTO/mgmt/HR would rather swallow the perf/cost hit of Cloud because it makes auditing, compliance, support, dr, security, training and hiring easier. Not to mention it usually provides much more flexibility during the PMF stage of a startup (where product roadmap, team talent/bandwidth and execution speed are often much more important drivers than the raw perf/cost ratio of your infra). There's definitely an argument to move (some) stuff off cloud later in the journey when flexibility (or dealing with flux/pivoting) becomes less of a primary driver and scale/cost start dominating.
  - margorczynski 4y ago
    "It's insane not to start your startup on the Cloud and even more so to stay on it" paraphrasing something I think I've read on HN some time ago. But I agree, it's best to treat the Cloud as an intermediate solution and once you stabilize and a clearer picture about requirements is available move at least partially to on-prem (especially the more expensive stuff like compute, storage is I think fine)
- kbrannigan 4y ago
  RDD or Resume Driven Development
  - rqtwteye 4y ago
    Realistically RDD is a very sane strategy for employees. When I read my company's job ads a few years ago I had serious doubts if I would get hired by them because my resume didn't have the buzzwords they wanted. Since then I always sprinkle in some new tech with each new project, not because it makes sense for the company, but for my resume and future employability.
mariusmg 4y ago
>I really don't understand why the industry seems to have lost sight of this. Because the industry is full of people who are chasing trends and keywords and to which the most important thing is to add those keywords to the CVs.
macNchz 4y ago
The dedicated server deployments I worked on at smallish software companies 10+ years ago wound up being really annoying. I enjoy sysadmin type stuff and this idea has tempted me, but I think it’s a false economy in most cases. The incremental cost of being on a cloud is totally worth it to me to have managed databases, automatic snapshots, hosted load balancers, plug and play block storage etc. I want to worry about whether our product is good, not about middle of the night hardware failures, spinning up a new server and having it not work right because there was no incentive to use configuration management with a single box, having a single runaway task cause OOM or full disk and break everything instead of just that task’s VM, fear of restarting a machine that has been up 1000 days etc.
- pas 4y ago
  Aaand that's why k8s seems like a good step in the right direction. It has all - okay, absolutely not all, but a good number of the important ones - of these concepts "built-in". Sure, the actual pile of Go(o) that drives it can be improved (and it is indeed improving). That said, the hard truth is that your approach is the correct one, almost all businesses (startups!) overbuild instead of focusing on providing value, identifying the actual niche, etc. (Yes, there's also the demand side of this, as the VC money inflated startups overbuild they want to depend on 3rd parties that can take the load, scale with them, SLAs and whatnot must be advertised.)
  - wkat4242 4y ago
    I don't agree, Ansible would be a much saner approach for a few dedicated boxes than k8s
    - riku_iki 4y ago
      What about deploying prebuilt docker image on dedicated server?
      - allarm 4y ago
        What about it? It can be easily done with Ansible. I managed a distributed infra with containers orchestrated by Ansible, worked fine.
Slartie 4y ago
"No one ever got fired for buying AWS"
- croo 4y ago
  You mean Oracle. That's another full circle right here...
avereveard 4y ago
if you need durability in the cloud you just pick any object storage service from the 3 bigs, and a db with multiple deployment with automatic replication and managed failover. getting the equivalent reliability with irons is a lot more expensive than renting "two dedicated servers" - now you might be fine with one server and a backup solution, and that's fair. but a sysad to create all that, even on a short contract for the initial setup and no maintenance, is going to go well beyond the cloud price difference, especially if there's a database in the mix and you care about that data.
- phpnode 4y ago
  I'm not saying there's no place for cloud, but that many workloads benefit from using dedicated hardware. It is often much simpler to deploy to a dedicated server rather than spending months rearchitecting your system (or even worse, cutting customer facing features!) to efficiently make use of many small cloud instances.
l5870uoo9y 4y ago
> trivially solved with one or two dedicated servers... For many even lot less than that. I run a small side project[1] that went viral a few times (30K views in 24 hours or so) and it is running on a single core CPU web server and a managed Postgres likewise on a single CPU core. It hasn't even been close to full utilization. 1: https://aihelperbot.com/ https://aihelperbot.com/
- deleted 4y ago
  [deleted]
jerf 4y ago
I am responsible for about 10-12 t3.medium-class servers (number includes redundant servers) and the services running on them. All of them are fairly overprovisioned for what they are, excepting that the monitoring system we use on them for some reason likes to consume about half-a-gig of RAM so I can't run on even smaller servers. I write my services in Go, so the CPU isn't sitting there spending 98% of its time chasing pointers and can use multiple CPUs as needed for its heavy-hitting tasks. It's a rough day for those servers when they pass 5% average CPU for the day. Could I switch some of them to lambda functions? Or switch to ECS? Or switch to some other cloud service du jour? Maybe. But the amount of time I spent writing this comment is already about six month's worth of savings for such a switch. If it's any more difficult than "push button, receive 100% reliably-translated service", it's hard to justify. Some of this is also because the cloud does provide some other services now that enable this sort of thing. I don't need to run a Kafka cluster for basic messaging, they all come with a message bus. I use that. I use the hosted DB options. I use S3 or equivalent, etc. I find what's left over is almost hardly worth the hassle of trying to jam myself into some other paradigm when I'm paying single-digit dollars a month to just run on an EC2 instance. It is absolutely the case that not everyone or every project can do this. I'm not stuck to this as an end-goal. When it doesn't work I immediately take appropriate scaling action. I'm not suggesting that you go rearchitect anything based on this. I'm just saying, it's not an option to be despised. It has a lot of flexibility in it, and the billing is quite consistent (or, to put it another way, the fact that if I suddenly have 50x the traffic, my system starts choking and sputtering noticeably rather than simply charging me hundreds of dollars more is a feature to me rather than a bug), and you are generally not stretching yourself to contort into some paradigm that is convenient for some cloud service but may not be convenient for you.
FlyingSnake 4y ago
The allure of our industry to mime FAANG leaders might be one of the reasons we're at this stage. Most products could be safely run on simpler setups. I suspect that VendorOps and complex tools like kubernetes are favored by complexity merchants which have arisen in the past decade. It looks great on resume and gives tech leaders a false sense of achievement. Meanwhile Stackoverflow, which is arguably more important than most startups, is chugging along on dedicated machines. 1: https://stackoverflow.blog/2016/02/17/stack-overflow-the-architecture-2016-edition/ https://stackoverflow.blog/2016/02/17/stack-overflow-the-arc...
- dgb23 4y ago
  Software development and related things are known to have fantastic often free online resources for autodidacts. It seems like the trend in this space is to jump directly to the highest layer of abstraction. Skipping the fundamentals and point to $buzzword tools, libraries and products. You see glimpses of this in different forms. One is social media threads which ask stuff like “How much of X do I need to know to learn Y?” Where X is a fundamental, possibly very stable technology or field and Y is some tool du jour or straight up a product or service.
- failbuffer 4y ago
  "complexity merchants" - I love this term. It aptly describes beneficiaries of earlier IT fads. Remember UML? Remember WS-*? One could come up with a long list of technologies that were a bonanza for billing consultant hours but no so great for getting to solutions.
  - switchbak 4y ago
    The debacle that was J2EE is hard to top (but I'm sure some have!). I still have battle scars from OSGi. CORBA belongs there. And perhaps even the semantic web stuff. Definitely XML!
    - 77pt77 4y ago
      SOAP, don't forget SOAP.
raducu 4y ago
Anybody tried mixing cloud + self hosted k8s on hetzner or is it too complicated to do?
- icedchai 4y ago
  You'd probably have some latency issues, plus waste a ton of $$$ on the bandwidth egress fees.
- ilrwbwrkhv 4y ago
  You've already lost if you install k8s.
7952 4y ago
It feels like artificial scarcity similar to what you see in high fixed cost monopolies. And that is exactly what most corporate IT departments are.
tstrimple 4y ago
I feel like many people use the cloud "wrong". If you're just standing up VMs then yeah, it's going to be slower and more expensive than renting a physical server. If you're just going to setup VMs and pretend you're still on-prem you're better off not switching. But I've got projects setup on PaaS services that literally cost a few bucks a month because it's only charging me for actual use and not idle time. I've got some small customers (tens of thousands of users) who I've helped move into the cloud who see the same thing. They have SPA websites backed by lambda functions and on-demand databases. If their website isn't being hit, it's not costing them anything at all. The only lock-in is the boiler plate code around the lambda activation which could be switched fairly easily to target any Functions as a Service platform.
- woadwarrior01 4y ago
  I'm curious to know what on-demand databases do you use with AWS Lambdas? Two options that I've evaluated are: Aurora Serverless v2, which isn't quite serverless in practice and DynamoDB. And the mental model for DynamoDB very different from that of traditional SQL databases.
  - tstrimple 4y ago
    I’m actually primarily building on Azure with Cosmos DB. The mental model is very different, but I think important if you want to build cheap cloud services. I think that’s the larger point. You have to think about doing things differently in the cloud otherwise you’re just deploying and managing VMs as if you were on-prem albeit with more scaling control. It feels like everyone who is complaining about how expensive cloud hosting is go into it thinking it’s just someone else hosting the VMs where as a VM is the option of last resort for me.
znpy 4y ago
> Even the mere suggestion of renting a dedicated server provokes scorn from devops teams. Have you ever had to manage one of those environments? The thing is, if you want to get some basic more-than-one-person scalability and proper devops then you have to overprovision by a significant factor (possibly voiding your savings). You're inevitably going to end up with a bespoke solution, which means new joiners will have a harder time getting mastery of the system and significant people leaving the company will bring their intimate knowledge of your infrastructure with them. You're back to pets instead of cattles. Some servers are special, after a while automation means "a lot of glue shell scripts here and there" and an OS upgrade means either half infra is KO for a while or you don't do OS upgrades at all. And in the fortunate case you need to scale up... You might find unpleasant surprises. And don't ever get me started on the networking side. Unless you're renting your whole rack and placing your own networking hardware, you get what you get. Which could be very poor in either functionalities or performances... Assuming you're not doing anything fancy.
- lelanthran 4y ago
  > The thing is, if you want to get some basic more-than-one-person scalability and proper devops then you have to overprovision by a significant factor If you want 100.0000% uptime, sure. But you don't usually. The companies that want that kind of uptime normally has teams dedicated to it anyway. And scaling works well on bare-metal too if you scale vertically - have you any idea the amount of power and throughput you can get from a single server? It's concerning to keep hearing about "scaling" when the speaker means "horizontal scaling". If you requirements are "scaling", then vertical scaling will take you far. If your requirements are "horizontal scaling on demand", then, sure, cloud providers will help there. But, TBH, few places need that sort of scaling.
  - baq 4y ago
    Just getting a highly available storage with any kind of performance requires you to overprovision by something like 12x (3 boxes with raid 1+0) which is frankly insane. And then you have to manage it and fix it when it inevitably breaks. The team who’s supposed to do that will be swamped by trying to keep the other 15 infrastructure services running and upgraded to a version which is only barely out of vendor support.
barbazoo 4y ago
As a developer I appreciate the compartmentalization, being able to make truly isolated changes. This is something that's not trivial for lots of folks, me included. I guess if one has the expertise in house to manage a K8s cluster on a raw machine, that'd be an option but I, personally, am not gonna touch that with a ten foot pole. So yeah with a big and capable enough infrastructure team that complexity could be abstracted away but in reality I (where I've worked) haven't seen that yet.
codegeek 4y ago
Because cloud is the new "No one gets fired for using IBM". If you are using dedicated servers somewhere, you cannot have the same reputation of say Using AWS for cloud. Everyone knows the AWSs, Google Cloud etc and no one gets fired for selecting/using them. I personally have run Dedicated Servers for our business in the earlier days but as we expanded and scaled, it became a lot easier to go with the Cloud providers to provision various services quickly even though the costs went up. Not to mention it is a lot easier to tell a prospective customer that you use "cloud like AWS" than "Oh we rent these machines in a data center run by some company" (which can actually be better but customers mostly wont get that). Audit, Compliance and others.
jgalt212 4y ago
> Even the mere suggestion of renting a dedicated server provokes scorn from devops teams. Because it's a threat to their jobs.
swatcoder 4y ago
The industry lost sight of a lot of things. We went through a growth boom, and like all of them before, it meant there were lots of inexperienced people being handed lots of money and urgent expectations. It’s a recipe for cargo culting and exploitative marketing. But growth is slowing and money is getting more expensive, so we’ll slow down and start to re-learn the old lessons with exciting new variations. (Like Here: managing bare metal scaling and with containers and orchestration) And the whole cycle will repeat in the next boom. That’s our industry for now.
- deleted 4y ago
  [deleted]
Spooky23 4y ago
Growth hides sins. I ran a large self-funded enterprise service at a loss for 4-5 years, which is possible if you are growing 30%. Today, cloud is similar - the time to market is quicker as there are less moving parts. When the economy tanks and growth slows, the beancounters come in and do their thing. It happens every time.
itomato 4y ago
I just want to go back to leasing an always-available Opteron with optional remote hands.
slt2021 4y ago
the most funny thing is that cloud engineering teams have no problem of using EC2 hosts, but somehow don't want to rent dedicated servers/VPC from regular datacenter companies. This only makes AWS richer at the expense of companies and cloud teams
- selcuka 4y ago
  > the most funny thing is that cloud engineering teams have no problem of using EC2 hosts, but somehow don't want to rent dedicated servers/VPC from regular datacenter companies. It is trivial to provision and unprovision an EC2 instance automatically, within seconds if your deployment needs to scale up or scale down. That's what makes it fundamentally different from a bare metal server. Now, I'm not denying that it might still be more cost effective when compared to AWS to provision a few more dedicated servers than you'll need, but when you have really unpredictable workloads it's not easy to keep up.
  - slt2021 4y ago
    the whole concept of a need to spin up more VM instances to meet increased load is flawed and is driven by cloud vendors to make money. if you are spinning up and shutting down VMs to meet demand curve - something is seriously wrong with your architecture. Did it ever occur to you, how come stackoverflow uses ~8 dedicated servers to serve entire world and doesn't need to spin up and shutdown VMs to meet global customer demand? -- When planning compute infrastructure, it is important to go back to basics and not fall for cloud vendor's propaganda
    - selcuka 4y ago
      > serve entire world and doesn't need to spin up and shutdown VMs to meet global customer demand? You said it yourself. Not all applications need to serve entire world, therefore demand will be lower when people go to sleep. Even with global applications there are regulations that require you to host applications and data in specific regions. Imagine an application that is used by customers in Europe, Australia, and the US. They need to be served from regional data centres, and one cluster will be mostly sleeping when the other is running (because of timezones). With dedicated servers you would waste 60-70% of your resources, while using something like EC2/Fargate you can scale down to almost 0 when your region is sleeping.
Akronymus 4y ago
> but highly scalable IME aiming for scalability is exceedingly wrong for most services/applications/whatever. And usually you pay so much overhead for the "scalable" solutions, that you then need to scale to make up for it.
jwr 4y ago
I think one of the reasons is that people confuse physical servers with manual administration. As I said, I do not do manual administration. Nothing ever gets configured on any server by hand. All administration is through ansible. I only have one ansible setup, and it can work both for virtualized servers and physical ones. No difference. The only difference is that virtualized servers need to be set up with terraform first, and physical ones need to be ordered first and their IPs entered into a configuration file (inventory). Of course, I am also careful to avoid becoming dependent on many other cloud services. For example, I use VpnCloud (https://github.com/dswd/vpncloud https://github.com/dswd/vpncloud) for communication between the servers. As a side benefit, this also gives me the flexibility to switch to any infrastructure provider at any time. My main point was that while virtualized offerings do have their uses, there is a (huge) gap between a $10/month hobby VPS and a company with exploding-growth B2C business. Most new businesses actually fall into that gap: you do not expect hockey-stick exponential growth in a profitable B2B SaaS. That's where you should question the usual default choice of "use AWS". I care about my COGS and my margins, so I look at this choice very carefully.
graymatters 4y ago
There are several reasons for that: - lack of knowledge how to maintain own servers on premises - MASSIVE PR by a handful of top tier cloud providers to instill the notion that cloud is the only solution, the best one and most advanced technologically and attractive silk set wise - hockey stick growth expectations of (past days) startups from investors, forming the mindset that it must be met from day one in deployment - “no one got fired for buying IBM” managers/tech leaders mentality, where IBM is replaced with “cloud” - difficulty obtaining true costs information of cloud vs on premises deployment. For example - a team of ~16 people already pays off the costs of dedicated sysadmin to maintain on premises servers - etc.

sinenomine 4y ago

> Even the mere suggestion of renting a dedicated server provokes scorn from devops teams. There is a method to the madness, here it is called "job-security-driven development".

dpweb 4y ago

Its big companies - their priorities are different. Saving a couple million dollars isn't worth the tradeoff of not being industry standrd (AWS, Google, MS). They hire tech consulting companies who sell their product, large deployments.

adrianmsmith 4y ago

I think it might be due to the incentives - the costs are borne by the company (thus irrelevant to the employee), however putting new cool technologies on their CV is very relevant to the employee.

bell-cot 4y ago
> ...company...employee... Yes - but stupid yearnings to "do what all the cool kids are doing now" are at least as strong in those who would normally be referred to as "managers", vs. "employees".
- adrianmsmith 4y ago
  Good point - and in the case of managers there's also the "nobody got fired for buying IBM" effect: outcome A) huge microservices/cloud spend goes wrong, well at least you were following best practices, these things happen, what can you do. outcome B) you went with a Hetzner server and something went wrong, well you are a fool and should have gone with microservices, enjoy looking for a new job. Thus encouraging managers to choose microservices/cloud. It might not be the right decision for the company, but it's the right decision for the manager, and it's the manager making the decision.
rr888 4y ago
Agreed. I love our physical in house servers, but not knowing AWS in depth really hurt my last job search.
- adrianmsmith 4y ago
  Yeah that's the problem. It's not just a "devs are idiots, they should think of their company and not be obsessed with their CV" - not having worked with the latest trendy tech has a very real cost for developers. (As does being a consultant wanting an extension and writing software that works, as I found out the hard way.)

onlyrealcuzzo 4y ago

> I really don't understand why the industry seems to have lost sight of this. There's a disconnect between founders and everyone else. Founders believe they're going to >10x every year. Reality is that they're 90% likely to fail. 90% of the time - you're fine failing on whatever. Some % of the 10% of the time you succeed you're still fine without the cloud - at least for several years of success, plenty of time to switch if ever necessary.

wongarsu 4y ago
Even if you 10x every year, going from year 1 to year 2 you switch from renting two 6-core servers to renting two 64 core servers, in year 3 you rent 20 servers, in year 4 you are renting a small room of racks in a colo space and add a server every other day (or you continue renting servers, if that's your style). Out of all the problems that arise from 10x-ing your organisation every year, this is a really minor one.
- ipaddr 4y ago
  The fear.. I'll need 20 servers in 3 years.. the reality: A 4 dollar vps is powerful enough until you hit millions and millions of customers.
  - l5870uoo9y 4y ago
    would prefer to spend 20-30-50 USD and get a managed deployment infrastructure with rollback, blue/green deployment, github integration and so. this setup can last a long time.
    - justsomehnguy 4y ago
      And all that to host what could be served by Hugo static site.
    - mring33621 4y ago
      I'd pay good money if you could truly teach me how to do that for recurring costs within a factor of 2 from cheap rental vps/box.

criley2 4y ago

I'll be the counter opinion for this. Let's say you're Sprocket Masters, and you have to run two front ends (a Sprocket Sales site and a Sprocket Repair site) and a backend connecting them. But ultimately your sprockets are made in your factory on-site, and the majority of your staff is related to the manufacture of sprockets. You're not a software company, fundamentally you make and sell Sprockets. The opinions here would be hire a big eng/IT staff to "buy and maintain servers" (PaaS is bad) and then likely "write a bunch of code yourself" (SaaS is bad) or whatever is currently popular here (last thread here it was "Pushing PHP over SFTP without Git, and you're silly if you need more" lol) But I believe businesses should do One Thing Well and avoid trying to compete (doing it manually) for things outside of their core competency. In this case, I would definitely think Sprocket Masters should not attempt to manage their own hardware and should rely on a provider to handle scaling, security, uptime, compliance, and all the little details. I also think their software should be bog-standard with as little in-house as possible. They're not a software shop and should be writing as little code as possible. Realistically Widget Masters could run these sites with a rather small staff unless they decided to do it all manually, in which case they'd probably need a lot larger staff.

lukeramsden 4y ago
You still have to do all that with cloud though - instead of firewalls and IP tables it’s VPCs, IAM, security groups, auto scaling groups, etc etc. The reason I think it’s still probably better for Sprocket Masters to go cloud is purely because those skills are easier to hire for.
arp242 4y ago
The core business for Sprocket Masters isn't really tech stuff: it's making sprockets. For those kind of businesses it makes a lot of sense; the tech stuff is required, but only in the sane sense that a phone line or electricity is. However, what I also see – and what I think the previous poster was talking about – are businesses where tech is at the core of the business, and there it often makes less sense, and instead of saving time it seems to cost time. There's a reason there are AWS experts: it's not trivial. "Real" servers also aren't trivial, but also not necessarily harder than cloud services.

esskay 4y ago

A lot of its the buzz. 'Cloud Hosting' is something everyone thinks they need, despite not grasping why, or that the chances are they arent getting cloud hosting at all, just a VPS. A solid dedicated server is 99% of the time far more useful than a crippled VPS on shared hardware but it obviously comes at an increased cost if you dont need all the resources they provide.

xnx 4y ago

devops people want things to be expensive and complicated to justify their salaries

k8sToGo 4y ago
Nice generalization. So far I haven’t noticed that if I spend more for the company that I also get paid more.
DrBenCarson 4y ago
Remember how Twitter lost 80% of its staff and still kept running?
- baq 4y ago
  And Twitter ceo turned his account private because they can’t figure out what the algorithm does
- wkat4242 4y ago
  For now... There's a lot of inertia obviously.
dsr_ 4y ago
Anybody with experience on the ops side of that is absolutely in favor of simple and easy to understand -- and that often gets you cheap as a side effect. Source: almost thirty years of ops.
sshine 4y ago
Having talked to AWS evangelists, it seems that they truly believe it cuts costs. In their reality this is true, because an alternative with the same guarantees and risk costs more. The implicit disagreement is that you often don’t need all those guarantees.
- marcosdumay 4y ago
  > because an alternative with the same guarantees and risk costs more I really doubt this. The one theme you see on almost every AWS proponent is some high amount of delusion about what guarantees AWS actually provides you.

cameronh90 4y ago

If you're ever trying to get big enterprise contracts, the kind of scalability, compliance and disaster recovery that providers like AWS/Azure enable are now just table stakes. Sure, you can get something working on Hetzner but be prepared to answer a lot more questions.

axelthegerman 4y ago
Yes fair point though I doubt some of these super distributed, scalable apps are able to properly execute a disaster recovery... And just because all the serverless components scale "infinitely" doesn't mean the patchwork stitching them together is. Agreed on your last point of enterprise asking for this which again is just sad that these business "requirements" dictate how to architect and host your software when another way might be the much better one
jwr 4y ago
I run a self-funded SaaS business, so I only deal with real problems, not imaginary ones. Disaster recovery is a very real problem. Which is why I test it regularly. In a scorched-earth scenario, I can be up and running on a secondary (staging) system or a terraformed cloud system in less than an hour. For my business, that is enough.

alex7734 4y ago

In addition to what has already been said in the other comments, devs have an incentive to push cloud tech against the interest of their employers so they can put the experience in their resume, since cloud tech is seen as more specialized and in higher regard and as such a dev experienced in it commands a higher salary. Plus who doesn't want to play with the newest, coolest toy on another's dime?

corford 4y ago
In my experience it's often the other way around. CTO/mgmt/HR would rather swallow the perf/cost hit of Cloud because it makes auditing, compliance, support, dr, security, training and hiring easier. Not to mention it usually provides much more flexibility during the PMF stage of a startup (where product roadmap, team talent/bandwidth and execution speed are often much more important drivers than the raw perf/cost ratio of your infra). There's definitely an argument to move (some) stuff off cloud later in the journey when flexibility (or dealing with flux/pivoting) becomes less of a primary driver and scale/cost start dominating.
- margorczynski 4y ago
  "It's insane not to start your startup on the Cloud and even more so to stay on it" paraphrasing something I think I've read on HN some time ago. But I agree, it's best to treat the Cloud as an intermediate solution and once you stabilize and a clearer picture about requirements is available move at least partially to on-prem (especially the more expensive stuff like compute, storage is I think fine)
kbrannigan 4y ago
RDD or Resume Driven Development
- rqtwteye 4y ago
  Realistically RDD is a very sane strategy for employees. When I read my company's job ads a few years ago I had serious doubts if I would get hired by them because my resume didn't have the buzzwords they wanted. Since then I always sprinkle in some new tech with each new project, not because it makes sense for the company, but for my resume and future employability.

mariusmg 4y ago

>I really don't understand why the industry seems to have lost sight of this. Because the industry is full of people who are chasing trends and keywords and to which the most important thing is to add those keywords to the CVs.

macNchz 4y ago

The dedicated server deployments I worked on at smallish software companies 10+ years ago wound up being really annoying. I enjoy sysadmin type stuff and this idea has tempted me, but I think it’s a false economy in most cases. The incremental cost of being on a cloud is totally worth it to me to have managed databases, automatic snapshots, hosted load balancers, plug and play block storage etc. I want to worry about whether our product is good, not about middle of the night hardware failures, spinning up a new server and having it not work right because there was no incentive to use configuration management with a single box, having a single runaway task cause OOM or full disk and break everything instead of just that task’s VM, fear of restarting a machine that has been up 1000 days etc.

pas 4y ago
Aaand that's why k8s seems like a good step in the right direction. It has all - okay, absolutely not all, but a good number of the important ones - of these concepts "built-in". Sure, the actual pile of Go(o) that drives it can be improved (and it is indeed improving). That said, the hard truth is that your approach is the correct one, almost all businesses (startups!) overbuild instead of focusing on providing value, identifying the actual niche, etc. (Yes, there's also the demand side of this, as the VC money inflated startups overbuild they want to depend on 3rd parties that can take the load, scale with them, SLAs and whatnot must be advertised.)
- wkat4242 4y ago
  I don't agree, Ansible would be a much saner approach for a few dedicated boxes than k8s
  - riku_iki 4y ago
    What about deploying prebuilt docker image on dedicated server?
    - allarm 4y ago
      What about it? It can be easily done with Ansible. I managed a distributed infra with containers orchestrated by Ansible, worked fine.

Slartie 4y ago

"No one ever got fired for buying AWS"

croo 4y ago
You mean Oracle. That's another full circle right here...

avereveard 4y ago

if you need durability in the cloud you just pick any object storage service from the 3 bigs, and a db with multiple deployment with automatic replication and managed failover. getting the equivalent reliability with irons is a lot more expensive than renting "two dedicated servers" - now you might be fine with one server and a backup solution, and that's fair. but a sysad to create all that, even on a short contract for the initial setup and no maintenance, is going to go well beyond the cloud price difference, especially if there's a database in the mix and you care about that data.

phpnode 4y ago
I'm not saying there's no place for cloud, but that many workloads benefit from using dedicated hardware. It is often much simpler to deploy to a dedicated server rather than spending months rearchitecting your system (or even worse, cutting customer facing features!) to efficiently make use of many small cloud instances.

l5870uoo9y 4y ago

> trivially solved with one or two dedicated servers... For many even lot less than that. I run a small side project[1] that went viral a few times (30K views in 24 hours or so) and it is running on a single core CPU web server and a managed Postgres likewise on a single CPU core. It hasn't even been close to full utilization. 1: https://aihelperbot.com/ https://aihelperbot.com/

deleted 4y ago
[deleted]

jerf 4y ago

I am responsible for about 10-12 t3.medium-class servers (number includes redundant servers) and the services running on them. All of them are fairly overprovisioned for what they are, excepting that the monitoring system we use on them for some reason likes to consume about half-a-gig of RAM so I can't run on even smaller servers. I write my services in Go, so the CPU isn't sitting there spending 98% of its time chasing pointers and can use multiple CPUs as needed for its heavy-hitting tasks. It's a rough day for those servers when they pass 5% average CPU for the day. Could I switch some of them to lambda functions? Or switch to ECS? Or switch to some other cloud service du jour? Maybe. But the amount of time I spent writing this comment is already about six month's worth of savings for such a switch. If it's any more difficult than "push button, receive 100% reliably-translated service", it's hard to justify. Some of this is also because the cloud does provide some other services now that enable this sort of thing. I don't need to run a Kafka cluster for basic messaging, they all come with a message bus. I use that. I use the hosted DB options. I use S3 or equivalent, etc. I find what's left over is almost hardly worth the hassle of trying to jam myself into some other paradigm when I'm paying single-digit dollars a month to just run on an EC2 instance. It is absolutely the case that not everyone or every project can do this. I'm not stuck to this as an end-goal. When it doesn't work I immediately take appropriate scaling action. I'm not suggesting that you go rearchitect anything based on this. I'm just saying, it's not an option to be despised. It has a lot of flexibility in it, and the billing is quite consistent (or, to put it another way, the fact that if I suddenly have 50x the traffic, my system starts choking and sputtering noticeably rather than simply charging me hundreds of dollars more is a feature to me rather than a bug), and you are generally not stretching yourself to contort into some paradigm that is convenient for some cloud service but may not be convenient for you.

FlyingSnake 4y ago

The allure of our industry to mime FAANG leaders might be one of the reasons we're at this stage. Most products could be safely run on simpler setups. I suspect that VendorOps and complex tools like kubernetes are favored by complexity merchants which have arisen in the past decade. It looks great on resume and gives tech leaders a false sense of achievement. Meanwhile Stackoverflow, which is arguably more important than most startups, is chugging along on dedicated machines. 1: https://stackoverflow.blog/2016/02/17/stack-overflow-the-architecture-2016-edition/ https://stackoverflow.blog/2016/02/17/stack-overflow-the-arc...

dgb23 4y ago
Software development and related things are known to have fantastic often free online resources for autodidacts. It seems like the trend in this space is to jump directly to the highest layer of abstraction. Skipping the fundamentals and point to $buzzword tools, libraries and products. You see glimpses of this in different forms. One is social media threads which ask stuff like “How much of X do I need to know to learn Y?” Where X is a fundamental, possibly very stable technology or field and Y is some tool du jour or straight up a product or service.
failbuffer 4y ago
"complexity merchants" - I love this term. It aptly describes beneficiaries of earlier IT fads. Remember UML? Remember WS-*? One could come up with a long list of technologies that were a bonanza for billing consultant hours but no so great for getting to solutions.
- switchbak 4y ago
  The debacle that was J2EE is hard to top (but I'm sure some have!). I still have battle scars from OSGi. CORBA belongs there. And perhaps even the semantic web stuff. Definitely XML!
  - 77pt77 4y ago
    SOAP, don't forget SOAP.

raducu 4y ago

Anybody tried mixing cloud + self hosted k8s on hetzner or is it too complicated to do?

icedchai 4y ago
You'd probably have some latency issues, plus waste a ton of $$$ on the bandwidth egress fees.
ilrwbwrkhv 4y ago
You've already lost if you install k8s.

7952 4y ago

It feels like artificial scarcity similar to what you see in high fixed cost monopolies. And that is exactly what most corporate IT departments are.

tstrimple 4y ago

I feel like many people use the cloud "wrong". If you're just standing up VMs then yeah, it's going to be slower and more expensive than renting a physical server. If you're just going to setup VMs and pretend you're still on-prem you're better off not switching. But I've got projects setup on PaaS services that literally cost a few bucks a month because it's only charging me for actual use and not idle time. I've got some small customers (tens of thousands of users) who I've helped move into the cloud who see the same thing. They have SPA websites backed by lambda functions and on-demand databases. If their website isn't being hit, it's not costing them anything at all. The only lock-in is the boiler plate code around the lambda activation which could be switched fairly easily to target any Functions as a Service platform.

woadwarrior01 4y ago
I'm curious to know what on-demand databases do you use with AWS Lambdas? Two options that I've evaluated are: Aurora Serverless v2, which isn't quite serverless in practice and DynamoDB. And the mental model for DynamoDB very different from that of traditional SQL databases.
- tstrimple 4y ago
  I’m actually primarily building on Azure with Cosmos DB. The mental model is very different, but I think important if you want to build cheap cloud services. I think that’s the larger point. You have to think about doing things differently in the cloud otherwise you’re just deploying and managing VMs as if you were on-prem albeit with more scaling control. It feels like everyone who is complaining about how expensive cloud hosting is go into it thinking it’s just someone else hosting the VMs where as a VM is the option of last resort for me.

znpy 4y ago

> Even the mere suggestion of renting a dedicated server provokes scorn from devops teams. Have you ever had to manage one of those environments? The thing is, if you want to get some basic more-than-one-person scalability and proper devops then you have to overprovision by a significant factor (possibly voiding your savings). You're inevitably going to end up with a bespoke solution, which means new joiners will have a harder time getting mastery of the system and significant people leaving the company will bring their intimate knowledge of your infrastructure with them. You're back to pets instead of cattles. Some servers are special, after a while automation means "a lot of glue shell scripts here and there" and an OS upgrade means either half infra is KO for a while or you don't do OS upgrades at all. And in the fortunate case you need to scale up... You might find unpleasant surprises. And don't ever get me started on the networking side. Unless you're renting your whole rack and placing your own networking hardware, you get what you get. Which could be very poor in either functionalities or performances... Assuming you're not doing anything fancy.

lelanthran 4y ago
> The thing is, if you want to get some basic more-than-one-person scalability and proper devops then you have to overprovision by a significant factor If you want 100.0000% uptime, sure. But you don't usually. The companies that want that kind of uptime normally has teams dedicated to it anyway. And scaling works well on bare-metal too if you scale vertically - have you any idea the amount of power and throughput you can get from a single server? It's concerning to keep hearing about "scaling" when the speaker means "horizontal scaling". If you requirements are "scaling", then vertical scaling will take you far. If your requirements are "horizontal scaling on demand", then, sure, cloud providers will help there. But, TBH, few places need that sort of scaling.
- baq 4y ago
  Just getting a highly available storage with any kind of performance requires you to overprovision by something like 12x (3 boxes with raid 1+0) which is frankly insane. And then you have to manage it and fix it when it inevitably breaks. The team who’s supposed to do that will be swamped by trying to keep the other 15 infrastructure services running and upgraded to a version which is only barely out of vendor support.

barbazoo 4y ago

As a developer I appreciate the compartmentalization, being able to make truly isolated changes. This is something that's not trivial for lots of folks, me included. I guess if one has the expertise in house to manage a K8s cluster on a raw machine, that'd be an option but I, personally, am not gonna touch that with a ten foot pole. So yeah with a big and capable enough infrastructure team that complexity could be abstracted away but in reality I (where I've worked) haven't seen that yet.

codegeek 4y ago

Because cloud is the new "No one gets fired for using IBM". If you are using dedicated servers somewhere, you cannot have the same reputation of say Using AWS for cloud. Everyone knows the AWSs, Google Cloud etc and no one gets fired for selecting/using them. I personally have run Dedicated Servers for our business in the earlier days but as we expanded and scaled, it became a lot easier to go with the Cloud providers to provision various services quickly even though the costs went up. Not to mention it is a lot easier to tell a prospective customer that you use "cloud like AWS" than "Oh we rent these machines in a data center run by some company" (which can actually be better but customers mostly wont get that). Audit, Compliance and others.

jgalt212 4y ago

> Even the mere suggestion of renting a dedicated server provokes scorn from devops teams. Because it's a threat to their jobs.

swatcoder 4y ago

The industry lost sight of a lot of things. We went through a growth boom, and like all of them before, it meant there were lots of inexperienced people being handed lots of money and urgent expectations. It’s a recipe for cargo culting and exploitative marketing. But growth is slowing and money is getting more expensive, so we’ll slow down and start to re-learn the old lessons with exciting new variations. (Like Here: managing bare metal scaling and with containers and orchestration) And the whole cycle will repeat in the next boom. That’s our industry for now.

deleted 4y ago
[deleted]

Spooky23 4y ago

Growth hides sins. I ran a large self-funded enterprise service at a loss for 4-5 years, which is possible if you are growing 30%. Today, cloud is similar - the time to market is quicker as there are less moving parts. When the economy tanks and growth slows, the beancounters come in and do their thing. It happens every time.

itomato 4y ago

I just want to go back to leasing an always-available Opteron with optional remote hands.

slt2021 4y ago

the most funny thing is that cloud engineering teams have no problem of using EC2 hosts, but somehow don't want to rent dedicated servers/VPC from regular datacenter companies. This only makes AWS richer at the expense of companies and cloud teams

selcuka 4y ago
> the most funny thing is that cloud engineering teams have no problem of using EC2 hosts, but somehow don't want to rent dedicated servers/VPC from regular datacenter companies. It is trivial to provision and unprovision an EC2 instance automatically, within seconds if your deployment needs to scale up or scale down. That's what makes it fundamentally different from a bare metal server. Now, I'm not denying that it might still be more cost effective when compared to AWS to provision a few more dedicated servers than you'll need, but when you have really unpredictable workloads it's not easy to keep up.
- slt2021 4y ago
  the whole concept of a need to spin up more VM instances to meet increased load is flawed and is driven by cloud vendors to make money. if you are spinning up and shutting down VMs to meet demand curve - something is seriously wrong with your architecture. Did it ever occur to you, how come stackoverflow uses ~8 dedicated servers to serve entire world and doesn't need to spin up and shutdown VMs to meet global customer demand? -- When planning compute infrastructure, it is important to go back to basics and not fall for cloud vendor's propaganda
  - selcuka 4y ago
    > serve entire world and doesn't need to spin up and shutdown VMs to meet global customer demand? You said it yourself. Not all applications need to serve entire world, therefore demand will be lower when people go to sleep. Even with global applications there are regulations that require you to host applications and data in specific regions. Imagine an application that is used by customers in Europe, Australia, and the US. They need to be served from regional data centres, and one cluster will be mostly sleeping when the other is running (because of timezones). With dedicated servers you would waste 60-70% of your resources, while using something like EC2/Fargate you can scale down to almost 0 when your region is sleeping.

Akronymus 4y ago

> but highly scalable IME aiming for scalability is exceedingly wrong for most services/applications/whatever. And usually you pay so much overhead for the "scalable" solutions, that you then need to scale to make up for it.

jwr 4y ago

I think one of the reasons is that people confuse physical servers with manual administration. As I said, I do not do manual administration. Nothing ever gets configured on any server by hand. All administration is through ansible. I only have one ansible setup, and it can work both for virtualized servers and physical ones. No difference. The only difference is that virtualized servers need to be set up with terraform first, and physical ones need to be ordered first and their IPs entered into a configuration file (inventory). Of course, I am also careful to avoid becoming dependent on many other cloud services. For example, I use VpnCloud (https://github.com/dswd/vpncloud https://github.com/dswd/vpncloud) for communication between the servers. As a side benefit, this also gives me the flexibility to switch to any infrastructure provider at any time. My main point was that while virtualized offerings do have their uses, there is a (huge) gap between a $10/month hobby VPS and a company with exploding-growth B2C business. Most new businesses actually fall into that gap: you do not expect hockey-stick exponential growth in a profitable B2B SaaS. That's where you should question the usual default choice of "use AWS". I care about my COGS and my margins, so I look at this choice very carefully.

graymatters 4y ago

There are several reasons for that: - lack of knowledge how to maintain own servers on premises - MASSIVE PR by a handful of top tier cloud providers to instill the notion that cloud is the only solution, the best one and most advanced technologically and attractive silk set wise - hockey stick growth expectations of (past days) startups from investors, forming the mindset that it must be met from day one in deployment - “no one got fired for buying IBM” managers/tech leaders mentality, where IBM is replaced with “cloud” - difficulty obtaining true costs information of cloud vs on premises deployment. For example - a team of ~16 people already pays off the costs of dedicated sysadmin to maintain on premises servers - etc.