17 ms·

Does your startup need complex cloud infrastructure?

• pillefitz 2y ago
  A compromise people seem to overlook: Use a single Lambda with internal routing.
  • cynicalsecurity 2y ago
    I'm sorry, a single Lambda for what exactly?
  • icedchai 2y ago
    This is my preferred approach for lambdas. A larger Lambda that handles URL routing on the "API" level instead of individual endpoint level.
  • ukuina 2y ago
    Is this using the Lambda as your entire service?
• threeseed 2y ago
  The classic HN catnip blog post: 1. New technology is bad and has no merit other than for resume. 2. Use old technology that I am comfortable with. 3. Insist that everyone should use old technology.
  • ghomem 2y ago
    Fake dichotomy. It is not old vs new, it is simple vs complex. The fact the older technology is simpler is just a coincidence.
    • mexicocitinluez 2y ago
      > The fact that older technology is simpler hahahahahahahahahaha. Yes, back in the days when all you could do on a website is read the text. Old technology was 1000% not simpler. What an insane & absolute statement to make in an enormous field just because you can't make a solid argument.
• datavirtue 2y ago
  It's still a badge of honor, bragging rights, for executives to declare that all their tech is in the cloud. Once this wears off we will get our fucking bare metal back.
  • marcosdumay 2y ago
    This is quickly turning into a bozo badge. (Even Gartner will say so.) So it's a risky thing to brag about right now.
    • datavirtue 2y ago
      I can't wait for the ooohhs and aaahhs when people start "going hybrid."
• hjaveed 2y ago
  After listening to @levelsio on Lex Friedman’s podcast, I became obsessed to simplify my deployments: Do startups really need complex cloud architecture? Inspired, I wrote a blog exploring simpler approaches and created a docker-compose template for deployment Curious to know your thoughts on how you manage your infrastructure. How do you simplify it? How do you balance?
  • KronisLV 2y ago
    > Curious to know your thoughts on how you manage your infrastructure. What I quite like about your repo: - there is a separate API and background job instance - there is a separate web image, to not always couple front end deployments to back end - there are specialized data stores like Redis (or maybe RabbitMQ or MinIO in a different type of project) - Dozzle seems nice https://dozzle.dev/ (I use Portainer mostly, but seems useful) What I think works quite nicely in general: - starting out with a monolithic back end but making it modular with feature flags (e.g. FEATURE_REPORTS, FEATURE_EMAILS, FEATURE_API), so that you can deploy vastly different types of workloads in separate containers BUT not duplicate your data model and don't need to extract shared code libraries (yet) and if you ever need to split the codebase into multiple separate ones, then it won't be *too* hard to do that - having a clear API (RESTful or otherwise) as the contract between a separate back end and front end deployment, so that even if your SPA technology gets deprecated (AngularJS, anyone?) then you can migrate to something, unlike when doing SSR and everything being coupled - the same applies to NOT having the same container build process have both the front end and back end build (I've seen a Java project install a specific Node version through Maven and then the build dragging on cause Maven ends up processing thousands of files as a part of the build) - using the right tool for the job: many might create full text search, key-value storage, message queues, JSON document storage, even blob storage all with PostgreSQL and that might be okay; others will go for separate instances of ElasticSearch, Redis, RabbitMQ, something S3 compatible and so on, probably a tradeoff between using well known libraries and tools vs building everything yourself against a single DB instance - in my experience, many projects out there are served perfectly fine by a single server so Docker Compose feels like the logical tool to start out with, if multiple instances indeed become necessary, there is always Docker Swarm (yes, still works, very simple), Hashicorp Nomad or K3s or one of the other more manageable Kubernetes distros - self-hosted (or self-hostable) software in general is pretty cool and gives you a bunch of freedom, though using managed cloud services will also be pleasant for many, more expensive upfront but less so in regards to your own time spent managing the stack; the former also lends itself nicely to being able to launch a local dev environment with the full stack, which feels like a superpower (being able to really test out breaking migrations, look at what happens with the whole stack etc.) - having some APM and tracing is nice, something like Apache Skywalking was pretty simple to setup, though there are more advanced options out there (e.g. cloud version of Sentry, because good luck running that locally) - having some uptime monitoring is also very nice, something like Uptime Kuma is just very pleasant to use - heck, if you really wanted to, you could even self-host a mail server: https://github.com/docker-mailserver/docker-mailserver (though that can be viewed as a hobbyist thing), or have MailCatcher / Inbucket or something for development locally
• mianos 2y ago
  Who is going to get a new job without k8s on their resume. :) Seriously, I think a lot of people do things the hard way to learn large scale infrastructure. Another common reason is 'things will be much easier when we scale to a massive number of clients', or we can dynamically scale up on demand. These are all valid to the people building this, just not as much to founders or professional CTOs.
  • sussexby 2y ago
    Just take a look at the level of complexity in home lab subreddits! I don’t quite get if people do it for interest, for love of the tech, or if they are technocratic and believe in levelling up their skill to get k8s on their CV like you say. All I think is “this looks painful to manage”!
    • from-nibly 2y ago
      K8s is painful to get started, and painful to learn. But once you have it up you can just keep adding stuff to it. I run a k8s cluster at home. Part of it yes, is to apply my existing skills and keep them fresh. But part of it is that kubernetes can be easier long term. Ive got magical hard drive storage with rook ceph. I can yoink a hard drive out of my servers and nothing happens to my workloads. I can do maintenance on one of the servers with 0 down time. All of my config for what I have deployed is in git. I manage VMS and kubernetes at work, and im not going to pretend that kubernetes isnt complex, but it's complex up front instead of down the road. VMs run into complexity when things change. I'm sure you can make VMS good but then why not use something like kubernetes, you will have to reinvent a lot of the stuff that's already in kubernetes. It's a hammer for sure and not everything is a nail, but it can be really powerful and useful even for home labs.
      • catdog 2y ago
        What you also can do is starting with just a single node, incredibly easy to install with e.g. https://k3s.io/ https://k3s.io/. You still have to invest the upfront effort to understand how it works but you can already reap a lot of benefits with a lot less complexity. Kubernetes does not force you into the distributed systems hell, you can go that route later, or never.
• qrush 2y ago
  Betteridge's law of headlines is an adage that states: "Any headline that ends in a question mark can be answered by the word no."
  • marcosdumay 2y ago
    But this one is phishing for a "no". That law explodes in contact with those.
• etaioinshrdlu 2y ago
  In my experience, having done it both ways, first on VM's, then on lots of fully or mostly managed services, I generally prefer the latter because systems tend to be a lot more "self-healing" - because they're someone elses responsibility. This has had a dramatic effect on improving my sanity and sleeping well at night. I only wish I could migrate to an even more fully managed stack that's more reliable and still less work. The cases where I haven't been able to are either too expensive or would be too difficult to migrate.
• maximumgeek 2y ago
  Simple answer. NO. Everyone is building like they are the next Facebook or Google. To be honest, if you get to that point, you will have the money to rebuild the environment. But, a startup should go with simple. I miss the days when RAILS was king just for this reason. The added complexity is overkill. Just keep it simple. Simple to deploy, simple to maintain, simple to test, etc. Sounds silly, but in the long run, it works.
• dboreham 2y ago
  Interesting. Expecting to read things I'd object to. But this is basically what I do, at least for smaller setups.
• buglungtung 2y ago
  I agree that we are overthinking about infrastructure. Boring stack like traditional RDMS, single server with regular backup, few bash script for deployment is fine for normal startup that targets to non-tech customer. They will serve you well at least one or two years, then you will know what should be improve. One of the big surprise is database like PostgreSQL can handle like 100tps very well with cheap hardware cost. That mean you can handle up to 86 millions transaction per day.
• csolorio 2y ago
  I've used Kamal for side projects and startups. Easy to deploy, simple commands for logging and configurable. Downside is its a one to one system. But I just use downsized servers.
• auspex 2y ago
  Honestly what you need: vulnerability scanning of your images. Fargate RDS
  • bigiain 2y ago
    I dunno. I've seen a _lot_ of business ideas fail, which could have much less expensively failed using PHP and MySQL on shared cPanel hosting than they did using AWS/Azure/GCP. Yeah, that won't scale to a million QPS, or even 10 QPS. But way more businesses fail because they never achieve 100 Queries Per Day, instead of failing because they fell over at 10 or 1,000 or 1,000,000 QPS. I mean, hell, Twitter (back in the day) was famous for The Fail Whale. Getting enough traffic is harder and more important than your "web scale architecture" for your startup. Making actual cash money off your traffic is harder and more important than your "web scale architecture" (ideally by selling them something they want, but making cash money through advertising or by impressing VCs with stories of growth and future value counts too). There is precisely _zero_ chance that if you ever get within 2 or 3 orders of magnitude of "a million QPS" - that the code you and your cofounder wrote won't have been completely thrown away and rewritten by the 20 or 100 person engineering department that is now supporting your "1000 QPS" business.
• sussexby 2y ago
  I think this goes for any technology group with any stage of company. I work in networking and genuinely of the product I sell, my customers only need a small amount of core functionality and default settings - the rest is “bells and whistles”. But still, no matter what, the odd customer demands they need all these complexities turned on for no discernible reason. IMO it’s a far better approach with any platform to deploy the minimum and turn things on if you need to as you develop. Incidentally, I’ve been exposed to “traditional” cloud platforms (Azure, GCP, AWS) through work and tried a few times to use them for personal projects in recent years and get bewildered by the number of toggles in the interface and strange (to me) paradigms. I recently tried Cloudflare Workers as a test of an idea and was surprised how simple it was.
• justinclift 2y ago
  > ... and Docker Swarm was deprecated.. I thought the same thing until recently. Apparently there's a "Docker Swarm version 2" around, and it was the original (version 1) Docker Swarm that was deprecated: https://docs.docker.com/engine/swarm/ https://docs.docker.com/engine/swarm/ Do not confuse Docker Swarm mode with Docker Classic Swarm which is no longer actively developed. Haven't personally tried out the version 2 Docker Swarm yet, but it might be worth a look at. :)
  • KronisLV 2y ago
    I use Swarm with Portainer, it’s quite a nice experience!
  • gawa 2y ago
    Yes, swarm is not deprecated. I haven't used it myself yet, but I read elsewhere that swarm offers an easy way to manage secrets with containers. Some people run their 1 container in a swarm cluster with 1 node just for this feature. I see it's even officially suggested as a Note in the doc: > Docker secrets are only available to swarm services, not to standalone containers. To use this feature, *consider adapting your container to run as a service. Stateful containers can typically run with a scale of 1 without changing the container code.* (Emphasis mine. From https://docs.docker.com/engine/swarm/secrets/ https://docs.docker.com/engine/swarm/secrets/ )
• mattbillenstein 2y ago
  Basically doing this for a small startup - there are some complexities around autoscaling task queues with gpus and whatnot, but the heart of it is on a single VM (nginx, webapp, postgres, redis). We're b2b, so there's very little traffic anyway. The additional benefit is devs can run all the same stuff on a Linux laptop (or Linux VM on some other platform) - and everyone can have their own VM in the cloud if they like to demo or test stuff using all the same setup. Bootstrapping a new system is checking in their ssh key and running a shell script. Easy to debug, not complex or expensive, and we could vertically scale it all quite a ways before needing to scale horizontally. It's not for everyone, but seed stage and earlier - totally appropriate imo.
  • teaearlgraycold 2y ago
    I did this type of setup but without even redis. Postgres can do anything.
    • mattbillenstein 2y ago
      True, I use it mainly for a few convenience things - holding ephemeral monitoring data, distributed locks, redis streams for some pub/sub stuff, sorted sets can be handy - things I could do in Postgres, but are a bit simpler in Redis.
  • normie3000 2y ago
    I love the simplicity of this approach. In your setup, how do you track config and updates of your VMs?
    • clvx 2y ago
      I like this but one of the issues with this approach is if no Docker images like traditional configuration management tool, you are going for a world of pain. Docker and Docker images have tons of best practices already defined for plenty of use cases. If it's already containerized; then, jumping to any orchestrator that supports OCI images is more about adjusting the business to a new set of operations.
• pnathan 2y ago
  if you take the time to understand k8s and have a straightforward k8s deployment, these things aren't really a problem - and you don't have to do the custom sysadmin timesinks that need to go into the "simple" suggestion. What is suggested here is "easy". But it is not simple: it proliferates custom work. I have had great success with a very simple kube deployment: - GKE (EKS works well but requires adding an autoscaler tool) - Grafana + Loki + Prometheus for logs + metrics - cert-manager for SSL - nginx-ingress for routing - external-dns for autosetup DNS I manage these with helm. I might, one day, get around to using the Prometheus Operator thing, but it doesn't seem to do anything for me except add a layer of hassle. New deployments of my software roll out nicely. If I need to scale, cut a branch for testing, I roll into a new namespace easily, with TLS autosetup, DNS autosetup, logging to GCP bucket... no problem. I've done the "roll out an easy node and run" thing before, and I regret it, badly, because the back half of the project was wrangling all these stupid little operational things that are a helm install away on k8s. So if you're doing a startup: roll out a nice simple k8s deployment, don't muck it up with controllers, operators, service meshes, auto cicds, gitops, etc. *KISS*. If you're trying to spin a number of small products: just use the same cluster with different DNS. (note: if this seems particularly appealing to you, reach out, I'm happy to talk. This is a very straightforward toolset that has lasted me years and years, and I don't anticipate having to change it much for a while)
  • catdog 2y ago
    > I manage these with helm. I might, one day, get around to using the Prometheus Operator thing, but it doesn't seem to do anything for me except add a layer of hassle. One big advantage of the operator is that its custom resources are practically kind of standard by now. This means helm charts for a lot of software ship those and integrating that piece of software into your monitoring is a matter of setting a few flags to true. The go to solution for a k8s monitoring setup is https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack/ https://github.com/prometheus-community/helm-charts/tree/mai...
    • pnathan 2y ago
      yeah, I know, that's the only reason I'm even thinking of using it. but tbqh I don't really install many things, as you can see...
• Animats 2y ago
  But it's so embarrassing if your startup is running on shared hosting, FCGI, Go programs, and MySQL, costing about $10 per month.
  • groestl 2y ago
    You immediately see there's no load ;)
    • Animats 2y ago
      That's not a joke. Go is a fast compiled language, and Go programs are self-contained executables. So you don't need containers. FCGI is an orchestration system, like Kubernetes. It's single-machine, but will start up and shut down processes as the load changes. A crashed process will be restarted. Host the web pages on a static page server, and use client-side Javascript for any dynamic stuff. Good for maybe 20-100 transactions per second. The database will be the bottleneck. Boring, but useful.
      • groestl 2y ago
        > 20-100 transactions per second In all seriousness, that is "no load". I know it fits 99% of all startups, and many larger companies too, but that's kind of the point. I wouldn't do it differently though, I think it's a perfectly fine architecture :)
        • t-writescode 2y ago
          > > 20-100 transactions per second > "no load" Ruby on Rails applications with even a modest amount of ActiveRecord work would like a word xD
          • sethammons 2y ago
            Couple thousand per second is expected on my Go services (per node) before any optimizations.
          • 2y ago
• akira2501 2y ago
  > 20-30 Lambda functions for different services Yes. This is the basis of privilege separation and differential rollouts. If you collapse all this down into a single server or even lambda you lose that. Once your service sees load you will want this badly. > SQS and various background jobs backed by Lambda Yes. This is the basis of serverless. The failure of one server is no longer a material concern to your operation. Well done you. > Logs scattered across CloudWatch Okay. I can't lie. CloudWatch is dogturds. There is no part of the service that is redeemable. I created a DyanmoDB table and created a library which puts log lines collected into "task records" into the table paritioned by lambda name and sorted by record time. Each lambda can configure the logging environment or use default which include a log entry expiration time. Then I created a command line utility which can query and or "tail" this table. This work took me 3 days. It's paid off 1000x fold since I did it. You do sometimes have to roll your own out here. CloudWatch is strictly about logging cold start times now. > Could this have been simplified to a single NodeJS container or Python Flask/FastAPI app with Redis for background tasks? Absolutely. Could this have been simplified into something far more fragile than what is described? Absolutely. Why you'd want this is entirely beyond me.
  • throwawaythekey 2y ago
    I agree that cloudwatch is dogturds, but want to dive deeper for illustrative purposes: Your dynamodb solution isn't foolproof. It has throughput limited to the partition granularity -> in your case the lambda name. It's also relatively expensive and fairly slow to query in bulk (DDB is designed for OLTP). I don't have direct experience here, but I expect slapping grafana on top of any disk basked source is likely to be cheaper, faster, and have better ergonomics. Once your logging is too much for a disk to handle (this will be later than you would've outgrown ddb, but before you would've outgrown cloudwatch) then you can bring something fancy in.
    • akira2501 2y ago
      > has throughput limited The event volume is not particularly large as we tend to process things in batch and rarely on the edge of an event. I also wouldn't, for example, log API requests using this mechanism. We're nowhere near this being an issue as 20-30 lambdas is not a particular problem for us. Choose a good naming convention and build your own deployment infrastructure and it's no sweat. > relatively expensive Large object compression and/or offload to s3 is baked into our dynamodb interface library. Not that this matters as almost all log records end up being less than 4kb anyways. > slow to query in bulk Which is why time is part of the key. You're not often looking back more than an hour. There's bulk export back onto campus servers if you wanted that anyways. TTL is default 1 day. Running a "tail" is absurdly cheap, much cheaper than CloudWatch's laughable rate for their similar feature, a miss is 1/2 a read unit, and a hit is almost never more than 2. > slapping grafana I didn't need "observability." I need current state and recent deltas. This is particularly true when any changes are made. Otherwise my logs are pure annoyance and don't generally provide value. We optimized for the exceptionally narrow case we felt the cloud underserved in and left it at that.
• throwaway89988 2y ago
  There are only relative few startups or non startups which need complex infrastructure from a technical point of view... In reality, there is a strong bias in favor of complex cloud infrastructure: "We are a modern, native cloud company" "More people mean (startup/manager/...) is more important" "Needing an architect for the cloud first CRUD app means higher bills for customers" "Resume driven development" "Hype driven development" ... in a real sense, nearly everyone involved benefits from complex cloud infrastructure, where from a technical POV MySQL and PHP/Python/Ruby/Java are the correct choice. One of the many reasons more senior developers who care for their craft burn out in this field.
• sverhagen 2y ago
  Yes, use boring technology, I'm all for that. But an application built in the high pressure environment of a startup also has the risk of becoming unmanageable, one or two years in. And to the extent you already have familiar tools to manage this complexity, I vote for using them. If you can divide and conquer your application complexity into a few different services, and you are already experienced in an appropriate application framework, that may not be such a bad choice. It helps focus on just one part of the application, and have multiple people work on the separate parts without stepping on each other. I personally don't think that should include k8s. But ECS/Fargate with a simple build pipeline, all for that. "Complex" is the operative word in the article's title.
  • threeseed 2y ago
    But it's never just ECS/Fargate is it. It's ECR, S3, ALB, CF etc. And at that point you've assembled a stack just as complex as doing it all inside a single k8s cluster.
    • sverhagen 2y ago
      Hence I said "I personally" and "already have familiar tools". Also, if you're fair... not all those AWS acronyms you're listing would be displaced by the single k8s cluster. (Maybe you weren't arguing to swap out complexity, rather that the complexity floodgates were open already anyway?)
      • threeseed 2y ago
        You can absolutely run object store, container serving, front end load balancing etc from a single k8s cluster. Very common in fact since many k8s clusters are air-gapped except for a single inbound edge node.
    • ghomem 2y ago
      And if one of those services is down, your entire application is down. You basically build a server made of abstract components ECR, S3, ALB, CF , all of which are able to fail.
• jumploops 2y ago
  Simple is robust. Focus on product market fit (PMF) and keep things as straightforward as possible. Create a monolith, duplicate code, use a single RDBMS, adopt proven tech instead of the “hot new framework”, etc. The more simple the code, the easier it is to migrate/scale later on. Unnecessary complexity is the epitome of solving a problem that doesn’t exist.
  • dgan 2y ago
    Can you expand om what of code duplication you deem reasonable?
    • jumploops 2y ago
      Early in a project you see a lot of similar code paths, and so it’s often tempting to take the logic from two or three e.g. API routes and merge the “clean” abstraction into single piece of logic both routes can call. Over-time this “clean” abstraction adopts a bunch of optional parameters based on the upstream API routes, leaving you with an omni-function that is more convoluted, and thus harder to change, than if the API routes weren’t overly optimized from the get-go. As a personal rule, I’ll let myself copy something 3 times before taking a step back and figuring out a “better” way.
      • dgan 2y ago
        A very reasonable approach indeed
  • kaptainscarlet 2y ago
    Yeah, I would focus on a better user experience over a beautiful backend architecture.
    • ghomem 2y ago
      ... and this!
  • ghomem 2y ago
• OutOfHere 2y ago
  After reading all the comments here, the conclusion is to start simple, then switch to k8s and later to cloud-native only when your business has grown to 1000 and then 1 million daily customers respectively.
  • number6 2y ago
    We have B2B-Customers around 700. It all runs on a single Server (not VM though). Since it's B2B we don't need zero downtime, updates at midnight are all right. A day before rollout they go through the staging server and the test environment, so no surprises the next morning. Before updates, the backups kick in, so if we need to recover from a bad update we can roll back. Sounds all 2000 and not very fancy but boring and profitable cuts for us
    • k8sagainand 2y ago
      The question is if you have so much buffer that it doesn't matter or if you could do a lot more but you just don't know. My ci/cd is doing a system test because everything is in containers. I can do full e2e tests and automatic rollouts without a downtime. What i can do, can everyone else do when i'm on holiday. How fast are you back if your server burns down tomorrow? How often have you tested that? Are your devs waiting regularly on things?
      • number6 2y ago
        > The question is if you have so much buffer that it doesn't matter or if you could do a lot more but you just don't know. Yes, we collect server metrics - that's pretty old-school > How fast are you back if your server burns down tomorrow? How often have you tested that? 25 Minutes - we test it once a year and we have third partys to check it. It's called an audit. They also check other cyber security related stuff. > My ci/cd is doing a system test because everything is in containers. I can do full e2e tests and automatic rollouts without a downtime. We have a staging system for this. > What i can do, can everyone else do when i'm on holiday. We also have documentation; is this really a big thing? > Are your devs waiting regularly on things? Code Reviews, these take time --- Are these real problems organizations have?
• csomar 2y ago
  In "serverless" defense, I'll put a one data point from myself. I built https://crates.live https://crates.live 4-5 years ago. I used a "complex" tech stack. A single page web app. Hosted in Github pages as static HTML/JS. For the server side, I used Cloudflare workers (Wasm) to run a GraphQL server (kind of). The result: It's still up after 5 years. I never looked back after I created the project. I do remember the endless other projects I did that have simply died now because I don't have time to maintain a server. And a server almost always end up crashing somehow. Another thing, Pieter Levels has successful small apps that relies more on centralized audiences than infrastructure. He makes cool money but it's nowhere near startup-expected levels of money/cash/valuations. He is successful in the indie game but it'll be a mistake to extrapolate that to the VC/Silicon Valley startup game.
  • archerx 2y ago
    To counter your point I have a site running since 2019 that is still up with no input from me or anybody, it’s a dynamic site too. It’s running on docker on a vps at digitalocean. If you build a rock solid configuration it will stand the test of time.
• jedberg 2y ago
  If you'll allow me, I'd like to shill my company for a second. We provide all the benefits of "single server deployment" while providing the scalability of the "30 lambdas" solution. You can even run the whole thing locally. We actually just did a Show HN about it: https://news.ycombinator.com/item?id=41502094 https://news.ycombinator.com/item?id=41502094
• myprotegeai 2y ago
  A single VM is all fine and well until your hacky go-fast code allows an issue with a single request to take down your service. Serverless requests are isolated and will limit the blast radius of your hacky code as you iterate quickly.
  • number6 2y ago
    Never saw a single request taking down a whole server. Killed a worker and the connection timed out, but never saw it take down the whole thing. Faulty input killing your logic - I saw this plenty, would Lambda really help here?
    • abrookewood 2y ago
      I've seen it plenty. A request to process an Excel file or generate a PDF etc. Basically anything generating or processing documents is a likely candidate. It might only affect a single application, but if you are running multiple apps on a box, it is often enough to cause an outage.
• udev4096 2y ago
  It highly depends on what you are developing. Just because one guy (levels or whatever his name is) is doing it, doesn't mean it fits for everyone
• throwaway81523 2y ago
  https://news.ycombinator.com/item?id=9581862 https://news.ycombinator.com/item?id=9581862 aww, yourdatafitsinram.com is now domain squatted.
  • daemonologist 2y ago
    https://yourdatafitsinram.net/ https://yourdatafitsinram.net/ is up (and looks to be approximately the same as the old .com was)
• aitchnyu 2y ago
  Whatever happened to EC2 with web/worker autoscaling? Is it outdated or unfashionable?
  • nitwit005 2y ago
    Just unfashionable.
  • abrookewood 2y ago
    The biggest issue with this is when you deploy multiple applications to a server (e.g. 5 apps on IIS or whatever) and one of them kills off the box when it behaves badly. You can auto-scale, but it takes time to provision new machines and until they are up, you are down. Once you've experienced this a few times, the desire to split out applications into micro-services gets pretty strong in order to limit the blast radius.
    • cynicalsecurity 2y ago
      I'm sorry, what? IIS? o_O
      • abrookewood 2y ago
        The web server that many of us are stuck with unfortunately ;)
    • ironhaven 2y ago
      Ah there's your problem. Don't use IIS on Windows Server 2008!
      • abrookewood 2y ago
        We don't always get a choice.
    • nitwit005 2y ago
      No matter what auto-scaling solution you pick, it'll take time to start fresh new instances.
• aryehof 2y ago
  “Scalability” seems to be perceived to be the most important thing for startups. It’s dream-driven development.
  • imiric 2y ago
    We often forget that scalability doesn't mean just scaling up. It also means scaling down to avoid wasting money on overprovisioned infrastructure when you don't need it. All businesses need to think about scalability, regardless of their size. If you're a startup, you likely want to be frugal with your infra costs, while still having the ability to quickly scale up when you need it. Those "simple" approaches everyone loves to suggest have no way of doing this.
    • cynicalsecurity 2y ago
      A single Hetzner bare metal server is going to be a few times cheaper than all of these scalability gimmicks while offering a significant productivity.
      • imiric 2y ago
        A single server of any kind is not a proper production environment, unless you're building a toy or demo service. You want at least one application and one database server, since they have different operational requirements. You might even want to have a separate web server, so that you can isolate your internal network from the internet. This is all web hosting 101, and has been standard practice for several decades. But wait, don't you want some form of redundancy/failover in case one of these servers catches fire? Alright, let's double this then. Make sure to setup your load balancer as well, which should probably run on a separate server. But wait, don't you also want some kind of staging environment, so that you can certify your releases before deploying them to production? Alright, let's double this again. And so on, and so on... Eventually you'll end up rebuilding the same features of those complex gimmicky tools, but do a much worse job at it, and you'll also have to maintain your custom tooling mess. Of course, if your company fails after a few months, none of this is worth considering. But if you plan to exist for the next few years, I would argue that your productivity would be considerably higher if you had just chosen that gimmicky tool from the start, or a very short time after it.
• jauntywundrkind 2y ago
  First, Lex Friedman is the dumbest motherfucker in podcasting. No brains at all, terrible ignorant thoughtless interactuons: just awful in every way. > But here's the truth: not every project needs Kubernetes, complex distributed systems, or auto-scaling from day one. Simple infrastructure can often suffice, When the hell can we be done with these self compromising losers? Holy shit! Enough! It doesn't save you anything doing less. People fucking flock to bot-Kubernetes because they can't hack it, because they suck, because they would prefer growing their own far worse far more unruly monster. A monster no one will ever criticize in public because it'll be some bespoke frivolous home grown alt-stack no one will bother to write a single paragraph on, which no one joining will grok understand or enjoy. It's just so dumb. Theres all these fools trying to say, oh my gosh, the emperor has no clothes! Oh my gosh! It might not be needed! But the alternative is a really running naked through the woods yourself, inventing entirely novel unpracticed & probably vastly worst less good means for yourself. I don't know why we keep entertaining & giving positions of privilege to such shit throwing pointless "you might not need it" scum sucking shits trying to ruin things like so, but never ever do they have positive plans and never ever do they acknowledge that what they are advocating is to take TNT to what everyone else is trying to practice, is collaborating on. Going it alone & DIY'ing your own novel "you might not need" to participate in a society stack is fucking stupid & these people don't have the self respect to face up to the tall dissent they're calling for. You'd have to be a fool to think you are winning by DIY'ing "less". Fucking travesty.
  • mexicocitinluez 2y ago
    > First, Lex Friedman is the dumbest motherfucker in podcasting. Amen. > I don't know why we keep entertaining & giving positions of privilege to such shit throwing pointless "you might not need it" scum sucking shits t Amen.
• throwaway984393 2y ago
  [dead]
• andrewstuart 2y ago
  Go super minimal: Postgres for everything including queuing Golang or nodejs/TypeScript for the web server Raw SQL to talk to Postgres Caddy as web server with automatic https certificates - No docker. - No k8s. - No rabbitmq. - No redis - No cloud functions or lambdas. - No ORM. - No Rails slowing things down.
  • cynicalsecurity 2y ago
    Nice, but MySQL is even simpler than PostgreSQL.
    • andrewstuart 2y ago
      I like the power of Postgres and I use many features and I find it simple. The goal is not for the technologies used to be simple or boring. The overall architecture is simple, the technologies used are powerful.
      • cynicalsecurity 2y ago
        Thank you. I appreciate the answer.
• qianli_cs 2y ago
  The issue with a long running server is that if your traffic is low, you’re paying for idle time all the time. So I’d prefer a serverless solution.
  • ghomem 2y ago
    ...and the funny thing is that it is still cheaper than cloud native even being up all the time and provides a predictable cost per month, unlike serverless where you can have big surprises. Check: https://logical.li/blog/emperors-new-clouds/ https://logical.li/blog/emperors-new-clouds/
• andrewstuart 2y ago
  I worked on a startup recently that had gone all in on AWS infrastructure, Lambda functions, managed database, IAM security. Man the infrastructure was absolutely massive and so much development effort went into it. They should have had a single server, a backup server, Caddy, Postgres and nodejs/typescript, and used their development effort getting the application written instead of futzing with AWS ad infinitum and burning money. But that's the way it is these days - startup founders raise money, find someone to build it and that someone always goes hard on the full AWS shebang and before you know it you spend most of your time programming the machine and not the application and the damn thing has become so complex it takes months to work out what the heck is going on inside the layers of accounts and IAM and policies and hundreds of lambda functions and weird crap.
  • datavirtue 2y ago
    Same here. The CTO was also engaging in resume-driven development. There is no rational discussion about what tech stack to use. Executives need to be able to point to a modern tech stack as a signal of their relevancy and competence. No one will be caught dead slinging bare metal and running on-prem databases right now. It's just the look. I built out a POC and was running it on bare metal for serious workloads under my desk at GE (12-factor). Management practically scrambled to get me cloud access. My setup was ephemeral and could be easily reproduced anywhere. The software was easily deployed on, or integrated with, cloud services. I just shrugged. I didn't care where my code ran, to them it was some epic priority to get it in the cloud and generate extra expenses.
  • icedchai 2y ago
    I've seen the same thing. Massive infrastructure for a site that could run on a small VM. More time was spent configuring infrastructure, Terraform, debugging IAM roles than building the actual code...
• jillesvangurp 2y ago
  It depends. I personally love cloud based solutions because they save me lots of time. But I'm highly selective in what I use and there are some solutions that are clearly counter productive because they are too complicated. I run a small, bootstrapped startup. We don't have enough money to pay ourselves and I make a living doing consulting on the side. Being budget and time constrained like that I have to be highly selective in what I use. So, I love things like Google cloud. Our GCP bills are very modest. A few hundred euros per month. I would move to a cheaper provider except I can't really justify the time investment. And I do like Google's UI and tools relative to AWS, which I've used in the past. I have no use for Kubernetes. Running an empty cluster would be more expensive than our current monthly GCP bills. And since I avoided falling into the micro-services pitfall, I have no need for it either. But I do love Docker. That makes deploying software stupidly easy. Our website is a Google storage bucket that is served via our load balancer and the Google CDN. The same load balancer routes rest calls to two vms that run our monolith. Which talk to a managed DB and managed Elasticsearch and a managed Redis. The DB and Elasticsearch are expensive. But having those managed saves a lot of time and hassle. That just about sums up everything we have. Nice and simple. And not that expensive. I could move the whole thing to something like Hetzner and cut our bills by 50% or so. Worth doing maybe but not super urgent for me. Losing those managed services would make my life harder. I might have to go back to AWS at some point because some of our customers seem to prefer that. So, there is that as well.
• taw1285 2y ago
  > 20-30 Lambda functions for different services My team of 6 engineers have a social app at around 1,000 DAU. The previous stack has several machines serving APIs and several machines handling different background tasks. Our tech lead is forcing everyone to move to separate Lambdas using CDK to handle each each of these tasks. The debugging, deployment, and architecting shared stacks for Lambdas is taking a toll on me -- all in the name of separation of concerns. How (or should) I push back on this?
  • langsoul-com 2y ago
    Does the tech lead have the CTO or CEO's graces for that decision? Why did the tech lead decide to move everything to lambda when you only have 1k DAU? Can they be reasoned with or is it lambda or the highway? You can pull put the stats and do comparison, note the wasted time, how it's not beneficial but rather detrimental. Note how long it now takes to debug for such a small codebase, then extrapolate that out. Having tons of lambdas is a massive pain in terms of debugging. Cloud watch is not that great to debug, and the debug tooling tends to be rather expensive, like data dog so not too much is invested. Or it's too resource intensive to setup open telemetry.
• imiric 2y ago
  This is a retreaded and often tiresome debate. I'll still throw my 2c in... Should you pick a complex framework from day one? Probably not, unless your team has extensive experience with it. My objection is towards the idea that managing infrastructure with a bespoke process and custom tooling will always be less effort to maintain than established tooling. It's the idea of stubbornly rejecting the "complexity" bogeyman, even when the process you built yourself is far from simple, and takes a lot of your time from your core product anyway. Everyone loves the simplicity of copying over a binary to a VPS, and restarting a service. But then you want to solve configuration and secret management, have multiple servers for availability/redundancy so then you want gradual deployments, load balancing, rollbacks, etc. You probably also want some staging environment, so need to easily replicate this workflow. Then your team eventually grows and they find that it's impossible to run a prod-like environment locally. And then, and then... You're forced to solve each new requirement with your own special approach, instead of relying on standard solutions others have figured out for you. It eventually gets to a question of sunken cost: do you want to abandon all this custom tooling you know and understand, in favor of "complexity" you don't? The difficult thing is that the more you invest in it, the harder it will be to migrate away from it. My suggestion is: start by following practices that will make your transition to the standard tooling later easier. This means deploying with containers from day 1, adopting the 12 factors methodology, etc. And when you do start to struggle with some feature you need, switch to established tooling sooner later than later. You're likely find that your fear of the unknown was unwarranted, and you'll spend less time working on infra in the long run.
  • bjornsing 2y ago
    This is a good articulation of the ambivalence I can feel around this. One approach that I’ve considered is to start with the standard tooling (k8s + gitops) from day one, but still run it in a single VM. Any thoughts?
    • shakiXBT 2y ago
      k3s single node + ArgoCD/Flux is what I would if I had to build infrastructure of a small startup by myself. Unfortunately it's HN so people are more likely to do everything in bash scripts and say a big "fuck you" to all new hires that would have to learn their custom made mess
• shahzaibmushtaq 2y ago
  Yes, your startup needs complex cloud infrastructure when your organizational infrastructure can afford it in terms of money, other resources and time. One domain, an idea, an easy-to-use development stack for a bootstrapped as well as funded startup is more than good enough to locate product-market fit. Alway remember this quote by Reid Hoffman “If you are not embarrassed by the first version of your product, you’ve launched too late.”
• Spotrate 2y ago
  In my case (Experience with Azure Development), I definitely would use cloud infrastructure. Cloud providers abstract a lot of difficult things away, have ok-ish documentation and have a UI where I can easily find relevant information or do some debugging. With tooling I have more experience with I move away from the UI, but it's so easy just to get something up and running. The difficult thing is not getting each of these individual tools up and running, but handling the interactions between them and unfortunately I don't feel comfortable enough to do Networking, SSL, Postgres, Redis, VM management and building / hosting containers at the same time. Costs in my case is not the highest priority: I can spend a month learning the ins and outs of a new tool, or can spend a few days learning the basics and host a managed version on a cloud provider. The cloud costs for applications at my scale are basically nothing compared to developer costs and time. In combination with LLMs who know a lot about the APIs of the large cloud providers, this allows me to focus on building a product instead of maintenance.
• deleted 2y ago
  [deleted]
• riiii 2y ago
  I've seen more than one start up go tits up because they were too focused on designing "Google scale ready" infrastructure.
  • Cwizard 2y ago
    And it has to be cloud agnostic because we can’t get locked in! I like the cloud but it is overused and misused a lot imo.
• negus 2y ago
  TLDR: I'm not too good with the infrastucture (and this couple of teams also), so you should also go on steam engines. Of cource it highly depends on the skills of the team. In a startup there could be no time to learn how to do infrastructure well. But having an infrastructure expert in the team can significantly improve the time to market and reduce the developer burnout and the tech debt growth rate.
  • seper8 2y ago
    TL;DR You shouldn't need to assemble a plane when your startups journey can be expected to only last a few kilometers and you really only need to carry a few boxes.
• ghomem 2y ago
  I went through sweat and tears with this on different projects. People wanting to be cool because they use hype-train-tech ending up doing things of unbelievably bad quality because "hey, we are not that many in the team" but "hey, we need infinite scalability". Teams immature to the point of not understanding what LTS means have decided that they needed Kubernetes because yes. I could go on. I currently have distilled, compact Puppet code to create a hardened VM of any size on any provider that can run one more more Docker services or run directly a python backend, or serve static files. With this I create a service on a Hetzner VM in 5 minutes whether the VM has 2 cores or 48 cores and control the configuration in source controlled manifests while monitoring configuration compliance with a custom Naemon plugin. A perfectly reproducible process. The startups kids are meanwhile doing snowflakes in the cloud spending many KEUR per month to have something that is worse than what devops pioneers were able to do in 2017. And the stakeholders are paying for this ship. I wrote a more structured opinion piece about this, called The Emperor's New clouds: https://logical.li/blog/emperors-new-clouds/ https://logical.li/blog/emperors-new-clouds/
  • dijit 2y ago
    I'm with you, but for me Cloud does have one major benefit: If you use it as IaaS, it's a lot quicker to get prototypes working than if you use anything else, including VPS's from other providers. Google Cloud in particular has very few vectors for lock-in, and follows more principle of least surprise. But once you have prototyped, you should ask the question about rebuilding it somewhere that is cheaper. Near infinite scalability of disk drives is nice, and snapshotting, and cloud in general can allow you to extend your prototype into taking production load and allowing you to measure what you will need; but leaning in to "cloud magick" (cloud run, lambdas, etc) will consume almost as much time to learn and debug as just doing it the old school way anyway. In my lived experience.
    • ghomem 2y ago
      I am not against the cloud. VMs are also cloud, unless you run them on your own servers. For instance, the Hetzner Cloud (mostly VMs, plus load balancers and disks) is so cheap and has such a nice CLI API that it competes aggressively with dedicated servers - I would definitely start any with VMs, not with iron. The biggest problem is the so called cloud native stuff which is both more expensive and more complex. There are contexts where it makes sense but for startups they are doing more harm than good.
• senko 2y ago
  Completely agree. Scaling (and relatedly, high availability) are premature optimizations[0] implemented (and authorized) by people hoping for that sweet hockey stick growth, cargo culting practices needed by companies several orders of magnitude larger. [0] https://blog.senko.net/high-availability-is-premature-optimization https://blog.senko.net/high-availability-is-premature-optimi...
• liampulles 2y ago
  There is a core 20% of kubernetes, which is deployments, pods services and the way it handles blue-green deployments and declarative based definitions, namespace seperation, etc. that is really good. Just keeping to those simple basics, using a managed cloud kubernetes service, and running your state (database) out of cluster is a good experience (IMO). It's when one starts getting sucked down the "cloud native" wormhole of all these niche open source systems and operators and ambassador and sidecar patterns, etc. that things go wrong. Those are for environments with many independent but interconnecting tech teams with diverse programming language use.
  • maeln 2y ago
    For a lot of company and project I worked on, this is the same conclusion I came to. 99% we only need / want is docker-compose++. Things like 0-downtime deployment out of the box, simple configuration system for replica set and other replication / distribution mechanism, and that is basically it. I which there was something that did just that, because kube comes with a lot of baggage, and docker-compose is a bit too basic for some important production needs.
    • wanderlust123 2y ago
      Why not use docker swarm?
    • salomonk_mur 2y ago
      The author posted almost exactly this. https://github.com/hadijaveed/docker-compose-anywhere https://github.com/hadijaveed/docker-compose-anywhere
  • globular-toast 2y ago
    For me this is all Kubernetes is. I feel like people are often talking about two different things in discussions like this. For me it's just a uniform way to deploy stuff that is better than docker compose. We pay pennies for the control plane and workers are just generic VMs with kubelet. But I think for many "kubernetes" means your second paragraph. It doesn't have to be like that at all! People should try settling up a k3s cluster and just learn about workloads, services and ingresses. That's all you need to replace a bunch of ad hoc VMs and docker stuff.
• rtpg 2y ago
  I quit my last job because of these kinds of shenanigans. I was brought in to help get a full system rewrite across the finish line. Of course the deployment story was pretty great! Lots of automated scripts to get systems running nicely, autoscaling, even a nice CI builder. The works. After joining, I found out all of this was to the detriment of so much. Nobody was running the full frontend/backend on their machine. There was a team of 5 people but something like 10-15 services. CI was just busted when I joined, and people were constantly merging in things that broke the few tests that were present. The killer was that because of this sort of division of labor, there'd be constant buck-passing because somebody wasn't "the person" who worked on the other service. But in an alternate universe all of that would be in the same repo. Instead, everything ended up coordinated across three engineers. A shame, because the operational story letting me really easy swap in a pod for my own machine in the test environment was cool! But the brittleness of the overall system was too much for me. Small teams really shouldn't have fiefdoms.
  • ghomem 2y ago
    > There was a team of 5 people but something like 10-15 services Puff! Talk about microservices! Or is it macropeople?! :-)
• roncesvalles 2y ago
  I've run a project for 6 years on a single $10/month VPS (I pay even less due to a perpetual discount I bagged from lowendtalk) run by a gameserver-focused VPS provider for about SIX years with about 99.999 reliability if you exclude the one time I fucked up a config and it was down for a whole day because I wanted to do a clean OS reinstall, and one other time when they changed my IP address (they gave me notice). VPS technology has come a very long way and is highly reliable. The disks on the node are set up in RAID 1 and the VM itself can be easily live migrated to another machine for node maintenance. You can take snapshots etc. To me, I would only turn to cloud infra not for greater reliability but more for collaboration and the operational housekeeping features like IAM, secrets management, infra-as-code etc, or for datacenter compliance reasons like HIPAA.
  • salomonk_mur 2y ago
    Which provider? Sounds great!
• mdavid626 2y ago
  Exactly. Keep it simple. We're running 1 monolith FastAPI service on EC2 with ECS (1 instance). Very simple, easy to debug and develop. Plus we have few lambdas for special tasks (like PDF generation), which run rarely, but are needed. Frontends are Vue projects served from a public S3 bucket. This setup might work for many years.
• throwpok 2y ago
  It doesn't matter how we build it if there are no users to use it. This is the real problem for many startups
• yieldcrv 2y ago
  yeah but I dont need Python Flask on my resume, I need docker, kubernetes and terraform on my resume I need it on my resume for every 2 year stint and 2-3 people on the team to vouch for it You’re saying “hey, let everyone know you worked on a tiny company’s low traffic product and how about you just don’t make half a million a year," all to save the company I work at a little money? until companies start interviewing for that its a dumb idea, I’m rarely making green field projects anywhere and other devs also are looking for maintainers of complex infrastructure
• bob1029 2y ago
  VMs, block & blob storage, DNS, IdP, domain registrar. These are the only things I have ever been comfortable using in the cloud. Once you get into FaaS and friends, things get really weird for me. I can't handle not having visibility into the machine running my production environment. Debugging through cloud dashboards is a shit experience. I think Microsoft's approach is closest to actually "working", but it's still really awful and I'd never touch it again. The ideal architecture for me after 10 years is still a single VM with monolithic codebase talking to local instances of SQLite. The advent of NVMe storage has really put a kick into this one too. Backups handled by snapshotting the block storage device. Transactional durability handled by replicating WAL, if need be. Dumbass simple. Lets me focus on the business and customer. Because they sure as hell don't care about any of this and wouldn't pay any money for it. All this code & infra is pure downside. You want as little of it as possible.
  • tstrimple 2y ago
    > VMs, block & blob storage, DNS, IdP, domain registrar. This is the most expensive way to build cloud services. When people talk about the cloud being more expensive than on-prem this is often the reason why. If you're just going to run VMs 24/7 there are better options.
• ssss11 2y ago
  EC2 Linux VM with node, sqlite, let’s encrypt cert and a domain name.
• konschubert 2y ago
  The backend doing the rendering for the 550 eink calendars that I have sold to far runs on a small, 10-Euro-a-month Hetzner server. Low operational costs are essential for a hardware business if you don't want to burden your customers with an ongoing subscription fee. Otherwise the business turns into some kind of pyramid scheme where you have to sell more and more units in order to keep serving your existing customers. I have a moral obligation towards my customers to keep running even if the sales stop at some point. So I always multiply my cost for anything with 10 years, and then decide if I am willing to bear it. If not, then i find another solution.
• Kydlaw 2y ago
  I recently moved to data engineering role where everything uses GCP services (think BigQuery, DataProc, Cloud Storage, ...) and wondered is all that was really necessary? What would be the simple yet robust infra for data eng? Not thought a lot about it for now, so I am curious if some of you have would have any insights.
  • ghomem 2y ago
    The same thing that happened to devops from 2017-2024 (see: https://logical.li/blog/devops/ https://logical.li/blog/devops/) is happening with dataops. Hype train and jargon based decisions are taking place. In the past years I was solving a data pipeline mess on a project which also had a devops AWS mess. First thing I was told was "what we need is a data lake". Decisions are sticky so take context into account.
• miguelaeh 2y ago
  I guess what some people do not understand is that K8s was created internally at Google, for managing their services and handling millions of users. For new projects that, with luck, will have a couple hundred users at the beginning it is just overkilling (and also very expensive). My approach is usually Vercel + some AWS/Hetzner instance running the services with docker-compose inside or sometimes even just a system service that starts with the instance. That's just enough. I like to use Vercel when deploying web apps because it is free for this scale and also saves me time with continuous deployment without having to ssh into the instances, fetch the new code and restart the service.
• jwr 2y ago
  The answer is "no, it doesn't". I've been running my SaaS first on a single server, then after getting product-market fit on several servers. These are bare-metal servers (Hetzner). I have no microservices, I don't deal with Kubernetes, but I do run a distributed database. These bare-metal servers are incredibly powerful compared to virtual machines offered by cloud providers (I actually measured several years back: https://jan.rychter.com/enblog/cloud-server-cpu-performance-comparison-2019-12-12 https://jan.rychter.com/enblog/cloud-server-cpu-performance-...). All in all, this approach is ridiculously effective: I don't have to deal with complexity of things like Kubernetes, or with cascading system errors that inevitably happen in complex systems. I save on development time, maintenance, and on my monthly server bills. The usual mantra is "but how do we scale" — I submit that 1) you don't know yet if you will need to scale, and 2) with those ridiculously powerful computers and reasonable design choices you can get very, very far with just 3-5 servers. To be clear, I am not advocating that you run your business in your home closet. You still need automation (I use ansible and terraform) to manage your servers.
  • stcroixx 2y ago
    The scaling thing is a great boogeyman. It preys on this optimism your software is going to be so successful in such a short amount of time which people want to believe.
  • mexicocitinluez 2y ago
    The answer is "it depends". Did you read the article or just the headline? Scroll down to the bottom, under the section "A few considerations" and try not to laugh. "A few considerations" turns out to be a pretty significant chunk of security work ESPECIALLY if you are storing/transmitting highly sensitive information. How do you handle something like HIPPA compliance when you're in this situation? There are 2 types of programmers: those that think they've seen everything and those that know they've seen next to nothing. And as such, these absolute takes are tiring.
    • pistoleer 2y ago
      > How do you handle something like HIPPA compliance when you're in this situation? I'm a dev who hasn't seen anything related to that. Since you bring it up, can you give some pointers on why something like a MySQL db coupled to a monolithic backend isn't good enough? What shortcomings did you experience? All of the things raised in the article seem possible to solve without the need for microservices.
• benterix 2y ago
  > Even GCP VMs and EC2 instances are reasonably priced. Really? EC2 instances are waaay overpriced. If you need a specific machibe for a relatively short time, sure, you can pick up one from the vast choice of available configurations, but if you need on for long-running workloads, you'll be much better of picking up one from Hetzner, by an order of magnitude. For one of the many examples, see this 5-year old summary (even more true today) by a CEO of a hardware startup: https://jan.rychter.com/enblog/cloud-server-cpu-performance-comparison-2019-12-12 https://jan.rychter.com/enblog/cloud-server-cpu-performance-...
• palebt 2y ago
  Docker Compose Anywhere looks cool. Looks similar, on principle, to [CapRover](https://caprover.com/ https://caprover.com/) which I highly appreciate.
• flanked-evergl 2y ago
  Building and operating your own car out of simple components is not simpler than buying a car off the shelf. Operating a bunch of simple low-level infrastructure yourself is not simpler than buying the capabilities off the shelf.
  • boesboes 2y ago
    Apples and oranges. I'd say it is more like: Using a trolley to move some stuff across the street is more simple than using a fleet of drones.
    • flanked-evergl 2y ago
      Running your own Postgres on your own server — implementing and testing your own backups, optimizing your own filesystem, managing encryption keys, managing upgrades, etc — is not simpler than using Google Cloud SQL, which does all of this for you at an SLA you will not be able to achieve if you will be focusing on your business, which is what you should do as a startup. Certainly you should not be running your own K8S cluster, but using Google Cloud Run is simpler than keeping your own server running. Even using Google Cloud Kubernetes Engine with autopilot is simpler than keeping your own server running.
• l5870uoo9y 2y ago
  > Pieter has built numerous successful micro-SaaS businesses by running his applications on single server, avoiding cloud infrastructure complexity... From what I understand he employees a dedicated system administrator to manage his fleet of VPS (updates, security and other issues that arise) for 1000s of USD per month.
• donatj 2y ago
  No. YAGNI In my time at my current job we've scaled PHP MySQL and Redis from a couple hundred active users to several hundred-thousand concurrent users. EC2+ELB, RDS (Aurora, Elasticache). Shell script to build a release tarball. Shell script to deploy it. Everyone goes home on time. In my 12+ years I've only had to work off hours twice. People really love adding needless complexity in my experience.
  • mexicocitinluez 2y ago
    > People really love adding needless complexity in my experience. No, people love thinking their experience is the same as everyone else's. Have you ever worked in healthcare? Do you have any idea what sort of requirements there are for storing sensitive information? >n my 12+ years I've only had to work off hours twice. Well that settles it. Then no one on the planet should need cloud infra if yuo didnt. And please, please don't tell me you've spent the last 12 years at the same place and have the gall to extend that to all software development.
    • donatj 2y ago
      That is a misinterpretation of what was said. I did not say all complexity is needless nor did I claim to have the one panacea. I presented my story of how we've actively kept our architecture simple, and noted we've had very few issues. I did not say our architecture is the architecture for everyone. Then I said > People really love adding needless complexity in my experience If the complexity is legally mandated, as in healthcare, it's by no means "needless". Legal compliance is a need. If the complexity is justified, has merit or value, it's not "needless". However, I've known a fair number of people who work on complicated kubernetes driven architectures that give them non-stop grief, and whose user base max out at ten-twenty active users. My point is just don't make things more complex than they need to be.
      • mexicocitinluez 2y ago
        > No. YAGNI Sounds pretty absolute to me. I mean, when asked "Does your startup need complex cloud infra" (which is a loaded question) and you say "No. YAGNI" that seems pretty unequivocal and not really fair to say I misinterpreted it. > My point is just don't make things more complex than they need to be. I agree. I just don't care for the absolute language (that I and others use sometimes). It made learning when I was just getting into this field really tough. My answer to that exact same question would be "It depends".
• icar 2y ago
  If your stack is Node.js, I highly recommend SSTv3 [0], which uses Pulumi under the hood and thus lets you deploy to any provider you want, be that cloud or docker in Hetzner. It's simple and can scale to complex if you want. I've had very good experience with it in medium size TS monorepos. [0]: https://sst.dev https://sst.dev
• api 2y ago
  Cloud Native is the J2EE of the 2010s and 2020s. It’s really brilliant. Sun would have been the one to buy Oracle if they’d figured out how to monetize FactorySingletonFactoryBean by charging by the compute hour and byte transferred for each module of that. That’s what cloud has figured out, and it’s easy to get developers to cargo cult complexity.
• vimto 2y ago
  It's funny because OPs solution was his docker-compose-anywhere, which is exactly what, from my experience, I've seen so many start-ups running with. Sure it works while you're running an MVP but it's incredibly brittle for running something in production as soon as the application grows in complexity. IMO the primary draw of k8s isn't necessarily "infinite scalability" but its resilience. I sometimes wonder how many of these post boil down to "I don't want to learn k8s can I just use this thing I already know?".
• alberth 2y ago
  Slight OT: I’m shocked at the complexity even for “simple” static hosting options. I recently attempted to move to a completely static site (just plain HTML/CSS/JS) on Cloudflare Pages, that was previously on a cheap shared webhost. Getting security headers setup, and forcing ssl, and www - as well as HSTS has been a nightmare (and still now working). When on my shared host, this was like 10 lines of config in an .htaccess file before.
• mannyv 2y ago
  No, it needs a simple cloud infrastructure.
• nerdjon 2y ago
  I think there is a middleground that to me it seems like this over simplifies both sides of this. For many of the "complex" things like lambdas there are frameworks like Serverless that makes managing and deploying it as easy (if not easier frankly) than static code on a VM. Not every workload also scales at the same time, we have seen new things that got very successful and crashed right out the door because it could not properly scale up. I agree that you don't need an over engineered "perfect" infrastructure, but just saying stick it on a VM also seems like it is too far of a swing in the other direction. That ignores the cost side of running several VM's vs the cost of smaller containers or lambdas that only run when there is actual use. Plus there is something to be said about easier local development which some things like Serverless and containers give you. You may not need to setup a full k8s cluster, but if you are going with containers why would you run your own servers vs sticking the container in something managed like ECS.
• solatic 2y ago
  Worrying about whether your web or app servers need or should use cloud architecture belies the much, much bigger consideration of how and where to store your data. Specifically, the economics of getting that data out of where you decide to put it first. Everything follows that decision. Want to run bare metal? OK, guess you're running your databases on bare metal. Do you have the DBA skills to do so? I would wager that an astounding number of founders who find themselves intrigued by the low cost of bare metal do not, in fact, have the necessary DBA skills. They just roll the dice on yet another risk in an already highly risky venture.
• samcat116 2y ago
  I'll say the same thing I always say on these kinds of posts. Both of the following can be true: - A lot of companies and startups can get by with a few modest sized VPSs for their applications - Cloud providers and other infrastructure managed services can provide a lot value that justifies paying for them.
• JSDevOps 2y ago
  Why would anyone make it complex?
• maccard 2y ago
  For all the people who are saying you don’t need X and Y - what is the simplest way to deploy a web app using TLS on a VPS/VM? Let’s say I’ve got a golang binary locally on my machine, or as an output of github actions. With Google Cloud Run/Fargate/DigitalOcean I can click about 5 buttons, push a docker image and I’m done, with auto updates, roll backs, logging access from my phone, all straight out of the box, for about $30/mo. My understanding with Hetzner and co is that I need to SSH (now i need to keep ssh keys secure and manage access to them) in for updates, logs, etc. I need to handle draining connections from the old app to the new one. I need to either manage https in my app, or run behind a reverse proxy that does tls termination, which I need to manage the ssl certs for myself. This is all stuff that gets in the way of the fact that I just want to write my services and be done with it. Azure will literally install a GitHub actions workflow that will autodeploy to azure container apps for you, with scoped credentials.
  • selfmodruntime 2y ago
    You can pretty easily self host a GitLab instance, host a kubernetes runner for your images and use Tailscale for SSH keys. This will most certainly cost you more than $30, but you can do it.
  • bruh2 2y ago
    Re: securing SSH keys; Nowadays most password managers can store SSH keys and integrate nicely with your SSH agent, making it essentially equivalent to logging in with a password. I use KeepassXC[1], and the workflow consists of opening the database using my master password, then just `ssh machine`, so in my book it's at the same level of comfort as a web interface for your cloud provider [1] https://keepassxc.org/docs/KeePassXC_UserGuide#_setting_up_ssh_agent_integration https://keepassxc.org/docs/KeePassXC_UserGuide#_setting_up_s...
  • zipy124 2y ago
    > For all the people who are saying you don’t need X and Y - what is the simplest way to deploy a web app using TLS on a VPS/VM? Depends on your defintion of simplest. In terms of set-up probably someting like https://dokku.com/ https://dokku.com/ . It's a simple self-hosted version of herokku, you can be up and running in literally minutes and because its compatable with herokku you can re-use lots of github action/ other build scripts. In terms of simple (low complexity and small sized components) just install caddy as your reverse-proxy which will do ssl certs and reverse proxy for you with extremely little, if any config. Then just have your github action push your containers there using whatever container set-up you prefer. This is usually a simple script on your build process like "build container -> push container to registry -> tell machine to get new image and run it" or even simpler just have your server check for updated images routinely if you don't want to handle communication between build script and server. That's the bare minimum needed. This takes a bit longer than a few minutes but you can still be done within an hour or two. Regardless of your choice it shouldn't take more than 1 working day, and will save you a lot of money compared to the big cloud providers. You can run as low as €4.51/month with hetzner and that includes a static IP and basically unlimited traffic. An EC2 instance with the same hardware costs about $23 a month for comparison (yes shared vs dedicated vCPU, but even the dedicated offer at hetzner is cheaper, and this is compared to a serverless set-up where loads are spikey, which is exactly how we can benefit from a shared vCPU situation).
• rco8786 2y ago
  > Does your startup need complex cloud infrastructure? 99.99% of the time. No.
• Nelkins 2y ago
  I've been looking at using Kamal for a side project. Seems to be similar in spirit. Has anybody used it, and if so, what do you think? https://github.com/basecamp/kamal https://github.com/basecamp/kamal
  • ghomem 2y ago
    I am curious about this too but haven't had the time to give it a try. Looking forward to hear about experiences.
• theptip 2y ago
  Even the book on Microservices says “First build the Monolith”. You don’t know how to split your system until you have actually got some traction with users, and it’s easier to split a monolith than to reorganize services. You may never need to split your monolith! Stripe eventually broke some stuff out of their Rails monolith but it gets you surprisingly far. You are not going to get easier to debug than a Django/Rails/etc monolith. I bit of foresight on where you want to go with your infra can help you though; I built the first versions of our company as a Django Docker container running on a single VM. Deploy was a manual “docker pull; docker stop; docker start”. This setup got us surprisingly far. Docker is nice here as a way of sidestepping dependency packaging issues, this can be annoying in the early stages (eg does my server have the right C header files installed for that new db driver I installed? Setup Will be different than in your Mac!) We eventually moved to k8s after our seed extension in response to a business need for reliability and scalability; k8s served us well all the way through series B . So the setup to have everything Dockerized made that really easy too - but we aggressively minimized complexity in the early stages.
  • corytheboyd 2y ago
    Yes! Also, use the damn framework, instead of rebuilding shitty versions of features it offers! One good seasoned person will outperform 10 non-seasoned people in this regard. It will add up over time. I think half the real reason people are soured to monoliths is because they are bad, poorly run monoliths.
  • randomdata 2y ago
    > Even the book on Microservices says “First build the Monolith”.\ And yet, funnily enough, the book on Monoliths says to break things up into smaller services! It says your data should be stored in its own service (possibly multiple services, if you need multi-paradigm access [e.g. relational, full-text search, etc.]). The user experience should use its own service. And, at very least, you should have another service in between (this is where Django and Rails usually fit). Optionally, it says, you will probably want to have additional services as well (auth, financial transitions, etc.)
• alamsterdam 2y ago
  For me (where our BE consists of maybe 100 endpoints) we’ve found the sweet spot to be Google AppEngine. Incredibly simple to deploy, we don’t really need to manage infrastructure or networking (although you can if you want), decent performance, plays well with other GCP services, great logging and observability, etc We’ve tried deploying services on K8s, Lambda/Cloud Run, but in the end, the complexity just didn’t make sense. I’m sure we could get better performance running our own Compute/EC2 instances, but then we need to manage that.
• jfisher4024 2y ago
  No product I’ve ever worked on has been successful enough to require the optimizations that microservices can provide. Part of the reason they weren’t successful was because my managers insist on starting with microservices. Starting with microservices prevents teams from finding product-market fit that would justify microservices.
• beaviskhan 2y ago
  Probably not - but by calling out EC2 instances as the way and then failing to mention patching or configuration management, this article loses some credibility for me. These considerations are not optional over any significant length of time, and will cause misery if not planned for. Bare minimum, script out the install of your product on a fresh EC2 instance from a stock (and up-to-date) base image, and use that for every new deploy.
  • leetrout 2y ago
    I strong agree this is the way. We run Spacelift workers with Auto Scaling Groups and pick up their new image ~monthly with zero hassle since everything is automated. Raw EC2 is just part of the story... Edit to add: I also recommend using Amazon Linux unless you _have_ to have RHEL / Cent / Rocky or Ubuntu. Just lean into the ecosystem and you can get so many great features (and yes, I ACK the vendor lock-in with this advice). A really cool feature is the ability to just flip on various AWS services like the systems manager session manager and get SSH without opening ports a-la wireguard.
  • hjaveed 2y ago
    For patch management particularly with EC2s, we use AWS Systems Manager Patch Manager.... fairly straightforward to setup once you configure a base image obviously, it's not cloud-native... but if you are using AWS EC2 it works
• dangus 2y ago
  While this advice is good for micro-SaaS, it’s only good for micro-SaaS. If you’re at any other kind of startup, your revenue is expected to grow by double digits. Your little startup will become large, and fast. That hacked together single server is going to bite you way sooner than you think, and the next thing you know you’ll be wasting engineer hours migrating to something else. Me personally, I’d rather just get it right the first time. And to be honest, all the cloud services out there have turned a complex cloud infrastructure into a quick and easy managed service or two. E.g., why am I managing a single VPS server when I can manage zero servers with Fargate and spend a few extra bucks per month? A single server with some basic stuff is great for micro-SaaS or small business type of stuff where frugality is very important. But if we shift the conversations to startups, things change fast.
• rirze 2y ago
  Yah this all sounds good until you realize you have to actually maintain those servers, apply security patches and inevitably run into configuration drift. Like all things, there's a good middle ground here-- use managed services where you can but don't over-architect features like availability & scaling. For example, Kubernetes is an heavy abstraction; make sure it's worth it. A lot of these solutions also increase dev cycles, which is not great early on.
• hintymad 2y ago
  It looks like the author specifically talks about the infra for an early-stage startup that has not found product-market fit yet. If a startup has product for consumers and does find the product-market fit, then I'd imagine two pieces of infrastructure that is hard to come by: EC2, and S3. Yes, EC2, the grandpa's infra that people either ignore or despise. But really, anyone can learn how to set up and run a k8s cluster, yet very few companies can offer something like EC2: full abstraction of the underlying servers, worry-free of provisioning new servers, and robust and versatile implementation of dynamic autoscaling. After all, all the k8s shit won't scale easily if we can't scale the underlying servers. And S3. S3 is just a wonderful beast. It's just so hard to get something that is so cheap yet offers practically unlimited bandwidth and worry-free durability. I'd venture to say that it's so successful that we don't have an open-source alternative that matches S3. By that I specifically mean that no open-source solution can truly take advantage of scale: adding a machine will make the entire system more performant, more resilient, more reliable, and cheaper per unit cost. HDFS can't do that because of its limitation on name nodes. Ceph can't do that because of it bottleneck on managing OSD metadata and RGW indices. MinIO can't do that because their hash-based data placement simply can't scale indefinitely, let alone ListObjects and GetObjects will have poll all the servers. StorJ can't do that because their satellite and metadata servers are still the bottleneck, and the list can go on.
• ozim 2y ago
  I would say WAF is also not that useful addition when you develop new applications. Especially if you use new frameworks and ORM. Most of crap hitting servers is old exploits targeting popular CMS. WAF is useful if you have to filter out traffic and you don’t know what might be exposed on your infra. Like that Wordpress blog that marketing set up 3 years ago and stopped adding posts and no one ever updated it.
• strzibny 2y ago
  I really subscribe to this kind of thinking, only I am team Kamal[0] instead of Docker Compose. Kamal 2 is around the corner and I think this version might even convince those that passed on Kamal 1. It's still just Docker but in a nice packaging. I'll be also updating my handbook[1] for the next version. [0] https://kamal-deploy.org https://kamal-deploy.org [1] https://kamalmanual.com/handbook https://kamalmanual.com/handbook
• mylh 2y ago
  No
• raghava 2y ago
  Points to be noted. 1. It took the end of ZIRP era for people to realize the undue complexity of many fancy tools/frameworks. The shitshow would have continued unabated as long as cheap money was in circulation. 2. Most seasoned engineers know for the fact that any abstractions around the basic blocks like compute, storage, memory and network come with their own leaky parts. And that knowledge and wisdom helps them make the suitable trade-offs. Those who don't grok them, shoot themselves in the foot. Anecdote on this. A small sized startup doing B2B SaaS was initially running all their workloads on cheap VPSs incurring a monthly bill of around $8K. The team of 4 engineers that managed the infrastructure cost about $10K per month. Total cost:$8K. They made a move to 'cloud native' scene to minimize costs. While the infra costs did come down to about $6K per month, the team needed new bunch of experts who added about another $5K to the team cost, making the total monthly cost $21K ($6K + $10K + $5K). That plus a dent to the developer velocity and the release velocity, along with long windows of uncertainty with regards to debugging complex stuff and challenges. The original team quit after incurring extreme fatigue and just the team cost has now gone up to about $18K per month. All in all, net loss plus undue burden. Engineers must be tuned towards understanding the total cost of ownership over a longer period of time in relation to the real dollar value achieved. Unfortunately, that's not a quality quite commonly seen among tech-savvy engineers. Being tech-savvy is good. Being value-savvy is way better.
  • ghomem 2y ago
    Thanks for sharing the story. Despite the whole TCO being higher, I wonder how the 8K to 6K reduction happened. On AWS, fargate containers way are more expensive than VMs and non fargate containers are kind of pointless as you have to pay for the VMs where they run anyway. Also auto scaling the containers - without making a mess - is not trivial. Thus, I'm curious. Perhaps it's Lambda? That's a different can of worms. I'm honestly curious.
    • deleted 2y ago
      [deleted]
    • raghava 2y ago
      > I wonder how the 8K to 6K reduction happened. As said, most of their workloads were on cheap VPSs before. Moved some to 'scale-to-zero' solutions, reduced the bloat in VMs, fixed some buggy IaC, also moved some stuff to the serverless scene. That got a decent ~20% reduction.