7 ms·
This is the right way to deliver software. Produce working product first, validate the idea, stabilize the business, start generating profit, and then you can
by lpapez 22d ago
This is the right way to deliver software.
Produce working product first, validate the idea, stabilize the business, start generating profit, and then you can start optimizing your costs.
In fact optimization is by far the easiest part of the process because there are many system programming experts on this HN thread who consider these optimizations to be trivial.
- vanviegen 22d agoOr optimize a bit earlier and prevent having to scale out to a bazillion systems.
- deleted 22d ago[deleted]
- bcrosby95 22d agoThe way I usually prevent having to scale out to a bazillion systems is never getting more than 10 users.
- steve_adams_86 22d agoI wonder why Cloudflare didn’t think of this
- kevin_thibedeau 22d agoThis is Broadcom's business model
- gerdesj 22d agoQuite. I was a VMware fanboi (25+ years, man and boy) I still look after a few VMware estates and a lot of Proxmox ones (that used to run VMware). Hilariously, VMware is described as "enterprise class", which I can only conclude means MVP and a bit wanky. Today I repaired a Proxmox HA + Ceph node using boring old normal Linux skills and as it turns out I have 30 years of those. Part way through a remote v8 to 9 upgrade I think I lost comms due to using OpenvSwitch for networking and despite using tmux for the upgrade session. Anyway, the Proxmox ISO was useless for rescue but the classic systemrescuecd worked nicely and I could run dpkg in a chroot. VMware "used" Linux and never really gave back. I don't miss fixing vCentres and all the other nonsense that "Enterprise" wankery has foisted on me over the years.
- taneq 22d agoWhen I was first getting into software dev, I thought 'enterprise' meant 'industrial grade', 'powerful', 'high-performing'. Then I actually met some enterprise software, and realised that it means 'expensive', 'bespoke', 'one-off', and usually 'janky'.
- phoghed 22d agoEnterprise means it has SSO and a support contract
- esseph 22d agoLol, so true
- jacquesm 22d agoIt means you are paying for a support contract. Whether you actually have one time will tell.
- PeterStuer 21d agoOnly seasoned engineers will understand you are not joking.
- phoghed 21d agoSorry, the very simple thing you’re trying to do is too complex and non-standard for our support team to handle. We’ll gladly sell you a consultant for $400/hr. He’ll work on modifying our system, and then we’ll sell those modifications to everyone else.
- valenterry 21d agoAnd in-transparent pricing and this annoying dude that asks you once in a while if everything is okay, usually before the renewal.
- 22d ago
- bch 22d agoThe Art of Production
- froh 22d agopro move. made my evening.
- forgot-my-pw 22d agoThe art of premature optimizations
- smolder 21d agoThe art is in knowing how to write software that doesn't perform like shit without doing all the work of measuring and refining. If you can save $100k in hardware costs in a couple days by just knowing what you're doing, that optimization is not premature.
- jodrellblank 21d agoAnd yet when Prof. Donald Knuth wrote that in 1974 paper[1] it was in this context: > "The improvement in speed from Example 2 to Example 2a is only about 12%, and many people would pronounce that insignificant. The conventional wisdom shared by many of today's software engineers calls for ignoring efficiency in the small; but I believe this is simply an overreaction to the abuses they see being practiced by pennywise-and-pound-foolish programmers, who can't debug or maintain their "optimized" programs. In established engineering disciplines a 12% improvement, easily obtained, is never considered marginal" also: > "In the late 1960's we witnessed a "software crisis", which many people thought was paradoxical because programming was supposed to be so easy. As a result of the crisis, people are now beginning to renounce every feature of programming that can be considered guilty by virtue of its association with difficulties. Not only go to statements are being questioned; we also hear complaints about floating-point calculations, global variables, semaphores, pointer variables, and even assignment statements. Soon we might be restricted to only a dozen or so programs that are sufficiently simple to be allowable" In a recent comment I mentioned a youtube interview with Rico Mariani, a performance engineer from Microsoft, and he said that he often got called into projects approaching their deadlines and not meeting their performance goals. In one anecdote he spent a couple of hours with a team and showed how their design could never meat the goal even with the fastest disks, CPUs, memory, and network. And commented how strange it is if they had spent a day at the start of the project whiteboarding out the design against hardware specs at the start of the project - and avoided months of wasted effort - that would be called "premature optimization". [1] https://dl.acm.org/doi/pdf/10.1145/356635.356640 https://dl.acm.org/doi/pdf/10.1145/356635.356640
- bigbuppo 22d agoAnd that's why I charge $10,000,000/user/mo.
- dakolli 22d agoYou can build foundations that aren't extermely optimal but have future optimisations in mind.
- scottlamb 22d agoYou're never going to get promoted with that attitude! I'm joking...but not entirely. It sounds impressive on a promo packet when you say you've saved 100 TB of RAM / $$$ through whatever technique. But it sounds a lot less impressive when you say if this system grows to this size in x years, I will have saved 100 TB, especially when no one yet knows how large the system will really be in that time or what the cost of RAM will be. I dunno, maybe if you say that x years ago, I made a decision that now is saving us 100 TB, that's kinda impressive, but you're also getting credit for it x years after you did the work. It also doesn't have the implication that it must be inherently complex/hard because some other smart person chose the other way. And there is a bias to care more about recent accomplishments. So I don't really think it'd be valued the same at all. Also, in general big tech (at least Google) prefers growing the userbase over improving efficiency. Periodically efficiency is rewarded, e.g. when RAM cost suddenly balloons or some big must-have feature has suddenly used up capacity planned for something else. You get rewarded for doing efficiency work on demand, not eagerly. I once got a $100 peer bonus for finding 100,000 cores that were essentially stranded by an accounting error in another team's migration script.
- Dylan16807 22d agoIt was already reasonably lean. If they had 10 bazillion systems, they now need somewhere between 6 and 8 bazillion systems.
- otterley 21d agoRemember that everything has an opportunity cost. Running a lot of servers might cost $10 million annually, but if the product team had to choose between a project that would recoup $5 million of that vs. an opportunity to earn $50 million ARR for the same amount of work, the logical answer would be obvious.
- vanviegen 21d agoThat's assuming the ops team has infinite capacity.
- Retric 21d agoIn my experience optimizations actually preformed tend to have ridiculously high ROI because they are so rarely prioritized. Better performance = saving money + better user experience.
- mtrovo 21d agoDepends on how you measure your ROI and how it could be different from how your company measure their ROI. The problem with optimizations is that you are competing in prioritization with other features. Reducing the baseline cost always has a limit of zero, while the upside from new features is infinite according to your leadership and investors, so it is very hard to argue against. In general it's challenging to convince a non-tech crowd of the importance of addressing any tech debt unless you can demonstrate a tangible financial impact on the product, such as delayed contracts or customer churn.
- 27183 21d ago> The problem with optimizations is that you are competing in prioritization with other features. I think companies often over-indulge in features nobody wants, needs, or cares about. I quit my previous company because they were forcing us to build something that had single digit weekly active users. It was utterly pointless, driven entirely by some half baked navel gazing harebrained ideas about what a "nontechnical user" might want. But nobody ever asked any real users. I estimate the company probably blew the greater part of $10M on this bullshit, not counting opportunity cost. > In general it's challenging to convince a non-tech crowd of the importance of addressing any tech debt unless you can demonstrate a tangible financial impact on the product, such as delayed contracts or customer churn. People like that are problematic not just because they don't understand tech debt. They also don't understand products. There are shitloads of people in the industry who market themselves as some kind of mystical gurus, are able to deliver impressive monologues talking over everyone on the zoom call, but contribute nothing else than a sense of urgency and frustration. If you find yourself in their company, better to just leave.
- hinkley 21d agoThis is perhaps the hardest nut to crack. We give perverse incentives to the marketing department to bring us the most customers they can, instead of the most appropriate customers. Someone told me at my second lead position that they had landed a top-tier customer for our demo-ware and the first words out of my mouth were, "FUCK ME". Not the response they were expecting, but then that guy never did end up understanding me the entire time we worked together. My bosses did though. When your system is new you're a loss leader for all of your customers. Every dollar they bring in costs you two, possibly more. You have to get to positive MRR before the venture capital runs out, and over a long enough time horizon, you will go for a new round and find out that a recession is about to start and the VC guys getting cagey is the first clue it's coming. Meanwhile if you spend all of your time and energy on reducing costs instead of increasing revenue, then your competitors catch up with you. Particularly if their funding rounds are half a cycle off from yours - one of you will be the last one to have gotten a cash infusion before the money dried up. Most recently I worked at a place that didn't understand my calls for sobriety until it was too late. So I got put in charge of a rear-guard action that was too little too late, and our customers all fled to much cheaper competitors who could do 80% of what we could for half the price. I learned some good stuff, but the company got bought by a competitor who scrapped those systems.
- casey2 22d agoThis reasoning assumes you have access to infinite runway. You don't.
- lpapez 22d agoExactly, and you need to start turning a profit before the end of that runway. Even if that means running code that is suboptimal.
- ramon156 22d agoi suppose you could say the same about buying a house. just make that initial 300.000 and from there on out its easy. everything looks better on paper
- topato 22d agoMy house is a ~700 sqft. condominium, gov. subsidized for lower income individuals, and even my mortgage is more than 300k… maybe I’m just basing my info off of coastal city prices, but is it possible to buy a reasonably nice home located in a reasonably nice amerikkkan city… for $300k in 2026?
- esseph 22d agoThat would buy you thousands of square feet and often several acres within 20 minute drive to a lot of US city downtown areas.
- 8n4vidtmkvmk 22d agoNot from what I've seen. The desirable neighborhoods are 500-600k. Suburbs outside major city
- esseph 22d agoYou didn't read what I said. I mentioned acres of land. You normally don't have multiple acres of land in the suburbs.
- sieabahlpark 22d ago[dead]
- MuffinFlavored 22d ago> Produce working product first, validate the idea, stabilize the business, start generating profit, not everybody is so lucky to be able to go in that order? The first part requires upfront capital/investment?
- phoghed 22d agoSo obviously you start at optimization
- tonymet 22d agoOnly if you have loads of capital
- nine_k 22d agoThis assumes that you have plenty of cash to burn in the process, which is approximately correct for VC-backed ventures, and for offshoots of large corporations that play a lomg game.
- hinkley 21d agoI have a mountain of bad things to say about Microsoft, but they definitely had loss leader products that never made a dime but pulled you into their ecosystem. So they funneled money from a couple of cash cows back into the other projects to keep it enticing.
- aeonfox 22d ago> start generating profit, and then you can start optimizing your costs Good thing they jumped on that as soon as they were profitable instead of burning cash. Oh wait... I think a distinction to draw here is that Cloudflare had relatively large capital raises and were almost immediately profitable¹. They had the luxury of throwing away money. Judicious optimisation makes sense for scrappy start-ups, especially when trivial optimisations like these could easily be farmed off to an agent. ¹ https://timeline.www.cloudflare.com/ https://timeline.www.cloudflare.com/
- sandeepkd 22d agoIts a yes if you do not know the domain space, query patterns well enough and also if the cost of optimization or time for optimization may have detrimental impact to business. In this case it most likely means that the crowd in the room did not anticipate much on this in early phases and no one in the room pointed these things out. The irony is that these performance and disk numbers are heavily discussed as a part of system design interviews. > In fact optimization is by far the easiest part of the process because there are many system programming experts on this HN thread who consider these optimizations to be trivia This is a misconception when you including roll out as a part of the change too, changing data once its running in production is hard, changing the data structure is even harder and when you talk about making changes in cache which is at the hot path its probably the hardest. Looking at the graph at the end it looks like it took them 4+ months to roll out the changes after optimization.
- switchbak 22d ago“changing data once its running in production is hard, changing the data structure is even harder” 100% agreement on this. There are a class of optimizations that can happen transparently. Those can happen at any time, and are fine to defer. Not all profiling and scalability improvements fall into this bucket. Some are very expensive to roll out, and ignoring these concerns can cause huge headaches down the line. Not fun to hear, but it’s definitely true. Even with LLMs, this can still be a huge challenge.
- tyre 21d ago> This is a misconception Pretty sure they were joking.
- brainless 22d agoI do not think Cloudflare was a less-than-peers optimized product when they launched. This is one of their blog posts which describes taking one aspect even further. I think Cloudflare became big only because they were so much more optimized than others that they offered some services for free that others were not offering. If running costs are high, you only burn (VC) cash and then you exit.
- majke 21d agoFirst version was a three layer VM monster, with the main entrance done as HTTP proxy written in PHP.
- Voultapher 21d agoThere is the entirely plausible option of the NSA indirectly bankrolling them to counteract the growing number of TLS connections. We terminate your TLS for free in our server and you don't have to change anything was a sweet deal for website operators _and_ those that want to look at unencrypted traffic. Given enough sustained funding they could undercut competitors and grow to what they are today.
- Sarkie 21d agoMake it work, make it fast, refactor
- adornKey 21d agoWhat I find surpring here is this being about DNS. Simple optimization should have been done maybe 1997. Letting it build up to 100 TB is noteworthy, but on the other hand for IT that's common. It's surprising but also totally expected...
- compounding_it 21d ago>optimization is by far the easiest part of the process Not if the whole thing is architected poorly but was a requirement of the hour so it became big. Then optimisation becomes an art, but definitely not the ‘easiest part of the process’
- DanielHB 21d agoYou underestimate the cost of "optimization", sometimes it means actually rebuilding large parts of the system. I would not say it is the "easiest" part, but it is usually not what will kill your business though.
- suncore 21d agoNot the right order if the optimization is a prerequisite for a positive business case. That happens more often than people think...
- saidnooneever 21d agoright or wrong, good or bad are all taste. from a business perspective, this might be considered the only way, but it is not. at large volume scale it becomes more, but often large scale is lacking optimisations in the first place. its not wrong in my eyes, but definitely not the only path to take.
- gbin 21d agoThis is true if you can scale out (ie you can add resources to your system). But for a robot for example, just adding a GPU can just flop your product completely: you need more battery, more weight, suddenly your unit economics is out of the window... Your next hardware iteration will be very slow to come and very expensive. So here, you better not have a system wasting too much resources pretty early on after the prototype phase.
- chii 21d ago> start generating profit, and then you can start optimizing your costs. this assumes you can generate profit before you can get optimized - what if profit generation is only possible with optimized software? A lot of online MMO-style games tend to require such optimizations as they scale into the size required to generate profit. Or, in the current era of ai, the cost of the capital investment is far exceeding the ability to generate profit off it. The optimization in how the resources gets used will be needed to cut the costs down, and allow increase in the scale of usage for the same hardware. That's where profits would lay. Of course, in order to achieve any of this, you'd need the runway to survive until such times. A small scale operator won't have this runway, and so die before they can accomplish anything profitable (or get big by begging for investor money to grow large - as we've seen in the past 20 years of tech).
- hinkley 21d agoI've seen a lot of people relearn the same couple of lessons over an over again. One, most of the things you believe scale linearly scale at nlogn because of physics. Storage is much worse, scaling at n^1.5. What fit in a single 4U case is orders of magnitude slower when it no longer fits in a single data center. Inserting records into a table with a billion rows is much slower than a table with ten thousand. Wipe the surprised Pikachu look off of your face. Second, every architectural choice needs to be reviewed every time traffic increases by an order of magnitude, and preferably before so that you have time to stew on possible solutions and allow serendipity the time it needs to find you a better answer. If you have made too many architectural decisions, you will find there aren't enough hours in a month to keep up, even if you never sleep. That's the real reason we practice Principle of Least Power. Not security, but job security. Keep the system small enough that you have time to keep the plates spinning while also implementing new features to attract or retain customers.
- aperture147 21d ago> This is the right way to deliver software. It's only right if you are not the one who pay the bills.
- hinkley 21d ago> In fact optimization is by far the easiest part of the process because there are many system programming experts on this HN thread who consider these optimizations to be trivial. <strange high pitched equivocal sounds> I do a lot of tuning work that others either can't be arsed to do, or more often are scared shitless of touching (learned helplessness). I tend to work on things that have to go right the first time in general, but I firmly believe that I learned that skill doing optimization work early in my career, not the other way around. Through a series of unfortunate events, my first specialization was performance, before I got good enough at anything else for people to listen to me. Ultimately, the single thing that fucks up performance work, far and above any other concerns or sources, is that the vast majority of developers want to skip from Make it Work to Make it Fast without going through Make it Right. Then they have to use dodgy 'optimization' tricks that embrittle the entire thing, destabilize production and make every new feature feel like walking a minefield. So what happens is that when someone who actually knows what the fuck they are doing comes along and wants to build a superhighway, they find the route littered with abandoned 'vehicles' that have sunk into the mud up to their door handles and left to cement in place. Clearing the proverbial route becomes a bigger technical challenge than building the road. And so you'll find that your peers have a bunch of ideas about what needs to be done but none of the stomach to do any of it. Which is why I say "stubbornness" is the most important tool in my performance toolbox, far above cunning.