26 ms·
Why Twitter didn’t go down: From a real Twitter SRE
- NaturalPhallacy 4y agoAll the people saying twitter is now dying remind me of "the walls closing in" on Trump, who is running for president in 2024: https://www.youtube.com/watch?v=rLEchPZm318 https://www.youtube.com/watch?v=rLEchPZm318
- fractalb 4y agoTwitter will soon experience the negative effect of the mass layoffs and die slowly. If it survives, please don't ascribe it to Elon's management skills. Don't forget the employees who are stretching themselves there to rescue Twitter.
- superdug 4y agoI will just wait for the bug the breaks the camels back here. There will be a necessary breaking change that there will be no support for that will cascade into the downtime the media is demanding at the moment.
- kar1181 4y agoFor all the hand wringing about keeping the lights on, Twitter more than doubled its headcount in 3 n bit years. I'm not sure anyone knows how much of a skeleton crew is required to keep things ticking along anymore given that sort of hiring binge.
- tzm 4y agoLooks like 28% of Twitter employees took the offer for 3 months of severance.. not the rumored 75% or 80%.
- newbieuser 4y agoConsidering the maintenance cost of a twitter-scale site, it is inevitable that even a very small-scale problem they will experience will become much larger with the snowball effect.
- wnevets 4y agoI know it became a meme but did people actually think it would just crash overnight? Aren't a lot of people on hacker news in tech?
- Existenceblinks 4y agoWell, debugging a company in prod mode like this is adventurous. Just like we remove code to see if things break, and put it back piece by piece until it's not. That's how some of us get rid of bloat. (Yes, I'm aware of orange and apple comparison)
- nokeya 4y agoSo, from 7500 people only 1 (ONE) person was responsible for cache, one of the most crucial parts of the service?
- mtejo 4y agoHey, sorry for the new account, i just like to try my best to keep my online identity separate. this for better or for worse has my real name on it. Hope this is interesting!
- cloudking 4y agoNice work! Does Twitter runs their own DCs or hosted somewhere else?
- devoutsalsa 4y agoAs of three years ago, Twitter had 2 to 3 data centers, and was moving some stuff to GCP. Not sure if current state of things.
- throwaway892238 4y ago> and was moving some stuff to GCP my condolences
- r721 4y ago>Mr. Musk is also considering shuttering one of Twitter’s three main U.S. data centers, a location known as SMF1 in Sacramento, which is used to store information needed to run the social media site, four people with knowledge of the effort said. If the data center in Sacramento is taken offline, it will leave the company with data centers in Atlanta and Portland, Ore., with potentially less backup computing capacity in case something fails. https://www.nytimes.com/2022/11/18/technology/elon-musk-twitter-workers-quit.html https://www.nytimes.com/2022/11/18/technology/elon-musk-twit...
- vonmoltke 4y agoThere are currently three data centers, hosting all of the real-time and most of the batch production load. Ad-hoc load and some production batch load was migrating to GCP, but this was being significantly curtailed in the months before the acquisition closed because it turned out to be very expensive, much more so than anticipated and more than the equivalent workloads had cost in the data centers.
- danielodievich 4y agoExcellent article. However!, software, like everything, is subject to laws of physics. Entropy always wins in the end. No matter how good the original engineering and planning, without maintenance it will all fall apart soon enough.
- enneff 4y agoRight. It’s not like if all the engineers walk out it’s suddenly going to fall over. It’s that when an issue comes up there may not be the expertise on hand to fix it. So the remaining twitter engineers should expect a rough time over the coming months.
- emodendroket 4y agoThere must be some plan for this, though, right? If it were me and I were trying to do more with less I'd plan to cut some features (for instance, are Spaces that critical to the operation? Because it seems like running them would be demanding) and try to migrate things to managed services to reduce the operational load.
- jason-phillips 4y ago> There must be some plan for this, though, right? To be this young and carefree
- hotpotamus 4y agoI don't have a DR plan for "mad billionaire buys the company", no, but I do agree with the OOM reaper logic - I start cutting the services that use the most resources that are least necessary.
- emodendroket 4y agoI don't mean that they had one sitting around, but anyone who's stayed with the company has to find this their most urgent priority, I would think, between the focus on cutting operational costs and the simple fact that they have far fewer people around.
- wankle 4y agoI can't help but feel it might not be legal to post all those details.
- HeyLaughingBoy 4y agoWhat law do you think is being broken?
- encryptluks2 4y agoBreach of contract possibly. Usually companies don't want you revealing their internal infrastructure publicly unless you get permission. Who knows though.
- emodendroket 4y agoWouldn't that be a civil claim and not actually a law being broken then?
- wankle 4y agoThere are entire sections of the law around trade secrets (intellectual capital).
- emodendroket 4y agoNothing here seems to approach anything like truly proprietary knowledge and the kind of information it contains is pretty similar to what goes up on engineering blogs (or what you'd describe in a job interview, if you prefer) all the time.
- wankle 4y agoYes exactly this. Maybe they did but I didn't find it stated in the writeup, which was actually pretty good. If it were my company though, I'd want the intellectual capital protected and we programmers usually have to sign non-compete agreements so I think my original comment stands, it was my opinion and initial gut reaction, not an indictment or judgement in any way.
- yieldcrv 4y agoI don't think it's that crazy. Private Equity and Wall Street doesn't have pressure for resilience, it has pressure for churn. As long as revenue and spending... exists... things are fine. Ideally they are numbers that grow. This doesn't lend itself to clearing out a development backlog, or engineers doing the most important things, it lends itself to rapid iteration and justifications for the iterations. So now that it is owned by someone that doesn't need churn and just needs to reduce cost, people can focus on discreet resilience factors, just like any small tightly held software operation. Many of Twitter's pet projects go away, the ad sales relationship have to get re-evaluated too, but the core consumer product that everyone sees can be made resilient and operate cheaper.
- danans 4y ago> Private Equity and Wall Street doesn't have pressure for resilience, it has pressure for churn. As long as revenue and spending... exists... things are fine. Ideally they are numbers that grow. This doesn't lend itself to clearing out a development backlog, or engineers doing the most important things, it lends itself to rapid iteration and justifications for the iterations. Did we read the same article, because it seemed clear that all that resilience work was done when the company was public shareholder owned, not under the new private owner.
- yieldcrv 4y agoI read the article, I wanted to use the space to point out that Elon's fortunate that it has been done, and also that the technology infrastructure things he needs to focus on aren't impossibly challenging to do, if people are expecting it to be. So operating Twitter with 80% fewer engineers isn't the voyeuristic suicide that many of us are hoping to be amused by.
- acdha 4y agoThe problem is that it’s run by someone who overpaid badly and needs to come up with significantly more money to pay for the debt he saddled the company with, and then his actions seriously disrupted ad revenue. That puts him more in the PE playbook of cutting costs as deeply as possible even at the extent of long-term growth. Unlike his other companies there isn’t strong government support to drive business for Twitter.
- uluyol 4y agoI wonder how big Twitter's infra costs are (capex). Based on the article, they need to build capacity > 2x peak loaf to tolerate one of the DCs failing. But if they instead had 4 and still only tolerated 1 failure, they'd only need capacity > 1.33x peak load. This doesn't account for the extra overhead associated with extra DCs, but it seems like there's opportunities for major effeciency wins.
- bigtones 4y agoWell Elon is planning on shutting down one of their three data centers, the Sacramento data center - so they must have a lot of extra capacity. https://www.datacenterdynamics.com/en/news/report-elon-musk-considers-shutting-twitters-sacramento-data-center-leaving-just-two/ https://www.datacenterdynamics.com/en/news/report-elon-musk-...
- midoridensha 4y agoGo Elon! Shutting down a datacenter will save a lot of energy. I applaud this move. (Though I suppose they could just sell it to some other company.) Now he just needs to shut down the other 2.
- hanafudafan 4y ago
- lolsoftware 4y agoI'm curious how/if the ostensible decline in active users is affecting things, and to what degree.
- user_ 4y ago> Twitter added 1.6M daily active users this past week, another all-time high https://twitter.com/elonmusk/status/1594865247323660290 https://twitter.com/elonmusk/status/1594865247323660290
- datalopers 4y agoInteresting how this summer all we heard from Musk was how Twitter was overflowing with bots and spam, but now that he runs the place he sure spends a lot of time bragging about all-time high "user" counts.
- acdha 4y agoWant to bet that’s not unconnected to sacking the people who tried to block bots?
- user_ 4y agoI'm sure it's connected, but not because of the bots. Millions of political moderates are stopping by to see what happens when a large-scale public forum begins to tolerate freedom of speech.
- astrange 4y agoInterestingly, my reports on racist posts are being accepted much more often than they were before, so this is in fact not happening. The site was also consumed in horrific discourse for the last week (topic being "is making chili for your neighbor racist, ableist and oppressing autistic people?") so all in all it seems to behaving as usual.
- acdha 4y agoExactly what kind of views do you think moderates were worried about not being tolerated? Going by the definition of the term it seems unlikely that anyone, much less millions, was previously deterred by policy.
- irrational 4y agoVery interesting article. But the constant misuse of “to” for “too” kept throwing me out of reading mode. I wonder why my brain does that instead of reading past it since I know what is actually meant.
- blamazon 4y agoInteresting, I didn't notice despite reading the article start too finish. The brain is so mysterious!
- stoniejohnson 4y agoIrony!
- smegsicle 4y ago> "to many" - 5 matches he did get it right one time though
- emodendroket 4y agoI think people's expectations became so exaggerated it was inevitable they wouldn't be lived up to. I'm sure Twitter will experience degradation from the drastic cost-cutting, but it was never going to happen overnight and I'm not sure why news outlets were saying that (except that their sources were employees with slightly inflated senses of their own importance, which we're all sometimes guilty of). And people became really invested in the idea that a site cannot possibly stay up without dedicated SREs, as though tons of tech sites (including big names like Amazon) don't just devolve this work to their on-call rotations.
- headsoup 4y agoBecause politics and ideologues are so prevalent and everyone's attaching 'the other' (both sides) for the own. And of course people in 'important' roles who've been laid off are going to say the company is doomed, these are the probably the worst source to go off. They're not going to say "oh yeah I didn't do much at all really, just bossed people around and spent my budget every year."
- AustinDev 4y agoI love this comment. It's so salient. It's probably hard for a lot of people on this board to digest but, I feel like most know where you're coming from.
- emodendroket 4y agoI don't even doubt that they did a lot of work, but as this article suggests, that's part of the reason why we _wouldn't_ expect the site to just go down in a couple days. If all the servers needed constant manual massaging that would speak more poorly to their work than the other way around.
- headsoup 4y agoI guess that highlights that for some time we just won't know the actual effects and everything right now is just posturing and speculating. If in six months (or suitable time period) we see stagnation in releases and technical issues increasing we'll have some data to infer from.
- raldi 4y agoThe most helpful thing to reflect on in these Twitter operational discussions is the difference between homeostasis and evolution. You can get rid of 80% of the work force and the existing homeostasis systems will keep things running smoothly despite known day-to-day chaos. Where you’re really going to run into trouble is inventing responses to novel chaos and gradually changing times.
- oska 4y agoWe had an obituary for Fred Brooks on here just the other day. I'd suggest that his thesis in The Mythical Man-Month conflicts with your comment above (that reduction in staff count for a software project has a good correlation with the ability to maintain it / evolve it / innovate on top of it).
- sangnoir 4y agoI've never heard of (or thought of) your interpretation of the corollary to Brooke's Law, but removing people from projects until they succeed and are on time seems like a bold strategy.
- oska 4y agoHappens regularly in the Free Software & Open Source world, where software projects are often forked by a single individual. Obviously enterprise software is on another level, but the principle remains that reduction to a smaller development team (for some period of time) does not necessarily correlate to a threat to viability and has often indeed been a reinjection of vitality.
- deleted 4y ago[deleted]
- judge2020 4y agoI'm sure we're going to see some sabotage accusations once this happens.
- emodendroket 4y ago
- ulkesh 4y agoAll this does is point out that smart people worked at Twitter who may now no longer work there, whether on their own accord, or due to Elon’s bulldogging tactics. Elon thinks he knows what he’s doing, but what he is going to be left with are people who are willing to work hard by his standards, but not necessarily smart. The simple truth is Elon knows nothing about the actual work involved in tech. He knows words or elicits help from others on what to say that sounds like tech speak (RPCs!), but when it comes to being truly knowledgeable in this space, he is losing his most valuable assets because of his amazingly poor managerial and ownership style. I know there are a lot of Elon fans on this site, and will disagree with all of this; but his abilities have not at all been proven. Yes, he knows how to spend money to claim credit for technical advances, but until he actually has his hands dirty in the muck of the hard work of tech, he will always be a glorified self-promoter with no substance. And Twitter will suffer for it.
- headsoup 4y agoNothing beats an internet random telling us how much a successful person is silly and also can't meet their own superior standards. It's not about being a fan or not, it's that you're not actually providing any real insight other than signalling how smart you are.
- theclansman 4y agoThese people behave just like the irrational fanboys, except they just do the exact opposite. Being a sheep and being a contrarian sheep are the same thing.
- intelVISA 4y agoI've no dog in this race but I'm excited to see what influence, if any, this will have on the topology of similarly inflated tech companies.
- Loughla 4y agoHonestly, I'm hopeful that other coastal tech companies do clean house. I genuinely believe it could lead to a resurgence in the lives of Midwest/middle American states and cities. There is an absolute vacuum of technology specialists in the middle of the US, because no one wants to "live in the middle of nowhere," and they don't want to earn less than FAANG (MAMAA?) salaries, when half of those salaries can give you an amazing life in the middle of the country (source: my piss-poor salary compared to yours).
- swellguy 4y agoI think the real question is: Twitter grew 3x on the headcount front with a flat stock price over the course of less than 5 years. What exactly where these thousands of employees actually doing and why did the previous CEO think what they were doing was worth hiring them for? That's just basic accountability from a stock holder or employee perspective. That's apparently a ton of money being wasted on nothing at all.
- mirzap 4y agoYeah, and they were even profitable with ~3k employees. Then the hiring spree started and they went negative. Even if there wasn't Musk they would have to let go at least 30% of the people.
- spaceman_2020 4y agoThe stock was rightly crashing when the company was public. A social media site that manges to lose money during the pandemic is truly mismanaged.
- halfmatthalfcat 4y agoThis is the real question? Your question has nothing to do with the blog post and if you take a look around, what Twitter did was literally done across the entire industry, hence all the layoffs recently. There was a hiring glut to take advantage of cheap capital during COVID recovery. The capital has dried up, glut has ended and a lot of people lost their jobs. Why is that so hard to see? None of this is unique in any way to Twitter.
- subroutine 4y agoA bunch of people just got axed from Twitter because covid cash dried up? Shit I thought it was because Elon took over and fired anyone who refused to work at the office instead of at home.
- halfmatthalfcat 4y ago
- yuppie_scum 4y agoI hope they consider moving to Kubernetes from Mesos.
- rubiquity 4y agoWhy
- robswc 4y agoIt was if I was in two realities the other day. I did experience a few bugs but twitter never went "down" for me, despite everyone claiming that 1) it would and 2) it already did. All 10 of the "top trends" for me were about how twitter was dead/dying and armchair experts on reddit said it was only a matter of hours at this point. Maybe it was close (in reference to external factors not implied in your very insightful post!) but its amazing how confident people are with opinions on events they have 0 insight into. Everyone knows how to solve the war in Ukraine or world hunger but how rare is the "consensus" (in terms of up-votes or popularity) right... just something that got me thinking. Thanks again for this article. I always love seeing details of tech ops!
- jeffbee 4y agoA few parts clearly did go down. 2FA login was just serving error codes all day a few days ago. On Saturday people were posting entire feature films, 2 minutes per tweet, because apparently their copyright content matching wasn’t working.
- School-Cotton 4y agoThose are hardly “Twitter going down”. They’re normal medium-severity outages that could have happened anywhere. Sure, it’s plausible they were made more likely by so many people leaving, but they’re not exactly the meltdown people predicted.
- jakevoytko 4y agoAt least one high-follower account on my timeline posted that their newest notifications were from October
- robswc 4y agoI wouldn’t be surprised. There were definitely a few bugs. I think ppl unhappy with Elon overplayed things though and now it will look silly. Our expectations were anchored by the idea that Twitter would cease to exist. So very real problems like 2FA going down look trivial compared to the former.
- totorovirus 4y agoIf Twitter survives and people shut up their mouths complaining after few months we'll see who was right.
- ReptileMan 4y agoAnd then the other big tech companies will take note. I personally know of highly compensated teams in VMware that do almost nothing.
- EagnaIonat 4y agoIt's already breaking down. HW/SW aside, a lot of the services had teams monitoring them, which don't exist anymore. Some of the side effects of that: - Trends are not working correctly. - Copyright reporting is not working. - Appealing flagged tweets isn't being responded to. It's only a matter of time before these get abused with no one to fix them.
- NaturalPhallacy 4y ago>- Trends are not working correctly. "Trends" were subject to Twitter's "trends blacklist" before; something that leaked a few years ago. Maybe they're working correctly now that they're unencumbered. Can you describe how they're not working now? >- Appealing flagged tweets isn't being responded to. Mine was responded to in a handful of hours. Much faster than I was expecting. >It's only a matter of time before these get abused with no one to fix them. "the walls are closing in" - https://www.youtube.com/watch?v=rLEchPZm318 https://www.youtube.com/watch?v=rLEchPZm318
- EagnaIonat 4y ago> Can you describe how they're not working now? The team would moderate junk trends. So some of the trends are literally just random words being spouted. Similar to early day twitter. > Mine was responded to in a handful of hours. Recently? That's impressive when the related teams are gone. > "the walls are closing in" I am not sure what a YT video about Trump has to do with my comment. I refer you "in comments": https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- dcj4 4y ago
- a-dub 4y agoi think i prefer a culture where one is rewarded for automating their job. moves like musk's are bad for the industry. they produce more systems with "job security" built in.
- TravHatesMe 4y ago> more systems with "job security" built in. How common is this?
- fragmede 4y agoThe "why hasn’t the site gone down" stuff is the wrong conversation. The engineers built a train engine, and this robber baron held up the train and forced the engineers off. Inertia will keep the train going for a good long while because they did such a good job of building the train. At some point the engin’s fluids will run out and need topping off. There's a warning system though so someone'll get notified before the engine blows up, so it's entirely possible this hardcore crew of H1Bs pulls it off. (Great job replacing the SSL cert!) At some point the train’s boiler will need retrofitting. A new team could successfully replace the boiler, but doing it while the engine keeps running just isn't an easy job. The handful of devs and SREs at Twitter will do their damnedest to keep the train, err, site running. They might even succeed. That doesn’t prove the site was over-engineered, it means they built a damn fine train engined that successfully pulled the whole train up and over the mountain pass. Twitter wasn't run like a lean start-up type business. Because wasn’t one. Society needs better than that. The vulture capitalism mindset is what's wrong with this county. What's the barest minimum I can pay people to work for me? Not; what can I pay them to let them live prosperous lives but what's the bare, subsistence minimum. America needs a middle class and these people were part of it. You don’t get a middle class by injudiciously paring things down to a not-even-skeleton crew that’s going to be worked till they burn out, only to get fired. Twitter was only $14 million from profitability in Q2, and any idiot can randomly fire people to find that much in that large an organization, but cutting costs like that is to misunderstand the situation entirely. Just like starving yourself isn't healthy and won’t improve your self image, randomly cutting costs like that saves money, but doesn’t result in a healthy business.
- zapdrive 4y agoI know most of my comment is going to be redundant, but I need to say it in order to point to it an year or two from now as an "I told you so!" Twitter is going to be here for a long long time. It's not going to suddenly shut down, it's not going to slowly decline, there is not going to be mass abandonment. Elon might be cocky, but at the end of the day he is a successful businessman. He didn't just sent out that loyalty pledge email out of cockyness. He really wanted to kick out everyone who doesn't believe in him and his beliefs. And he sent that email well after understanding how Twitter runs and putting his loyal people from Tesla ans SpaceX in charge of operations. The woke crowd needs to get out of denial and start coping with the fact that Twitter is not the bastion of unopposed woke ideology anymore.
- faizmokhtar 4y agoWhat kind of woke ideology exactly? Sorry not familiar the current climate in USA.
- Bud 4y ago
- acknorabotr 4y agoIt's a form of progressive ideology that is largely based on symbolic actions and relies on mechanisms like shame and shunning, struggle sessions, etc. So instead of actually doing any sort of real tangible action to solve problems, you mostly spend your time denouncing your enemies, including the ones in your own ranks that are not pure enough. It's very much associated with the left-wing parties in many Western countries, who have for the most part traded their historical defense of working classes for this new brand of performative progressive politics.
- linuxhansl 4y agoDefinition of woke: alert to injustice in society, especially racism Somehow, some folks are now saying this is bad.
- kaba0 4y ago
- poulsbohemian 4y agoI did SRE consulting work for a phase of my career... as the author points out, these systems are scaled out and resilient, but what happens next is entropy. Team sizes shrink, everything starts to be viewed through a cost cutting / savings lens, overtaxed staff start ignoring problems or the long-term view because they are in firefighting mode, it becomes hard to attract new talent because the perception is "the good times are over." Things start to become brittle and/or not get the attention needed, contractors are brought in because they are cheaper and/or bits get outsourced to the cheapest bidder... the professional care and attention like the author clearly brought just starts to shift over time. Consultants like me are brought in to diagnose what's wrong - the good staff could write our briefs, they know what's going on - and generally we slap a band-aid on the problem because management just wants to squeeze whatever value they can out of the assets rather than actually improve anything.
- bigiain 4y agoNew owner: "Nothing ever goes wrong with the cache! It just works, look at the status logs. Why are we even paying those guys to look after it!" Also new owner: "What even is Mesos? Why are we running something called Aurora? Obviously pure bloat. Fire the lot of them."
- radu_floricica 4y agoYeah, but that's not what's happening here, is it? Musk is pretty obviously looking for a corporate culture shift, not turning the company into a cash cow - and for this to work, entropy needs only to work slower than half a year. Which this article argues pretty convincingly is going to happen.
- mirzap 4y ago
- henrikschroder 4y agoYour reply is indistinguishable from a religious person exclaiming that "God is omniscient and moves in mysterious ways!" Stop worshipping billionaires.
- didip 4y agoI am mostly curious about the Mesos layer itself. Mesos is dead. So you need in-house expertise to patch it without being able to leverage community knowledge. Does Twitter retain enough people to manage Mesos?
- wyclif 4y agoI was surprised to hear that a cutting edge social media app uses Mesos. Why did they choose that over other options?
- rickette 4y agoTwitter didn't start yesterday. There was a time when Mesos was all the hotness and k8s was this new thing that looked promising but wasn't nearly production ready. Apple is also a big Mesos user, but also moving to k8s.
- __warlord__ 4y agoIIRC Mesos was an internal tool (at Twiiter) that got released under the Apache umbrella later on. Then Mesosphere (a company) wanted to bring it to the enterprise market but at the time was competing with Kubernetes... and we all know who won.
- minhazm 4y agoThey were one of the first users of Mesos, but they didn't create it. > Mesos began as a research project in the UC Berkeley RAD Lab by then PhD students Benjamin Hindman, Andy Konwinski, and Matei Zaharia, as well as professor Ion Stoica. > The social networking site Twitter began using Mesos and Apache Aurora in 2010, after Hindman gave a presentation to a group of Twitter engineers. https://en.wikipedia.org/wiki/Apache_Mesos https://en.wikipedia.org/wiki/Apache_Mesos
- vonmoltke 4y agoAurora, a container manager on top of Mesos, was developed in-house. Mesos itself was not.
- squegles 4y agoNot sure when OP left, but Twitter has 3 datacenters now.
- mtejo 4y agoOver the summer, Yeah there is three but I didn't think the 3rd was really running yet. It was kind of just there with some small things dripping in.
- dchia 4y agoWhy do people assume that things would break so easily? I'm pretty sure Twitter (ex)team did tons of scalability testing with loads that are 10x or even 100x. Plus random 100x to 1000x spikes to account for breaking news. And all of these are done automatically, constantly, and consistently over the years.
- hairofadog 4y agoRegardless of what’s up with the tech stack, I find it difficult to want to be a part of whatever this is: https://mobile.twitter.com/elonmusk/status/1594500655724609536 https://mobile.twitter.com/elonmusk/status/15945006557246095... (screenshot: https://raw.githubusercontent.com/aboxwithrocksinit/test-bucket/c43139083b1a2d51ca5a0f8f03adcab9295a6efd/New%20Note.png https://raw.githubusercontent.com/aboxwithrocksinit/test-buc...) If this is the new town square, you can forward my mail to a cabin in the woods.
- throwawaylinux 4y agoI think that's pretty hilarious. It has caught the ire of a lot of furious unhinged puritans too, which has actually been the funniest thing about it for me.
- HyperSane 4y agoIt would be funny for Musk to share privately with friends. It is a deeply strange and inappropriate thing for the CEO of a billion dollar company to post publicly. Something is deeply wrong with Musk's decision making process.
- Operative0198 4y agoHe can get away with posting just about any silly thought because he has no board of directors to answer to and no one can fire him. Big difference when you are thinking of him as a CEO.
- HyperSane 4y agoI understand how he can get a way with it, just not his thought process.
- s1artibartfast 4y agoWhat would stop you from posting that? Are they internal or external barriers?
- areoform 4y agoThere's a pattern of diverging expectations here, one is the non-technical/naïve one, - Twitter is going to go down tomorrow and it's all over. RIP. The second is, - Twitter is going to experience a failure cascade over time. The third is, - It's all going to be fine. I suspect that the real question is, how many individual wires can break before the cable holding the suspended platform snaps? I am not that good of a developer, but watching Twitter I can't help but be reminded of Arecibo, except at a larger, more abstract scale. There was no single massive event that caused the failure, rather a series of factors and events, tiny cables breaking that eventually leads to a failure cascade that then causes the suspended platform to crash. From what I can tell, in the past week or so, - Twitter's copyright system failed - Two Factor Authentication broke down (it seems to be back up?) - (anecdata) Tweets have been loading sporadically for me and other people, sometimes we try to open a tweet and it says that it doesn't exist. Happens more frequently with new/recent tweets. - (unconfirmed) Twitter's managed account backend is behaving "strangely." For e.g., "One of my campaign managers logged in last week and found all our paused creatives from the past 6 years had been reactivated." from https://www.teamblind.com/post/i-told-my-team-to-pause-our-750kmonth-twitter-ads-budget-last-week-4dnbo1Ft ———— Friends have told me other similar stories Are these failures symptomatic of a larger problem, or are they well-isolated parts misbehaving? Can Twitter even experience a failure cascade like Arecibo? Can that be paused/stopped? I am asking this question because I don't know. And I'd like to develop a better mental model to understand what happens next.
- WatchDog 4y agoAlso anecdote, but twitter has historically been pretty unreliable for me. Without a reliable twitter systems status history pre-acquisition, the reports of failures, like the issues with the 2FA system, don't mean a whole lot.
- jakear 4y agoDefinitely this. I regularly see people complaining about ancient bugs as if they were new after the acquisition. Nope, you just are engaged enough to notice them now.
- 4y ago
- bushbaba 4y agoIf there’s no code changes, reliability is greatly increased. Probably 99% of all outages and incidents are from a bad code change. Within most companies you don’t want to pause innovation.
- throwaway892238 4y agoFrom this operations engineer's perspective, there are only 3 main things that bring a site down: new code, disk space, and 'outages'. If you don't push new code, your apps will be pretty stable. If you don't run out of disk space, your apps will keep running. And if your network/power/etc doesn't mysteriously disappear, your apps will keep running. And running, and running, and running. The biggest thing that brings down a site is changes. Typically code changes, but also schema/data changes, infra/network/config changes, etc. As long as nothing changes, and you don't run out of disk space (from logs for example), things stay working pretty much just fine. The trick is to design it to be as immutable and simple as possible. There are other things that can bring a site down, like security issues, or bugs triggered by unusual states, too much traffic, etc. But generally speaking those things are rare and don't bring down an entire site. The last thing off the top of my head that will absolutely bring a site down over time, is expired certs. If, for any reason at all, a cert fails to be regenerated (say, your etcd certs, or some weird one-off tool underpinning everything that somebody has to remember to regen every 360 days), they will expire, and it will be a very fun day at the office. Over a long enough period of time, your web server's TLS version will be obsoleted in new browser versions, and nobody will be able to load it.
- illiac786 4y agoI think that without code push they won’t be able to maintain compatibility. With updated APIs from third parties, new hardware, new encryption requirements from clients or browsers etc. It’s a slow descent into chaos indeed.
- ranguna 4y agoAnd vulnerabilities.
- lr4444lr 4y agoYou didn't mention data scale. Just because the disks have room, doesn't mean the data access patterns in perfectly stable code will perform well at continual multiples if old data isn't somehow moved to colder storage.
- marstall 4y ago
- deleted 4y ago[deleted]
- zxcvbn4038 4y agoNow that Elon has cleaned house, when is Twitter hiring again? It might be a good opportunity to get in on the new ground floor.
- UberFly 4y agoThe new HARDCORE ground floor.
- HyperSane 4y ago
- wruza 4y agoMaybe it’s worth the money for guys who are okay with that, otherwise who would work for him?
- blisterpeanuts 4y agoIs this based on personal experience, or are you going by some well known publicly available stories about Musk? I've heard pretty much every possible opinion about the guy, but the ones that impress me the most are that he loves to go deep into technical topics with people who know their stuff, regardless of their credentials. That, to me, is a sign of a man with deep humility and desire to learn. At the same time, he's not considered a very diplomatic person and evidently is rather hard to get along with both in his personal life and professionally. Yet, his accomplishments are substantial. So I would go with maverick, different thinker, fk your feelings, kind of guy who not only isn't afraid to break a few eggs to make an omelette, but probably isn't even aware he's breaking the eggs :)
- HyperSane 4y ago"Is this based on personal experience, " It is based on the fact that be publicly fired very competent people on Twitter for contradicting or criticizing him. https://timesofindia.indiatimes.com/gadgets-news/this-is-what-elon-musk-has-to-say-to-people-twitter-fired-for-criticising-him/articleshow/95554777.cms https://timesofindia.indiatimes.com/gadgets-news/this-is-wha... https://www.businessinsider.com/tesla-elon-musk-ruthlessly-fired-anyone-who-disagreed-spacex-report-2021-8 https://www.businessinsider.com/tesla-elon-musk-ruthlessly-f... He also publicly mocked all the people he fired https://www.ndtv.com/world-news/elon-musks-dig-at-sacked-employees-i-apologise-for-firing-geniuses-3523990 https://www.ndtv.com/world-news/elon-musks-dig-at-sacked-emp...
- dboreham 4y ago...because it's only been a week and it takes longer for things to rot badly?
- jdminhbg 4y agoI think this post is aimed at non-technical "RIP Twitter" posters from last week who are surprised it didn't fall apart in 24 hours.
- npn 4y agoThere was a long twitter thread from some one proclaimed 10+ years SRE listing a lot of things that could bring the server down. Lot of them are pretty far-fetched and outdated. Too bad I did not bother to save the link.
- Karrot_Kream 4y agoYou're thinking of https://twitter.com/MosquitoCapital/status/1593541177965678592 https://twitter.com/MosquitoCapital/status/15935411779656785.... Remember, on Twitter your likes and follower counts are completely independent of your actual skill area and level. Most of that list reminded me of 2009-era site operations, nothing like what operating a high-traffic site is like in 2023.
- randyrand 4y agoservers don’t really rot if they don’t rely on 3rd party cloud software. Linux doesn’t upgrade itself. the web is very standards based. Hacker news has pretty much been running for decades on the same software hardware stack
- rubiquity 4y agoGreat post. Thanks for the insight on how mature Twitter’s Cache infrastructure tooling is. Having done similar work at a different place where you’re exposed to the realities of racks and racks of machines I could appreciate the evolution of the stack and knocking off problems one by one. I suspected that the hopes and fears of Twitter’s quick demise were overstated and shared[0] some of those thoughts last week, on Twitter of course. Also hi from another infrastructure person in SD! 0 - https://twitter.com/bitsandhops/status/1593637241527578625 https://twitter.com/bitsandhops/status/1593637241527578625
- mradek 4y agoTangent but I do hope that Musks trim down results in orgs that have less “executives” and a layered cake of a org structure, and more autonomous small teams that execute on shared overarching initiatives. I really don’t understand why so many tech companies have like 8 layers of engineering levels. If the argument is that you need more money so more levels, just have a bigger band. Don’t chase titles they don’t mean shit. Not to mention the management stacks that seem to just hang out in meetings and take pvt. I haven’t worked at a proper startup but I’ve been on projects where a dozen or so people rebuilt apps used by tens of millions of people in a few months, or launched completely new applications for bigger companies. Now that I work in a tech company (not big tech, but still a multibillion dollar corp) I’ve noticed that since IPO we have added a ton of bureaucracy whereas back in the day we were small teams building completely new and at times complex features. Literally was in a meeting earlier this month where people were patting each other on the back because we added a single attribute to a table. I’m obviously reducing the entire initiative to a small thing but it kind of explains all we had to do. It’s soul crushing but with the economy the way it is I must deal with it. Hell I’m down to show up next Monday and work for Elon if he wants a go getter. At least I’ll get to DO stuff. Or any other startup in SF if they are hiring.
- shitloadofbooks 4y agoA lot of those upper layers and the bureaucracy which is seen as "bloat by the bottom layers comes about from compliance. The compliance burden grows as the business grows (e.g PCI, insurance and government requirements kick in at certain thresholds). Sure engineering adding a new widget to the site might increase profits by 12%, but all that bureaucracy can prevent the company losing its payment provider or breaching a government regulation which might cause the company to close overnight. So if the stakes are +12% vs -100%, who is actually doing the most important work? I don't disagree with your desire to work somewhere lean and task-focused, but I think it's almost impossible for that to happen anywhere but small(er) workplaces.
- chrismarlow9 4y agoIf 10 engineers can build 5 features in a week, then why can't 50 build 25? We easily understand why as engineers on HN, and even managers and investors understand why. But that wont stop them from trying to solve that problem. And some of them do! All of these big FANG companies are as big as they are because at some point they figured out how to solve this problem. But just like time and growth break architectures and software, they also break processes, and they eventually encounter a level where they can't solve it anymore. Then they stagnate, then they either die or solve the new problems. They have to grow to keep investors, and to grow they have to keep investors. That's why it doesn't stop at 12 dudes building an app. But sometimes it does just fine. It's really just a matter of who's investing and what returns they're looking for.
- AtlasBarfed 4y agoTwitter is a lot of drama, but ultimately this is a common private equity play: 1) take private 2) fire/layoff until things break, patch up / rehire with cheaper labor. Repeat as necessary 3) spit shine/repackage what's left with a theoretically more appealing balance sheet. 4) resell to another sucker...uh, buyer, or take public again. Fundamentally Musk bought 44 billion for 5 billion of annual revenue, and presumably 5 billion in costs. Unlikely to add revenue in twitter's model, he's cutting costs. Honestly, at this point, is there revolutionary technology that is needed to keep the lights on at twitter? Do you need graphQL gods and SREs that would ace any Amazon raising the bar and master of silicon valley interviewing? Nah. He'll honestly probably outsource a ton of the upkeep. It's ugly, but the Twitter board and shareholders took their money and ran, and abandoned the product and the workforce. They could have backed out and let Elon Musk off the hook of his dumb contract with them, but they just wanted the money, and sold to someone that just wants to get his money out of it too.
- dehrmann 4y agoThere are a few key differences. PE tries not to massively overpay, it'll put more effort into targeted layoffs, and it'll maintain enough of a "business as usual" posture to not scare away advertisers. Edit: they also put in a CEO whose full-time job is being CEO of that company. But yes, this is very bad execution of the PE playbook.
- midoridensha 4y agoElon thinks he's brilliant at everything he tries his hand at, so he thought he'd jump into the private-equity arena and show everyone how brilliant he is.
- npn 4y agoChinese developers are famous for making apps serving 1.4 billion people, with at most 1/3 salary of US, and the willingness to do 996 a whole year around. I would be surprised that no company in the west take advanced of that.
- 4y ago
- treme 4y agoturns out CEO with exceptional track record & success in 2 different industries is some what competent. I'm pretty surprised to see the occasional redditesque "Elon is an idiot" rhetoric on hn tho.
- bearmode 4y agoIn spite of him, not because of him.
- deleted 4y ago[deleted]
- deleted 4y ago[deleted]
- ggm 4y agoThe article is good, and informative but a little odd in one respect. It uses air-quotes to introduce concepts like "rack" and "shelf" in a DC, but does O(logn) notation mid-flow. If you don't know what a rack is, how are you meant to know what the Order of scaling function means? Thats a highly computer sciences specific notation, and if you grok O(n) you know what a rack, a host, a DC is.
- mtejo 4y agoHah fair point, I was trying to write for a more general audience. Part of blogging for me is to improve my written communication skills. Appreciate this!
- ggm 4y agoSomething like "scales linearly" or "scales exponentially" probably covers it, for whatever case it was (I forget) really good article btw. really enjoyed reading it. might pay to flag when systems you talk about are twitter-internal or are open-source. People love that kind of thing. "wow: twitter uses the gnu C compiler" type thing.
- housecarpenter 4y agoI get your point that both of these are specialized notations/terminologies, but it's entirely possible for somebody to understand algorithmic complexity without knowing what "a rack, a host, a DC" is, there's no inherent connection between the two. As somebody whose interests tend to be more theoretical than practical, myself prior to reading the article would be an example.
- ggm 4y agoAs the OP said "I was trying to write for a more general audience."
- clayhacks 4y agoHe says at the end his automation allowed him to automate more. A great positive feedback loop. I see the inverse is also true. Where having no automation makes automating new things harder. Without enough engineers I think they will hit some issue and just not have the bandwidth to fix it in a reasonable amount of time
- mkl95 4y agoI have seen this happen twice at a smaller scale. Turnover is too high quantitatively and qualitatively to survive by hiring new people, so Musk's "solution" will probably be to oursource a lot of stuff to some consultancy firm. Twitter will almost inevitably rot within 5 years. But don't expect anything noticeable overnight. Edit: also the reason Musk wants to shut down all those microservices has nothing to do with money, it's that he doesn't have enough manpower to maintain them. If Twitter is like most non FAANG out there those "microservices" are coupled with each other, so good luck.
- csomar 4y agoDid the OP confirm Elon theory that most of the stuff is not needed? > For four of those years I was the sole SRE for the Cache team. There was a few before me, and the whole team I worked with, where a bunch came and went. But for four years I was the one responsible for automation, reliability and operations in the team. I designed and implemented most of the tools that are keeping it running so I think I’m qualified to talk about it. (There might be only one or two other people) If you only need one person for the caching department (which is, as I understand, is critical as it delivers most of the data); then maybe you need a handful other dozen engineers and there you have a functional Twitter. That or the OP is full of himself. Kinda like Musk?
- mapme 4y agoOne SRE, many SWE. Also have fun asking someone to be permanently oncall with one person on the team. The cache clusters size are also described here for anyone who wants a good technical read over speculation. https://www.usenix.org/system/files/osdi20-yang.pdf https://www.usenix.org/system/files/osdi20-yang.pdf
- csomar 4y agoThe OP claims he did the implementation (so he was the software engineer too?): > I designed and implemented most of the tools that are keeping it running so I think I’m qualified to talk about it.
- cmckn 4y agoI read this as they built the “tools” (automation, orchestration, monitoring, etc.) for this system, not the system itself; which aligns with the common definition of SRE.
- mapme 4y agoSRE is a mix of both. The expectation is you are able to write and understand any code the team is responsible for.
- okdood64 4y ago
- harriet1 4y ago[dead]
- npn 4y agoLet don't kid ourselves here, even if Twitter crashes and needs a few weeks or even a few months to recover it doesn't matter. People who leaved will comeback again, after a year everything will be like nothing ever happened. And even if Twitter lost 50% of it users and revenues, it is still more profitable than have to pay the salary of 7000 people.
- kaba0 4y agoMore profitable than what? It was and is losing money, plus now it also has some serious debt on top. With staff that actually know their shit (and a sane CEO), it may be possible to turn it profitable, but I don’t think it will happen as things currently stand.
- npn 4y ago7000 people x $300k = $2.1B And that's salary alone. The operation cost to serve all those people is a lot more. Twitter revenue last year was around $5B.
- uwagar 4y agoautomation is indeed key. IRC networks run with no obvious employees and salarys and been running fine since the dawn of the internet. i see Twitter as a centralised, walled IRC. the # was pinched from IRC if u remember.
- kaba0 4y agoAn IRC network server can serve what, a few hundred thousands of concurrent users’ text messages on a single PC. Twitter has around 200 million daily user, peaking up to a billion, running on their own server farm. They are not remotely comparable, especially if we note that read-write dbs are notoriously hard to scale.
- uwagar 4y agothank you, i was trying to be provocative to learn more. otoh, to get an automation going is like climbing mount everest the first time? very hard but makes it easier to scale subsequent attempts?
- mesozoic 4y agoAs an SRE I know when people aren't working things rarely break (not ever but much more rare)
- hot_gril 4y agoYou can usually do more with less if the remaining few are up to the challenge. It will come at some cost, like having to "move fast and break things" more. There can even be a positive side if the previous situation had a lot of friction due to "too many cooks," as I've experienced on several teams. What's more worrisome is the alleged 80hr work weeks. Yes, two SWEs on a team aren't as good as one SWE working double hours, since SWE-time is known not to scale that way. But burning people out, aside from being unethical, is risky when nobody is ready to fill the spot. Musk has a point, but it feels more like he's doing this out of financial desperation. I'll bet it starts going down a bit more often, nothing too severe. Bigger issue might be inability to roll out new features, especially if they dig a horrible tech debt hole. Maybe someone has more details, I haven't kept up, but it seems like the $8 verification rollout got botched because they cut corners on actually verifying the subscribers.
- justahuman74 4y ago> cut corners on actually verifying the subscribers Did they actually verify anything other than the ability to pay $8? It seems wild to me that they thought it would work out just fine
- blisterpeanuts 4y agoFeels like one of those "ideas that sounded good in the conference room" that turns out to have unintended consequences. Hopefully they can fix this and have an effective way to verify identity... or else just give it up and go back to the old, creaky, somewhat effective method since people are used to it.
- hot_gril 4y agoIt sounds like an ok idea that someone didn't execute properly, unless the idea really was to give the checkmark without real validation.
- resonious 4y ago> This left a lot wondering what exactly was going on with all those engineers and made it seem like it was all just bloat. I was partly expecting the rest of the article to explain to me why exactly it wasn't just bloat. But it goes on talking about this 1~3-person cache SRE team that built solid infra automation that's really resilient to both hardware and software failures. If anything, the article might actually persuade me that it was all bloat.
- H8crilA 4y agoThis is a PR article that tries to push the idea that Twitter is OK (to work at, and maybe also to buy ad space from). Damage control.
- Lutger 4y agoI read it as the exact opposite: the reason Twitter is still ok* is not because all these people were just browsing reddit at work. You can't just gut Twitter to run on only a couple of hundred people and still expect the same results in the longterm. Twitter was not a leftist commie welfare company as Musk and its fans want it to be. It was actually the fine work SRE (amongst others) put into it that makes it still tick along as it does...for now. * actually some things are already breaking, but it will take some time for the real damage to surface on a technical level
- pas 4y agoIt could have been both! :) And seriously, the gaming industry is infamous for its "year round crunch mode". It can be done. Many sectors are "hard work" (ie. public sector, fundamentally non-tech gigacorps with no real in-house IT competence, etc.) Twitter was an innovator. I remember getting into Scala party because of some open source code library from them back then, I don't know, probably around Scala 2.8. That was absolutely unnecessary for their core business. All of this can be done in bad C++, or plain old Java 7, or whatever, for low cost, built in Bangalore. Now that would of course likely not withstand this kind of change storm that Elon caused. But that's a different story. (Also what Elon probably doesn't understand on a technical level (but likely guessed from a business aspect), is how flexible Twitter is. In this regard he hit arrived at the right time. Twitter is fundamentally the same as it was, now it's time to innovate with it, and the architecture he found is exactly the right one for that.)
- textbookrental 4y agoI guess I am just a classic HN reader, so I wonder why Twitter needed so many people. Also, I find it pretty interesting how this is playing out so far. I'm starting to think that Mr. Enron Musk is going to pull this off.
- cheri9 4y ago[dead]
- tyiz 4y agoShe mentioned you: https://youtu.be/g-voQsFY6SE https://youtu.be/g-voQsFY6SE She was assistant to the gender health support board.
- new2yc 4y agoSomeone please post this as a win for open source, nsidb. I don't know how https://lichess.org/blog/Y3u1mRAAACIApBVn/settlement-reached-in-stockfish-v-chessbase https://lichess.org/blog/Y3u1mRAAACIApBVn/settlement-reached...
- bawolff 4y agoI think it would be more surprising if it did go down immediately. Presumably in the turmoil nobody is deploying new code. Sure shit happens, but i imagine that twitter is mature enough that they aren't having weekly critical incidents, especially during times where nothing is changing. I still think twitter is going to have some disaster, but give it a few weeks.
- aryehof 4y agoThere will always be those that think that Twitter can be written by a single programmer in a day or so. How hard can it be for such a simple idea after all, given that there are hundred of tutorials producing a version in just about every language just a search away. Keep it running? One part-time should surely do it?
- yalogin 4y agoThe reason why Twitter hasn’t crashed is it’s well written and well orchestrated. Once a bug comes in and crashes something that is when the chaos starts. It’s almost a guarantee that the current crop do not know how to fix the bug. It will be interesting to see how they handle that.
- trollied 4y ago> It’s almost a guarantee that the current crop do not know how to fix the bug Seeing lots of comments like this. Why do you feel it’s ok to say that, when you don’t actually work there or have intimate knowledge of who is currently working there? The pure speculation and nonsense surrounding Twitter at the moment is plain awful.
- yalogin 4y agoIt’s just statistical probability. The number of employees left is small and the amount of code is huge. It’s not a huge leap to say that a bug is unlikely to be in the area of expertise of the folks that are left. I would say it’s the probability is greater than 50%. You should notice the probability is not proportional to the employees still remaining. That is because I am guessing some good folks are still there.
- b99andla 4y agoWhy would twitter go down due to staff layoffs? The stuff I build would keep running for years without my involvement. Do you mean that other systems are hand cranked?
- throwaway5371 4y agois twitter going to be profitable if it can run with 10% of the workforce?
- pie314isi 4y agoSo, this? https://www.youtube.com/watch?v=3ztFaYvgDAg https://www.youtube.com/watch?v=3ztFaYvgDAg
- theknocker 4y ago[dead]
- eshork 4y agoI read ~60% of your post, and noticed it repeats, so I stopped reading. Not sure if you are aware
- dsq 4y agoIts possible to teach a horse to eat less and less each day. They just have this unfortunate tendency to die just after you succeed in weaning them off food altogether. A horse, of course, is not same as a reduction in force. Engineers don't eat hay, neither does a horse have a 401(k).
- octodog 4y agoThanks for the write-up OP! Appreciate it.
- rvba 4y ago> When I joined the team the first project I had was to swap old machines that were being retired for new machines. There were no tools or automation to do this, I was given a spreadsheet with server names. Writing a program to store a list of servers to be swapped instead of keeping them in a spreadsheet sounds a bit like buying a brewery when you want to drink 1 beer. Program used by a team of one sounds like over-engineering.
- ojosilva 4y agoThis spreadsheet was, from the OPs narrative, the master-list used by the whole operations team. An operations team that is now reduced to "ashes" - therefore not a bad idea at all to have your server inventory centralized.
- karol 4y agoI don't understand why would a mature platform just fall down without its engineering team. There are surely a lot of small fires but 80/20 rule with regards to triaging should keep the lights on when properly applied. Speaking as an experienced single tech founder.
- skc 4y agoTwitter pretty much passed it's first real test with flying colors since Elon took over. The world cup is on and the site didn't collapse. That is a huge win and while I don't believe Twitter will ever be profitable, I think Elon will be feeling rather smug right now with his skeleton staff.
- eBombzor 4y agoWas this a thing people actually believed? Other than the typical room temperature iq Twitter user.
- PeterStuer 4y agoIn every software company of over 50 employees I have seen up close it would be fairly to identify 50% of people that had minimal to negative contribution. That said, it takes some time to get to know the real workings of a company, knowledge required to select the right 50%. That said, afaict the current mo is keep the lights on for the current offering while a new team builds a new product for that enormous user base that offers something much more profitable.
- andy_ppp 4y agoThere seems to be a load of people in this thread making statements on both sides that a) the article doesn’t prove there were too many people at twitter b) the article implies there were too many people at Twitter c) Twitter is a simple app why so complicated etc. I would say this; Twitter at this scale is extremely complicated. We don’t have enough information to know what all these roles were doing and how many were truly superfluous. The article gives an interesting insight into something extremely complicated- it might be a slice repeated in a lot of areas. I’m of the opinion that you can do a lot more with less but there’s no need to send emails to everyone in the organisation being an arsehole and essentially accusing them of being lazy, then rehiring people who never worked there just for the meme. Anyway Musk will blow up some rockets, put enormous pressure on people to fix them, and come up smelling of roses again even if he treats a load of people like crap along the way. His fans will say he’s a genius and his enemies will say he’s a dickhead. And so it goes.
- Dave3of5 4y agoI don't think it will fail on a technical level. As this article says lots of engineering has gone in to make the thing pretty resilient. I would also say that there are still enough engineers who work there who can figure out what's gone wrong and "turn it on and off again" or w/e makes it splutter back to life. In terms of changes to the Platform ditto. It's not difficult to make these changes that a team of 100's of devs who are not 100% aware of what's there already can't figure it out. I've taken over systems that I knew very little of and were pretty big (not as big as twitter) and I managed for years to make changes without drastically breaking stuff. In any event if they do break stuff they will be able to fix what they have broken. No, the real failure here is the massive debt burden and the fact that there is no way that twitter can ever service that amount of debt. Note that before EM took over it was ticking along with a relatively small loss. If they had cut headcount by maybe 10% they would have been break even easily. There is no way that's possible with $4million of interest per day. They have to radically change the way they monetise the platform to get to that level. I don't think they will ever get there and Musk will sell off at a bargain basement price at some point in the future to pay back the debt.
- andreyk 4y agoYep, agree. Twitter's revenue was primarily from ads, and now I'd bet their ads revenue has dropped a huge amount. Given how Musk is behaving now (reinstating Trump etc.), and losing many pivotal sales people, and the firing of a ton of people in charge of dealing with hate speech and such, it seems unlikely advertisers will return now.
- _3u10 4y agoI don’t think given their ad sales that the sales people were that pivotal.
- hnbad 4y agoAs I understand it, most of the ad slots were sold at a yearly sales event. This time most advertisers were cautious because the company couldn't respond to questions of how Elon Musk's takeover would affect the company (and specifically the type of content alongside which the ads would appear) as this was happening before he took over but after the sale was practically a done deal. This left them with only a fraction of the ad slots "pre-sold" compared to a normal year. Additionally it seems a lot of advertisers have cautioned their ad departments to halt spending on Twitter ads for now because they're still unclear about wtf Elon Musk is up to (not to mention the mass impersonation fiasco that allegedly resulted in share prices of several companies being affected negatively). It's not just sales people, Twitter seems to have gotten rid of most of its external communications people to the point where journalists can't even find anyone to ask for statements other than Elon Musk himself (which may be intentional given that he likes to present himself as the face of his companies). But if Twitter's ad sales really did rely so much on these sales events, I'd wager that the sales people largely existed to communicate directly with long-standing big-spend advertising partners. It's hard to overstate how much these types of deals hinge on "social glue" rather than cold hard numbers (which can be interpreted/reframed to be interpreted however you want as long as both sides are happy).
- fatneckbeardz 4y agoever notice that the truly competent people can explain things in really simple language?
- Marazan 4y agoOnce thing I've seen that I think is overlooked in the whole "is Twitter's tech stack inefficient" chat is the nature of the service. A lot of the "Twitter could be run off my laptop" style comments seem to come from people who run, effectively, Read-Only services. They might serve data at thousands of queries per second but the data itself is _slow_. It is video files or music streams or other data that updates infrequently or, if there is dynamic data involved, it comes from a very small number of fixed sources. Twitter deals with thousands of posts per second that are subject to huge fluctuations in the density _and_ those posts have to be disseminated to the millions of users. Twitter is a two sided problem.
- jiggawatts 4y agoI've heard this elsewhere, and when Twitter was originally founded in 2006 it wouldn't have been a particularly hard problem. Twitter processes roughly 10K tweets per second. Even if you bloat out the text quite a lot to account for encoding overheads, metadata, etc, etc... and assume that each one is 10KB, then this is just 1 Gbps. A single NIC on an old server. Okay, I get it, Twitter needs a lot of data too. Lists of users, etc... Twitter has 450 million monthly active users, which sounds like a lot, but even if there's 1MB of profile data per user such as who they are following, that's just 400 TB. That's... not that much these days. A large-but-not-enormous database cluster. Sure, there's "historical" data, but that can be compressed and put on cheap cold storage, like S3 or whatever. Give me a few million annually as an opex budget, a small team of decent developers, and I can guarantee you I could whip up a cloud-hosted service that can process tweets at Twitter's scale. Obviously, what I can't replicate is the much larger set of tools and systems behind the scenes that are used for moderation, analytics, ad sales, etc...
- Marazan 4y agoThat's looking at Twitter from the other end, as a Write-Only service with non-realtime requirements on the read layer. Twitter is both sides. Tweets are replicated out in milliseconds. Twitter search is lightning fast, comprehensiveband powerful. o Old tweets are not just glaciered into S3!
- Nokinside 4y agoMore than 50% of large distributed system downtime is caused by configuration errors, failed software updates, or network attacks. With configuration error, or failed software update everything can go down at once, or there are cascades of restarts that kill the system.
- reacharavindh 4y agoHas anybody suggested the idea of rewriting twitter in Rust yet to Elon? Memory safe, needing fewer servers(hence less $), and the story sells itself to someone like Elon who jumps on such impulses... ^ is a snark. I'm not suggesting that they should..
- harriet1 4y ago[dead]
- jawadch93 4y ago[dead]
- YokoZar 4y ago> drove some large cost savings projects Having worked as SRE on efficiency projects for BigCo, it is not at all uncommon to recover more than your lifetime salary in company savings with only a few months work or even less. The scale of things is so immense that even a slightly better handling of things can lead to outsize returns. Laying off someone like that, rather than putting them on full-time efficiency work, is an obvious waste.
- notyourday 4y agoNews at 11: people who waste 90% of their work time on HN while collecting a-la FAANG salaries because they are used to bullshitting their management and getting away with it are terrified of the management that can look at the code.
- HeavyStorm 4y agoObjectively, the article proves that there was bloat since all things are automated.
- nigamanth 4y agoTwitter won't go down that easily. Besides, Elon Musk hasn't bought it for no reason. I'm sure that he has his plans with Twitter to implement new features and whatnot to restore it. Till now, he's fired a lot of engineers probably because: a) he doesn't need so many. b) he thinks he can do the job himself.
- dekhn 4y agoThe article didn't address what happens when Mesos goes down. How much of the team running the underlying cluster infrastructure remains?
- Kye 4y ago>> "How the cache’s keep running" It's a little thing, but it's the little things that cascade into big things that require teams of people to fix. Here, we see this person who's quite proud of his automations making the simple, but harmless error of using a possessive instead of a plural. Humans are good at reading through this sort of thing. Computers are not. Somewhere deep in all his scripts is a misplaced ', `, or ’ waiting to break something.
- deleted 4y ago[deleted]
- randomsearch 4y agoIs it essential for Twitter to run their own datacentres?
- shagymoe 4y agoWhen I was learning programming, I "wrote" an internal app where a supervisor input some information in a form, it connected to an oracle database and then printed out an outbound manifest for a truck full of car parts. I put "wrote" in quotes because what I had actually done was install Apache and PHP Nuke on a Windows NT server and then modified an existing form page to do what I wanted. I wrote that application in 2002 and never had a problem with it. I never restarted Apache or the server or did any maintenance. I didn't even upgrade Windows NT if I'm being honest. Windows NT became unsupported in something like 2006 and I left the company in 2010. I received a call in 2018 from the last person still working there who knew who I was. It had finally fallen over and they wanted to know if I could help. That was the first time it ever had an issue.
- Giorgi 4y agoIt did not go down because huge companies don't usually go due to reddit woke culture.
- redeyedtreefrog 4y agoSpaceX and Tesla made affordable, reusable rockets, and affordable, high performance electric cars with a long range. Those things have obvious value, but required solving big engineering challenges to make them a reality. Musk doesn't seem to have any clear goal with twitter. There are vague ideas about making it subscription-funded rather than advertising-funded, and about having less censorship, but neither are things the majority of people want. I imagine the site will mostly continue to more-or-less work, despite all the layoffs. They still have thousands of staff. The network effect of Twitter is so big that people will continue to use it even if fail whales become more common. Others suggest that it will soon crash and burn, or that Musk will get bored and sell it for a few billion. Or that Musk is a genius who will make some sort of amazing Twitter 2.0 that does for social media what Tesla did for electric cars. But without any appealing long term vision, and with an owner who bought it to satisfy their ego rather than with any real plan, the reality may be more boring. I imagine it will just languish for many years, with occasional manufactured drama, and occasional downtime, but no real innovation. Maybe to be eventually supplanted by something else in 10 years or so.
- mattbrewsbytes 4y ago> Or that Musk is a genius who will make some sort of amazing Twitter 2.0 Based on texts exposed via the lawsuit about the purchase, I don't think this is the case. I don't think he, or his advisors, understand that the product at twitter (and other social media) is content moderation. You can have a vision of whatever type of content or pricing scheme you want but without solid content moderation you will lose advertisers, gain lawsuits (people were posting movies the other day) and lose users because the "feed" becomes a muddied mess. Users are only really the product when you can moderate their content to have profit via advertisers.
- bdcravens 4y agoI wonder if the Twitter situation may be the event that causes the industry to push back on large technical organization sizes. I wonder how many reading this are starting to feel nervous about their own roles.
- ineedasername 4y agoI wasn't really expecting a large failure quite yet. I'm guessing there are enough people left with secondary or tertiary knowledge of systems to keep things ticking over for now. The question is how long that state of affairs can continue before some combination of worker burnout or deferred maintenance will stretch things too thin. Then, all-hands emergency work to get things running could contribute even more to burnout -> resignations etc. They could stay ahead of things if they hire into areas related to infra maintenance that got hit the hardest, before issues reach the point of cascading failure. Or maybe put everything not critical to keeping the site up & running on hold as remaining devs get a bit of cross training using the limited people w/ institutional knowledge that remain as trainers. For myself, I don't see much value in Twitter, in terms of net social value. It's format seems pretty much designed for only the most surface level discussions, part of what I believe leads to some of its toxicity: it's simply too hard to have conversations complex enough to invite enough discourse for that to tip the productive/toxic ratio a bit more positive. So I've been rooting for Twitter to fail for years. I'm not rooting against Musk: I like his other businesses and at least a portion of their ongoing success is tied to his persona. (At least before the Twitter stuff: His actions there may impact his relationship with major banks that his other businesses will rely on, and Musk & JP Morgan, a bank that wasn't included in the deal & therefore won't be hurt by it, is already on somewhat negative terms with Musk. The twitter deal has added a few more to that list, and other banks have undoubtedly taken note. Some bank will always finance the regular sorts of things any large corporation need for him, but they're all going to be pricing in some additional risk. That's not really a big deal, it's just that I think Musk's persona has previously been a net positive for his companies and now it's lost at least a little bit of that.)
- tiffanyh 4y agoFail Whale. I think people just miss seeing the fail-whale graphic. https://www.techopedia.com/definition/1987/fail-whale https://www.techopedia.com/definition/1987/fail-whale
- ellen364 4y agoLinear extrapolation seems to be a common theme in the comments. People take a product they know and then essentially say “Twitter only has X times as many users, surely they only need X times as many engineers. The other employees were excess!” I haven’t worked on Twitter-scale products. But I do wonder if it’s safe to draw a straight line between services that have middling and large scales. Things don’t necessarily work the same at the extremes.
- raspberry1337 4y agoMusk makes leftists lose their shit and all rational functioning, even on Hacker News; that is my reflection.
- socialismisok 4y agoNot just leftists. His fans are equally out of touch in the other direction.
- AtNightWeCode 4y agoAnybody working in the biz knows about technical debt. At Twitter that will skyrocket now. My firm believe and experience tells me that keeping everything somewhat up to date over time always pays off. The problem is not that things will break. Things break ALL the time. The problem is more about avoiding cascading effects and the time it takes to fix stuff.
- csmpltn 4y agoWhy Twitter didn’t go down, you're asking. Short answer? H1Bs: https://twitter.com/elonmusk/status/1593899029531803649/photo https://twitter.com/elonmusk/status/1593899029531803649/phot...
- SMAAART 4y ago> if you have a cluster of servers where serving 1000 requests a second might cost $1000, you can instead use the cache to store the responses and serve it from that cache server instead. Then you would have a small cluster for $100 and a cheap and large cache cluster of servers maybe for another $100. The numbers are just examples to illustrate the point. Reddit needs to implement this if they want to get to the next level.