8 ms·
To everyone who is angry: calm down. Github’s servers are constantly on fire as their usage increased something like 50x due to LLM sloppers pushing large amoun
by ieie3366 1mo ago
To everyone who is angry: calm down. Github’s servers are constantly on fire as their usage increased something like 50x due to LLM sloppers pushing large amounts of trash code
Unless you have a emergency hotfix (you don’t), go hit the gym or walk outside.
If a work tool going down triggers you enough mentally to start angrily ranting online, it’s a sign you need to chill out and focus more on your health
- cesarb 1mo ago> go hit the gym or walk outside. My mind immediately went to this classic xkcd: https://xkcd.com/303/ https://xkcd.com/303/
- denysvitali 1mo agoStop blaming LLMs: https://damrnelson.github.io/github-historical-uptime/ https://damrnelson.github.io/github-historical-uptime/
- mococa 1mo ago> increased something like 50x due to LLM This should not be an excuse since they probably can use AI to fix it /partially sarcasm
- eddieroger 1mo agoFor the amount of money enterprises are paying to GitHub, there is a reasonable (and contractual) expectation of uptime. I don't think my boss would find it to fun if I bailed work to hit the gym just because GitHub was down.
- deleted 1mo ago[deleted]
- iamcoder18 1mo agoYes, but GitHub enterprise has different uptime (https://us.githubstatus.com/posts/dashboard https://us.githubstatus.com/posts/dashboard) and actually does have an SLA. Edit: it seems GHE is also affected by the outage and isn't much better.
- twistedpair 1mo agoSince upgrading to Cloud Enterprise... why didn't my GH uptime get any better? Guess on-prem GHE server is the only way to do that.
- cyansmoker 1mo agoAs an enterprise user I can tell you that: - we are getting no benefit beyond getting excuses replies to our emails - right or wrong, but we depend heavily on GitHub actions so this has a very real impact on us
- tedivm 1mo agoThis, like all of their dashboards, is complete bullshit. As a GitHub enterprise user i am absolutely affected by the outage today, and have been affected by all of the GitHub Cloud outages. There is no separate enterprise platform unless you buy their self hosted server product.
- eddieroger 1mo agoYeah, I know, and it's down.
- AgentK20 1mo agowhat if I do have an emergency hotfix for a production outage though ;-;
- askonomm 1mo agoThat's an overly optimistic view of things. For a lot of us GitHub is critical infrastructure, which if it goes down loses us and our customers money.
- AlexandrB 1mo agoBoomer opinion but trusting third parties to be critical infrastructure, especially with no SLA in sight, will always end in tears. "The cloud" is very convenient, but its providers will never care about your infrastructure or your customers as much as you will.
- BryanBeshore 1mo agoTruly a boomer opinion. Respect!
- Maxion 1mo agoOlder millennial here and I agree. If you rely on Github you should at least be have your processes such that you can work around it.
- whynotmaybe 1mo agoThat's the reason why my build agent is self-hosted
- raffael_de 1mo ago+1 (as a millenial) ... especially given that setting up a git server for non-OSS company code isn't too much of a challenge really. also, no need to self-denigrate this reasonable opinion in preemptive obedience.
- otterley 1mo agoGitHub is way more than just a git repo host. It manages code reviews, merge (pull) requests, and has an entire CI/CD workflow engine in it. Replicating all that is a challenge that most orgs are not up to.
- raincole 1mo agoOr, you know, it's perfectly reasonable and natural to feel angry when a service you paid for gets worse over time. > Unless you have a emergency hotfix (you don’t) Oh, we do. Given the sheer number of users, it's almost guaranteed someone is on fire ever time GitHub is down. Statistics is a very charming branch of reality.
- leishman 1mo ago> Github’s servers are constantly on fire as their usage increased something like 50x due to LLM sloppers pushing large amounts of trash code Which means they have a pricing problem. Rate limit non-paying users.
- anoplus 1mo agoRight. It is not an excuse and they were notorious for outages before LLM era. Also, LLM can serve Github engineers as well, so we are all playing in the same field.
- cluckindan 1mo agoThey already do
- rzzzt 1mo agoI can't check the commit history of some OSS projects without hitting a rate limit if I'm not signed in. My request rate is one request per (arbitrary time interval) at that point.
- xnorswap 1mo agoJust take away free Actions and it would surely solve a lot of their scale issues. It's absurd that I have dozens of repos, many with GH Actions that run CI, test and then package and push to prod/package managers, and haven't paid GH anything.
- chrisjj 1mo agoNo. It means they have a capacity problem.
- s_dev 1mo agoWhat if we're not angry and just perplexed? Do we calm down further or can we still ask questions?
- ethagnawl 1mo agoNo questions. Only bench press.
- Topfi 1mo agoNo one ever has to commit anything time sensitive.
- sausagefeet 1mo agoActually, I do have an important fix for a deal we are trying to close that does need to go out. And I am paying GitHub to host this. Once or twice, I can see, but GitHub's SLA is getting worse than just hosting it myself, and that's the entire reason I pay GitHub.
- world2vec 1mo agoGeez, sorry that the paid service my company paid for is down and I can't do my work.
- jjice 1mo ago> Github’s servers are constantly on fire as their usage increased something like 50x due to LLM sloppers pushing large amounts of trash code I sympathized with them when they said this a handful of months ago, but then I saw this [0] page that shows how it's been shot for years prior (which tracks with my memory). I feel for the GH engineers that have to deal with this, especially the SREs. I also don't hate the downtime right now, as I'll make a cup of coffee and do something else. I will say though, I did have a hotfix a week or so ago during the Actions outage, which really was a pain. You're right that getting angry and ranting isn't the right reaction here, but I do no give them the LLM load excuse. I don't give them an out for having awful uptime during work hours for a product we pay quite a bit for as an org. [0] https://damrnelson.github.io/github-historical-uptime/ https://damrnelson.github.io/github-historical-uptime/
- debarshri 1mo agoIt looks like right after Microsoft acquisition this started going off. This is the outcome of violating single responsibility principle in business.
- Topfi 1mo agoGithub being under the CoreAI division probably also doesn't help the engineers prioritize addressing infrastructure issues and makes using LLM load as an excuse feel self-inflicted. Akin to feeling sorry when a pyromaniacs house burns down...
- twistedpair 1mo agoThey _are_ the LLM coding agent vendor, and are _owned_ by MS, the ~majority~ biggest shareholder of OpenAI. How can you NOT consider that 20x+ scaling in your capacity roadmap projections, where you are trying to get everyone to use these agents as part of your core OKRs? Or, maybe your #1 IT priority was moving everything to Azure instead ;)
- rafram 1mo ago> the majority owner of OpenAI Largest shareholder (27%), not majority.
- javier2 1mo agoWhile I agree in principle, Github with their CI is very critical to many parties.
- sarjann 1mo agoThis has to be rage bait, this is a critical piece of infrastructure for many people. Stuff going down can lead to deployments failing and as you mentioned in many cases emergency hotfix's. The idea that we should be fine with this unreliability is just amazing. It's not a mental health issue to have problems when important infra fails.
- john_strinlai 1mo ago>The idea that we should be fine with this unreliability is just amazing. the comment doesnt say you should be "fine" with the unreliability. they are saying people shouldnt get so emotionally worked up over it. which, while i wouldn't phrase it in the way the parent did, i agree with the direction of their point.
- sarjann 1mo agoExcept he tries justifying it. "Github’s servers are constantly on fire as their usage increased something like 50x due to LLM sloppers pushing large amounts of trash code" "Unless you have a emergency hotfix (you don’t), go hit the gym or walk outside." with assumptions like above. If it was, "hey they f*ed up but there's no point having an overly emotional reaction" that would be fine. But he seems to be justifying this. Especially with Github's record up to now of unreliability I think it's completely fair to be annoyed.
- zahlman 1mo agoI don't even see a whole lot of emotional worked-up-ness, though. Just continuing disappointment.
- bhouston 1mo ago> To everyone who is angry: calm down. Github’s servers are constantly on fire Is this rage bait? Isn't Google, Amazon, and all the other services in the world similarly impacted by LLM's? Isn't Claude, OpenAI, etc.? Why can they handle the load but not Github?
- dimgl 1mo agoWhat if you, you know, have a job?
- phtrivier 1mo agoI understand the need to level-head the discussion and calming everyone. However, your "(you don't)" comment is not going to calm down all the people, who, you know, DO have a hotfix to push now, and DO have an angry customer that could not care less for which part of our infrastructure is breaking _their_ workflow. The only things would calm everyone down is guarantee that Microsoft would be paying for _our_ SLA breach compensation. But they don't. And I don't think anyone is paying their GH bill with a prorata of the number of time the platform was actually available. (I'm also aware that the wording of the contract probably clearly says that you should not use GitHub for anything critical, that Microsoft is only a small startup in their garage, that you can't credibly expect 90% uptime anyway, and that it's all the fault of LLM slop ! Bad LLM slop ! Also, please buy our LLMs to generate more slop, please.)
- fn-mote 1mo agoIt's past time to move your critical infrastructure off Github then, no?
- phtrivier 1mo agoIt's not exactly done in a pinch, but yeah, I guess someone must be working on disrupting CIs by the audacious value proposition of "we'll work when you need to".
- maccard 29d agoThat effectively means “stop using GitHub” though. Which is definitely worth considering but also kind of insane to think as the best option.
- c2403 1mo agoIn this case, perhaps they shouldn't allow AI slop or promote usage of AI agents...
- madeforhnyo 1mo ago> due to LLM sloppers pushing large amounts of trash code Which they encouraged by pushing Copilot down everyone's throat
- slowin 1mo agoWell, they’re a critical piece of infrastructure with terrible stability. I’m in the process of migrating us off GitHub now. I’m not a Meta fan but it’s interesting that they manage to keep their systems up with an order of magnitude more traffic. GitHub’s uptime is inexcusable.
- AustinDev 1mo agoAlternatively, don't use GH for critical infrastructure they are no longer reliable, simple as.
- TZubiri 1mo agois the infrastructure of the free tier shared with that of paid users? That might be the issue. Otherwise if usage of paid users scales, then it would be fine.
- john_strinlai 1mo ago>is the infrastructure of the free tier shared with that of paid users? github enterprise is fully operational at the moment
- nope1000 1mo agoAt least enterprise EU is partially affected https://eu.githubstatus.com/posts/dashboard https://eu.githubstatus.com/posts/dashboard We cannot pull any actions images (or how it is called) for example.
- neo_doom 1mo agoAgreed. This impatience culture is so toxic. Been down less than 30 mins.
- anonymars 1mo agoThat's cool, I mean if you'd gotten here earlier you could have said "been down less than 5 minutes" So with the "elapsed time" out of the way (now over an hour, incidentally), how about "how long will it be down?"
- neo_doom 1mo agoEnded up being down for about 4 hours. Hope you were able to get out and touch grass.
- anonymars 1mo agoFor some people losing half a day of work is kind of important. Hope you were able to get out and find some empathy https://news.ycombinator.com/item?id=49331481 https://news.ycombinator.com/item?id=49331481
- cb33 1mo ago2 hours later - still down. Is this "impatience culture" or incompetence culture at this point?
- anonymousab 1mo agoAllowing and accepting the LLM load is a willful choice they made and are making at the expense of their users, including their paying enterprise users. They could easily tighten things up in that regard, and make a choice that is right for their main users at the expense of The MS corpo mandate/mission. It is a choice to do otherwise.
- pamcake 1mo agoAllowing and accepting? More like encouraging and instigating.
- constant_flux 1mo agoGH shouldn't have promoted LLM usage the way they have if they didn't have the infrastructure to support it. Regardless of how you feel about the code quality (which you have zero evidence of), if GH has made commitments to supporting broad LLM usage, they need to back that up with the proper hardware and without whatever fragile SDLC processes they have.
- pamcake 1mo ago> GH shouldn't have promoted LLM usage the way they have No qualifiers necessary. But try arguing with a lawnmower...
- stetrain 1mo ago> as their usage increased something like 50x due to LLM sloppers pushing large amounts of trash code Github and their parent company are active participants in pushing LLM-driven coding.
- DharmaPolice 1mo agoThis might be naïve but wouldn't the appropriate response be to reduce access/rate limit free/new accounts in order for service to be maintained for everyone else?
- nerdix 1mo agoWouldn't be surprised if we lost free private repos because of the slopocalypse. Though most don't seem to be shy about sharing their slop with the world so I'm not sure if that will actually increase stability.
- deleted 1mo ago[deleted]
- sharts 1mo agoIt has nothing to do with load.
- otterley 1mo agoThe CTO said otherwise: https://github.blog/news-insights/company-news/addressing-githubs-recent-availability-issues-2/ https://github.blog/news-insights/company-news/addressing-gi...
- karissa 1mo agohttps://damrnelson.github.io/github-historical-uptime/ https://damrnelson.github.io/github-historical-uptime/
- otterley 1mo agoFirst, that source is unreliable, particularly for before the acquisition. (We don’t know whether the data is accurate, it’s based on GitHub’s own self reporting.) Secondly, the scale is WAY off and makes it look far worse than it is. It makes it look like 99.5% availability is practically zero availability. Finally, GitHub’s availability is not a binary all-or-nothing proposition. They report incidents on a granular level, for example, webhook firing can be impaired while Git hosting may be working fine.
- maccard 29d ago0.5% downtime is an hour a week. If that happens for 10 minutes a day at 11pm PT it doesn’t matter, if it happens for half a working day once a month, it’s a disaster.
- OtomotO 1mo agoYou know, I am not employed. I have little time in my day to spend with my daughter. But I also have responsibilities. And one client has decided to bet on github. It's the only client I've ever worked with who has their code on github. EVERY other client has hosted their own gitforge or used bitbucket. So now I am not angry because some critical piece of US-american infrastructure is down all the time. I am angry because instead of spending time with my daughter I have to work on this later, because there are due dates and "Well, fucking GitHub was down" ain't gonna cut it.
- malfist 1mo agoWhy do you get to tell me how I should feel?
- kilroy123 1mo agoAs a _paying_ customer, I disagree.
- semiquaver 1mo agoWhy would you take what they say at face value? I believe that this is basically a lie covering up deeper rot.
- catigula 1mo agoFYI this is an argument to not use Github, not an argument to use it.
- _joel 1mo agoIt's every other day now. We pay for a service, we expect service.
- bradhe 1mo ago> due to LLM sloppers pushing large amounts of trash code hey bud your bias is showing
- robofanatic 1mo ago> constantly on fire as their usage increased something like 50x due to LLM sloppers pushing large amounts of trash code I find it hard to believe that github APIs don't have rate limits to handle traffic spikes.
- thejosh 1mo agoit's been trash for years, their status page just doesn't reflect that.
- deleted 1mo ago[deleted]
- armSixtyFour 1mo agoHard disagree, people are paying GitHub a lot of money. If they are facing an influx of commits from bots maybe they need to start metering commits from those accounts and charge them money. This is a self imposed problem.
- bmitc 1mo agoIf we all go to the gym every time GitHub is down, we're all gonna get ripped. GitHub was down all day just last week. They get paid millions of dollars by my company. They need to get it together.
- bluerooibos 1mo ago> calm down If people and companies are paying for critical infra they expect it to be available - it's not MY problem that their servers are on fire - last time I checked, having too much demand was a good problem to have and they've had over a year to fix it and come up with a plan. I'm so glad I migrated to GitLab months ago.