7 ms·
GitHub is once again down
- olivia-banks 6mo agoAt least it happened after I did my work for the day... jfc!
- jrm4 6mo agoDo your part; remind people that Github is not git. Git is decentralizable and people should know this.
- MattIPv4 6mo agoHitting 500s when trying to push branches and create PRs.
- belter 6mo ago[dead]
- inaros 6mo agoEvery day more Microsofty...they should rename to "Your Repository Needs To Restart To Apply Updates"
- amarant 6mo agoLol, someone should make a pre -commit hook that reboots your computer with a message like this!
- corvad 6mo agoJust wait until github comes up with an outage tuesday.
- bartread 6mo ago"It looks like you're trying to develop some software. Would you like help? - Get help with developing the software - Just develop the software without help [ ] Don't show me this tip again"
- Waterluvian 6mo agoIt's now safe to turn off your expectations.
- kenhwang 6mo agoI wonder what the average career tenure of the userbase here is now, because Github was slow and flaky well before Microsoft got involved. Maybe it wasn't as noticeable when Github had less features, but our CI runners and other automation using the API a decade ago always had weekly issues caused by Github being down/degraded.
- jmtulloss 6mo agoThe best stretch Github ever had was post-acquisition when Nat Friedman as CEO.
- morkalork 6mo agoWould you like to setups repository backups with OneDrive?
- rvz 6mo agoGitHub goes down at least once a week as I said before. [0] thanks to Copilot, Tay.ai and Zoe chatbots wrecking the platform instead of humans maintaining it. If there was a prediction market for when GitHub experiences an outage every week, then you would make a lot of money. [0] https://news.ycombinator.com/item?id=47487881 https://news.ycombinator.com/item?id=47487881
- serf 6mo ago>GitHub goes down at least once a week as I said before. [0] thanks to Copilot, Tay.ai and Zoe chatbots wrecking the platform instead of humans. there are tens of thousands of stupid scripts hosted on github itself that have scheduled progmatic pushes or pulls to repos via cron jobs with millions and millions of users -- yeah LLMs accelerate the fire but let's not pretend that GH was some bastion of real-user-dom somehow at some point.
- pak9rabid 6mo agoVegas should start taking bets
- Imustaskforhelp 6mo agoYou have GOT to be kidding me.
- dylan604 6mo agoNo, got to be kidding me day is next week.
- Imustaskforhelp 6mo agoHey dylan, we got the got to be kidding me day today when claude showed outage (-:
- ahstilde 6mo agogithub is at one nine, basically: https://news.ycombinator.com/item?id=47428035 https://news.ycombinator.com/item?id=47428035
- Imustaskforhelp 6mo ago9% ? /s (though To be honest I genuinely wouldn't be surprised if things go down so bad too at this point either)
- abound 6mo agoUnironically, I think 9% uptime would be "one-tenth of a nine".
- brookst 6mo agoAre you saying 9.999% isn’t four nines?
- munchler 6mo agoCan’t tell if this is intended as humor, but I LOL’ed.
- the_real_cher 6mo agoIt unarguably is.
- mememememememo 6mo agoReliability where N is the number of nines: 1 - 10 ^ -N (multiply by 100 for percent) So 9% is 0.09 for the calc 1 - 10 ^ -N = 0.09 So 10 ^ -N = 0.91 So N = -log10 0.91 So 0.09 (9%) reliability is 0.0409586077 of a nine. And running it thru... a tenth of a nine is 0.2056717653 or about 20.57% reliability
- mememememememo 6mo ago90% would be one 9 following the sequence back. 99.99 99.90 99.00 90.00
- corvad 6mo agoAnd this is why I self host a lot of my Git stack with Gerrit...
- mememememememo 6mo agoOr just make sure you git fetch repos into $other-place. That helps with Git not so much issues etc.
- corvad 6mo agoYeah, I think especially Git mirrors can go a long too for maintaining availability and also for reducing load off main infra.
- steeleduncan 6mo agoWhat has changed at GitHub to cause this?
- smartmic 6mo agoAIpocalypse. Eaten too much Copilot dog food.
- bartread 6mo agoPerhaps even AIslopalypse.
- zahlman 6mo agoI've been using "slopocalypse". People already know AI is responsible, but slop existed before — e.g. conventionally generated SEO spam. It's just... so much worse now.
- bartread 6mo ago"Slopocalypse": yeah, I like that. Easier to pronounce too. At any rate, it seems like GitHub is back up now, so we'll see how long that lasts.
- KaiserPro 6mo agoLooking at the status, its not one long outage, but lots of little ones, microslops if you will.
- adzm 6mo agoWeird Al needs to capitalize on this whole AI/Al thing
- voidfunc 6mo agoAzure
- altairprime 6mo ago
- dsm4ck 6mo agoMicroslop at it again
- workfromspace 6mo agoPlease don't use that term; it makes them look bad :p /s
- kraemahz 6mo agoOh, they've gone beyond micro. It's Macroslop now!
- GiorgioG 6mo agoI'm going to blame Claude Code!
- packetlost 6mo agoI've been sitting here waiting for a critical deploy to happen via GitHub Actions (I know, hour fault, we should have left ages ago). My patience for this bullshit is gone, I'm going to be pushing very hard to get us off of GitHub entirely except for public code mirrors going forward. Edit: oh look, their site says all good, but I still have jobs stuck. What a pile of garbage. I'm so sick of this.
- mememememememo 6mo agoDown? No sir we are not down. There are elevated error rates and degraded performance.
- xtracto 6mo agoAn isolated group of customers are experiencing elevated error rates and degraded performance. FTFY. (I've read AWS word it like that)
- mememememememo 6mo agoTechnucally a set is a subset of itself.
- karim79 6mo agoThe update to .NET framework went badly and we need to reinstall Windows.
- odiroot 6mo agoThey invented the perfect solution to stop supply chain attacks.
- anematode 6mo agoCredential stealers hate this one trick!
- bartread 6mo agoFuck sake. Again? Sorry, I realise this comment isn't up to HN's usual standards for thoughtfulness and it is perhaps a bit inflammatory but... look, I'd bet the majority of us on this site rely on GitHub and I can't be the only one becoming incredibly frustrated with its recent unreliability[0]? (And, yes, I did enough basic data analysis to confirm that it IS indeed getting worse versus a year, two years, and three years ago, and is particularly bad since the start of this year.) [0] EDIT: clearly not from looking at the rest of the comments in this discussion.
- zahlman 6mo ago> I realise this comment isn't up to HN's usual standards for thoughtfulness > And, yes, I did enough basic data analysis to confirm Perhaps you'd consider showing us that analysis? That sounds like it would make a pretty substantive, thoughtful comment.
- KaiserPro 6mo ago> consider showing us Gaze upon the tapestry in which github paints it's failure with a thin copper red thread: https://www.githubstatus.com/ https://www.githubstatus.com/
- bartread 6mo ago@KaiserPro has pasted the link to someone else's heatmap, which is really good. Mine was just an Excel spreadsheet with a graph that I'd intended to write a blog about but then got demotivated on because I was too busy with other things and I saw that heatmap as well. Maybe I will do a proper write up next time GitHub has an outage and I'm blocked by it.
- nasretdinov 6mo agoMust be Tuesday then
- 3842056935870 6mo ago[dead]
- sc__ 6mo agoMicroslop
- hirako2000 6mo agoI'm glad I moved over to forgejo. Being selfhosted, the UI loads faster. Most importantly, the thing is always responsive.
- mfenniak 6mo agoAs a developer working on Forgejo -- glad you like it!
- hirako2000 6mo agoIt hosts all the repositories backing applycreatures, we ran dozens of git projects on the same instance, have teams, you guys did a phenomenal work. I would say it's even easy to customise. https://foja.applycreatures.com https://foja.applycreatures.com Edit: it has a wonderful API so I posted the link it may tempt some to ditch MS/Azure hub.
- corvad 6mo ago[dead]
- paxys 6mo agoTook a full 8 years for a Microsoft acquisition to go to shit, which is probably a record. Kudos to the Github team for holding out this long.
- xeonmc 6mo agoHow fast was Skype?
- cm2187 6mo agoMSFT acquired Skype in 2011, so I would say only a few months: https://trends.google.com/trends/explore?date=all&geo=GB&q=skype&hl=en-US https://trends.google.com/trends/explore?date=all&geo=GB&q=s...
- htrp 6mo agoTo be fair a bunch of this is because the CEO after Nat Friedman (Thomas Dohmke) was pushed out in August 25.
- workfromspace 6mo agoWho was also the last CEO, right? Is this a coincidence?
- carlosft 6mo agoAnd a ton of the top end ruby staff have left. Many of them ended up at shopify. There is a growing about of non ruby/rails code at github, but most of the system that people think of when they think github are ruby/rails.
- ambicapter 6mo agoShopify is on the AI-everything train as well, we'll see how that goes.
- ahartmetz 6mo agoNat Friedman is a grifter who always shows up and says "I'm with these guys" when somebody or something is successful. I don't think that he was responsible for anything good.
- guywithabike 6mo agoThe worst part of all this is that GitHub's CTO and VP of Engineering sent out the usual "here's what we'll do to fix things" letter to their larger customers and, without exaggeration, it boiled down to: 1) "Here's a bunch of stuff we already did!" which... clearly isn't working, and 2) "We're continuing our Azure migration." also clearly not working. So needless to say, if you depend on GitHub for critical business operations, you need to start thinking about what a world without GitHub looks like for your business and start working your way toward that. I know my confidence in GitHub's engineering leadership is at rock bottom.
- packetlost 6mo agoI second this. I'm done.
- Eji1700 6mo agoI could sorta see a situation where the reality is "we're in the middle of a miserable transition and it'll clean up when we're done" but I don't think anyone has confidence that's all it is at this point.
- everforward 6mo agoEven that doesn’t really make sense to me, unless they’ve done it in a way where everything has to move at once. Everywhere I’ve worked, if a migration is causing this much downtime then you kill the migration or slow it down. If every change has a 10% chance of bringing the site down, you only do a change every week or two until you can work out the kinks.
- deleted 6mo ago[deleted]
- rileymichael 6mo agolooking forward to the `addressing-githubs-recent-availability-issues-3` news post
- esafak 6mo agoMicrosoft products are so human, they stop working weekly as if they're observing some sort of sabbath ...
- duped 6mo agoDoes github not do any kind of blue/green rollouts or what
- pylua 6mo agoAnyone else notice other Microsoft cloud services ( for instance inside azure ) with bad performance also? I can’t be specific but we are constantly complaining.
- pbkompasz 6mo agoVibe check?
- overgard 6mo agoI remember back in the early Windows XP era when things got so bad that Microsoft basically had to make a hard pivot towards security and reliability. I think they may need to do that once again. Almost every product of theirs feels like a dumpster fire. GitHub is down constantly, Windows 11 is a nightmare and instead of patching things they're adding stupid features nobody asked for. I think they need to stop and really look closely at what they're prioritizing.
- ekropotin 6mo agoRemember when GitHub was cool? Pepperidge Farm remembers.
- whalesalad 6mo agoI remember. My GitHub user ID is #5907, account created 2008-04-08T20:27:36Z. I think it is inevitable that all good things come to an end, but it's still a bummer to see.
- Freedom2 6mo agoAs do I. Mine is even earlier as well!
- rdedev 6mo agoMy vscode slop session stopped in between. Maybe it's for the better
- wrqvrwvq 6mo ago"the food is terrible and such small portions!"
- jeppester 6mo agoAt this rate it will be a matter of time before a "Github is up" parody site reaches the top of HN
- proc0 6mo agoIt's starting to really look like the AI effect. It might be coincidence but I've noticed a lot more downtime and bad software lately. The last Nvidia drivers gave me a blue screen (last week or so), and speaking about Windows, I froze updates last year because it was clear they were introducing a bunch of issues with every update (not to mention unwanted features). I like AI but actually not for coding because code quality is correlated to how well you understand the underlying systems you're building on, and AI is not really reasoning on this level at all. It's clearly synthesizing training data and it's useful in limited ways.
- boxingdog 6mo ago[dead]
- newbish 6mo agoI think maybe it's not that GitHub is using AI, but that the amount of AI slop going into GitHub may be more than they expected.
- someperson 6mo agoGitHub has been unreliable since before AI. Though it's definitely gotten far worse. Seemingly the decline started with the Microsoft acquisition in 2018, and subsequent "unlimited private repository" change in 2019 (to match Gitlab's popular offer)
- hparadiz 6mo ago
- stevepotter 6mo agoI'm just going to stand by until Microsoft is back in everyone's good graces again by releasing some oss software that we all swoon over
- torben-friis 6mo agoIs that a cyclical thing? I went through the VS code+typescript good graces era but I didn't know there were previous cycles.
- RevEng 6mo agoNot that I remember. VS Code was a surprising turnabout for a company that was both adamantly closed and threw FUD around like monkeys in a zoo.
- philipallstar 6mo agoYou've been here before!
- ransom1538 6mo agoDid MS finish the Hotmail transition?
- wenbin 6mo agoI guess vibe coding can't solve such problem for now...
- s_u_d_o 6mo agoCan this downtime be quantified to actual monetary losses?
- cyanydeez 6mo agoThat's just like your Vibe man; can you just copilot your wayout of these problems?
- zelphirkalt 6mo agoMan, a while ago I thought: "It happens often, alright, but every 2 weeks? Sounds like a slight exaggeration." But it really is every 2 weeks, isn't it? If I imagine in a previous job anything production being down every 2 weeks ... phew, would have had to have a few hard talks and course corrections.
- genewitch 6mo agoi once fixed a site going down several times a year with two t1.micro instances in the same region as the majority of traffic. Instantly solved the problem for what, $20/month? Another site was constantly getting DDoS by Russians who were made we took down their scams on forums, that had to go through verisign back then, not sure who they're using now. They may have enough aggregate pipe it doesn't matter at this point
- newbish 6mo agoSo am I the only one thinking that maybe GitHub is succumbing to the weight of AI slop that's coming in from all the vibecoding, clawbots, and other AI workflows?
- TimReynolds 6mo agoWould be easier for them to just tell us when it’s up these days
- tholford 6mo agoSetting up a Gitea instance is approachable, especially with agent assistance. https://about.gitea.com/ https://about.gitea.com/
- jbmilgrom 6mo agoh8 github so much. ahhhhhh
- messe 6mo ago[flagged]
- justinholmes 6mo agoMoved to self hosted crowci instead of actions.
- lousken 6mo agoGithub CEO must be on HN, right? If so, any comments? They have not even bothered to implement entra login when they have their competitors login for years, do they even know what their product is? Or are you just a middle man for slop?
- mayhemducks 6mo agoDoes anyone else ever think "that code I just pushed into my repo just took down all of github..." whenever it goes down around the same time you sync your changes?
- butterlesstoast 6mo agoThat Go rewrite continues to rear its head, eh?
- gchamonlive 6mo agoJust moved a project of mine to Gitlab. Created this very simple component with codex that will keep a mirror updated on GitHub for me, so I can focus development on Gitlab. https://gitlab.com/gabriel.chamon/ci-components/-/tree/main/templates/github-mirror https://gitlab.com/gabriel.chamon/ci-components/-/tree/main/...
- 0xbadcafebee 6mo agoI'm surprised nobody has tried to throw together a commercial alternative to GitHub. 50% of it is available as FOSS, the other 50% you can vibecode in a month (you can vibecode reliably, Microsoft/Google just suck at it). Afaict, reason we all keep using GitHub is it has a million features and isn't as ugly, difficult and slow as GitLab. (sorry GitLab, I love your handbook, hate your UX)
- nomilk 6mo agoWhy don't companies with chronic outages mimic their stack from top to bottom (i.e. starting with a new domain), then before making a change, make the change on the duplicate stack and blast it with mock requests. Might catch 90% of problems before they make it into the real stack? E.g. every step of GitHub's migration to Azure could be mimicked on the duplicate stack before it's implemented on the primary stack. Is this just considered too much work? (I doubt cost would be the issue, because even if it costs millions, it would pay for itself in reduced reputational damage from outages). EDIT: downvotes - why? - I think this is a good idea (I'd do it for my sites if outages were an issue).
- worik 6mo agoTesting? Who needs it when you have Copilot!
- drewbug01 6mo ago> EDIT: downvotes - why? - I think this is a good idea (I'd do it for my sites if outages were an issue). Because that's a monumental amount of work, and extraordinarily difficult to retrofit into a system that wasn't initially designed that way. Not to mention the unstated requirement of mirroring traffic to actually exercise that system (given the tendency of bugs to not show up until something actually uses the system).
- nomilk 6mo ago> that's a monumental amount of work Agree, but look at the alternative; GitHub is constantly being savaged by users who (quite reasonably) expect uptime. Ignoring impacts on morale and reputation, damage to their bottom line alone might tens (hundreds?) of millions per year. > mirroring traffic yeah, I agree that's difficult, but it need to not be exact to still be useful.
- hedora 6mo agoDownvotes are probably because that is what companies without chronic outages do. If you'd ever worked on a codebase as terrible as I imagine GH's internals are and looked at the git history, you'd find two things: 1) fixing it would require rolling back 100's-1000's of engineer-years of idiocy that make things like testing or refactoring untenable 2) many prior engineers got part of the way through such improvements before leaving or being kicked out. Their efforts mostly just made it worse, because now you never know what sort of terribleness to expect when you open an unfamiliar file.
- rco8786 6mo agoGH is ripe for a disruptor right now.
- gverrilla 6mo agoWhat's the successor?
- darig 6mo ago[dead]
- sathish316 6mo agoHow much of this is due to Microsoft Culture of not innovating and buying leading companies with their revenue from windows/office and slowly destroying the aspects of companies that made them great in the first place? Is all the recent GitHub downtime entirely attributable to GitHub AI Copilot related development? How hard can it be to reduce the blast radius of new AI features to not affect the core parts of hosting repositories? Because of Copilot everywhere, The UX has become bad and I had to click all over the place and on my profile to find repositories.
- jiveturkey 6mo agoI sure hope they created a restore point first.
- deleted 6mo ago[deleted]
- unmayx 6mo ago[dead]
- zzo38computer 6mo agoI do not care about much of it other than the git and API. I also sometimes use the Issues, although only with the API. But if it stops working sometimes, that is not too significantly a problem since the files can be sent after they start to work again; it does not have to be immediately.
- lesoutilsia 6mo ago[dead]
- zombot 6mo agoGood. Users should be abandoning this vibe-coded hellscape.
- ygritte 6mo agoYou're underestimating Microsoft. They're fully capable of fucking up without the help of AI. Although I heard it's really good at that.