16 ms·
It's been hours :( I have sympathy for the on-call team trying to resolve it, most of us have been there done that. But seems something is systematically goin
by alamsterdam 1mo ago
It's been hours :(
I have sympathy for the on-call team trying to resolve it, most of us have been there done that.
But seems something is systematically going wrong at GH
- yashap 1mo agoIt's honestly insane how terrible their reliability is. Over the past 4 weeks, 8 days with GitHub Actions outages, many of them multi-hour outages: >3 hours on each of July 9th, July 20th and today, and 1.5 hrs on July 23rd. Outages happen, but this many outages so close together, and so many of them so major/long lasting, something is systematically wrong for sure. It's been seriously hamstringing our ability to ship code at my company.
- alamsterdam 1mo agoMon and Dad (MS) are (generally) pretty solid with uptime. What is happening at GH? Rate of change trying to keep up with new challengers? Over-reliance on AI? Engineers trying to debug slop?
- yashap 1mo agoYeah who knows, would be interesting to hear an inside take if any readers here are also GH devs! They're at 93.91% uptime over the past 90 days, according to https://mrshu.github.io/github-statuses/ https://mrshu.github.io/github-statuses/ , and that doesn't even include today's outage yet. A glorious one nine of reliability.
- marcprux 1mo agoI see two nines in there…
- warmwaffles 1mo agoDon't throw shade, there are two nines in that percentage for now. Miles a part though.
- alamsterdam 1mo ago911 has one! :)
- reilly3000 1mo agoYea just try requesting an SLA credit from them. According to THEIR numbers it’s 99.995%. It’s intolerable and I am going to put my full weight behind stripping as much work as we can from GHA as possible, even if leaving GitHub itself is effectively logistically and contractually impossible.
- gitgud 1mo ago> They're at 93.91% uptime over the past 90 days, according to https://mrshu.github.io/github-statuses/ https://mrshu.github.io/github-statuses/ , and that doesn't even include today's outage yet. I don’t love GitHub, but that number is a little misleading… That’s the intersection uptime of all GitHub services, most of which I (and most users) do not care about; code spaces, copilot, packages etc… When you take out those uptimes, it becomes a lot higher. I’ll admit there seems to be a lot more incidents than usual though…
- simoncion 1mo ago> What is happening at GH? In large part, the move from AWS to Azure. Azure's just bad.
- alamsterdam 1mo agowasn't that years ago? or was that Skype? hahaha
- simoncion 1mo agoAFAIK, the move to Azure is still ongoing. Rumor has it that it was delayed for a very long time after the Github acquisition because Azure was too bad to move Github to. Azure remains too bad to move Github to, but the relevant executives have pretty clearly decided that that's no longer a good reason to delay the move.
- PsylentKnight 1mo agoBased on previous posts I've seen about this, IIRC the timing seems to imply that it has more to do with them being hammered with AI slop than it does with the Azure transition. Who really knows though
- simoncion 1mo ago> ...IRC the timing seems to imply that it has more to do with them being hammered with AI slop... It can be both things. I've worked professionally with AWS, Azure, and GCP, and Azure is just really flaky and unreliable. It's easily the worst of the three.
- mococa 1mo ago> But seems something is systematically going wrong at GH Yes, we call it: Microslop.
- tempaccount420 1mo agoThey're one rewrite in Rust away from fixing everything. (jk)
- hinkley 1mo agoTime to touch some grass. Better use of my time and energy than twisting the remaining things on my todo list today to make more progress on them than I have managed. I should have taken a long lunch but I rebased the hell out of a PR instead.
- niwtsol 1mo ago"most of us have been there done that" - so true. That feeling in your gut when you realize something you just did caused an outage is pretty unique.