7 ms·

I don't think this portents anything great for software in general. We're a good year+ into the use LLMs for all major bits of software that we all rely upon a

by zehaeva 1mo ago

I don't think this portents anything great for software in general. We're a good year+ into the use LLMs for all major bits of software that we all rely upon and GitHub here is down to one 9 of uptime. I've been using GitHub for a _long_ time, my first commits there go back to August 2009!, and I honestly don't recall GitHub going down as much as it has in the last year. I'm sure there's other things happening in the background, but I can not help but believe that this is directly correlated with the increase of LLM usage. Though I would love to hear someone else's pet theory how a rock of the internet went from four+ nines of uptime to maybe one.

askonomm 1mo ago
To me this correlates more to them being bought by Microsoft, a company known for being seemingly incapable of creating quality software to the point that it's not even funny anymore, and also known for sloppifying all the products they touch.
- IshKebab 1mo ago
  I don't think so. GitHub was bought by Microsoft 8 years ago and people have only started complaining about its uptime in the last year or so - exactly correlating with the surge in LLM use.
  - pluralmonad 1mo ago
    Didn't github also migrate at azure recently? That certainly can't help.
    - deleted 1mo ago
      [deleted]
  - onraglanroad 1mo ago
    I don't think so. I've seen those complaints for more than a year. I have an Ops background and I strongly suspect they were given a stupid timeline for the Azure migration. I've got to believe Microsoft have decent Ops people but the management wanted to move faster than was reasonable and screwed it up. Move one thing at a time and double check it all works and you can do a migration like this.
    - macintux 1mo ago
      Hotmail redux.
    - tempest_ 1mo ago
      Even on a large timeline there is just a lot of risk when migrating legacy infra to an entirely new system, especially if the original arch grew "organically" and has tones of edge cases. I always figured these problems were directly related to the migration.
    - crabbone 1mo ago
      I've lived through two acquisitions by the world-largest companies (and few more smaller ones). Here's my impression of what often happens in situations like this: * There's a huge disconnect between the actual product and what was sold to the buyer. It could be that the product was a borderline fraud, or it could be that the product was actually much better quality than the expectation on the buyer's side, but the buyer isn't interested in most of the product. * Fear spreads in the acquired company that their product will be discontinued or reshaped into something else. * The pay is good, probably much better than before the company was bought. * Many internal teams end up lacking real purpose and try to insert themselves into every new internal project only to obfuscate their irrelevance. This leads to some pathological developments, where the teams previously working on acquired products start doing a lot of useless, for-show work. Internal initiatives sprout like mushrooms after a summer rain, but they are all plagued by very broad (and mostly irrelevant) team involvement, duplication of existing products / services, and fear of being discovered. And while there are plenty of such initiatives, their role is to be a superficial distraction. In reality, everyone is afraid to touch the old code or do any sensible integration because it could lead to blanket firing of a lot of people. The middle-management behavior becomes a sort of exchange of favors, where everyone is afraid that the other can blackmail them into losing their job, and so everyone is trying to be extra nice by offering a slice of a pie to another manager. What this leads to, in reality, is insane inertia (often despite highly shortened software release cycles), astronomic amounts of unmitigated tech. debt, opacity in communication with management, persecution of those who genuinely want to improve the system. Based on this, my prediction is that Github will not survive. Just like Skype didn't. Somehow or other, Microsoft will find a way to replace the product with... MS Outlook with a new skin.
  - axod 1mo ago
    https://damrnelson.github.io/github-historical-uptime/ https://damrnelson.github.io/github-historical-uptime/ Seems pretty conclusive. Very similar story when they bought skype.
    - gtowey 1mo ago
      It's not the whole story. The biggest change is actually the internal rules for how downtime was reported, it wasn't actually such a large change in the actual reliability then.
    - willio58 1mo ago
      God that's sad. I almost feel bad for all the engineers there, though I'm sure they made good money and probably left
      - axod 1mo ago
        Do they still have any engineers? From the downtime it seems like they don't any more. Yep - the competent ones likely all left.
    - inigyou 1mo ago
      Hosting that on GitHub Pages is ballsy.
    - zamalek 1mo ago
      That's missing a some of the LLM era? Maybe the author is working on a logarithmic view?
    - Izkata 1mo ago
      Heatmap of the past year: https://isgithubcooked.com/ https://isgithubcooked.com/ It got worse around April/May this year, while that graph stops at February.
  - miyoji 1mo ago
    > people have only started complaining about its uptime in the last year or so I'm sorry but this made me laugh out loud. That isn't true at all, this has been going on for years. This conversation[0] from six years ago has discussion about the outages starting to become much more frequent in December 2019. It has never gotten better in that time, it's just continually degraded. [0] https://news.ycombinator.com/item?id=22935941 https://news.ycombinator.com/item?id=22935941
    - everfrustrated 1mo ago
      Yup. GitHub has easily the worst human-noticable downtime for all SaaS services I've used going back over 10 years. Most people work around it by self-hosting github (which has other problems but uptime aint one).
      - inigyou 1mo ago
        What problems does it have?
    - IshKebab 1mo ago
      Ok I should have said people have only started constantly complaining in the past year or so. Of course there was the odd complaint in the past, but now I see a thread about it at least ever month on HN, and loads of articles about migration off GitHub.
  - Melatonic 1mo ago
    Isnt it also in the past year when they started to move Github fully to microsoft infra? Off whatever they were doing before
- shevy-java 1mo ago
  It's true, Microsoft made github worse, but the more recent issues seem to have to do a lot more with Microsoft selling its soul to AI. Microsoft really appears to have gotten dumber as they became dependent on AI. Most recent example: they used to promote Win11 and 32GB RAM. Now they are down to 8GB silently ... this is quite hilarious. There are now so many side effects that you see degradation in so many other areas. Or the gaming industry: it is not quite dying but it is taking a huge hit with skyrocketing RAM prices. Consoles selling less is an example here. It's quite fascinating how deadly disruptive AI is now.
  - stingraycharles 1mo ago
    This really needs to have a lot more evidence that it’s because GitHub’s code is being written by AI vs them being bombarded by activity from all kinds of AI agents all over the world vs they were already on a trajectory of quality loss.
  - scandals 1mo ago
    >they used to promote Win11 and 32GB RAM. Its common knowledge that Win11 needs 32gb RAM, but im glad to hear that Microsoft is aware of it. Looking forward to this 8gb variant :)
  - Shorel 1mo ago
    The 32GB issue is more about one internal factor: the wasteful reliance on Electron and other JS runtimes for UI, and two external factors: the very high price of RAM, and the increasing adoption of Linux. So they will make Windows run on 8GB because people can't purchase cheap RAM anymore, and they want to stop people migrating to Linux. This is not directly caused by AI slop, only indirectly through RAM prices.
- dreamcompiler 1mo ago
  Microsoft is the General Motors of software.
  - HeWhoLurksLate 1mo ago
    at present I'd almost put them at Stellantis levels of bad
    - dreamcompiler 1mo ago
      Good point. I'm amazed that people are out there buying new Jeeps and Ram trucks. But I'm equally amazed people are out there buying 365 too.
      - deleted 1mo ago
        [deleted]
- acuteaura 1mo ago
  GitHub Actions was always a mess. Because it is a Microsoft product. If you want to have a horrible time, try reading some of the runner code. It's early 2010s-style Windows-First, MSFT C# crud that has trouble not racing several threads to inconclusive status codes.
  - microgpt 1mo ago
    [flagged]
    - acuteaura 1mo ago
      I'm not sure if this is a joke or not, because it has all the signals of a joke, but it might be closer to true than I'd like to admit. At my work, the average quality has gone up with "blindly trusting Opus". The failure modes are horrendous and the code is verbose as shit, but it still works better. Most "engineers" aren't very good at writing code.
  - crabbone 1mo ago
    Yes. That was my impression too. Github Actions aren't a good product in many ways. It's also very complex and requires quite a bit more infrastructure than the rest of Github. Unfortunately, I don't think that the commercial side of the product would allow it to improve in the direction of better quality / uptime. It's cursed to be forever like MS Outlook: whenever it changes it's for the worse, even though it was never good.
    - acuteaura 1mo ago
      More literally a Microsoft product. As in, rebadged Azure Pipelines.
porridgeraisin 1mo ago
For the longest time, I didn't appreciate the "AI-induced traffic" excuse. But seriously, I checked the rough github egress for our lab versus an old log from 2024, and there's an order of magnitude or two difference. From asking around, it seems people all have the gh cli tool and let it loose with parallel tool calls and e.g LLM's polling Actions in a background bash while loop with sleep $TOO_FEW_SECONDS. Some people use a variety of skills where the agent makes a commit every few code changes, and uses Issues for its memory/log. And they have O(5) sessions at the same time doing all kinds of crap. It's the same with PR checks/PRs. Recently we also saw continued usage throughout the night as well, which did not exist pre coding agents. Loops or whatever they call cron jobs in the harnesses these days is the reason. It must be adding up.
- bloppe 1mo ago
  Ya they're getting boned
- crote 1mo ago
  I honestly don't care either way. Either they are using AI as an excuse to hide their incompetence, or they are constantly going down due to the AI they have been promoting. If your platform can't handle the use patterns of AI, then perhaps don't go around telling everyone to use AI for everything? It's a self-inflicted wound, you could also just not do this. Too bad Microsoft has bet its future on AI not being a giant bubble, huh?
  - habinero 1mo ago
    Someone else quoted a 14x increase in load due to AI hammering github. I wouldn't be at all shocked if that was true, considering how much AI tools are DDOSing the entire internet. That plus migrating clouds is insanely difficult to manage. They're almost certainly drowning in traffic and trying to keep up.
    - inigyou 1mo ago
      There's no evidence the internet DDOS is AI companies
gershy 1mo ago
I guess github is kind of a shared garden. Interesting that, like in game theory, if everyone is using it too much, no one gets to use it.
- jitbit 1mo ago
  tragedy of the commons, exactly
- steve-atx-7600 1mo ago
  charge more for it. its fine if free customers dont have service. dont break paying customers
  - gershy 1mo ago
    Completely agreed
  - InsideOutSanta 1mo ago
    This. I use GitHub for free, and I'm incredibly grateful for the amount of stuff they provide. I don't mind my Actions going down once in a while.
BigTuna 1mo ago
It's probably not attributable to AI in the way that you're thinking - Github has been absorbing an exponential increase in usage, and that increase is mostly due to new AI-related projects being created and worked on. Though I'm sure some of the blame can go to internal slop code.
cortesoft 1mo ago
Are these outages caused by introduced bugs, though, or by load issues? As someone who has spent many years working in high load environments, this is not an uncommon pattern. You design a system and it works great. It can handle failures, load spikes, it is horizontally scalable, things are great. You think you figured it out. And then load keeps increasing and you suddenly hit a tipping point where everything keeps failing, and you cant keep up. The things that you thought were perfectly horizontally scalable turn out to have a bottleneck you didn’t even think about until you got to a truly massive scale. Your systems suddenly don’t have the excess capacity to handle load spikes or catchup work, so suddenly any failure cascades and recovery is more and more difficult. You can’t solve the problem with additional hardware, and your perfect scalable design actually can’t scale any more. This doesn’t have to be about GitHub using LLMs in their code to still be related to LLMs. GitHub gets a lot more commits now because of LLMs and probably get a lot more reads because of LLMs as well. It could be that the extra usage just pushed them past one of those capacity thresholds.
- Syntaf 1mo ago
  There was a great article awhile back that shed some light on just how dysfunctional azure is as a platform: https://news.ycombinator.com/item?id=47616242 https://news.ycombinator.com/item?id=47616242 If I had to guess it's because Github is sitting on top on infrastructure held up by toothpicks and duct tape
  - toomuchtodo 1mo ago
    Which is somewhat humorous because it was arguably more stable when they ran on their own hosted colo infra before moving to Azure. This was a choice versus keeping the infra compartmentalized and using Azure for elastic overflow compute needs. I'm sure marketing and bonuses rest on throwing it all on the Azure quicksand though. GitHub Will Prioritize Migrating to Azure Over Feature Development - https://news.ycombinator.com/item?id=45517173 https://news.ycombinator.com/item?id=45517173 - October 2025 (63 comments)
    - hirako2000 1mo ago
denysvitali 1mo ago
Microsoft acquisition which forced to migrate all to Azure.
- zehaeva 1mo ago
  I can totally buy that this is a larger contributor! Thank you, I wasn't aware that it was going on right now.
- judge2020 1mo ago
  And yet they still have a completely separate IDP for employees https://github.okta.com https://github.okta.com .
  - sitzkrieg 1mo ago
    most companies do. do you mean they’re not using AD only?
    - judge2020 1mo ago
      Most companies onboard and consolidate IDPs fairly quickly because it's quite easy to change the SSO configuration for services over just a few years. My only guess is there's some circular dependency they want to avoid by keeping Okta.
tylerdavis 1mo ago
They mentioned earlier this year that they were beginning the migration to Azure and that it would take a couple years. I would assume it has more to do with that migration then anything else.
paulsutter 1mo ago
It is definitely and absolutely caused by LLMs. I must do 20x more GitHub operations now, and since the agents know GitHub far better than me, I'm using more advanced features. Multiply this times all of us.
Melatonic 1mo ago
The funny thing is that if companies wanted they could probably use AI to instead increase uptime. Keep existing QA teams (instead of replacing them) and then use AI for better and more timely monitoring (and messaging even) and as an additional Always-Testing™ layer of QA
- tofuahdude 1mo ago
  "Just use AI" is definitely not the solve for the problems Github faces.
  - pdimitar 1mo ago
    Shhh! Do you hear that? It's the collective scream of middle managers screaming in terror. Jokes aside, yes, quite correct. Even frontier models like Fable have very obvious limits.
nickspag 1mo ago
someone from github posted a usage graph from the last year on twitter a while ago and they were serving like 14x more requests in a matter of months. it's frankly impressive they've kept up.
- honkostani 1mo ago
  [dead]
UncleOxidant 1mo ago
I wonder if GitHub actions was a bad idea? Like maybe it's being abused for other kinds of compute besides just builds? And even builds themselves can require a lot of compute. I've only recently had a repo there where I wanted to do builds to make a release (both linux binaries and WASM) and whenever I do that tag and wait a few minutes for those builds to finish I think about all the other projects/repos out there on GitHub doing the same. I'm really kind of surprised they let us do that - like, why didn't they just have you upload the binaries after building on your local machine?
- judge2020 1mo ago
  > why didn't they just have you upload the binaries after building on your local machine? You can do that already with GH Releases. Actions is if you want CI/CD managed by GitHub. And you can also use your own machines via self-hosted runners.
- crabbone 1mo ago
  They let you do it because GitLab was first to let you do that. MS had to match the offering of their most noticeable competitor. They also felt like thay had to one-up them... well, to win the competition. So, from the sales point of view, Github Actions was, at the minimum, an alright idea. Not brilliant, but quite obvious and expected. From the engineering standpoint, however, this is a disaster on many levels. But, that never stopped Microsoft before. They don't try to win the market by making an objectively better product, their tactics are and always have been to make a product that can claim (with an asterisk) to be able to do a lot of things the customer wanted only to discover afterwards that those promises were phony.
JohnTHaller 1mo ago
The last month GitHub hit four 9s of uptime was November 2024
- judge2020 1mo ago
  Yeah, GH's availability has been a joke for many years, maybe even for a decade now.
TacticalCoder 1mo ago
> We're a good year+ into the use LLMs for all major bits of software that we all rely upon and GitHub here is down to one 9 of uptime. With what to show for it? If GH did 10x in volume/git commits, it's all LLM sloppy-pasta. Where's the 10x productivity? Where are the amazing apps?
- crote 1mo ago
  They have delivered some great shareholder value!
nullbio 1mo ago
It's caused by human laziness and corner cutting. LLMs can write buggy code or incomplete architectures as much as humans can, but standards have lowered. It's not the LLMs are not capable of also fixing these same issues, but that's additional work.
skinfaxi 1mo ago
> I don't think this portents anything great for software in general. I think it is a natural progression in technology. Failure rates were high for initial aircraft designs and safety improved over time, for instance.
- jeltz 1mo ago
  But in this case it is the other way round, failure rate is going up.
ransom1538 1mo ago
Now GITHUB gets to experience debugging the GHA in prod. From my experience, good luck.
mhitza 1mo ago
General outage was uncommon, but unicorns and octocat popping up for particular pages/repos was common enough for me in the past.
crabbone 1mo ago
I saw a graph of Github outages somewhere. The drop in uptime strongly correlates with the introduction of Github Actions. To be fair, it's a huge chunk of functionality, also, hard to make reliable. LLMs might have an effect, but the issues definitely started before LLMs were used for code generation, so, as much as I don't like the AI-generated code, I'd have to admit that it's probably not the cause here.

by zehaeva 1mo ago

askonomm 1mo ago

To me this correlates more to them being bought by Microsoft, a company known for being seemingly incapable of creating quality software to the point that it's not even funny anymore, and also known for sloppifying all the products they touch.

IshKebab 1mo ago
I don't think so. GitHub was bought by Microsoft 8 years ago and people have only started complaining about its uptime in the last year or so - exactly correlating with the surge in LLM use.
- pluralmonad 1mo ago
  Didn't github also migrate at azure recently? That certainly can't help.
  - deleted 1mo ago
    [deleted]
- onraglanroad 1mo ago
  I don't think so. I've seen those complaints for more than a year. I have an Ops background and I strongly suspect they were given a stupid timeline for the Azure migration. I've got to believe Microsoft have decent Ops people but the management wanted to move faster than was reasonable and screwed it up. Move one thing at a time and double check it all works and you can do a migration like this.
  - macintux 1mo ago
    Hotmail redux.
  - tempest_ 1mo ago
    Even on a large timeline there is just a lot of risk when migrating legacy infra to an entirely new system, especially if the original arch grew "organically" and has tones of edge cases. I always figured these problems were directly related to the migration.
  - crabbone 1mo ago
    I've lived through two acquisitions by the world-largest companies (and few more smaller ones). Here's my impression of what often happens in situations like this: * There's a huge disconnect between the actual product and what was sold to the buyer. It could be that the product was a borderline fraud, or it could be that the product was actually much better quality than the expectation on the buyer's side, but the buyer isn't interested in most of the product. * Fear spreads in the acquired company that their product will be discontinued or reshaped into something else. * The pay is good, probably much better than before the company was bought. * Many internal teams end up lacking real purpose and try to insert themselves into every new internal project only to obfuscate their irrelevance. This leads to some pathological developments, where the teams previously working on acquired products start doing a lot of useless, for-show work. Internal initiatives sprout like mushrooms after a summer rain, but they are all plagued by very broad (and mostly irrelevant) team involvement, duplication of existing products / services, and fear of being discovered. And while there are plenty of such initiatives, their role is to be a superficial distraction. In reality, everyone is afraid to touch the old code or do any sensible integration because it could lead to blanket firing of a lot of people. The middle-management behavior becomes a sort of exchange of favors, where everyone is afraid that the other can blackmail them into losing their job, and so everyone is trying to be extra nice by offering a slice of a pie to another manager. What this leads to, in reality, is insane inertia (often despite highly shortened software release cycles), astronomic amounts of unmitigated tech. debt, opacity in communication with management, persecution of those who genuinely want to improve the system. Based on this, my prediction is that Github will not survive. Just like Skype didn't. Somehow or other, Microsoft will find a way to replace the product with... MS Outlook with a new skin.
- axod 1mo ago
  https://damrnelson.github.io/github-historical-uptime/ https://damrnelson.github.io/github-historical-uptime/ Seems pretty conclusive. Very similar story when they bought skype.
  - gtowey 1mo ago
    It's not the whole story. The biggest change is actually the internal rules for how downtime was reported, it wasn't actually such a large change in the actual reliability then.
  - willio58 1mo ago
    God that's sad. I almost feel bad for all the engineers there, though I'm sure they made good money and probably left
    - axod 1mo ago
      Do they still have any engineers? From the downtime it seems like they don't any more. Yep - the competent ones likely all left.
  - inigyou 1mo ago
    Hosting that on GitHub Pages is ballsy.
  - zamalek 1mo ago
    That's missing a some of the LLM era? Maybe the author is working on a logarithmic view?
  - Izkata 1mo ago
    Heatmap of the past year: https://isgithubcooked.com/ https://isgithubcooked.com/ It got worse around April/May this year, while that graph stops at February.
- miyoji 1mo ago
  > people have only started complaining about its uptime in the last year or so I'm sorry but this made me laugh out loud. That isn't true at all, this has been going on for years. This conversation[0] from six years ago has discussion about the outages starting to become much more frequent in December 2019. It has never gotten better in that time, it's just continually degraded. [0] https://news.ycombinator.com/item?id=22935941 https://news.ycombinator.com/item?id=22935941
  - everfrustrated 1mo ago
    Yup. GitHub has easily the worst human-noticable downtime for all SaaS services I've used going back over 10 years. Most people work around it by self-hosting github (which has other problems but uptime aint one).
    - inigyou 1mo ago
      What problems does it have?
  - IshKebab 1mo ago
    Ok I should have said people have only started constantly complaining in the past year or so. Of course there was the odd complaint in the past, but now I see a thread about it at least ever month on HN, and loads of articles about migration off GitHub.
- Melatonic 1mo ago
  Isnt it also in the past year when they started to move Github fully to microsoft infra? Off whatever they were doing before
shevy-java 1mo ago
It's true, Microsoft made github worse, but the more recent issues seem to have to do a lot more with Microsoft selling its soul to AI. Microsoft really appears to have gotten dumber as they became dependent on AI. Most recent example: they used to promote Win11 and 32GB RAM. Now they are down to 8GB silently ... this is quite hilarious. There are now so many side effects that you see degradation in so many other areas. Or the gaming industry: it is not quite dying but it is taking a huge hit with skyrocketing RAM prices. Consoles selling less is an example here. It's quite fascinating how deadly disruptive AI is now.
- stingraycharles 1mo ago
  This really needs to have a lot more evidence that it’s because GitHub’s code is being written by AI vs them being bombarded by activity from all kinds of AI agents all over the world vs they were already on a trajectory of quality loss.
- scandals 1mo ago
  >they used to promote Win11 and 32GB RAM. Its common knowledge that Win11 needs 32gb RAM, but im glad to hear that Microsoft is aware of it. Looking forward to this 8gb variant :)
- Shorel 1mo ago
  The 32GB issue is more about one internal factor: the wasteful reliance on Electron and other JS runtimes for UI, and two external factors: the very high price of RAM, and the increasing adoption of Linux. So they will make Windows run on 8GB because people can't purchase cheap RAM anymore, and they want to stop people migrating to Linux. This is not directly caused by AI slop, only indirectly through RAM prices.
dreamcompiler 1mo ago
Microsoft is the General Motors of software.
- HeWhoLurksLate 1mo ago
  at present I'd almost put them at Stellantis levels of bad
  - dreamcompiler 1mo ago
    Good point. I'm amazed that people are out there buying new Jeeps and Ram trucks. But I'm equally amazed people are out there buying 365 too.
    - deleted 1mo ago
      [deleted]
acuteaura 1mo ago
GitHub Actions was always a mess. Because it is a Microsoft product. If you want to have a horrible time, try reading some of the runner code. It's early 2010s-style Windows-First, MSFT C# crud that has trouble not racing several threads to inconclusive status codes.
- microgpt 1mo ago
  [flagged]
  - acuteaura 1mo ago
    I'm not sure if this is a joke or not, because it has all the signals of a joke, but it might be closer to true than I'd like to admit. At my work, the average quality has gone up with "blindly trusting Opus". The failure modes are horrendous and the code is verbose as shit, but it still works better. Most "engineers" aren't very good at writing code.
- crabbone 1mo ago
  Yes. That was my impression too. Github Actions aren't a good product in many ways. It's also very complex and requires quite a bit more infrastructure than the rest of Github. Unfortunately, I don't think that the commercial side of the product would allow it to improve in the direction of better quality / uptime. It's cursed to be forever like MS Outlook: whenever it changes it's for the worse, even though it was never good.
  - acuteaura 1mo ago
    More literally a Microsoft product. As in, rebadged Azure Pipelines.

porridgeraisin 1mo ago

For the longest time, I didn't appreciate the "AI-induced traffic" excuse. But seriously, I checked the rough github egress for our lab versus an old log from 2024, and there's an order of magnitude or two difference. From asking around, it seems people all have the gh cli tool and let it loose with parallel tool calls and e.g LLM's polling Actions in a background bash while loop with sleep $TOO_FEW_SECONDS. Some people use a variety of skills where the agent makes a commit every few code changes, and uses Issues for its memory/log. And they have O(5) sessions at the same time doing all kinds of crap. It's the same with PR checks/PRs. Recently we also saw continued usage throughout the night as well, which did not exist pre coding agents. Loops or whatever they call cron jobs in the harnesses these days is the reason. It must be adding up.

bloppe 1mo ago
Ya they're getting boned
crote 1mo ago
I honestly don't care either way. Either they are using AI as an excuse to hide their incompetence, or they are constantly going down due to the AI they have been promoting. If your platform can't handle the use patterns of AI, then perhaps don't go around telling everyone to use AI for everything? It's a self-inflicted wound, you could also just not do this. Too bad Microsoft has bet its future on AI not being a giant bubble, huh?
- habinero 1mo ago
  Someone else quoted a 14x increase in load due to AI hammering github. I wouldn't be at all shocked if that was true, considering how much AI tools are DDOSing the entire internet. That plus migrating clouds is insanely difficult to manage. They're almost certainly drowning in traffic and trying to keep up.
  - inigyou 1mo ago
    There's no evidence the internet DDOS is AI companies

gershy 1mo ago

I guess github is kind of a shared garden. Interesting that, like in game theory, if everyone is using it too much, no one gets to use it.

jitbit 1mo ago
tragedy of the commons, exactly
steve-atx-7600 1mo ago
charge more for it. its fine if free customers dont have service. dont break paying customers
- gershy 1mo ago
  Completely agreed
- InsideOutSanta 1mo ago
  This. I use GitHub for free, and I'm incredibly grateful for the amount of stuff they provide. I don't mind my Actions going down once in a while.

BigTuna 1mo ago

It's probably not attributable to AI in the way that you're thinking - Github has been absorbing an exponential increase in usage, and that increase is mostly due to new AI-related projects being created and worked on. Though I'm sure some of the blame can go to internal slop code.

cortesoft 1mo ago

Are these outages caused by introduced bugs, though, or by load issues? As someone who has spent many years working in high load environments, this is not an uncommon pattern. You design a system and it works great. It can handle failures, load spikes, it is horizontally scalable, things are great. You think you figured it out. And then load keeps increasing and you suddenly hit a tipping point where everything keeps failing, and you cant keep up. The things that you thought were perfectly horizontally scalable turn out to have a bottleneck you didn’t even think about until you got to a truly massive scale. Your systems suddenly don’t have the excess capacity to handle load spikes or catchup work, so suddenly any failure cascades and recovery is more and more difficult. You can’t solve the problem with additional hardware, and your perfect scalable design actually can’t scale any more. This doesn’t have to be about GitHub using LLMs in their code to still be related to LLMs. GitHub gets a lot more commits now because of LLMs and probably get a lot more reads because of LLMs as well. It could be that the extra usage just pushed them past one of those capacity thresholds.

Syntaf 1mo ago
There was a great article awhile back that shed some light on just how dysfunctional azure is as a platform: https://news.ycombinator.com/item?id=47616242 https://news.ycombinator.com/item?id=47616242 If I had to guess it's because Github is sitting on top on infrastructure held up by toothpicks and duct tape
- toomuchtodo 1mo ago
  Which is somewhat humorous because it was arguably more stable when they ran on their own hosted colo infra before moving to Azure. This was a choice versus keeping the infra compartmentalized and using Azure for elastic overflow compute needs. I'm sure marketing and bonuses rest on throwing it all on the Azure quicksand though. GitHub Will Prioritize Migrating to Azure Over Feature Development - https://news.ycombinator.com/item?id=45517173 https://news.ycombinator.com/item?id=45517173 - October 2025 (63 comments)
  - hirako2000 1mo ago

denysvitali 1mo ago

Microsoft acquisition which forced to migrate all to Azure.

zehaeva 1mo ago
I can totally buy that this is a larger contributor! Thank you, I wasn't aware that it was going on right now.
judge2020 1mo ago
And yet they still have a completely separate IDP for employees https://github.okta.com https://github.okta.com .
- sitzkrieg 1mo ago
  most companies do. do you mean they’re not using AD only?
  - judge2020 1mo ago
    Most companies onboard and consolidate IDPs fairly quickly because it's quite easy to change the SSO configuration for services over just a few years. My only guess is there's some circular dependency they want to avoid by keeping Okta.

tylerdavis 1mo ago

They mentioned earlier this year that they were beginning the migration to Azure and that it would take a couple years. I would assume it has more to do with that migration then anything else.

paulsutter 1mo ago

It is definitely and absolutely caused by LLMs. I must do 20x more GitHub operations now, and since the agents know GitHub far better than me, I'm using more advanced features. Multiply this times all of us.

Melatonic 1mo ago

The funny thing is that if companies wanted they could probably use AI to instead increase uptime. Keep existing QA teams (instead of replacing them) and then use AI for better and more timely monitoring (and messaging even) and as an additional Always-Testing™ layer of QA

tofuahdude 1mo ago
"Just use AI" is definitely not the solve for the problems Github faces.
- pdimitar 1mo ago
  Shhh! Do you hear that? It's the collective scream of middle managers screaming in terror. Jokes aside, yes, quite correct. Even frontier models like Fable have very obvious limits.

nickspag 1mo ago

someone from github posted a usage graph from the last year on twitter a while ago and they were serving like 14x more requests in a matter of months. it's frankly impressive they've kept up.

honkostani 1mo ago
[dead]

UncleOxidant 1mo ago

I wonder if GitHub actions was a bad idea? Like maybe it's being abused for other kinds of compute besides just builds? And even builds themselves can require a lot of compute. I've only recently had a repo there where I wanted to do builds to make a release (both linux binaries and WASM) and whenever I do that tag and wait a few minutes for those builds to finish I think about all the other projects/repos out there on GitHub doing the same. I'm really kind of surprised they let us do that - like, why didn't they just have you upload the binaries after building on your local machine?

judge2020 1mo ago
> why didn't they just have you upload the binaries after building on your local machine? You can do that already with GH Releases. Actions is if you want CI/CD managed by GitHub. And you can also use your own machines via self-hosted runners.
crabbone 1mo ago
They let you do it because GitLab was first to let you do that. MS had to match the offering of their most noticeable competitor. They also felt like thay had to one-up them... well, to win the competition. So, from the sales point of view, Github Actions was, at the minimum, an alright idea. Not brilliant, but quite obvious and expected. From the engineering standpoint, however, this is a disaster on many levels. But, that never stopped Microsoft before. They don't try to win the market by making an objectively better product, their tactics are and always have been to make a product that can claim (with an asterisk) to be able to do a lot of things the customer wanted only to discover afterwards that those promises were phony.

JohnTHaller 1mo ago

The last month GitHub hit four 9s of uptime was November 2024

judge2020 1mo ago
Yeah, GH's availability has been a joke for many years, maybe even for a decade now.

TacticalCoder 1mo ago

> We're a good year+ into the use LLMs for all major bits of software that we all rely upon and GitHub here is down to one 9 of uptime. With what to show for it? If GH did 10x in volume/git commits, it's all LLM sloppy-pasta. Where's the 10x productivity? Where are the amazing apps?

crote 1mo ago
They have delivered some great shareholder value!

nullbio 1mo ago

It's caused by human laziness and corner cutting. LLMs can write buggy code or incomplete architectures as much as humans can, but standards have lowered. It's not the LLMs are not capable of also fixing these same issues, but that's additional work.

skinfaxi 1mo ago

> I don't think this portents anything great for software in general. I think it is a natural progression in technology. Failure rates were high for initial aircraft designs and safety improved over time, for instance.

jeltz 1mo ago
But in this case it is the other way round, failure rate is going up.

ransom1538 1mo ago

Now GITHUB gets to experience debugging the GHA in prod. From my experience, good luck.

mhitza 1mo ago

General outage was uncommon, but unicorns and octocat popping up for particular pages/repos was common enough for me in the past.

crabbone 1mo ago

I saw a graph of Github outages somewhere. The drop in uptime strongly correlates with the introduction of Github Actions. To be fair, it's a huge chunk of functionality, also, hard to make reliable. LLMs might have an effect, but the issues definitely started before LLMs were used for code generation, so, as much as I don't like the AI-generated code, I'd have to admit that it's probably not the cause here.