40 ms·
Incident with Issues and Pull Requests
- gabrielizaias 3y agoHere we go again…
- melx 3y agoShout out to all devs that put the deps of their programs on GH... Right now I can ignore that my PRs show 500 error, or old code best case scenario. But... I cannot build and ship due to project's dependency depending on some stuff hosted on GH.
- mminer237 3y agogit says my code pushed fine and everything is synced, but Github is not showing any of my changes. Edit: 10 minutes later, the Github finally shows the push, but triggers still aren't working. Edit #2: Things are working normally now.
- hideoncode 3y ago[flagged]
- dbingham 3y agoWhat is going on over there? Third day in a row is... kind of impressive.
- frde 3y agoMy money is on some significant backend architecture migration gone wrong without a viable way to roll back the time machine :)
- lallysingh 3y agoAzure?
- capableweb 3y agoFeels like an organization as big as Microsoft must surely have some sort contingency plan in place before doing such a large migration. Right?
- isbvhodnvemrwvn 3y agoSales handing out Office365 discounts and trying to convince people that AWS and GCP is going to steal their data, judging by companies I worked for that used Azure.
- hardware2win 3y agoWasnt it true? Thas Amazon abused their AWS position and stole their competitors data, so thats why Germany's retail businesses are building their own Clouds
- jacooper 3y agoAny links? Interested on reading more about this
- belmont_sup 3y agoHere’s one https://www.wsj.com/articles/amazon-scooped-up-data-from-its-own-sellers-to-launch-competing-products-11587650015 https://www.wsj.com/articles/amazon-scooped-up-data-from-its... The gist is that yes it’s true. They’d come out with their own Amazon Basics branded stuff and push it to the top.
- MattIPv4 3y agoLoading github.com is returning a 500 for me currently, so seems like more than just issues/pull requests. Also seeing actions fail with 500s on assorted steps.
- zachallaun 3y agoSimilar issues for me. I can load github.com and my profile, but visiting a repository (or trying to git pull a repo with the https origin) returns a 500.
- jamespetercook 3y agoThird day in a row this has affected my productivity
- nickthesick 3y agoHave they said what has been up lately?
- Vermyndax 3y agoThey are far overdue for issuing a statement.
- capableweb 3y agoThey're having so many outages that they've moved from making statements after each event to aggregating them all and write about all of them on a month-by-month basis. I'm not joking: https://github.blog/tag/github-availability-report/ https://github.blog/tag/github-availability-report/
- KomoD 3y agoThey've had that for years.
- capableweb 3y agoI don't remember when it started getting worse, but if I do a guess, I'd say around 2018 sometime, which I'd also guess is when they started pushing more changes too.
- mostafah 3y agoIt started for me a few minutes before the status page showed something. Which is understandable, of course. But strange that it keeps happening almost every day now.
- longwave 3y agoYeah, I was getting 500s for about three minutes before they posted the status update. I guess it's good that they at least update the status page in a timely fashion, but the third day in a row of downtime is not exactly good service.
- zomglings 3y agoHave to give it to them for how useful their status page is. Other products we use play all kinds of word games to downplay issues so they don't have to show them on the status page, which is extremely annoying.
- darrenkopp 3y agoI can confirm this as well. I started seeing 500 errors intermittently when trying to view pages, so I checked status page and saw everything was green. Status page started showing the incident within about 3 minutes of when I started seeing issues. Clearly that's all based on happenstance of when I was landing on GitHub's website, but I have found that of all the status page's by large companies, GitHub's is almost always showing an incident as soon as I start noticing issues myself.
- rhymeswithjazz 3y agoAnother day, another outage. I'm getting 500 for any repo.
- bhouston 3y agoYup, hitting me and all over twitter: https://twitter.com/search?q=github https://twitter.com/search?q=github
- daniaal 3y agoTrying to access repos is returning 500 for me also
- megadopechos 3y agoVery helpful 500 page: "In the meantime, try refreshing."
- talboren 3y agoIs it time to move back to Jenkins? :X
- dboreham 3y agoNow that gitea supports act_runner (and hence a reasonably usable clone of GH actions), Jenkins is dead.
- deleted 3y ago[deleted]
- justinclift 3y agoOh, that's really good news. :)
- michaelmure 3y agoDaily reminder that https://github.com/MichaelMure/git-bug https://github.com/MichaelMure/git-bug could use some help :-D
- isaacdl 3y agoUnfortunately, I can't see what this is because GitHub is down.
- michaelmure 3y ago... of course. Mirror there: https://gitlab.com/MichaelMure/git-bug https://gitlab.com/MichaelMure/git-bug
- joostlek 3y agoI was greeted with a "Please give us this UUID when you report this bug" twice and I thought it was a Github breaking repo. love it
- greenie_beans 3y agothis is the third day in a row this is a problem yet they're framing it as a new outage.
- bluehatbrit 3y agoI'd say it's reasonable to list it as separate outages on the status page as it's really a representation of "is github available and working as expected". Even if it is the same issue, when they manage to mitigate it (or it goes away) I'd want to see that everything is now available from a user perspective. That said, they're getting to the point where they really need to make some larger post about this. It seems reasonable to assume it is all from one root cause.
- zomglings 3y agoNot just issues and PRs - I had to try multiple times before I was able to successfully push code to a repo over SSH. This is the error I was seeing: ERROR: fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists.
- timvdalen 3y agoYeah, same. I've since been able to push my branch, but now I can't open a PR.
- candiddevmike 3y agoEveryone thinking they've been laid off and notified via git push. Just GitHub being GitHub.
- acyou 3y agoDoes anyone host their own git repos in an enterprise environment? How do you do it and what are some good resources for learning how to do this?
- tommy_axle 3y agoYes, we do this using https://gitea.io/en-us/ https://gitea.io/en-us/ on a private server (firewall, backups and a replica) for most projects. Github is only used when it's required by a stakeholder.
- cube2222 3y agoYou can just use GitHub Enterprise if you want GitHub, but self-hosted.
- trollied 3y agoYes, gitlab. Self-hosted, behind a VPN.
- Kelteseth 3y agoThis. Rock solid for the last 4 years.
- aeyes 3y agoYes using gitolite, it just works. In the past I worked at a company which used the commercial solution from JFrog, I don't remember ever having problems with git availability as a user.
- c16 3y agoThe temptation to create `isgithubup.com` and return "surprisingly yes." on the rare occasion it actually is up.
- ta1243 3y agohost it on githubpages
- atl4s 3y agoJust lost a merge commit to dev/null. This is getting tiresome
- capableweb 3y agoHow??? You do the merge, which either creates a new commit for the change, or appends the commits to your existing tree. Then you push that to the remote. If the push fails, you can just push again, it's not lost. And if the merge failed, you didn't have any merge commit to begin with.
- misnome 3y agoThere used to be a pretty consistent bug that if an on-site PR merge failed but you clicked "Retry", that it just did a basic non-squash full-merge discarding all your commit message work, often requiring a revert to tidy things up. It could be similar to that.
- edgyquant 3y agoMay have been using the GitHub interface
- snapcaster 3y agoYou can do a lot of operations on the github web UI nowadays. Could have been that
- deleted 3y ago[deleted]
- SettembreNero 3y agoand i thought i broke `master`...
- yeck 3y agoI wasn't even able to get to repos or users/orgs for a while (though as I write this it seems like that is coming back).
- remorses 3y agoI cannot even push code
- cloudking 3y agoAnecdotal - I've been using Gitlab for a few years on some projects, and haven't experienced any downtime issues with them of this magnitude.
- goodoldneon 3y agoSame. Overall, I like GitLab a lot more than GitHub. I wonder how much of GitHub's popularity is buoyed by its status as the defacto home for OSS
- cloudking 3y agoIt has good parity with GitHub on core functionality and a generous free tier.
- jgadelange 3y agoI can still remember https://about.gitlab.com/blog/2017/02/10/postmortem-of-database-outage-of-january-31/ https://about.gitlab.com/blog/2017/02/10/postmortem-of-datab... (though I can't rembember reading of any big problems since then)
- dbingham 3y agoYeah, Gitlab has had its share of downtime. Github has had shockingly little until the last year or two. These things come and go. Of course, if this continues happening it may be time for the OSS community seriously consider migrating en masse. [Edit to fix grammar - thanks for the corrections!]
- tnorthcutt 3y agoFWIW the typical phrase is "en masse" (which is French). The English translation is "in mass".
- CharlesW 3y agoThe English translation would be something like, “all together, at the same time”. “In mass” isn’t an English idiom.
- VWWHFSfQ 3y ago> Codespaces is experiencing degraded performance. We are continuing to investigate. Imagine not only not being able to push your code, but also not even being able to _write your code_ at all. And so many orgs rely on Actions to even be able to deploy. Geez. I personally believe that the cloud sucks.
- gabrielgio 3y agoIf you're putting all your eggs in one basket you are the only one to blame I guess. It's sill weird to me how many and how much companies relly on Github infra.
- booleanbetrayal 3y ago500's on every repo page + CLI errors on pull / push. Completely useless at the moment.
- Vasniktel 3y agoSo nice to meet you all here again, gents - this is becoming a regular thing.
- buglungtung 3y agoI'm considering host a gitea instance backup all of my repos. I have an important fix that need to be deployed right now but there is no way to deploy it in a normal way with our CI which one was setup with Github Action. Fortunately I have a instruction to bypass CI and build the source by myself. But again, Github defeat me because our release workflows are depend on GitOps which are effected by Github issue. Ahhhhhhhhhh I have to build the docker image, push it to ECR then update a YAML template to make EKS apply the new changes It's 9PM in my timezone and I'm waiting for my patches are up. A frustrating incident
- galleywest200 3y agoGitea's ability to create a local repository as mirror of a remote repository is great for this. You can stay on Github and have your code regularly mirrored locally.
- jmkni 3y agoIf only Git had this sort of functionality built in…
- silverwind 3y agoPretty sure git can not continuously pull/push the mirrored repo like this feature does, by default every 10 minutes.
- edgyquant 3y agoThis is a one line cron job
- ijustlovemath 3y agoJust a mounted drive over sshfs...
- dboreham 3y ago
- mr90210 3y agoOn the bright side, maybe, just maybe the open source community realises that such centralisation might not be the best solution for hosting code.
- samwillis 3y agoHugs to the GitHub Opps and SRE teams right now! Also hugs to any Devs, Opps or SREs directly effected by this outside GitHub. Looking forward to a post-mortem on the last few days, I'm sure it will be a really interesting read.
- martiuk 3y agoAnyone experienced with asking for service credits for enterprise customers? It looks like you have to prove that they have missed their SLAs
- intunderflow 3y agoGitHub outages are very reliable, as I live in Europe and they always come in the afternoon they're a great reminder to go get lunch It's a feature, not a bug!
- willsmith72 3y agoHow i miss the euro lifestyle
- agos 3y agoin CET they're more around afternoon coffee break time
- est 3y agoThis gives me some the fail-whale vibes. But the blue bird didn't fail like this these days, strangely.
- melx 3y agoI'm going to self-host my git repos. Any recommendations? The git+nginx would suffice but it does not offer GUI. I need one to see the changes proposed (aka PRs). Gitea is nice, but a bit overkill for my needs. I don't need CI, files hosting, issues, team members, releases, wiki, forking/watching/staring, etc.
- dboreham 3y agoimho Gitea is so smooth to deploy and manage that it's worthwhile even if you don't need its advanced features.
- justinclift 3y agoGitea is pretty light on resources so even though it has a extra functionality you don't need it's not really a resource pig. Unlike say, GitLab.
- melx 3y agoGitea requires a database which is unwanted feature for "git server" in my little world. I'm just looking for "website" (read: interface) that list files over HTTPS, with the ability to show nice looking diffs. Some sort of ssh keys(?) to prevent unauthorized access etc.
- justinclift 3y agoMaybe something like gitolite then? https://github.com/sitaramc/gitolite https://github.com/sitaramc/gitolite I've not used it personally though, so no idea how well it works in practise.
- melx 3y agoThanks! Gitolite doesn't offer any UI for what I see. Also to set it up is too complicated. I just found about "git instaweb" (developed by Git authors, but it's extra OS package) - it works locally, probably what I was looking for (minus the 100% width page layout). BTW There's Gitblit is someone's fancy hosting Java app.
- dugmartin 3y agoI've had some actions queued for multiple days now on certain repos, but not others. I've cancelled them and restarted them during the green status intervals but they all go back to "Queued". I've also cancelled them and then made slight documentation tweaks to get new commit hashes on the branches and it still goes to queued.
- WesolyKubeczek 3y agoI feel that with dynamic like this, someone could make a page showing the number of days since the last github incident. It would show a very prominent zero and be a static page with no logic whatsoever.
- gtrax 3y agoWe are back to Windows 95, which reliably crashed after 49.7 days: https://www.cnet.com/culture/windows-may-crash-after-49-7-days/ https://www.cnet.com/culture/windows-may-crash-after-49-7-da...
- ryandvm 3y agoMaybe they should put CoPilot in charge of ops...
- talboren 3y agoBack to biz
- rvz 3y agoThree days in a row of outages, in less than a week of unreliability after yesterday's downtime of GitHub Actions [0]. Really at this point, you just might as well consider self hosting and it is looking very chronic with GitHub falling apart and self-hosting was indeed the sensible idea just like how the other open source projects have done for years. GitHub is going just great, and centralizing everything to GitHub really was a good idea wasn't it? [1] /s [0] https://news.ycombinator.com/item?id=35887029 https://news.ycombinator.com/item?id=35887029 [1] https://news.ycombinator.com/item?id=22867803 https://news.ycombinator.com/item?id=22867803
- ta1243 3y agoThere's no way a company could run its own source management system and have an uptime approaching 23 hours a day -- that's like one nine of uptime! (/s of course)
- moltar 3y agoThee days in a row!! Is this the result of all the job cuts?