9 ms·
GitHub incident 2022-03-23
- deleted 4y ago[deleted]
- jonnybarnes 4y ago2nd day in a row isn’t it?
- momothereal 4y agoYes: https://news.ycombinator.com/item?id=30767635 https://news.ycombinator.com/item?id=30767635
- fishnchips 4y agoYesterday they had two.
- stepri 4y agoAnd 6 days ago: https://news.ycombinator.com/item?id=30711269 https://news.ycombinator.com/item?id=30711269
- rvz 4y agoIt is. 24 hours later [0] and I only expected it to happen once every month. Looks like it is getting worse. Oh dear. Not a good idea to go 'all in' on GitHub. [0] https://news.ycombinator.com/item?id=30767821 https://news.ycombinator.com/item?id=30767821
- max23_ 4y agoLooks like the same services that were affected in yesterday incident.
- bob1029 4y agoWe are scheduling a call with an enterprise sales person next week. If I can get all the Github features I had as of ~2020, but on an instance that wont get hit by the public cloud/update bus, I would be exceptionally happy. The only complaints we have are regarding availability. If we can fix that one problem, this is a perfect product in our view.
- andruby 4y agoHow do you evalute running your own gitlab instance?
- paskozdilar 4y ago
- etimberg 4y agoThe quality of GH seems to be slipping
- Trasmatta 4y agoI've actually been pretty impressed with the quality of the product and new features over the past couple of years, but it seems to be having a lot of stability issues recently.
- etimberg 4y agoI've liked the new features too, especially after so many years of not many features. Maybe they've moved too fast now
- xtracto 4y agoFunny that it happened since they were acquired by Microsoft... reminds me of Hotmail, Skype, LinkedIn, Rare, among several others.
- deleted 4y ago[deleted]
- deleted 4y ago[deleted]
- amelius 4y agoI hope it doesn't affect security ...
- iBotPeaches 4y agoIt seems like we haven't had a non-robot status update on the status page in days since this what seems like daily occurrence. I figure at this point we'd get something of why this is happening. I also don't appreciate our builds freezing, unable to be cancelled and then eating up hundreds of minutes.
- lucasyvas 4y agoBilling should always be built on a "ping" IMO and not start/stop hooks. The latter is shockingly bad for customers during times of unreliability. The former sounds stupid and requires more infrastructure from the one offering the service, but I think it's more fair. I haven't used GA in a way where it actually costed me anything, but having minutes just tick away while you can't do anything is really stupid if that's the case. Edit: Another sane solution would probably be to record outage periods and have Billing automatically reconcile for every customer when invoicing. This would require them to admit the outage durations however, so it may be flawed from a human perspective.
- drusepth 4y agoThe "ping" solution is an interesting one that I haven't seen proposed before. At what rate would you do these pings? I don't know how upgrading/downgrading works at GitHub but if they do any sort of refund/credit when you downgrade, it seems like there's some interesting implications for abusing the system (e.g. upgrading/downgrading between pings for "free" service if the time between them is too long) versus performance (e.g. how do you update all users per ping in a timely manner if the time between them is too short?). Would love to read up more on this approach; seems interesting!
- easton 4y agoDo they give you the minutes back if there's an incident during the period where a job is running?
- no_wizard 4y agoYou will have to contact them for them to credit you, that's what we did
- cube2222 4y agoLooks like they really want to get a PR deployed, but there's still not enough duct tape on it.
- Xarodon 4y agoThis has been a pretty rough week for GitHub
- stuff4ben 4y agoGithub Enterprise hasn't been faring too well at my work either this week. When you work on both open and closed source products and GH and GHE are both down, it leads to a very unproductive week.
- jrowley 4y agoDoes GitHub enterprise result in dedicated instance or any better availability?
- jon-wood 4y agoIt Depends. GitHub Enterprise is confusingly both a "call us for pricing" tier of GitHub the website, and also an on-premise version of GitHub that you can run as an appliance in your own data centre. The first of those is ultimately just GitHub and so has the same outages, the second is running on your own hardware so (shouldn't be) tied to the website's availability.
- bewuethr 4y agoThere are multiple products: self-hosted (Enterprise Server) and hosted by GitHub (Enterprise Cloud). I don't know about uptime guarantees, but you can buy Premium or Premium Plus support with 30-minute SLA or a dedicated account manager.
- nimbius 4y agohttps://www.githubstatus.com/history https://www.githubstatus.com/history 21 incident outages in just 3 months. At this rate the benefits of running your own gitea or gitlab are starting to become competitive.
- mrkurt 4y agoIf you want companies to be honest on their status pages (I do!), you can't just count incidents like that. Status pages can be an amazing place to communicate all kinds of problems. Most issues have a relatively narrow impact, but the impacted people _still_ benefit from seeing them listed.
- jmartens 4y agoHow can we solve this as customers, or push the vendor to do better?
- mrkurt 4y agoUse vendors who do a good job communicating status, basically. I don't think you can change AWS behavior. But if you find a hosting company who does an amazing job with their status updates, put some apps there (_my_ company does an ok job with status page updates, we're getting better, it's not amazing yet).
- encryptluks2 4y agoStop being a customer of crappy vendors
- copperx 4y agoWhat cloud provider does better status pages than AWS?
- drusepth 4y agoThe snarky answer is "literally all of them", but one real answer is that I've been pretty happy with GCP's status reporting for the past year-ish I've used them. I've only noticed a few incidents, but every time I've checked the status it was already updated. They also occasionally provide workarounds on the live incident pages if you need to be back up before the issue is fixed on their end.
- rvz 4y agoAgain? Last time that happened was 24 hours ago? [0] It is really getting unreliably bad. Like I said before, having a self-hosted backup seems to make more sense. [0] https://news.ycombinator.com/item?id=30767821 https://news.ycombinator.com/item?id=30767821
- blueplanet200 4y agoI hope they figure out what’s going on every morning. Heard from inside they don’t know why the db dies everyday but restarting it fixes it.
- cube00 4y agoBreak out the early morning restart cron job.
- Kostic 4y agoEarly morning in which timezone?
- afterburner 4y agoGaryOldman.gif
- glenneroo 4y agoWhen the least amount of users are online?
- gaoshan 4y agoHere you go, Github: 0 4 * * * /etc/init.d/postgresql restart I'll take an architect position as compensation, but only if there is equity.
- exikyut 4y agoWhat's "the db"? It sounds like something of small to medium scale if you can just restart it like that. In any case, why not just relocate some vendor engineers on site for a bit? Or, better, why does the vendor not have a small presence in the corner? Sounds like whatever "the db" is it's probably some (objectively) small but very scary thing that's currently on fire and people are trying to figure out how to put it out without crashing the plane and also making too many waves internally, which is probably even harder. So asking about making vendor noises is (as useful as it may be) probably going down the wrong path - in much the same way this is probably not related to the outages (it may well be, but from the outside it's all coincidence anyway).
- einpoklum 4y agoThe page at the link is not much more informative than the link itself :-(
- okareaman 4y agoWhat's the difference between GitHub and GrubHub? GrubHub delivers
- intunderflow 4y agoWith how often these happen we might as well sticky this thread for the next one
- koolba 4y agoI really wish they would add the word “outage” to these titles. “Incident” alone makes me think something got hacked or leaked.
- arez 4y agoThat's SRE lingo --> https://sre.google/sre-book/managing-incidents/ https://sre.google/sre-book/managing-incidents/
- zufallsheld 4y agoIt's also itil lingo, which predates sre.
- mtnops 4y agoIt's NIMS - FEMA lingo, which predates ITIL. Which was developed in USFS wildland firefighting, which predates FEMA. It's incident management all the way down.
- zacharynewton 4y ago"The Simpsons already did it"
- toastal 4y agoAnd to think Git can easily be decentralized. I wonder if the community could fork GitHub to fix it. Oh, it's not open source. Devs must be too busy working on more 'social' features like "For You (Beta)" to milk the attention economy.
- higeorge13 4y agoThe usual services (actions) again down around the same time. This is embarrassing.
- grumple 4y agoAgain?! Jeez. I wish I had customers this tolerant.
- frjalex 4y agoLooking at the "GitHub" prefix in the title, I was half-expecting this to point to a report explaining the outage a week ago... But rest assured, it is a new outage!
- teekert 4y agoOh I thought it was about the one from yesterday :)
- aaaaaaaaata 4y agoAre their CI/CD toys that shiny that people still willingly choose them even with all the issues? I find myself regularly asking this — about every major SaaS used for critical ops stuff like this.
- teekert 4y agoWork choose GitHub (we are a MicroSoft shop), I have to say, I like GitHub a lot. The disruptions have been annoying sometimes, that's true. But due to the nature of Git I could always just keep working.
- deleted 4y ago[deleted]
- annexrichmond 4y agoI thought it was going to be a Postmortem. I couldn't have been more wrong!
- mirekrusin 4y agoWhat's the best crowdsourced status monitor?
- eckza 4y agohttps://outage.bingo/ https://outage.bingo/
- mirekrusin 4y ago+1 :)
- mirekrusin 4y agoStatus page says only degraded performance. It's a nice way of putting it. I'm trying to run github action for couple of hours now. They don't work at all. But apparently this means they run, but in infinite time, hence == degraded performance, nice.
- raffraffraff 4y agoIt's just a way to avoid SLA breaches. "Of course it wasn't down! It was just infinitely slow!"
- eatonphil 4y agoGithub Actions are back for me now.
- mfashby 4y agoI'm inclined to look at tools like fossil again, for it's distributed issue tracking and wiki capability https://fossil-scm.org/home/doc/trunk/www/index.wiki https://fossil-scm.org/home/doc/trunk/www/index.wiki
- edgyquant 4y agoI had forgotten about that, thanks!
- JonChesterfield 4y agoFossil is faultless for a team size of one. I've been using it for nearly a decade, doing totally non-optimal things like using versions released years apart on different OSs with the same database. I also ctrl-c it when I spot a typo in a commit message and check in binaries. Never missed a beat. As headcount goes up I think the inability to locally rewrite history into easily reviewable patches would be sorely missed. So it's git for team stuff and fossil for my own.