15 ms·
HN was down
- 1vuio0pswjnm7 6y agoIt did not seem to affect the Firebase feed.
- symisc_devel 6y agoHacker News is hosted at M5 and they are having a network outage: http://status.m5hosting.com/pages/incident/5407b8e2b002442510000272/604fa9a35aef05052e955afe http://status.m5hosting.com/pages/incident/5407b8e2b00244251... edit: Unrelated to the Azure outage.
- fotta 6y agoI'm surprised that a site as big as HN is only hosted in one place.
- cm2187 6y agoYou should look at stackoverflow's hosting!
- aspectmin 6y agoIs this described somewhere? :)
- cm2187 6y agoThe most recent resource I found. I think they basically use a rack in one datacentre. https://meta.stackexchange.com/questions/10369/which-tools-and-technologies-are-used-to-build-the-stack-exchange-network https://meta.stackexchange.com/questions/10369/which-tools-a...
- giantrobot 6y agoThis is the newest version of their architecture I've seen [0]. Compare to an overview from 2009 [1]. tl;dr StackOverflow's architecture is fairly simple and has done mostly vertical scaling (more powerful machines) and bare metal servers rather than virtual servers. They also realize their use patterns are read-heavy so there's a lot of caching and they take advantage of CDNs for static content which completely offloads that traffic off their main servers. [0] https://nickcraver.com/blog/2016/02/17/stack-overflow-the-architecture-2016-edition/ https://nickcraver.com/blog/2016/02/17/stack-overflow-the-ar... [1] http://highscalability.com/stack-overflow-architecture http://highscalability.com/stack-overflow-architecture
- Sahbak 6y agoSometimes, it pays off being extremely simple. In HN, it definitely does
- mromanuk 6y agoAfter this event, they should switch to two servers in different DC.
- centimeter 6y agoHaving two servers is a lot more than 2x as complicated and expensive as having 1 server.
- jpwgarrison 6y agoA wise colleague recently explained to me that if you build HA things HA from the start, it's only a little more than 2x. If you try to make an _existing_ system HA, it's 3x at best. HN is not a paid service, they can be down for a few hours per year, no problem. We're not all going to walk away in disgust.
- joshmanders 6y agoNot to mention the HNStatus[0] twitter account has so few tweets[1] I don't think it's even worth it. [0]: https://twitter.com/HNStatus https://twitter.com/HNStatus [1]: Last tweet since before today's incident was 2 years ago, 4 years ago since the one before that.
- johannes1234321 6y agoWhen going to two you need to handle split brain some way probably, otherwise you end up with an database state hard to merge, thus you better get three, so two can find consensus, or at least an external arbitration node, deciding on who is up. At that point you have lots of complexity ... while for HN being down for a bit isn't much of a (business) loss. For other sites that maths probably is different. (I assume they keep off-site backups and could recover from there fairly quickly)
- jsty 6y agoUntil 2018 at least it was ... wait for it ... a single server! https://news.ycombinator.com/item?id=18496344 https://news.ycombinator.com/item?id=18496344 (Anyone know if that's still the case?)
- dang 6y agoOne production server and one failover (in the same data center, obviously).
- betamaxthetape 6y agoI assume there are off-site backups, though? Asking as someone who was impacted by the OVH fire last week, and I didn't have recent backups and therefore lost data.
- Aperocky 6y agoHN is probably very small. Curious as to the minimum size of the backend that will hold up the website. There may need to be read replicas, but maybe not even that is needed.
- _joel 6y agoThey only have one server, iirc.
- voxadam 6y agoAnd, if I'm not mistaken, the site is single threaded.
- aspectmin 6y agoWould love to see the HN architecture.
- krapp 6y agoarclanguage.org hosts the current version of Arc Lisp, including an old version of the forum, but HN has made a lot of changes locally that they won't disclose for business reasons. There's an open source fork at https://github.com/arclanguage/anarki https://github.com/arclanguage/anarki, but it doesn't have any direct relationship with HN.
- mike_d 6y agoSingle threaded LISP application running on a single machine. Ta-da.
- dang 6y agoThe application is multi-threaded. But it runs over a green-thread language runtime, which maps everything to one OS thread. That's a significant distinction because if you swap the underlying implementation then the same application should magically become multithreaded, which is exactly the plan.
- deleted 6y ago
- mwcampbell 6y agoRunning on a single server is cheaper, and nobody loses money if HN is down (as far as I know), so it makes sense.
- nodesocket 6y agoNot sure why HN would still be a hosted at a 3rd tier provider. A few EC2 instances (multi-zone) behind a application load balancer should do the trick.
- colinmhayes 6y agoBecause it doesn't make any money
- heavyset_go 6y agoYC's companies get free advertising and job listings on the front page.
- mike_d 6y agoYou just described not making money.
- nodesocket 6y agoWho cares it’s a few hundred a month to host on AWS
- mike_d 6y agoThen where will people go to learn about AWS outages?
- dang 6y agoThey're first-tier to us.
- 1vuio0pswjnm7 6y agoHN is also available through Cloudflare but that seems to depend on M5. Don't take my word for it. Test it for yourself: printf 'GET / HTTP/1.1\r\nHost: news.ycombinator.com\r\nConnection: close\r\n\r\n' \ |openssl s_client -connect cloudflare.com:443 -ign_eof -servername news.ycombinator.com
- slig 6y agoCloudflare only proxies dynamic websites.
- robocat 6y agoCloudFlare will proxy whatever site you configure, even if it is static. Static websites will get the best speed boost from locally served assets (much reduced latency from the local POP) because the page itself can be cached (presuming headers on origin site are correctly set). Especially for page requests from international users.
- slig 6y agoSorry, I wasn't clear. What I meant was that since HN is dynamic its dynamic content is not usually cached. I mentioned specifically "dynamic" sites on my parent comment because as of this month Cloudflare can host static pages.
- dang 6y agoWe stopped using Cloudflare a few years ago. https://news.ycombinator.com/item?id=18188832 https://news.ycombinator.com/item?id=18188832 https://news.ycombinator.com/item?id=21799045 https://news.ycombinator.com/item?id=21799045
- ksec 6y ago>The desire to have fewer moving parts This actually got me thinking. Do we really need CDN? This is one of those thing we take and use without actually thinking whether we could do without it. Interesting thought experiment.
- tootie 6y agoWe had some AWS network issues today too...
- rattray 6y agoIs HN fully back? Looks like this was a little less than 3 hours total, is that right?
- bombcar 6y agoSo strange that this coincided with Azure authentication eating it.
- mikiem 6y agoUnrelated issues, but I did hear from our other clients that O365 was having issues at the same time as our network outage affected HN and many others.
- fabbari 6y agoIt's seems an odd coincidence of this and the Azure AD outage -- I was trying to get to HN to see what people were saying about it!
- PenguinCoder 6y agoDefinitely. My thought was "HN is hosted on Azure"? So I went looking into their hosting provider, and lo, they were down too. M5 might be Azure hosted... couldn't confirm that.
- deadmetheny 6y agoGood to see posting purely for karma isn't just a Reddit thing.
- yawnxyz 6y agomy fingers automatically just start typing in "news.y" when I'm idle, I definitely didn't know what to do when greeted with a 404! Is there any way to put the HN homepage on an edge cache so at least the homepage shows up? Or am I admitting that I'm addicted to checking HN too many times a day?
- h2odragon 6y agoAdmitting you have a problem is the first step to recovery, right? I'm sure I've heard that. Dunno how it's supposed to help.
- rossdavidh 6y agoYes, that's what you're admitting. :) Not that you're alone in that...
- breckinloggins 6y agoIt's gotten so bad for me that I'm down to just "n". I think I have a problem.
- slater 6y agoi was gonna say, check out that n00b that has to type all the way to "news.y" for the browser autocomplete! :D /s
- axaxs 6y agoLOL, same. In fact, every site I visit often is one char + enter in the browser. With the exception of W, being east of the Mississippi every station starts with W. That got me to thinking about 'first letter advantages.' If a site has a first letter not currently in use, I'm much more likely to visit it more often(mostly out of boredom, sure). V and X are still available if anyone is wondering. Zillow got Z!
- madjam002 6y agoOh man this hits home so much
- 6y ago
- alvatech 6y agoI think I have tried to visit HN for more than 10 times in last 2 hours and failed. This made me realize how much I'm addicted to HN
- jorl17 6y agoI keep 3 pinned tabs in my browser: - Reddit (my main source of addiction) - HackerNews (the second source of addiction) - Cookie Clicker (a rather recent addition that I'm slightly embarassed of) At a point in time I also had facebook, but I've since stopped going there (maybe once a week).
- ethbr0 6y agoJust wait until you find out Reddit and Cookie Clicker have the same endgame...
- willis936 6y agoJust cheat. It'll break the spell quickly. Also, check out universal paperclips if you haven't already. it has a definite end. You likely won't play more than maybe 10-20 hours.
- jorl17 6y agoI've played Universal Paperclips from start to finish 4 times! I loved it. In fact, I loved it so much the last time around that I wanted to have another game "somewhat like it" in the background -- that's where the recent Cookie Clicker tab came from. I always recommend Universal Paperclips to people who don't like cookie clicker games, because I fell in love with it the first time I tried it (heard of it from the Hello Internet podcast)
- mikewarot 6y agoI played Universal Paperclips, and converted a HectoVerse (100 Universes) to paperclips. Long covid sucks.
- StrictDabbler 6y ago
- 2bitencryption 6y agoAzure AAD also had an outage at this time - perhaps linked in some domino effect, or perhaps a coincidence? https://status.azure.com/en-us/status https://status.azure.com/en-us/status
- williesleg 6y agoLooks like a startup opportunity
- koolba 6y agoThe title should be updated to "Productivity was up".
- j_walter 6y agoNot sure that is true...trying to find other info as to why HN was down led to more productivity lost here
- bombcar 6y agoCan't sign into Azure Portal, let's check HN, oh that's down too, hmm is my internet up ... Huge rabbit hole
- deleted 6y ago[deleted]
- MattGaiser 6y agoWith Azure also going down, lots of people were probably scrambling to figure out what blew up.
- drusepth 6y agoHN is one of the few sites I always keep zoomed-in (around 200%), which led to me finding an interesting bug in Chrome while HN was down: Chrome's internal "This site can't be reached" page uses the zoom level of the site you would be visiting (if it were up), rather than Chrome's default zoom. Screenshot: https://i.imgur.com/VwFtgQh.png https://i.imgur.com/VwFtgQh.png
- RaketenStadt 6y agoThe font-size on HN is barely readable, I'm working on an accessible skin for the HN frontend that addresses this. I'm targeting WCAG 2.0. Keep an eye out for the "Show HN" coming soon!
- p1necone 6y agoAre you using a high dpi monitor but not using > 100% display scaling in your OS or something? It's roughly the same size as most other sites for me. (And pretty much all browsers have a zoom function for exactly this, it feels like a totally separate frontend would be more hassle to use than just ctrl + scroll wheel once)
- RaketenStadt 6y agoNo I'm not. The font-size for most text on the site is 10pt and 9pt. Zoom doesn't fix line lengths of 1500 characters and terrible color contrast. The link to the site guidelines is 7pt with a contrast that fails WCAG 2.0. No wonder no one reads them.
- account42 6y ago> Zoom doesn't fix line lengths of 1500 characters That depends on your browsers zoom implementation. Firefox is able to zoom the text/element sizes while keeping the page width the same on HN.
- TylerE 6y agoOn Win10 with default settings, fonts on most sites are totally comfortable to me. HN is readable - just - but it's definitely on the small side. The complete lack of some sort of horizontal constraint doesn't help either. 200 character lines are no bueno for reading.
- aritmo 6y agoThere was a noticeable increase in productivity during the last hour or so.
- ibraheemdev 6y agoIt says a lot that @hnstatus has not tweeted since 2018.
- dangwu 6y ago@HNStatus tweeted about the outage 3 hours ago.
- TheRealNGenius 6y agoThat's the point...
- dangwu 6y agoAh, my bad. I was wrongly interpreting OP’s comment.
- hnrodey 6y agoHad me quite confused because I'm also having home internet issues. I was trying to get my laptop to switch to my mobile hotspot and HN is one of the sites I used to test connectivity because a) it's almost always available and b) loads very quick. A bit of a mindfuck trying to assess my actual internet connectivity via a site that was also down : )_
- divbzero 6y agoDitto. HN is so reliable and light on JavaScript that I typically use it to test my connection. I thought my connection was down earlier but guess this was the rare case where it was HN. (Other comments suggest it was a network outage at M5 where HN is hosted.)
- blakehaswell 6y agoMe too. I was trying to browse HN on my phone earlier and my first instinct was that my WiFi was having a moment. It's a testament to how reliable HN is.
- k__ 6y agoI was trying to read some news while training in the basement, where I don't have very good Wi-Fi. Usually HN is one of the pages that work better down there, haha.
- aasasd 6y agoThe common method of testing connectivity is opening Bing. Because it's guaranteed to not be cached in the browser.
- nhylated 6y agoFound some use for Bing!
- beaconstudios 6y agoHa, I do this with yahoo.com. I never have a reason to visit it otherwise.
- coding123 6y agoI never expect HN to be down... I asked my wife - hey is the internet down? She said - no, it's working for me. I clicked on another site and my mouth dropped.
- PhilosAccnting 6y agoI have a massive learning project[1], and I think 2/3 of my "to get through as soon as sensible" content is news.ycombinator links. Needless to say, this site is my own personal StackOverflow, and I think there's something about ingratitude bouncing around in my mind somewhere. [1]https://github.com/PhilosAccounting/ts-learning https://github.com/PhilosAccounting/ts-learning
- bfostbfostbfost 6y agoWow, looks like a wealth of knowledge. Forked it for myself, only for reference, hope that is ok. Just seems like a ton of great info that I’d love to comb through myself. Cheers.
- PhilosAccnting 6y agoTotally okay, though I added the PDFs/videos to gitignore. I'm mildly paranoid about IPs[1]! [1]https://gainedin.site/ip/ https://gainedin.site/ip/
- bfostbfostbfost 6y agoMakes sense, well I will be following the TechSplained project, good luck!
- deleted 6y ago[deleted]
- deleted 6y ago[deleted]
- tempestn 6y agoIs this related to the big Microsoft outage?
- JasonFruit 6y agoI thought pg posted "Memphis".
- spondyl 6y agoI didn't notice unfortunately due to the Azure outage blowing everything up :(
- southerntofu 6y agoWho needs so many 9's when there's actually interesting content we keep coming back for?
- ibraheemdev 6y ago> Back now. Got to write some code for a change... - 5:39 https://twitter.com/HNStatus/status/1371576822748487683 https://twitter.com/HNStatus/status/1371576822748487683
- tartoran 6y agoSeeing HN unresolved was a bit weird as it is the best performing site I ever visit on my low bandwidth phone so several times I thought the problem was on my end. But in the end it helped me realize how frequently I dial into HN while it was down. I have a bit of a problem and I think I need to turn on that no procrastination flag on.
- vincentmarle 6y agoWe also had issues with our YC application earlier today, was that related to this issue?
- tpowell 6y agoI've been on this site 12+ years, and I don't ever remember it being down. I assumed we were under nuclear attack.
- protomikron 6y agoCurious, what is the uptime of HN - is there some data about that? My guess is around 99.9% ... but maybe that's too optimistic?
- deleted 6y ago[deleted]
- simonebrunozzi 6y agoWhy too optimistic? Probably closer to 4 9s. With this outage of ~2 hours, we are at ~99.97% for this year. (I am not aware of any other downtime during 2021) Rule of thumb (I strongly prefer minutes/year instead of 9s, to get an immediate sense of how good the availability is): 99.9% : down for 525 minutes / year, or roughly ~10 hours 99.99% : down for 52 minutes / year, or roughly ~1 hour 99.999% : down for 5 minutes / year
- protomikron 6y agoYes, but it would be nice to read some "official" numbers backed by HN's monitoring (although I'm on HN quite/too often, I would not notice every downtime).
- tomxor 6y agoFunny, my first thought was "oh no they've blacklisted VPNs", can't remember when HN was ever down!
- kenm47 6y agoIt's 3pm.... do you know where your servers are?
- enobrev 6y agoThe one time I'm actually reading HN for actually relevant information for actual work, it's down for half a day. Made for a great excuse to take a nap.
- fotta 6y agoSomething that I learned from this is that HN has a status Twitter. Rarely used though, which is a testament to the team. https://twitter.com/HNStatus/status/1371525940656803848?s=20 https://twitter.com/HNStatus/status/1371525940656803848?s=20
- EasyTiger_ 6y agoOnly they never posted anything during the outage there
- mikiem 6y agoFounder and CEO of M5 Hosting here. We did have a network outage today that affected Hacker News. As with any outage, we will do an RCA and we will learn and improve as a result. I'm a big fan of HN and YC in general, we host of other YC alum, and I have taken a few things through YC Startup School. During this incident, I spoke to YC personally when they called this morning.
- nas 6y agoWe have been using M5 Hosting for one of our servers since 2011. They have been extremely reliable up until today. Based on what was posted about the Hacker News server setup, we have something similar. We have a "warm spare" server in a different data center. We use Debian, not FreeBSD. We are in the process of slowly moving to a distributed system (distributed DB) that is going to make fallover easier. However, that kind of setup is orders of magnitudes more complex than the current (manual fallover) setup. I really wonder if the planned design is going to be more reliable in practice. Complexity is almost always a bad idea, in my experience. Distributed systems are just fundamentally very complicated.
- mikiem 6y agoOh hi! Thank you for the kind words. I cant tell who you are by your name here, but if you've been with us since 2011, we have certainly spoken. Are you using our second San Diego data center for your failover location? If you and I aren't already talking directly, ask to speak with Mike in your ticket.
- PaybackTony 6y agoI had used M5 some years ago to host an online rent payment / property management app. Have nothing but positive things to say about that experience. We once had an outage that was our own fault on our single server and they had someone go in, in the middle of the night, to reboot it for us and we weren't even on an SLA.
- mikiem 6y agoThank you for sharing your positive experience! We can power cycle power outlets remotely and can connect a console (ip kvm)... and we are staffed 24x7.... in case you need another server. Thanks again!
- deepsun 6y agoIf only they used Kubernetes! /s
- 0xbadcafebee 6y agoSince I couldn't get to HN, I wrote up how to make the site resilient to outages: https://gist.github.com/peterwwillis/ce2bfaba7fc72e4af44c28135ab3db1e https://gist.github.com/peterwwillis/ce2bfaba7fc72e4af44c281... tl;dr 1 server x 2 providers, different regions, replicate content
- comprev 6y agoCan't decide if the Gist is tongue-in-cheek or actually serious...
- nickthemagicman 6y agoThis site is so reliable, I thought my I.P. had gotten banned.
- peanut_worm 6y agoAnd now it looks like there is an outage at reddit
- tiffanyh 6y ago@dang Would love an updated post on what the current hardware / software stack that’s running HN. It’s been years since I’ve seen a post/comment on this topic. Are you still running FreeBSD, on a few high frequency cores (iirc)?
- vulcan01 6y agoHe commented on this a couple hours ago: https://news.ycombinator.com/item?id=26469566 https://news.ycombinator.com/item?id=26469566
- eternalban 6y agoTIL HN uses "mirrored magnetic for logs (UFS)". Is there a privacy policy posted anywhere? What's in these logs? Magnetic is for long term storage. How far back does it go?
- coder543 6y agoAt the bottom of (almost) every HN page is a link titled "Legal" which contains the privacy policy: https://www.ycombinator.com/legal/ https://www.ycombinator.com/legal/ (I don't have any relationship to HN)
- eternalban 6y agoThanks! The section on ("HN Information") - does this include e.g. IP address under "any submissions or comments that you have publicly posted to the Hacker News site"? My naive reading of that would say "no". But is that correct? "If you create a Hacker News profile, we may collect your username (please note that references to your username in this Privacy Policy include your Hacker News ID or another username that you are permitted to create in connection with the Site, depending on the circumstances), password, email address (only if you choose to provide it), the date you created your account, your karma (HN points accumulated by your account in response to submissions and comments you post), any information you choose to provide in the “about” field, and any submissions or comments that you have publicly posted to the Hacker News site (“HN Information”)."
- Koshkin 6y agoIt’s OK. The skies haven’t fallen. (And I was able to accomplish something today.)
- technick 6y agoHN is my homepage and when it wouldn't load, I told my coworkers the internet was down and took a 3 hour lunch.