8 ms·
Stack Overflow is a cacheless, 9-server on-prem monolith
- didntreadarticl 4y agoAnd runs on .NET One of the only well known sites to do so, I think?
- profile53 4y agoI think most things Microsoft run on .net incl. parts of bing and office online.
- didntreadarticl 4y agoah yes, of course
- mytailorisrich 4y agoJoel Spolsky used to work for Microsoft and all his products were developed using the MS ecosystem, I believe.
- tyingq 4y agoNot caching the questions and answers makes sense to me, as I imagine the hit rate wouldn't be terribly good. I would guess, though, that they somehow cache things like the sidebar list of blog articles, featured items, "Hot Network Questions", etc.
- banana_giraffe 4y agoThey do in fact cache some things like that, they've had caching issues in the past (and again recently, I think) with the wrong cache being used in some situations: https://meta.stackexchange.com/a/235277 https://meta.stackexchange.com/a/235277
- tony-allan 4y agoIn the diagram [1], I can see why you might design it that way if starting from scratch but it works as is so why change it. Is there a particular reason to suggest a change to the architecture? [1] https://twitter.com/sahnlam/status/1629713954225405952/photo/1 https://twitter.com/sahnlam/status/1629713954225405952/photo...
- default-kramer 4y agoThe word "should" might be confusing here. I didn't read it as the author recommending a change; rather the author first proposes "Given what I know about Stack Overflow, they must be doing something like this, right?" Then boom comes the surprising revelation.
- borland 4y agoDiagram 1 has the comment "What I think it should be". It's easy to interpret that as "stackoverflow should change to be like this", but I think it was meant to be more like "If I had to guess how stackoverflow works, this is what I think it would look like". It's amazing how much performance and scalability you can get out of computers, if you don't burden them with 100x overhead caused by shoveling data between microservices all the time :-)
- quanticle 4y agoIt's easy to interpret that as "stackoverflow should change to be like this", but I think it was meant to be more like "If I had to guess how stackoverflow works, this is what I think it would look like". That's not a better interpretation. It says something (something not good) about the mindset of modern software engineers that the first thing they think of when they look at a website like StackOverflow is a n-layer microservice architecture, with more moving components than a Swiss chronometer.
- asalahli 4y agoIt is absolutely a better interpretation. The former signals arrogance while the latter shows that OP accepts the inferiority of their guess in comparison to the actual architecture.
- selcuka 4y agoIt is ironic that many questions on Stack Overflow are about various cloud services, hyped-up technologies, and problems caused by over-engineering.
- oconnore 4y ago"The medium is the message" wins again.
- didntreadarticl 4y agovarious cloud services This question does not appear to be about programming, Closed. hyped-up technologies subjective, Closed problems caused by over-engineering Opinion-based, Closed.
- motoxpro 4y agoI know you were joking but I am so glad stuff like that is not on SO. It would look like Quora which is the scourge of the internet.
- rhaway84773 4y agoThis! Quora is a great example of what SO would be if it wasn’t for their moderation. That being said, I think most would agree it might have gone too far and there may be value in trying to tweak it a bit to make it easier to contribute.
- docandrew 4y agoI’m always puzzled when I’m using SO to help diagnose some obscure problem in my tech stack and I see a bunch of “hot questions” in the sidebar about whether dwarf armor can deflect magic bullets, or what the energy capacity of a Stormtrooper’s laser rifle is, etc.
- docandrew 4y agoNot knocking any interest, just really curious that there is such a wide range of topics on there.
- yamrzou 4y agoIs it hosted on the cloud?
- didntreadarticl 4y agoNope, on-prem https://twitter.com/alexcwatt/status/1544876135711916035?lang=en https://twitter.com/alexcwatt/status/1544876135711916035?lan...
- atonse 4y agoEven though I love their simplicity as an example of how to be pragmatic and not over-engineer, do remember that they’ve tuned their code to the point that they built an ORM that is one of the fastest in the NET world. I used it and it was awesomely lightweight. It’s as much an example of how far world class talent can go, as it is about doing more with less.
- didntreadarticl 4y agoDapper! I used it a while back and it was a single class that bundled query results straight into a list of objects by emitting low level CLR bytecode Looks like its expanded a little since then https://github.com/DapperLib/Dapper https://github.com/DapperLib/Dapper
- KyeRussell 4y agoYep. Following some of the SO folks on Twitter a while back, I remember watching them do all sorts of things with .NET that didn’t feel remotely “necessary” for a Q&A website. It’s not like you can pull people off the street and have them get away with infrastructure this simple.
- cntainer 4y ago> It’s not like you can pull people off the street and have them get away with infrastructure this simple I know that in many cases simple != easy but I can't help feeling sad while reading this. When I started my career cloud wasn't yet mainstream bu as a beginner I was able to deploy and configure a nginx proxy and loadbalance between 2-3 backend servers without too much effort. It wasn't some kind of rocket science. I guess the current issue is that cloud has been marketed so much that nobody who's just starting out in the industry even has a second thought about using it by default. What can I say, great job from the cloud providers in capturing their customers as soon as they get in front of the store.
- jameshart 4y agoGreat, now you have an nginx reverse proxy as a load balancer in front of a few servers. Now sort out log storage, certificate expiry, access controls, patch management, health monitoring, and remote administration, update it whenever you add or remove backend servers for maintenance, and make sure to synch it up to DNS, and you’ve almost got the same capability as an AWS ELB. Except yours doesn’t have high availability or horizontal autoscaling. Getting all of that stuff right actually kind of gets close to rocket science. Which can be worth doing… but just be aware that Amazon will happily sell you a rocket kit.
- PaulKeeble 4y agoMicroservices remains mostly an organisational pattern to scale development teams not necessarily the system performance. Microservices add a lot of complexity and overhead.
- mupuff1234 4y ago"Normal" sized services should be adequate enough for that purpose.
- threeseed 4y agoMicroservices became a synonym for Services Orientated Architecture years ago. It's almost always relatively normal sized services split by functional area e.g. Auth, Cache etc.
- smitty1e 4y agoSOA was all XML at the API level. Serverless is RESTful, lower-level, and coupled to the cloud-provider's menu of containers, key/value store, authentication, &c. "Synonymous" seems a stretch.
- guywhocodes 4y agoWhen concept X becomes hot as an alternative to Y, everyone absolutely have to do X or be square. But for large enough companies it's cheaper to influence the industry so that X = Y and we relegate Y to Y', where Y' only contains our bad memories of Y. This is the "Enterprise Technology Adaptation Strategy".
- bryancoxwell 4y agoIt’s also one of the few sites I use that regularly goes down for maintenance.
- deleted 4y ago[deleted]
- ThatMedicIsASpy 4y agosteam would be the biggest for me
- faizmokhtar 4y ago"What I think it should be" That's a little bit arrogant no?
- didntreadarticl 4y agoThey mean preconception
- KyeRussell 4y agoQuite the opposite. It’s what mere morals think it’d be, vs what the extraordinary talent has gotten away with.
- cntainer 4y agoImagine trying to present this kind of architecture to a room full of executives already sold on the "benefits" of kubernetes, big data, serverless, etc.
- prng2021 4y agoHah. I get your point but it would be an easy sell for them. The impossible sell would be to engineers. Executives would just compare operating costs estimates.
- cntainer 4y agoGood point, for normal executives (whatever that means). In my little bubble most executives I have to deal with believe themselves to be on par with solution/enterprise architects and they like to show this by saying stuff like: "Let's use microservices and kubernetes for better scalability, everybody's doing it..."
- threeseed 4y agoThe use case is simple i.e. web front end, thin app layer, database. So if you were to implement this same architecture using Kubernetes or Serverless it would be as equally simple as a bunch of Ansible or Puppet scripts.
- cntainer 4y agoAgreed, but would most engineers understand that they can keep the simplicity of the solution if the underlying infrastructure is based in the cloud/serverless/etc? Fro my limited experience many engineers fall in the trap of adding accidental complexity to an otherwise simple architecture just by trying to use the latest/coolest cloud architecture trend. Monolith in the cloud on kubernetes? Speak no such abomination. Of course we have to do microservices, the more the better. How can we scale otherwise? SQL DB? What is this, 2010? Of course we're going to use Cosmos DB, how else could we get "single-digit millisecond response times, automatic and instant scalability, along with guarantee speed at any scale". Of course I'm exaggerating for dramatic effect but I rarely see teams disciplined enough to keep cloud architectures simple and clean.
- kichik 4y agoIs there a website that tracks outages of other websites like Stack Overflow over years? I know some that tell you if it's down right now, but not over years. I have a subjective feeling that Stack Overflow is down a lot more than other websites. I don't see that ever mentioned in the discussion of cloud vs on-prem which makes the discussion seem lacking.
- didntreadarticl 4y agohttp://stats.pingdom.com/w2oc4thvox7s/73676/history http://stats.pingdom.com/w2oc4thvox7s/73676/history
- ilyt 4y agoWith a caveat that pingdom will mark "a connection from pingdom server from other side of the world to the server" as downtime, even if the target and your ISP, and the ISP of your ISP had no problems.
- capableweb 4y agoSeems to be testing from just one location, as far as I can tell? Randomly, packets time out on the internet, I would take this random dashboard with a grain of salt, we cannot be sure SO had a outage just because one request happen to fail.
- didntreadarticl 4y agoOn the other hand, if they had a 'down for maintenance' page up, pings would still work
- Spooky23 4y agoThat’s an engineering choice not cloud vs. cloud. How many services are down when AWS us-east has a problem?
- kichik 4y agoTrue. But cloud makes it a lot easier. In some cases it's built-in, like S3. In others it's a checkbox like RDS Multi-AZ. And if you need to roll your own, multi-AZ or even multi-region is much more straightforward than renting another rack somewhere. I have personally seen Stack Overflow be "under maintenance" or straight up down a lot more than I have seen entire us-east-1 down.
- lifeisstillgood 4y agoThe main takeaway is that the questions searched for are so widely distributed that there is no need for a cache layer - they are nothing but long tail. At that point there is no 'cloud' design that can help. Its either one database (or maybe just shard everything onto thousands of distributed nodes) But the point I am trying to make is that kubernetes and microservices etc are based on idea of winners - power laws. One tweet everyone wants to read. One search term, one viral video. Then again. This is just a question of taste - the taste of the dev lead. What (s)he feels is best approach. Take another company doing the same thing and different approach might emerge.
- pickledish 4y agoI mean, kubernetes or microservices don’t care how the data reads are distributed, right? That problem is a database-level thing whereas k8s is infrastructure, you can run any kind of database with any kind of sharding you want on it. I feel like it might be more accurate to say something like “the value of caching is based on the idea of winners” for example
- lifeisstillgood 4y agoYes. I think basically I would not have done it like that but they did, it's wildly successful so fair play - it's taste that makes the difference
- tiffanyh 4y ago> Removed Redis 4 years ago; average latency remained unchanged at 20ms. A hidden taken away is that NVMe storage databases are so fast, they are comparable to in-memory (redis) databases these days.
- kkielhofner 4y agoThrowing 1.5TB of RAM in the SQL Server (server) has to help too!
- tiffanyh 4y ago> [1.5TB of RAM] that is a third of the entire Q&A dataset. Yes, but maybe not as much as you’d think. https://twitter.com/sahnlam/status/1629713961951330304 https://twitter.com/sahnlam/status/1629713961951330304
- kkielhofner 4y agoIt's probably debatable in this case but from what I know of Postgres (as one example) the general thinking seems to be "throw as much (relatively cheap) RAM at it as you can", tune some of the default (conservative) memory consumption params, and let Postgres eat the RAM. See the various parameters here[0] - it's complicated but from my understanding you can pretty quickly run into performance issues depending on some not-exactly obvious variables in dataset size, specific queries, etc. Of course Postgres != SQL Server but the concepts are likely similar. That said you won't catch me every researching this because I've never used SQL Server and never will :). [0] - https://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server https://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Serv...
- ec109685 4y agoSource material is from 2022, so title should include that disclaimer.
- eduction 4y agoThe best cache is the one built into the database. People seem to forget that the major rdbmses have sophisticated cache strategies of their own and that handing them more RAM (and ensuring they are configured to use it for query or other cache) is usually a good first strategy before trying to second guess and reinvent the cache outside the db. Thread says SO allocates 1.5TB RAM to SQL Server. Sounds wise.
- MrFoof 4y agoMakes sense. Traditional RDBMSs are basically a buffer cache and a query optimization engine. If the data is sitting in memory, and you've tuned extracting the data from memory as fast as possible, job done.
- winrid 4y agoNot just a RDBMSs. Any modern DB, document store, or kv store will use a buffer cache.
- likeabbas 4y agoIt's all about the load though. SO is probably 95% Read-Only which makes sense for removing the cache layer. If you had a more writes, then they would need an external cache to offset the read load.
- tfehring 4y agoI don’t follow. Holding the total server load constant, why wouldn’t a read-heavy workload benefit more from caching than a more balanced read/write workload?
- bitwize 4y agoThat defies the laws of physics. How can they be web scale without cloud and microservices?
- another2another 4y agoI want to upvote you, but you forgot MongoDB, which is the most fundamental law of web scale.
- bitwize 4y agoWe all know that /dev/null is an adequate substitute, as long as it gets those kickass benchmark numbers.
- cosmotic 4y agoIt's not cacheless. There are countless caches throughout (including what appears to be ~1TB of memory in the database server), just not a dedicated cache machine.
- Sammi 4y agoIt think OP is only referring to server architecture. And as you say there is no cache server. So cacheless server architecture.
- ElectricalUnion 4y agoBy this definition almost all non-toy applications under non-toy OSes have caches, because of CPU caches and registers.
- foobazzy 4y agoPlease ignore my lack of understanding a bit here. I'm genuinely trying to learn. I've always heard (and it made sense to me) that to reduce latency of requests from across the globe, you might want to have read replicas or caches spread on global infrastructure. Then how is it that stack overflow is fast here when the db is on-prem, 7 seas across from me? Any amount of RAM should not account for the distance, right?
- spiffytech 4y agoYou can put a big dent in the impact of the speed of light if you keep round-trips to a minimum. This is one advantage of server-rendered HTML (though that's not the only option you have). It also helps that StackOverflow is light on interactivity. You load a page, read for a minute, then maybe click a vote button or open a textarea to discuss. As long as the text and styles load quickly, you won't notice if progressive enhancement scripts take a little more time to load.
- Fire-Dragon-DoL 4y agoIsn't stackoverflow, incidentally, one of the websites who would benefit the most from caching, given their content supposedly is going to be static the majority of the time?
- infomaniac 4y agoThis is addressed in one of the linked tweets.
- stby 4y agoThis one: https://twitter.com/sahnlam/status/1629713961951330304?s=20 https://twitter.com/sahnlam/status/1629713961951330304?s=20
- jonas-w 4y agoThe linked url [0] is also a great visualization with a bit more data than the twitter image. [0] https://stackexchange.com/performance https://stackexchange.com/performance
- ctvo 4y agoThe folks over at SO picked a stack (C#, SQL Server, IIS), and optimized the heck out of it to keep this "simplicity". Much of SO is custom built from the ground up to push performance and stay within the purity of the canonical .net stack. It isn't clear to me this is a model that would work elsewhere, or should be held up as something to be replicated. Did they save time? Did they save money? Did this help make SO a wildly successful company? Did it allow them to deliver features to customers faster?
- Yeroc 4y agoIt's worth reminding people what is actually possible with a relatively simple architecture. There's a vast number of websites and services with a very small fraction of the traffic of Stack Overflow with a much more complicated architecture simply because everyone thinks you need Kubernetes etc to scale out.
- inhumantsar 4y agoThat's the point though. If you want to focus your engineering time on optimization and code quality, then of course you can scale to SO's size with 9 servers and a simple architecture. If you're still growing and more interested in delivering tons of features quickly, and/or don't have the ability to attract world leading talent, then a more complicated architecture with clear boundaries is often a better call than delivering relatively few features with obsessive rigor in a monolithic codebase.
- Yeroc 4y agoYou'll only need the expertise to perform in-depth optimization if you're scaling to the level of Stack Overflow though. For the vast majority of sites it won't be a concern. The simpler architecture should be the default because local method calls are easily 2 orders of magnitude faster than a network roundtrip (yes I know that's not the whole story). I'm not sure how or why creating clear boundaries without the RPC crutch suddenly became insurmountable.
- tylergetsay 4y agoI don't think its that much more complicated than Wikimedia, which does 5x the traffic: https://meta.wikimedia.org/wiki/Wikimedia_servers https://meta.wikimedia.org/wiki/Wikimedia_servers
- ksec 4y agoAnd somehow Wikipedia require thousands of severs.
- ElectricalUnion 4y agoWikipedia servers much heavier multimedia content around 20x more often (in page views), with a vastly highier write load.
- mike_hearn 4y agoIt's a useful reality check. Dedicated machines are fast and you can do a lot without much software complexity. People mention the StackOverflow guys optimizing their software, but their CPU utilization is 5% so they have a lot of headroom to be less optimized. Probably they just enjoyed it and could spend time on that, so why not? At KotlinConf in April I'll be giving a talk on two-tier architecture, which is the StackOverflow simplicity concept pushed even further. Although not quite there yet for social "web scale" apps like StackOverflow, it can be useful for many other kinds of database backed services where the users are a bit more committed and you're less dependent on virality. For example apps where users sign a contract, internal apps, etc. The gist is that you scrap the web stack entirely and have only two tiers: an app that acts as your frontend (desktop, mobile) and an RDBMS. The frontend connects directly to the DB using its native protocols and drivers, the user authentication system is that of the database. There is no REST, no JSON, no GraphQL, no OAuth, no CORS, none of that. If you want to do a query, you do it and connect the resulting result stream directly to your GUI toolkit's widgets or table view controls. If what you want can't be expressed as SQL you use a stored procedure to invoke a DB plugin e.g. implemented with PL/Java or PL/v8. This approach was once common - the thread on Delphi the other day had a few people commenting who still maintain this type of app - but it fell out of favor because Microsoft completely failed to provide good distribution systems, so people went to the web to get that. These days distributing apps outside the browser is a lot easier so it makes sense to start looking at this design again. The disadvantages are that it requires a couple more clicks up front for end users, and if they have very restrictive IT departments it may be harder for them to get access to your app. In some contexts that doesn't matter much, in others it's fatal. The tech for blocking DoS attacks isn't as good, and you may require a better RDBMS (Postgres is great but just not as scalable as SQL Server/Oracle). There are some others I'll cover in my talk along with proposed solutions. The big advantage is simplicity with consequent productivity. A lot of stuff devs spend time designing, arguing about, fighting holy wars over etc just disappears. E.g. one of the benefits of GraphQL over plain REST is that it supports batching, but SQL naturally supports even better forms of batching. Results streaming happens for free, there's no need to introduce new data formats and ad-hoc APIs between frontend and DB, stored procedures provide a typed RPC protocol that can integrate properly with the transaction manager. It can also be more secure as SQL injection is impossible by design, and if you don't use HTML as your UI then XSS and XSRF bugs also become impossible. Also because your UI is fully installed locally, it can provide very low latency and other productivity features for end users. In some cases it may even make sense to expose the ability to do direct SQL queries to the end user, e.g. if you have a UI for browsing records then you can allow business analysts to supply their own SQL query rather than flooding the dev's backlog with requests for different ways to slice the data.
- wlonkly 4y agoWhen I look up www.stackoverflow.com, I get Fastly IPs. I feel like using a CDN has to count as some cache?
- bluedino 4y agoNot that long ago (2016) they had: Servers: SQL Servers (Stack Overflow Cluster) 2 Dell R720xd Servers SQL Servers (Stack Exchange “…and everything else” Cluster) 2 Dell R730xd Servers, each with: Web Servers 11 Dell R630 Servers Service Servers (Workers) 2 Dell R630 Servers 1 Dell R620 Server Elasticsearch Servers (Search) 3 Dell R620 Servers HAProxy Servers (Load Balancers) 2 Dell R620 Servers Redis Servers (Cache) 2 Dell R630 Servers VM Servers (VMWare, Currently) 2 Dell FX2s Blade Chassis, each with 2 of 4 blades populated 4 Dell FC630 Blade Servers (2 per chassis) 2 Equalogic SAN PS6000-series Machine Learning Servers (Providence) 2 Dell R620 Servers Machine Learning Redis Servers (Still Providence) 3 Dell R720xd Servers LogStash Servers 6 Dell R720xd Servers HTTP Logging SQL Server 1 Dell R730xd Development SQL Server 1 Dell R620 Network: 2x Cisco Nexus 5596UP core switches (96 SFP+ ports each) 10x Cisco Nexus 2232TM Fabric Extenders (2 per rack) 2x Fortinet 800C Firewalls 2x Cisco ASR-1001 Routers 2x Cisco ASR-1001-x Routers 6x Cisco 2960S-48TS-L Management network switches (1 Per Rack) https://nickcraver.com/blog/2016/03/29/stack-overflow-the-hardware-2016-edition/ https://nickcraver.com/blog/2016/03/29/stack-overflow-the-ha...