6 ms·
Reining in the thundering herd: Getting to 80% CPU utilization with Django
- vvatsa 5y agoya, I pretty much agree with 3 suggestions at the end: * use uWSGI (read the docs, so many options...) * use HAProxy, so very very good * scale python apps by using processes.
- catillac 5y agoFamous last words, but I get the sense that the need to handle this sort of load on Clubhouse is plateauing and will decline from here. The app seems to have shed all the people that drew other people initially and lost its small, intimate feel and has turned into either crowded rooms where no one can say anything, or hyper specific rooms where no one has anything to say. Good article though! I’ve dealt with these exact issues and they can be very frustrating.
- hnarayanan 5y agoThis
- latchkey 5y agoIf it is just a backend, why not port it over to one of the myriad of cloud autoscaling solutions that are out there? The opportunity cost of spending time figuring out why only 29 workers are receiving requests over adding new features that generate more revenue, seems like a quick decision. Personally, I just start off with that now in the first place, the development load isn't any greater and the solutions that are out there are quite good.
- _tom_ 5y agoThey are on a back-end that does auto-scaling. They stated that they had problems when scaling up past 1000 nodes. Now, maybe they could have fixed that issue instead, but going from 29 to 58 workers is easy, it's not the same going to 29,000 to 58,000. And 1000 hosts vs 500 is a non-trivial cost.
- PaywallBuster 5y agocontainers would've solved it one process per container, easy peasy
- motoboi 5y agoyou now containers are just processes, right? This is what they did, but because they didn't need to schedule other jobs on the same machine, kubernetes or even docker would be overkill. In this case, simple VM orchestration seems like a fine solution.
- PaywallBuster 5y agoIndeed, but you wouldn't be thinking about instance sizes, how many processes per instance and wondering if you're hitting kernel limits with all the issues coming up
- motoboi 5y agoWhy not?
- zbentley 5y agoYou'd probably be worrying more about instance sizes if you ran a single executor per container; the memory overhead of your app would become a problem very quickly unless it's startup footprint was quite small.
- PaywallBuster 5y agoThat's what they're doing now. One app pool with one worker x number of cores
- zbentley 5y agoI assumed they're managing all those workers under one parent process which compiled their codebase on start. Perhaps that assumption was in error.
- lddemi 5y agoAuthor here. We do and did use autoscaling heavily but at a certain scale we just ran out of headroom on the smaller instance types we were using. Jumping to a much larger instance types meant that we will likely never run into those headroom issues again, plus solves other problems like faster spin up, better sidecar connection pooling and allows for a much higher hit rate on per instance caching.
- TekMol 5y agoDid you consider switching form CPython to Pypy?
- latchkey 5y agoYou were autoscaling a single threaded process. You had 1000 connections coming in and scaling 1000 workers for those connections. Everything was filtered through gunicorn and nginx, which just adds additional latencies and complexity, for no real benefit. What I'm talking about is just pointing at something like AppEngine, Cloud Functions, etc... (or whatever solution AWS has that is similar) and being done with it. I'm talking about not running your own infrastructure, at all. Let AWS and Google be your devops so that you can focus on building features.
- ddorian43 5y agoNow you just 5x their costs.
- latchkey 5y agoNot if you do it right. a) you get to fire the devops person, which saves $150k+ a year. b) you add appropriate caching layers in front of everything. c) you spend time adding features, which generate revenue. I've done all of this before at scale. This whole case study was written about work I did [1]. Two devs, 3 months to release, first year was $80m gross revenue on $500/month cloud bills. Infinite scalability, zero devops. [1] https://cloud.google.com/customers/gearlaunch https://cloud.google.com/customers/gearlaunch
- TekMol 5y agoPerformance is the only thing that is holding me back to consider Python for bigger web applications. Of the 3 main languages for web dev these days - Python, PHP and Javascript - I like Python the most. But it is scary how slow the default runtime, CPython, is. Compared to PHP and Javascript, it crawls like a snake. Pypy could be a solution as it seems to be about 6x faster on average. Is anybody here using Pypy for Django? Did Clubhouse document somewhere if they are using CPython or Pypy?
- sdze 5y agouse PHP ;)
- nsizx 5y agoSo much this. Practically any other option is better than Python for web development if you're looking for performance.
- void_mint 5y agoBy this logic, why not Java, C++, Rust, Go, C#? They’re all web-capable and blow the doors off PHP, Python, etc.
- sdze 5y agoC#, Java, C++ need application servers, no? "Serverless" scales infinitely due to its simpler request/response lifecycle.
- void_mint 5y agoServerless is too overloaded a term to have any meaning. I'm not really seeing how Python or PHP "scales infinitely" in any way that C#, Java, C++ couldn't.
- tored 5y agoPHP is usually easier to scale because it just a matter of how many webservers. e.g. apache or nginx, you choose to deploy. This also possible with other platforms, but can be a bit trickier to get right. For large PHP setups it is usually the number of database connections that is the limiting factor, however that is why historically the replicated MySQL databases was such a good fit for PHP, thus only creating a limit for writes on the master.
- void_mint 5y ago
- tbrock 5y agoAside: AWS only allows registering 1000 targets in a target group… i wonder if thats the limit they hit. If so, its documented.
- dilyevsky 5y agoKinda funny they decided paying a ton of money to aws was ok but paying for nginx plus was not
- Spivak 5y agoI kinda get that honestly. It’s why I’ll spend $20 without even thinking for take out but not spend $2 for an app. It’s because the cost off the software is way way more than the money. It’s a commitment to actually use it and integrate it, deal with their sales team, talk to purchasing, handle licensing, and introducing friction to replacing it or using tools that don’t integrate well because “well we already pay for it.” Licensing also complicates deployments substantially when you’re doing lots of autoscaling. And on top of that Nginx Plus is also expensive as hell.
- dilyevsky 5y agoDon’t you have to integrate cloud? This whole post is about having to put a bunch of workaround bc the cloud can’t scale apparently
- ClumsyPilot 5y ago"It’s why I’ll spend $20 without even thinking for take out but not spend $2 for an app." I pay for apps, its not a healthy attotude
- rowanG077 5y agoThe buy in into AWS is much, much larger then using a piece of software though.
- deleted 5y ago[deleted]
- spullara 5y agoThe difference people see, as far as I can tell, is that AWS is charging you cost+ and pure software companies need to charge for value or die.
- stingraycharles 5y agoTangent, but I always had a different understanding of the “thundering herd” problem; that is, if a service is down for whatever reason, and it’s brought back online, it immediately grinds to a halt again because there are a bazillion requests waiting to be handled. And the solution to this problem is to slowly, rate-limited, bring the service back online, rather than letting the whole thundering herd go through the door immediately.
- Ozzie_osman 5y agoYea you are right. It could be a service being down and requests piling up, or a cache key expiring and many processes trying to regenerate the value at the same time, etc. I think the article just used this phrase to describe something else. (Great article otherwise).
- taylorhughes 5y agoPhrase borrowed from excellent uWSGI docs https://uwsgi-docs.readthedocs.io/en/latest/articles/SerializingAccept.html https://uwsgi-docs.readthedocs.io/en/latest/articles/Seriali...
- ambicapter 5y agoFunny reading this comment after reading the article > So many options meant plenty of levers to twist around, but the lack of clear documentation meant that we were frequently left guessing the true intention of a given flag. And then reading your link, they complain >inside the docs< that the docs aren't complete. I have no idea what to believe anymore :D
- remram 5y agoThe uWSGI docs also say, in the section called "uWSGI developers are fu*!ing cowards": "why --thunder-lock is not the default when multiprocess + multithread is requested? This is a good question with a simple answer: we are cowards who only care about money." Strange read.
- lmilcin 5y ago1M requests per minute on 1000 web instances is not an achievement, it is a disaster. It is ridiculous people brag about it. Guys, if you have budget maybe I can help you up this by couple orders of magnitude.
- smashed 5y agoTo be honest the article does realize this, first blaming it on the poor hindsight from original developer (co-founder) and in the conclusion about maybe rewriting the whole thing. It seemed to be all about how to extract the most performance from the lemon they had to deal with. I found the linked reference really informative too: https://rachelbythebay.com/w/2020/03/07/costly/ https://rachelbythebay.com/w/2020/03/07/costly/
- lmilcin 5y agoI don't know Python or how complex their domain is but the number of workers suggests to me it is not that complex and their application spends most of its time switching contexts and in inefficient frameworks. Per my experience most applications that mostly serve documents from databases should be able to take on at least 10k requests per second on a single node. this is 600k requests per minute on one node, compared to their 1M per 1000 nodes. This is what I am typically getting from a simple setup with Java, WebFlux and MongoDB with a little bit of experience on what stupid things not to do but without spending much time fine tuning anything. I think bragging about performance improvements when your design and architecture is already completely broken is at the very least embarrassing. > poor hindsight from original developer (co-founder) Well, you have a choice of technologies to write your application in, why chose one that sucks so much when there are so many others that suck less? It is not poor choice, it is lack of competency. You are co-founder and want your product to succeed? Don't do stupid shit like choosing stack that already makes reaching your goal very hard.
- nomdep 5y agoSo do you think using Django is stupid? I guess you think the same about every product that uses Ruby on Rails?
- stu2010 5y agoInteresting to see this. It sounds like they're not on AWS, given that they mentioned that having 1000 instances for their production environment made them one of the bigger deployments on their hosting provider. If not for the troubles they experienced with their hosting provider and managing deployments / cutting over traffic, it possibly could have been the cheaper option to just keep horizontally scaling vs putting in the time to investigate these issues. I'd also love to see some actual latency graphs, what's the P90 like at 25% CPU usage with a simple Gunicorn / gevent setup?
- ksec 5y agoI was wondering that too, but there aren't that many common cloud provider that has 96 vCPU offering. I am also wondering on 144 Workers, on 96 vCPU which is not 96 CPU Core but 96 CPU thread. So effectively 144 Workers on 48 CPU Core possibly running at sub 3Ghz Clock Speed. But it seems they got it to work out in the end. ( May be at the expense of latency )
- mst 5y agoAssuming you're running a system where normal request/response handling blocks on database queries it's often optimal to have more workers than available cpu threads and 1.5x is a common rule of thumb to try first.
- j4mie 5y agoIf you’re delegating your load balancing to something else further up the stack and would prefer a simpler WSGI server than Gunicorn, Waitress is worth a look: https://github.com/pylons/waitress https://github.com/pylons/waitress
- JanMa 5y agoInteresting to read that they are using Unix sockets to send traffic to their backend processes. I know that it's easily done when using HaProxy but I have never read about people using it. I guess the fact that they are not using docker or another container runtime makes sockets rather simple to use.
- kvalekseev 5y agoIt's standard way to connect things in UNIX and provides better performance. For example postgresql tcp+ssl is 175% slower than socket https://momjian.us/main/blogs/pgblog/2012.html#June_6_2012 https://momjian.us/main/blogs/pgblog/2012.html#June_6_2012
- lttlrck 5y agoBut domain sockets only work between processes on the same machine, why would SSL be used in that case?
- mst 5y agoI do that every chance I can get. At a guess, it's probably most loved by people picking old school simple architectures that aren't the sort of thing that goes viral.
- kvalekseev 5y agoHAProxy is a beautiful tool but it doesn't buffer requests that is why NGINX is recommended in front of gunicorn otherwise it's suspectible to slowloris attack. So either cloubhouse can be easily DDOS'd right now or they have some tricky setup that prevents slow post reqests reaching gunicorn. In the blog post they don't mention that problem while recommend others to try and replace NGINX with HAPRoxy.
- lddemi 5y ago1. HAProxy does support request buffering https://cbonte.github.io/haproxy-dconv/2.2/configuration.html#4-option%20http-buffer-request https://cbonte.github.io/haproxy-dconv/2.2/configuration.htm... 2. our load balancer buffers requests as well
- kvalekseev 5y agoFrom HAProxy mailing list about http_buffer_request option https://www.mail-archive.com/haproxy@formilux.org/msg23074.html https://www.mail-archive.com/haproxy@formilux.org/msg23074.h... > In fact, with some app-servers (e.g. most Ruby/Rack servers, most Python servers, ...) the recommended setup is to put a fully buffering webserver in front. Due to it's design, HAProxy can not fill this role in all cases with arbitrarily large requests. A year ago I was evaluating recent version of HAProxy as buffering web server and successfully run slowloris attack against it. Thus switching from NGINX is not a straightforward operation and your blog post should mention http-buffer-request option and slow client problem.
- polote 5y agoI wouldn't be very proud of writing an article like that. Usually engineering blogs exists to show that there are fun stuff to do in a company. But here it just seems they have no idea, what they are doing. Which is fine, I'm classifying myself in the same category. Reading the article I don't feel like they have solved their issue, they just created more future problems
- luhn 5y agoUnfortunately HAProxy doesn't buffer requests*, which is necessary for a production deployment of gunicorn. And for anybody using AWS, ALB doesn't buffer requests either. Because of this I'm actually running both HAProxy and nginx in front of my gunicorn instances—nginx in front for request buffering and HAProxy behind that for queuing. If anybody is interested, I've packaged both as Docker containers: HAProxy queuing/load shedding: https://hub.docker.com/r/luhn/spillway https://hub.docker.com/r/luhn/spillway nginx request buffering: https://hub.docker.com/r/luhn/gunicorn-proxy https://hub.docker.com/r/luhn/gunicorn-proxy * It does have an http_buffer_request option, but this only buffers the first 8kB (?) of the request.
- Twirrim 5y agoCouldn't Apache httpd just do all of that for you? mod_buffer provides request buffering, and mod_proxy_balancer provides load balancing capabilities.
- luhn 5y agoCan Apache do request queuing?
- Twirrim 5y agoYes, under MPM, you have a listen backlog. If there's a spare worker available to process a request, they'll be picked off the backlog. https://httpd.apache.org/docs/2.4/mod/mpm_common.html#listenbacklog https://httpd.apache.org/docs/2.4/mod/mpm_common.html#listen...
- luhn 5y agoI don't think that's what I'm looking for, that's queuing at the front of the pipe but I need it queuing at the end of the pipe. Apache should be buffering and queuing lots of connections (with a timeout) and sending them single-file in gunicorn. This lays out what I'm trying to achieve: https://aws.amazon.com/builders-library/using-load-shedding-to-avoid-overload/ https://aws.amazon.com/builders-library/using-load-shedding-...
- deleted 5y ago[deleted]
- jhgg 5y agoThis is somewhat suspect. At my place of work, we operate a rather large Python API deployment (over an order of magnitude more QPS than the OP's post). However, our setup is... pretty simple. We only run nginx + gunicorn (gevent reactor), 1 master process + 1 worker per vCPU. In-front of that we have an envoy load-balancing tier that does p2c backend selection to each node. I actually think the nginx is pointless now that we're using envoy, so that'll probably go away soon. Works amazingly well! We run our python API tier at 80% target CPU utilization.
- lddemi 5y agoglad you are seeing such awesome performance with gevent+envoy! which part of our experience do you think is suspect?
- jhgg 5y agoSo, in guincorn default mode (sync), the mode I'm assuming you're using. This means you really have 1 process handling 1 request at a time. The "thundering herd" problem really only applies to connection acceptance. Which is to say, that in the process of accepting a connection, it is possible to wake all idle processes that are waiting for a connection comes in (they will wake and hit EAGAIN and then go back to waiting.) Busy processes that are servicing requests (not waiting on the accept call) will not be woken, since they aren't waiting on a new request to come in. The "thundering herd" problem as I understand it, can indeed waste CPU cycles, but only on processes that aren't doing much anyways. I do however believe that `accept()` calls have been synchronized between processes on Linux for a while now to prevent spurious wakeups. You should verify you're actually doing spurious wakeups by using `strace` and seeing if you are seeing a bunch of `accept()` calls returning EAGAIN. In gunicorn, `sync` mode does exhibit a rather pathological connection churn, because it does not support keep-alive. Generally, most load balancing layers already will do connection pooling to the upstream, meaning, your gunicorn processes won't really be accepting much connections after they've "warmed up". This doesn't apply in sync mode unfortunately :(. Connection churn can waste CPU. Another thing to also note is that if you have 150 worker processes, but your load balancer only allows 50 connections per upstream, chances are 100 of your processes will be sitting there idle. Something just doesn't feel quite right here. EDIT: I do see mention of `gthread` worker - so you might be already able to support http-keepalives. If this is the case, then you should really have no big thundering herd problem after the LB establishes connections to all the workers.
- trinovantes 5y agoI've always used nginx for my servers. Is HAProxy that much better to consider learning/switching?
- ram_rar 5y ago> Python's model of running N separate processes for your app is not as unreasonable as people might have you believe! You can achieve reasonable results this way, with a little digging. I have been through this journey, we eventually migrated to Golang and it saved a ton of money and firefighting time. Unfortunately, python community hasnt been able to remove GIL, it has its benefits (especially for single threaded programs), but I believe the cost (lack of concurrent abstractions. async/await doesn't cut it) far outweigh it. Apart from what the article mentions, other low hanging fruits worth exploring are [1] Moving under PyPy (this should give some perf for free) [2] Bifurcate metadata and streaming if not already. All the django CRUD stuff could be one service, but the actual streaming should be separated to another service altogether.
- jstrong 5y agoI read the article and could not believe that was their takeaway. sometimes people are determined to vindicate their technology choices, no matter what.
- petargyurov 5y ago> Which exacerbated another problem: uWSGI is so confusing. It’s amazing software, no doubt, but it ships with dozens and dozens of options you can tweak. I am glad I am not the only one. I've had so many issues with setting up sockets, both with gevent and uWSGI, only to be left even more confused after reading the documentation.
- tarasglek 5y agoHave to wonder how well haproxy works vs balancing by making gunicorn listen via SO_REUSEPORT and letting the kernel balance instead (ala https://talawah.io/blog/extreme-http-performance-tuning-one-point-two-million/ https://talawah.io/blog/extreme-http-performance-tuning-one-...)