10 ms·
HTTP/2 rapid reset attack impacting Nginx products
- chidi0202 3y ago[dead]
- dang 3y agoRelated. Others? HAProxy is not affected by the HTTP/2 Rapid Reset Attack - https://news.ycombinator.com/item?id=37837043 https://news.ycombinator.com/item?id=37837043 - Oct 2023 (31 comments) The largest DDoS attack to date, peaking above 398M rps - https://news.ycombinator.com/item?id=37831062 https://news.ycombinator.com/item?id=37831062 - Oct 2023 (461 comments) HTTP/2 Rapid Reset: deconstructing the record-breaking attack - https://news.ycombinator.com/item?id=37831004 https://news.ycombinator.com/item?id=37831004 - Oct 2023 (22 comments) HTTP/2 zero-day vulnerability results in record-breaking DDoS attacks - https://news.ycombinator.com/item?id=37830998 https://news.ycombinator.com/item?id=37830998 - Oct 2023 (69 comments) The novel HTTP/2 'Rapid Reset' DDoS attack - https://news.ycombinator.com/item?id=37830987 https://news.ycombinator.com/item?id=37830987 - Oct 2023 (103 comments)
- rewmie 3y agoThanks for the helpful summary. It does wonder to provide context to such an important topic.
- tialaramex 3y agoIt's been interesting to see who is affected and who isn't and their rationale.
- msmith 3y agoAlso Go's HTTP/2 packages - https://news.ycombinator.com/item?id=37863419 https://news.ycombinator.com/item?id=37863419
- nomaxx117 3y agoI posted another one the other day which didn't get any traction but probably goes in the list: https://news.ycombinator.com/item?id=37835295 https://news.ycombinator.com/item?id=37835295
- jchw 3y agoI don't know if a post to HN has been made (don't think so), but Envoy released 1.27.1 in response to Rapid Reset as well. https://www.envoyproxy.io/docs/envoy/v1.27.1/version_history/v1.27/v1.27.1 https://www.envoyproxy.io/docs/envoy/v1.27.1/version_history...
- codetrotter 3y agoHehe, when I heard about the attack a couple of days ago I was interested to know if Nginx was affected and did a search on Google for the CVE of that attack followed by the name of Nginx. I didn’t find anything relevant so I assumed that Nginx was not affected. Turns out that was not a good assumption :p
- herpderperator 3y agoIf you read the article, you'll see that the default configuration is not affected.
- codetrotter 3y agoI know. But not everyone uses the default configuration.
- k_roy 3y agothe article literally points out the somewhat rare configuration you would have to have to be exploited by this. The way this headline is worded is pretty misleading, and you clearly sucked it up.
- codetrotter 3y agoHere’s the thing. I use Nginx. Some of the configurations in which I use Nginx were mostly copy-pasted from recommendations from third-parties. Hence, my initial assumption which had been that no action was needed when I didn’t find anything mentioning Nginx with this attack when I searched a few days ago, needed to be revisited. Because when I saw the op article it turned out that there was reason for me to have a closer look at my Nginx instances. To see if any of the configs that had been recommended by third-parties involved changing values that could lead to this attack being able to affect me. > you clearly sucked it up Clearly not. Or at least, not without reason.
- ziddoap 3y agoCVEs aren't restricted to only consider default configs.
- sickofparadox 3y agoImportant to note that unless your Nginx instance has a special (read: very high) keepalive limit configured, Nginx has a fairly reasonable defense against HTTP/2 rapid reset attack by default, as the article says. Still, interesting to see the response to these attacks.
- ChrisArchitect 3y agoWhy the submission OP? Lots of discussion and submissions related to this over the last few days, not to mention this submitted 2 days ago
- jacquesm 3y agoBecause Nginx is a very widespread high performance web server that initially seemed not to be impacted but now it turns out that it is.
- tmpX7dMeXU 3y agoThere is no new revelation here.
- k_roy 3y agoExcept it's only impacted in a pretty non-standard config. Not that it's not important to disseminate the knowledge, but the chosen title here is deliberatley sensational
- jacquesm 3y agoIt impacts the product, there is nothing sensational about that. If it wasn't impacted that's what they would have said and with Nginx being the tool of choice for many high volume installations you can bet that there are many thousands running in non-standard configs.
- eastdakota 3y agoFrom some first-hand experience over the last few months… these suggestions and patch will help prevent a single client from overwhelming an NGINX server, but it will do little to stop even a modest botnet from generating enough requests to be a problem. Keeping some state on IPs and downgrading those that exceed limits to HTTP/1.1 I believe is the only effective defense. Tuning those thresholds to get them right is… challenging.
- 22c 3y agoIf the only viable fix is to downgrade clients to an earlier protocol, do you take that to mean that there is a fundamental weakness in the protocol itself?
- phendrenad2 3y agoHow does HTTP/1.1 stand up to the current attack?
- eastdakota 3y agoNot impacted.
- deleted 3y ago[deleted]
- andrewstuart 3y agoAnyone know if it affects Caddy?
- MallocVoidstar 3y agoPatched in 2.7.5: https://github.com/caddyserver/caddy/releases/tag/v2.7.5 https://github.com/caddyserver/caddy/releases/tag/v2.7.5 I think it might also require a patched version of Go.
- kelthan 3y agoThere are new versions of Go which have this CVE patched already published and available for download.
- jprd 3y agoYou could 'caddy upgrade' pretty quickly to get the patch (servers had updated go), though the release number bump didn't happen immediately. Running the same now, or pulling a new binary, using xcaddy, etc. will get you 2.7.5 which also includes some other small fixes not related to rapid reset.
- ComputerGuru 3y agoI’m stuck trying to figure out if this is technically desired behavior or not. If you were retroactively designing http/2 with this knowledge, would you have done anything different?
- kelthan 3y agoYes, because it's exploitable. Systems that require very high cognitive load on their human operators (whether machines, programming languages, etc.) are alway destined to fail. Human beings are not good at doing boring, repetitious work that requires them to stay focused: lapses will occur. And hackers are going to find and exploit those gaps. So the best way to avoid those problems is to build solutions or specifications in which those gaps are not even possible. Modern software systems are some of the most complex systems that humans have ever invented--and they just keep getting more complex over time as we layer new things on top of them. Think of a really high Jenga tower with lots of holes in the base. That means we need to strive to keep things simple. That may mean making decisions that prevent common or severely impacting failure cases from being possible, even if it is a little more difficult to do, or eliminates an esoteric use-case. This is even more true when writing specifications that others will implement and/or be expected to conform to.
- j16sdiz 3y agoI guess the parent post want to know if there are any _specific_ and _effective_ change for this kind of attacks. In this case, simplifying the protocol won't help -- the vulnerability is: 1. Backend servers can't cancel immediately (this is no protocol problem) 2. The client can make concurrent request in a connection (This is the goal of Http/2) 3. The concurrency is pre determined, there is no way for the server to throttle without user-visible error. 4. The client can cancel any request mid-flight (removing this is equally bad, security-wise) Unless you are removing the concurrency, making the protocol simpler won't fix it. The protocol designer need adversary mindset, not a simpler mind
- tedunangst 3y agoYou should be able to cancel a request, but you don't get it back (as in, can't send another) until the server acks the cancel.
- nimbius 3y agoFYI this is for the commercial nginx product, hastily purchased by F5 a few years back when software load balancers were annihilating their hardware offering. Curious to see f5 still playing games with their own cve disclosure on the bigip product though...assigning it a mitre cw400 is just lying. https://my.f5.com/manage/s/article/K000137106 https://my.f5.com/manage/s/article/K000137106
- gshulegaard 3y agoBoth http2_max_concurrent_streams and keepalive_requests (the configuration parameters discussed in this article) are configuration parameters available in open source nginx: http://nginx.org/en/docs/http/ngx_http_v2_module.html#http2_max_concurrent_streams http://nginx.org/en/docs/http/ngx_http_v2_module.html#http2_... http://nginx.org/en/docs/http/ngx_http_core_module.html#keepalive_requests http://nginx.org/en/docs/http/ngx_http_core_module.html#keep... So are limit_conn and limit_req: https://nginx.org/en/docs/http/ngx_http_limit_conn_module.html https://nginx.org/en/docs/http/ngx_http_limit_conn_module.ht... https://nginx.org/en/docs/http/ngx_http_limit_req_module.html https://nginx.org/en/docs/http/ngx_http_limit_req_module.htm... So it pertains to both NGINX and NGINX+.
- 1vuio0pswjnm7 3y agoIf someone asked me how to "speed up the web", I would not suggest "use HTTP/2". I would remove ads and other garbage. As a decades long non-popular browser and TCP client user, I can testify this works very effectively. I prefer to have full control over the resources that I request, whether text or binary, so no auto-loading resources, no Javascript-requested resources and no HTTP/2 "server push". The clients I use do not auto-load resources, run Javascript nor carry out "server push". Works great for me. Web is not slow. According to HTTP/2 proponents, the protocol originated at an online advertising services company and was developed by companies that profit from sale and delivery of online advertising, HTTP/2 was designed to "speed up the web". I respect that opinions on HTTP/2 may differ. If someone loves HTTP/2, then I respect that opinion. In return I ask that others respect opinions that may differ from their own, including mine. NB. This comment speaks only for the web user submitting it. It does not speak for other web users. IMHO, no HN commenter can speak for other web users either. Thank you.
- gear54rus 3y agoWell you ain't getting no-JS webpages because it's too useful so maybe HTTP/2 in the end? Adblock helps with ad I hear.
- wvenable 3y agoI would say that a text-only experience is valid I don't think it's how the majority of people want to use the web. Users want a rich multimedia experience. If HTTP/2 speeds up a rich multimedia web experience then it may legitimately be one way to "speed up the web" for someone who expects that level of experience. I don't think it's fair to criticize a protocol for who designed it. The specification is out there for anyone to interpret and if there is specific complaint in it's design then make that.
- veeti 3y agoHTTP2 Server Push has been deprecated already https://developer.chrome.com/blog/removing-push/ https://developer.chrome.com/blog/removing-push/
- otabdeveloper4 3y ago
- getcrunk 3y ago> layer 4 monitoring and alerting tools What do you guys use? Anything foss and not an applicance?
- amelius 3y ago> this vulnerability can be exploited to execute a denial-of-service attack Title should contain this info.
- bullen 3y agoJust use HTTP/1.1, it's the final protocol. Nothing Google or Microsoft does will dethrone it. Forget the browser; use a C or Java client and HTTP. If they block port 80, just use another port. They cannot win.
- knorker 3y agoWho is "they"? Is there some kind of evil back room conspiracy to make the web faster, using open standards?
- bullen 3y agoThey are those that disable HTTP. Like it was not enough to make HTTPS default, they need to eradicate the opposition. The list is too long to enumerate but they all have one thing in common; they profit from root certificates. The web is not faster, it's bloated. It's only open if you can understand it and implement it. HTTP/1.1 is the fastest, most open, web you'll ever have; because it is small.
- knorker 3y ago> They are those that disable HTTP. Who's disabling HTTP? > Like it was not enough to make HTTPS default, they need to eradicate the opposition. I think you need to elaborate your world view by many paragraphs before I can understand what you're trying to say. You're against HTTPS? Plain TLS wrapping HTTP? > they profit from root certificates. The (web) root certificate industry has never been weaker than it is today, thanks to free root CAs like LetsEncrypt. > HTTP/1.1 is the fastest, most open, web you'll ever have; because it is small. So run it. Nobody's stopping you. I won't. Aside from firesheep, the fact that you thanks to Snowden we no longer have to be called a conspiracy theorist to believe in Echelon, and with all the webpage ad and bitcoin mining injectors, or just plain DPI middleboxes, I don't think HTTP is a reasonable default. Google used to provide "Conncetion: Close" (with the typo), to work around middle boxes that did shenanigans to headers, but luckily only compared some bad checksum of the header, not string comparison. HTTP/1.1 is open and simple, sure, but it's also being interfered with on a massive scale. Companies out there are selling ad injector boxes to ISPs. They only work on HTTP. You don't have ads on your blog? Well, you do now, for the visitors from some ISPs. Now HTTP/2 and HTTP/3, I see more of your point. Encryption is just table stakes to get a working website, at this point, but nobody's obligated to race to eliminate every RTT. Plain HTTPS is fine. CSS sprites is still (last I checked) a bit faster than individual resources over HTTP/2 or 3. But if I'm making a photo album showing 50 thumbnails at once, then I'm unlikely to use plain simple HTTPS with individual resources. I'd at least have to choose between CSS sprites or HTTP/2/3. It would just make for a poor user experience otherwise. And if you're not making a website for your users, then what is it for?
- blackbeans 3y agoWhat about the old and proven Apache? Is it affected?