6 ms·

How Rust Lets Us Monitor 30k API calls/min

97 points by cfabianski 6y ago

qrczeno 6y ago
That was a real issue we were struggling to solve. Feels like Rust was the right tool for the right job.
eggsnbacon1 6y ago
They didn't mention Java as a possible solution, even though its GC's are far better than anything else out there. I have nothing against Rust but if I was at a startup I would save my innovation points for where they're mandatory
- JoshTriplett 6y ago
  > I have nothing against Rust but if I was at a startup I would save my innovation points for where they're mandatory An article published today, addressing that exact point: https://tim.mcnamara.nz/post/621040767010504704/spend-your-novelty-budget-on-rust https://tim.mcnamara.nz/post/621040767010504704/spend-your-n...
  - andrewzah 6y ago
    Ah yes, a 236 word "article", that says to choose "boring old technology" and also to use Rust in the same breath. This article should mention that rust isn't close to ready when it comes to web backends. As much as I love Rust, if I were running a startup or even a decently sized company I would always choose Rails. Now -that- is boring, old... and mature technology. Certain components could get re-written in Rust, certainly, but there's no reason to ignore a mature ecosystem from the start.
    - zozbot234 6y ago
      > This article should mention that rust isn't close to ready when it comes to web backends. Actix-web works just fine. They got a new maintainer team involved that has been spending some time getting rid of all the insane unsafety that was in the code before.
      - andrewzah 6y ago
        Yes. It "works". However deciding to use Actix/Warp means throwing away years and years of work in the rails and ruby world. Rails is mature, robust, and has a huge ecosystem with rubygems. Rust (when it comes to web stuff) is not. "it works" does not pass my litmus test. With Actix/Warp I have to implement stuff by hand that either comes by default with rails or already exists in a gem. I like Rust but I'm not a zealot. People way overestimate performance when they barely have any traffic to begin with as a small startup or even a medium sized company. You could even use rails, and use Rust to write ruby gems instead of going with actix/warp/etc. > insane unsafety This was overblown. Yes, the author didn't respond appropriately, but unsafe isn't inherently dangerous. This is a stupid misconception within the Rust community and caused a lot of unnecessary drama around Actix.
- dep_b 6y ago
  It's not like Elixir doesn't benefit from a battle hardened VM since it's older and has been used in these kind of high volume scenarios before Java was.
  - dnautics 6y ago
    not to mention a GC that you basically don't have to hire an expensive FTE "enterprise dev" to tweak and generally give you good p99s.
- andrewzah 6y ago
  Rust doesn't have a GC; it uses an ownership model instead. Ultimately in a startup it comes to what people are comfortable with. I personally would use Rust for something like this because I am comfortable with Rust. It's a perfect use case for it as well, imho.
- TheTank 6y ago
  One step beyond good Java GCs is to write fully zero-GC Java code. The advantage of it is complete control over your performance which means your software is going to be consistently fast. The disadvantage is that it is relatively difficult to obtain. If you want to see an example of fully zero-GC Java, you can check out QuestDB on Github [1] - Disclaimer I work for QuestDB. [1]https://github.com/questdb/questdb https://github.com/questdb/questdb
  - masklinn 6y ago
    > One step beyond good Java GCs is to write fully zero-GC Java code. The advantage of it is complete control over your performance which means your software is going to be consistently fast. The disadvantage is that it is relatively difficult to obtain. I don't know that it's actually possible in the general case, as Java's support for value types remains wholly insufficient. IIRC the ixy folks never managed to remove all allocations from the java version.
    - bluestreak 6y ago
      It is possible, we've done it.
- lostcolony 6y ago
  They ruled out a language because it had stop the world GC, and even if it removed the bottleneck, it would likely become one later. Java has the same issue. Rust does not. Not sure why they'd consider Java, given that concern.
  - eggsnbacon1 6y ago
    Shenandoah and ZGC collectors have worse case pauses of ~10ms and average pauses of 0.5ms . The average pause is faster than malloc() sometimes in C, so you won't really be introducing more latency than C does. The other option is to avoid allocating memory at all, which you could do in C/Rust but also in Java. The vast majority of shops given the choice for low/no allocation performance use Java anyways (HFT)
    - lostcolony 6y ago
      But those GCs also don't guarantee all garbage has been collected, nor how much processing time you'll get before they run again. So op could still end up stuck with their code barely executing, due to memory and CPU pressure, and throughput/latency drops to zilch. "The vast majority of shops" is an interesting metric given the vast majority had to pick a language before Rust existed. Java and trying to minimize allocations, vs C/C++, I'd probably choose Java too. Java trying to minimize allocations (no way to guarantee you've done it right), vs Rust (which does guarantee no GC)...I'd probably pick Rust.
      - eggsnbacon1 6y ago
        > But those GCs also don't guarantee all garbage has been collected, nor how much processing time you'll get before they run again. So op could still end up stuck with their code barely executing, due to memory and CPU pressure, and throughput/latency drops to zilch. You could malloc() and free() so much that the code doesn't have time to do anything too. And these operations aren't bounded in time either. Just using C doesn't save you from memory allocation, its just done manually instead of automatic. In every system I've worked on you won't have this kind of GC pressure unless you do something profoundly bad > The vast majority of shops" is an interesting metric given the vast majority had to pick a language before Rust existed. true
- zozbot234 6y ago
  They did mention Go that also has a modern GC, and does not need to run your app under a software VM. But ultimately, even Go is just another GC language. They had no real need for GC and the like so why use Go, let alone Java?
  - eggsnbacon1 6y ago
    Go's GC is notoriously low tech. Its design dates back to papers published in the 70s. Java's Parallel collector is most similar to Go's GC and its deprecated for removal due to poor performance. Go collects VERY frequently to keep average pause times low, which hurts throughput. It also has pathological worst-case pause times, which is what the author ran into. My suggestion of Java is just because its been used for high performance REST for decades. Rust has Actix and a few other frameworks, in Java you have 20+ options. Its a lot easier to get something off the ground when you're 99% sure you won't have to build anything except some glue code. Yes, Rust is faster, but is 30% better performance worth treading the wilds while you're trying to keep a startup afloat, I don't think so personally. Using Java with Shenandoah or ZGC collector would directly handle their GC issue with rather boring technology
didroe 6y ago
I'm one of the engineers that worked on this. It was the first Rust production app code I've written so it was a really fun project.
- mamcx 6y ago
  One of the interesting effects of using rust is saving money! I also migrate a F#/.NET ecommerce backend and can run in less RAM/CPU that make my bills lower.
  - chrischen 6y ago
    Does Rust offer much of the same language features as F#?
    - penagwin 6y ago
      Disclaimer: not a rust of .net dev, but my impression is they’re very different in what they’re trying to. It’d would be like comparing C to Python (as an example, not as an analogy). Rust is closer to a super fancy C, compiles natively and was made with a heavily focus on certain types of memory safety. F# has syntax more like Haskell/ML, and is compiled to a bytecode instead of an executable. It runs on .NET and everything that entails.
      - chrischen 6y ago
        That's actually why I'm asking about that since the OP is transitioning backend logic from F# to Rust. So either Rust can cover a lot of the language features or he has a use case that really warrants that performance (and F# on .NET is not exactly non-performant).
    - mamcx 6y ago
      Near. Exist some fancy features that are not present, but the practical side is covered...
  - raphinou 6y ago
meritt 6y ago
Sorry, I must be missing something in this blog post because the requirements here sound incredibly minimal. You just needed an HTTP service (sitting behind an Envoy proxy) to process a mere 500 requests/second (up to 1MB payload) and pipe them to Kinesis? How much data preparation is happening in Rust? It sounds like all the permission/rate-limiting/etc happens between Envoy/Redis before it ever reaches Rust? I know this comes across as snarky but it really worries me that contemporary engineers think this is a feat worthy of a blog post. For example, take this book from 2003 [1] talking about Apache + mod_perl. Page 325 [2] shows a benchmark: "As you can see, the server was able to respond on average to 856 requests per second... and 10 milliseconds to process each request". And just to show this isn't a NodeJS vs Rust thing, check out these webframework benchmarks using various JS frameworks [3]. The worst performer on there still does >500 rps while the best does 500,000. It's 2020, the bar needs to be much higher. [1] https://www.amazon.com/Practical-mod_perl-Stas-Bekman/dp/0596002270 https://www.amazon.com/Practical-mod_perl-Stas-Bekman/dp/059... [2] https://books.google.com/books?id=i3Ww_7a2Ff4C&pg=PT356&lpg=PT356 https://books.google.com/books?id=i3Ww_7a2Ff4C&pg=PT356&lpg=... [3] https://www.techempower.com/benchmarks/#section=data-r19&hw=ph&test=db&l=zik0sf-1r https://www.techempower.com/benchmarks/#section=data-r19&hw=...
- BubRoss 6y ago
  I hope some day doing something trivial using rust will no longer warrant a hacker news post to promote a startup.
- lostcolony 6y ago
  They list out what is being done by the service - "It would receive the logs, communicate with an elixir service to check customer access rights, check rate limits using Redis, and then send the log to CloudWatch. There, it would trigger an event to tell our processing worker to take over." That sounds like a decent amount of work for a service, and without more detail it's very hard to say whether or not a given level is efficient or inefficient (we don't know exactly what was being done; we can assume that they're using pretty small Fargate instances though since the Node one came in at 1.5G). They also give some number; 4k RPM was their scaleout point for Node (that's not necessarily the maximum, but the point they felt load was sufficiently high to warrant a scaleout; certainly, their graph shows an average latency > 1 second). Rewriting in Rust, that number was raised to 30k RPM; 100 mb of memory, < 40ms average latency (and way better max), and 2.5% of CPU. Given all that, it sounds like, yes, GC was the issue (both high memory and CPU pressure), and with the Rust implementation (no GC) they're nowhere near any CPU or memory limit, and so the 30k is likely a network bottleneck. That said, while I agree that sounds like a terrible metric on the face of it, with what data they've provided (and without anything else), it also sounds like it may be due to they're just operationally dealing with very large amounts of traffic. They may want to consider optimizing the network pipe; not familiar enough with Fargate, but if it's like EC2, there may be a sizing of cpu/memory that also gives you a better network connection (EC2 goes from 1 GBPS to a 10 GBPS network card at one instance type)
hobbescotch 6y ago
Having never dealt with issues relating to garbage collection before, how do you go about diagnosing GC issues in a language where that’s all handled for you?
- the8472 6y ago
  In the java realm you have very fine-grained GC logging that provides insight into the overall behavior of the GC and its different subcomponents. Then there are recording/debugging facilities that allow you to trace allocations, how long objects live, analyze the entire heap (including unreachable but not yet collected objects). And higher-level monitoring APIs separate from the logging and debugging stuff. You can also choose between collectors with different characteristics, trading between overall heap size, latency, utilization of CPU cores and other factors.
- 013a 6y ago
  There are some general tricks that are language-agnostic, like allocating a huge "buffer" object when the app starts, the size of which is some significant portion of the memory you allow the process to use, which always has a reference, then storing references to other objects you need in that big object. In other words, circumvent the garbage collector. Of course, this has its own issues, but I've seen it done in e.g. Go before. Its likely you'll inevitably end up with leaks, but if your service is fungible and can tolerate restarts, basically what you're doing is moving the "GC Pause" to be a "Container Restart" pause, which may be slower, but would happen less often. Some languages have ways to manually call the GC (Node is not one of them, afaik).
  - the8472 6y ago
    Node allows you to call global.gc() if you enable that functionality with a separate argument. In many cases this is an anti-pattern that would make your application behavior worse rather than better, that's why you have to opt into that.
  - eggsnbacon1 6y ago
    > There are some general tricks that are language-agnostic, like allocating a huge "buffer" object when the app starts I've only seen this done in Go :)
foxknox 6y ago
500 requests a second.
- jvehent 6y ago
  Which any programming language can handle easily. The architecture here is more interesting than the language choice.
akoutmos 6y ago
Great article and thanks for sharing! There are a couple of things that stand out at me as possible architecture smells (hopefully this comes across as positive constructive criticism :)). As someone who has been developing on the BEAM for long time now, it usually sticks out like a sore thumb any time I see Elixir/Erlang paired with Redis. Not that there is anything wrong with Redis, but most of the time you can save yourself the additional Ops dependency and application network hop by bringing that state into your application (BEAM languages excel at writing stateful applications). In the article you write that you were using Redis for rate limit checks. You could have very easily bundled that validation into the Elixir application and had for example a single GenServer running per customer that performs the rate limiting validation (I actually wrote a blog post on this using the leaky bucket and token bucket algorithms https://akoutmos.com/post/rate-limiting-with-genservers/ https://akoutmos.com/post/rate-limiting-with-genservers/). Pair this with hot code deployments, you would not lose rate limit values across application deployments. I would be curious to see how much more mileage you could have gotten with that given that the Node application would not have to make network calls to the Elixir service and Redis. Just wanted to share that little tidbit as it is something that I see quite often with people new to the BEAM :). Thanks again for sharing!
- eggsnbacon1 6y ago
  I would push rate limiting to the load balancer, HAProxy or Nginx, but that's just me. If you have a round-robin LB in front you just set each instance to limit at 1/nodes rate, that way you don't have to share any state. If you're load balancing on IP hash you can set each instance to limit at full rate and not worry about it. Shared state in rate limiting becomes a bottleneck very quickly. If you're trying to mitigate spam/DDOS you could easily get 100,000 requests a second. You're going to max out your shared state db way faster than 10gig lines
  - akoutmos 6y ago
    That is definitely a valid route to go so long as your rate limiting is not dependent on much business logic. If rate limiting is per user or per user per instance/service, I would personally bring that kind of concern into the application where it is closer to the persistence layer where those things are defined (and again handling the business logic inside per customer GenServers). I have never used this product so just speculation. But I imagine there is some sort of auth token that valid agents send to tell Bearer that this is a valid/invalid request so that things can be trivially rejected to mitigate a DoS/DDoS to an extent.
trimbo 6y ago
> After some more research, we appeared to be another victim of a memory leak in the AWS Javascript SDK. Did you try using the kinesis REST API directly: https://docs.aws.amazon.com/kinesis/latest/APIReference/API_PutRecord.html https://docs.aws.amazon.com/kinesis/latest/APIReference/API_...
DevKoala 6y ago
There is a couple things I see in this post that I wouldn’t do at all, and I maintain a couple services with orders of magnitude higher QPS. I feel that replacing Node.js with any compiled language would have had the same positive effect.
- ecoqba11 6y ago
  Totally!
newobj 6y ago
500qps. i think the more interesting story here is what language/framework COULDN'T do this, than which one could.
cybervasi 6y ago
GC of 500 request/s could not have possibly caused a performance issue. Most likely the problem was due to JS code holding on to the 1MB requests for the duration of the asynchronous Kinesis request or a bug in the Kinesis JS library itself. With timeout of 2 minutes, you may end up with up to 30K/min x 2min x 1mb = 60GB RAM used. GC would appear running hot during this time but it is only because it is has to scrape more memory somewhere while up to 60gb is being in use.
zerubeus 6y ago
Feels like a HN post being upvoted just bcz it contains Rust in the title (after reader the article) ...

97 points by cfabianski 6y ago

That was a real issue we were struggling to solve. Feels like Rust was the right tool for the right job.

They didn't mention Java as a possible solution, even though its GC's are far better than anything else out there. I have nothing against Rust but if I was at a startup I would save my innovation points for where they're mandatory

JoshTriplett 6y ago
> I have nothing against Rust but if I was at a startup I would save my innovation points for where they're mandatory An article published today, addressing that exact point: https://tim.mcnamara.nz/post/621040767010504704/spend-your-novelty-budget-on-rust https://tim.mcnamara.nz/post/621040767010504704/spend-your-n...
- andrewzah 6y ago
  Ah yes, a 236 word "article", that says to choose "boring old technology" and also to use Rust in the same breath. This article should mention that rust isn't close to ready when it comes to web backends. As much as I love Rust, if I were running a startup or even a decently sized company I would always choose Rails. Now -that- is boring, old... and mature technology. Certain components could get re-written in Rust, certainly, but there's no reason to ignore a mature ecosystem from the start.
  - zozbot234 6y ago
    > This article should mention that rust isn't close to ready when it comes to web backends. Actix-web works just fine. They got a new maintainer team involved that has been spending some time getting rid of all the insane unsafety that was in the code before.
    - andrewzah 6y ago
      Yes. It "works". However deciding to use Actix/Warp means throwing away years and years of work in the rails and ruby world. Rails is mature, robust, and has a huge ecosystem with rubygems. Rust (when it comes to web stuff) is not. "it works" does not pass my litmus test. With Actix/Warp I have to implement stuff by hand that either comes by default with rails or already exists in a gem. I like Rust but I'm not a zealot. People way overestimate performance when they barely have any traffic to begin with as a small startup or even a medium sized company. You could even use rails, and use Rust to write ruby gems instead of going with actix/warp/etc. > insane unsafety This was overblown. Yes, the author didn't respond appropriately, but unsafe isn't inherently dangerous. This is a stupid misconception within the Rust community and caused a lot of unnecessary drama around Actix.
dep_b 6y ago
It's not like Elixir doesn't benefit from a battle hardened VM since it's older and has been used in these kind of high volume scenarios before Java was.
- dnautics 6y ago
  not to mention a GC that you basically don't have to hire an expensive FTE "enterprise dev" to tweak and generally give you good p99s.
andrewzah 6y ago
Rust doesn't have a GC; it uses an ownership model instead. Ultimately in a startup it comes to what people are comfortable with. I personally would use Rust for something like this because I am comfortable with Rust. It's a perfect use case for it as well, imho.
TheTank 6y ago
One step beyond good Java GCs is to write fully zero-GC Java code. The advantage of it is complete control over your performance which means your software is going to be consistently fast. The disadvantage is that it is relatively difficult to obtain. If you want to see an example of fully zero-GC Java, you can check out QuestDB on Github [1] - Disclaimer I work for QuestDB. [1]https://github.com/questdb/questdb https://github.com/questdb/questdb
- masklinn 6y ago
  > One step beyond good Java GCs is to write fully zero-GC Java code. The advantage of it is complete control over your performance which means your software is going to be consistently fast. The disadvantage is that it is relatively difficult to obtain. I don't know that it's actually possible in the general case, as Java's support for value types remains wholly insufficient. IIRC the ixy folks never managed to remove all allocations from the java version.
  - bluestreak 6y ago
    It is possible, we've done it.
lostcolony 6y ago
They ruled out a language because it had stop the world GC, and even if it removed the bottleneck, it would likely become one later. Java has the same issue. Rust does not. Not sure why they'd consider Java, given that concern.
- eggsnbacon1 6y ago
  Shenandoah and ZGC collectors have worse case pauses of ~10ms and average pauses of 0.5ms . The average pause is faster than malloc() sometimes in C, so you won't really be introducing more latency than C does. The other option is to avoid allocating memory at all, which you could do in C/Rust but also in Java. The vast majority of shops given the choice for low/no allocation performance use Java anyways (HFT)
  - lostcolony 6y ago
    But those GCs also don't guarantee all garbage has been collected, nor how much processing time you'll get before they run again. So op could still end up stuck with their code barely executing, due to memory and CPU pressure, and throughput/latency drops to zilch. "The vast majority of shops" is an interesting metric given the vast majority had to pick a language before Rust existed. Java and trying to minimize allocations, vs C/C++, I'd probably choose Java too. Java trying to minimize allocations (no way to guarantee you've done it right), vs Rust (which does guarantee no GC)...I'd probably pick Rust.
    - eggsnbacon1 6y ago
      > But those GCs also don't guarantee all garbage has been collected, nor how much processing time you'll get before they run again. So op could still end up stuck with their code barely executing, due to memory and CPU pressure, and throughput/latency drops to zilch. You could malloc() and free() so much that the code doesn't have time to do anything too. And these operations aren't bounded in time either. Just using C doesn't save you from memory allocation, its just done manually instead of automatic. In every system I've worked on you won't have this kind of GC pressure unless you do something profoundly bad > The vast majority of shops" is an interesting metric given the vast majority had to pick a language before Rust existed. true
zozbot234 6y ago
They did mention Go that also has a modern GC, and does not need to run your app under a software VM. But ultimately, even Go is just another GC language. They had no real need for GC and the like so why use Go, let alone Java?
- eggsnbacon1 6y ago
  Go's GC is notoriously low tech. Its design dates back to papers published in the 70s. Java's Parallel collector is most similar to Go's GC and its deprecated for removal due to poor performance. Go collects VERY frequently to keep average pause times low, which hurts throughput. It also has pathological worst-case pause times, which is what the author ran into. My suggestion of Java is just because its been used for high performance REST for decades. Rust has Actix and a few other frameworks, in Java you have 20+ options. Its a lot easier to get something off the ground when you're 99% sure you won't have to build anything except some glue code. Yes, Rust is faster, but is 30% better performance worth treading the wilds while you're trying to keep a startup afloat, I don't think so personally. Using Java with Shenandoah or ZGC collector would directly handle their GC issue with rather boring technology

didroe 6y ago

I'm one of the engineers that worked on this. It was the first Rust production app code I've written so it was a really fun project.

mamcx 6y ago
One of the interesting effects of using rust is saving money! I also migrate a F#/.NET ecommerce backend and can run in less RAM/CPU that make my bills lower.
- chrischen 6y ago
  Does Rust offer much of the same language features as F#?
  - penagwin 6y ago
    Disclaimer: not a rust of .net dev, but my impression is they’re very different in what they’re trying to. It’d would be like comparing C to Python (as an example, not as an analogy). Rust is closer to a super fancy C, compiles natively and was made with a heavily focus on certain types of memory safety. F# has syntax more like Haskell/ML, and is compiled to a bytecode instead of an executable. It runs on .NET and everything that entails.
    - chrischen 6y ago
      That's actually why I'm asking about that since the OP is transitioning backend logic from F# to Rust. So either Rust can cover a lot of the language features or he has a use case that really warrants that performance (and F# on .NET is not exactly non-performant).
  - mamcx 6y ago
    Near. Exist some fancy features that are not present, but the practical side is covered...
- raphinou 6y ago

meritt 6y ago

Sorry, I must be missing something in this blog post because the requirements here sound incredibly minimal. You just needed an HTTP service (sitting behind an Envoy proxy) to process a mere 500 requests/second (up to 1MB payload) and pipe them to Kinesis? How much data preparation is happening in Rust? It sounds like all the permission/rate-limiting/etc happens between Envoy/Redis before it ever reaches Rust? I know this comes across as snarky but it really worries me that contemporary engineers think this is a feat worthy of a blog post. For example, take this book from 2003 [1] talking about Apache + mod_perl. Page 325 [2] shows a benchmark: "As you can see, the server was able to respond on average to 856 requests per second... and 10 milliseconds to process each request". And just to show this isn't a NodeJS vs Rust thing, check out these webframework benchmarks using various JS frameworks [3]. The worst performer on there still does >500 rps while the best does 500,000. It's 2020, the bar needs to be much higher. [1] https://www.amazon.com/Practical-mod_perl-Stas-Bekman/dp/0596002270 https://www.amazon.com/Practical-mod_perl-Stas-Bekman/dp/059... [2] https://books.google.com/books?id=i3Ww_7a2Ff4C&pg=PT356&lpg=PT356 https://books.google.com/books?id=i3Ww_7a2Ff4C&pg=PT356&lpg=... [3] https://www.techempower.com/benchmarks/#section=data-r19&hw=ph&test=db&l=zik0sf-1r https://www.techempower.com/benchmarks/#section=data-r19&hw=...

BubRoss 6y ago
I hope some day doing something trivial using rust will no longer warrant a hacker news post to promote a startup.
lostcolony 6y ago
They list out what is being done by the service - "It would receive the logs, communicate with an elixir service to check customer access rights, check rate limits using Redis, and then send the log to CloudWatch. There, it would trigger an event to tell our processing worker to take over." That sounds like a decent amount of work for a service, and without more detail it's very hard to say whether or not a given level is efficient or inefficient (we don't know exactly what was being done; we can assume that they're using pretty small Fargate instances though since the Node one came in at 1.5G). They also give some number; 4k RPM was their scaleout point for Node (that's not necessarily the maximum, but the point they felt load was sufficiently high to warrant a scaleout; certainly, their graph shows an average latency > 1 second). Rewriting in Rust, that number was raised to 30k RPM; 100 mb of memory, < 40ms average latency (and way better max), and 2.5% of CPU. Given all that, it sounds like, yes, GC was the issue (both high memory and CPU pressure), and with the Rust implementation (no GC) they're nowhere near any CPU or memory limit, and so the 30k is likely a network bottleneck. That said, while I agree that sounds like a terrible metric on the face of it, with what data they've provided (and without anything else), it also sounds like it may be due to they're just operationally dealing with very large amounts of traffic. They may want to consider optimizing the network pipe; not familiar enough with Fargate, but if it's like EC2, there may be a sizing of cpu/memory that also gives you a better network connection (EC2 goes from 1 GBPS to a 10 GBPS network card at one instance type)

hobbescotch 6y ago

Having never dealt with issues relating to garbage collection before, how do you go about diagnosing GC issues in a language where that’s all handled for you?

the8472 6y ago
In the java realm you have very fine-grained GC logging that provides insight into the overall behavior of the GC and its different subcomponents. Then there are recording/debugging facilities that allow you to trace allocations, how long objects live, analyze the entire heap (including unreachable but not yet collected objects). And higher-level monitoring APIs separate from the logging and debugging stuff. You can also choose between collectors with different characteristics, trading between overall heap size, latency, utilization of CPU cores and other factors.
013a 6y ago
There are some general tricks that are language-agnostic, like allocating a huge "buffer" object when the app starts, the size of which is some significant portion of the memory you allow the process to use, which always has a reference, then storing references to other objects you need in that big object. In other words, circumvent the garbage collector. Of course, this has its own issues, but I've seen it done in e.g. Go before. Its likely you'll inevitably end up with leaks, but if your service is fungible and can tolerate restarts, basically what you're doing is moving the "GC Pause" to be a "Container Restart" pause, which may be slower, but would happen less often. Some languages have ways to manually call the GC (Node is not one of them, afaik).
- the8472 6y ago
  Node allows you to call global.gc() if you enable that functionality with a separate argument. In many cases this is an anti-pattern that would make your application behavior worse rather than better, that's why you have to opt into that.
- eggsnbacon1 6y ago
  > There are some general tricks that are language-agnostic, like allocating a huge "buffer" object when the app starts I've only seen this done in Go :)

foxknox 6y ago

500 requests a second.

jvehent 6y ago
Which any programming language can handle easily. The architecture here is more interesting than the language choice.

akoutmos 6y ago

Great article and thanks for sharing! There are a couple of things that stand out at me as possible architecture smells (hopefully this comes across as positive constructive criticism :)). As someone who has been developing on the BEAM for long time now, it usually sticks out like a sore thumb any time I see Elixir/Erlang paired with Redis. Not that there is anything wrong with Redis, but most of the time you can save yourself the additional Ops dependency and application network hop by bringing that state into your application (BEAM languages excel at writing stateful applications). In the article you write that you were using Redis for rate limit checks. You could have very easily bundled that validation into the Elixir application and had for example a single GenServer running per customer that performs the rate limiting validation (I actually wrote a blog post on this using the leaky bucket and token bucket algorithms https://akoutmos.com/post/rate-limiting-with-genservers/ https://akoutmos.com/post/rate-limiting-with-genservers/). Pair this with hot code deployments, you would not lose rate limit values across application deployments. I would be curious to see how much more mileage you could have gotten with that given that the Node application would not have to make network calls to the Elixir service and Redis. Just wanted to share that little tidbit as it is something that I see quite often with people new to the BEAM :). Thanks again for sharing!

eggsnbacon1 6y ago
I would push rate limiting to the load balancer, HAProxy or Nginx, but that's just me. If you have a round-robin LB in front you just set each instance to limit at 1/nodes rate, that way you don't have to share any state. If you're load balancing on IP hash you can set each instance to limit at full rate and not worry about it. Shared state in rate limiting becomes a bottleneck very quickly. If you're trying to mitigate spam/DDOS you could easily get 100,000 requests a second. You're going to max out your shared state db way faster than 10gig lines
- akoutmos 6y ago
  That is definitely a valid route to go so long as your rate limiting is not dependent on much business logic. If rate limiting is per user or per user per instance/service, I would personally bring that kind of concern into the application where it is closer to the persistence layer where those things are defined (and again handling the business logic inside per customer GenServers). I have never used this product so just speculation. But I imagine there is some sort of auth token that valid agents send to tell Bearer that this is a valid/invalid request so that things can be trivially rejected to mitigate a DoS/DDoS to an extent.

trimbo 6y ago

> After some more research, we appeared to be another victim of a memory leak in the AWS Javascript SDK. Did you try using the kinesis REST API directly: https://docs.aws.amazon.com/kinesis/latest/APIReference/API_PutRecord.html https://docs.aws.amazon.com/kinesis/latest/APIReference/API_...

DevKoala 6y ago

There is a couple things I see in this post that I wouldn’t do at all, and I maintain a couple services with orders of magnitude higher QPS. I feel that replacing Node.js with any compiled language would have had the same positive effect.

ecoqba11 6y ago
Totally!

newobj 6y ago

500qps. i think the more interesting story here is what language/framework COULDN'T do this, than which one could.

cybervasi 6y ago

GC of 500 request/s could not have possibly caused a performance issue. Most likely the problem was due to JS code holding on to the 1MB requests for the duration of the asynchronous Kinesis request or a bug in the Kinesis JS library itself. With timeout of 2 minutes, you may end up with up to 30K/min x 2min x 1mb = 60GB RAM used. GC would appear running hot during this time but it is only because it is has to scrape more memory somewhere while up to 60gb is being in use.

zerubeus 6y ago

Feels like a HN post being upvoted just bcz it contains Rust in the title (after reader the article) ...