6 ms·

YJIT is the most memory-efficient Ruby JIT

228 points by panic 3y ago

JohnBooty 3y ago
Wow, Shopify continues to make some heroic improvements here. Kudos, kudos, kudos. Thanks, Shopify folks. One thing I didn't see discussed in the article was YJIT's memory usage relative to CRuby, the baseline non-JIT version of Ruby. It is certainly possible I missed it; that's been known to happen! Anyway, the news there is very good. We can see detailed information here: https://speed.yjit.org/memory_timeline.html#railsbench https://speed.yjit.org/memory_timeline.html#railsbench Currently Railsbench consumes a peak of ~95MB with CRuby, and a peak of ~110MB with YJIT. So, YJIT delivers 70% more performance while consuming 16% more RAM here. That is a tradeoff I think most people would gladly accept in most scenarios. =) Real-world speedups will be less, since a "real" web application spends much of its time waiting for the database and other external resources. As the article notes, Shopify's real-world observed storefront perf gain is 27.2%. YJIT is a success and its future is even brighter.
- shadowbanned0 3y ago
  [dead]
- deleted 3y ago
  [deleted]
- jez 3y ago
  The Memory Usage on Benchmarks section shows CRuby as one of the bars. It is the lowest across the board. YJIT, being implemented in CRuby, uses at least as much memory as CRuby in all the benchmarks (you could maybe imagine one day JIT’d code using less memory in specific situations by compiling code to a more efficient algorithm that skips allocations the interpreter would make, but it appears that is not the case yet today).
compumike 3y ago
Also for a practical tip on YJIT memory usage, note that there is a "--yjit-exec-mem-size" option, see https://github.com/ruby/ruby/blob/master/doc/yjit/yjit.md#command-line-options https://github.com/ruby/ruby/blob/master/doc/yjit/yjit.md#co... for more details. (This command-line argument is mentioned in the paper https://dl.acm.org/doi/10.1145/3617651.3622982 https://dl.acm.org/doi/10.1145/3617651.3622982 but not in this blog post about the paper.) At Heii On-Call https://heiioncall.com/ https://heiioncall.com/ we use: ENV RUBY_YJIT_ENABLE=1 ENV RUBYOPT=--yjit-exec-mem-size=16 in our Dockerfile for our Rails processes.
- booleanbetrayal 3y ago
  Any recollection on how you arrived at the --yjit-exec-mem-size value? We've been running YJIT in production for some time, but haven't looked into tuning this at all.
  - JohnBooty 3y ago
    Not parent poster and do not have production YJIT experience. =) My guess is that you would monitor `RubyVM::YJIT.runtime_stats[:code_region_size]` and/or `RubyVM::YJIT.runtime_stats[:code_gc_count]` so that you can get a feel for a reasonable value for your application, as well as know whether or not the "code GC" is running frequently. https://github.com/ruby/ruby/blob/master/doc/yjit/yjit.md#performance-tips-for-production-deployments https://github.com/ruby/ruby/blob/master/doc/yjit/yjit.md#pe...
    - compumike 3y ago
      That’s exactly right. Our code_region_size levels off a bit under 8 MB, so we set the limit to 16. In practice we see code_gc_count stays at 0.
      - booleanbetrayal 3y ago
        Will have to look into this. Thanks for the suggestion!
- JohnBooty 3y ago
  Wow, that's interesting and it seems a little crazy? From the docs: When JIT code size (RubyVM::YJIT.runtime_stats[:code_region_size]) reaches this value, YJIT triggers "code GC" that frees all JIT code and starts recompiling everything. Compiling code takes some time, so scheduling code GC too frequently slows down your application. Increasing --yjit-exec-mem-size may speed up your application if RubyVM::YJIT.runtime_stats[:code_gc_count] is not 0 or 1. https://github.com/ruby/ruby/blob/master/doc/yjit/yjit.md#command-line-options https://github.com/ruby/ruby/blob/master/doc/yjit/yjit.md#co... It just dumps all the JIT-compiled code? I'd expect to see some kind of heuristic or algorithm there... LFU or something. The internals of a JIT are essentially black magic to me, and I know the people working on YJIT are super talented, so I am sure there is a good reason why they just dump everything instead of the least-frequently used stuff. Maybe the overhead of trying frecency outweighs the gains, maybe they just haven't implemented it yet, or maybe it's just a rarely-reached condition. (I hope a YJIT team member sees this, I'm super curious now)
deleted 3y ago
[deleted]
- deleted 3y ago
  [deleted]
Freaky 3y ago
> We were very generous in terms of warm-up time. Each benchmark was run for 1000 iterations, and the first half of all the iterations were discarded as warm-up time, giving each JIT a more than fair chance to reach peak performance. 1,000 iterations isn't remotely generous for JRuby, unfortunately - JVM's Tier-3 compilation only kicks in by default around 2,000, and full tier-4 is only considered beyond 15,000. I've observed this to have quite a substantial effect, for instance bringing manticore (JRuby wrapper for Apache's Java HttpClient) down from merely "okay" performance after 10,000 requests to pretty much matching the curb C extension under MRI after 20,000. You can tweak it to be more aggressive, but I guess this puts more pressure on the compiler threads and their memory use, while reducing the run-time profiling data they use to optimize most effectively. It perhaps also risks more churn from deoptimization. I kind of felt like I'd be better off trying to formalise the warmup process. It's rather a shame that all this warmup work is one-shot. It would be far less obnoxious if it could be preserved across runs - I believe some alternative Java runtimes support something like that, though given JRuby's got its own JIT targetting Java bytecode I dare say it would require work there as well.
- maxime_cb 3y ago
  It is enough iterations for these VMs to warm up on the benchmarks we've looked at, but the warm-up time is still on the order of minutes on some benchmarks, which is impractical for many applications.
- Twirrim 3y ago
  Agreed, the JVM's thresholds for method compilation is higher than the test strategy seemed to account for. Also, quoting from the site: TruffleRuby eventually outperforms the other JITs, but it takes about two minutes to do so. It is also initially quite a bit slower than the CRuby interpreter, taking over 110 seconds to catch up to the interpreter’s speed. This would be problematic in a production context such as Shopify’s, because it can lead to much slower response times for some customers, which could translate into lost business. There's always ways around that, for example pushing artificial traffic at a node as part of a deployment process, prior to exposing it to customers. I've known places that have opted for that approach, because it made the best sense for them. The initial latency hit of JIT warm-up wasn't a good fit for their needs, while every other aspect of using a JIT'd language was. As ever, depends on the trade-off if it's worth the extra work to do that. e.g. if I could see after 5-10 minutes that TruffleRuby was, say, 25% faster than YJIT, then that extra engineering effort may be the right choice. edit: Some folks throw traffic at nodes before exposing them to customers to ensure that their caches are warm, too. It's not necessarily something limited to JIT'd languages/runtimes.
titzer 3y ago
Nice work! JIT compilers are a multi-dimensional tradeoff space: memory consumption, startup time, and execution time. I've been experimenting with different ways of visualizing these three dimensions in a tradeoff space. I like the iteration time over time (showing warmup time) but that graph takes a lot of space and is necessarily per-benchmark (and only works with benchmarks with clearly repeated iterations). In my recent JIT paper I went for a scatter plot of two dimensions (setup time and code quality). It's here if anyone's curious: https://arxiv.org/abs/2305.13241 https://arxiv.org/abs/2305.13241
- pjmlp 3y ago
  I always like collecting such papers, thanks.
alberth 3y ago
Seems like Truffle Ruby is 50-100% faster in most tests. But uses 5-10x more memory.
xiaodai 3y ago
efficient jit is an oxymoron in the ruby world
larodi 3y ago
If only Perl crowd focused on stuff like JIT improvement and cross compilations, rather than Raku madness, the lang could've been running strong in 2023. Hate to admit it, but rubyists got this all better.
- lizmat 3y ago
  I can assure you, nobody in the Perl crowd is doing any Raku.
  - larodi 3y ago
    And never did.
the-alchemist 3y ago
Oh, benchmarking. Great work, but maybe I'm missing something: I'm more interested in "performance per MB", i.e., per dollar USD, versus "how much memory does it use?" I feel like that's not fair to systems that will trade performance and memory usage if given the chance. If you give Linux 10GB of memory and only use 2GB, it'll use the rest as a page cache, or memory cache, or anything. Unused RAM is wasted RAM. Same for the JVM. Maybe it's deep in the weeds, but I don't see any setting of max memory (-Xmx, or -mx in the Java world). Same as any operation system, if you give it more memory, it'll use it. Also, just like you'd want to generate and pre-warm your caches, you'd want to use the same for a JIT, a cache of sorts. I'd also like to see these benchmarks using GraalVM native image, which tends to use a lot less memory and reach peak speeds much much faster (seconds, instead of minutes).

228 points by panic 3y ago

Wow, Shopify continues to make some heroic improvements here. Kudos, kudos, kudos. Thanks, Shopify folks. One thing I didn't see discussed in the article was YJIT's memory usage relative to CRuby, the baseline non-JIT version of Ruby. It is certainly possible I missed it; that's been known to happen! Anyway, the news there is very good. We can see detailed information here: https://speed.yjit.org/memory_timeline.html#railsbench https://speed.yjit.org/memory_timeline.html#railsbench Currently Railsbench consumes a peak of ~95MB with CRuby, and a peak of ~110MB with YJIT. So, YJIT delivers 70% more performance while consuming 16% more RAM here. That is a tradeoff I think most people would gladly accept in most scenarios. =) Real-world speedups will be less, since a "real" web application spends much of its time waiting for the database and other external resources. As the article notes, Shopify's real-world observed storefront perf gain is 27.2%. YJIT is a success and its future is even brighter.

shadowbanned0 3y ago
[dead]
deleted 3y ago
[deleted]
jez 3y ago
The Memory Usage on Benchmarks section shows CRuby as one of the bars. It is the lowest across the board. YJIT, being implemented in CRuby, uses at least as much memory as CRuby in all the benchmarks (you could maybe imagine one day JIT’d code using less memory in specific situations by compiling code to a more efficient algorithm that skips allocations the interpreter would make, but it appears that is not the case yet today).

compumike 3y ago

Also for a practical tip on YJIT memory usage, note that there is a "--yjit-exec-mem-size" option, see https://github.com/ruby/ruby/blob/master/doc/yjit/yjit.md#command-line-options https://github.com/ruby/ruby/blob/master/doc/yjit/yjit.md#co... for more details. (This command-line argument is mentioned in the paper https://dl.acm.org/doi/10.1145/3617651.3622982 https://dl.acm.org/doi/10.1145/3617651.3622982 but not in this blog post about the paper.) At Heii On-Call https://heiioncall.com/ https://heiioncall.com/ we use: ENV RUBY_YJIT_ENABLE=1 ENV RUBYOPT=--yjit-exec-mem-size=16 in our Dockerfile for our Rails processes.

booleanbetrayal 3y ago
Any recollection on how you arrived at the --yjit-exec-mem-size value? We've been running YJIT in production for some time, but haven't looked into tuning this at all.
- JohnBooty 3y ago
  Not parent poster and do not have production YJIT experience. =) My guess is that you would monitor `RubyVM::YJIT.runtime_stats[:code_region_size]` and/or `RubyVM::YJIT.runtime_stats[:code_gc_count]` so that you can get a feel for a reasonable value for your application, as well as know whether or not the "code GC" is running frequently. https://github.com/ruby/ruby/blob/master/doc/yjit/yjit.md#performance-tips-for-production-deployments https://github.com/ruby/ruby/blob/master/doc/yjit/yjit.md#pe...
  - compumike 3y ago
    That’s exactly right. Our code_region_size levels off a bit under 8 MB, so we set the limit to 16. In practice we see code_gc_count stays at 0.
    - booleanbetrayal 3y ago
      Will have to look into this. Thanks for the suggestion!
JohnBooty 3y ago
Wow, that's interesting and it seems a little crazy? From the docs: When JIT code size (RubyVM::YJIT.runtime_stats[:code_region_size]) reaches this value, YJIT triggers "code GC" that frees all JIT code and starts recompiling everything. Compiling code takes some time, so scheduling code GC too frequently slows down your application. Increasing --yjit-exec-mem-size may speed up your application if RubyVM::YJIT.runtime_stats[:code_gc_count] is not 0 or 1. https://github.com/ruby/ruby/blob/master/doc/yjit/yjit.md#command-line-options https://github.com/ruby/ruby/blob/master/doc/yjit/yjit.md#co... It just dumps all the JIT-compiled code? I'd expect to see some kind of heuristic or algorithm there... LFU or something. The internals of a JIT are essentially black magic to me, and I know the people working on YJIT are super talented, so I am sure there is a good reason why they just dump everything instead of the least-frequently used stuff. Maybe the overhead of trying frecency outweighs the gains, maybe they just haven't implemented it yet, or maybe it's just a rarely-reached condition. (I hope a YJIT team member sees this, I'm super curious now)

deleted 3y ago

[deleted]

deleted 3y ago
[deleted]

Freaky 3y ago

> We were very generous in terms of warm-up time. Each benchmark was run for 1000 iterations, and the first half of all the iterations were discarded as warm-up time, giving each JIT a more than fair chance to reach peak performance. 1,000 iterations isn't remotely generous for JRuby, unfortunately - JVM's Tier-3 compilation only kicks in by default around 2,000, and full tier-4 is only considered beyond 15,000. I've observed this to have quite a substantial effect, for instance bringing manticore (JRuby wrapper for Apache's Java HttpClient) down from merely "okay" performance after 10,000 requests to pretty much matching the curb C extension under MRI after 20,000. You can tweak it to be more aggressive, but I guess this puts more pressure on the compiler threads and their memory use, while reducing the run-time profiling data they use to optimize most effectively. It perhaps also risks more churn from deoptimization. I kind of felt like I'd be better off trying to formalise the warmup process. It's rather a shame that all this warmup work is one-shot. It would be far less obnoxious if it could be preserved across runs - I believe some alternative Java runtimes support something like that, though given JRuby's got its own JIT targetting Java bytecode I dare say it would require work there as well.

maxime_cb 3y ago
It is enough iterations for these VMs to warm up on the benchmarks we've looked at, but the warm-up time is still on the order of minutes on some benchmarks, which is impractical for many applications.
Twirrim 3y ago
Agreed, the JVM's thresholds for method compilation is higher than the test strategy seemed to account for. Also, quoting from the site: TruffleRuby eventually outperforms the other JITs, but it takes about two minutes to do so. It is also initially quite a bit slower than the CRuby interpreter, taking over 110 seconds to catch up to the interpreter’s speed. This would be problematic in a production context such as Shopify’s, because it can lead to much slower response times for some customers, which could translate into lost business. There's always ways around that, for example pushing artificial traffic at a node as part of a deployment process, prior to exposing it to customers. I've known places that have opted for that approach, because it made the best sense for them. The initial latency hit of JIT warm-up wasn't a good fit for their needs, while every other aspect of using a JIT'd language was. As ever, depends on the trade-off if it's worth the extra work to do that. e.g. if I could see after 5-10 minutes that TruffleRuby was, say, 25% faster than YJIT, then that extra engineering effort may be the right choice. edit: Some folks throw traffic at nodes before exposing them to customers to ensure that their caches are warm, too. It's not necessarily something limited to JIT'd languages/runtimes.

titzer 3y ago

Nice work! JIT compilers are a multi-dimensional tradeoff space: memory consumption, startup time, and execution time. I've been experimenting with different ways of visualizing these three dimensions in a tradeoff space. I like the iteration time over time (showing warmup time) but that graph takes a lot of space and is necessarily per-benchmark (and only works with benchmarks with clearly repeated iterations). In my recent JIT paper I went for a scatter plot of two dimensions (setup time and code quality). It's here if anyone's curious: https://arxiv.org/abs/2305.13241 https://arxiv.org/abs/2305.13241

pjmlp 3y ago
I always like collecting such papers, thanks.

alberth 3y ago

Seems like Truffle Ruby is 50-100% faster in most tests. But uses 5-10x more memory.

xiaodai 3y ago

efficient jit is an oxymoron in the ruby world

larodi 3y ago

If only Perl crowd focused on stuff like JIT improvement and cross compilations, rather than Raku madness, the lang could've been running strong in 2023. Hate to admit it, but rubyists got this all better.

lizmat 3y ago
I can assure you, nobody in the Perl crowd is doing any Raku.
- larodi 3y ago
  And never did.

the-alchemist 3y ago

Oh, benchmarking. Great work, but maybe I'm missing something: I'm more interested in "performance per MB", i.e., per dollar USD, versus "how much memory does it use?" I feel like that's not fair to systems that will trade performance and memory usage if given the chance. If you give Linux 10GB of memory and only use 2GB, it'll use the rest as a page cache, or memory cache, or anything. Unused RAM is wasted RAM. Same for the JVM. Maybe it's deep in the weeds, but I don't see any setting of max memory (-Xmx, or -mx in the Java world). Same as any operation system, if you give it more memory, it'll use it. Also, just like you'd want to generate and pre-warm your caches, you'd want to use the same for a JIT, a cache of sorts. I'd also like to see these benchmarks using GraalVM native image, which tends to use a lot less memory and reach peak speeds much much faster (seconds, instead of minutes).