16 ms·

Well, if you have a couple of decades of experience with low-level programming, you know that AOT compilation and non-moving pointers carry intrinsic runtime ov

by pron 15d ago

Well, if you have a couple of decades of experience with low-level programming, you know that AOT compilation and non-moving pointers carry intrinsic runtime overheads that manifest as programs grow large and complex, and run for a long time. It's these very overheads that moving collectors and JIT compilers are designed to reduce, and it's also the very thing benchmarks don't measure. A TCMalloc runtime is almost the same size as Java's most sophisticated GC, and it still can't keep up because of the fundamental overheads. In low-level programming we try to avoid these overheads by avoiding dynamic dispatch and dynamic heap memory, but it gets harder as the program evolves. It's true that even in such programs you could, in principle, reach the same level of performance of Java in C++, but in practice it's very, very hard. This is why most large and long-running programs have abandoned low-level programming languages. It's easy to get excellent performance when the code is small, regular, and new, but over time it gets harder and harder. In general, low-level programming languages yield relatively fast small programs, but relatively slow large programs, and with Java/C# it's generally the opposite. The low-level control that helps performance when you're small, starts hurting it when you're big.

afdbcreid 15d ago
I know that. I also know that JITs cannot optimize to the same amount as LLVM due to the time limit, and that C++ and Rust are allocating much, much less than Java and even C# or Go, so a faster allocation scheme is much less needed there. I'm not saying that faster allocation or fragmentation cannot yield gains for some specific programs, but even in those cases it's usually possible to alleviate the costs with wise organization of allocations (including using arenas etc. in some places), and they're also offloaded from the better-optimizing compiler.
- pron 15d ago
  > I know that. I also know that JITs cannot optimize to the same amount as LLVM due to the time limit Yeah, this is not true, and there's no time limit. I mean, maybe some JIT compilers, like JavaScript's have a time limit, but their goal is to run JS at an acceptable speed. Java's JIT is intended to reduce the runtime overheads of AOT compilers, and the only way to do that is by optimising significantly more than AOT compilers, obviously not less (otherwise, we'd just always use an AOT compiler). You can easily see why there's no time limit if you understood how Java's optimising JIT works. First, code is run in the interpreter and some profiles are collected, then a non-optimising JIT runs and continues to collect profile, and finally the optimising JIT runs. The vast majority of the time is spent waiting for profiles to collect, and so if compilation itself runs, say, even 3x slower, it won't even be perceptible. Also, because we have profiles, we don't have to compile much of the program at all, because we know what the hot spots are. Initialisation code that runs once is never compiled (remember, the focus is long-running programs, exactly those that low-level languages have trouble with). Finally, the reason sophisticated JIT compilers can optimise more - which is why they're used in the first place - is thanks to speculative optimisation. AOT compilers need to spend a lot of time on optimisation, and even then they are limited, because they need to prove that the program transformation is valid (i.e. that there's no miscompilation). The power of JIT compilers is that they don't. They only need to speculate that a certain profile will continue to be in effect. So if so far some virtual call always hits a certain target, they can go ahead and inline it (not only to the cost of a regular call, but to no call at all, and then they optimise the whole inlined code). If they're wrong, a fault triggers and they decompile the relevant subroutine going back to the interpreter and non-optimising compiler. > and that C++ and Rust are allocating much, much less than Java and even C# or Go, so a faster allocation scheme is much less needed there This is true, but the causaility here is that the reason we avoid allocation in C++ is precisely because it's so slow. > but even in those cases it's usually possible to alleviate the costs with wise organization of allocations The problem is that this is true in principle. In practice this is certainly true in smaller programs. In larger programs, this work is not easy at all, and you find yourself doing harder and harder work just to keep up. > including using arenas etc. in some places One of the reasons I'm excited about Zig (I'm a low-level programmer) is that it makes arenas much more viable. Arenas in C++ and especially Rust are not really a pleasure to work with, and they're viral and a constant maintenance burden. BTW, the reason moving GCs are so fast is that they work quite similarly to arenas. > and they're also offloaded from the better-optimizing compiler. It's a worse-optimising compiler. In C++, I use templates to achieve similar optimisation to what Java does, and in Zig I can use comptime, and again, it's certainly possible but it's hard work. You can't let the templates explode all over the codebase, and, as it evolves, you have to go back and profile and take out the ones that no longer help, replacing them with new ones. Just to tell you a bit about me, I was a C++ programmer for many years, and when Java showed up, like many, I was sceptical. When I saw that the JIT + moving collector hypothesis actually accomplishes its goal in reducing the overheads we were seeing in C++ in many situations, I went to work on the JVM. Back then there were still latency tradeoffs due to GC pauses, but GC pauses no longer exist as of three years ago. Now, a lot of people, including some of the world's top compiler and memory management experts, believe that the vision of using JITs and moving collectors to address the performance problem of low-level languages is working exceedingly well. We can certainly argue about under which conditions Java wins and under which C++ (or Zig it Rust etc.) win and how common they are, but people who think low-level languages win across the board or almost across the board clearly don't know what's going on. Early on it was people who were sceptical about how effectively JITs and moving collectors could do their job in practice (even though the theory was clear), but these days I think it's mostly people who haven't struggled with performance issues in low-level languages long enough, and just see that for small or young programs they work fine. They always were. Writing a new program in C++ was never harder than writing a new program in Java, and the performance was great (and people weren't concerned about memory safety in particular). The problems came later - in the 5th year, the 10th year, etc.., when the cost of evolution and trying to keep performance good were piling up.
  - joas_coder 7d ago
    [dead]
afdbcreid 15d ago
Also: > This is why most large and long-running programs have abandoned low-level programming languages. That's not true, as evidenced by the fact that this move has started before extremely sophisticated JIT compilers or garbage collectors were invented. The reason was not because managed languages were faster or even had equal speed, but because of the costs associated with memory unsafety (not just security), exactly what Rust prevents (which was of course not available then). You can see empirical evidence of this, for example, by the post about Aurora DSQL rewrite in Rust (https://www.allthingsdistributed.com/2025/05/just-make-it-scale-an-aurora-dsql-story.html https://www.allthingsdistributed.com/2025/05/just-make-it-sc...). One notable quote: > But after a few weeks, it compiled and the results surprised us. The code was 10x faster than our carefully tuned Kotlin implementation – despite no attempt to make it faster. To put this in perspective, we had spent years incrementally improving the Kotlin version from 2,000 to 3,000 transactions per second (TPS). The Rust version, written by Java developers who were new to the language, clocked 30,000 TPS. You also ignore the impact of memory usage, where unmanaged language have an even greater edge (yes I know it is possible to optimize managed languages' memory consumption as well. Not to the same amount and often at the expense of speed).
- pron 15d ago
  > That's not true, as evidenced by the fact that this move has started before extremely sophisticated JIT compilers or garbage collectors were invented. I don't know how long you've been programming, but that's not true. In the late nineties and early aughts I was working on large, performance-critical, soft- and hard-realtime systems, and we only started moving away from C++ when Java started beating its performance. > The reason was not because managed languages were faster or even had equal speed, but because of the costs associated with memory unsafety (not just security), exactly what Rust prevents (which was of course not available then). That's a myth, and a fairly recent one. Sure, there were non-performance-sensitive programs written in slow languages for a long time. But the industry was mostly using C++ for anything that needed to be big and fast, and back then "memory safety" was mostly just another type of bug. It was nowhere near reason enough to use slow languages, which is why we didn't use them. Lack of memory safety is a serious problem, but the claim that it's the biggest issue with C++, let alone the one that's always been considered the biggest issue, is just a myth. Back then it was certainly considered no bigger an issue than the language complexity, compilation time, and even performance issues in large, long-running programs. > You can see empirical evidence of this, for example, by the post about Aurora DSQL rewrite in Rust I talk to the people at AWS, and this is not the evidence you think it is. First, their problem was primarily with GC pauses, and it was before pauseless GCs. Second, the codebase isn't very big. Third, because Java and C++/Rust offer similar performance - sometimes one wins, sometimes another - you expect to see exactly that. I can tell you that we recently wrote a distributed cache in both Java and Rust simultaneously (using the pauseless GC). The Java version achieved twice the throughput of the Rust version, and significanly better latency across all percentiles. So sure, on the smaller end, there are programs where Rust would be 2x as fast as Java, there are programs where Java would be 2x as fast as Rust, and on average they're about the same. But over time, Java's advantage starts to show as it makes it easier to keep the good performance over years of evolution.

by pron 15d ago

afdbcreid 15d ago

I know that. I also know that JITs cannot optimize to the same amount as LLVM due to the time limit, and that C++ and Rust are allocating much, much less than Java and even C# or Go, so a faster allocation scheme is much less needed there. I'm not saying that faster allocation or fragmentation cannot yield gains for some specific programs, but even in those cases it's usually possible to alleviate the costs with wise organization of allocations (including using arenas etc. in some places), and they're also offloaded from the better-optimizing compiler.

pron 15d ago
> I know that. I also know that JITs cannot optimize to the same amount as LLVM due to the time limit Yeah, this is not true, and there's no time limit. I mean, maybe some JIT compilers, like JavaScript's have a time limit, but their goal is to run JS at an acceptable speed. Java's JIT is intended to reduce the runtime overheads of AOT compilers, and the only way to do that is by optimising significantly more than AOT compilers, obviously not less (otherwise, we'd just always use an AOT compiler). You can easily see why there's no time limit if you understood how Java's optimising JIT works. First, code is run in the interpreter and some profiles are collected, then a non-optimising JIT runs and continues to collect profile, and finally the optimising JIT runs. The vast majority of the time is spent waiting for profiles to collect, and so if compilation itself runs, say, even 3x slower, it won't even be perceptible. Also, because we have profiles, we don't have to compile much of the program at all, because we know what the hot spots are. Initialisation code that runs once is never compiled (remember, the focus is long-running programs, exactly those that low-level languages have trouble with). Finally, the reason sophisticated JIT compilers can optimise more - which is why they're used in the first place - is thanks to speculative optimisation. AOT compilers need to spend a lot of time on optimisation, and even then they are limited, because they need to prove that the program transformation is valid (i.e. that there's no miscompilation). The power of JIT compilers is that they don't. They only need to speculate that a certain profile will continue to be in effect. So if so far some virtual call always hits a certain target, they can go ahead and inline it (not only to the cost of a regular call, but to no call at all, and then they optimise the whole inlined code). If they're wrong, a fault triggers and they decompile the relevant subroutine going back to the interpreter and non-optimising compiler. > and that C++ and Rust are allocating much, much less than Java and even C# or Go, so a faster allocation scheme is much less needed there This is true, but the causaility here is that the reason we avoid allocation in C++ is precisely because it's so slow. > but even in those cases it's usually possible to alleviate the costs with wise organization of allocations The problem is that this is true in principle. In practice this is certainly true in smaller programs. In larger programs, this work is not easy at all, and you find yourself doing harder and harder work just to keep up. > including using arenas etc. in some places One of the reasons I'm excited about Zig (I'm a low-level programmer) is that it makes arenas much more viable. Arenas in C++ and especially Rust are not really a pleasure to work with, and they're viral and a constant maintenance burden. BTW, the reason moving GCs are so fast is that they work quite similarly to arenas. > and they're also offloaded from the better-optimizing compiler. It's a worse-optimising compiler. In C++, I use templates to achieve similar optimisation to what Java does, and in Zig I can use comptime, and again, it's certainly possible but it's hard work. You can't let the templates explode all over the codebase, and, as it evolves, you have to go back and profile and take out the ones that no longer help, replacing them with new ones. Just to tell you a bit about me, I was a C++ programmer for many years, and when Java showed up, like many, I was sceptical. When I saw that the JIT + moving collector hypothesis actually accomplishes its goal in reducing the overheads we were seeing in C++ in many situations, I went to work on the JVM. Back then there were still latency tradeoffs due to GC pauses, but GC pauses no longer exist as of three years ago. Now, a lot of people, including some of the world's top compiler and memory management experts, believe that the vision of using JITs and moving collectors to address the performance problem of low-level languages is working exceedingly well. We can certainly argue about under which conditions Java wins and under which C++ (or Zig it Rust etc.) win and how common they are, but people who think low-level languages win across the board or almost across the board clearly don't know what's going on. Early on it was people who were sceptical about how effectively JITs and moving collectors could do their job in practice (even though the theory was clear), but these days I think it's mostly people who haven't struggled with performance issues in low-level languages long enough, and just see that for small or young programs they work fine. They always were. Writing a new program in C++ was never harder than writing a new program in Java, and the performance was great (and people weren't concerned about memory safety in particular). The problems came later - in the 5th year, the 10th year, etc.., when the cost of evolution and trying to keep performance good were piling up.
- joas_coder 7d ago
  [dead]

afdbcreid 15d ago

Also: > This is why most large and long-running programs have abandoned low-level programming languages. That's not true, as evidenced by the fact that this move has started before extremely sophisticated JIT compilers or garbage collectors were invented. The reason was not because managed languages were faster or even had equal speed, but because of the costs associated with memory unsafety (not just security), exactly what Rust prevents (which was of course not available then). You can see empirical evidence of this, for example, by the post about Aurora DSQL rewrite in Rust (https://www.allthingsdistributed.com/2025/05/just-make-it-scale-an-aurora-dsql-story.html https://www.allthingsdistributed.com/2025/05/just-make-it-sc...). One notable quote: > But after a few weeks, it compiled and the results surprised us. The code was 10x faster than our carefully tuned Kotlin implementation – despite no attempt to make it faster. To put this in perspective, we had spent years incrementally improving the Kotlin version from 2,000 to 3,000 transactions per second (TPS). The Rust version, written by Java developers who were new to the language, clocked 30,000 TPS. You also ignore the impact of memory usage, where unmanaged language have an even greater edge (yes I know it is possible to optimize managed languages' memory consumption as well. Not to the same amount and often at the expense of speed).

pron 15d ago
> That's not true, as evidenced by the fact that this move has started before extremely sophisticated JIT compilers or garbage collectors were invented. I don't know how long you've been programming, but that's not true. In the late nineties and early aughts I was working on large, performance-critical, soft- and hard-realtime systems, and we only started moving away from C++ when Java started beating its performance. > The reason was not because managed languages were faster or even had equal speed, but because of the costs associated with memory unsafety (not just security), exactly what Rust prevents (which was of course not available then). That's a myth, and a fairly recent one. Sure, there were non-performance-sensitive programs written in slow languages for a long time. But the industry was mostly using C++ for anything that needed to be big and fast, and back then "memory safety" was mostly just another type of bug. It was nowhere near reason enough to use slow languages, which is why we didn't use them. Lack of memory safety is a serious problem, but the claim that it's the biggest issue with C++, let alone the one that's always been considered the biggest issue, is just a myth. Back then it was certainly considered no bigger an issue than the language complexity, compilation time, and even performance issues in large, long-running programs. > You can see empirical evidence of this, for example, by the post about Aurora DSQL rewrite in Rust I talk to the people at AWS, and this is not the evidence you think it is. First, their problem was primarily with GC pauses, and it was before pauseless GCs. Second, the codebase isn't very big. Third, because Java and C++/Rust offer similar performance - sometimes one wins, sometimes another - you expect to see exactly that. I can tell you that we recently wrote a distributed cache in both Java and Rust simultaneously (using the pauseless GC). The Java version achieved twice the throughput of the Rust version, and significanly better latency across all percentiles. So sure, on the smaller end, there are programs where Rust would be 2x as fast as Java, there are programs where Java would be 2x as fast as Rust, and on average they're about the same. But over time, Java's advantage starts to show as it makes it easier to keep the good performance over years of evolution.