5 ms·

The code in compilers is the closest to your typical app you can get in a benchmark like SPEC, eveerything else is actually far more specialized. Compiler code

The code in compilers is the closest to your typical app you can get in a benchmark like SPEC, eveerything else is actually far more specialized. Compiler code is full of small basic blocks, lots of branches, indirect memory access; it's actually harder to get good performance for such code, both for CPUs and compilers (that was part of the death of Itanium too).

jcranmer 1mo ago
That's true of most of the applications in SPECint (SPECfp is a different matter); there's nothing special about compilers there. Where compiler code is going to get really unusual, I suspect, is that compilers tend to be a little mono-focused on relatively few data structures. I know I was able to get measurable (single-digit percent!) performance differences in LLVM making very small tweaks to layout in llvm::Value. By contrast, when I was working on Thunderbird, the only similarly small change I could think to make that kind of difference would be to "oops, all string functions are now a cross-DLL call" (and even then, only because string handling is so dominant in that kind of application). Another kind of difference is that the compiler-based benchmarks are going to be quite light in virtual or indirect function calls (there's more of an emphasis on switch-based dispatching than vtable-based dispatching in most compiler implementations), which is going to make it a poorer proxy for some kinds of applications.
- 14113 1mo ago
  Seconding this - having worked on LLVM and Firefox, the performance tuning of each application was very different. Even measuring the performance of an application like Firefox (in a meaningful way) is non-trivial, wheras compilers are much more approachable with traditional profilers (either tracing or sampling).
lstodd 1mo ago
Death of Itanium was that it was a) VLIW and b) Intel was too arrogant. So it went to the same destination as later Larrabee and ATI/AMD attempts at VLIW GPUs. That is, nowhere. Also you are wrong and anyone sizing up an arch to put their loads onto must first try that load on it and not rely on "bah, compilers compile on it".
- speed_spread 1mo ago
  VLIW works for DSP applications, it's not an instant dead end. It's a good fit in cases where code path and memory accesses are predictable, like shader code.
  - Symmetry 1mo ago
    Essentially every cell phone out there has a VLIW DSP like Qualcomm's Hexagon cores (though AFAIK Qualcomm is the only one who lets you run your on programs on their DSP).
- pjmlp 1mo ago
  Itanium only died because AMD exists, and due to various licensing reasons they were allowed to come up with AMD64.
  - wtallis 1mo ago
    In a hypothetical world where AMD wasn't allowed to do AMD64, and Intel stayed committed to Itanium: Itanium would still have sucked, and both PowerPC and SPARC would have out-sold Itanium by an even wider margin than they did in this reality. Itanium could only have succeeded if AMD64 wasn't possible and literally all of the competing 64-bit architectures were killed off by their owners so they could jump on the Itanium bandwagon. Itanium managed to kill off PA-RISC and Alpha and (high-end) MIPS roadmaps, but it still had competitors that were not just viable but actually more successful.
    - pjmlp 1mo ago
      Except you would never had Windows running on either PowerPC and SPARC. Remember, the very first Windows XP 64 bit release was on Itanium.
      - my123 1mo ago
        Windows NT on PowerPC was an actual product, although running in 32-bit mode.
14113 1mo ago
I mildly disagree - depending on your definition of "typical app". Most applications have much greater use of multi-processing and concurrent cross-thread (or cross-process) communication. Compilers, aside from high-level parallelism across modules, tend to be quite single-threaded applications. If you're solely interested in single-core performance, then I would agree that they are a good stress test, but I think for a processor that is being sold on it's parallelism, they are not a great benchmark.
- adrian_b 1mo ago
  In the history of the SPEC benchmarks, the compiler benchmarks, like gcc, have been the best predictor of CPU performance for the applications that cannot benefit from array operations, so they cannot use the vector or matrix instruction set extensions. The reason is that for the other benchmarks the CPU vendors have always succeeded sooner or later, to tweak their compilers and compiling options, or even the hardware of the CPUs, in order to get improved benchmark results that nonetheless are not indicative of the improvements in other applications. On the other hand, the compiling benchmarks, like with gcc, and now also with clang, are too diverse in CPU resource usage and no special feature of the CPU has a significantly greater weight than others, so special tricks to enhance the benchmark results have never been found. When looking at the past SPECint results, the values of the gcc benchmark remain the most reliable relative performance estimator. I doubt that this will change in the near future. Moreover, the multi-threaded compiling benchmark is also very useful, because it matches exactly a real-world workload that is extremely frequently encountered. Due to the great clock frequency difference between running a benchmark on a single thread and running it on all available threads, the single-threaded results have a very poor correlation with the multi-threaded results.

jcranmer 1mo ago

That's true of most of the applications in SPECint (SPECfp is a different matter); there's nothing special about compilers there. Where compiler code is going to get really unusual, I suspect, is that compilers tend to be a little mono-focused on relatively few data structures. I know I was able to get measurable (single-digit percent!) performance differences in LLVM making very small tweaks to layout in llvm::Value. By contrast, when I was working on Thunderbird, the only similarly small change I could think to make that kind of difference would be to "oops, all string functions are now a cross-DLL call" (and even then, only because string handling is so dominant in that kind of application). Another kind of difference is that the compiler-based benchmarks are going to be quite light in virtual or indirect function calls (there's more of an emphasis on switch-based dispatching than vtable-based dispatching in most compiler implementations), which is going to make it a poorer proxy for some kinds of applications.

14113 1mo ago
Seconding this - having worked on LLVM and Firefox, the performance tuning of each application was very different. Even measuring the performance of an application like Firefox (in a meaningful way) is non-trivial, wheras compilers are much more approachable with traditional profilers (either tracing or sampling).

lstodd 1mo ago

Death of Itanium was that it was a) VLIW and b) Intel was too arrogant. So it went to the same destination as later Larrabee and ATI/AMD attempts at VLIW GPUs. That is, nowhere. Also you are wrong and anyone sizing up an arch to put their loads onto must first try that load on it and not rely on "bah, compilers compile on it".

speed_spread 1mo ago
VLIW works for DSP applications, it's not an instant dead end. It's a good fit in cases where code path and memory accesses are predictable, like shader code.
- Symmetry 1mo ago
  Essentially every cell phone out there has a VLIW DSP like Qualcomm's Hexagon cores (though AFAIK Qualcomm is the only one who lets you run your on programs on their DSP).
pjmlp 1mo ago
Itanium only died because AMD exists, and due to various licensing reasons they were allowed to come up with AMD64.
- wtallis 1mo ago
  In a hypothetical world where AMD wasn't allowed to do AMD64, and Intel stayed committed to Itanium: Itanium would still have sucked, and both PowerPC and SPARC would have out-sold Itanium by an even wider margin than they did in this reality. Itanium could only have succeeded if AMD64 wasn't possible and literally all of the competing 64-bit architectures were killed off by their owners so they could jump on the Itanium bandwagon. Itanium managed to kill off PA-RISC and Alpha and (high-end) MIPS roadmaps, but it still had competitors that were not just viable but actually more successful.
  - pjmlp 1mo ago
    Except you would never had Windows running on either PowerPC and SPARC. Remember, the very first Windows XP 64 bit release was on Itanium.
    - my123 1mo ago
      Windows NT on PowerPC was an actual product, although running in 32-bit mode.

14113 1mo ago

I mildly disagree - depending on your definition of "typical app". Most applications have much greater use of multi-processing and concurrent cross-thread (or cross-process) communication. Compilers, aside from high-level parallelism across modules, tend to be quite single-threaded applications. If you're solely interested in single-core performance, then I would agree that they are a good stress test, but I think for a processor that is being sold on it's parallelism, they are not a great benchmark.

adrian_b 1mo ago
In the history of the SPEC benchmarks, the compiler benchmarks, like gcc, have been the best predictor of CPU performance for the applications that cannot benefit from array operations, so they cannot use the vector or matrix instruction set extensions. The reason is that for the other benchmarks the CPU vendors have always succeeded sooner or later, to tweak their compilers and compiling options, or even the hardware of the CPUs, in order to get improved benchmark results that nonetheless are not indicative of the improvements in other applications. On the other hand, the compiling benchmarks, like with gcc, and now also with clang, are too diverse in CPU resource usage and no special feature of the CPU has a significantly greater weight than others, so special tricks to enhance the benchmark results have never been found. When looking at the past SPECint results, the values of the gcc benchmark remain the most reliable relative performance estimator. I doubt that this will change in the near future. Moreover, the multi-threaded compiling benchmark is also very useful, because it matches exactly a real-world workload that is extremely frequently encountered. Due to the great clock frequency difference between running a benchmark on a single thread and running it on all available threads, the single-threaded results have a very poor correlation with the multi-threaded results.