58 ms·

https://github.com/triton-lang/triton/pull/7298#discussion_r2202281596 https://github.com/triton-lang/triton/pull/7298#discussion_r... > By disassembly of ptxa

https://github.com/triton-lang/triton/pull/7298#discussion_r2202281596 https://github.com/triton-lang/triton/pull/7298#discussion_r... > By disassembly of ptxas, it is indeed hard-coded that they have logic like: strstr(kernel_name, "cutlass"). > it is likely that, this is an unstable, experimental, aggressive optimization by NVIDIA, and blindly always enabling it may produce some elusive bugs.

temp0826 1y ago
Thanks for a little context, this is not my wheelhouse at all (never even heard of this project) and I could not make heads or tails of the title or the linked PR.
frogblast 1y ago
Often not elusive bugs, but elusive performance. GPU compilers are hard: Once you've done the basics, trying to do further transforms in a mature compiler will almost always produced mixed results. Some kernels will go faster, some will go slower, and you're hoping to move the balance and not hit any critical kernel too hard in your efforts to make another go faster. An optimization with a universal >=0 speedup across your entire suite of tests is a really hard thing to come by. Something is always going to have a negative speedup. My experience is with non-Nvidia GPU systems, but this feels like a familiar situation. They probably found something that has great outcomes for one set of kernels, terrible outcomes for another, and no known reliable heuristic or modeling they could use to automatically choose.
- Eridrus 1y ago
  A saner design would turn this optimization into a documented flag that anyone can opt into.
  - rcoveson 1y ago
    Speaking from a place of long-term frustration with Java, some compiler authors just absolutely hate exposing the ability to hint/force optimizations. Never mind that it might improve performance for N-5 and N+5 major releases, it might be meaningless or unhelpful or difficult to maintain in a release ten years from now, so it must not be exposed today.
    - MichaelZuo 1y ago
      That seems valid for customers expecting a warranty or support. But they should allow it if customers waive all such in writing.
      - Dylan16807 1y ago
        Warranty and support specifically for that flag? Because I don't see how general warranty and support requires keeping any hint flags forever.
        shadowpho 1y ago
        If you remove the hint flag peoples build will break
        Dylan16807 1y ago
        Doesn't need to, it can acknowledge and ignore the hints.
        shadowpho 1y ago
        True, but there might be more problems — like if you drop support their run time will be slow because they rely on this flag and they are unhappy
        Dylan16807 1y ago
        The premise of removing the flag is that it's useless or a problem. If it's still causing a big speed boost somewhere then you need to figure something out, but the core scenario here is that it's obsolete.
    - recursivecaveat 1y ago
      I once exposed a "disableXYZOptimization" flag to customers so they could debug a easier without stuff getting scrambled. Paid for my gesture for the next year signing off on release updates, writing user guide entries, bleh.
- godelski 1y ago
  > An optimization with a universal >=0 speedup across your entire suite of tests is a really hard thing to come by. Something is always going to have a negative speedup. Maybe a common example of this is that people can write matrix matrix multiplication kernels that outperform standard implementations (also in BLAS for CPU). But that's not a General Matrix Matrix multiply. Is the speedup still there for spare matrices? Larger ones? Small ones? Ones that aren't powers of 2? Non-square? And so on. You can beat the official implementation in any one of these but good luck doing it everywhere. In fact, you should beat the official method because you don't have the overhead to check which optimization you should use. It's easy to over simplify a problem and not even realize you have done so. There's always assumptions being made and you should not let these be invisible.

temp0826 1y ago

Thanks for a little context, this is not my wheelhouse at all (never even heard of this project) and I could not make heads or tails of the title or the linked PR.

frogblast 1y ago

Often not elusive bugs, but elusive performance. GPU compilers are hard: Once you've done the basics, trying to do further transforms in a mature compiler will almost always produced mixed results. Some kernels will go faster, some will go slower, and you're hoping to move the balance and not hit any critical kernel too hard in your efforts to make another go faster. An optimization with a universal >=0 speedup across your entire suite of tests is a really hard thing to come by. Something is always going to have a negative speedup. My experience is with non-Nvidia GPU systems, but this feels like a familiar situation. They probably found something that has great outcomes for one set of kernels, terrible outcomes for another, and no known reliable heuristic or modeling they could use to automatically choose.

Eridrus 1y ago
A saner design would turn this optimization into a documented flag that anyone can opt into.
- rcoveson 1y ago
  Speaking from a place of long-term frustration with Java, some compiler authors just absolutely hate exposing the ability to hint/force optimizations. Never mind that it might improve performance for N-5 and N+5 major releases, it might be meaningless or unhelpful or difficult to maintain in a release ten years from now, so it must not be exposed today.
  - MichaelZuo 1y ago
    That seems valid for customers expecting a warranty or support. But they should allow it if customers waive all such in writing.
    - Dylan16807 1y ago
      Warranty and support specifically for that flag? Because I don't see how general warranty and support requires keeping any hint flags forever.
      - shadowpho 1y ago
        If you remove the hint flag peoples build will break
        Dylan16807 1y ago
        Doesn't need to, it can acknowledge and ignore the hints.
        shadowpho 1y ago
        True, but there might be more problems — like if you drop support their run time will be slow because they rely on this flag and they are unhappy
        Dylan16807 1y ago
        The premise of removing the flag is that it's useless or a problem. If it's still causing a big speed boost somewhere then you need to figure something out, but the core scenario here is that it's obsolete.
  - recursivecaveat 1y ago
    I once exposed a "disableXYZOptimization" flag to customers so they could debug a easier without stuff getting scrambled. Paid for my gesture for the next year signing off on release updates, writing user guide entries, bleh.
godelski 1y ago
> An optimization with a universal >=0 speedup across your entire suite of tests is a really hard thing to come by. Something is always going to have a negative speedup. Maybe a common example of this is that people can write matrix matrix multiplication kernels that outperform standard implementations (also in BLAS for CPU). But that's not a General Matrix Matrix multiply. Is the speedup still there for spare matrices? Larger ones? Small ones? Ones that aren't powers of 2? Non-square? And so on. You can beat the official implementation in any one of these but good luck doing it everywhere. In fact, you should beat the official method because you don't have the overhead to check which optimization you should use. It's easy to over simplify a problem and not even realize you have done so. There's always assumptions being made and you should not let these be invisible.