6 ms·

My heard hurts - i was stupid enough to think that SIMD was a CPU only thing - I don't understand why it would be ported to GPU - huge kudos to managing to surp

by 6r17 1mo ago

My heard hurts - i was stupid enough to think that SIMD was a CPU only thing - I don't understand why it would be ported to GPU - huge kudos to managing to surprise me

hingler36 1mo ago
Welcome to the lucky 10,000! SIMD is actually a pretty integral part of how GPUs are able to work efficiently, it's part of why there's such a strong focus on branchless programming in the field.
chlorion 1mo ago
GPUs work on vectors and matrices very often, that's what they are good at, so it makes a lot of sense that they can operate with SIMD I think!
ismailmaj 1mo ago
There is something very SIMD-coded in GPU programming which is coalesced stores/loads, if a warp (32 threads) handles contiguous memory, it will create ~4 transactions instead of 32.
monocasa 1mo ago
GPU "cores" are basically what a CPU would call SIMD lanes. So a GPU with 1024 'CUDA cores' might be structured as 16 relatively independent pieces that a CPU might call a core, each with a 64 wide SIMD unit.
- mathisfun123 1mo ago
  32 wide - only AMD has a 64 wide mode
- y1n0 1mo ago
  64 what? Bits/bytes/something bigger?
  - corysama 1mo ago
    When you dig through the CUDA developer docs instead of the promotional materials, you can develop a view of Nvidia GPUs as having 8-128 processing cores, each with 4 hyperthreads, running 32-lane SIMD for almost everything. Where a lane is 32 bits wide. The promotional material likes to label the individual lanes as “cores” because it sounds more impressive. And, it’s not entirely incorrect. Even the dev docs use the marketing terminology. The description I gave above needs a bit of piecing together.
    - dataflow 1mo ago
      This is a fantastic explanation, thanks for writing it. It also makes me wonder something: where exactly is the biggest difference between a 32-core x86 CPU (AVX512 basically being 16 32-bit lanes) and (say) an NVIDIA GPU with ~8-16 processing cores? Like why can't the CPU compete against a GPU like that for GPU-y tasks - or can it?
      - monocasa 1mo ago
        Not corysama, but I'll take a stab: The major one is that there's one layer of indirection that exists on GPUs that doesn't really on CPUs. There's one giant vector register file per for each of these processing cores (that'll be something like 2048 rows x 32 lanes x 32 bits). An individual shader invocation might only need say, 16 rows. While there's hardware for issuing 4 hyperthreads at any given time, there can be a variable number of thread states in the register file. So for the case of each invocation only needing 16 rows, you might be able to fit 128 hyperthread states into processing core. Those four hyperthreads then hardware schedule those 128 states and will execute any that are ready, as well as allocate more from other scheduling hardware as gaps in the register file appear from shaders completing. Because of this massive amount thread state, you don't depend nearly as much on a cache hierarchy to deal with DRAM latency. There's ostensibly some other thread state sitting around that can be serviced while others wait for the hundreds of cycles of latency to access DRAM. So the whole model of how you account for the discrepancy between ALU cycle time, and DRAM latency changes versus a CPU. Where a modern CPU spends a lot of area on complex cache hierarchies, speculation, etc, to hide the latency to memory, a GPU focuses on having a lot of thread state around and a lot of ALUs, but balanced ideally, so there's always ALU work to do while other thread states are waiting on memory. Now, over time, GPUs have gotten more complex hardware, and more complex cache hierarchies to cover the cases that aren't handled well by extremely long access times. But those tend to be very explicit. Additionally, CPU vector files have gotten more similar to GPU cores as architectural features like lane masking/predicates have been added to have the equivalent of CUDA threads in the same warp that take different paths through control flow blocks. That's a lot of what people mean when they say that AVX-512 adds a lot more than just 512-bit registers. The K mask registers let you do a lot of GPU shader tricks to have effective partial residency, and not have to use all the lanes if the data doesn't line up with that.
MindSpunk 1mo ago
It's not really obvious unless you go in depth of the details on modern GPU architecture. GPUs aren't really SIMD, they're SIMT (single instruction multiple thread). The silicon looks a lot like SIMD, but the programming model is different. If you go look at AMD's ISA docs (they're public) you'll see you don't have the equivalent of a __mm256 register like on x86. Each 'thread' just deals with single scalar values like int32 of float32. The hardware, however, groups 32 or 64 threads together which all run the same program and runs them together. Each 'thread' loosely maps to a SIMD lane. The SIMD is implicit, not explicit. The main difference is that the 'SIMD' execution is somewhat opaque to the program. You just write plain scalar code and the hardware model dispatches it efficiently to SIMD execution units. It's not really an abstraction because to extract maximum performance you have to understand how it works. You can use this kind of programming model on a CPU too, Intel did it with [0] ISPC. It's a C-like language that has execution semantics similar to GPU shader languages but compiles to regular CPU code, and maps threads to your CPUs SIMD lanes like a GPU. [0] https://ispc.github.io/ https://ispc.github.io/
- mathisfun123 1mo ago
  > GPUs aren't really SIMD, they're SIMT (single instruction multiple thread) False. If they were threads they'd have their own PC. They do not - only the warp has a PC. > You just write plain scalar code and the hardware model dispatches it efficiently to SIMD execution units. Absolutely not. If you don't write coalesced loads, bank-conflict free, predication-free, cooperative code you will get worse than CPU performance.
  - MindSpunk 1mo ago
    Yes, of course writing naive code assuming each lane in a thread group is a real thread is going to cause problems, but I didn't feel like I needed to go into that level of detail replying to someone just learning about GPU internals. I tried to cover this loosely by mentioning how you need to know how it works for maximum performance. If you want to get more pedantic you also need to look at your target hardware and their specific micro-architectural quirks and features to get the best performance. AMD specifically benefits a lot from exploiting the scalar unit over the vector unit, you save loads of register file space if you can keep data in SGPRs over VGPRs. There's lots of traps you can fall into where you can load data from buffers into SGPRs but they get promoted to VGPRs because the scalar unit lacks an opcode for like one math operation you did to the value somewhere. While each lane isn't truly a thread because it doesn't have its own PC the programming model definitely tries to make it seem that way. The threads can terminate at different points too. And again, the ISA isn't a vector ISA. Your register values are scalar.

by 6r17 1mo ago

My heard hurts - i was stupid enough to think that SIMD was a CPU only thing - I don't understand why it would be ported to GPU - huge kudos to managing to surprise me

hingler36 1mo ago

Welcome to the lucky 10,000! SIMD is actually a pretty integral part of how GPUs are able to work efficiently, it's part of why there's such a strong focus on branchless programming in the field.

chlorion 1mo ago

GPUs work on vectors and matrices very often, that's what they are good at, so it makes a lot of sense that they can operate with SIMD I think!

ismailmaj 1mo ago

There is something very SIMD-coded in GPU programming which is coalesced stores/loads, if a warp (32 threads) handles contiguous memory, it will create ~4 transactions instead of 32.

monocasa 1mo ago

GPU "cores" are basically what a CPU would call SIMD lanes. So a GPU with 1024 'CUDA cores' might be structured as 16 relatively independent pieces that a CPU might call a core, each with a 64 wide SIMD unit.

mathisfun123 1mo ago
32 wide - only AMD has a 64 wide mode
y1n0 1mo ago
64 what? Bits/bytes/something bigger?
- corysama 1mo ago
  When you dig through the CUDA developer docs instead of the promotional materials, you can develop a view of Nvidia GPUs as having 8-128 processing cores, each with 4 hyperthreads, running 32-lane SIMD for almost everything. Where a lane is 32 bits wide. The promotional material likes to label the individual lanes as “cores” because it sounds more impressive. And, it’s not entirely incorrect. Even the dev docs use the marketing terminology. The description I gave above needs a bit of piecing together.
  - dataflow 1mo ago
    This is a fantastic explanation, thanks for writing it. It also makes me wonder something: where exactly is the biggest difference between a 32-core x86 CPU (AVX512 basically being 16 32-bit lanes) and (say) an NVIDIA GPU with ~8-16 processing cores? Like why can't the CPU compete against a GPU like that for GPU-y tasks - or can it?
    - monocasa 1mo ago
      Not corysama, but I'll take a stab: The major one is that there's one layer of indirection that exists on GPUs that doesn't really on CPUs. There's one giant vector register file per for each of these processing cores (that'll be something like 2048 rows x 32 lanes x 32 bits). An individual shader invocation might only need say, 16 rows. While there's hardware for issuing 4 hyperthreads at any given time, there can be a variable number of thread states in the register file. So for the case of each invocation only needing 16 rows, you might be able to fit 128 hyperthread states into processing core. Those four hyperthreads then hardware schedule those 128 states and will execute any that are ready, as well as allocate more from other scheduling hardware as gaps in the register file appear from shaders completing. Because of this massive amount thread state, you don't depend nearly as much on a cache hierarchy to deal with DRAM latency. There's ostensibly some other thread state sitting around that can be serviced while others wait for the hundreds of cycles of latency to access DRAM. So the whole model of how you account for the discrepancy between ALU cycle time, and DRAM latency changes versus a CPU. Where a modern CPU spends a lot of area on complex cache hierarchies, speculation, etc, to hide the latency to memory, a GPU focuses on having a lot of thread state around and a lot of ALUs, but balanced ideally, so there's always ALU work to do while other thread states are waiting on memory. Now, over time, GPUs have gotten more complex hardware, and more complex cache hierarchies to cover the cases that aren't handled well by extremely long access times. But those tend to be very explicit. Additionally, CPU vector files have gotten more similar to GPU cores as architectural features like lane masking/predicates have been added to have the equivalent of CUDA threads in the same warp that take different paths through control flow blocks. That's a lot of what people mean when they say that AVX-512 adds a lot more than just 512-bit registers. The K mask registers let you do a lot of GPU shader tricks to have effective partial residency, and not have to use all the lanes if the data doesn't line up with that.

MindSpunk 1mo ago

It's not really obvious unless you go in depth of the details on modern GPU architecture. GPUs aren't really SIMD, they're SIMT (single instruction multiple thread). The silicon looks a lot like SIMD, but the programming model is different. If you go look at AMD's ISA docs (they're public) you'll see you don't have the equivalent of a __mm256 register like on x86. Each 'thread' just deals with single scalar values like int32 of float32. The hardware, however, groups 32 or 64 threads together which all run the same program and runs them together. Each 'thread' loosely maps to a SIMD lane. The SIMD is implicit, not explicit. The main difference is that the 'SIMD' execution is somewhat opaque to the program. You just write plain scalar code and the hardware model dispatches it efficiently to SIMD execution units. It's not really an abstraction because to extract maximum performance you have to understand how it works. You can use this kind of programming model on a CPU too, Intel did it with [0] ISPC. It's a C-like language that has execution semantics similar to GPU shader languages but compiles to regular CPU code, and maps threads to your CPUs SIMD lanes like a GPU. [0] https://ispc.github.io/ https://ispc.github.io/

mathisfun123 1mo ago
> GPUs aren't really SIMD, they're SIMT (single instruction multiple thread) False. If they were threads they'd have their own PC. They do not - only the warp has a PC. > You just write plain scalar code and the hardware model dispatches it efficiently to SIMD execution units. Absolutely not. If you don't write coalesced loads, bank-conflict free, predication-free, cooperative code you will get worse than CPU performance.
- MindSpunk 1mo ago
  Yes, of course writing naive code assuming each lane in a thread group is a real thread is going to cause problems, but I didn't feel like I needed to go into that level of detail replying to someone just learning about GPU internals. I tried to cover this loosely by mentioning how you need to know how it works for maximum performance. If you want to get more pedantic you also need to look at your target hardware and their specific micro-architectural quirks and features to get the best performance. AMD specifically benefits a lot from exploiting the scalar unit over the vector unit, you save loads of register file space if you can keep data in SGPRs over VGPRs. There's lots of traps you can fall into where you can load data from buffers into SGPRs but they get promoted to VGPRs because the scalar unit lacks an opcode for like one math operation you did to the value somewhere. While each lane isn't truly a thread because it doesn't have its own PC the programming model definitely tries to make it seem that way. The threads can terminate at different points too. And again, the ISA isn't a vector ISA. Your register values are scalar.