6 ms·

Honestly feels like a misdesign in ARM. Where does it ever make sense for Device memory to not be data prefetchable, but allow instruction prefetch? It should I

by eqvinox 10d ago

Honestly feels like a misdesign in ARM. Where does it ever make sense for Device memory to not be data prefetchable, but allow instruction prefetch? It should IMHO disable all prefetch…

achierius 10d ago
(Total guess) this feels like an attempt at restricting what parts of the hardware pipeline need to know what; if prefetches need to key off of both the device bit and the nx bit, then both of those need to be piped into the frontend, whereas if device is only relevant for data prefetches then only the nx bit needs to be available there (while device would be piped into the backend where the data prefetcher lives).
- adrian_b 10d ago
  Not really. To the front end, one needs to pipe only appropriate logical combinations of the flags that describe the memory properties, not the flags as they are stored in some control register. So a control register should contain flags that are meaningful for the programmer, while the kind of flags that hardware happens to need can be generated with a few logic gates from them and routed through hardware wherever they are needed. It makes more sense to classify the memory in a few types, which must be specified by the programmer, including a "device memory" a.k.a. "memory-mapped peripherals" type, instead of having to specify for each memory area a long set of attributes about each kind of access that may be allowed, or not. This is how it is done in x86-64.
  - sleirsgoevy 9d ago
    x86_64 seems not to care much. I've had a UART mapped as normal memory, and it somehow still worked.
    - adrian_b 9d ago
      Did you really map it as normal memory? Standard UARTs are accessed in the x86 I/O space, which is uncacheable and strongly ordered, not in the normal memory space. Non-standard UARTs that are on PCIe add-on cards might be mapped into memory, but when the computer boots, the BIOS already maps the PCIe memory as uncacheable, with the Memory Type Range Registers. So if you do nothing, you get uncacheable memory for your UART, as you should. To map the UART in normal memory, i.e. write-back cacheable, you must do this explicitly in the UART device driver, either by allocating a dedicated MTRR for it, or more likely, by locking a memory page for it in the virtual address translation tables and using the PAT bits to set the memory type (Page Attribute Table). If you really mapped the UART as normal memory, i.e. write-back cacheable, it can work correctly only if you are incredibly lucky, because the writes to the UART control registers will not happen when you do them in your program, so I do not believe that you did that. With normal memory, not only you cannot predict when the UART registers will be written, but if you do multiple writes to the same register, all but the last will be omitted, so there is no way for the UART to work correctly. On x86-64, any memory-mapped peripheral must be mapped into uncacheable memory, except for memory banks that are located on PCIe cards, like the GPU memories, which should be mapped as write-combining memory.
phire 10d ago
The problem is that unlike data prefetch, the so-called "instruction prefetch" is not actually prefetch at all. It's simply speculative execution. Which doesn't look any different to regular execution. The fetcher has no idea that its predicted branch is about to invalidated and flushed, otherwise it would never have issued that fetch. Actually, on a modern OoO core, [0] it's very rare for the instruction fetcher to not be doing speculative fetches. Even when it's not predicting a branch, the fact that it has "predicted" the lack of a branch is speculative in itself. It assumes it didn't fetch a branch in the last cycle, but it can't be sure until after instruction decoding, which takes at least 2 cycles (more on larger L1i caches). About the only time the instruction fetcher is not doing speculative fetching is for a single cycle after each miss-predicted branch. [0] Or even something technically in-order, like the Cortex A53 cores here. They might issue in-order, but because of how they implement dual issue, they look somewhat close to a simple OoO core... I suspect they actually do register renaming. And (most importantly) importantly they have a branch predictor.
- repiret 10d ago
  No, the problem really is the prefetch itself when there are undesirable side-effects if the bus sees that memory read.
  - phire 10d ago
    Well yes. That is why the "prefetch" is a problem. But the original question was asking why disabling data prefetching to a memory region didn't automatically disable instruction prefetching at the same time. And the answer is that speculative execution is a completely different mechanism that I'm not even sure can be disabled, at least not per memory region.
    - renox 10d ago
      Why do you say this? The post is quite clear that marking the memory region as NX fix the issue caused by speculative execution.
- sleirsgoevy 9d ago

by eqvinox 10d ago

Honestly feels like a misdesign in ARM. Where does it ever make sense for Device memory to not be data prefetchable, but allow instruction prefetch? It should IMHO disable all prefetch…

achierius 10d ago

(Total guess) this feels like an attempt at restricting what parts of the hardware pipeline need to know what; if prefetches need to key off of both the device bit and the nx bit, then both of those need to be piped into the frontend, whereas if device is only relevant for data prefetches then only the nx bit needs to be available there (while device would be piped into the backend where the data prefetcher lives).

adrian_b 10d ago
Not really. To the front end, one needs to pipe only appropriate logical combinations of the flags that describe the memory properties, not the flags as they are stored in some control register. So a control register should contain flags that are meaningful for the programmer, while the kind of flags that hardware happens to need can be generated with a few logic gates from them and routed through hardware wherever they are needed. It makes more sense to classify the memory in a few types, which must be specified by the programmer, including a "device memory" a.k.a. "memory-mapped peripherals" type, instead of having to specify for each memory area a long set of attributes about each kind of access that may be allowed, or not. This is how it is done in x86-64.
- sleirsgoevy 9d ago
  x86_64 seems not to care much. I've had a UART mapped as normal memory, and it somehow still worked.
  - adrian_b 9d ago
    Did you really map it as normal memory? Standard UARTs are accessed in the x86 I/O space, which is uncacheable and strongly ordered, not in the normal memory space. Non-standard UARTs that are on PCIe add-on cards might be mapped into memory, but when the computer boots, the BIOS already maps the PCIe memory as uncacheable, with the Memory Type Range Registers. So if you do nothing, you get uncacheable memory for your UART, as you should. To map the UART in normal memory, i.e. write-back cacheable, you must do this explicitly in the UART device driver, either by allocating a dedicated MTRR for it, or more likely, by locking a memory page for it in the virtual address translation tables and using the PAT bits to set the memory type (Page Attribute Table). If you really mapped the UART as normal memory, i.e. write-back cacheable, it can work correctly only if you are incredibly lucky, because the writes to the UART control registers will not happen when you do them in your program, so I do not believe that you did that. With normal memory, not only you cannot predict when the UART registers will be written, but if you do multiple writes to the same register, all but the last will be omitted, so there is no way for the UART to work correctly. On x86-64, any memory-mapped peripheral must be mapped into uncacheable memory, except for memory banks that are located on PCIe cards, like the GPU memories, which should be mapped as write-combining memory.

phire 10d ago

The problem is that unlike data prefetch, the so-called "instruction prefetch" is not actually prefetch at all. It's simply speculative execution. Which doesn't look any different to regular execution. The fetcher has no idea that its predicted branch is about to invalidated and flushed, otherwise it would never have issued that fetch. Actually, on a modern OoO core, [0] it's very rare for the instruction fetcher to not be doing speculative fetches. Even when it's not predicting a branch, the fact that it has "predicted" the lack of a branch is speculative in itself. It assumes it didn't fetch a branch in the last cycle, but it can't be sure until after instruction decoding, which takes at least 2 cycles (more on larger L1i caches). About the only time the instruction fetcher is not doing speculative fetching is for a single cycle after each miss-predicted branch. [0] Or even something technically in-order, like the Cortex A53 cores here. They might issue in-order, but because of how they implement dual issue, they look somewhat close to a simple OoO core... I suspect they actually do register renaming. And (most importantly) importantly they have a branch predictor.

repiret 10d ago
No, the problem really is the prefetch itself when there are undesirable side-effects if the bus sees that memory read.
- phire 10d ago
  Well yes. That is why the "prefetch" is a problem. But the original question was asking why disabling data prefetching to a memory region didn't automatically disable instruction prefetching at the same time. And the answer is that speculative execution is a completely different mechanism that I'm not even sure can be disabled, at least not per memory region.
  - renox 10d ago
    Why do you say this? The post is quite clear that marking the memory region as NX fix the issue caused by speculative execution.
sleirsgoevy 9d ago