4 ms·

Am I misreading the bitmask code? It looks like (in addition to a few other ideas) it's using the old "stick a few extra bits in an aligned pointer", but it see

Am I misreading the bitmask code? It looks like (in addition to a few other ideas) it's using the old "stick a few extra bits in an aligned pointer", but it seems to be only manipulating high bits, whereas aligned pointer zeroes are low-order bits. I'd suggest a heavier investment in testing infrastructure.

pixelesque 2y ago
It looks like it can use both traditional "tagged pointer" alignment bits AND virtual address bits...
- couchand 2y ago
  I see that in the readme but I don't see where that's handled in the bitmasking. It appears to be universally masking high-order bits here: https://github.com/irrustible/ointers/blob/1961f75bbb9818d72f59a3601997bf8a34952005/src/lib.rs#L460-L464 https://github.com/irrustible/ointers/blob/1961f75bbb9818d72...
  - formerly_proven 2y ago
    Read the function above the function you linked to.
    - couchand 2y ago
      Oh that's frightening they shift the entire pointer by the alignment
      - db48x 2y ago
        Why? Alignment causes some low–order bits to be zero. Shifting the pointer to the right drops those zeros on the floor, leaving high–order bits zero (or sign extended or whatever). Then you can put your tag up there instead. Shifting the value left by the same amount drops the tag and recovers the same pointer for use.
        vlovich123 2y ago
        Any idea why it's preferred to do the alignment route instead of storing the bits in the upper part of the pointer & masking?
        db48x 2y ago
        Personal preference, architectural differences, phase of the moon, etc, etc.
        couchand 2y ago
        My guideline with pointer tricks is: if you're not just masking bits it's too complicated for real use.
Sharlin 2y ago
The stored representation is packed such that all the stealable bits are contiguous. To get the original pointer value it’s unpacked first.
masklinn 2y ago
64 bit architectures don't actually have 64 bit address spaces, both AMD64 and ARM64 have 48 bit address spaces by default (some CPUs have extensions you can enable to request larger address spaces e.g. LVA on ARM64 and 5LP / LA57 on AMD64 but that's opt-in on a per-process basis). So while you have 3 bits available at the bottom of the pointer, there are 16 at the top. That's a lot more payload you can smuggle. There are even CPU extensions which tell the processor to ignore some of that (Linear Address Masking for Intel, Upper Address Ignore for AMD, and Top Byte Ignore for ARM).
- vlovich123 2y ago
  Small correction. That's true for 4-level paging where virtual memory is capped at 256 TiB (LAM48). There's a 5-level extension that reduces the wasted space to 7 bits (LAM57) to allow for 128 PiB of virtual space. I have no idea what the purpose of the extension is that when I don't believe you can get that much secondary storage attached to a CPU unless you go to tape which is pointless for virtual mapping.
  - masklinn 2y ago
    ... I literally mentioned five-level paging in my comment. But it's an opt-in extension, you need the kernel to support it, and then you need to ask the kernel to enable it on your behalf. So it's not much of an issue, you just have to not to use this sort of extensions (it's not the only one, ARM64 also has a similar one though it only goes up to 52 bit address spaces) with 16~19 bits of tagging.
    - LegionMammal978 2y ago
      You do need to opt-in when compiling the kernel, but on Linux it doesn't take anything particularly special on the program's side to enable it. The rule is that non-fixed mmap() will only ever yield 48-bit addresses, until the program specifies a 57-bit address as a hint, after which it can yield 57-bit addresses at will. (This rule has lots of curious consequences for the location of the vDSO, when 5-level paging is enabled and an ELF program uses big addresses for its segments.)

pixelesque 2y ago

It looks like it can use both traditional "tagged pointer" alignment bits AND virtual address bits...

couchand 2y ago
I see that in the readme but I don't see where that's handled in the bitmasking. It appears to be universally masking high-order bits here: https://github.com/irrustible/ointers/blob/1961f75bbb9818d72f59a3601997bf8a34952005/src/lib.rs#L460-L464 https://github.com/irrustible/ointers/blob/1961f75bbb9818d72...
- formerly_proven 2y ago
  Read the function above the function you linked to.
  - couchand 2y ago
    Oh that's frightening they shift the entire pointer by the alignment
    - db48x 2y ago
      Why? Alignment causes some low–order bits to be zero. Shifting the pointer to the right drops those zeros on the floor, leaving high–order bits zero (or sign extended or whatever). Then you can put your tag up there instead. Shifting the value left by the same amount drops the tag and recovers the same pointer for use.
      - vlovich123 2y ago
        Any idea why it's preferred to do the alignment route instead of storing the bits in the upper part of the pointer & masking?
        db48x 2y ago
        Personal preference, architectural differences, phase of the moon, etc, etc.
      - couchand 2y ago
        My guideline with pointer tricks is: if you're not just masking bits it's too complicated for real use.

Sharlin 2y ago

The stored representation is packed such that all the stealable bits are contiguous. To get the original pointer value it’s unpacked first.

masklinn 2y ago

64 bit architectures don't actually have 64 bit address spaces, both AMD64 and ARM64 have 48 bit address spaces by default (some CPUs have extensions you can enable to request larger address spaces e.g. LVA on ARM64 and 5LP / LA57 on AMD64 but that's opt-in on a per-process basis). So while you have 3 bits available at the bottom of the pointer, there are 16 at the top. That's a lot more payload you can smuggle. There are even CPU extensions which tell the processor to ignore some of that (Linear Address Masking for Intel, Upper Address Ignore for AMD, and Top Byte Ignore for ARM).

vlovich123 2y ago
Small correction. That's true for 4-level paging where virtual memory is capped at 256 TiB (LAM48). There's a 5-level extension that reduces the wasted space to 7 bits (LAM57) to allow for 128 PiB of virtual space. I have no idea what the purpose of the extension is that when I don't believe you can get that much secondary storage attached to a CPU unless you go to tape which is pointless for virtual mapping.
- masklinn 2y ago
  ... I literally mentioned five-level paging in my comment. But it's an opt-in extension, you need the kernel to support it, and then you need to ask the kernel to enable it on your behalf. So it's not much of an issue, you just have to not to use this sort of extensions (it's not the only one, ARM64 also has a similar one though it only goes up to 52 bit address spaces) with 16~19 bits of tagging.
  - LegionMammal978 2y ago
    You do need to opt-in when compiling the kernel, but on Linux it doesn't take anything particularly special on the program's side to enable it. The rule is that non-fixed mmap() will only ever yield 48-bit addresses, until the program specifies a 57-bit address as a hint, after which it can yield 57-bit addresses at will. (This rule has lots of curious consequences for the location of the vDSO, when 5-level paging is enabled and an ELF program uses big addresses for its segments.)