19 ms·

Seems pretty dubious to do this without adding support for having both 4KB and 16KB processes at once to the Linux kernel, since it means all old binaries break

by devit 2y ago

Seems pretty dubious to do this without adding support for having both 4KB and 16KB processes at once to the Linux kernel, since it means all old binaries break and emulators which emulate normal systems with 4KB pages (Wine, console emulators, etc.) might dramatically lose performance if they need to emulate the MMU. Hopefully they don't actually ship a 16KB default before supporting 4KB pages as well in the same kernel. Also it would probably be reasonable, along with making the Linux kernel change, to design CPUs where you can configure a 16KB pagetable entry to map at 4KB granularity and pagefault after the first 4KB or 8KB (requires 3 extra bits per PTE or 2 if coalesced with the invalid bit), so that memory can be saved by allocating 4KB/8KB pages when 16KB would have wasted padding.

mgaunard 2y ago
why does it break userland? if you need to know the page size, you should query sysconf SC_PAGESIZE.
- akdev1l 2y ago
  Assumptions in the software. Jemalloc is infamous for this: https://github.com/sigp/lighthouse/issues/5244 https://github.com/sigp/lighthouse/issues/5244
- ndesaulniers 2y ago
  Ossification. If the page size has been 4k for decades for most OS' and architectures, people get sloppy and hard code that literal value, rather than query for it.
- Dwedit 2y ago
  Emulating a processor with 4K size pages becomes much higher performance if you can use real addresses directly.
- fweimer 2y ago
  It should not break userland. GNU/Linux (not necessarily Android though) has supported 64K pages pretty much from the start because that was the originally page size chosen for server-focus kernels and distributions. But there are some things that need to be worked around. Certain build processes determine the page size at compile time and assume it's the same at run time, and fail if it is not: https://github.com/jemalloc/jemalloc/issues/467 https://github.com/jemalloc/jemalloc/issues/467 Some memory-mapped files formats have assumptions about page granularity: https://bugzilla.redhat.com/show_bug.cgi?id=1979804 https://bugzilla.redhat.com/show_bug.cgi?id=1979804 The file format issue applies to ELF as well. Some people patch their toolchains (or use suitable linker options) to produce slightly smaller binaries that can only be loaded if the page size is 4K, even though the ABI is pretty clear in that you should link for compatibility with up to 64K pages.
- PhilipRoman 2y ago
  Replacing compile time constants with function calls will always bring some trouble, suddenly you need to rearrange your structures, optimizations get missed (in extreme cases you can accidentally introduce a DIV instruction), etc. So it is not surprising that code assumes 4k pages.
  - mgaunard 2y ago
    Any code that does divisions and modulos with non-constants that are known to be powers of 2 should do the optimization manually. Even for non-powers-of-two there are also techniques to speed up divisions if the same divisor is used repeatedly.
username81 2y ago
Shouldn't there be some kind of setting to change the page size per program? AFAIK AMD64 CPUs can do this.
- saagarjha 2y ago
  Yes, ARM CPUs can do it too.
fouronnes3 2y ago
Could they upstream that or would that require a fork?
phh 2y ago
Google/Android doesn't care much about backward compatibility and broke programs released on Pixel 3 in Pixel 7. (the interdiction of 32bit-only apps is 2019 on Play Store, Pixel 7 is first 64bits only device, while Google still released 32bits only device in 2023...). They quite regularly break apps in new Android versions (despite their infrastructure to handle backward compatibility), and app developers are used to brace themselves around Android & Pixel releases
- reissbaker 2y ago
  Generally I've found Google to care much more about not breaking old apps compared to Apple, which often expects developers to rebuild apps for OS updates or else the apps stop working entirely (or buy entirely new machines to get OS updates at all, e.g. the Intel/Apple Silicon transition). Google isn't on the level of Windows "we will watch for specific binaries and re-introduce bugs in the kernel specifically for those binaries that they depend on" in terms of backwards compatibility, but I wouldn't go so far as to say they don't care. I'm not sure whether that's better or worse: there's definitely merit to Apple's approach, since it keeps them able to iterate quickly on UX and performance by dropping support for the old stuff.
Veserv 2y ago
Having both 4KB and 16KB simultaneously is either easy or hard depending on which hardware feature they are using for 16KB pages. If they are using the configurable granule size, then that is a system-wide hardware configuration option. You literally can not map at smaller granularity while that bit is set. You might be able to design a CPU that allows your idea of partial pages, but there be dragons. If they are not configuring the granule size, instead opting for software enforcement in conjunction with always using the contiguous hint bit, then it might be possible. However, I am pretty sure they are talking about hardware granule size, since the contiguous hint is most commonly used to support 16 contiguous entrys (though the CPU designer is technically allowed to do whatever grouping they want) which would be 64KB.
- stingraycharles 2y ago
  I’m a total idiot, how exactly is page size a CPU issue rather than a kernel issue? Is it about memory channel protocols / communication? Disks have been slowly migrating away from the 4kb sector size, is this a same thing going on? That you need to actual drive to support it, because of internal structuring (i.e. how exactly the CPU aligns things in RAM), and on some super low level 4kb / 16kb being the smallest unit of memory you can allocate? And does that then mean that there’s less overhead in all kinds of memory (pre)fetchers in the CPU, because more can be achieved in less clock cycles?
  - IshKebab 2y ago
    The CPU has hardware that does a page table walk automatically when you access an address for which the translation is not cached in the TLB. Otherwise virtual memory would be really slow. Since the CPU hardware itself is doing the page table walk it needs to understand page tables and page table entries etc. including how big pages are. Also you need to know how big pages are for the TLB itself. The value of 4kB itself is pretty much arbitrary. It has to be a small enough number that you don't waste a load of memory by mapping memory that isn't used (e.g. if you ask for 4.01kB you're actually going to get 8kB), but a large enough number that you aren't spending all your time managing tiny pages. That's why increasing the page size makes things faster but waste more memory. 4kB arguably isn't optimal anymore since we have way more memory now than when it was de facto standardised so it doesn't matter as much if we waste a bit. Maybe.
lxgr 2y ago
> all old binaries break and emulators which emulate normal systems with 4KB pages Would it actually affect the kind of emulators present on Android, i.e. largely software-only ones, as opposed to hardware virtualizers making use of a CPU's vTLB? Wine is famously not an emulator and as such doesn't really exist/make sense on (non-x86) Android (as it would only be able to execute ARM binaries, not x86 ones). For the downvote: Genuinely curious here on which type of emulator this could affect.
Zefiroj 2y ago
The support for mTHP exists in upstream Linux, but the swap story is not quite there yet. THP availability also needs work and there are a few competing directions. Supporting multiple page sizes well transparently is non-trivial. For a recent summary on one of the approaches, TAO (THP Allocation Optimization), see this lwn article: https://lwn.net/Articles/974636/ https://lwn.net/Articles/974636/
aaron695 2y ago
[dead]

by devit 2y ago

mgaunard 2y ago

why does it break userland? if you need to know the page size, you should query sysconf SC_PAGESIZE.

akdev1l 2y ago
Assumptions in the software. Jemalloc is infamous for this: https://github.com/sigp/lighthouse/issues/5244 https://github.com/sigp/lighthouse/issues/5244
ndesaulniers 2y ago
Ossification. If the page size has been 4k for decades for most OS' and architectures, people get sloppy and hard code that literal value, rather than query for it.
Dwedit 2y ago
Emulating a processor with 4K size pages becomes much higher performance if you can use real addresses directly.
fweimer 2y ago
It should not break userland. GNU/Linux (not necessarily Android though) has supported 64K pages pretty much from the start because that was the originally page size chosen for server-focus kernels and distributions. But there are some things that need to be worked around. Certain build processes determine the page size at compile time and assume it's the same at run time, and fail if it is not: https://github.com/jemalloc/jemalloc/issues/467 https://github.com/jemalloc/jemalloc/issues/467 Some memory-mapped files formats have assumptions about page granularity: https://bugzilla.redhat.com/show_bug.cgi?id=1979804 https://bugzilla.redhat.com/show_bug.cgi?id=1979804 The file format issue applies to ELF as well. Some people patch their toolchains (or use suitable linker options) to produce slightly smaller binaries that can only be loaded if the page size is 4K, even though the ABI is pretty clear in that you should link for compatibility with up to 64K pages.
PhilipRoman 2y ago
Replacing compile time constants with function calls will always bring some trouble, suddenly you need to rearrange your structures, optimizations get missed (in extreme cases you can accidentally introduce a DIV instruction), etc. So it is not surprising that code assumes 4k pages.
- mgaunard 2y ago
  Any code that does divisions and modulos with non-constants that are known to be powers of 2 should do the optimization manually. Even for non-powers-of-two there are also techniques to speed up divisions if the same divisor is used repeatedly.

username81 2y ago

Shouldn't there be some kind of setting to change the page size per program? AFAIK AMD64 CPUs can do this.

saagarjha 2y ago
Yes, ARM CPUs can do it too.

fouronnes3 2y ago

Could they upstream that or would that require a fork?

phh 2y ago

Google/Android doesn't care much about backward compatibility and broke programs released on Pixel 3 in Pixel 7. (the interdiction of 32bit-only apps is 2019 on Play Store, Pixel 7 is first 64bits only device, while Google still released 32bits only device in 2023...). They quite regularly break apps in new Android versions (despite their infrastructure to handle backward compatibility), and app developers are used to brace themselves around Android & Pixel releases

reissbaker 2y ago
Generally I've found Google to care much more about not breaking old apps compared to Apple, which often expects developers to rebuild apps for OS updates or else the apps stop working entirely (or buy entirely new machines to get OS updates at all, e.g. the Intel/Apple Silicon transition). Google isn't on the level of Windows "we will watch for specific binaries and re-introduce bugs in the kernel specifically for those binaries that they depend on" in terms of backwards compatibility, but I wouldn't go so far as to say they don't care. I'm not sure whether that's better or worse: there's definitely merit to Apple's approach, since it keeps them able to iterate quickly on UX and performance by dropping support for the old stuff.

Veserv 2y ago

Having both 4KB and 16KB simultaneously is either easy or hard depending on which hardware feature they are using for 16KB pages. If they are using the configurable granule size, then that is a system-wide hardware configuration option. You literally can not map at smaller granularity while that bit is set. You might be able to design a CPU that allows your idea of partial pages, but there be dragons. If they are not configuring the granule size, instead opting for software enforcement in conjunction with always using the contiguous hint bit, then it might be possible. However, I am pretty sure they are talking about hardware granule size, since the contiguous hint is most commonly used to support 16 contiguous entrys (though the CPU designer is technically allowed to do whatever grouping they want) which would be 64KB.

stingraycharles 2y ago
I’m a total idiot, how exactly is page size a CPU issue rather than a kernel issue? Is it about memory channel protocols / communication? Disks have been slowly migrating away from the 4kb sector size, is this a same thing going on? That you need to actual drive to support it, because of internal structuring (i.e. how exactly the CPU aligns things in RAM), and on some super low level 4kb / 16kb being the smallest unit of memory you can allocate? And does that then mean that there’s less overhead in all kinds of memory (pre)fetchers in the CPU, because more can be achieved in less clock cycles?
- IshKebab 2y ago
  The CPU has hardware that does a page table walk automatically when you access an address for which the translation is not cached in the TLB. Otherwise virtual memory would be really slow. Since the CPU hardware itself is doing the page table walk it needs to understand page tables and page table entries etc. including how big pages are. Also you need to know how big pages are for the TLB itself. The value of 4kB itself is pretty much arbitrary. It has to be a small enough number that you don't waste a load of memory by mapping memory that isn't used (e.g. if you ask for 4.01kB you're actually going to get 8kB), but a large enough number that you aren't spending all your time managing tiny pages. That's why increasing the page size makes things faster but waste more memory. 4kB arguably isn't optimal anymore since we have way more memory now than when it was de facto standardised so it doesn't matter as much if we waste a bit. Maybe.

lxgr 2y ago

> all old binaries break and emulators which emulate normal systems with 4KB pages Would it actually affect the kind of emulators present on Android, i.e. largely software-only ones, as opposed to hardware virtualizers making use of a CPU's vTLB? Wine is famously not an emulator and as such doesn't really exist/make sense on (non-x86) Android (as it would only be able to execute ARM binaries, not x86 ones). For the downvote: Genuinely curious here on which type of emulator this could affect.

Zefiroj 2y ago

The support for mTHP exists in upstream Linux, but the swap story is not quite there yet. THP availability also needs work and there are a few competing directions. Supporting multiple page sizes well transparently is non-trivial. For a recent summary on one of the approaches, TAO (THP Allocation Optimization), see this lwn article: https://lwn.net/Articles/974636/ https://lwn.net/Articles/974636/

aaron695 2y ago

[dead]