6 ms·

Being a user of both Linux and PostrgreSQL, I'm very interested in this issue, but I only understand some of the words... Could everybody wiser than me tell me

Being a user of both Linux and PostrgreSQL, I'm very interested in this issue, but I only understand some of the words... Could everybody wiser than me tell me if I should be concerned and the possible implications of these decisions? Should I invest in alternative platforms?

treenyc 12y ago
same here. Was a FreeBSD user, but find hard to find VM a few years ago that support FreeBSD, may switch back, if this issue is NOT addressed.
- pjc50 12y ago
  If you're running in a VM, the VM may impose its own constraints on the disk write and virtual memory issues.
- dfox 12y ago
  First issue is relevant only on systems that have more than one NUMA node, which is probably every meaningful physical server and essentially no VM (at least on Xen, multiprocessor VMs are single NUMA node), as it does not make much sense to advertise NUMA topology to guest VMs. Second issue is relevant for postgresql mostly only if you use very large shared_buffers which anyway is not recommended for general workloads. Writing page that exists on disk and was not read short time before is not especially common thing to do.
  - rodgerd 12y ago
    NUMA can absolutely ping you in virtual servers, but without access to the hypervisor you'll never know why it's happening (JVMs straddling NUMA regions have caused me pain in the past, when the guest was split across memory regions).
    - dfox 12y ago
      The point is that kernel inside VM guest knows nothing about NUMA, so it cannot do any kind of NUMA optimizations hence such optimizations cannot hurt performance as they do not happen at all.
      - rodgerd 12y ago
        Actually, KVM can allow you to create NUMA domains inside the guest, for better or worse.
ars 12y ago
These are very minor issues and will be noticed by very few. If you are running a huge database and need every possible bit of performance this will matter, otherwise it's not something to worry about.
artumi-richard 12y ago
You probably don't need to worry about it, it's just further proof that the postgresql devs do it right, and you're in safe hands.
fauigerzigerk 12y ago
It is something that could be helpful to have at the back of your mind if you're tasked with optimizing postgres and OS settings for big workloads. But it's one of a myriad of little things, not something that could inform a platform decision. It's much more interesting for kernel devs than it is for postgresql users. I think what this shows is that issues related to interactions between RAM, caches and CPU cores are becoming a lot more complex on all platforms.
Orva 12y ago
In NUMA front Linux is, as far as I know, leaps and bounds ahead of FreeBSD, Solaris and Windows. All four platforms offer ways to tune process and memory allocations by hand, but Linux is only platform that puts lot of work to make automatic NUMA scheduling actually work. It is actually pretty difficult problem. Something to read if you are interested: http://queue.acm.org/detail.cfm?id=2513149 http://queue.acm.org/detail.cfm?id=2513149 http://lwn.net/Articles/591995/ http://lwn.net/Articles/591995/ http://lwn.net/Articles/568870/ http://lwn.net/Articles/568870/
Spooky23 12y ago
Back in ye olden times when I was an Informix DBA, we had to worry about stuff like this for storage. It was always a fight with the storage guys, because they wanted to use their fancy Veritas File System for optimizing disk utilization, and us prima-donna DBAs wanted raw LUNs and allow the database engine to manage our disk, because it maximized our transaction throughput. Some DBAs even wanted whole disks allocated, so they could control where data lived from a disk geometry POV. There were (mostly) valid arguments for doing this, most of which have gone away over the years. This is an issue like my disk issue -- corner cases that need to be thought about in situations where you are investing lots of engineering effort into your databases. If you don't have a couple of angry DBAs whom you're always arguing with, you don't need to worry about this.

treenyc 12y ago

same here. Was a FreeBSD user, but find hard to find VM a few years ago that support FreeBSD, may switch back, if this issue is NOT addressed.

pjc50 12y ago
If you're running in a VM, the VM may impose its own constraints on the disk write and virtual memory issues.
dfox 12y ago
First issue is relevant only on systems that have more than one NUMA node, which is probably every meaningful physical server and essentially no VM (at least on Xen, multiprocessor VMs are single NUMA node), as it does not make much sense to advertise NUMA topology to guest VMs. Second issue is relevant for postgresql mostly only if you use very large shared_buffers which anyway is not recommended for general workloads. Writing page that exists on disk and was not read short time before is not especially common thing to do.
- rodgerd 12y ago
  NUMA can absolutely ping you in virtual servers, but without access to the hypervisor you'll never know why it's happening (JVMs straddling NUMA regions have caused me pain in the past, when the guest was split across memory regions).
  - dfox 12y ago
    The point is that kernel inside VM guest knows nothing about NUMA, so it cannot do any kind of NUMA optimizations hence such optimizations cannot hurt performance as they do not happen at all.
    - rodgerd 12y ago
      Actually, KVM can allow you to create NUMA domains inside the guest, for better or worse.

ars 12y ago

These are very minor issues and will be noticed by very few. If you are running a huge database and need every possible bit of performance this will matter, otherwise it's not something to worry about.

artumi-richard 12y ago

You probably don't need to worry about it, it's just further proof that the postgresql devs do it right, and you're in safe hands.

fauigerzigerk 12y ago

It is something that could be helpful to have at the back of your mind if you're tasked with optimizing postgres and OS settings for big workloads. But it's one of a myriad of little things, not something that could inform a platform decision. It's much more interesting for kernel devs than it is for postgresql users. I think what this shows is that issues related to interactions between RAM, caches and CPU cores are becoming a lot more complex on all platforms.

Orva 12y ago

In NUMA front Linux is, as far as I know, leaps and bounds ahead of FreeBSD, Solaris and Windows. All four platforms offer ways to tune process and memory allocations by hand, but Linux is only platform that puts lot of work to make automatic NUMA scheduling actually work. It is actually pretty difficult problem. Something to read if you are interested: http://queue.acm.org/detail.cfm?id=2513149 http://queue.acm.org/detail.cfm?id=2513149 http://lwn.net/Articles/591995/ http://lwn.net/Articles/591995/ http://lwn.net/Articles/568870/ http://lwn.net/Articles/568870/

Spooky23 12y ago

Back in ye olden times when I was an Informix DBA, we had to worry about stuff like this for storage. It was always a fight with the storage guys, because they wanted to use their fancy Veritas File System for optimizing disk utilization, and us prima-donna DBAs wanted raw LUNs and allow the database engine to manage our disk, because it maximized our transaction throughput. Some DBAs even wanted whole disks allocated, so they could control where data lived from a disk geometry POV. There were (mostly) valid arguments for doing this, most of which have gone away over the years. This is an issue like my disk issue -- corner cases that need to be thought about in situations where you are investing lots of engineering effort into your databases. If you don't have a couple of angry DBAs whom you're always arguing with, you don't need to worry about this.