7 ms·

NUMA can cause really crappy performance. We deployed a Go based LLM gateway in Kubernetes deployed on a server with hundreds of CPU cores. We didn't explicitly

by lukax 3mo ago

NUMA can cause really crappy performance. We deployed a Go based LLM gateway in Kubernetes deployed on a server with hundreds of CPU cores. We didn't explicitly set GOMAXPROCS so Go runtime scheduled goroutines over different CPUs and it constantly used 200% CPU and GC was causing latency spikes. Then we set GOMAXPROCS 8 and all performance issues went away. Until recently Kubernetes didn't work well with NUMA.

gopalv 3mo ago
> Kubernetes deployed on a server with hundreds of CPU cores Was that a Power9 or some sort of IBM machine? Not all NUMA is the same, ccNUMA from the Intel is a different beast from the PPC version of the same.
re-thc 3mo ago
Is this on AMD? I wonder if it's all to do with NUMA or their CCD architecture etc (well these days Intel and everyone also does it to some extent).
- deleted 3mo ago
  [deleted]
- toast0 3mo ago
  Hundreds of cores is likely two sockets and so you've got NUMA there. Scaling to large core counts has a lot of gotchas.
- Twirrim 3mo ago
  Intel suffers just as much when NUMA enters the picture, even prior to CCD style architecture. That extra latency hop across to the other core to get at memory is absolutely crippling, especially in a hot loop. It requires very careful handling, while being this kind of invisible element (unless you know to look for it, nothing will draw your attention to it)
CarRamrod 3mo ago
There is one instance where the NUMA performance never disappoints: https://www.youtube.com/watch?v=Cqd1Gvq-RBY https://www.youtube.com/watch?v=Cqd1Gvq-RBY
- drunkboxer 3mo ago
  There are in fact two instances https://www.youtube.com/watch?v=ZBKm1MBsTbk https://www.youtube.com/watch?v=ZBKm1MBsTbk
strifey 3mo ago
Heck, we saw crazy performance degradation with redis when its memory usage exceeded a single NUMA block. Not much to be done about that at the k8s level when redis is single-threaded. Have to be super conscious of the underlying hardware at that point.
- 0x457 3mo ago
  In that case I run one redis instance per NUMA domain. On my home server I essentially split machine in two and treat it as two distinct machines. PCIe devices attached to a proper domain etc.
nanibot 3mo ago
I don't see how GOMAXPROCS alone can help here though. You would have to use Topology Manager (single-node policy) to avoid cross-NUMA allocations. This is in addition to other managers - Memory and CPU Manager. CPU Pinning (via CPU Manager's Static Policy) will also be required to ensure your processes don't just get a CFS quota/share but are actually pinned onto specific CPU cores.
pjmlp 3mo ago
Some lessons are never lost it seems, back when Windows NT was recent, we had to lock threads/processes to specific CPUs on SMP machines (affinity), exactly for similar reasons.

by lukax 3mo ago

gopalv 3mo ago

> Kubernetes deployed on a server with hundreds of CPU cores Was that a Power9 or some sort of IBM machine? Not all NUMA is the same, ccNUMA from the Intel is a different beast from the PPC version of the same.

re-thc 3mo ago

Is this on AMD? I wonder if it's all to do with NUMA or their CCD architecture etc (well these days Intel and everyone also does it to some extent).

deleted 3mo ago
[deleted]
toast0 3mo ago
Hundreds of cores is likely two sockets and so you've got NUMA there. Scaling to large core counts has a lot of gotchas.
Twirrim 3mo ago
Intel suffers just as much when NUMA enters the picture, even prior to CCD style architecture. That extra latency hop across to the other core to get at memory is absolutely crippling, especially in a hot loop. It requires very careful handling, while being this kind of invisible element (unless you know to look for it, nothing will draw your attention to it)

CarRamrod 3mo ago

There is one instance where the NUMA performance never disappoints: https://www.youtube.com/watch?v=Cqd1Gvq-RBY https://www.youtube.com/watch?v=Cqd1Gvq-RBY

drunkboxer 3mo ago
There are in fact two instances https://www.youtube.com/watch?v=ZBKm1MBsTbk https://www.youtube.com/watch?v=ZBKm1MBsTbk

strifey 3mo ago

Heck, we saw crazy performance degradation with redis when its memory usage exceeded a single NUMA block. Not much to be done about that at the k8s level when redis is single-threaded. Have to be super conscious of the underlying hardware at that point.

0x457 3mo ago
In that case I run one redis instance per NUMA domain. On my home server I essentially split machine in two and treat it as two distinct machines. PCIe devices attached to a proper domain etc.

nanibot 3mo ago

I don't see how GOMAXPROCS alone can help here though. You would have to use Topology Manager (single-node policy) to avoid cross-NUMA allocations. This is in addition to other managers - Memory and CPU Manager. CPU Pinning (via CPU Manager's Static Policy) will also be required to ensure your processes don't just get a CFS quota/share but are actually pinned onto specific CPU cores.

pjmlp 3mo ago

Some lessons are never lost it seems, back when Windows NT was recent, we had to lock threads/processes to specific CPUs on SMP machines (affinity), exactly for similar reasons.