5 ms·

Are you talking about the NIC interrupts or all hardware interrupts? If the later - how do you configure that?

by srd 10y ago

peterwwillis 10y ago
It wouldn't matter because all your interrupts would be handled by a single core, becoming a bottleneck. It also doesn't matter because heavy i/o will take up both system and user cpu, and the 90/10+ split (is the app even multi core??) puts you into full utilization, which is fine for bulk jobs, terrible for high performance requests. Even a single machine at 100% can (in unfortunate circumstances) cause domino effects. Better to build in excess capacity as a buffer for unexpected spikes, which means managing your clusters to not stack jobs which could compete for resources, but also not stack jobs that could unintentionally starve other jobs - this requires intelligent load balancing that's application and job specific. Or a cluster dedicated to specific jobs (which they have, ironically)
- user5994461 10y ago
  You can assign different interrupts to different cores. That's another advanced optimization. e.g. One core per network card.
  - SEJeff 10y ago
    And you can go one step further using receive flow steering[1] or transmit flow steering. Most modern performance oriented network cards (Intel 10G, Solarflare 10G, anything from Mellanox, Chelsio, etc) surface each of these receive queues differently and can be seen as different on the right hand column in /proc/interrupts. You can distribute said rx/tx queues on different cores (ideally) on the same socket (but potentially a different core) as the application for minimum latency. Linux has some really impressive knobs[2] for optimizing these sorts of weird workloads. [1] https://lwn.net/Articles/382428/ https://lwn.net/Articles/382428/ [2] https://www.kernel.org/doc/Documentation/networking/scaling.txt https://www.kernel.org/doc/Documentation/networking/scaling....

Are you talking about the NIC interrupts or all hardware interrupts? If the later - how do you configure that?

by srd 10y ago

Are you talking about the NIC interrupts or all hardware interrupts? If the later - how do you configure that?

It wouldn't matter because all your interrupts would be handled by a single core, becoming a bottleneck. It also doesn't matter because heavy i/o will take up both system and user cpu, and the 90/10+ split (is the app even multi core??) puts you into full utilization, which is fine for bulk jobs, terrible for high performance requests. Even a single machine at 100% can (in unfortunate circumstances) cause domino effects. Better to build in excess capacity as a buffer for unexpected spikes, which means managing your clusters to not stack jobs which could compete for resources, but also not stack jobs that could unintentionally starve other jobs - this requires intelligent load balancing that's application and job specific. Or a cluster dedicated to specific jobs (which they have, ironically)

user5994461 10y ago
You can assign different interrupts to different cores. That's another advanced optimization. e.g. One core per network card.
- SEJeff 10y ago
  And you can go one step further using receive flow steering[1] or transmit flow steering. Most modern performance oriented network cards (Intel 10G, Solarflare 10G, anything from Mellanox, Chelsio, etc) surface each of these receive queues differently and can be seen as different on the right hand column in /proc/interrupts. You can distribute said rx/tx queues on different cores (ideally) on the same socket (but potentially a different core) as the application for minimum latency. Linux has some really impressive knobs[2] for optimizing these sorts of weird workloads. [1] https://lwn.net/Articles/382428/ https://lwn.net/Articles/382428/ [2] https://www.kernel.org/doc/Documentation/networking/scaling.txt https://www.kernel.org/doc/Documentation/networking/scaling....