Wednesday 23 October 2013

using /proc/irq/#/smp_affinity to shield cpus from IRQs

IRQs (interrupt requests) are a request for service at a hardware level from the kernel.

When an IRQ arrives the kernel switches to interrupt context and loads the ISR (interrupt service routine) for the IRQ number which will process the interrupt.

IRQs have an affinity mask which defines which CPUs the ISR can run on. This is defined in /proc/irq/#/smp_affinity

The irqbalance service distributes IRQs across the processors in their associated affinity masks on a multiprocessor system. It uses /proc/irq/#/smp_affinity if it exists, or otherwise falls back to /proc/irq/default_smp_affinity/

The affinity mask in /proc/irq/#/smp_affinity and /proc/irq/default_smp_affinity is a 32-bit hex bitmask of CPUs.
    eg: ffffffff means all 32 CPUs 0-31
If there are more than 32 cores, we build up multiple comma separated 32-bit hex masks.
    eg: ffffffff,00000000 means CPUs 32-63

The irqbalance service uses the environment variable IRQBALANCE_BANNED_CPUS to tell it which CPUs it can't use for ISRs and IRQBALANCE_BANNED_INTERRUPTS to tell it which IRQs to ignore,

IRQBALANCE_BANNED_CPUS follows the same comma separated 32-bit hex format as /proc/irq/#/smp_affinity
IRQBALANCE_BANNED_INTERRUPTS is a space separated list of integer IRQs.

We can shield some CPUs from being interrupted in order to dedicate them to our own tasks.

We can also pin a network data consuming process onto a certain CPU and set the associated NIC (network interface controller) IRQs affinity mask to the same CPU so that they can share cache lines.



No comments:

Post a Comment