login
Search
Hi
I have a system that consists of 14 processoes .Under high load one processor shows 100% usage and rest show normal behaviour.Driver is NAPI compliant.So is it a driver issue or scheduler issue?. Kernel is 2.6. The output of "top" commands show that the cpu that is 100% consumed is running ksoftirqd and it is 0% idle and 100% SI.
Interrupts are evenly distributed over the system
supposed to?
that sounds quite normal if you switch off the NIC's interrupt and change to polling mode, which is what NAPI is doing to avoid IRQ storms on high traffic.
There are cheap ethernet chipsets which force the CPU to do a lot of work like calculating checksums or copying around data, and there are more intelligent server NICs that have engines for this in hardware and are more CPU friendly. Which ethernet chipset do you have? Have you tried using multiple interfaces in parallel?
You ask "So is it a driver issue or scheduler issue?". Which driver in which version and which scheduler and version are you using? "Kernel is 2.6." is a _very_ broad description.
no. of Interrupts reduction
In e1000(implemented in kernel 2.6.21)How interrupts no. have been reduced because I didnot find anything in the handler routine of this driver relevant between the line that netif_rx_schedule_prep and netif_rx_schedule. Usually this is the place where receive interrupts are disabled. Or some different mechanism has been implemented in order to reduce the no. of interrupts in the case of NAPI.
NAPI Disabled
After some work I found out that My NIC card does not support some of the feature like support for MSI-X,RSS etc. so the issue will remain there.Now I disabled the NAPI.now even in low traffic all the NIC interrupts are falling on a single cpu.
In my system no irqbalance thread is running.But by default smp_affinity value is ffff so I hope that it should be even over all the cpu.but all NIC interrupts maps to any one cpu at one time. If I try to throw it to multiple cpu by writing something like 3 or 7 to smp_affinity it crashes.though default is ffff.
Design Issue
Kernel 2.6.21
Driver e1000
Yes,The cpu that will receive the interrupt for the very first time will schedule the softirq(net_rx_action) and will execute on the same processor .Moreover the data is comming from only one interface so no chance of this softirq to sehedule simultaneously on different cpu for other interface (as there is only one interface). Fine---this is how NAPI works ------is there any other way around so that even if there is one interface its processing can be distributed over more than one cpu. some changes in the driver part??
LOAD BALANCE
NAPI is disabled
my driver code sets eth0 interrupt affinity to all cores only when NAPI is enabled?
why all interrupts are getting to a single cpu irrespective of the load ?Though /proc/irq/45/smp_affinity is ffff. then why all the interrupts are not getting upon all the cores?
when I try to do somerhinf #echo 0f > /proc/irq/45/smp_affinity system gets crashed.why?
This seems to be a well-known
This seems to be a well-known behavior that the CPU utilization is not evenly distributed across multiple cores under high load. I've developed a very naive kernel module that will dispatch the received packets to other cores. The performance gain was about 50-200% in term of packet rate depending on hardware configuration and traffic pattern. The code is under GPL but lack of comments and documentations. Let me know if you're interested.
Code
If you have some code then please share with me . For now I put a lock in driver and now it ig getting executed over all cpus --I mean load gets distributed but the problem now I face is that performance has gone down.