KernelTrap - 2.6.21

Additional CFS Benchmarks

Jeremy — Mon, 17 Sep 2007 14:59:12 +0000

"After posting some benchmarks involving cfs, I got some feedback, so I decided to do a follow-up that'll hopefully fill in the gaps many people wanted to see filled," Rob Hussey began. He added, "this time around I've done the benchmarks against 2.6.21, 2.6.22-ck1, and 2.6.23-rc6-cfs-devel (latest git as of 12 hours ago)." Rob briefly summarized, "the only analysis I'll offer is that both sd and cfs are improvements, and I'm glad that there is a lot of work being done in this area of linux development. Much respect to Con Kolivas, Ingo Molnar, and Roman Zippel, as well all the others who have contributed."

Referring to a chart in which the blue line represented the CFS process scheduler, and the green line represented the SD "staircase" process scheduler, Ingo Molnar noted, "heh - am i the only one impressed by the consistency of the blue line in this graph? :-) [ and the green line looks a bit like a .. staircase? ]" He acknowledged some slowdown in CFS compared to SD in one of the benchmarks, "-ck1 is 0.8% faster in this particular test." Ingo then explained, "many things happened between 2.6.22-ck1 and 2.6.23-cfs-devel that could affect performance of this test. My initial guess would be sched_clock() overhead." In further testing he applied a low-res-sched-clock that resulted in better performance for CFS leading him to conclude, "the performance difference between -ck and -cfs-devel seems to be mostly down to the more precise (but slower) sched_clock() introduced in v2.6.23 and to the startup penalty of freshly created tasks." When asked if the low-res-sched-clock was likely to be merged, Ingo replied:

"I don't think so - we want precise/accurate scheduling before performance. (otherwise tasks working off the timer tick could steal away cycles without being accounted for them fairly, and could starve out all other tasks.) Unless the difference was really huge in real life - but it isn't."

Linux: Continuing 2.6.20.y -stable

Jeremy — Sat, 11 Aug 2007 19:58:22 +0000

Greg KH and Chris Wright have been maintaining a -stable 2.6.x.y patchset for the 2.6.x and 2.6.(x-1) kernels since March of 2005. Thus, with the current stable release being 2.6.22, they maintain -stable patches for 2.6.22 and 2.6.21. 2.4 stable kernel maintainer Willy Tarreau noted the currently high patch rate in each of the 2.6 -stable trees and decided to maintain -stable patches against the 2.6.20 tree until things calm down. Adrian Bunk also continues to maintain a -stable 2.6.16 branch of the Linux kernel. Willy explained about his new 2.6.20 -stable patches:

"I proposed Chris and Greg to continue issuing a few more 2.6.20 releases during the time needed for 2.6.21 and 2.6.22 to show a significant drop in their patch rates, which hopefully will be just a matter of a few releases.

"My goal is *not* to do all the hard work they do, but just to backport from their patches those which are meaningful for 2.6.20. For this reason, 2.6.20 releases will always be slightly late and should not contain patches not merged in more recent releases."

Linux: Debugging With "git bisect"

Jeremy — Fri, 13 Jul 2007 15:10:53 +0000

Following up to a bug report against the 2.6.22 kernel, Andrew Morton and Linus Torvalds offered some tips on how to debug kernel problems. Andrew first pointed to netconsole.txt for instructions on setting up a netconsole, "when the machine has stalled, see if you can get a task trace with ALT-SYSRQ-t. This will require CONFIG_MAGIC_SYSRQ=y and possibly setting ignore_loglevel on the kernel boot command line."

Linus Torvalds suggested "git bisect" as an alternative, "[it] will take some time, but is really a lot easier" He explains, "there's almost 7000 commits in between 2.6.21 and 22, but that still means that in about fourteen recompiles/reboots, "git bisect" should tell us where your problem starts, which will hopefully make it obvious what the problem is (or at least pinpoint it a *lot*)." He goes on to detail how to install git, obtain the latest kernel, and run "git bisect", "doing a git bisect isn't really that hard, but fourteen compiles/reboots will take some time (well, the compiles will, the reboots aren't that bad). But even if you're not a git user, it really is very simple". Specifically, he notes, "start the 'git bisect' with 'git bisect good v2.6.21', 'git bisect bad v2.6.22', and it will pick a kernel version about half-way between the two points, and you can now start testing. For each kernel you try, if it boots fine, do 'git bisect good', otherwise boot into a working kernel, and then do 'git bisect bad'. Git will then pick the next 'halfway' kernel for that case."

Linux: Reviewing the Tickless Kernel for x86-64

Jeremy — Thu, 12 Jul 2007 17:47:02 +0000

Included in Andrew Morton's potential 2.6.23 merge list [story] were a series of patches to make the x86-64 architecture tickless. Andi Kleen, the x86-64 maintainer replied, "I'm sceptical about the dynticks code. It just rips out the x86-64 timing code completely, which needs a lot more review and testing. Probably not .23." Linus Torvalds agreed, "we are *not* going to do another 'rip everything out, and replace it with new code' again. Over my dead body. We're going to do this thing gradually, or not at all." He went on to explain "the patch-set itself actually looks fine, as far as I'm concerned. But it does seem to have that 'enable everything in one go' problem. I'd much rather see one time source at a time being converted, and enabled then and there, so that when people report problems and do a bisection, if it was HPET that broke, you get the commit that changed HPET."

In response to the pains caused by the original dyntick merge in 2.6.21, Ingo Molnar acknowledged, "we had 12 -hrt/dynticks merge related regressions between 2.6.21-rc1 and -final, and 4 after final." He went on to point out, "it's all pretty quiet today on the dynticks regressions front. (there are no open regressions in either the upstream i386 code or in the devel patches we are aware of)." As to the source of the bugs, he explained, "the majority of the above bugs were in the infrastructure code. (the worst was the generic resume/suspend one fixed in 2.6.21.2) And sadly, a fair number of the infrastructure bugs we introduced during the frentic clockevents/dynticks rewrites/redesigns we did between .20 and .21. That was a royally stupid mistake for us to do - instead of patiently waiting for the bugs to be shaken out we destabilized the infrastructure. (it was a 'lets make this thing so nice that it's impossible to reject' instintic gut reaction.)" Linus replied, "one thing I'll happily talk about is that while 2.6.21 was painful, you and Thomas in particular were both very responsible about the thing, so no, I'm not at all complaining or worried about it in that sense! I just really _really_ wish we could have two fairly stable releases in a row. I think 2.6.22 has the potential to be a pretty good setup, and I'd really like to avoid having another 2.6.21 immediately afterwards."

Linux: 2.6.22 Kernel Released

Jeremy — Mon, 09 Jul 2007 00:19:16 +0000

Linux creator Linus Torvalds announced the official release of the 2.6.22 kernel, "it's out there now (or at least in the process of mirroring out - if you don't see everything, give it a bit of time)." He summarized the changes since 2.6.22-rc7 [story]:

"Not a whole lot of changes since -rc7: some small architecture changes (ppc, mips, blackfin), and most of those are defconfig updates. Various driver fixes: new PCI ID's along with some ide, ata and networking fixes (for example - the magic wireless libertas ioctl's got removed, they may be re-added later, hopefully in a more generic form, but in the meantime this doesn't make a release with new interfaces that aren't universally liked)."

The previous stable kernel, 2.6.21, was released a little over two months ago on April 25'th [story]. An overview of all the changes merged into the latest version of the kernel is maintained in the Kernel Newbies wiki. Included in the list of changes are the SLUB allocator which replaced the slab allocator, a new wireless stack, a new firewire stack [story], and support for the Blackfin architecture. Source level changes can be tracked via the gitweb interface to Linus' kernel tree.

Linux: 2.6.22 Coming Soon

Jeremy — Tue, 03 Jul 2007 04:56:44 +0000

Linux creator Linus Torvalds released the 2.6.22-rc7 kernel saying, "it's hopefully (almost certainly) the last -rc before the final 2.6.22 release, and we should be in pretty good shape. The flow of patches has really slowed down and the regression list has shrunk a lot." He briefly summarized the changes in this latest release candidate, "the patches are mostly trivial fixes, a few new device ID's, and the appended shortlog really does pretty much explain it," adding, "final testing always appreciated, of course".

Linux: 2.6.21-ck1, Performance Patchset

Jeremy — Sat, 05 May 2007 12:15:48 +0000

Con Kolivas [interview] continues to maintain the performance oriented -ck patchset that he started in early 2004 [story], "this patchset is designed to improve system responsiveness and interactivity. It is configurable to any workload but the default -ck patch is aimed at the desktop and -cks is available with more emphasis on serverspace." In Con's latest release, 2.6.21-ck1, he notes that he has updated the patchset to include his improved SD cpu scheduler [story], "the staircase-deadline cpu scheduler has replaced the old staircase design in this version."

Con goes on to explain, "the staircase-deadline cpu scheduler can be set in either purely forward-looking mode for absolutely rigid fairness and cpu distribution according to nice level, or it can allow a small per-process history to smooth out cpu usage perturbations common in interactive tasks by enabling this sysctl. While small fairness issues can arise with this enabled, overall fairness is usually still strongly maintained and starvation is never possible. Enabling this can significantly smooth out 3d graphics and games." Swap prefetch [story] is also among the patches included in the -ck patchset.