"As some of the latency junkies on lkml already know, commit 8e3e076 in v2.6.26-rc2 removed the preemptible BKL feature and made the Big Kernel Lock a spinlock and thus turned it into non-preemptible code again. This commit returned the BKL code to the 2.6.7 state of affairs in essence," began Ingo Molnar. He noted that this had a very negative effect on the real time kernel efforts, adding that Linux creator Linus Torvalds indicated the only acceptable way forward was to completely remove the BKL. Ingo explained:

"This task is not easy at all. 12 years after Linux has been converted to an SMP OS we still have 1300+ legacy BKL using sites. There are 400+ lock_kernel() critical sections and 800+ ioctls. They are spread out across rather difficult areas of often legacy code that few people understand and few people dare to touch. It takes top people like Alan Cox to map the semantics and to remove BKL code, and even for Alan (who is doing this for the TTY code) it is a long and difficult task."

Ingo went on to describe how the BKL works, how it differs from other locking mechanisms, and why this complicates removing it permanently from the kernel. He noted that the various dependencies of the lock are lost in the haze of 15 years of code changes, "all this has built up to a kind of Fear, Uncertainty and Doubt about the BKL: nobody really knows it, nobody really dares to touch it and code can break silently and subtly if BKL locking is wrong." He then suggested "changing the rules of the game", creating a "kill-the-BKL" branch which "turns the BKL into an ordinary albeit somewhat big mutex, with a quirky lock/unlock interface called 'lock_kernel()' and 'unlock_kernel()'."

"About 45% architecture updates (counting the include files too), about 30% drivers, and about 25% odds-and-ends. The odds-and-ends are mainly Documentation, filesystems (mostly cifs) and core kernel (scheduler updates etc)," said Linux creator Linus Torvalds, announcing the 2.6.26-rc2 kernel. He added, "if you read the shortlog and get the feeling that most of it is pretty boring small details, you'd be right. There is little exciting there." He continued:

"A fairly small part of it, but quite possibly the most noticeable one, is how the semaphore changes impacted the BKL (the old 'big kernel lock' that is still used for some legacy code, for you non-core people out there), which in the past had different versions ('regular', 'preemptable'). A few months ago we dropped the regular BKL version, but in 2.6.25-rc1 we then had performance (and then correctness) issues with the interaction between the semaphore implementation and the preemptable BKL, so we're back to the old regular version for now."

"So this merge window was somewhat rocky in the sense that there was a lot of arguments about it, but at the same time I at least personally think that from a technical angle, we had somewhat less scary stuff going on than has been almost the rule lately," noted Linux creator Linus Torvalds, announcing the 2.6.26-rc1 kernel. He continued:

"Lots of changes, but nothing that really feels all that fragile to me. Famous last words. I expect that the x86 PAT support (which has been long in the making) has the potential to have some issues, but the obvious problems were hashed out long ago, and while the merge window already showed one bug, that one was fairly benign and quickly fixed."

Linus highlighted, "another feature that is notable not for its size, but because people have tried to get me to merge it for some long is kgdb support. Which really turned out pretty small and clean, once people started putting their effort into making it so." He concluded, "so go out and test it. The diffstat and shortlogs are too big to post here (7500+ commits and the compressed full patch is 8.5MB in size), but one interesting tidbit I found was that during this *one* merge window, we had almost 800 different authors."

"This is starting to get beyond frustrating for me," complained David Miller of the latest merge window, launching what turned into a very lengthy and ongoing discussion about the Linux kernel development process. The concept of a regular "merge window" was first discussed in July of 2005 with the release of the 2.6.14-rc4 kernel, following the 2005 Developers' Summit. From 2.6.14 on, the release of each official 2.6.y kernel has been followed by a two week period during which major changes are merged into the kernel, followed by a 2.6.y-rc1 release. David complained that this particular merge window has been more painful than others, "the tree breaks every day, and it's becoming an extremely non-fun environment to work in. We need to slow down the merging, we need to review things more, we need people to test their [...] changes!"

During the lengthy discussion, Linux creator Linus Torvalds explained:

"The notion that we should even _try_ to aim to slow things down, that one I find unlikely to be true, and I don't even understand why anybody would find it a logical goal? Of course, you will have fewer new bugs if you have fewer changes. But that's not a goal, that's a tautology and totally uninteresting. A small program is likely to have fewer bugs, but that doesn't make something small 'better' than something large that does more. Similarly, a stagnant development community will introduce new bugs more seldom. But does that make a stagnant one better than a vibrant one? Hell no. So what I'm arguing against here is not that we should aim for worse quality, but I'm arguing against the false dichotomy of believing that quality is incompatible with lots of change."

"It's been long promised, but there it is now," began Linux creator Linus Torvalds, announcing the 2.6.25 Linux kernel. He continued, "special thanks to Ingo who found and fixed a nasty-looking regression that turned out to not be a regression at all, but an old bug that just had not been triggering as reliably before. That said, that was just the last particular regression fix I was holding things up for, and it's not like there weren't a lot of other fixes too, they just didn't end up being the final things that triggered my particular worries." Linus added:

"The full changelog from 2.6.24 is 7.5M, with a 12MB compressed patch. Tons and tons has changed, but if you've been following the -rc releases, you'll already know about the big things. The changes from the last rc (-rc9) are fairly small and mostly pretty trivial, and the shortlog is appended. So it's mostly one-liners, with some updates to drivers (net and usb) and to networking that are a bit larger (although a number of the driver updates are things like just new ID's etc)."

More information about the latest release can be found on the KernelNewbies Linux 2.6.25 wiki page.

"Finally found it ... the patch below solves the sparsemem crash and the test system boots up fine now," announced Ingo Molnar. He described the patch as fixing a "memory corruption and crash on 32-bit x86 systems. If a !PAE x86 kernel is booted on a 32-bit system with more than 4GB of RAM, then we call memory_present() with a start/end that goes outside the scope of MAX_PHYSMEM_BITS." He included a source snippet with the loop that could corrupt memory, "depending on what that memory is, we might crash, misbehave or just not notice the bug." Ingo went on to note that the bug was first introduced with sparsemem support in the 2.6.16 kernel:

"I believe this was the reason why my many bisection attempts were unsuccessful: the bug pattern was not stable and seemingly working kernels had the memory corruption too. It was pure luck that v2.6.24 'worked' and v2.6.25-rc9 broke visibly."

Linux creator Linus Torvalds replied, "good job. I've pushed this out, and will let this simmer at least overnight to see if there are any brown-paper-bag issues (either with this or with some last changes from Andrew), but I'm happy, and I think I'll do the real 2.6.25 tomorrow."

"I really don't want to do this, and I was actually hoping to release 2.6.25 last weekend (which is why -rc9 is a few days late - just me hoping to not do another -rc at all), but I've done an -rc9," Linus Torvalds said, announcing the 2.6.25-rc9 kernel. "The changes in -rc9 are pretty small (shortlog appended)," he continued, "and 60% of them are m68k updates - mostly defconfigs. And some doc updates. But there's some network driver updates (tg3 and wireless hostap stand out), some late XFS patches and a mvsas driver update (the mvsas driver is new in 2.6.25, so that's not going to regress anything ;). The rest is mostly one-liners, with a few reverts going on." Linus then explained why he was putting out another release candidate:

"The reason for not doing a 2.6.25 is that some people are making noises about slab/page-alloc setup issues, and I wanted something out this week, but didn't feel comfy doing a final release.

"That said, I think I'll have to do 2.6.25 early next week regardless, because we can't just keep holding things back forever. At some point it will have to turn into a 2.6.25.x issue, and the developers with stuff pending for the next version need to be able to start merging."

"The latest feature release GIT 1.5.5 is available at the usual places," began Git maintainer Junio Hamano, adding "we kept this cycle just slightly over two months, as the previous 1.5.4 cycle was painfully tooooo long."

Git is a distributed version control system that was originally written by Linus Torvalds in April of 2005. It was written to be only a temporary replacement for BitKeeper, which Linus had been using to manage kernel source code since February of 2002. Junio Hamano took over maintainership of Git in July of 2005, and the tool has long since become quite popular outside of even Linux kernel development. Regarding the latest stable release, Junio highlighted some of the changes, including:

"Comes with git-gui 0.10.1; bunch of portability improvement patches coming from an effort to port to Solaris has been applied; 'git fetch' over the native git protocol used to make a connection to find out the set of current remote refs and another to actually download the pack data. We now use only one connection for these tasks; 'git commit' does not run lstat(2) more than necessary anymore; bash completion script (in contrib) are aware of more commands and options; a catch-all 'color.ui' configuration variable can be used to enable coloring of all color-capable commands, instead of individual ones such as 'color.status' and 'color.branch'; bash completion's prompt helper function can talk about operation in-progress (e.g. merge, rebase, etc.); 'git help' can use different backends to show manual pages and this can be configured using 'man.viewer' configuration; 'git gui' learned an auto-spell checking; 'git checkout' and 'git remote' are rewritten in C; two conflict hunks that are separated by a very short span of common lines are now coalesced into one larger hunk, to make the result easier to read."

"No cute April 1st shenanigans, just a regular -rc release that happened to come up today because I was waiting for the input layer oops-fixes to be ready and tested," began Linus Torvalds, announcing the 2.6.25-rc8 kernel on April 1st. He continued, "the bulk of the fixes are the usual random one-liners. [...] A lot of the one-liners are some sparse cleanups, which is probably unnecessary noise at this point, but when Al sends me a series I just tend to apply it because his patches tend to be rather careful and basically always correct." Linus added:

"The big thing that is actually *noticeable* to most people is that this should fix the two top regressions: we've had some suspend-resume regressions due to the stupid ACPI _PTS ordering issues, and while the cleanups were left, the ordering changes were reverted. So that should fix issues for some people (of course, the people who had it fixed are unhappy, but regressions are worse). The other thing that bit a number of people and is now fixed (and that also probably often showed up as a suspend/resume regression) was some 'struct device' lifetime changes that broke the input layer. Thanks to people who debugged that one."

"So this hopefully continues closing various regressions, and most of the changes are pretty small (ie diffstat shows a lot of oneliners). The biggest patches are the trivial powerpc defconfig updates which show up pretty clearly in the dirstat, ie if it weren't for those, the arch/ updates would hardly show up at all," began Linus Torvalds, announcing the 2.6.25-rc7 Linux kernel. He noted that the ps2esdi driver was removed after being marked broken for years, and a new metronomefb.c driver was added for the E-Ink Metronome controller. Linus continued:

"Apart from those, most of the changes really are fairly small and spread out. The scheduler got some tweaking, the memstick driver got some TLC, and cifs and reiserfs had some fixes. The shortlog has more details, but it boils down to some reverts, some docbook fixes, some sparse annotation fixups, a number of trivial patches, and a healthy sprinkling of small fixups."

In summary, Linus suggested, "give it a good testing, because we're hopefully now well on our way towards that eventual real 2.6.25 release!"