KernelTrap - merge window

Active Merge Windows

Jeremy — Fri, 02 May 2008 20:41:55 +0000

"This is starting to get beyond frustrating for me," complained David Miller of the latest merge window, launching what turned into a very lengthy and ongoing discussion about the Linux kernel development process. The concept of a regular "merge window" was first discussed in July of 2005 with the release of the 2.6.14-rc4 kernel, following the 2005 Developers' Summit. From 2.6.14 on, the release of each official 2.6.y kernel has been followed by a two week period during which major changes are merged into the kernel, followed by a 2.6.y-rc1 release. David complained that this particular merge window has been more painful than others, "the tree breaks every day, and it's becoming an extremely non-fun environment to work in. We need to slow down the merging, we need to review things more, we need people to test their [...] changes!"

During the lengthy discussion, Linux creator Linus Torvalds explained:

"The notion that we should even _try_ to aim to slow things down, that one I find unlikely to be true, and I don't even understand why anybody would find it a logical goal? Of course, you will have fewer new bugs if you have fewer changes. But that's not a goal, that's a tautology and totally uninteresting. A small program is likely to have fewer bugs, but that doesn't make something small 'better' than something large that does more. Similarly, a stagnant development community will introduce new bugs more seldom. But does that make a stagnant one better than a vibrant one? Hell no. So what I'm arguing against here is not that we should aim for worse quality, but I'm arguing against the false dichotomy of believing that quality is incompatible with lots of change."

Defaulting To 4K Stacks

Jeremy — Wed, 23 Apr 2008 01:42:24 +0000

Andrew Morton replied to a commit message making 4k stacks the default, saying, "this patch will cause kernels to crash." Ingo Molnar replied, "what mainline kernels crash and how will they crash? Fedora and other distros have had 4K stacks enabled for years." He added, "we've conducted tens of thousands of bootup tests with all sorts of drivers and kernel options enabled and have yet to see a single crash due to 4K stacks." During the lengthy discussion it was suggested that nfs+xfs+raid kernel configurations, and using ndiswrapper are the most common reasons for overflowing a 4K stack size.

Andi Kleen questioned the usefulness of 4k stacks, "as far as I can figure out they are not [a worthy goal]. They might have been a worthy goal on crappy 2.4 VMs, but these times are long gone." Arjan van de Ven suggested that though the 2.6 VM is much improved over the 2.4 VM, fragmentation with 8K stacks remains an unsolvable problem, "it's basic math; the Linux VM gets to deal with both short and long lasting allocations; no matter how hard you try to get some degree of fragmentation; especially due to the 15:1 acceleration you get due to the lowmem issue. And before you say 'you should use 64 bit on such machines'; I would love it if more people used 64 bit linux. Sadly the adoption rate of that is not very good still.... by far ;(" In another email, Arjan listed two advantages to 4K stacks, "1) less memory consumption in the lowmem zone (critical for enterprise use, also good for general performance), and 2) kernel stacks at 8K are one of the most prominent order-1 allocations in the kernel; again with big-memory systems the fragmentation of the lowmem zone is a problem (and the distros that ship 4K stacks went there because of customer complaints)".

Kgdb Light

Jeremy — Sun, 10 Feb 2008 02:08:48 +0000

"While this is probably one of the last days of the merge window, please still consider pulling the 'kgdb light' git tree," began Ingo Molnar, explaining:

"This is a slimmed-down and cleaned up version of KGDB that i've created out of the original patches that we submitted two weeks ago. I went over the kgdb patches with Thomas and we cut out everything that we did not like, and cleaned up the result. KGDB is still just as functional as it was before (i tested it on 32-bit and 64-bit x86) - and any desired extra capability or complexity should be added as a delta improvement, not in this initial merge."

Ingo noted that the previous merge request modified 41 files, while this new merge request modifies only 22 files. Among the changes, he highlighted, "removed _all_ critical path impact, even if KGDB is enabled and active; removed all the lowlevel serial drivers; added a redesigned and cleaned up version of the 'KGDB over polled consoles' approach; removed the longjump code; removed the module symbol hacks; removed the GTOD/clocksource hacks; removed the softlockup hacks; removed the toplevel Makefile changes; removed the might_sleep scheduler hack; and did lots of other cleanups and rewrites as well." Ingo summarized, "as a result, this kgdb series has _obviously_ zero impact on the kernel, because it just does not touch any dangerous codepath. From this point on KGDB can evolve in small, well-controlled baby steps, as all other kernel code as well. And the resulting kgdb is still very functional: it can still break into a kernel (via SysRq-G), can catch crashes, can single-step, etc. It's already a quite usable first step."

Quote: Amazed How Little Gets Broken

Jeremy — Mon, 04 Feb 2008 04:05:59 +0000

"We all wear the brown paper bag on occasion, and with the 'merge maelstrom' during each merge window, I'm quite frankly amazed at how _little_ stuff gets broken overall."

Kernel Rate of Change

Jeremy — Sun, 03 Feb 2008 08:11:44 +0000

"I re-ran some statistics the other day on our kernel development rate, and changed my formula after Andrew accused me of severely undercounting the rate of change," noted Greg KH during a discussion about the stability of the Linux kernel while undergoing significant changes. He continued, "turns out that as of 2.6.24-rc8 for the 2.6.24 kernel release we did: lines added per day, 4945; lines removed per day, 2006; lines modified per day, 1702". Greg added:

"And note, that is real stuff, not renames or file moves at all, git handles not reporting that. That's for the 99 days that it took to do 2.6.24-rc8 (I need to re-run the scripts now that 2.6.24 is out.) It's fricken scarily amazing that things are still working at all... Just something to make you all sleep well at night :)"

Core Driver Patches in the 2.6.25 Merge Window

Jeremy — Wed, 30 Jan 2008 02:58:46 +0000

Prefacing a series of 196 patches, Greg KH noted, "due to the low level nature of these patches, and because they touch so many different parts of the kernel, a number of the subsystem maintainers have asked me to get them in first to make merging other trees easier." Linus Torvalds agreed and quickly merged the patches into his tree. Greg summarized the many changes:

"They can be broken down into these major areas: Documentation updates (language translations and fixes, as well as kobject and kset documentation updates.); major kset/kobject/ktype rework and fixes; struct bus_type has been reworked to now handle the lifetime rules properly, as the kobject is properly dynamic; struct driver has also been reworked, and now the lifetime issues are resolved; the block subsystem has been converted to use struct device now, and not 'raw' kobjects; nozomi driver is added; lots of class_device conversions to use struct device instead."

Scheduler Merges for 2.6.25

Jeremy — Sat, 26 Jan 2008 15:02:10 +0000

Ingo Molnar posted a merge request for the latest git scheduler tree summarizing, "it contains various enhancements to the scheduler - find the full shortlog is below. 96 commits from 19 authors - scheduler developers have been busy again. :-/" He added, "the scheduling behavior of the kernel to normal users should not change over v2.6.24, but there are a good number of new features and enhancements under the hood." Ingo went on to list a number of these new features, including:

"Various instrumentation and debugging enhancements from Arjan van de Ven; Peter Zijlstra's RT time limit and RT throttling code for the RT scheduling class; Paul E. McKenney's preemptible RCU code; refcount based CPU-hotplug rework by Gautham R Shenoy; there's serious interest in running RT tasks on enterprise-class hardware, so Steven Rostedt and Gregory Haskins wrote a large number of enhancements to the RT scheduling class and load-balancer; Peter Zijlstra's high-resolution scheduler tick code; [...] and a good number of other, smaller enhancements."