Andrey Savochkin leads the development of the kernel portion of OpenVZ, an operating system-level server virtualization solution. In this interview, Andrey offers a thorough explanation of what virtualization is and how it works. He also discusses the differences between hardware-level and operating system-level virtualization, going on to compare OpenVZ to VServer, Xen and User Mode Linux.

Andrey is now working to get OpenVZ merged into the mainline Linux kernel explaining, "virtualization makes the next step in the direction of better utilization of hardware and better management, the step that is comparable with the step between single-user and multi-user systems." The complete OpenVZ patchset weighs in at around 70,000 lines, approximately 2MB, but has been broken into smaller logical pieces to aid in discussion and to help with merging.

With the release of the 2.6.16 Linux kernel, Adrian Bunk reiterated his previously debated intention of maintaining the 2.6.16.y kernel tree well into the future. The first 2.6.x.y release was 2.6.8.1 by Linus Torvalds [story], a quick one line fix for NFS. The idea was revisted a few months later in October of 2004 [story], but didn't actually gain momentum until March of 2005 [story] [story]. Beginning with the 2.6.11 kernel, the process was formalized with Greg KH and Chris Wright officially maintaining 2.6.x.y releases [story] until 2.6.(x+2) is released. For example, stable patches will be applied to the current 2.6.16.y kernel by Greg and Chris until 2.6.18 is released sometime well in the future.

Adrian's plan is to pick up the development of the 2.6.16.y kernel at that point, maintaining it much as the 2.4 kernel tree is is maintained [interview]. His intention is to maintain the tree as long is it is used and people contribute patches. The earlier debate on this idea was met with mixed reactions. At that time Greg KH cautioned, "the time and energy to do this for a long period of time is huge. If I were you, I would listen to the people who have and do maintain these kinds of kernels, it's not a simple job by any means."

A kernel crash dump is a snapshot of system state taken at the time that the kernel crashed, useful for finding and debugging the problem that caused the crash in the first place. There is no standard mechanism for automatiaclly collecting a crash dump on Linux, but there are a number of existing projects working toward efficiently meeting this goal. A "Linux Kernel Dump Summit" was recently mentioned on the lkml, with participants from some of the many crash dump projects looking to standardize the dump process and information collected. A followup email noted, "as memory size grows, the time and space for capturing kernel crash dumps really matter." It went on to examine partial dumps, and full dumps that are compressed. The former risks not collecting information necessary for proper debugging, while the latter risks greatly increasing the amount of time required to collect a dump.

There are a number of existing projects for collecting automatic kernel crash dumps on Linux, including Linux Kernel Crash Dump (LKCD), Mini Kernel Dump (mkdump), kdump, and diskdump (detailed here). Some of these projects also include tools for examining the obtained dumpfiles. Other projects focus just on tools for analyzing kernel crash dumps, including the perl-based Alicia (the Advanced LInux Crash-dump Interactive Analyzer) and Red Hat's crash analysis tool "loosely based on the SVR4 UNIX crash command, but significantly enhanced by completely merging it with the GNU gdb debugger."

In a conversation that began as a request to include the SAS Transport Layer in the mainline Linux kernel, there was an interesting thread regarding specifications. Linux creator Linus Torvalds began the discussion saying, "a 'spec' is close to useless. I have _never_ seen a spec that was both big enough to be useful _and_ accurate. And I have seen _lots_ of total crap work that was based on specs. It's _the_ single worst way to write software, because it by definition means that the software was written to match theory, not reality."

Linus went on to list two reasons to avoid specifications when writing software. First, "they're dangerously wrong. Reality is different, and anybody who thinks specs matter over reality should get out of kernel programming NOW." Second, "specs have an inevitable tendency to try to introduce abstractions levels and wording and documentation policies that make sense for a written spec. Trying to implement actual code off the spec leads to the code looking and working like CRAP." As a "classic example" he pointed to the OSI model, "we still talk about the seven layers model, because it's a convenient model for _discussion_, but that has absolutely zero to do with any real-life software engineering. In other words, it's a way to _talk_ about things, not to implement them. And that's important. Specs are a basis for _talking_about_ things. But they are _not_ a basis for implementing software."

The git directory content manager used to manage the Linux kernel source tree [story] continues to develop at a rapid pace [story]. Keeping up with the latest changes, Jeff Garzik released an updated version of his Kernel Hacker's Guide To Git. He explains, "several changes in git-core have made working with git a lot easier, so be sure to re-familiarize yourself with the development process."

Jeff's short guide is broken into four major sections, 'getting started' which talks about installing the software and getting a copy of the linux kernel source tree, 'basic tasks' which offers examples of keeping up to date with the latest code and merging in your own changes, 'branches' offering examples of creating, using and merging branches, and 'miscellaneous debris' which mentions applying patches from an mbox file and syncronizing tags. Further documentation on the various git commands can be found in the git man pages.

Following the piratical release of 2.6.14-rc2, a brief discussion looked at the advantages of using git to grab the latest version of the kernel code. A small break in service as the master.kernel.org server was situated in its new home [story] caused the 2.6.14-rc2 patch to not show up right away, and led to people pointing out the advantages of using git. When the ketchup script [story] was proposed as an alternative, it was illustrated how git can keep you up to date with the kernel down to a patch by patch level, or with a specific checkpoint. Linus further explained how git can be used to first track down that a bug was introduced between for example rc1-git3 and rc1-git4, and then to use "git-bisect" to further isolate the problem to a specific change.

As for -rc2, Linus noted, "not a whole lot o' excitement, ye scurvy dogs, but it has t' ALSA, LSM, audit and watchdog merges that be missed from -rc1, and a merge series with Andrew. But on t' whole pretty reasonable - you can see t' details in the shortlog (appended)." Evidently Monday the 19'th of September was International Talk Like A Pirate Day.

Linux creator Linus Torvalds sent a reminder to the Linux Kernel Mailing List that the merge window for 2.6.14 is coming to and end. "As per the new merge policies that were discussed during LKS in Ottawa earlier during the summer," Linus explained, "I'm going to accept new stuff for 2.6.14 only during the first two weeks after 2.6.13 was released." The new development policy was first discussed on the lkml with the release of 2.6.13-rc4 [story], and further elaborated with the release of 2.6.13 [story].

The 2.6.13 stable kernel was released on August 28'th [story]. "That release was ten days ago," Linus said, "so you've got four more days before I don't want any big merges." He went on to note that following the merge cutoff 2.6.14-rc1 will be released. "We certainly already have enough for 2.6.14," Linus noted, "but I just wanted to remind people that if they expected me to merge your work, you're getting closer to the cut-off point."

Andrew Morton [interview] provided an update on the current development status of the Linux kernel. As of his announcement, the latest development release is 2.6.13-git5, with 2.6.14 expected around October 7'th. At this time, Andrew is tracking 144 bugs though he notes, "I haven't culled these yet - some may be fixed." Indeed, a number of replies indicated that several of the listed bugs have been fixed.

As for what will likely be merged in the next couple of weeks and be part of the upcoming 2.6.14 release, Andrew listed several filesystems including relayfs [story], v9fs [story], and FUSE [story]. Regarding the latter he noted that he was, "fed up with arguing - any remaining problems can be fixed up in-tree if anyone can think of how to fix them." As for much anticipated Reiser4, Andrew summarized, "Stuck. Last time we discussed this I asked the reiser4 team to develop and negotiate a bullet-point list of things to be addressed. Once that's agreed to, implement it and then we can merge it. None of that has happened and as far as I know, all the review feedback which was provided was lost."

Petr Baudis announced the creation of a homepage for git, the directory content manager used to manage the Linux kernel. Git was originally written by Linus Torvalds in early April of 2005 [story], and is now maintained by Junio Hamano [story]. Other online resources available for the tool include a tutorial that walks through the process of setting up and using git, a man page, and the gitweb interface providing easy browsing of the many kernel trees managed by git. The new webpage explains:

"GIT falls into the category of distributed source code management tools, similar to Arch or Darcs (or, in the commercial world, BitKeeper). Every GIT working directory is a full-fledged repository with full revision tracking capabilities, not dependent on network access to a central server."

The git directory content manager was born in early April of 2005 [story], less than a week after the announcement that BitKeeper would no longer be available free of charge to kernel developers [story]. The tool was originally written by Linux creator Linus Torvalds, and rapidly evolved with the help of an active developer community which Linus repeatedly noted he hoped would eventually take over [story]. A scant two months after development on the tool began, the 2.6.12 kernel was released, managed by git [story].

Now, Linus has announced that Junio Hamano is the new git maintainer. "I always said I didn't really want to maintain it in the long run," Linus explained, "and maybe some of you thought I was just saying that, especially as the weeks dragged out to over three months, but hey, that's just because this thing ended up being a bit bigger and more professional than I originally even envisioned." He went on to note that Junio was the obvious choice, and that this change means he will be able to return his full focus to the Linux Kernel mailing list.

Junio thanked those involved, described his development methods, and discussed the upcoming 1.0 release. Regarding 1.0, he suggested that all features are in, and that he is primarily looking for bugfixes and documentation updates. Following the announcement, Ryan Anderson posted an updated "Git 1.0 Synopis", a brief overview of the tool and its features. His document begins, "Git, sometimes called 'global information tracker', is a 'directory content manager'. Git has been designed to handle absolutely massive projects with speed and efficiency, and the release of the 2.6.12 and (soon) the 2.6.13 version of the Linux kernel would indicate that it does this task well."

Jesper Juhl submitted a small patch to bring the kernel/module.c source file closer in line with the kernel's CodingStyle document. Specifically, he quoted from the CodingStyle document, "don't put multiple statements on a single line unless you have something to hide," which goes on to give an example of how such statements can cause confusion:

         if (condition) do_this;
           do_something_everytime;

2.6 maintainer Andrew Morton [interview] quickly replied, "there are about 88 squillion of these in the kernel. I think it would be a mistake for me to start taking such patches, sorry." David Miller countered, "I disagree. Putting statements on the same line as the if statement hides bugs and makes the code harder to read." Andrew replied, "we all know that, but this means that we spend the next two years fielding an ongoing dribble of trivial patches which distract from real work."

A short discussion followed in which Andrew agreed, "well I suppose I could live with a few REALLY REALLY BIG patches to do this to lots of files, but if it's the old death-by-1000-cuts, I'm gonna call uncle this time." Jesper agreed to begin working on the large patches, to which Andrew repeated, "OK, a few 100k-400k patches would suit." A similar discussion was raised here.

The Linux Kernel Archives are perhaps most familiar through their web interface, http://kernel.org/. The latest release of the Linux kernel is easily found here, along with patches by various Linux kernel hackers, and mirrors of other popular free and open source projects. Countless people worldwide happily rely on this archive without giving much thought to the effort behind it.

In a recent announcement to the Linux Kernel Mailing List, H. Peter Anvin detailed a recent upgrade of the infrastructure behind kernel.org. The new servers were donated by Hewlett-Packard, and are each quad Opterons with 24 gigabytes of RAM and 10 terabytes of disk space. Internet Systems Consortium, Inc. donates the bandwidth in the form of two independent gigabit-connected datacenters, PAIX Palo Alto and e200paul in San Francisco. H. Peter Anvin, Nathan Laredo, and Kees Cook all donate time to maintain the archives. KernelTrap recently spoke with Peter Anvin to learn more about the history behind the Linux Kernel Archives.