KernelTrap

Linux: 2.6.34-rc4, "Hunting A Really Annoying VM Regression"

Jeremy — Tue, 13 Apr 2010 05:16:30 +0000

"It's been two weeks rather than the usual one, because we've been hunting a really annoying VM regression that not a lot of people seem to have seen, but I didn't want to release an -rc4 with it," began Linux creator Linus Torvalds, announcing the 2.6.34-rc4 Linux kernel. He explained, "we had the choice of either reverting all the anon-vma scalability improvements, or finding out exactly what caused the regression and fixing it. And we got pretty close to the point where I was going to just revert it all." Linus continued:

"Absolutely _huge_ kudos to Borislav Petkov who reported the problem and was able to not just reliably reproduce it, but also test new patches to try to narrow things down at a moments notice. The thing took ten days of emails flying back and forth, and Borislav was there all the time, day and night, through several patches that tried to fix it (several real bugs, but not the one he hit) and lots of patches to just add instrumentation to get us nearer to the cause of the problem. And finally, today, confirmation that we actually nailed the problem. So if anybody has been seeing a oops (or sometimes a GP fault) in page_referenced(), that should be gone now."

As for the rest of the changes, Linus noted, "the bulk of the changes come from drivers - a new network driver (cxgb4), but also updates to the radeon and nouveau drivers. And then there is the random updates everywhere." Read on for the full changelog.

Linux: Properly Creating And Testing Patches

Jeremy — Fri, 09 Apr 2010 20:18:27 +0000

"If you're wondering why I'm taking a long time to respond to your patches,", began Theodore Ts'o on the linux-ext4 mailing list, in a thread that offered much insight into how and why to properly submit and test patches. "Patches that are accepted into mainline should do one and only one thing," Ted continued, "so if someone suggests that you make changes to your submitted patch, ideally what you should do is to resubmit the patch with the fixes --- and not submit a patch which is a delta to the previous one." He also noted that patch submitters often greatly outnumber maintainers dictating a higher standard of quality, "consider that for some maintainers, there may be 10 or 20 or 30 or more patch submitters in their subsystem. With that kind of submitter-to-maintainer ratio, the patch submitter simply has to do much more of the work, since otherwise the subsystem maintainer simply can't keep up."

Ted went on to acknowledge, "I happen to believe that we need to encourage newcomers to the kernel developer community, and so I spend more time mentoring people who are new to the process." He noted that his time was finite however, and that patches are accepted more quickly when they are easy to review and integrate. Regarding the filesystem for which the patches had been submitted, he added, "Ext4 is actually quite stable at this point. Very large numbers of people are using it, and most users are quite happy." For this reason, he pointed out that it is even more critical that the patches merged be of high quality. That said, he continued, "there is no such thing as code which is not buggy. For any non-trivial program, it's almost certain there are bugs. [...] Ext4 is not exempt from these fundamental laws of software engineering. 'Code is always buggy until the last user of the program dies'." He tied this back to the importance of testing patches before submitting, "keep in mind that the maxim that code which is not buggy also applies to your patches."

Linux: Memory Compaction

Jeremy — Sat, 03 Apr 2010 00:11:38 +0000

Mel Gorman posted the seventh version of his Memory Compaction patches asking, "are there any further obstacles to merging?" The patches, first posted in May of 2007, provide a mechanism for moving GFP_MOVABLE pages into a smaller number of pageblocks, reducing externally fragmented memory. Mel explains that 'compaction' is another method of defragmenting memory, "for example, lumpy reclaim is a form of defragmentation as was slub 'defragmentation' (really a form of targeted reclaim). Hence, this is called 'compaction' to distinguish it from other forms of defragmentation."

The core compaction patch explains that memory is compacted in a zone by relocating movable pages towards the end of the zone:

"A single compaction run involves a migration scanner and a free scanner. Both scanners operate on pageblock-sized areas in the zone. The migration scanner starts at the bottom of the zone and searches for all movable pages within each area, isolating them onto a private list called migratelist. The free scanner starts at the top of the zone and searches for suitable areas and consumes the free pages within making them available for the migration scanner. The pages isolated for migration are then migrated to the newly isolated free pages."

Linux: Removing The Big Kernel Lock

Jeremy — Thu, 01 Apr 2010 14:52:12 +0000

Arnd Bergmann noted that he's working on removing the BKL from the Linux kernel, "I've spent some time continuing the work of the people on Cc and many others to remove the big kernel lock from Linux and I now have [a] bkl-removal branch in my git tree". He went on to explain that his branch is working, and lets him run the Linux kernel, "on [a] quad-core machine with the only users of the BKL being mostly obscure device driver modules." Arnd noted that this effort has a long history, "the oldest patch in this series is roughly eight years old and is Willy's patch to remove the BKL from fs/locks.c, and I took a series of patches from Jan that removes it from most of the VFS."

Arnd noted that his patch adds a global mutex to the TTY layer, which he called the 'Big TTY Mutex' and described as, "the basic idea here is to make recursive locking and the release-on-sleep explicit, so every mutex_lock, wait_event, workqueue_flush and schedule in the TTY layer now explicitly releases the BTM before blocking." Alan Cox suggested that this portion of the patch was best dropped for now, "it would be nice to get the other bits in first removing BKL from most of the kernel and building kernels which are non BKL except for the tty layer. That (after Ingo's box from hell has run it a bit) would reasonably test the assertion that the tty layer has no BKL requirements that are driven by [code] external to tty layer code." Andrew Morton suggested that the patches be pushed upstream to their appropriate maintainers for an additional sanity check, "Seems that there might be a few tricksy bits in here. Please do push at least the non-obvious parts out to the relevant people."

Website Updated

Jeremy — Wed, 22 Jul 2009 04:44:01 +0000

KernelTrap has been updated to Drupal 6. We're running a fair bit of custom code, so please let me know if you find anything broken or if any of your favorite features are missing. And yes, this upgrade is the first step toward getting regular posts on the website again.

Linux: First Release Of nftables

Jeremy — Wed, 01 Apr 2009 21:05:48 +0000

Netfilter maintainer Patrick McHardy recently announced a first alpha-release of nftables, slated to eventually replace iptables as the standard Linux packet filtering engine. Nftables aims to simplify the kernel ABI, reduce code duplication, improve error reporting, and provide more efficient execution, storage and updates of filtering rules. Patrick began with a high level overview of the three pieces that comprise the firewall, "the kernel provides a netlink configuration interface, as well as runtime ruleset evaluation using a small classification language interpreter. libnl contains the low-level functions for communicating with the kernel, the nftables frontend is what the user interacts with." An insightful overview can be found on lwn.net.

Patrick explained that data is represented internally in a generic fashion, "meaning it's possible to use any matching feature (ranges, masks, set lookups etc.) with any kind of data." He went on to add, "the kernel doesn't have a distinction between matches and targets anymore, operations can be arbitrarily chained, fixing a common complaint that multiple rules are required to f.i. log and drop a packet. Terminal operations will stop evaluation of a rule, even if further operations are specified." Speaking about the the userspace frontend, he noted, "the classification language is based on a real grammar that is parsed by a bison-generated parser (currently, it might have to be replaced) and converted to a syntax tree." Patrick continued, "the frontend supports both dealing with only a single rule at a time for incremental operations, as well as parsing entire files. In the later case verification is performed on all rules and changes are only made after full validation. Currently not implemented, but planned, is transactional semantic where changes are rolled back when the kernel reports an error."