"It's been two weeks rather than the usual one, because we've been hunting a really annoying VM regression that not a lot of people seem to have seen, but I didn't want to release an -rc4 with it," began Linux creator Linus Torvalds, announcing the 2.6.34-rc4 Linux kernel. He explained, "we had the choice of either reverting all the anon-vma scalability improvements, or finding out exactly what caused the regression and fixing it. And we got pretty close to the point where I was going to just revert it all." Linus continued:

"Absolutely _huge_ kudos to Borislav Petkov who reported the problem and was able to not just reliably reproduce it, but also test new patches to try to narrow things down at a moments notice. The thing took ten days of emails flying back and forth, and Borislav was there all the time, day and night, through several patches that tried to fix it (several real bugs, but not the one he hit) and lots of patches to just add instrumentation to get us nearer to the cause of the problem. And finally, today, confirmation that we actually nailed the problem. So if anybody has been seeing a oops (or sometimes a GP fault) in page_referenced(), that should be gone now."

As for the rest of the changes, Linus noted, "the bulk of the changes come from drivers - a new network driver (cxgb4), but also updates to the radeon and nouveau drivers. And then there is the random updates everywhere." Read on for the full changelog.

The task of tracking regressions between kernel releases [story] has been picked up by Michal Piotrowski who maintains a "known regressions" wiki page at Kernel Newbies. The list is divided into sections and mailed out to the lkml after each release candidate.

Following the release announcement of the 2.6.21 Linux kernel [story], Adrian Bunk noted that he no longer planned to track regressions [story]. He explained, "if we would take 'no regressions' seriously, it might take 4 or 5 months between releases due to the lack of developer manpower for handling regressions. But that should be considered OK if avoiding regressions was considered more important than getting as quick as possible to the next two week regression-merge window."

Linus Torvalds disagreed with Adrian's view that increasing the length of the release cycle would improve stability, "regressions _increase_ with longer release cycles. They don't get fewer." He went on to add, "you are ignoring the reality of development. The reality is that you have to balance things. If you have a four-month release cycle, where three and a half months are just 'wait for reports to trickle in from testers', you simply won't get _anything_ done. People will throw their hands up in frustration and go somewhere else." He continued:

"Do you really think bugs get fixed faster just because there wasn't a release? Quite the reverse. Bugs get _found_ faster thanks to a release (simply because you tend to get more information thanks to more users), giving the stable people more information, causing the bugs to be able to be found and fixed _more_quickly_ in the stable release than if we had waited for four months to release 2.6.21."

Adrian Bunk posted a couple lists of known regressions that found their way into the 2.6.21-rc1 kernel [story] since the release of the 2.6.20 kernel [story]. Adrian notes that his lists only include bugs that are not yet fixed in Linus' -git tree. In an updated version of his lists he included 19 known regressions, including links to bugzilla or the appropriate mailing list discussion thread. The lists track who submitted the bug, who is currently handling it, who caused it if known, a link to a patch that fixes the problem if any, and the current status.