Linux news

Btrfs Online Resizing, Ext3 Conversion, and More

Submitted by Jeremy
on January 17, 2008 - 8:06am
Linux news

Chris Mason announced version 0.10 of his new Btrfs filesystem, listing the following new features, "explicit back references, online resizing (including shrinking), in place conversion from Ext3 to Btrfs, data=ordered support, mount options to disable data COW and checksumming, and barrier support for sata and IDE drives". He noted that the disk format in v0.10 has changed, and is not compatible with the v0.9 disk format. Regarding back reference support, Chris explained, "the core of this release is explicit back references for all metadata blocks, data extents, and directory items. These are a crucial building block for future features such as online fsck and migration between devices. The back references are verified during deletes, and the extent back references are checked by the existing offline fsck tool." He then detailed the new Ext3 to Btrfs conversion utility:

"The conversion program uses the copy on write nature of Btrfs to preserve the original Ext3 FS, sharing the data blocks between Btrfs and Ext3 metadata. Btrfs metadata is created inside the free space of the Ext3 filesystem, and it is possible to either make the conversion permanent (reclaiming the space used by Ext3) or roll back the conversion to the original Ext3 filesystem."

2.6.24-rc8, "A Final Shakedown"

Submitted by Jeremy
on January 16, 2008 - 6:05am
Linux news

"I do hate doing -rc's for so long, but I hate releasing when not feeling it's simmered enough even more. And the changes since -rc7 are bigger than the changes between -rc6 and -rc7 were (partly probably because people were still on vacation between -rc6 and -rc7, so we had something of a small trickle come in afterwards)," Linus Torvalds began, explaining why he posted another release candidate rather than the official 2.6.24 kernel. He continued, "that said, the changes here really aren't that big, and the shortlog is fairly boring. So I'm pretty sure this is the last -rc, and the final 2.6.24 will probably be out next weekend or so. But in the meantime, let's give this a final shakedown, and see if we can fix any last regressions still." Linus went on to summarize the changes:

"Drivers, networking, some arch updates, and ACPI. A fair number of really small commits. I honestly can't really improve on the appended shortlog - there isn't any over-arching theme, except for 'lots of small boring fixes'. Which is as it should be, of course."

Further Oops Insights

Submitted by Jeremy
on January 14, 2008 - 2:44pm
Linux news

"[The] text below is mostly for the benefit of newbies - it's more along the lines of 'how to get from [a] bug report to the source of [the] bug', with more details than normal," began Al Viro, offering a full review of another Linux kernel oops in an effort to educate more people on how this is done. Al's walk through included a patch to fix the bug that caused the oops. He noted:

"This might be worth doing on [a] more or less regular basis, especially if more people join the fun; everyone [has] their own set of tricks in [this] area and making it easier to gather might help a lot of people. It's not just about oops-tracing per se, of course - Arjan's site gives a nice collection of those, so that makes an obvious starting point."

Speeding Up Fsck With Metaclustering

Submitted by Jeremy
on January 14, 2008 - 6:52am
Linux news

"This patch speeds up e2fsck on Ext3 significantly using a technique called Metaclustering," stated Abhishek Rai. In an earlier thread he quantified this claim, "this patch will help reduce full fsck time for ext3. I've seen 50-65% reduction in fsck time when using this patch on a near-full file system. With some fsck optimizations, this figure becomes 80%." Most criticism so far has been in regards to formatting issues with the patch preventing it from being easily tested, resolved in the latest postings. It was also cautioned that the patch affects a significant amount of ext3 code, and thus will require very heavy testing. Abhishek described how the patch offers its significant gains for e2fsck:

"Metaclustering refers to storing indirect blocks in clusters on a per-group basis instead of spreading them out along with the data blocks. This makes e2fsck faster since it can now read and verify all indirect blocks without much seeks. However, done naively it can affect IO performance, so we have built in some optimizations to prevent that from happening. Finally, the benefit in fsck performance is noticeable only when indirect block reads are the bottleneck which is not always the case, but quite frequently is, in the case of moderate to large disks with lot of data on them. However, when indirect block reads are not the bottleneck, e2fsck is generally quite fast anyway to warrant any performance improvements."

Decoding Oops

Submitted by Jeremy
on January 9, 2008 - 7:49am
Linux news

"This week, a total of 49 oopses and warnings have been reported, compared to 53 reports in the previous week," Arjan van de Ven noted, sending out a list of the week's top 10 kernel oopses. Al Viro suggested, "FWIW, people moaning about the lack of entry-level kernel work would do well by decoding those to the level of 'this place in this function, called from <here>, with so-and-so variable being <this>' and posting the results." This was met by multiple requests for documentation on how to actually decode an oops. Linus Torvalds explained:

"It's actually not necessarily at all that trivial, unless you have a deep understanding of the code generated for the architecture in question (and even then, some oopses take more time to figure out than others, thanks to inlining and tailcalls etc). If the oops happened with a kernel you generated yourself, it's usually rather easy. Especially if you said 'y' to the 'generate debugging info' question at configuration time."

Linus went on to detail how to debug a random oops reported on the lkml, "you will generally have to disassemble the hex sequence given in the oops (the 'Code:' line), and try to match it up against the source code to try to figure out what is going on." He then offered a number of tips on how this is best accomplished, continuing with an example walking through one of the reports oops. Al Viro replied describing his own methods of accomplishing the same thing, walking through of another oops and isolating a bug.

2.6.24-rc7, A Lot of Rather Small Changes

Submitted by Jeremy
on January 7, 2008 - 3:09am
Linux news

"It's been two weeks since rc6, but let's face it, with xmas and new years (and birthdays) in between, there hasn't actually been a lot of working days, and the incremental patch from -rc6 is about half the size of the one from rc5->rc6," began Linus Torvalds, announcing the release of the 2.6.24-rc7 Linux kernel. He then quipped, "and I'll be charitable and claim it's because it's all stabilizing, and not because we've all been in a drunken stupor over the holidays." Linus quickly summarized the changes:

"The shortlog (appended below) is short and fairly informative. It's all really just a lot of rather small changes. The diffstat shows a lot of one- and two-liners, with just a few drivers (and the Cell platform) getting a bit more attention, and the SLUB support of /proc/slabinfo showing up as a blip."

Sleepy Linux

Submitted by Jeremy
on January 4, 2008 - 8:11am
Linux news

"Current Linux versions can enter suspend-to-RAM just fine, but only can do it on explicit request. But suspend-to-RAM is important, eating something like 10% of [the] power needed [compared to an] idle system. Starting suspend manually is not too convenient," began Pavel Machek, describing an idea he referred to as Sleepy Linux. He continued, "[starting suspend manually] is not an option on multiuser machines, and even on single user machines some things are not easy: 1) Download this big chunk in Mozilla, then go to sleep; 2) Compile this, then go to sleep; 3) You can sleep now, but wake me up in 8:30 with mp3 player". Pavel provided a simple not-fully-functional patch, then described his proposed solution:

"Today's hardware is mostly capable of doing better: with correctly set up wakeups, machine can sleep and successfully pretend it is not sleeping -- by waking up whenever something interesting happens. Of course, it is easier on machines not connected to the network, and on notebook computers."

2.4.36 Stable Release

Submitted by Jeremy
on January 3, 2008 - 3:24am
Linux news

"New year, new kernel: Linux 2.4.36 is finally ready and has been checked long enough to be released. Quite a bunch of bugs, build errors and security issues have been fixed since 2.4.35, but all of those fixes were merged into 2.4.35-stable," 2.4 maintainer Willy Tarreau stated, announcing the latest 2.4 stable Linux kernel. He noted, "I should say that I'm quite satisfied of this dual-branch release model which proves to be very successful at separating quick fixes from changes which require more thorough testing." Willy went on to add:

"Concerning future versions, I have nothing pending in the queue anymore. I will then go on with 2.4.36.X when bug fixes come in, and only open 2.4.37 when I get something which I do not consider suitable for 2.4.36.X."

The previous 2.4.35 stable kernel was released in July of 2007. Source level changes can be viewed through the linux-2.4 gitweb interface.

Dusting Off the 0.01 Kernel

Submitted by Jeremy
on January 2, 2008 - 1:35pm
Linux news

Abdel Benamrouche announced that he has updated the original 0.01 Linux kernel to compile with GCC-4.x, allowing it to run on emulators such as QEMU and Bochs. After applying his series of small patches, Abdel explains that the 0.01 kernel can be built on a system running the 2.6 Linux kernel. He added that he's successfully ported bash-3.2, portions of coreutils-6.9, dietlibc-0.31 (instead of glibc), bin86-0.16.17, make-3.81, ncurses-2.0.7, and vim-7.1 all to run on his modified 0.01 kernel.

Collecting Kernel Oops Data

Submitted by Jeremy
on December 17, 2007 - 4:55pm
Linux news

"The http://www.kerneloops.org website collects kernel oops and warning reports from various mailing lists and bugzillas," noted Arjan van de Ven, announcing the new website. He included a summary of the top 10 oopses collected in the past 7 days noting, "this is the first such report that I'm posting; Please let me know if this is useful or not."

Feedback was positive. Andrew Morton commented, "well that would have been fun to write." Steven Richter expressed some concern about the tool counting the same bug report duplicate times when found in different places. Arjan aggreed, "this is true however it's .. a hard issue. It's really hard to distinguish a duplicate report from two reports of the same bug." Another concern was in separating oops generated by 2.6.X-rcY kernels from 2.6.X-rcY-mmZ kernels. Arjan noted, "finding what exact kernel version an oops is from is... surprisingly hard. And to be honest, bugs against -mm are still very interesting, since they'll be the next mainline after all".

2.6.24-rc5, No Final Before Xmas

Submitted by Jeremy
on December 12, 2007 - 7:02am
Linux news

"It's been a week, and I promised to be a good boy and try to follow my release rules, so here is the next -rc," Linus Torvalds said, announcing the 2.6.24-rc5 kernel. He noted:

"Things _have_ slowed down, although I'd obviously be lying if I said we've got all the regressions handled and under control. They are being worked on, and the list is shrinking, but at a guess, we're definitely not going to have a final 2.6.24 out before xmas unless santa puts some more elves to work on those regressions. So any elves out there - please keep working."

Linus added that there were no major changes in the latest release candidate, stating that because of this it wasn't worth posting a diffstat, "it only highlights a textually big PA-RISC revert, and the powerpc defconfig updates. And the Blackfin SPI driver. The rest is largely random noise in various subsystems (drivers/net, xfs filesystem, and arch updates are some of the areas that show more changes)."

2.6.24-rc4, "A Bit Late"

Submitted by Jeremy
on December 4, 2007 - 7:55am
Linux news

"We should have one week between -rc releases, but I was gone for a week over thanksgiving (as were some other kernel developers), so this one is a bit late. It's been almost the rule rather than the exception, but I promise I'll be better..." began Linus Torvalds, announcing the 2.6.24-rc4 kernel. He noted, "there aren't a lot of exciting changes here, but there's still a _lot_ more churn than I really hoped for at the -rc4 stage. Blackfin, MIPS and Power do stand out in the diffstats, but ARM and x86 got some updates too." Linus continued:

"And we had some ACPI churn (processor throttling etc), along with various driver updates: ATA, IDE, infiniband, SCSI, USB and network drivers.. And on the filesystem side, cifs, NFS, ocfs2 and proc. Ugh. Too much. [...] That said, none of the changes are really _exciting_ or really scary. And we should have fixed a number of regressions, although more certainly remain."

Modular IO Schedulers

Submitted by Jeremy
on November 26, 2007 - 2:23pm
Linux news

Adrian Bunk posted a patch to make Linux IO schedulers a non-modular option, which would require one IO scheduler to be selected at compile time. He suggested, "there isn't any big advantage and doesn't seem to be much usage of modular schedulers." He added that removing the option to make IO schedulers modular would save 2kB on each kernel image. Jens Axboe did not like the patch, "big nack, I use it all the time for testing. Just because you don't happen to use it is not a reason to remove it." When Adrian noted that no distros seemed to be making IO schedulers available as modules, Jens suggested that this was a mistake and quipped, "it's been a long time since I considered a distro .config a benchmark/guideline of any sort."

Adrian went on to ask for the technical reasons for continuing to support four different IO schedulers, expressing concern that it could lead to bugs in individual schedulers going unreported. Jens explained that he was aiming for the perfect IO scheduler, but at this time different IO schedulers offer better results for different workloads, "with some hard work and testing, we should be able to get rid of [the anticipatory scheduler]. It still beats cfq for some of the workloads that deadline is good at, so not quite yet." Arjan van de Ven offered, "there is at least one technical reason to need more than one: certain types of storage (both big EMC boxes as well as solid state disks) don't behave like disks and have no seek penalty; any cpu time spent on avoiding seeks is wasted on those, so for these devices one really wants to use a different IO scheduler, one which is much lighter weight". Jens then acknowledged, "there's always a risk with 'duplication', like several drivers for the same hardware. I'm not disputing that."

CFS Scheduler -v24 Backports

Submitted by Jeremy
on November 19, 2007 - 1:00pm
Linux news

Ingo Molnar announced that version 24 of his Completely Fair Scheduler patch is now available backported to the 2.6.24-rc3, 2.6.23.8, 2.6.22.13, and 2.6.21.7 kernels. He noted that there have been significant changes since the last backport, "36 files changed, 2359 insertions(+), 1082 deletions(-). That's 187 individual commits from 32 authors." Ingo noted, "99% of these changes are already upstream in Linus's git tree and they will be released as part of v2.6.24. (there are 4 pending commits that are in the small 2.6.24-rc3-v24 patch.)" He also highlighted some of the more significant improvements:

"Improved interactivity via Peter Ziljstra's 'virtual slices' feature. As load increases, the scheduler shortens the virtual timeslices that tasks get, so that applications observe the same constant latency for getting on the CPU. (This goes on until the slices reach a minimum granularity value).

"CONFIG_FAIR_USER_SCHED is now available across all backported kernels and the per user weights are configurable via /sys/kernel/uids/. Group scheduling got refined all around."

Git on Windows

Submitted by Jeremy
on November 19, 2007 - 3:15am
Linux news

"This lovely dark 4am is as good an occasion as any to offer to you the 5th issue of the msysGit Herald, the not-quite-biweekly news letter to keep you informed about msysGit, the effort to bring one of the most powerful Source Code Management systems to the poor souls stuck with Windows," began Johannes Schindelin on the git mailing list. He noted that the project was finally concentrating on getting git to work on Windows, having finally gotten the installer working. The Git on MSys project home page notes,

"Unfortunately, Git on Windows is only officially supported using Cygwin. However, there is a fork (hopefully to be merged with 'official' git real soon now) which enables you to compile git using MinGW/MSys. It is a little bit tricky to get ahold of everything needed (MSys, iconv, Tcl/Tk, gcc, make, zlib, regex, etc.), so this project tries to provide a single .zip (actually, a 7-Zip packed installer) which you can unpack, and by double-clicking on msys.bat everything is set. You can start right away to hack on your favourite Source Code Management tool."