KernelTrap - DragonFlyBSD

Quote: No Known Bugs

Jeremy — Sat, 09 Aug 2008 17:37:17 +0000

"As of now there are no known bugs, though I'm sure that will change as more DragonFly users start using the filesystem :-)"

Comparing HAMMER And Tux3

Jeremy — Thu, 07 Aug 2008 15:25:34 +0000

"The big advantage Hammer has over Tux3 is, it is up and running and released in the Dragonfly distro," began Daniel Phillips, offering a comparison between the two filesystem. He continued, "the biggest disadvantage is, it runs on BSD, not Linux, and it so heavily implements functionality that is provided by the VFS and block layer in Linux that a port would be far from trivial. It will likely happen eventually, but probably in about the same timeframe that we can get Tux3 up and stable." This led into a lengthy and interesting technical discussion between Daniel and HAMMER author Matthew Dillon, comparing the design of the two filesystems.

Matthew reviewed the Tux3 notes and replied, "it sounds like Tux3 is using many similar ideas [as HAMMER]. I think you are on the right track. I will add one big note of caution, drawing from my experience implementing HAMMER, because I think you are going to hit a lot of the same issues. I spent 9 months designing HAMMER and 9 months implementing it. During the course of implementing it I wound up throwing away probably 80% of the original design outright." Daniel noted that he's been working on the Tux3 design for around ten years, "and working seriously on the simplifying elements for the last three years or so, either entirely on paper or in related work like ddsnap and LVM3." Matthew cautioned, "I can tell you've been thinking about Tux for a long time. If I had one worry about your proposed implementation it would be in the area of algorithmic complexity. You have to deal with the in-memory cache, the log, the B-Tree, plus secondary indexing for snapshotted elements and a ton of special cases all over the place. Your general lookup code is going to be very, very complex. My original design for HAMMER was a lot more complex (if you can believe it!) then the end result. A good chunk of what I had to do going from concept to reality was deflate a lot of that complexity." The friendly conversation offers a very detailed look at the design choices made in each of these file systems.

DragonFly BSD 2.0, HAMMER Filesystem

Jeremy — Tue, 22 Jul 2008 01:48:18 +0000

"Hurrah! 2.0 has been released!" said Matthew Dillon, announcing the eighth major release of DragonFly BSD. This release is the first to include HAMMER, a new clustering filesystem that already boasts an impressive list of features, including: "crash recovery on-mount, no fsck; fine-grained snapshots, snapshot management, snapshot-support for filesystem-wide data integrity checks; historically accessible by default; mirroring: queueless incremental mirroring, master to multi-slave; undo and rollback; reblocking; multi-volume, maximum storage capacity of 1-Exabyte." Other highlighted changes in this release include, "native fairq-queue implementation using ALTQ, for PF", and "native connection state recovery to PF, so router reboots do not drop active TCP connections."

The latest version of DragonFly BSD can be downloaded from a mirror. The download page explains:

"DragonFly CDs are 'live', which means that the CD will boot your system and let you log in as root (no password). You can use this feature to check for hardware compatibility and play with DragonFly a little before actually installing it on your hard drive."

Quote: Haphazard Mathematical Algorithms

Jeremy — Tue, 22 Jul 2008 01:29:44 +0000

"I wasn't even trying to invent a new protocol or anything, I was simply fixing the haphazard mathematical algorithms the clearly non-mathematically-oriented programmers built into those crufty clients."

HAMMER Performance and Mirroring

Jeremy — Fri, 20 Jun 2008 15:28:11 +0000

Matthew Dillon continues to make significant progress on his HAMMER clustering filesystem for DragonFly BSD. He labeled the latest release 56c, noting that it, "represents an additional significant improvement in performance, [also including] bug fixes and most of the final media changes." A significant improvement in write performance was obtained by making the filesystem block size automatically increase from 16K to 64K when a file grows to larger than 1 MB. One remaining media change is required to optimize mtime and atime storage, at which point HAMMER will go into testing and bug fixing mode. Matt noted, "HAMMER's performance is extremely good now, and its system cpu overhead has dropped to roughly the same that we get from UFS", adding, "HAMMER is now able to sustain full disk bandwidth for bulk reads and writes. HAMMER continues to have far superior random-write performance, whether the system caches are blown out or not." Discussing future plans for the filesystem, Matt noted, "I could go on and on, there's so much that can be done with this filesytem :-)" Regarding one of these plans, he offered:

"I am not going to promise it, but there is a slight chance I will be able to get mirroring working by the release. I figured out how to do it, finally. Basically the solution is to add another field to the B-Tree's internal elements... the 'most recent' transaction id, and to propogate it up all the way to the root of the tree. The mirroring code can then optimally scan the B-Tree and pick out all records that have changed relative to some transaction id, allowing it to quickly 'pick up' where it left off and construct a record-level mirror over a fully asynchronous link, without any queueing. You can't get much better then that, frankly. "

HAMMER's B+Tree Implementation

Jeremy — Wed, 18 Jun 2008 02:52:03 +0000

"HAMMER makes no modifications to the B-Tree whatsoever on the front-end. When you create, delete, rename, write, etc... when you do those operations HAMMER caches them in a virtualization layer in memory and doesn't make any modifications to its on-media data structures (or their in-memory representations) at all until the meta-data is synced to disk," DragonFly BSD creator Matthew Dillon explained, comparing HAMMER, his clustering filesystem, to a wiki summary of Reiser4's implementations. He continued:

"HAMMER uses a modified B+Tree for its on-disk representation, which is a B-Tree with only keys at internal nodes and only records at the leafs. This was done to reduce structural bloat, allow for a leaf->leaf linking optimization in the future, and for other reasons. [...] HAMMER's internal nodes have a left and right bounding element. A standard B+Tree only has a left bounding element. By adding a right bounding element HAMMER can cache pointers into its B+Tree and 'pick up' searches, insertions, and deletions relative to the cached pointers instead of having to start at the root of the tree. More importantly, it can pickup searches, insertions, and deletions at internal nodes, not just leaf nodes. So I can cache a proximity pointer and if I do a good job I never have to traverse the B+Tree above that point."