Leslie Hawthorn, a Program Manager in Google's Open Source team, gave a talk at BSDCAN 2008 on Google's ongoing Summer of Code project. She started by explaining what the open source team does, including enforcing license compliance, hosting over 700,000 open source projects with Google Code, academic research, funding open source development, and community outreach including the sponsorship of conferences such as BSDCan. She went on to discuss how she got started running the project after its initial launch in 2005.
Having sponsored four summer of code's now, Leslie noted that Google has had over 1,500 "graduates" and over 2,000 mentors involved, coming from over 98 countries and working with over 175 open source projects. By the end of the currently in progress 2008 Summer of Code, the project will have provided over 10 million US dollars in funding, generating over 6 million lines of code.
Randall Stewart of Cisco Systems gave a talk titled SCTP, what it is and how to use it, discussing the Stream Control Transmission Protocol (SCTP). A paper that was displayed on the overhead projecter before the talk began summarized:
"Integrated into FreeBSD 7.0 -- first standardized by the Internet Engineering Task force (IETF) in October of 2000, in RFC 2960 and later updated by RFC 4960. SCTP is a message oriented protocol providing reliable end to end communication between two peers in an IP network."
Randall explained that SCTP is an alternative protocol to TCP, UDP. To describe SCTP, he suggested you start with TCP features, including: reliable retransmission, congestion control, flow control, connection oriented, and selective acknowledgements. You then add to it more features, including: "association" 4-way handshake, framing and ordered service, multistreaming, multihoming, and reachability.
Pawel Dawidek first ported ZFS to FreeBSD from OpenSolaris in April of 2007. He continues to actively port new ZFS features from OpenSolaris, and focuses on improving overall ZFS stability. During the introduction to his talk at BSDCan, he explained that his goal was to offer an accessible view of ZFS internals. His discussion was broken into three sections, a review of the layers ZFS is built from and how they work together, a look at unique features found in ZFS and how they work internally, and a report on the current status of ZFS in FreeBSD.
The BSDCan website notes that Pawel is a FreeBSD committer, adding:
"In the FreeBSD project, he works mostly in the storage subsystems area (GEOM, file systems), security (disk encryption, opencrypto framework, IPsec, jails), but his code is also in many other parts of the system. Pawel currently lives in Warsaw, Poland, running his small company."
BSDCan 2008 officially started this morning at 9AM with an opening talk by the event's organizer, Dan Langille. However, in reality the event has already been running for two days, with the FreeBSD tutorials having started on the 14'th. After arriving in Ottawa yesterday afternoon and finding my room in a 20 story University of Ottawa residence, I wandered down to the Royal Oak Pub for early registration, meeting several dozen BSD hackers from all over the world.
This morning's opening talk was well attended, filling up first with clusters of laptop users around the power outlets along both walls. By 15 minutes after the hour, the room was completely full, and Dan started with a humorous slideshow of example letters he's been receiving ever since posting the words "letter of invitation" somewhere on the BSDCan website two year back. Coming primarily from Nigeria, the letter's authors often claim to represent large groups of developers, yet always coming from "disposable" email addresses. After some laughs, he launched into his opening keynote.
"I'd like to send a small update on my progress on the Performance Tracker project," noted Erik Cederstrand on the FreeBSD -current mailing list. He continued, "I now have a small setup of a server and a slave chugging along, currently collecting data. I'm following CURRENT and collecting results from super-smack and unixbench." The project performs regular benchmarks of the FreeBSD -current source tree using Unixbench and Super Smack, allowing you to chart the results over time. Erik highlighted an example of a visible change in performance when the generic kernel moved from the 4BSD scheduler to the ULE scheduler on October 19th, 2007.
Kris Kennaway responded favorably, then noted, "one suggestion I have is that as more metrics are added it becomes important for an 'at a glance; overview of changes so we can monitor for performance improvements and regressions among many workloads." He went on to suggest, "at some point the ability to annotate the data will become important (e.g. 'We understand the cause of this, it was r1.123 of foo.c, which was corrected in r1.124. The developer responsible has been shot.")" Erik agreed with both recommendations, and noted that he would continue to work in that direction.
A recent thread on the FreeBSD -current mailing list discussed the stability of ZFS on FreeBSD. Scott Long noted that ZFS requires proper tuning to be stable:
"I guess what makes me mad about ZFS is that it's all-or-nothing; either it works, or it crashes. It doesn't automatically recognize limits and make adjustments or sacrifices when it reaches those limits, it just crashes. Wanting multiple gigabytes of RAM for caching in order to optimize performance is great, but crashing when it doesn't get those multiple gigabytes of RAM is not so great, and it leaves a bad taste in my mouth about ZFS in general."
ZFS was committed in April of 2007 by Pawel Dawidek who notes that he is using ZFS quite successfully on all of his systems. He then cautioned, "of course all this doesn't mean ZFS works great on FreeBSD. No. It is still an experimental feature." In response to some negative comments about ZFS on FreeBSD, Pawel noted, "in my opinion people are panicing in this thread much more than ZFS:) Let try to think how we can warn people clearly about proper tunning and what proper tunning actually means. I think we should advise increasing KVA_PAGES on i386 and not only vm.kmem_size. We could also warn that running ZFS on 32bit systems is not generally recommended."
"This report covers FreeBSD related projects between July and October 2007," began the latest FreeBSD Quarterly Status Report, posted by Brad Davis. He included a summary of the recent Google Summer of Code projects noting, "lots of participants are working getting their code merged back into FreeBSD." Regarding the upcoming FreeBSD 7.0 release he noted, "the bugs in the FreeBSD HEAD branch are being shaked out and it is being prepared for the FreeBSD 7 branching. If your are curious about what's new in FreeBSD 7.0 we suggest reading Ivan Voras' excellent summary."
Among the many projects discussed in the status report was work by Marko Zec on network stack virtualization, "the network stack virtualization project aims at extending the FreeBSD kernel to maintain multiple independent instances of networking state. This allows for networking independence between jail-like environmens, each maintaining its private network interface set, IPv4 and IPv6 network and port address space, routing tables, IPSec configuration, firewalls, and more." Another project discussed was the porting of Linux KVM, "a software package that can be used to create virtual machines fully emulating x86 hardware on top of machines supporting Intel VT-x or AMD-V virtualization extensions." The report noted, "Linux KVM has been ported to FreeBSD as a loadable kernel module, using the linux-kmod-compat port (in /usr/ports/devel/) to reuse as much as possible of the original source code, plus an userspace client consisting in a modified version of qemu, that uses KVM for the execution of its guests."
Andrew Doran posted some threading benchmark results to NetBSD's tech-kern mailing list, following up to some benchmarks he'd posted earlier. The results compared NetBSD -current with FreeBSD -current, and the Linux 2.6.21 kernel. Kris Kennaway was surprised by the results, and ran his own benchmarks with minimal configuration changes, summarizing, "this measurement shows that FreeBSD is performing 70-80% better than NetBSD in this 4 CPU configuration. This is in contrast to Andrew's findings which seem to show NetBSD performing 10% better than FreeBSD on a 4 CPU system (a very old one though)." He added, "the drop-off above 8 threads on FreeBSD is due to non-scalability of mysql itself. i.e. it comes from pthread mutex contention in userland."
Kris ran additional benchmarks with PostgreSQL instead of MySQL, showing much improved scalability above 8 threads, "postgresql is much more scalable than mysql on this workload and doesn't have silly scaling bottlenecks inside the application (cf the tail of the FreeBSD curve for mysql which is where pthread mutex contention kicked in)." He continued his testing, and found that on older 4CPU P3 hardware NetBSD did outperform FreeBSD, "but only by 3-4% (in particular I am not seeing the ~10% difference that Andrew observes on his 4*p3 700MHz). Given the age of the hardware and the fact that I am not seeing it on other workloads or on modern hardware it might just be due to a small scheduling difference on this configuration."
"Congratulations to the successful students and their FreeBSD Project mentors for participating in another productive Google Summer of Code," Murray Stokely noted on the -announce FreeBSD mailing list. He offered an interesting summary of all of this year's student projects, adding:
"This program encourages students to contribute to an open source project over the summer break with generous funding from Google. We have had a total of over 50 successful students working on FreeBSD as part of this program in 2005, 2006, and 2007. These student projects included security research, improved installation tools, filesystems work, new utilities, and more. Many of the students have continued working on their FreeBSD projects even after the official close of the program. We have gained many new FreeBSD committers from previous summer of code projects already, and more are in the process."
The sendfile() facility allows a regular file to be sent out to a stream socket. The system call was first implemented in FreeBSD 3.0. This provides many performance benefits for various server appliances. Andre Opperman has implemented an improved sendfile() facility for FreeBSD that has so far shown 45% less CPU usage without TCP segmentation offload and 83% less CPU usage with TCP segmentation offload. This is a great improvement over the previous implementation.
Colin Percival continues the discussion regarding the shared-cache vulnerability inherent in multi-core processors [story], offering potential mitigation techniques in the form of fixes to the FreeBSD schedulers. Based on Percival's original discovery, information leakage between threads which share a processor core and the subsequent opportunity to monitor memory access patterns can be prevented by eliminating the co-scheduling of threads that have differing privileges. Additionally, Percival advises that a currently scheduled in-kernel thread should be capable of telling its siblings (who would likely run with the same privileges) to sleep in cases when it is handling sensitive data in a "non-oblivious" manner - IPSec being a good example of this. This would further secure sensitive data from monitoring. For these two solutions, he suggests the use of p_candebug(9) for first and an as yet unimplemented IPI (Interprocessor Interrupt) mechanism for the second:
Robert Watson of FreeBSD core has provided an up-to-date list of remaining non-MPSAFE system calls present in FreeBSD. Asynchronous I/O, extended attributes, and mount related calls appear to need some looking over. This review is a necessary step for the improvement and merging of new code for FreeBSD's POSIX.1e compliant auditing code; work that will eventually allow comprehensive real-time system event detection and monitoring features. In reference to the required work to get these straggling system calls multi-processor safe, Robert offers:
"There's probably quite a bit of low-hanging fruit in the compatibility ABI system call tables. In particular, quite a few calls can probably be marked MPSAFE on the basis that they call MPSAFE code and do little or no work. In many cases we are probably unnecessarily acquiring or recursing Giant as things stand. In other places, more work will be required, where the compat code actually implements substantial services or calls, such as additional file system system calls not present in the FreeBSD interface."
Read below for specifics and the surprisingly short list of remaing non-MPSAFE calls.
Earlier this month, Alexander Leidinger unveiled a set of FreeBSD kernel source documentation generated with the help of Doxygen. The Doxygen documentation system extracts code structure from source files producing file/directory indexes, struct reference and function descriptions, include dependency graphs, function call graphs, and other useful information. While the output is not entirely perfect, this can be a very helpful visual aid to browsing through source code. Among other notable FreeBSD subsystems, Alexander has generated html linked PDF documentation for GEOM, crypto, virtual memory, netgraph, crypto, cam and net80211. Read on for the original announcement and additional comments from Poul-Henning Kamp.
Colin Percival, a FreeBSD committer and security team member, has found a local exploit against the current implementation of Intel's Hyper-Threading Technology. "Hyper-Threading, as currently implemented on Intel Pentium Extreme Edition, Pentium 4, Mobile Pentium 4, and Xeon processors, suffers from a serious security flaw," Colin explains. "This flaw permits local information disclosure, including allowing an unprivileged user to steal an RSA private key being used on the same machine. Administrators of multi-user systems are strongly advised to take action to disable Hyper-Threading immediately."
Colin will present the details behind the attack at BSDCan 2005 at 10:00 AM EDT on May 13'th. "At the conclusion of my talk I will also be releasing a paper describing the attack and possible mitigation strategies," Colin explains. The flaw affects all operating systems, and for a secure multi-user environment essentially requires that Hyper-Threading be disabled. More information can be found on Colin's web page on the topic. The formentioned paper can be downloaded here in pdf format.
Scott Long posted the latest bi-monthly status report, covering FreeBSD development for the first three months of 2005. Scott begins:
"The first quarter of 2005 has been extremely active in both FreeBSD-CURRENT and -STABLE. With FreeBSD 5.4 in the final RC stage and an anticipated branch of FreeBSD-6 this summer we have seen a lot of performance improvements in 5 and a couple of exciting new features in 6."
FreeBSD 5.4 [forum] is expected to be released by the end of this month, focusing primarily on minor feature and performance improvements. As for 6.0 [forum], the status report explains, "in contrast to FreeBSD 5.0, the goal is to take a more incremental approach to major changes, and not wait for years to get as many features in as possible. FreeBSD 6.0 will largely be an evolutionary change from the 5.x series, with the largest changes centered around multi-threading and streamlining the filesystem and device layers."