Tables are cluttered with laptops, servers, switches, cables and cords as the 2006 OpenBSD hackathon continues in Calgary, Canada. Small groups of developers talk and debate around LCD screens, while others work individually on their own projects. Behind the scenes, a donated 10 megabit wireless connection provides Internet access to all. IP addresses and DNS are provided by stock bind and dhcpd processes running on an OpenBSD server. Among other things, the infrastructure area hosts an HP DL385 with 24 GB of memory that was recently donated by HP, a G5, several Sun Blade 2000's, and an assortment of PowerPC, Alpha and Opteron-based servers. A console server provides serial connections to the servers along with logs of what went on on the serial console, useful for debugging. Power issues on the first day were resolved by evenly spreading the servers and many laptops across the available circuits in the hackathon room. Chris Kuethe explained, "the whole point of the infrastructure is that it's not supposed to be exciting, it's just supposed to be there, like a light switch."
I have spoken with another 28 OpenBSD developers from Turkey, Iceland, Ireland, Germany, Sweden, Switzerland, Denmark, Australia, Austria, Hungary, the US, and Canada. Efforts are being made on ACPI, the VFS subsystem, link-layer authentication, OpenBGPD, tcpdump, XFree86, pf, CARP, dvmrpd as a replacement for mrouted, OpenRCS, OpenCVS, the USB layer, prebinding, ipsecctl, 10 gig Ethernet support, link layer path mtu discovery, several new and improved drivers, amd64 large memory support, new CD and DVD recording features for cdio, improvements to mg, support for new architectures, numerous new and updated ports, and much more.
The 2006 OpenBSD Hackathon, c2k6, is well underway in a conference room at a hotel in downtown Calgary, Canada. The event started yesterday, May 27th, attended by nearly 50 OpenBSD developers from all over the globe. OpenBSD creator Theo de Raadt [interview] is thrilled by what is already proving to be another successful event, "I don't think anybody else does this, developers suspend their lives for a week to focus entirely on just development." Theo explains that he doesn't get much coding done himself at these hackathons, but instead focuses on ensuring beneficial communication between developers, an obvious advantage to assembling so much talent in a single room.
Walking among the cluttered tables, I've been talking with the high energy attendees of this year's hackathon, learning who's here and what they're working on. In this first installment I've talked to 18 developers from France, Switzerland, Germany, the UK, the Netherlands, Australia, Brazil, Dominica, the US, and Canada. They each talk a little about how they discovered OpenBSD and what they're working on here at the hackathon, including introducing new ports, support for SD devices, local OpenCVS functionality, improvements to OpenNTPD, improved SCSI controller support, initial support for the UltraSparc III architecture, and much more. The hackathon continues around the clock through June 2nd.
One new attendee of this year's OpenBSD hackathon was Fernando Gont, a diverse individual from Argentina whose current job titles include teacher, technical writer, system administrator and network researcher. His presence at the hackathon was the result of an internet-draft he wrote about some flaws in the ICMP protocol, flaws he discovered while writing the "Security Considerations" of a different internet-draft titled "TCP's reaction to soft errors" for the IPv6 Operations working group. In researching that earlier draft, he considered various attacks against TCP using ICMP error messages, and proposed some extra validation that could be done as prevention. Following up, Fernando reviewed the IETF specifications for ICMP and TCP and was surprised to discover that they didn't propose similar validation checks, ultimately deciding to write his latest internet-draft highlighting the security impact.
Fernando was interested in discussing the ideas with his peers, but was concerned about vendors trying to patent his suggested fixes. He'd read some comments by OpenBSD creator Theo de Raadt [interview] which led him to believe that he could safely talk with Theo about his ICMP discoveries. Theo was impressed by the ideas, and as Fernando was already heading to BSDCan, Theo helped arrange for him to stay in Canada longer to attend CanSecWest and the OpenBSD hackathon. At the hackathon, Fernando worked around the clock to implement some of his suggested fixes into the OpenBSD networking stack, during which time I spoke with him.
The ICMP flaw is in the design of the protocol, not in any specific implementation. Theo explains, "here we have a 20 year old protocol, a part of the Internet infrastructure that hasn't been touched in 10 years and we were all sure was right, and now is cast in doubt." He went on to add, "these things have to be done carefully. We can't ignore the problem, which is what the IETF and the other vendors are telling us to do."
People have started trickling into the hackathon rooms as the morning wears on. The music is louder than yesterday, and discussions continue around the various tables. A CTV television crew arrives, circling the room taking footage that will be distributed throughout Canada. One day earlier, a photographer arrived getting photos for an upcoming four page spread in Forbes magazine. Discussions about tomorrow's tear down start, reflecting on how much effort and time is involved in packing everything up. But the primary focus remains on the many projects currently in progress that people still hope to get finished. At least, finished enough.
One of the projects that has multiple people involved is PF, OpenBSD's packet filter. The packet filter's original author, Daniel Hartmeier [interview], talked about his ongoing efforts and reflected on the evolution PF has seen in the past few years. Mike Frantzen talked about his work on improving the PF optimizer. Henning Brauer described his work to allow PF to filter on interface groups. And Ryan McBride [interview] spoke about his efforts to turn pfctl into more of a compiler offering a number of useful benefits.
I arrived in Calgary this afternoon, and headed straight downtown to the hotel in which the OpenBSD hackathon is taking place. Walking through the fancy hotel's front door, the concierge steered me upstairs to the hackathon rooms. A few minutes later, I entered into a dimly lit room crowded with tables, people, wires, switches, laptops and servers. Each table is covered with a white cloth, with ethernet cables dropping down from the ceiling, suspended by brown tape.
The hackathon is taking place in two rooms, attended by around 60 OpenBSD developers. Obtaining an accurate count of the attendees is difficult, as people have arrived from all over the world and are still working in different timezones, populating the room around the clock. I was warmly greeted, and given free access to wander around asking questions and taking notes. At first glance, it seemed there was very little order to the event, but speaking with OpenBSD creator Theo de Raadt [interview] he was able to point from table to table explaining what each person or group of people were doing.
Theo explained that he's personally not working on any specific hacking himself, but instead is involved in everything. "I'm involved with lots of ideas," he explained, "shooting them down, changing them, approving them, pointing them to someone they should talk to..." He described his primary role as, "accelerating communication." Indeed, the one week hackathon is designed for just that, accelerating communication between OpenBSD developers who have gathered from all over the world. Ideas are flowing, and code is being written.
It's getting late in the hackathon. I can tell, because I have to think hard to remember what day it is. (Thursday, or at least, that's what somebody in the lobby told me). Except for occasional breaks for food and sleep, I have hardly looked-up from my laptop for 3 days now. I've been chasing a bug through the multi-processor initialization code (certain features, including SSE instructions, were not initialzed properly on secondary processors), and having finally fixed it, I finally had the chance to get up and take a look around.
And wow. A lot has happened in 3 days.
One good sign that you're at a hackathon is that you've just powered-on a machine by shorting the appropriate power switch pins with a house key. Some motherboards, especially prototype ones, don't fit nicely into a case. So they live on cardboard boxes, and get powered-on with house keys.
At a hackathon, you make do with what you have.
One of the virtues of having all the developers together in one place is that some bugs get fixed that may otherwise have languished for a while. There a bug, for instance, in the Zaurus APM resume. It showed up most recently during a late-night excursion to a nearby pub. The pub in question happened to have free wireless, so a few Zauri came along for the trip. A conversation between 3 Zaurus-wielders ensued:
It's early in the hackathon, and many of the week's big projects haven't started yet. In fact, many of the big projects haven't even been conceived of yet. Typically, Day 1 is about getting set up, getting connected, and getting hacking on something, and though much of this hacking occurs on laptop, sometimes, you need access to the Big Iron.
There's two hack rooms this year. The second room is the "Big Iron" room. You can tell by the temperature—several degrees warmer than any other room in the Hotel. The source of this temperature difference lies underneath a pair of banquet tables lining one wall of the room. There you will find an eclectic stack of hardware, including a 1U dual AMD64, a CATS (ARM), a Via C3 (i386 with integrated crypto), a few SunBlades (sparc64), and some large i386es. In fact, just about any piece of hardware that isn't available in laptop form makes its way to the Big Iron room, stashed under one of these tables.
It's the day before the hackathon—infrastructure day. For Hackers, infrastructure means power and Internet. The hackathon infrastructure has evolved over the years. Initially, it was Theo's living room, and a laptop acting as a wireless gateway. Needless to say, this approach didn't scale. Today, the hackathon's infrastructure needs are somewhat more significant, hence the 0-day set up ritual.
The ninth OpenBSD Hackathon will be happening in Canada next week, in Calgary, Alberta. The week long event will begin on May 21'st and run through May 28'th, attended by around 60 OpenBSD developers from all over the world. I will also be attending for the first few days, providing live coverage on KernelTrap.org, speaking to developers and observing some of the magic as it happens from May 22'nd through May 24'th.
To get a better feel for the upcoming event, I spoke to several OpenBSD developers. Henning Brauer, an OpenBSD contributer from Germany described the event as, "the highlight of the year for most of our developers." Nikolay Sturm, also from Germany, added, "hackathons are focused on two aspects, they are an important technical event and they are *the* OpenBSD social event." Peter Valchev, who lives in Calgary and is very involved in making the hackathon happen, talked about the value of face-to-face communication, "normally, we have to sit down and write a long explanation email in order to communicate, and people are in different timezones, so the feedback is often less than fast. Being able to go directly up to somebody and perhaps even work together on a task in real-time, is a big plus." Thierry Deval, an OpenBSD developer from Belgium added, "the reduction in distance and time augments the dialog between developers working in related areas, and some new projects can even spontaneously emerge on their own."
The OpenBSD project has long been associated with security. Indeed, thanks to proactively and regularly auditing its code, the project's web site is able to boast "only one remote hole in the default install, in more than 8 years," and another page states "our aspiration is to be NUMBER ONE in the industry for security (if we are not already there)." However, security is not the only focus of OpenBSD, as reflected in the project's slogan which reads, "Free, Functional and Secure." All three of these words are strongly backed by OpenBSD developers.
If you speak with OpenBSD creator Theo de Raadt for any length of time, you will quickly realize just how important freedom is to the project. For example, freedom was the driving force behind the now ubiquitous OpenSSH, developed within the OpenBSD project. It has also lead to the development of OpenNTPD, OpenCVS, and the widely used pf Packet filter [story]. In recognition of these many contributions, Theo recently received the 2004 Free Software Award from the Free Software Foundation. The freedom that the OpenBSD team works so hard for comes without any strings, patents, or conditions, distributed under the BSD license.
Currently, the OpenBSD project is focusing on wireless networking technology, working to convince hardware manufacturers to make the firmware for their wireless cards freely distributable. It sounds simple enough, but the effort has taken much persistence and perseverance. Many of today's corporations require the signing of non-disclosure agreements and other legal red tape prior to making firmware or documentation available, requirements that don't measure up to OpenBSD's standards for freedom.
A recent discussion on the OpenBSD -misc mailing list focused on the project's efforts to initiate communication with Texas Instruments to try and get them to offer the firmware for one of their wireless chipsets under an open license [story]. The goal is not to get the company to open source their firmware, just to license it in such a way that it can be legally distributed with OpenBSD, an operating system that prides itself on being 100% free. TI is only one of many companies currently being approached in this recent effort [story].
In order to gain the attention of these vendors, many hundreds of letters have been sent and phone calls made to numerous contacts. The idea being to let the vendors know that there is a large concerned user base that is going to decide how they will spend their money based on the vendor's willingness to work with open source software. Already this effort has successfully initiated communication between OpenBSD creator Theo de Raadt [interview] and several wireless chip vendors, with two of them recently agreeing to offer their firmwares under a freely redistributable license. At this time, Symbol, Zydas, and Atmel have opened their firmwares so that OpenBSD can redistribute them. Intel and Conexant are discussing the possibilty. And only Texas Instruments has remained silent.
In order to better understand why OpenBSD has decided this is important, I approached Theo de Raadt with a few questions. In reply he fully explains the issue, talking about how successful this form of activism has been for OpenBSD in the past, and offering specifics on exactly what they are trying to accomplish. He summarizes, "the open source community has support for all the ethernet chipsets, all the scsi chipsets, all the raid chipsets, so why should we not have support for all the wireless chipsets?" Read on for the full interview.
A vulnerability in TCP, the transmission control protocol, recently received some exposure in the media. Paul Watson released a white paper titled Slipping In The window: TCP Reset Attacks at the 2004 CanSecWest conference, providing a much better understanding of the real-world risks of TCP reset attacks.
To better understand the reality of this threat, KernelTrap spoke with Theo de Raadt [interview], the creator of OpenBSD, an operating system which among other goals proactively focuses on security. In this article, we aim to provide some background into the workings of TCP, and then to build upon this foundation to understand how resets attacks work.
This is the first article in a two part series. The second article will look into how TCP stacks can be hardened to defend against such attacks. Toward this goal, we spoke with members of the OpenBSD team to learn what they have done so far, and what further plans they have to minimize the impact of reset attacks.
The upcoming release of OpenBSD 3.3 on May 1'st will include, among many other improvements, a notably enhanced version of PF, OpenBSD's stateful packet filter. Some of the more significant enhancements to PF include: 'queues', allowing for per-rule bandwidth control [story]; 'pool options', allowing one to utilize multiple uplinks and to intelligently redirect traffic to multiple servers; 'anchors', which allow one to divide packet filtering rule lists into logical pieces; 'tables', efficiently allowing for very large lists; and other parser improvements that make an already friendly syntax more human readable.
PF replaced its predecessor, IPF, with the release of OpenBSD 3.0 in December of 2001. Since that time, this impressive and relatively new packet filter has grown a faithful following (myself included), and continues to evolve rapidly with each new OpenBSD release. Perhaps the greatest compliment, developers have begun to port PF to other operating systems. Back in January, Joel Wilsson announced his effort to port PF to NetBSD. And more recently, Pyun YongHyeon announced his port for FreeBSD.
I approached Pyun to learn more about his recent porting efforts. In the following article he explains why he began working on this port, and what FreeBSD users can expect from the project. Additionally, I spoke with PF creator Daniel Hartmeier [interview], PF developer Henning Brauer, and OpenBSD creator Theo de Raadt [interview]. They all reflect on these recent porting efforts, as well as the exciting new features found in OpenBSD's PF.