The NetBSD Foundation Press Release: NetBSD and the Google "Summer of Code" 2007 Summary

Project: Automated Testing Framework (ATF)

Student: Julio M. Merino Vidal
Mentor: Martin Husemann

Summary and goal:  NetBSD has a set of regression tests in its source tree, but unfortunately there is no easy way to run them all in an unattended fashion. This means the tests are not being run as often as they should. Some of these tests raise errors when executed, which proves that they are useful in catching errors but that their execution must be automated to be really useful. Other projects, such as Monotone and GCC, have already demonstrated the value of having an extensive test suite that is periodically run. The goal of the ATF project was to develop a testing framework to easily define test cases and run them in a completely automated way. It was a very important design goal to allow the end user to run the complete test suite without the need for development tools, so that he could ensure that the whole system behaved correctly on a new installation. Part of this project's goals also included the conversion of some of the existing regression tests to the new framework to demonstrate its functionality and usefulness.

Results:  At the time of the deadline the first public release of the package, ATF 0.1, as well as a patch set to merge ATF into the NetBSD source tree and convert the regress/bin and regress/sys/fs/tmpfs to the new infrastructure was published. Keep in mind that the infrastructure itself, ATF, is a standalone package not tied to NetBSD -- it already works on many other different operating systems, including FreeBSD, Solaris and Linux, hoping that other operating systems and applications can take advantage of it. Development has continued after the Summer of Code deadline. The ATF 0.2 release was published just a month after 0.1 and already includes many improvements and corrections. It is expected that ATF will be ready for integration into the NetBSD source tree soon, hopefully after one or two more releases (before the end of the year). However, it can already be found in pkgsrc as devel/atf, a package aimed at those people that would like to follow its development.

Project: Porting ZFS

Student: Oliver Gould
Mentor: Dieter Baron

Summary and goal:  The primary goal of this project was to port volume emulation (ZVOL) functionality in order to mount FFS file systems. This entailed an analysis of OpenSolaris and FreeBSD ZFS implementations, which was intended to develop, additionally, a plan for completing porting ZFS to NetBSD.

Results:  Not all goals were met, though significant progress has been made such that meaningful results are expected shortly. From the outset, the disk(9)-interfacing components of ZFS have been a primary focus. At this point major bugs have not yet been eliminated. Many userland, compatibility, and internal system components behave normally. A significant amount of time was spent researching (reading sources, configuring tests, etc) ZFS components and their related interfaces. Once the immediate bugs are vetted, ZFS will be harnessed with ATF & RUMP to implement tests present in ztest(1). The idea being to use a single framework to test ZFS at the component and file system levels. The student has learned a lot during this project and intends to continue the ZFS work.

Project: A Framework For Enforcing QoS Inside the NetBSD UVM

Student: Sumantra R. Kundu
Mentor: William Stouder-Studenmund

Summary and goal:  For delay sensitive systems such as streaming multimedia servers and back-end database systems, servicing the reader processes in a timely fashion is more important than the servicing the writers. This is because applications are blocked from processing unless the data is made available to them by the underlying operating system (OS). Thus, readers need to be serviced instantaneously while it does no harm for the writers to wait for a certain amount of time. The aim of this project was to prioritize the readers over the writers so that the transactions-per-second (TPS) of latency sensitive processes can be improved when the system is either experiencing lots of page requests and/or flurry of write activities.

Results:  Sumantra made good progress up to the midterm. He produced a new priority scheduler that is designed to monitor disk performance in real-time and also seems to give better performance in and of itself. Unfortunately he became ill after the midterm review. Due to the seriousness of his illness Google has granted Sumantra an extension and his mentor has now received some new work.

Project: Running Kernel File Systems in Userspace

Student: Antti Kantee
Mentor: William Stouder-Studenmund

Summary and goal:  The initial goal of the project was to be able to compile and run the same FFS code both in the kernel and in userspace. The benefits were testing and development: file system code could be developed in userspace with userspace tools and dropped into the kernel for final adjustments.

Results:  Almost all NetBSD kernel file systems can be compiled, mounted and run in userspace. The work is already a part of the NetBSD source tree. The major future undertaking is making the userspace version run multithreaded so that kernel file system locking and spl can be tested in userspace. See http://www.NetBSD.org/docs/puffs/rump.html for more information and the current status.

Project: Improving the pkgsrc Build System

Student: Jörg Sonnenberger
Mentor: Johnny C. Lam

Summary and goal:  There are two parts to this project. The first part is pbulk, a replacement for the bulk build infrastructure from Hubert Feyrer et al. This is an integral part of pkgsrc as it is used to automatically create binary packages for users. It is also the primary regression test to find issues in the infrastructure and specific packages. The design goal of pbulk was to create a modular system focused around build jobs (similar to make):

The second part of the project is the cross-compiling support for modular Xorg

Results:  Pbulk can be found in pkgtools/pbulk in the main pkgsrc tree and is actively used for the DragonFly BSD bulk builds. The HOWTO-pbulk document in pkgsrc/doc describes the steps to use pbulk on DragonFly and NetBSD. More work to refine the scripts and to cover other areas will follow. Modular Xorg was made cross compilable in a number of small steps:

The result is that as long as a native build of modular Xorg would support the platform, it can be cross-compiled as well. The infrastructure is generic enough that more packages can be supported as long as they have a cross-compiling friendly infrastructure. Jörg wrote a description about "How to cross compile modular Xorg on NetBSD"

Project: Hardware Monitoring and HAL Port

Student: Jáchym Holeček
Mentor: Quentin Garnier

Summary and goal:  The aim of this project was to develop a kernel event notification framework to notify userland of hardware changes e.g. a new USB device being added. A userland daemon was to be developed which would observe events and dispatch actions as configured. A secondary aim was to get a preliminary port of the hardware abstraction layer working with the new framework.The kernel autoconfiguration framework was modified to keep cfdata structure (which basically describes potential devices in the system) as a simple plist instead of a small jungle of C structures. Because of this autoconfig(9) had to be rewritten and device locators were converted to the more flexible plist format. Due to the huge impact (which was not accounted for) of these kernel changes throughout the device drivers and MD device discovery code the project goals were changed to leave the HAL port for later and to only include a preliminary version of the device notification framework.

Results:  Both of the new goals were achieved along with a rewrite of drvctl(4) and a basic implementation of the deviced(8) daemon. Currently the more exotic ports are being converted over to the new framework. After this has finished the project will be synced with the latest NetBSD sources and will be committed. Once the autoconfig framework has been committed the event notification framework and HAL port will be revisited.