Unigroup organizes monthly meetings, each of which contains a main presentation on a wide variety of topics. In this month's event, happening on Thursday, July 17th, 2014, I will present the FreeBSD Test Suite and its backing testing framework, Kyua, both of which are supported by the Google Open Source Programs Office.
Today's blog post features the key concepts behind the FreeBSD Test Suite and what you can expect from Thursday’s meeting. If you are attending, please do not forget to RSVP by July 17th!
The FreeBSD Test Suite
FreeBSD is a Unix-like, free, general purpose operating system with a large codebase — over 5 million lines according to Ohloh. In order to easily ascertain the quality of the system and to ensure that such quality does not regress over time, the foundations of a test suite and a collection of tests were needed.
With funding from a Google Summer of Code project in 2007, I got involved in writing a test suite framework for NetBSD known as ATF, parts of which were rewritten under the Kyua project name starting in 2010. Both ATF and Kyua have always been standalone components able to work on any Unix-like operating system. Until recently, NetBSD was the major consumer of these testing tools, but in 2013 they spilled into FreeBSD to equip the system with its own test suite.
The goals of the FreeBSD Test Suite are to assist developers in modifying the system, to assist end users in validating that the system works according to documented expectations, and to assist the release engineering team in vetting new releases and to put the shiny-new Kyua framework to use in a production-quality project.
Currently, the FreeBSD Test Suite is part of both FreeBSD 11.0-CURRENT (the development branch) and of stable/10 (the stable branch that will yield 10.1-RELEASE). The test suite currently holds about 570 test cases — a pretty small number considering its scope, but decent enough given that the test suite foundations are still under active development.
Kyua: the test suite glue
Kyua is a runtime engine for test suites, mostly engineered towards testing operating systems. In general terms, a test suite defines the layout of its test programs and its test cases using a declarative language that Kyua is in charge of. Based on this definition, Kyua allows to execute the tests in a controlled environment and to generate user-friendly and machine-parseable reports of the results. Continuous integration facilities are left to better-suited third-party systems such as Jenkins.
Kyua is able to run test programs implemented in a variety of languages with support for various different testing libraries. In particular, Kyua can run ATF-based test programs (written in either C, C++ or POSIX shell), legacy test programs (those that just exit with 0 or 1 depending on the test's success), and TAP-compliant test programs. It should be possible, and is in fact planned, to support other backends like GoogleTest.
The upcoming talk
In Thursday’s meeting, I will be presenting all of the above and much, much more.
The session will start with a bit of history about my involvement with the BSDs. I’ll talk about the goals of the FreeBSD Test Suite while comparing those with its NetBSD counterpart, presenting the Kyua project, outlining the current state of the test suite, showcasing Jenkins and possibly performing some live demonstrations. Expect code samples.
If you happen to be in the New York City area on the 17th, RSVP and join us for the session!
By Julio Merino, Google Site Reliability Engineering
