I’m currently working on the flexible LAVA deployment blueprint. This blueprint is about untying our hands (both as developers and administrators) from having to provide Debian packages for our releases. This is especially important our major new dependency, celery, is not available in Debian. The idea is to deploy both production and development releases from pypi using a mixture of release tarballs and source branches. This way we can do very frequent releases, test new features in isolation (one of the features of the deployment tool is an ability to install multiple instances of LAVA on one machine, ideally we’d get an instance for each merge request if we wish so).

The scrip has been improving and is now semi-production (see TODO for known issues) ready. It would be a shame to break production though so I wanted to do some testing before we actually use this. Since instances and global configuration are at odds I needed to make every component instance aware. This was not that hard, most of the things are using lava-server manage (or have been converted to do so) and thus get automatic configuration management for free. Other things, like the dispatcher did not and had to be improved (thanks for the quick patch mwhudson).

With all the pieces aligning I had only one thing left to do. Deploy an instance in a fresh virtual machine, connect it to a board and do some testing! This is something I have carefully avoided as the dispatcher is the most complicated (for practical reasons) components of the LAVA stack and the one that is, how shall I put it, least elegant.

One of the things you will notice is that you need to prepare a master image according to the instructions. This step is not automated and is quite error prone. Everyone ends up with a different master image. While going through the pain of doing this I learned a few interesting things. I learned how to rebuild an arm initrd on intel (copy qemu-arm-static to your rootfs, mount proc+sys and chroot inside, finally invoke mkimage with all the arguments you can read from linaro-media-create (grep for mkimage), yuck!) I learned how to boot from nfs while using initrd (you need nfs-common in your ramdisk, add “ip=dhcp boot=nfs root=/dev/nfs nfsroot=192.168.1.40:/path/to/rootfs” to kernel command line, add insecure_locks to your nfs export line), I finally learned how to edit u-boot boot script (edit the plain text file and run mkimage with different options).

I have finally come up with a rather nice solution. Let me summarize the benefits:

1. I only need a few megabytes on the SD card for LAVA, the rest can be devoted to testing so a 4GB card is more than enough to test all of our images.
2. The SD card will not wear out so badly and the critical parts are less likely to be damaged (and are faster to recover too)
3. The tests cannot harm our master root filesystem. This has happened before and it’s something we had to recover from manually.
4. I don’t get duplicate MAC addresses anymore, I can identify each board uniquely once it powers on.
5. We can experiment with remote booting and other new features
6. We can script the creation of all of the required components.
7. Each board could tell LAVA that it has been powered on and ask the system for things to do. LAVA could power down boards to save money and energy while the test queue is empty.
8. We could put intelligent piece on a board, consistently, so that we don’t have to rely on fragile serial port for communication. We could have standard APIs for boards!
9. We could stop caring if a board is locally attached or behind NAT on the other side of the planet.

The only thing that made my setup different from the one in the lab was remote power control. In the lab we can recover from a failed test by power cycling a board. As long as the boot loader was not damaged we could still regain control. Unfortunately the hardware we used for that is rather pricey and I could not afford it. About a year ago I experimented with USB hubs that could replace expensive server hardware. You see, the USB specification defines that all hubs have an ability to turn a port on and off (it also includes LED control but I have not seen many hubs that actually implement that feature). Sadly none of the hubs I have at home implement this specification correctly. Details differ but technically what they do is turn the port off (it goes away as far as we are concerned) but keeps the power on. The other downside is that most USB hubs don’t provide enough power to run a board without additional power brick. That idea faded but now it got back in somewhat other form. An old colleague from Poland built a simple ARM-based device that acts as a USB cable with … USB control port for power. The board is small (half of the things I have on mine are not needed and are simply there because we reused hardware from some other project). It has a few LEDs thaw we could use for signaling (master image vs test image, recovery, etc). The software that runs on the micro controller is open and you can get the source from launchpad. Since I did not need the mini-USB port on my Panda I plugged it to a 10W iPad USB power adapter. I connected the other end to my LAVA machine. The board is nice enough to show up as another serial port, you can echo a single ASCI digit there, the first bit of that digit controls power, the other three control LEDs on the board itself. I can turn my panda off with echo 0 > /dev/ttyACM0

So there you have it. I’ll post a separate article about the details of the master image and how it can change and improve our workflow. Now it’s time to run some tests!

LAVA is a broad project and to make it more manageable for day-to-day engineering we have separated it into a collection of smaller projects, focused on a narrow topic. Today LAVA has the following key sub-projects:

• LAVA Test is a framework and a command line tool for wrapping existing test programs and translating results into a common format.
• LAVA Dispatcher is a framework and a command line tool for controlling test execution on a particular machine, including deployment of the initial environment.
• LAVA Dashboard is a web application for storing and viewing test results. It also has a simple data mining and reporting features.
• LAVA Scheduler is another web application for managing a collection of devices and processing a queue of test jobs to perform.

There are many more smaller parts, all of which you can see on the LAVA project page but I will not cover them directly here. To explain how the four key pieces work I will take you back to the May 2010 when this project started.

Originally the Validation / QA effort started out as two small projects: launch-control and abrek. The former was responsible for the so-called “validation dashboard” – a web application that makes sense of test results while the latter was a test execution/wrapper tool that allowed us to run an existing test, gather the result and upload it to the dashboard.

Abrek would allow you to customize how a test is installed, how to invoke and finally how to translate between the text output of the test program and the dashboard data format. Simple tests take a few minutes to wrap with this framework and we quickly got a few interesting, open source tests and benchmarks integrated.

During that early period one of the most fundamental interface was formed (that is still around today) that is, the concept of dashboard bundles, files containing machine-readable test results. We investigated existing formats but after a lot of mind-storming and thought experiments we realized that none of them could express the kind of data that we were interested in. We used plain text files with hierarchical data structure encoded as JSON. The idea was IMHO quite interesting because unlike virtually all other consolidated QA or CI systems that I came across before the actual data can be freely created, edited and moved around by conventional means and is not trapped in a complex database hidden behind proprietary APIs. A bundle can be written by any program that can output text, transmitted between systems by email, thumb drive or anything in between, uploaded to a dashboard, downloaded and re-sent somewhere else for processing.

Each bundle would contain one or more “test run” that describes the outcome of running a test somewhere. In addition to storing typical “pass/fail/skip/etc” status we wanted to be able to representing benchmarks and performance measurements. As soon as you think of benchmarks you will find the need to store the hardware and software context that describe the environment in which the test was performed. To formalize the format and validate bundles for correctness we have used JSON Schema. You can read the actual schema if you are interested in that level of detail. The schema is maintained alongside helper APIs in a library called linaro-python-dashboard-bundle (I really wish we could simple drop the “python” part now). Apart from what I already mentioned bundles can have arbitrary text or binary attachments, key-value attributes, references to code branches and a few other interesting features. So far we managed to map all of the tests and results we encountered in a meaningful way but if you think that, based on your experience, something is missing do let us know – the earlier we know the better.

If you are interested in looking at some real-life bundles you can see them using the integrated bundle viewer built into the dashboard. Note, that while bundle format is well-defined, various programs generate different bundles: LAVA dispatcher uses one bundle to store results of all the tests that were ran in one go on a single system (example, remember to click on bundle viewer tab), a small script that processes measurements from various benchmarks built with specific version of the toolchain stores source code references and uses external attachments not to duplicate large amounts of data (example) while Abrek introspects software and hardware context for a particular test (example).

This Abrek/Launch Control duo was the smallest building block that allowed us to do testing and retrieve the results later. But that’s not the full story today.

Background note: a few months later Paul Larson invented the fantastic project name that we use today and we gradually transitioned from launch-control to LAVA Dashboard and from Abrek to LAVA Test.

To make testing reliable and predictable we wanted to run each test in a clean, pristine environment. This also allowed us to do simple Continuous Integration on the daily Linaro images that were being produced by various parts of the build system created by our infrastructure team. The details of how we actually do that are interesting but not essential here. The idea is to describe which image to deploy, which tests to install and run and pass this data to the LAVA Dispatcher. The dispatcher would then encapsulate all the magic of automating the deployment. In theory anyone could implement the required APIs to have dispatchers running tests on development boards, silicon simulators, QEMU, virtual machines, off-the-shelf x86 boxes, servers and laptops. All that matters is the machine readable description of the test job.

While I purposefully skip the details, this was and still is one of the most challenging part of LAVA. Doing automated deployment and recovery is hard. There are lots of practical problems to solve, unexpected issues and hardware mis-design side-effects that make it very difficult to get this right. The details warrant a separate blog post which I will surely write. You may want to look at example Android test job and example Ubuntu test job straight from the history of our public scheduler instance.

As long as you have one board or other device and are happy with using lava-dispatcher to describe and deploy your test environment, lava-test to execute the test program and lava-dashboard to store and visualize the results you would not need more components. Since we were tasked with building and maintaining the Validation Lab that holds many different development boards, from all the Linaro members, we had to have another component that would allow us to manage the pool of available devices. This component is known as the LAVA Scheduler.

The scheduler has an extremely simple API: you can submit a job that will be passed down to the dispatcher that is responsible for a particular board. Apart from describing the actual test you can also specify which board or board class to use. Simple as that.

As we were developing the scheduler we quickly noticed that, as a web application, it would have to copy or somehow share various parts of the dashboard code base to get the level of integration we wanted. We have thus decided to split the dashboard into two parts: the actual dashboard-specific code and the generic webapp plumbing that allows us to reuse UI elements, RPC services, user database and a few other things. This component is now called LAVA Server. Whenever you install any of the web-facing parts you will also get the server to host them. In retrospective this decision was very good as it allowed us to quickly add new applications to the server and simply not have to worry about deployment complexity or integration issues.

As our server-side parts offer XML-RPC APIs we wanted to allow people to use them directly from the command line for experiments or simple scripting. We have created a set of command-line tools, called lava-something-tool by convention, that expose each XML-RPC method of the corresponding server-side component.

Today we have two such tools: lava-dashboard-tool and lava-scheduler-tool. Since we liked the concept lava-server we did the same thing here and wrote lava-tool to have a consistent command line tool framework.

This covers the core parts of LAVA. Remember that you can quickly install everything, apart from the dispatcher, from our PPA. You may also be interested in the server side API documentation by looking at the API help page of our public LAVA instance.

In the next installment I will take a closer look at LAVA Test and show you how to wrap a simple test program so that LAVA can use it.

This is my first blog post for a long while and my very first post wearing my Linaro LAVA hat. Over the next several weeks I will introduce various topics related to LAVA, describe the overall architecture and design, listing all the participating components and finally drilling down to some practical examples.

TL;DR: LAVA-the-stack aims to make systematic, automatic and manual quality control more approachable for projects of all sizes.

Some people say I talk to much so I believe I may also write too much at times, I will gladly welcome all feeeback, please comment on what you find curious, interesting, lacking or inaccurate.

Currently LAVA is a brand. People use it to address topics ranging from test wrappers, data formats, the validation lab test farm in Cambridge UK, the official validation website, one of the dozen lava modules/projects on Launchpad and everything in between. This is all good as it makes the brand stronger but it sometimes makes it hard to communicate successfully and unambiguously with others.

For me LAVA is a collection of participating components, the overall idea and evolving architecture that allows us to make testing, quality control and automation a little bit better than it is currently, that is… nowhere.

We, the free and open source community, have traditionally neglected quality. The perceived quality has improved over time, partially due to experience and improving awareness and partially to indirect tool support. Try to remember back before we had reliable build systems, distributed version control systems, sensible and widely deployed bug trackers. Times were fun indeed.

Good software feels good. It can have lots of bugs but as long as it feels good people are going to be happy using it. Bugs get fixed over time but that may not improve quality in any way. Quality is elusive. The distinctive properties vary with context and recipient. End user applications, system tools, developer tools, libraries, kernel drivers, packaging, documentation, artwork and design, community, translations, compatibility, migration path, feature set, web infrastructure, …even project name. All that, together, determines how we think of a particular piece of software.

To think that one can automate “quality” is naive. Quality is pretty much impossible to automate. It can be regarded as a process, much like some people look at security. Having said that, we are at the very early stages of working towards good quality. Usually a combination of excellence in development skills, taste and intuition coupled with hitting the right target audience/feature set combination is responsible for the most successful, “high quality” free software projects out there. There are some best practices (usually oscillating around libraries and tool kits, less frequently around programming languages), several libraries, few tools and virtually no automation.

Automation is hard to do, it takes time, it has a steep learning curve and almost always requires infrastructure, deployment plans, tinkering and even writing custom tools. This is where we fail. We need the git or bzr for quality, the launchpad, bugzilla for automation, the facebook for social benchmarking and feedback system. The desire to pursue quality should to be as natural as using a modern version control system, build system, documentation system, bug control system.

Our goal at the Linaro Validation team is to build a stack of tools and components that make automated testing and benchmarking, manual testing, continuous integration, reporting, analysis and data mining as useful, easy to deploy and extend as we can.

We may never get there but at least we’re trying and doing it in the open.

In the next post I will describe what components of LAVA exist today and how they relate to one another. Stay tuned.