Integration tests and code coverage in Rust

At Thistle, we treat software update mechanisms as a critical piece of security infrastructure. To be able to apply updates as dynamically as possible, our system has to be resilient and reliable.

To elevate the level of trust in our platform, we built an inclusive integration test suite. Using rust and a few other handy tools we are able to aggregate and track coverage results on these end-to-end integration tests. A similar approach was taken by Filippo Valsorda a few years ago using golang.

Testing in Rust

Rust provides a fantastic testing infrastructure. Simply by adding a #[test] tag, we can define testing functions that will be ran using the cargo test command. This toolkit is ideal for unit tests, as it makes it possible to quickly write a test for a given function.

Because of the nature of our update client, we need to rely more on integration-tests than unit tests - and while rust provides first-class support for unit tests, integration tests have to be built and executed manually.

Integration tests

All the work described in this article has been summarised in a sample repo. This repository includes a sample program being tested (mentioned here as software-under-test), as well as the testing harness that we will describe below.

We start by defining two crates: the first one is our software-under-test. This sample project is only composed of a main.rs file, executing 3 functions printing text. The second crate defines the integration test harness.

In the most simple scenario, we can run the integration test by running the two following commands. This will build the software-under-test, and run the integration test suite. This is the default recipe of the integration-test makefile.

$ cargo build -p mycode 
$ cargo test -p integration-test

The test harness here simply tests the stdout/stderr output of the software-under-test. In this case, a helper function run_ensure_outputs has been implemented to automatically run and test the program's stderr.

In this case, the software-under-test is really simple, but we use a similar framework to test the behavior of our update client. In addition to testing on expected logs, we added functionalities to support reading local files, so that we can test a posteriori the state of the filesystem, and for instance, verify that we appropriately updated a given file.

Collect coverage of integration tests

In order to collect coverage statistics for our integration test, we need to build our software-under-test with the instrument-coverage compilation flag. Setting this flag will automatically output coverage files when running our software-under-test, in the .profraw format.

$ RUSTFLAGS="-C instrument-coverage" cargo build -p mycode  
$ cargo test -p integration-test

The makefile coverage target is doing exactly this - and it also adds a CARGO_TARGET_DIR variable in order to optimize caching between normal and coverage runs, and it resets the traces folder so that every run will generate a new batch of coverage file.

Analyzing results

Now that we have run our integration test suite against the software-under-test, and collected multiple .profraw files, we can aggregate them and extract statistics.

We can use multiple third-party tools to perform these tasks, as defined in the figure above. We particularly liked using grcov, a tool from Mozilla that is used to perform coverage statistics on Firefox. This tool can be used locally to measure coverage in a clear web interface. See the makefile target coverage-report.

Online coverage tools such as coveralls can also be used. On the example repository, we used grcov to aggregate the .profraw files into a single lcov file, and we upload the output to coveralls. See the coveralls report for the test repository.

We also use the llvm-cov tool to obtain CLI useable statistics, such as the total coverage percentage.

Note: It is also possible to use a CI job to put the coverage percentage of each commit of a pull request on a comment. See the coverage-percentage-gh make target. It sets the COVERAGE_STAT environment variable, which can then be posted as a comment on the pull request using this gist.

Conclusion

This simple yet powerful testing infrastructure helps us maintain the consistency of our update mechanism. It also helps us release stable software, so that our update platform is as reliable as possible for our customers.

The second part of this blog post will explain how we run end-to-end tests on an emulated arm64 device, including bootloader - using qemu and running in complete isolation with Docker! This integration is also running on our CI to perform integrity checks on our software. More on this coming soon!