Super-charge Your Python Builds – Test What Matters, Stop Testing What Doesn't

Developers looking to ship code need to integrate their changes into a project safely. Continuous Integration practices enable the frequent and verified (built and tested) contribution of code to shared repositories. But as projects grow, so do their test suites and the time it takes to run them. The increasing time that it takes developers to integrate their changes can lead to a drop in productivity, as they are blocked waiting for builds to complete. Small and contained changes do not require all tests to be run, since most of them simply aren't relevant.

As such, we realised this is an opportunity to improve the efficiency of testing Python projects, so we built and open-sourced Partial Testing – functionality enabling developers to run the tests that matter, and ignore those that don’t. We use coverage data gathered from previous runs to identify which tests are relevant for each new change or pull-request, and, as a consequence, developer productivity improves, release cycles shorten, and compute resource usage drops too.

How Does it Work?

1. Master branch builds run the entire test suite while recording coverage via coveragepy>=5. This ensures that coverage data remains up-to-date, and that all tests are still sometimes run;
2. Non-master branches introduce changes and their builds use the saved coverage data to determine which tests are relevant to this change;
3. Finally, we profit by running only the relevant tests. We use the excellent pytest framework, which takes the list of test files from step #2 as input.

For simplicity, we consider a file as the smallest possible changed unit (instead of doing it at the line level). That is, if a line has changed, it is treated as if the entire file has changed. This approach reduces the complexity of Partial Testing and provides a wider safety net against filtering out tests that were actually relevant.

Figure 1. Our CI Pipeline

Source: Man Group. Illustrative example – for information only.

As shown on Figure 1, once a file has been modified, a few different testing scenarios arise according to the file type. It is also possible that a combination scenario is required.

Any Python project can benefit from Partial Testing by making its builds faster – the only requirements are Coverage.py and Python itself.

Impact on Builds

Man Group builds generally have a separate stage for unit and integration tests, which is useful for various reasons:

• It helps to quickly identify if a build might have failed due to external reasons;
• It facilitates the use of specific environments, or nodes, for each stage – further improving build times.

The impact from Partial Testing can be seen in both kinds of tests, unit and integration, as shown by Figure 2 and 3 where we’ve plotted the execution time for the test stages of the last builds of a given project.

Figure 2. Master Branch versus Non-Master Branch: Unit Test Times

Source: Man Group.

Figure 3: Master Branch versus Non-Master Branch: Integration Test Times

Source: Man Group.

As each pull-request now only runs a subset of the entire test suite, we achieve a 60% throughput improvement. Faster builds gives us happier developers!

It is worth noting that because only a subset of tests are run, we cannot determine the change in test coverage that a pull-request introduces. However, the master branch builds run the entire test suite and so provide us with this information as needed.

Final Thoughts

A detailed example of how to get started with Partial Testing and set it up for your continuous-integration pipeline is available on Man Group’s github.

In the future, Partial Testing could be extended to execute the relevant tests in real-time, as the user types to modify their code, speeding up the feedback cycle. This might bring up interesting questions about how to keep the reference coverage data up-to-date as the user types, but that might not be that important for this use case.

At Man Group, we are committed to contributing back to the open-source community and have made this package available for everyone: github.com/man-group/partialtesting. We welcome contributions, do please send us suggestions or pull-requests.