Super-charge Your Python Builds – Test What Matters, Stop Testing What Doesn't

Developers looking to ship code need to integrate their changes into a project safely. Continuous Integration practices enable the frequent and verified (built and tested) contribution of code to shared repositories. But as projects grow, so do their test suites and the time it takes to run them. The increasing time that it takes developers to integrate their changes can lead to a drop in productivity, as they are blocked waiting for builds to complete. Small and contained changes do not require all tests to be run, since most of them simply aren't relevant.

As such, we realised this is an opportunity to improve the efficiency of testing Python projects, so we built and open-sourced Partial Testing – functionality enabling developers to run the tests that matter, and ignore those that don’t. We use coverage data gathered from previous runs to identify which tests are relevant for each new change or pull-request, and, as a consequence, developer productivity improves, release cycles shorten, and compute resource usage drops too.

How Does it Work?

  1. Master branch builds run the entire test suite while recording coverage via coveragepy>=5. This ensures that coverage data remains up-to-date, and that all tests are still sometimes run;
  2. Non-master branches introduce changes and their builds use the saved coverage data to determine which tests are relevant to this change;
  3. Finally, we profit by running only the relevant tests. We use the excellent pytest framework, which takes the list of test files from step #2 as input.

For simplicity, we consider a file as the smallest possible changed unit (instead of doing it at the line level). That is, if a line has changed, it is treated as if the entire file has changed. This approach reduces the complexity of Partial Testing and provides a wider safety net against filtering out tests that were actually relevant.

Figure 1. Our CI Pipeline

Source: Man Group. Illustrative example – for information only.

As shown on Figure 1, once a file has been modified, a few different testing scenarios arise according to the file type. It is also possible that a combination scenario is required.

Any Python project can benefit from Partial Testing by making its builds faster – the only requirements are Coverage.py and Python itself.

Impact on Builds

Man Group builds generally have a separate stage for unit and integration tests, which is useful for various reasons:

  • It helps to quickly identify if a build might have failed due to external reasons;
  • It facilitates the use of specific environments, or nodes, for each stage – further improving build times.

The impact from Partial Testing can be seen in both kinds of tests, unit and integration, as shown by Figure 2 and 3 where we’ve plotted the execution time for the test stages of the last builds of a given project.

Figure 2. Master Branch versus Non-Master Branch: Unit Test Times

Source: Man Group.

Figure 3: Master Branch versus Non-Master Branch: Integration Test Times

Source: Man Group.

As each pull-request now only runs a subset of the entire test suite, we achieve a 60% throughput improvement. Faster builds gives us happier developers!

It is worth noting that because only a subset of tests are run, we cannot determine the change in test coverage that a pull-request introduces. However, the master branch builds run the entire test suite and so provide us with this information as needed.

Final Thoughts

A detailed example of how to get started with Partial Testing and set it up for your continuous-integration pipeline is available on Man Group’s github.

In the future, Partial Testing could be extended to execute the relevant tests in real-time, as the user types to modify their code, speeding up the feedback cycle. This might bring up interesting questions about how to keep the reference coverage data up-to-date as the user types, but that might not be that important for this use case.

At Man Group, we are committed to contributing back to the open-source community and have made this package available for everyone: github.com/man-group/partialtesting. We welcome contributions, do please send us suggestions or pull-requests.

Important information

Opinions expressed are those of the author and may not be shared by all personnel of Man Group plc (‘Man’). These opinions are subject to change without notice, are for information purposes only and do not constitute an offer or invitation to make an investment in any financial instrument or in any product to which the Company and/or its affiliates provides investment advisory or any other financial services. Any organisations, financial instrument or products described in this material are mentioned for reference purposes only which should not be considered a recommendation for their purchase or sale. Neither the Company nor the authors shall be liable to any person for any action taken on the basis of the information provided. Some statements contained in this material concerning goals, strategies, outlook or other non-historical matters may be forward-looking statements and are based on current indicators and expectations. These forward-looking statements speak only as of the date on which they are made, and the Company undertakes no obligation to update or revise any forward-looking statements. These forward-looking statements are subject to risks and uncertainties that may cause actual results to differ materially from those contained in the statements. The Company and/or its affiliates may or may not have a position in any financial instrument mentioned and may or may not be actively trading in any such securities. This material is proprietary information of the Company and its affiliates and may not be reproduced or otherwise disseminated in whole or in part without prior written consent from the Company. The Company believes the content to be accurate. However accuracy is not warranted or guaranteed. The Company does not assume any liability in the case of incorrectly reported or incomplete information. Unless stated otherwise all information is provided by the Company. Past performance is not indicative of future results.

20/0760/D/GL/R/W

Please update your browser

Unfortunately we no longer support Internet Explorer 8, 7 and older for security reasons.

Please update your browser to a later version and try to access our site again.

Many thanks.