Technology > Super-charge Your Python Builds - Test What Matters, Stop Testing What Doesn't

TECH ARTICLE | 4 MIN

Super-charge Your Python Builds - Test What Matters, Stop Testing What Doesn't

August 18, 2020

With PartialTesting, we can speed up build-times and reduce compute requirements – run the tests that matter, ignore those that don’t!

Developers looking to ship code need to integrate their changes into a project safely. Continuous Integration practices enable the frequent and verified (built and tested) contribution of code to shared repositories. But as projects grow, so do their test suites and the time it takes to run them. The increasing time that it takes developers to integrate their changes can lead to a drop in productivity, as they are blocked waiting for builds to complete. Small and contained changes do not require all tests to be run, since most of them simply aren't relevant.

As such, we realised this is an opportunity to improve the efficiency of testing Python projects, so we built and open-sourced Partial Testing – functionality enabling developers to run the tests that matter, and ignore those that don’t. We use coverage data gathered from previous runs to identify which tests are relevant for each new change or pull-request, and, as a consequence, developer productivity improves, release cycles shorten, and compute resource usage drops too.

How Does it Work?

  1. Master branch builds run the entire test suite while recording coverage via coveragepy>=5. This ensures that coverage data remains up-to-date, and that all tests are still sometimes run;
  2. Non-master branches introduce changes and their builds use the saved coverage data to determine which tests are relevant to this change;
  3. Finally, we profit by running only the relevant tests. We use the excellent pytest framework, which takes the list of test files from step #2 as input.

For simplicity, we consider a file as the smallest possible changed unit (instead of doing it at the line level). That is, if a line has changed, it is treated as if the entire file has changed. This approach reduces the complexity of Partial Testing and provides a wider safety net against filtering out tests that were actually relevant.

Figure 1. Our CI Pipeline

Source: Man Group. Illustrative example – for information only.

As shown on Figure 1, once a file has been modified, a few different testing scenarios arise according to the file type. It is also possible that a combination scenario is required.

Any Python project can benefit from Partial Testing by making its builds faster – the only requirements are Coverage.py and Python itself.

Impact on Builds

Man Group builds generally have a separate stage for unit and integration tests, which is useful for various reasons:

  • It helps to quickly identify if a build might have failed due to external reasons;
  • It facilitates the use of specific environments, or nodes, for each stage – further improving build times.

The impact from Partial Testing can be seen in both kinds of tests, unit and integration, as shown by Figure 2 and 3 where we’ve plotted the execution time for the test stages of the last builds of a given project.

Figure 2. Master Branch versus Non-Master Branch: Unit Test Times

Source: Man Group.

Figure 3: Master Branch versus Non-Master Branch: Integration Test Times

Source: Man Group.

As each pull-request now only runs a subset of the entire test suite, we achieve a 60% throughput improvement. Faster builds gives us happier developers!

It is worth noting that because only a subset of tests are run, we cannot determine the change in test coverage that a pull-request introduces. However, the master branch builds run the entire test suite and so provide us with this information as needed.

Final Thoughts

A detailed example of how to get started with Partial Testing and set it up for your continuous-integration pipeline is available on Man Group’s GitHub.

In the future, Partial Testing could be extended to execute the relevant tests in real-time, as the user types to modify their code, speeding up the feedback cycle. This might bring up interesting questions about how to keep the reference coverage data up-to-date as the user types, but that might not be that important for this use case.

At Man Group, we are committed to contributing back to the open-source community and have made this package available for everyone: github.com/man-group/partialtesting. We welcome contributions, do please send us suggestions or pull-requests.

For further clarification on the terms which appear here, please visit our Glossary page.

This information is communicated and/or distributed by the relevant Man entity identified below (collectively the "Company") subject to the following conditions and restriction in their respective jurisdictions.

Opinions expressed are those of the author and may not be shared by all personnel of Man Group plc (‘Man’). These opinions are subject to change without notice, are for information purposes only and do not constitute an offer or invitation to make an investment in any financial instrument or in any product to which the Company and/or its affiliates provides investment advisory or any other financial services. Any organisations, financial instrument or products described in this material are mentioned for reference purposes only which should not be considered a recommendation for their purchase or sale. Neither the Company nor the authors shall be liable to any person for any action taken on the basis of the information provided. Some statements contained in this material concerning goals, strategies, outlook or other non-historical matters may be forward-looking statements and are based on current indicators and expectations. These forward-looking statements speak only as of the date on which they are made, and the Company undertakes no obligation to update or revise any forward-looking statements. These forward-looking statements are subject to risks and uncertainties that may cause actual results to differ materially from those contained in the statements. The Company and/or its affiliates may or may not have a position in any financial instrument mentioned and may or may not be actively trading in any such securities. Unless stated otherwise all information is provided by the Company. Past performance is not indicative of future results.

Unless stated otherwise this information is communicated by the relevant entity listed below.

Australia: To the extent this material is distributed in Australia it is communicated by Man Investments Australia Limited ABN 47 002 747 480 AFSL 240581, which is regulated by the Australian Securities & Investments Commission ('ASIC'). This information has been prepared without taking into account anyone’s objectives, financial situation or needs.

Austria/Germany/Liechtenstein: To the extent this material is distributed in Austria, Germany and/or Liechtenstein it is communicated by Man (Europe) AG, which is authorised and regulated by the Liechtenstein Financial Market Authority (FMA). Man (Europe) AG is registered in the Principality of Liechtenstein no. FL-0002.420.371-2. Man (Europe) AG is an associated participant in the investor compensation scheme, which is operated by the Deposit Guarantee and Investor Compensation Foundation PCC (FL-0002.039.614-1) and corresponds with EU law. Further information is available on the Foundation's website under www.eas-liechtenstein.li.

European Economic Area: Unless indicated otherwise this material is communicated in the European Economic Area by Man Asset Management (Ireland) Limited (‘MAMIL’) which is registered in Ireland under company number 250493 and has its registered office at 70 Sir John Rogerson's Quay, Grand Canal Dock, Dublin 2, Ireland. MAMIL is authorised and regulated by the Central Bank of Ireland under number C22513.

Hong Kong SAR: To the extent this material is distributed in Hong Kong SAR, this material is communicated by Man Investments (Hong Kong) Limited and has not been reviewed by the Securities and Futures Commission in Hong Kong.

Japan: To the extent this material is distributed in Japan it is communicated by Man Group Japan Limited, Financial Instruments Business Operator, Director of Kanto Local Finance Bureau (Financial instruments firms) No. 624 for the purpose of providing information on investment strategies, investment services, etc. provided by Man Group, and is not a disclosure document based on laws and regulations. This material can only be communicated only to professional investors (i.e. specific investors or institutional investors as defined under Financial Instruments Exchange Law) who may have sufficient knowledge and experience of related risks.

Switzerland: To the extent this material is made available in Switzerland the communicating entity is:

  • For Clients (as such term is defined in the Swiss Financial Services Act): Man Investments (CH) AG, Huobstrasse 3, 8808 Pfäffikon SZ, Switzerland. Man Investment (CH) AG is regulated by the Swiss Financial Market Supervisory Authority (‘FINMA’); and
  • For Financial Service Providers (as defined in Art. 3 d. of FINSA, which are not Clients): Man Investments AG, Huobstrasse 3, 8808 Pfäffikon SZ, Switzerland, which is regulated by FINMA.

United Kingdom: Unless indicated otherwise this material is communicated in the United Kingdom by Man Solutions Limited ('MSL') which is a private limited company registered in England and Wales under number 3385362. MSL is authorised and regulated by the UK Financial Conduct Authority (the 'FCA') under number 185637 and has its registered office at Riverbank House, 2 Swan Lane, London, EC4R 3AD, United Kingdom.

United States: To the extent this material is distributed in the United States, it is communicated and distributed by Man Investments, Inc. (‘Man Investments’). Man Investments is registered as a broker-dealer with the SEC and is a member of the Financial Industry Regulatory Authority (‘FINRA’). Man Investments is also a member of the Securities Investor Protection Corporation (‘SIPC’). Man Investments is a wholly owned subsidiary of Man Group plc. The registration and memberships described above in no way imply a certain level of skill or expertise or that the SEC, FINRA or the SIPC have endorsed Man Investments. Man Investments Inc, 1345 Avenue of the Americas, 21st Floor, New York, NY 10105.

This material is proprietary information and may not be reproduced or otherwise disseminated in whole or in part without prior written consent. Any data services and information available from public sources used in the creation of this material are believed to be reliable. However accuracy is not warranted or guaranteed. © Man 2025