Parallel Tests
TeamCity is now capable of parallelizing the execution of your tests by distributing them across multiple build agents, thus minimizing the overall duration of tests. The tests of a build can be automatically split into batches, and each batch will run on a separate build agent. This feature addresses a popular use case when a build consequently runs many independent tests on the same agent while they could technically be running in parallel, utilizing resources of multiple agents. Previously, to emulate such behavior, some users would configure several build configurations and join them in a chain with parallel connections. This approach works, but sometimes requires implementing a non-trivial logic of distributing tests across batches.
In TeamCity 2022.04, the tests' distribution logic is provided by TeamCity itself. In addition, such build runners as Maven, Gradle, IntelliJ IDEA Project, and .NET are capable of automatic filtering of the tests on the agent without the need to change the settings of build steps.
The new Parallel tests build feature solves the task of parallel tests execution on different agents. To enable this feature in an existing build configuration which is already configured to run build steps with tests, go to Build Configuration Settings | Build Features, click Add build feature, and choose the Parallel tests type. You will be prompted to set the number of batches, which also means the number of parallel agents to use in a build.
tip
Note: Only tests methods from different classes (or test cases) can run in parallel on different agents.
Before TeamCity splits tests into batches to run in parallel, it needs to gather test statistics of at least one preceding build. This information helps subdivide tests in semi-equally sized batches (based on tests duration) so that the total build time is as short as possible. If you enable this feature in a freshly added build configuration, its first build will run in the normal mode; when it finishes and produces test reports, TeamCity will be able to split the second one.
TeamCity takes into account not only the latest tests run, but also the history of your tests: the next builds with parallel tests will also contribute to this statistics and allow TeamCity to make smarter decisions.
If the test statistics is available, then the following happens when a new build is triggered:
TeamCity generates copies of the current build configuration according to the number of batches specified in the build feature. These build configurations will have the same build steps as the original one. In the future, the changes to the build steps of the original configuration will be propagated to the generated ones automatically.
The triggered build will be transformed into a composite build with dependencies on the builds from the generated build configurations.
As soon as the first dependency build starts, the composite build will start too.
tip
The settings of the original build configuration are not affected by the Parallel tests build feature.
On the other hand, the number and settings of the generated build configurations are fully controlled by the Parallel tests build feature. The generated build configurations are read-only by default and are not intended to be modified manually. The generated build configurations are also placed into a subproject that is hidden. If the project has versioned settings enabled, the generated build configurations will not be committed to the VCS repository.
A build of a generated build configuration will run the same set of build steps as defined in the original build configuration. If some of these steps are of the Maven, Gradle, IntelliJ IDEA Project, or .NET type, and they were executing some tests, then these build runners will automatically run only the fraction of the tests corresponding to the current batch.
tip
If the original build configuration has deployment steps, these steps will be performed the same number of times as the number of batches.
If you run tests differently, you can still enable the Parallel tests build feature for your configuration and benefit from automatic test division: you will obtain the information about the tests to execute from the build parameters that will be provided by the build feature.
Automatic execution of a batch of tests instead of the whole set of tests is only supported if the following requirements are met:
In some cases the tests are executed in such a way that TeamCity cannot affect their execution anyhow. For instance, they can be generated on the fly, or they can be reported by a third-party build runner, or imported from a file, and so on.
In such cases, the user will need to implement custom test execution logic to execute only one batch of tests. The Parallel tests build feature simplifies the task by providing a number of build parameters. These parameters are:
Parameter name | Description |
---|---|
teamcity.build.parallelTests.currentBatch | Contains the current batch number starting with 1 |
teamcity.build.parallelTests.totalBatches | Contains the total number of configured batches |
system.teamcity.build.parallelTests.excludesFile | Contains a path on the agent to a text file with tests which should be excluded from execution |
The format of the file with excluded tests is as follows:
#version=1
#algorithm=<name of the algorithm used to split tests, optional>
#current_batch=<number of the current batch, same as teamcity.build.parallelTests.currentBatch parameter>
#total_batches=<total number of batches, same as teamcity.build.parallelTests.totalBatches parameter>
#suite=<suite name, can be empty>
<new line separated list of test classes>
Here version
represents the file format version. It is implied that the custom tests' execution logic checks the version and reports an error or fails a build if the version has an unexpected value.
In the future, new keywords starting with the #
character can be added to the file. All such unrecognized keywords should be ignored.
Note: Java and .NET test frameworks usually report tests to TeamCity in the following format:
[<suite name>: ]<fully qualified test class name>.<test method>[<test arguments>]
Where <suite name>
and <test arguments>
are optional and not always present.
For example, the following Java Test class:
package org.example.tests;
class TestCase1 {
public void testMethod1() { ... }
public void testMethod2() { ... }
}
will produce the following test names in TeamCity:
org.example.tests.TestCase1.testMethod1
org.example.tests.TestCase1.testMethod2
Then the parameter system.teamcity.build.parallelTests.excludesFile
will point to a text file with the following content:
#version=1
#current_batch=1
#total_batches=3
#suite=
org.example.tests.TestCase1
tip
Note: the granularity of tests filtering is per class (or test case), not per test method.
The build step with custom tests' execution logic should use this file and filter out all the tests that belong to the classes mentioned there. All the other tests should be executed.
The Code coverage statistics will be inaccurate for builds with parallel tests because it will be collected for a fraction of tests executed by the current batch.
After enabling parallel tests in a build configuration, it will stop publishing artifacts. Consider having a separate build configuration that publishes artifacts.
A newly added test which is not yet known to TeamCity will run in each batch during the first run.
When TeamCity divides tests into batches, it only takes into account the duration of the test itself. The duration of setUp/tearDown or any other preparation methods is not know to TeamCity, therefore the duration of batches may not be equal.
An agent selected in the custom build dialog for a build with parallel tests will be ignored because the build will be transformed into a composite one after triggering.
Parameters published by the build steps via the setParameter service message, as well as runner specific parameters, such as
maven.project.version
, won't be available in a composite build with parallel tests.When it comes to the build configurations limit in the TeamCity Professional version, the automatically generated build configurations are counted as normal build configurations.
The Enforce Clean Checkout action does not work for build configurations with parallel tests configured.
A subsequent start of a build with parallel tests won't reuse already existing builds of the generated build configurations even if there were no new VCS commits.
A failed test can be executed by a different batch on the next run, showing that the test is newly failed while in fact it was already failing in some other batch.
Builds with parallel tests triggered in a branch won't use the build steps from this branch; the build steps from the default branch will be used instead.
Thanks for your feedback!