Visual Studio Tests

The Visual Studio Tests build runner integrates functionality of the MSTest framework and VSTest console runner. Support for both frameworks enables TeamCity to execute tests and automatically import their test results.

The Visual Studio Test runner requires Visual Studio Test Agent or Microsoft Visual Studio installed on the build agent.

Visual Studio Tests runner settings

Option	Description
Test engine type	Select the tool used to run tests: VSTest or MSTest.
Test engine version	Select the version of the tool from the drop-down menu. By default, the available VSTest and MSTest installations are autodetected by TeamCity: MSTest versions 2005-2017 are supported VSTest versions 2012-2017 are supported You can specify a custom path to the test runner here as well. TeamCity parameters are supported. You can specify the agent property `teamcity.dotnet.vstest.VS_VERSION.install.dir` pointing to a VSTest distribution path and use it here, where `VS_VERSION` can be 11.0, 12.0, 14.0, 15.0, or 16.0. The property will have priority over registry values.
Test file names	This field is mandatory for VSTest and optional for MSTest. Specify the new-line separated list of paths to assemblies to run tests on in the included assemblies list. Exclude assemblies from the test run by specifying paths to them in the corresponding field. Wildcards are supported. Paths to the assemblies must be relative to the build checkout directory.
Run configuration file	(Optional) Specify the typical `.runsettings` file for VSTest or `.testsettings` file for MSTtest. You can use the file browser .
Additional command line parameters	Enter additional command line parameters for the selected test engine. Note that tests run with MSTest are not reported on-the-fly. The Microsoft Developer Network lists the available options for VSTest and MSTest.

The rest of settings will vary depending on the engine to run tests with:

VSTest Settings
MSTest Settings

VSTest Settings

Option	Description
Target platform	Select the platform bitness. Note that specifying x64 target platform will force the `vstest.console` process to be run in the isolated mode
Framework	If the default is specified, vstest.console will select the target framework automatically. You can also choose the .NET platform manually using the drop-down.
Test names	(optional) Of all tests discovered in the included assemblies, only the tests with the names matching the provided values will be run. For multiple values, use new line. If the field is empty, all tests will be run. See details in the Microsoft documentation. Cannot be used with the option below.
Test case filter	Run tests that match the given expression. See details in the Microsoft documentation. Cannot be used with the option above.
Run in isolation	Run the tests in an isolated process
Use real-time test reporting	If this checkbox is selected, a custom TeamCity test logger will be used for real-time reporting. See the next section for details. If not selected, VSTest will report tests to TeamCity after the run (the discovery of custom logger will not be attempted). When using this option, it is recommended to check the number of tests in the produced `trx` file against the build log. In case of incosistencies, switch to the custom logger.

Custom test logger

VSTest.Console supports custom loggers, that is libraries that can handle events that occur when tests are being executed.
TeamCity has a custom logger that provides real-time test reporting. The logger must be installed manually on the agent machine, as it requires dlls to be copied to the Extensions directory of the VSTest.Console. No agent restart is needed when the custom logger is installed.

To install the custom logger:

1. Download the custom logger.

2. Extract the contents of the downloaded archive on the agent machine to the following path:

for VisualStudio 2019 update:

PROGRAM_FILES(x86)\Microsoft Visual Studio\2019<Edition>\Common7\IDE\Extensions\TestPlatform\Extensions

for VisualStudio 2017 update 5 onwards:

PROGRAM_FILES(x86)\Microsoft Visual Studio\2017<Edition>\Common7\IDE\Extensions\TestPlatform\Extensions

for VisualStudio 2017 up to update 4:

PROGRAM_FILES(x86)\Microsoft Visual Studio\2017<Edition>\Common7\IDE\CommonExtensions\Microsoft\TestWindow\Extensions

for VisualStudio 2015:

PROGRAM_FILES\Microsoft Visual Studio 14.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\Extensions

for VisualStudio 2013:

PROGRAM_FILES\Microsoft Visual Studio 12.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\Extensions

for VisualStudio 2012:

PROGRAM_FILES\Microsoft Visual Studio 11.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\Extensions

3. Check that the custom logger was installed correctly by executing vstest.console.exe /ListLoggers in the console on the agent machine. If the logger was installed correctly, you will see the logger with FriendlyName TeamCity listed:

VSTest.TeamCityLogger.TeamCityLogger
Uri: logger://TeamCityLogger
FriendlyName: TeamCity{info}

MSTest settings

Option	Description
MSTest metadata	Enter a value for `/testmetadata:file` argument. See details in the Microsoft documentation.
Testlist from metadata to run	Edit the Testlist. Every line will be translated into `/testlist:line` argument. See details in the Microsoft documentation.
Test	Names of individual tests to run. This option will be translated into the series of `/test:` arguments. See details in the Microsoft documentation.
Unique	Run the test only if one unique match is found for any specified test in test section.
Results file	Enter a value for the `/resultsfile:file_name` command-line argument. Parameter references are supported. It is recommended to leave the field blank to generate a temporary `\.trx` file in the build temporary directory. Note that the directory may be cleaned between the builds. To save the test run results to a named non-default file, enter a value for the `/resultsfile:file_name` command-line argument. If the path specified is relative, it will be relative to the build checkout directory. If the specified file already exists in the checkout directory, the build agent will attempt to delete the file. If the agent fails to delete it, the build will fail. To create a `.trx` report in the build temporary directory, use `%system.teamcity.build.tempDir%.` If you need to generate your results file in the checkout directory, consider adding the Swabra build feature to your configuration. If the path specified is absolute, it will be used "as is".

Last modified: 19 May 2023