CI/CD Test Automation

Test automation is a central part of any CI/CD pipeline and an essential step toward more frequent releases. TeamCity provides a wide range of features that help you get the most from your automated tests, including test reordering, test parallelization, smart failure analysis, quality gates, and many others. With built-in support for popular testing frameworks for .NET, Java, Python, JavaScript, and other programming languages, TeamCity paves the way to better code and a more reliable release process for your entire software stack.

Tailored to your workflow

Software testing workflows are as diverse as software itself. Orchestrating Docker containers, configuring test environments, changing the scope of testing based on various parameters, and setting up quality gates are just a few common activities that you may want to do in your CI/CD tool. The good news is that TeamCity supports all of these scenarios, allowing you to easily set up automated pipelines tailored to your specific testing workflow.

Integrated with all popular build and test tools

TeamCity integrates with build tools and test frameworks like no other CI/CD solution. It parallelizes the execution of your tests, identifies flaky tests, and keeps the history of every single test it runs, including the build log, duration, order of execution, code coverage, and static analysis results, as well as other data. As TeamCity collects more test statistics, it becomes indispensable for understanding the kinds of changes that typically break your tests.

Suitable for projects of any size and complexity

If you’ve ever moved from one CI/CD tool to another, you know the process can take months and months of work. With TeamCity, you’ll never have to worry about your company outgrowing your build system and having to switch to another one. Our solution is built to handle the largest and most complex projects. For example, inside JetBrains, one single installation of TeamCity is used by over 1,000 developers who run over 5,000 builds in parallel.

Boost your productivity with advanced test reporting

By automating your tests and running them regularly, you and your team benefit from prompt feedback on your changes. As a result, any new bugs can be caught sooner and fixed faster, while the change is still fresh in your mind.

With TeamCity, you can configure automated testing as part of your CI/CD pipelines. Test results are provided on the fly to ensure fast feedback. When tests fail, TeamCity provides details of the associated code changes, together with stack traces, logs, and build artifacts, to help you kick-start the debugging process. Flaky tests and newly failed tests are highlighted, and if some tests are fixed in later builds, the test results page is automatically updated with the corresponding information.

In addition to being viewable in the web UI, test results can be sent to you via email, Slack, or other messengers, or be delivered directly to your IDE. This, again, gives you and your team immediate feedback on your latest changes without having to wait for the build step to finish.

Core test automation features

Failure analysis and code coverage

TeamCity comes with a variety of build runners that integrate with tools used to build, test, and deploy software. When a test suite is run with the help of a build runner, TeamCity is able to understand events happening during the build, check code coverage, provide on-the-fly reports, collect statistics, and analyze trends.

Sometimes it may feel impractical to convert a testing script into TeamCity build steps, and you may want to use the generic Command Line build runner instead. If you decide to go this route, you can add TeamCity service messages to your script’s output to inform TeamCity of the progress, and still get all of its smart features. For example, providing information about test names and their duration will allow TeamCity to calculate how tests can be parallelized. For more details, see the Service Messages section of our documentation.

Depending on your specific requirements, TeamCity lets you define when a build should be marked as successful or failed. For example, if it is important that your unit test coverage not decrease from one build to the next, you can make code coverage metrics a build step failure condition.

For managing test failures, TeamCity allows you to assign investigations to your team members or mute tests that you don’t need to fix immediately. Muted tests are ignored on subsequent pipeline runs, allowing your automated test process to continue uninterrupted without manual changes to the pipeline.

We develop build runners for all popular tools and programming languages, including:

  • Gradle
  • Kotlin
  • Maven
  • .NET
  • Node.js
  • Python
  • And many others

In addition to the JetBrains-supported build runners, there are over 100 free plugins and a comprehensive API that lets you integrate TeamCity with your own tools and scripts.

Parallel tests

Intelligent test parallelization is a unique feature of TeamCity that lets you greatly speed up your testing process. When multiple build agents are available, TeamCity can automatically divide tests into equal batches based on test duration and execute them in parallel. If your tests use custom scripts instead of TeamCity’s build runners, you can use service messages to inform TeamCity of their progress so that it can calculate the optimal way to parallelize tests and use it to generate special instruction files to help you set up the parallelization.

For more details, see the Parallel Tests section of our documentation.

Performance monitoring

It is often not easy to understand whether the hardware or the cloud instances you run your tests on are optimal for your projects. TeamCity’s Performance Monitor automatically collects statistics on CPU, memory, and hard drive performance and helps you understand whether there are any bottlenecks that hinder your pipelines.

For more details, see the Performance Monitor section of our documentation.

Testing code without committing it to a VCS

TeamCity makes it possible to test your code without committing it to a version control system. This is particularly useful if your development takes place in one main branch, as it minimizes the chance of breaking the build and blocking the work of other team members.

The easiest way to test your changes without committing them is to use the Remote Run feature of the TeamCity plugin for IntelliJ-based IDEs or Visual Studio. Teams that host their code in Git have an additional option of creating a Git patch, uploading it to TeamCity, and then running a personal build using the TeamCity UI.

Flexible configuration of automated test pipelines

TeamCity provides a variety of build triggers to initiate tests automatically, and it allows running tests manually from the TeamCity UI or your IDE. Depending on what triggered the build and what happened earlier in the pipeline, you can add conditional logic to your testing pipeline to streamline your process and control when the tests are run. For example, you can schedule nightly builds to run a full suite of tests, run only unit tests on every commit, or specify the exact scope of testing when running a pipeline manually.

More TeamCity features to complement your testing workflow

Automatic CI/CD infrastructure scaling

Having a large CI/CD system with many projects can make it challenging to accurately determine the number of build agents required to run builds and tests. TeamCity can help you implement a scalable CI/CD setup by automatically launching new build agents in the cloud, dynamically increasing the capacity of your pipelines. Supported cloud platforms include Amazon EC2, Microsoft Azure, Google Cloud, VMWare vSphere, Kubernetes, and others.

Docker integration

TeamCity supports and can accelerate all software development workflows that utilize Docker. For instance, it can run tests in a specific environment version, ensuring consistent and reproducible results, or conduct integration tests that require orchestrating multiple Docker containers. Regardless of how you use Docker in your CI/CD pipelines, TeamCity can track all events that occur during the build and provide real-time reports, test analysis, and other insights.

Automatic merging of changes

TeamCity has the ability to automatically merge changes into a designated destination branch if the build is successful. It is also possible to set up a series of merge operations in cascade. Combined with the ability to test code before committing it to a VCS, this allows you to utilize the gated commit pattern.

FAQ: Automated testing with TeamCity

Is TeamCity’s test parallelization a free feature?

Yes, test parallelization is available for free in all TeamCity editions, including the free self-hosted edition. In TeamCity On-Premises, you can split tests across as many build agents as are allowed by your license.

What operating systems are supported by TeamCity?

TeamCity is available for Windows, Linux, and macOS, and it can run as a Docker container on any platform that supports a Docker runtime. You can also use it as a managed service, TeamCity Cloud, which does not require you to install and maintain anything yourself.

Can TeamCity automatically retry failed tests?

Yes, with the help of the Retry build trigger, TeamCity can automatically add the build to the build queue again after it fails. This helps you deal with situations where the code is correct but tests fail because of network problems or other external factors.

Does TeamCity support UI testing frameworks?

TeamCity is a versatile CI/CD tool that can be used with a wide range of tools and test frameworks, including those used for UI testing. To learn more about using TeamCity for UI testing, check out our TeamCity UI: How Do We Test It? blog post.