What's New in TeamCity 2019.2
New flexible clean-up rules
Automatic build clean-up has been present in TeamCity since the early versions. It allows conveniently deleting old and no longer necessary build data. While the provided customization options are quite easy to configure, they cover only the most common cases and do not allow for fine-tuning.
With this release, we are introducing more options for flexible control of the clean-up process. In addition to existing "base rule" for a project or build configuration, you can now create multiple "keep rules" to specify what builds and data to preserve during the clean-up. The keep rules are more fine-grained and can cover cases like keeping all the builds with a certain tag (for example, release) or in a certain branch. While using the new keep rules requires a better understanding of different kinds of builds and their data, it also offers greater flexibility.
The Clean-up Rules section of Project Settings allows managing base and keep rules for the current project and for its subprojects and build configurations.
When adding a keep rule, you can specify:
Build data to preserve: history, artifacts, logs, statistics, or everything.
Builds range: what time interval or what number of last builds will be affected by the rule.
Optionally, filter the builds by their status, tags, and branches. Choose if to limit the builds to personal or non-personal only.
You can also choose if the rule is applied per each matching branch individually or to all the builds in the selected branches as a single set.
Example of a keep rule:
Read more in Clean-up.
Metrics reporting
TeamCity now provides Prometheus-compatible server metrics displayed on the Diagnostics | Metrics tab and available via the app/metrics API endpoint.
To access the metrics, make sure your TeamCity user account has the "View usage statistics" permission assigned to it.
The metrics include:
CPU, memory, and system load
time spent on processing a build queue
number of active user sessions,
and more
The full list is available via https://<teamcity-server-host>/app/metrics.
Some of the metrics are experimental, but you can already access them by adding the ?experimental=true query string to the endpoint URL or enabling "Show experimental metrics" option on the Metrics tab.
To collect the metrics, we suggest that you use a Prometheus database or a combination of Telegraf& InfluxDB. For a visual representation of the metrics, you can use Grafana or any other similar solution.
Example of TeamCity metrics represented in Grafana:
Code highlighting in build runner scripts
We’ve added automatic code highlighting and line numbering for scripts of the following build runners: Command Line, Ant, PowerShell, Docker, NAnt, and Rake, as well as for the configuration of Amazon EC2 images.
To improve readability, you can now apply soft wraps to the code by clicking 
next to the input form.
Example of Dockerfile highlighting in TeamCity:
Support of EC2 launch templates
TeamCity now supports Amazon EC2 launch templates for cloud instances. Many of the EC2 users appreciate the launch templates since they allow reusing a once defined launch specification for all new instances, which eliminates the need to describe the launch settings every time new instances are requested.
If your cloud profile is connected to the Amazon server, TeamCity will automatically detect launch templates available on this server. Simply select a template as an image Source and specify its version, and TeamCity will request instances based on the template parameters.
Read more in Setting Up TeamCity for Amazon EC2.
Restoring backup on TeamCity startup
If you start the TeamCity server instance for the first time and want to restore the backed up data of your previous TeamCity installation, you can now restore the backup right from the startup screen UI.
Read more in Restoring TeamCity Data from Backup.
Running personal builds on unified diffs
Now you can run a personal build containing your local changes based on a diff patch uploaded via the TeamCity UI or via REST API.
Previously, it was possible to run personal builds, based on your local changes, only by integrating your TeamCity server with some supported IDE. Now you can save your patch as a .diff file in a unified format (for example, using IntelliJ IDEA or the git diff command) and upload it directly to the server.
Read more about this feature in Personal Build.
Note that this is an experimental feature. TeamCity provides a stable parsing of unidiff files generated by Git and IntelliJ IDEA. Binary changes in non-binary files are not supported.
Extended REST API for cloud profiles
TeamCity REST API now provides the .../app/rest/cloud/profiles, .../app/rest/cloud/images, and .../app/rest/cloud/instances endpoints that expose the same cloud integration details as those provided in the TeamCity UI.
Predefined build parameters for pull requests
We have added a few predefined build parameters to expose valuable information on pull requests, so it can be used in the settings of a build configuration or in build scripts:
Support of cross-platform dotCover
TeamCity now allows collecting code coverage for .NET Core projects on Linux and macOS by supporting cross-platform JetBrains dotCover, version 2019.2.3+.
dotCover 2019.2.3 for Windows is bundled with TeamCity. If you need to collect code coverage under non-Windows platforms, add the cross-platform dotCover tool under Administration | Tools and enable the dotCover coverage in the .NET CLI build step. If you want to use cross-platform dotCover under Windows as well, make sure the agents have .NET Framework SDK 4.6.1+ installed.
Now you can also run code coverage analysis with dotCover inside a Docker container, with the Docker Wrapper extension.
Updates for multinode setup
User-level actions on secondary nodes
In previous versions of TeamCity, secondary nodes provided a read-only interface. It was not possible to add builds to the queue, tag/pin builds, or perform any other user-level actions. With this release, it changes. Now, if a secondary node is granted any responsibility (that is it does not act as a read-only server), it will enable build actions for users. Currently, supported user-level actions are:
Triggering a build, including a custom or personal one
Stopping a build
Pinning/tagging/commenting a builds
Deleting a build
Assigning investigations and muting build problems and tests
Marking a build as successful/failed
Merging sources and label sources actions
See the full list of supported actions in our tracker: TW-62749.
Administrator-level actions are not yet available on secondary nodes. Use the main server if you need to change the server configuration.
Optimized server-side patches download on agents
Since this release, agents can download server-side patches from secondary nodes — not only from the main server, as it was before.
Server-side patches are mostly used when an agent cannot find a VCS client executable (for example, Git or Perforce) on an agent machine. In this case, the agent will request the server to create a patch with VCS changes and send it to the agent. Now, if you assign the "VCS repositories polling " and "Processing data produced by builds " responsibilities to a secondary node, the agents will be able to request patches from this node as well, which significantly reduces the load on the main server.
Updates for DSL-based projects
TeamCity Kotlin DSL receives the following updates:
DSL offers an alternative way to configure a build chain in a pipeline style (read more).
You can now customize the DSL generation behavior using context parameters configured in the TeamCity UI (read more).
If the "Store secure values outside of VCS" option is enabled for a project, TeamCity allows you to see all the project tokens and generate new ones on the Version Settings | Tokens tab (read more).
New features of experimental UI
TeamCity 2019.2 introduces the redesigned Build Details and Agents pages, as well as other experimental features.
Experimental Build Details Page
In our attempt to rethink the approach to displaying build details, we have redesigned the Build Details page so it provides better visualization and gives quick access to all other projects via the sidebar.
You can instantly preview all the previous builds and their details, without leaving the current build page:
A visualized build timeline reflects the duration of each stage and indicates build problems:
Click any build stage to open the corresponding line of the build log. Note that in the new UI, even a long log can be displayed directly in the preview, with no need to download it.
Apart from the build timeline and build log, the Overview tab gives quick access to build problems, tests, changes, and dependencies. The corresponding tabs have also been updated and now offer new features:
The Changes tab displays more information about changes in the build. You can separately browse user and artifact changes, and optionally display changes in build settings. Click any change to preview its details.
The Tests tab allows quickly switching between failed, ignored, and succeeded tests. Click a test to view its details and assign an investigation.
The Dependencies tab provides three alternative modes of displaying the build dependencies: visual timeline, structured list, and build chain.
Experimental Agents page
The experimental Agents page loads faster for a large number of agents and allows quickly switching between agent details. You can use the sidebar to browse the agent pool hierarchy and search agents and pools by name. The Reports section provides statistics about running, idle, and disconnected agents in every pool.
Other new features of experimental UI
Comparison of two builds
With the Compare Builds feature, you can select two builds from the same configuration and review information about their parameters, revisions, statistics, and tests side-by-side. This allows for easier monitoring and is especially helpful when multiple users manage and monitor builds. For example, if a build has no changes in the project code but fails for no obvious reason, you can compare this build with the last successful build and analyze their differences to find the most probable cause of the failure.
To compare a build with another, open the Actions menu of this build, click Compare with, and select a build for comparison.Expanded build preview in the build list
TeamCity now allows you to preview the most important build results directly in the list of builds. On the Build Configuration Home page, click a build line in the list to see its details.Favorite Projects section
We have migrated the Projects section that gives quick access to your favorite projects.Redesigned Changes pop-up
Build changes are now sorted chronologically and grouped by changes in code and changes in artifact dependencies. You can also filter changes by their author and toggle the display of changes made in the build configuration settings.Information on build queueing reason
If a build has been staying in a queue for too long, you can now see a reason for the delay by clicking the "In queue" label when previewing the build in Trends or in the build list.
Other improvements
The branch filter has been added for artifact dependencies, retry triggers, and the Pull Requests build feature.
Autodetection of Dockerfile changes: once the Docker Compose build runner creates a Docker image on an agent, it is stored on this agent to be used for the later builds. Now, TeamCity will automatically detect that the Dockerfile, used for building the image, has changed and will force the Docker Compose step to rebuild this image.
You can now enforce the interval of VCS changes check, specified in Global Settings, as a minimum polling interval for all VCS roots on the server. This way, Project Administrators will only be able to set intervals that are larger than the default one. This helps restrict the frequency of polling requests thus offloading the server.
Project administrators with the enabled "Change user / group notification rules in project" permission can edit notification rules for users and user groups assigned to their projects.
Now you can re-run personal builds, just like regular builds: open the context menu of a finished build and click Re-run this build.
TeamCity can automatically manage the amount of memory used by the
git fetchprocess.
If you have previously used theteamcity.git.fetch.process.max.memoryinternal property to set the memory amount available for fetching in each VCS root, you can now disable it to delegate the detection of memory consumption to the TeamCity server. You can control the limit of available memory via theteamcity.git.fetch.process.max.memory.limitproperty.You can configure more than one SSH key for a build. It is useful if the build needs to authenticate in several external systems or repositories. To use multiple SSH keys in a build:
On the project level: specify these keys in SSH Keys.
On the build configuration level: add several SSH Agent build features, one per each key.
TeamCity uses improved rules for resolving assigned investigations and unmuting muted problems and tests in active branches. Now it waits until the build problem (or failed test) is fixed in all active branches before unmuting it or resolving its investigation.
The retry build trigger now can trigger a new build with the same revision and put it to the queue top.