TeamCity Monitoring and Diagnostics

TeamCity provides a variety of diagnostic tools and indicators to monitor and troubleshoot the server. These tools make it easier to identify and investigate problems and, if needed, report issues on your server.

This article describes the diagnostics tools available in Administration | Diagnostics. You can also find more details in the nested articles of this section.

Troubleshooting

This tab provides a number of indicators helping you to detect and address issues with the TeamCity server performance.

CPU & Memory Usage

This section displays the data provided by the TeamCity Memory Monitor, which regularly checks available memory and submits a warning if the memory consumption grows too high. See also information on configuring memory settings for the TeamCity server.

Depending on your operating system and Java settings, the list of displayed properties below may vary.

CPU usage

TeamCity process CPU usage shows the CPU time used by the main TeamCity process averaged over a period of time. Other processes consuming CPU resources (databases, Maven server, VCS) are not included.
Garbage collection shows the time spent on cleaning the server memory averaged over a period of time. High numbers over long periods of time usually indicate insufficient server memory.
System load displays CPU load information based upon the CPU queue length averaged over a period of time, i.e. the sum of the number of runnable entities queued to the available processors and the number of entities running on the available processors averaged over a period of time (1 minute)
Overall CPU usage shows the recent CPU usage for the whole system with a value in the [0.0,1.0] interval. A value of 0.0 means that all CPUs were idle during the recent period of time, while a value of 1.0 means that all CPUs were actively running 100% of the time during the recent period

Memory usage

Total heap displays the total amount of memory used by TeamCity to store all data, including temporary data about to be collected. Represents all heap memory pools (young and old generations) in terms of java. Garbage collection runs quickly and frequently.
Data shows the amount of memory used by TeamCity to store persistent data. Represents old generations only in terms of java. Garbage collection runs slowly and infrequently.

Troubleshooting

Debug Logging

This allows changing the internal TeamCity server logging settings.

Hangs and Thread Dumps

The server thread dump can be viewed in the browser or saved to a file.

If you experience memory problems, this section provides an option to dump a memory snapshot.

Server Restart

This section allows restarting the server from the UI.

Java Configuration

This section informs you on the Java installed on your server and the configured JVM options.

VCS Status

This tab displays the information on the monitored VCS roots, including their checking for changes status and duration.

You can filter the available VCS roots by the checking for changes duration.

Metrics

TeamCity provides server load metrics which can be used for monitoring the TeamCity server health.

The Metrics tab displays all supported metrics, their tag parameters, and current values.

The <TeamCity_server_URL>/app/metrics endpoint provides the metrics in a Prometheus format, ready for importing to monitoring solutions with a Prometheus support (for example, to Grafana). Note that server metrics can be obtained only by a user with the "View usage statistics" permission.

The experimental tag for metrics is not reported starting with TeamCity 2023.05. The ?experimental=true URL parameter for metrics in the Prometheus format still works, and some of the metrics still have the experimental status. If you find any of the experimental metrics useful and would want them to be graduated to the supported metrics, let us know via our support channel.

Server Logs

This tab allows you to view and download the available TeamCity server logs, as well as saved thread dumps and memory dumps.

Internal Properties

This tab displays the TeamCity server internal properties and allows modifying them.

Logging Presets

TeamCity uses the log4j library for logging internal server activities. In this section you can view and download the available presets, as well as upload new presets, which can then be enabled on the Diagnostics | Troubleshooting | Debug Logging.

It is also possible to change the logging configuration manually.

Caches

This tab shows you the caches of the TeamCity processes stored in <TeamCity Data Directory>/system/caches. Resetting some caches is performed by the server during the clean-up automatically, but sometimes you might need to clear caches manually using the reset link.

vcsContentCache — TeamCity maintains vcsContentCache cache for the sources to optimize communications with the VCS server. The caches are reset during the clean-up time. To resolve problems with sources update, the caches may need to be reset manually.
search — resetting this cache is required when enabling search by build log.
git — contains the bare clone of the remote Git repository used by TeamCity.
buildsMetadata — resetting this cache is required to reindex the TeamCity NuGet feed.

Versioned Settings Caches

The following caches are utilized when you keep settings of a TeamCity project in a version control system. See this article for more information: Storing Project Settings in Version Control

dslDependenciesMaven contains downloaded Maven dependencies specified by users in the pom.xml file of a Kotlin DSL configuration.
generatedVersionedSettings stores generated configs cache to prevent excessive DSL runs.
kotlinDslData stores internal data related to configurations and results of Kotlin DSL runs. This cache is used by TeamCity to maintain Kotlin DSL configurations, for example support UI patches.
pluginsDslCache contains Kotlin DSL extensions from plugins such as sources, compiled JARs, and documentation. Additionally, this cache stores Maven repository that provides Maven dependencies.
versionedSettings contains downloaded versioned settings (the content of the .teamcity folder).
versionedSettingsIncrementalMode supports incremental compilation for Kotlin DSL.

Search

Displays information on the TeamCity data index related to the search.

Browse Data Directory

This tab shows the files in the TeamCity Data Directory and allows you to upload new files.

Last modified: 23 May 2023