Reporting Issues

Best Practices When Reporting Issues

Following these guidelines will ensure timely response and effective issue resolution. Check Feedback for appropriate ways to contact us.

When submitting a request for support, include the following general information:

• The TeamCity version in use, including the build number (this can be found in the footer and the teamcity-server.log). It is also worth including the Operating System and Java version being used.

• Document any pattern of issue occurrence (first time, recurring, and so on), if it was mitigated in the past, whether there were any recent environment changes. Be Specific: note exact times, error messages, and describe both the expected and actual behavior.

• Whenever possible, include related screenshots of the TeamCity UI (always include the entire page and the browser URL in the capture) and provide details on the related settings files, actual values, REST API entity representations.

• When including an error message or other related text messages (as text, not as an image), include the complete message and where it was/is being observed. If the message is more than a few lines long, attach it to the ticket as a file.

• If sending more than three files, package the files into a single archive.

• If you are sending a file larger than 10Mb, follow the guide for Uploading Large Data Archives.

Other helpful tips for specific issues:

• If necessary, attach/upload an archive with TeamCity server logs (see details, ideally, the entire <server_home>\logs directory with all the directories inside. If impractical, all the files updated around or later than the issue time); if related to the build time or agent behavior, attach the entire <agent_home>\logs directory.

• For performance/slowness/delays issues, take a set of (10+ spread across the issue time) server or agent thread dumps during the issue occurrence and make sure to send us the dumps with the related data.

• Consider checking if the issue exists in our YouTrack Issue Tracker. We identify most of the bugs and edge-conditions here. There is a chance your issue has already been fixed in a later TeamCity version.

• If you have an issue with your build logic, make sure it is a TeamCity-related issue (that is it does not reproduce in the same environment without TeamCity) and include details of your investigation.

• When reporting build procedure issues, try to reproduce them without TeamCity on the agent machine in the same environment and let us know the results.

• If specific builds are affected, include the complete build logs (downloaded via a dedicated link from the build results' Build Log tab).

• When replacing or masking sensitive data in logs, note the replacement patterns used.

• If there is an antivirus installed and/or if there is a network proxy or reverse proxy in front of the TeamCity server, include this information in your request.

• List any non-bundled plugins installed.

• Note any previous modifications of the TeamCity Data Directory or the database.

• Do not include large portions (above 10Kb) of the textual data into the email text — rather attach it in a file.

Consider the following guidelines for posts:

• When posting new details about the already reported issue, update previous posting on the topic instead of creating new ones. However, if you need to create a duplicate topic, make sure to note all previous postings on the same topic you have made or found.

• Post one issue per submission, the most important issue first. If you do make more than one submission, make a note of the other issues if they are related.

Check the sections below for common cases and specific information to collect and send to us.

Slowness, Hangings and Low Performance

If TeamCity is running slower than you would expect, please use the notes below to locate the slow process and send us all the relevant details if the process is a TeamCity one.

OutOfMemory Problems

If you experience problems with TeamCity consuming too much memory or "OutOfMemoryError"/"Java heap space" errors in the log, do the following:

• Determine what process encounters the error (the actual building process, the TeamCity server, or the TeamCity agent). You can track memory and CPU usage by TeamCity with the charts on the Administration | Server Administration | Diagnostics page of your TeamCity web UI.

• If the problem is on the server's side, check you have increased memory settings from the default ones for using the server in production (see the section).

• If the problem is on the build process' side, set "JVM Command Line Parameters" settings in the build runner. Increase the value for the -Xmx JVM option: for instance, -Xmx1200m. Note that Java Inspections builds may specifically need increasing the -Xmx value.

• If the problem is on the TeamCity server's side and increasing the memory size does not help, please report the case for us to investigate. For this, while the server is high on memory consumption, take several server thread dumps as described above, get the memory dump (see below) and all the server logs including threadDumps-* sub-directories, archive the results, and send them to us for further analysis. Make sure that the -Xmx setting is less than 8Gb before getting the dump:

  • if a memory dump (hprof file) is created automatically, the java_xxx.hprof file is created in the process startup directory (<TeamCity Home>/bin or <TeamCity Agent home>/bin);

  • for the server, you can also take memory dump manually when the memory usage is at its peak. Go to the Administration | Server Administration | Diagnostics page of your TeamCity UI and click Dump Memory Snapshot.

  • another approach to take a memory dump manually is to use the jmap standard JVM command line utility of the full JVM installation of the same version as the Java used by the process. Example command line is:

    jmap -dump:file=<file_on_disk_to_save_dump_into>.hprof <pid_of_the_process>

See how to change JVM options for the server and for agents.

"Too many open files" Error

1. Determine what computer it occurs on

2. Determine the process which has opened a lot of files and the files list (on Linux use lsof, on Windows you can use handle or TCPView for listing sockets)

3. If the number is under thousands, check the OS and the process limits on the file handles (on Linux use ulimit -n) and increase them if necessary. Note that default Linux 1024 handles per process is way too small for a server application like TeamCity. Increase the number to at least 16000. Check the actual process limits after the change as there are different settings in the OS for settings global and per-session limits (e.g. see the post)

If the number of files is large and looks suspicious and the locking process is a TeamCity one (the TeamCity agent or server with no other web applications running), then, while the issue is still occurring, grab the list of open handles several times with several minutes interval and send the result to us for investigation together with the relevant details.

Note that you will most probably need to reboot the machine with the error after the investigation to restore normal functioning of the applications.

Agent does not connect to the server

Please refer to Common Problems.

Logging events

The TeamCity server and agent create logs that can be used to investigate issues.

For detailed information, refer to the corresponding sections:

• TeamCity Server Logs

• Viewing Build Agent Logs

Patch Application Problems

In case the server-side checkout is used, the "patch" that is passed from the server to the agent can be retrieved by:

• add the property system.agent.save.patch=true to the build configuration.

• trigger the build. the build log and the agent log will contain the line "Patch is saved to file ${file.name}"Get the file and supply it with the problem description.

Build triggers debug logging

To collect all build triggers debug logs in TeamCity 2021.1 and later, switch the logging preset on the Administration | Diagnostics page to debug-triggers, reproduce the problem and then collect all the teamcity-triggers.log files.

In TeamCity versions before 2021.1, it is only possible to enable debug logging for a VCS trigger defined in a specific build configuration:

1. Take a default logging preset file <TeamCity Home>/conf/teamcity-server-log4j.xml and save it with some other name under the <TeamCity Data Directory>/config/_logging/ directory.

2. Modify the resulting file as follows:

   • add a new appender to log VCS trigger related events to a separate file:

   <appender name="ROLL.VCS.TRIGGER" class="jetbrains.buildServer.util.TCRollingFileAppender"> <param name="file" value="${teamcity_logs}/teamcity-vcs-trigger.log"/> <param name="maxBackupIndex" value="20"/> <layout class="org.apache.log4j.PatternLayout"> <param name="ConversionPattern" value="[%d] %6p [%15.15t] - %30.30c - %m%n"/> </layout> </appender>

   • add a new category with the name jetbrains.buildServer.buildTriggers.vcs.AllBranchesVcsTrigger_<build configuration id> as follows:

   <category name="jetbrains.buildServer.buildTriggers.vcs.AllBranchesVcsTrigger_MyBuildConfigurationId"> <priority value="DEBUG"/> <appender-ref ref="ROLL.VCS.TRIGGER"/> </category>

   Note: in the example above, MyBuildConfigurationId is the ID of the build configuration with a VCS trigger which we want to debug.

3. Switch the logging preset on the Administration | Diagnostics page to the newly created preset file.

No server restart is required. If the configuration is correct, TeamCity will create the teamcity-vcs-trigger.log log file in a few minutes. The file will contain the debug log related to the selected configuration only.

Logging for .NET Runners

To investigate process launch issues for .NET-related runners, enable debugging as described below. The detailed information will then be printed into the build log. It is recommended not to have the debug logging for a long time and revert the settings after investigation.

Aa alternative way to enable the logging is as follows: Add the teamcity.agent.dotnet.debug=true configuration parameter in the build configuration or on the agent and run the build.

1. Open the <agent home>/plugins/dotnetPlugin/bin directory.

2. Make a backup copy of teamcity-log4net.xml.

3. Replace teamcity-log4net.xml with the content of teamcity-log4net-debug.xml.

Remote Run Problems

The changes that are sent from the IDE to the server on a remote run can be retrieved from the server .BuildServer/system/changes directory. Locate the <change_number>.changes file that corresponds to your change (you can pick the latest number available or deduce the URL of the change from the web UI). The file contains the patch in the binary form. Please send it with the problem description.

Logging in IntelliJ IDEA/Platform-based IDEs

To enable debug logging for the IntelliJ Platform-based IDE plugin, include the following fragment into the Log4j configuration of the <IDE home>/bin/log.xml file.

After changing this file, restart the IDE. The TeamCity plugin debug logs are saved into idea-teamcity\* files and will appear in the logs' directory of the IDE settings (<IDE settings/Data Directory>/system/log directory).

TeamCity Visual Studio Add-in issues

dotCover Issues

To collect additional logs generated by JetBrains dotCover, add the teamcity.agent.dotCover.log configuration parameter to the build configuration. This parameter should store an absolute or relative path to an empty agent directory where you want to keep dotCover logs. If you need to publish these logs as build artifacts, add the same directory path to the list of artifact paths.

JVM Crashes

On a rare occasion of the TeamCity server or agent process terminating unexpectedly with no apparent reason, it can happen that this is caused by a Java runtime crash.
If this happens, the JVM regularly creates a file named hs_err_pid*.log in the working directory of the process. The working directory is usually <TeamCity server home>/bin or <agent home>/bin.
Under Windows, when running as a service, it can be other like C:\Windows\SysWOW64. You can also search the disk for the recent files with "hs_err_pid" in the name. See also the related Fatal Error Log section in this document.

Please send this file to us for investigation and consider updating the JVM for the server (or for agents) to the latest version available.

If you get the "There is insufficient memory for the Java Runtime Environment to continue. Native memory allocation (malloc) failed to allocate..." message with the crash or in the crash report file, make sure to switch to 64-bit JVM or reduce the -Xmx setting below 1024m, see details in the memory configuration section.

Build Log Issues

While investigating issues related to a build log, we might need the raw binary build log as stored by TeamCity. It can be downloaded via the Web UI from the Build Log build's tab: select "Verbose" log detail and use the "raw messages file" link in the upper right corner.

IntelliJ IDEA Inspections

The inspections result from a TeamCity build may not match inspections from a local run in IntelliJ IDEA.

To help us investigate issues with inspections, do the following:

1. Add system.teamcity.dont.delete.temp.result.dir=true to the configuration parameters

2. Add %system.teamcity.build.tempDir%/inspection*result/** => inspections-reports-data-%build.number%.zip rule to Artifact paths

3. Add %system.teamcity.build.tempDir%/idea-logs/** => inspections-reports-idea-logs-%build.number%.zip rule to Artifact paths

4. Add -Didea.log.path=%system.teamcity.build.tempDir%/idea-logs/ to the runner's JVM command line parameters field.

5. Run a new build.

6. Send us inspections-reports-*.zip files.

Uploading Large Data Archives

Files under 10 MB in size can be attached right into the tracker issue (if you do not want the attachments to be publicly accessible, limit the attachment visibility to "teamcity-developers" user group only).
You can also send small files (up to 2 MB) via email: teamcity-support@jetbrains.com or via online form (up to 20 MB). Please do not forget to mention your TeamCity version and environment and archive the files before attaching.

Large files can be uploaded via https://uploads.jetbrains.com/. Please let us know the exact filename after the upload.

If you cannot upload a large file in one go, try splitting the file into parts and upload them separately.