Common Problems

If a build fails or otherwise misbehaves in TeamCity but you believe it should not, the first thing to do is to check whether the issue is related to TeamCity or not.

To do that, follow this procedure:

1. Find a way to run the task from a command prompt. Make sure it works on the TeamCity agent machine, under the same user as the TeamCity agent runs under, with the same environment the agent receives. If necessary, run the TeamCity agent under a different user or tweak its environment.

2. When the command runs OK, configure the same command in a TeamCity build using the Command Line runner with the custom script setting.

3. If that works, try another runner if that feels applicable.

Here are details on the approach:

Check that the build runs fine from the command prompt when run on the same machine as the TeamCity agent and under the same user that the agent is running, with the same environment variables and the same working directory, same architecture (32/64 bit) command line.

If the TeamCity build agent is run as a service (for example, it is installed as a Windows service), try running the TeamCity agent under a regular user with administrative permissions from the command line. See also Windows Service limitations.

If this fixes the issue, you can try to figure out why running under the service is a problem for the build. Most often this is service-specific and is not related to TeamCity directly. Also, you can set up the TeamCity agent to be run from the console all the time (for example, configure an automatic user logon and run the agent on the user logon).

Here are the detailed steps you can use to run a build from the command line.

Assuming you have a configured build in TeamCity which is failing, do the following:

• run the build in TeamCity and see it misbehaving

• disable the agent so that no other builds run on it. This can be done while the build is still in progress

• log in to the agent machine using the same user as the one running the TeamCity agent (check the right user in the machine processes list)

• stop the agent

• in a command line console, cd to the checkout directory of the build in question (the directory can be looked up in the beginning of the build log in TeamCity)

• run the build with a command line as you would do on a developer machine. This is runner-dependent. (For some runners you can look up the command line used by TeamCity in the build log, see also the logs\teamcity-agent.log file for the command line used by TeamCity)

• if the build fails — investigate the reason as the issue is probably not TeamCity-related and should be investigated on the machine.

• if it runs OK, continue

• in the same console window cd to the <TeamCity agent home>/bin directory and start TeamCity agent from there with the agent start command

• ensure the runner settings in TeamCity are appropriate and should generate the same command line as you used manually. For example, use the Command Line build step with the Custom script option and the same command which can be saved in a .sh or .bat file and run from the command prompt

• run the build in TeamCity selecting the agent in the Run custom build dialog

• when finished, enable the agent

If the build succeeds from the console but still fails in TeamCity, use a command line runner in TeamCity to launch the same command as in the console.

If it still behaves differently in TeamCity, most probably this is an environment or a tool issue.

If the command line runner works but the dedicated runner does not while the options are all the same, create a new issue in our tracker detailing the case. Please attach all the build step settings, the build log, all agent logs covering the build, the command you used in the console to run the build and the full console output of the build.

If you experience slow builds, the first thing to do is to check the build log to see if there are some long operations or the time is just spread over the entire process. You can compare build logs of slower and faster builds to figure out what the difference is. You can also run the build from the console on the same machine as detailed above to see if there is any difference between the build run from the console and the build in TeamCity.

If the slowness is spread over all the operations, the agent machine resources (CPU, disk, memory, network) are to be analyzed during the build to see if there is a bottleneck in any of those. The Performance Monitor build feature lets you see how much CPU, memory and disk I/O is used by the build and at which stage.

If build process is hanging, then "View thread dump" link on the running build results page can help with understanding why it happens.

If there is some long operation and it is a TeamCity-related one (before the start or after the end of the actual build process), the TeamCity agent and server are to be analyzed (logs and thread dumps).

If you want to turn to us with the issue, make sure to describe the visible effects, detail the process of investigation and attach the build log, full agent logs and other data collected.

First start of agent after installation or TeamCity server upgrade/plugin installation can take time as agent downloads updates form the server and autoupgrades.

Regularly, agent should become connected in 1 to 10 minutes, depending on the agent/server network connection speed.

If the agent is not connected within that time, check the name of the agent (as configured in conf\buildAgent.properties file) and check the tabs under the Agents page:

• the agent is under Connected — the agent is ready to run builds

• the agent is under Disconnected — the agent was connected to the server, but became disconnected. Check the "Inactivity reason" in the table. If the reason is "Agent has unregistered (will upgrade)", then wait for several more minutes

• the agent is under Unauthorized — all the agents connected to the server for the first time should be authorized by a server administrator

If the agent stays in the state for more than 10 minutes and you have a fast network connection between the agent and the server, do the following:

• check the related agent machine to ensure that the agent process is running and serverURL in conf\buildAgent.properties is correct (and that the server is reachable by that URL from the machine);

• check that all the related environment requirements are met;

• check agent logs (teamcity-agent.log, launcher.log, upgrade.log) for any related messages/errors;

• check server logs (teamcity-server.log) for any messages/errors mentioning agent name or IP.

If you cannot find the cause of the delayed agent upgrade in the logs, contact us and provide the full agent and server logs. Be sure to check/include the state of the agent processes (java ones) on the agent machine.

If you encounter a case when artifacts are preserved while they should have been removed by the server clean-up process, check the following:

• the clean-up rules of the build configuration, artifacts section

• presence of the icon "This build is used by other builds" in the build history line (prior to Pin action/icon on Build History)

• build's Dependencies tab, "Delivered Artifacts" section. For every build configuration, check whether "Prevent dependency artifacts clean-up" is turned ON (this is default value). If it is, then the build's artifacts are not cleaned because of the setting.

Read more on clean-up settings.

There are two kinds of Maven-related issues commonly seen in the TeamCity build configurations:

• Error message on "Maven" tab of build configuration: "An error occurred during collecting Maven project information ... "

• Error message in build configuration with Maven dependencies trigger activated: "Unable to check for Maven dependency Update ..." If the build configuration produces successful builds despite displaying such error messages, these errors are likely to be caused by the server-side Maven misconfiguration.

To collect information for the Maven tab, or to perform Maven dependencies check (for the trigger), TeamCity runs the embedded Maven. The execution is performed on the server machine, and any agent-side maven settings are not accessible. TeamCity resolves the settings.xml files on the server-side separately, as described in Maven Server-Side Settings.

It makes sense to check if the server-side settings.xml files contain correct information about remote repositories, proxies, mirrors, profiles, credentials, and so on.

If you encounter the error, it means the settings stored in the TeamCity Data Directory are in an inconsistent state. This can occur after manual change of the files or if newer version of TeamCity starts to report the inconsistencies.
To resolve the issue, you can edit the file noted in the message on the server file system. (Make sure to create backup copy of the file before any manual edits). Usually server restart is not necessary for the changes to take effect.

VCS root with id "XXX" does not exist

The build configuration or template reference a VCS root which is not defined in the system. Remedy actions: Restore the VCS root or create a new VCS root with the id noted or edit the file noted in the message to remove the reference to the VCS root.

If the TeamCity web UI cannot be accessed after installation, you might be running TeamCity on a port that is already in use by another program. Check and configure your TeamCity installation.

If you are experiencing issues with partial TeamCity NuGet Feed (for example, missing NuGet packages), you might have to reindex the TeamCity NuGet Feed.

To force TeamCity to reindex all available packages and reset the NuGet package list, navigate to the server Administration | Diagnostics | Caches and use the buildsMetadata Reset link.

You might want to use tools which require some manual interaction during the build procedure executed on the TeamCity agent. This is not a TeamCity-specific problem, so it should be approached using generic means.

Under Windows, you might want to configure TeamCity agent to run not as a Service, but with access to the desktop by configuring automatic user logon, related details.

There is no simple solution for Extended Validation (EV) code signing as the feature is built in for a reason. There is some discussion on the issue on stack overflow. The appropriate solution seems to implement a dedicated service with own authorization approach and sign the binaries through it.

By default, Tomcat sets a 4096 bytes (4KB) limit to the request and response HTTP headers. If a header includes too many cookies (for example, from the proxy servers between an end user and TeamCity) or their size is too large, users might stumble on the 400: Request Header too large or 502: Bad Gateway errors when browsing TeamCity. To identify the cause of this problem, we recommend that you inspect the affected header with the browser developer console.

To resolve this issue, increase the Tomcat header limit by editing the maxHttpHeaderSize parameter value for all relevant connectors in the $TOMCAT_HOME/conf/server.xml file. For example, the following sample sets the parameter value to 65536 to increase the limit to 64KB.

<Connector port="8111" protocol="org.apache.coyote.http11.Http11NioProtocol"
           connectionTimeout="60000"
           redirectPort="8543"
           useBodyEncodingForURI="true"
           tcpNoDelay="1"
           maxHttpHeaderSize="65536"
/>