Common Problems
Build fails or behaves differently in TeamCity but not locally
If the build fails or otherwise misbehave in TeamCity but you believe it should not, please:
check that the build runs fine on the same machine as TeamCity agent and under the same user that the agent is running, with the same environment variables and the same working directory.
if TeamCity build agent is installed as a Windows service, try running TeamCity agent Setting up and Running Additional Build Agents. See also Known Issues.
If this fixes the issue, you can try to figure out why running under the service is a problem for the build. Most often this is service-specific and does not relate to TeamCity directly. Also, you can setup TeamCity agent to be run from console all the time (e.g. configure automatic user logon and run the agent on user logon).
The following approach can be used for this test: For a configured build in TeamCity which is failing:
run the build and see it misbehaving
disable the agent so that no other builds run on it. This can be done while the build is still in progress
log in to the agent machine using the same user as TeamCity agent runs under
stop the agent
in a command console, cd to the checkout directory of the build in question (the directory can be looked up in the beginning of the build log in TeamCity)
run the build with a command line as you would do on a developer machine. This is runner-dependent. (for some runners you can look up the command line used by TeamCity in the build log)
if the build fails - investigate the reason as the issue is probably not TeamCity specific and should be investigated on the machine.
if it runs OK, continue
in the same console window cd to <TeamCity agent home>\bin and start TeamCity agent from there with
agent startcommandensure the runner settings in TeamCity are appropriate and should generate the same command line as you used manually
run the build in TeamCity selecting the agent in Run custom build dialog
when finished, enable the agent
You can also find the command that TeamCity used to launch the build in logs\teamcity-agent.log agent log file.
If the build succeeds from console but still fails in TeamCity, please use a command line runner in TeamCity to launch the same command as in the console. If it still behaves differently in TeamCity most probably this is an environment or tool issue. If the command line runner works good but dedicated runner does not while using all the same options, please create a new issue in our tracker detailing the case. Please attach all the build step settings, the build log, all agent logs covering the build, the command you used in console to run the build and the full console output of the build.
Build is slow under TeamCity
If you experience slow builds, the first thing to do is to check the build log to see if there are some long operations or the time is just spread over the entire process. You can compare build logs of slower and faster builds to figure out what the difference is. You can also run the build from console on the same machine as detailed Build-fails-or-behaves-differently-in-TeamCity-but-not-locally to see if there is any difference between the build from console and build in TeamCity.
If the slowness is spread over all the operations, agent machine resources (CPU, disk, memory, network) are to be analyzed during the build to see if there is a bottleneck in any of those. If there is, the process loading the resource is to be found and investigated (e.g. with the help of the thread dump taken via "View thread dump" link on the running build results).
If there is some long operation and it is a TeamCity-related one (before start or after end of the actual build process), TeamCity agent and server are to be analyzed (logs and thread dumps).
If you want to Feedback with the issue, please describe the visible effects, detail the process of investigation and attach build log, full agent logs and other data collected.
Agent does not upgrade with "Agent has unregistered (will upgrade)"
This means the agent is downloading the new agent version and should become connected shortly. The time necessary to upgrade mostly depends on the network link between the agent and the server. After downloading the files upgrade can take 1-3 minutes.
If the agent stays in the state for more then 10 minutes and you have fast network connection between the agent and the server, please look into Viewing Build Agent Logs. If you cannot find the cause of the delayed agent upgrade in the logs, Feedback and attach the agent logs and the server logs.
Artifacts of a build are not cleaned
If you encounter a case when artifacts of a build are not cleaned, please check:
the cleanup rules of the build configuration in question, artifacts cleanup section
presence of the icon "This build is used by other builds" in the build history line (prior to Pin action/icon on Build History)
build's Dependencies tab, "Delivered Artifacts" section. For every using build configuration, check whether "Prevent dependency artifacts clean-up" is turned ON (this is default value). If it does, then the build's artifacts are not cleaned because of the setting. Read more on Clean-Up.
"out of memory" error with internal (HSQLDB) database hsqldb_oom
If while TeamCity server statup you encounter errors like:
"error in script file line: ... out of memory""java.sql.SQLException: out of memory"
Try Installing and Configuring the TeamCity Server. If this does not help, most probably this means that you have encountered internal database corruption. You can try to deal with corruption based on notes on HSQLDB documentation.
A way to attempt a manual database restore:
stop the TeamCity server
remove <TeamCity Data Directory>/system/buildserver.data file and replace it with zero-size file of the same name. Please backup the removed file beforehand.
start the TeamCity server
However, if the database does not recover automatically, chances that it can be fixed manually are minimal.
Unfortunately, internal (HSQL) database is not stable enough for production use and we highly recommend using Setting up an External Database for TeamCity non-evaluation usage. If you encountered database corruption, you can restore last good backup or drop builds history and users, but preserve the settings, see Migrating to an External Database.
The transaction... log is full
This error can occur during cleanup, when the external database is MS SQL or Sybase. In this case we recommend increasing the transaction log for the TeamCity database. The log size can be 1 - 16 GB depending on the number of build agents in the system and the number of tests all agents report daily.
Slow download from TeamCity server
If you experience slow seeps when downloading artifacts from TeamCity, try checking the speed on the server machine, downloading from localhost. If the seeps is OK for localhost, the issue can be in the network configuration or OS/hardware settings when combined with TeamCity(Tomcat) settings.
You can try the following approach: in <TeamCity home>\conf\server.xml, change default part (port number may be different)
<Connector port="8111" protocol="HTTP/1.1" connectionTimeout="60000" redirectPort="8543" useBodyEncodingForURI="true" />
to:
<Connector port="8111" protocol="org.apache.coyote.http11.Http11NioProtocol" connectionTimeout="60000" redirectPort="8543" useBodyEncodingForURI="true" socket.txBufSize="64000" socket.rxBufSize="64000" tcpNoDelay="1" />
and restart TeamCity server. If this helps, please let us know via Feedback. If it does not, please revert the settings. To investigate the case you might need to find an administrator with appropriate network-related issues investigation skills to look into the case.
Failure to publish artifacts to server behind IIS reverse proxy
This problem is only relevant to configurations that involve IIS reverse proxy between build server and agents. Sometimes a build agent can be found in infinite loop trying to publish build artifacts to server. Build log looks like this:
[11:15:05]Publishing artifacts [11:15:05][Publishing artifacts] Collecting files to publish: [toZip/** => artifact.zip] [11:15:06][Publishing artifacts] Creating archive artifact.zip (9s) [11:15:06][Creating archive artifact.zip] Creating C:\BuildAgent\temp\buildTmp\ZipPreprocessor2847146024236637664\artifact.zip [11:15:15][Creating archive artifact.zip] Archive was created, file size 32142324 bytes [11:15:15][Publishing artifacts] Sending toZip/** [11:15:25][Publishing artifacts] Sending toZip/** [11:15:39][Publishing artifacts] Sending toZip/** [11:15:48][Publishing artifacts] Sending toZip/** [11:16:01][Publishing artifacts] Sending toZip/** [11:16:16][Publishing artifacts] Sending toZip/**
meanwhile teamcity-agent.log is filled with 404 responses from IIS:
[2012-08-01 12:04:55,514] WARN - jetbrains.buildServer.AGENT - <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"> <html xmlns="http://www.w3.org/1999/xhtml"> <head> <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1"/> <title>404 - File or directory not found.</title> <style type="text/css"> <!-- body{margin:0;font-size:.7em;font-family:Verdana, Arial, Helvetica, sans-serif;background:#EEEEEE;} fieldset{padding:0 15px 10px 15px;}
The most common cause for this is maxAllowedContentLength setting (in IIS) either
is set to too small value
is left uncofigured and so defaults to 30000000 bytes (<30 Mb)
So any aritfact larger than maxAllowedContentLength is discarded by IIS Check the settings value and try to rerun your build