Configuring Running Builds Node
It is possible to use a separate TeamCity server instance to process traffic coming from the TeamCity agents. This allows moving the related load to a separate machine from the main TeamCity server and improve general performance of the TeamCity server when handling hundreds of concurrent, actively logging running builds. In general, you do not need the two-node installation unless you have more than 400 agents connected to a single server. Using the running builds node allows you to significantly increase the amount of agents which the setup can handle.
Overview of two-node configuration
The same TeamCity server distribution is used for the main and running builds nodes, the role is selected based on the startup settings. When a TeamCity server is started as the running builds node, its functionality is reduced to only one task - process real time data coming from the running builds. The primary setup is running the two server nodes on different machines.
Several important notes about two-node configuration:
the server started as the Running Builds Node preprocesses it and stores it in the database and on the disk in the TeamCity Data Directory
in a two-node installation the main TeamCity server handles all other tasks: user interface, VCS-related activity, management of agents, scheduling builds on agents, etc.
both the main TeamCity server and the Running Builds Node require access to the same data directory (which must be shared) and to the same database
firewall settings should allow access to Running Builds Node from the agents and from the main TeamCity server (the main TeamCity server also communicates with the running builds node by HTTP)
Shared Data Directory
Ensure that both machines where TeamCity server nodes will be installed can access the same TeamCity data directory in the read / write mode. Using dedicated network file storage with good performance is recommended.
Note that Running builds node mostly writes data under TeamCity data directory, while main TeamCity server can do both read / write operations, and also scan directories during cleanup. So performance of file operations can be more important for the main server. As such to achieve better performance main server can either reside on the machine where data directory is located, or can use some kind of block level data storage (SAN). The same data directory can be then shared with Running builds node by SMB or NFS.
Tuning storage and network performance is recommended: make sure to review performance guidelines for your storage solution. For example, increasing MTU for the network connection between the server and the storage usually has positive effect on artifacts transfer speed.
Note: if you use Windows and one of your nodes accesses the TeamCity data directory via a mapped network drive, this may lead to the inability to access the data directory if TeamCity is running as a service. This happens because Windows services cannot work with mapped network drives. On the other hand, TeamCity does not support the UNC format (\\host\directory) for the data directory path. To workaround this problem, you can use mklink as it allows making a symbolic link on a network drive:
mklink /d "C:\<path to mount point>" "\\<host>\<shared directory name>\"
Installation
On both machines, install TeamCity software as usual: download a distribution package, unpack it or follow the installation wizard.
Configure the TEAMCITY_DATA_PATH environment variable on both machines, make sure it points to the shared data directory.
On the Running Builds Node machine, add additional arguments to the TEAMCITY_SERVER_OPTS environment variable: export
TEAMCITY_SERVER_OPTS=-Dteamcity.server.role=running-builds-node -Dteamcity.server.rootURL=<node url> <your regular options if you have them>, where<node url>is the URL of the Running Builds node. This URL must be accessible by both the agents and main server. If you do not have an HTTP proxy installed in front of the TeamCity servlet container and you did not change the port in the servlet container during the installation, then by default this URL will be:http://<your host name>:8111/
Startup & shutdown
Both the main TeamCity server and Running Builds Node can be started / stopped using regular TeamCity scripts (teamcity-server.bat, teamcity-server.sh) or Windows services.
The Running Builds Node server uses the same approach to logging as the main server. You can check how the startup is doing in the <TeamCity installation directory>/logs/teamcity-server.log file. You can also open <Running Builds Node URL> in your browser, there you should see regular TeamCity startup screens.
Enabling Running Builds Node
When both the main server and Running Builds Node are up and running, the Administration | Nodes Configuration page on the main TeamCity server displays the Running Builds Node:
By default, the running builds node is disabled and all traffic produced by running builds is still handled by the main server. Once you enable the running builds node, all newly started builds will be routed to this node. The existing running builds will continue being executed on the main server.
And vice versa, when the running builds node has been disabled, only the newly started builds will be switched to the main server, the builds that were already running on the second node will continue running there.
The Running Builds Node on the Administration | Nodes Configuration page displays how many builds are currently assigned to this node. After a Running Builds Node is enabled, eventually all of the running builds will be switched to it.
Restarting Servers while Builds are Running
The Running Builds Node as well as the main TeamCity server can be stopped or restarted while builds are running there. If agents cannot connect to the running builds node for some time, they will reroute their data to the main server. If the main server is unavailable either, agents will keep their data and resend it once the servers reappear.
Upgrade & Downgrade
Both the main TeamCity server and the Running Builds Node must be of exactly the same version.
To upgrade:
stop the running builds node (if you forget to do that, you'll be warned during upgrade)
start upgrade on the main TeamCity server as usual
proceed with upgrade
check everything is OK, agents are connecting etc. (since the running builds node is not available anymore, the agents will reroute their data to the main server)
upgrade TeamCity installation on the running builds node to the same version
start the running builds node and check that it is connected using the Nodes Configuration page on the main server
To downgrade:
shutdown the main server and the running builds node
restore data from backup (only if the data format has been changed during upgrade)
downgrade TeamCity software on the main server
start the main TeamCity server and check that everything works
downgrade TeamCity software on the the running builds node to the same version as the main server
start the the running builds node
As usually with TeamCity, agents perform upgrade / downgrade automatically.
Cleanup
The TeamCity cleanup task runs on the main TeamCity server only. In a two-node configuration as well as in a single node configuration, it can work at the same time while the servers are handling data from the running builds.
Backup & Restore
Backup through TeamCity web interface can be done on the main TeamCity server only. Backup from the command line can be done on either the running builds node or on the main TeamCity server.
Restore can be done on either of these nodes, but only if all of the servers using the TeamCity database and data directory are stopped.
Current limitations
Only one running builds node can be configured for the main TeamCity server.
If you use non-bundled TeamCity plugins, some fraction of them might be not compatible with the running builds node setup as few TeamCity API endpoints stop working in the setup (only the obsolete API related to build message processing listeners is affected). Also, there is no possibility to write plugins specific to the running builds node.