Configuring Secondary Node

In addition to the main server, you can start a secondary TeamCity server looking at the same database and data directory as the main one. In TeamCity 2018.1 the secondary server provides user interface in read-only mode and does not perform any other background activity.

With secondary node it is possible to set up a High-Availability TeamCity installation that will have zero read downtime, i.e. when the main server is unavailable or is performing an upgrade, requests will be routed to the secondary node. Such setup requires installing both the main server and the secondary node behind a reverse proxy that should be configured to route requests to the main server while it's available and to the secondary one in other cases.

In TeamCity 2018.2 the secondary server provides user interface in read-only mode and can optionally perform VCS repositories polling.

The sections below describes the secondary node configuration.

Secondary Node Overview

The same TeamCity server distribution is used for the main and secondary nodes. The primary setup is running the two server nodes on different machines.

The secondary node uses the license from the main server and does not require a separate license .

When a TeamCity server is started as the secondary node, it allows users read operations: view the build information, download artifacts, etc. during the downtime of the main server, e.g. during upgrade. When both main and secondary nodes are active, the secondary node constantly watches for different events produced by the main server and refreshes its internal state. So users whould see actual state on the secondary node at any point of time.

It is recommended that both the main TeamCity server and the secondary node have the exactly the same version. However, the main server and the secondary one can be running different versions when the main server is upgraded. See the Upgrade section below.

The secondary node has a number of limitations compared with the main server:

The secondary node does not allow changing the server configuration, start builds, or change state of the server anyhow. Therefore the pages are served in the read only mode, and not all the Administration area pages are available. There are no actions like Run build, etc.
Currently, only bundled and a limited set of some other plugins are loaded by the secondary server, so some functionality provided by external plugins can be missing.
Users may need to re-login when they are routed to the secondary node if they didn't use Remember Me.

Starting Secondary Node

A secondary node needs to access the same TeamCity Data Directory and the same external database as the main TeamCity Server. The typical data directory mounting options are SMB and NFS. TeamCity uses the data directory as a regular file system so all the basic file system operations should be supported. The backing storage and way of mounting should not impose strict IO operations count or IO volume limits and should provide performance like that of a local drive.

Besides the data directory shared with the main server, the secondary node requires a 'local' data directory where it stores some caches, unpacked external plugins and some other configuration. By default, the 'local' data directory is automatically created under the <Main Data Directory>/nodes/<node ID> path during the first start. You can override its location using-Dteamcity.node.data.path property in the TeamCity start-up scripts.

To start a secondary node along with the main TeamCity Server, do the following:

Install the TeamCity software as usual: download a distribution package, unpack it or follow the installation wizard.
Configure the location of the TeamCity Data Directory on the secondary Node, make sure it points to the shared data directory.
In 2018.1.x, add additional arguments to the TEAMCITY_SERVER_OPTS environment variable: TEAMCITY_SERVER_OPTS= -Dteamcity.server.nodeId=<nodeID> -Dteamcity.server.role=read-only-serverIn 2018.2.x, add additional arguments to the TEAMCITY_SERVER_OPTS environment variable: TEAMCITY_SERVER_OPTS=-Dteamcity.server.nodeId=<nodeID>where <nodeID> is the name of your server, which will be displayed on the Administration | Nodes Configuration page.
(Optional) you can configure Secondary Node URL via -Dteamcity.server.rootURL=<node url > .

It is possible to run several secondary nodes at the same time. Before starting a secondary node, make sure that the database is configured to accept enough parallel connections to handle requests from all nodes.

Startup & Shutdown

Both the main TeamCity server and the secondary node can be started / stopped using regular TeamCity scripts (teamcity-server.bat, teamcity-server.sh) or Windows services.

The secondary node server uses the same approach to logging as the main server. You can check how the startup is doing in the <TeamCity installation directory>/logs/teamcity-server.log file. You can also open <secondary node URL> in your browser and you should see the regular TeamCity startup screens.

Proxy Configuration

The configurations examples below will proxy requests to the main server while it's available and to the secondary in other cases. If you did not setup TeamCity server behind reverse proxy before, make sure to review our How To... on this topic.

For example, the following https://docs.nginx.com/nginx/admin-guide/load-balancer/http-health-check configuration will route requests to the secondary node only when the main server is not available or when the main server responds with 503 status code, which generally means that it is starting or is performing an upgrade.

http {
    upstream backend {
        server teamcity-main.local:8111 max_fails=0; # full internal address of the main server;
        server teamcity-ro.local:8111 backup; # full internal address of the secondary  node;
    }
 
    server {
        location / {
            proxy_pass      	http://backend;
            proxy_next_upstream error timeout http_503 non_idempotent;
         }
    }
} 

Note that http://nginx.org/en/ does not support active https://docs.nginx.com/nginx/admin-guide/load-balancer/http-health-check/ (which is a better way to configure High Availability installation) and may experience DNS resolution issues. Consider using NGinx Plus or HA Proxy.

HA Proxy configuration example:

resolvers dns
   nameserver dns %IP%:%PORT%
   resolve_retries       3
   timeout resolve       1s
   timeout retry         1s
defaults
   mode http
frontend http-in
   bind *:80
   default_backend tc-ha
backend tc-ha
   option httpchk GET /login.html
   default-server inter 2s fall 2 rise 2
   server tc_main_node %TC_MAIN_NODE_URL:8111% check resolvers dns
   server tc_ro_node %TC_RO_NODE_URL:8111% backup check resolvers dns

Enabling VCS Polling on Secondary Node

Usually the main TeamCity server polls the VCS repositories for changes to detect new commits.

Since TeamCity 2018.2, you can delegate VCS polling to the secondary node. To do it, navigate to the Administration | Nodes Configuration page on the main server and enable this responsibility for the secondary node. It may take some time for the main server to finish the polling activities in progress, then the secondary node will pick up this task. You can disable this responsibility at any moment and the main server will start polling VCS repositories.

If you have commit hooks configured on the main server, no changes are required: the hooks will continue working if the VCS polling is delegated to the secondary node.

Upgrade

It is recommended that both the main TeamCity server and the secondary node should be of exactly the same version. However, the main server and the secondary one can be running different versions, e.g. when the main server has been upgraded. W hen the versions of the secondary node and the main server are different, the corresponding health report will be displayed on both nodes.

When upgrading to a minor version (a bugfix release), both servers should be running without issues as the TeamCity data has the same format.

When upgrading the main server to the major version, its TeamCity data format will be changed and data upgrade will be performed while the secondary node is running provided the secondary node is not performing any write operations. If the write operations are detected on the secondary node, a warning will be displayed and the main server upgrade will be suspended until the secondary node is stopped. After the main server is upgraded to a new major version, the secondary node shows a health report when it detects that the data was upgraded and stops processing new events, i.e. will not show new builds. The secondary node must be upgraded after the main server upgrade to be able to process new events.

TeamCity agents will perform upgrade / automatically.

Last modified: 20 April 2023