Using AWS Aurora Database Cluster

Overview

When using an AWS Aurora cluster with TeamCity pointing to the cluster end-point as the database server, it is important to understand what happens when an AWS Aurora cluster fails over.

Both AWS Aurora DB instances are rebooted (so for a short period of time TeamCity entirely loses connection to the cluster) and

• the original DB instance is started in read-only mode (the new reader instance);

• the former failover instance is the new writer, and the cluster endpoint DNS record is changed to point to the new writer instance.

By default, TeamCity JVM caches DNS name lookups, which essentially means that TeamCity will stay connected to the original DB instance until DNS cache expires. This in turn leads to the database connection pool on the TeamCity side to be populated with the connections to the new reader.

It will take some time for the JVM-specific cache in TeamCity to expire and for the invalid connections to be evicted from the pool.

General Recommendations

When working with a failover cluster, it is recommended to decrease the JVM-specific DNS caching on TeamCity by setting the TTL to 60:

1. Add the -Dsun.net.inetaddr.ttl=60 JVM option to the environment variable.

2. Restart TeamCity for the changes to take effect.

note

You may also wish to manually flush your OS-specific DNS cache, but this is the action to be taken each time Aurora fails over.

Forcing TeamCity to Connect to New Writer

You can force TeamCity to connect to the new writer either manually or automatically.

To force the connection manually:

• Reboot the new DB reader instance again: the reboot will take up to 2 minutes which is sufficient for the DNS cache to expire as well as for the invalid connections to be evicted from the pool.

• Alternatively, restart TeamCity manually and all the connections will be created anew.

To force TeamCity to automatically connect to the new primary instance as soon as it's up and running:

note

The following options may affect your TeamCity server performance.

1. Configure the database connection pool to use a special validation query, so that the connections to the DB instance are tested before and/or after use and if a connection to the read-only database is detected, they are evicted from the pool.
   For this, add the following lines to the <TeamCity Data Directory>/config/database.properties:
   For Aurora MySQL:

   testOnBorrow=true
   testOnReturn=true
   testWhileIdle=true
   timeBetweenEvictionRunsMillis=60000
   validationQuery=select case when @@read_only + @@innodb_read_only \= 0 then 1 else (select table_name from information_schema.tables) end as `1`

   For Aurora PostgreSQL:

   testOnBorrow=true
   testOnReturn=true
   testWhileIdle=true
   timeBetweenEvictionRunsMillis=60000
   validationQuery=select case when not pg_is_in_recovery() then 1 else random() / 0 end

2. Restart the TeamCity server. Once you do that, the specified validation query will be executed for all connections whenever they are borrowed from or returned to the pool, and also every 1 minute (60000 milliseconds) for idle connections, raising an error for each connection to the read-only database.

Using SSL Connection to Aurora MySQL Cluster

Since Amazon Aurora MySQL version 2 (compatible with MySQL 5.7 and higher), you can determine if TeamCity uses a secure SSL connection to your DB cluster by running the respective query. If necessary, you can change the connection mode on the Amazon server side.

If you are using Amazon Aurora MySQL version 1 (compatible with MySQL 5.6 and lower), you can check if the connection between TeamCity and the DB cluster is secure by inspecting its traffic with a packet analyzer like tcpdump.
To change the mode of the SSL connection (for example, to enforce or disable it), pass the respective sslMode parameter to the MySQL Connector/J JDBC driver by adding it to the <TeamCity data directory>/config/database.properties file:

• Either add a separate line:

  connectionProperties.sslMode=<value>

  or

• Append the ?sslMode=<value> argument to the connection string. For example:

  connectionUrl=jdbc:mysql://<hostname>:3306/<dbname>?sslMode=<value>

See the list of supported connection mode values in Connector/J configuration properties.