TeamCity On-Premises 2023.11 Help

Setting Up TeamCity for Amazon EC2

TeamCity Amazon EC2 integration allows TeamCity auto-scale its building resources by automatically starting and stopping cloud-hosted agents on-demand, depending on the current build queue workload.

Common Information

You can set up various types of EC2 integrations in TeamCity. Depending on the settings and the source you use, cloud AWS-hosted agents can run on:

Overview image
  • Multiple identical instances cloned from the same Amazon Machine Image (AMI). Can be launched as On-Demand or spot instances.

  • A single permanent EC2 instance managed by TeamCity. Can be shared between multiple TeamCity servers.

  • A set of spot instances requested from AWS (Spot Fleet).

Prerequisites

This section describes the steps that you must perform in your AWS account before setting up a cloud profile in TeamCity UI.

Create and Setup an EC2 Instance

  1. Open the Amazon EC2 console.

  2. Create the required instance. Refer to these Amazon tutorials for more information: Linux, Windows, macOS.

  3. Install an agent on your instance. Depending on the OS type, the required steps may vary. See these articles for more information:

  4. Install any additional software required to run build steps and the TeamCity agent itself: JDKs and JREs, SDKs, runtime frameworks, building tools such as Git and Docker, and so on.

  5. Run the agent and ensure it successfully connects to your TeamCity server and is compatible with all required configurations.

  6. Configure the system to automatically run a TeamCity agent when an instance starts.

  7. For Windows instances, follow these steps: Additional configuration for Windows agents.

  8. Restart your instance to ensure the build agent starts automatically and can connect to the server.

Create an AMI

Skip this section if you want the instance created in the previous step to directly connect to a TeamCity server and serve as a stand-alone agent machine. Otherwise, if you want TeamCity to automatically scale the number of active cloud agents based on the current workload, create an AMI from this instance.

  1. Stop the build agent. For Windows instances, stop the agent service but leave its startup type Automatic.

  2. Remove all temporary and excess data from your instance: installation wizards, downloaded archives, build logs, and so on.

  3. Optional: Delete the <agent_home>/logs and <agent_home>/temp directories.

  4. Optional: Delete the <agent_home>/conf/amazon-* file.

  5. Remove the following properties from the <agent_home>/conf/buildAgent.properties file:

    • name — TeamCity will automatically assign unique names to your instances.

    • serverUrl — for the EC2 integration plugin, this property is safe to remove since you will provide a server URL for all instances when setting up a cloud profile. Other plugins may require this property to be present and set to a correct value.

    • (optional) authorizationToken — new cloud agents are authorized automatically.

  6. Stop the instance.

  7. On the overview page of your instance, click Actions | Image and Templates | Create Image.

    Create an AMI

Setup EC2 Integration in TeamCity

After you have created a required instance or AMI, you can set up cloud profiles in TeamCity UI.

Create a Cloud Profile

A cloud profile is a collection of general settings for TeamCity to start virtual machines.

  1. Navigate to Administration | <Required Project> | Cloud Profiles. If you want cloud agents in this profile to be available globally, choose the <Root project>. Profiles owned by individual projects can be used to spawn agents that can be used only in these projects.

  2. Click Create new profile.

  3. Set Cloud type to "Amazon EC2".

  4. Enter your profile name and optional description.

  5. Choose between authentication via access key/secret pair or credentials stored locally on the server machine. Regardless of the selected mode, a user or IAM role used by TeamCity to access AWS resources must have all permissions listed in this section: Required IAM Permissions.

    This option lets you specify credentials that TeamCity will use to access your AWS resources. Using static credentials is the least secure approach, we recommend the Use default credential provider chain option instead.

    5.1. Go to the AWS Identity and Access Management (IAM) dashboard.

    5.2. Switch to the Users tab and find a user whose credentials can be used by TeamCity to access your EC2 instances and AMIs.

    5.3. Switch to the Security Credentials tab and scroll to the Access keys section.

    5.4. Create a new key. Paste its ID and secret to the related fields in TeamCity UI.

    The Use default credential provider chain option allows TeamCity to look for AWS credentials stored on the server machine. Typically, the config file with your AWS credentials is located at ~/.aws/config on Linux or macOS, or at C:\Users\USERNAME\.aws\config on Windows. This approach is more stable and secure compared to using static access keys.

    See the following article for more information about locally stored AWS credentials: Configuration and credential file settings.

  6. Choose an AWS region in which your instances are hosted.

  7. Set up the agent limit. This number specifies the overall limit for agents created from all cloud images of this profile.

  8. Specify the TeamCity server URL. This value will be automatically passed to agents' buildAgent.properties files. If not specified, agents will use the same value as on the Administration | Global Settings page.

  9. Specify the set of criteria for winding down active cloud agents. You can choose how long agents can remain idle and (or) how long they can perform actual building routines.

    Agents terminate conditions
  10. Click Apply changes to save the profile and exit the profile settings page.

Add a Cloud Image

Cloud profiles specify global settings, such as authorization credentials and instance regions. Each profile can have one or multiple images that store settings related to the specific type of cloud instance that should be started. You can add as many images to a profile as your needs dictate. However, note that the total number of agents started by all images cannot exceed the limit set in the profile settings (and the number of agents permitted by your license).

  1. Click the Add image button.

  2. Specify the optional image name.

  3. Choose the required image type.

    Choose an AMI that TeamCity will use to spawn identical instances.

    3.1. Check the Use launch template option if you want TeamCity to import and use a specific launch template. When the default/latest version of the template updates on the server, TeamCity detects these changes and updates the running instances.

    3.2. Choose a required AMI.

    • Own AMI — TeamCity scans a collection of AMIs available under credentials specified in the cloud profile settings. You can browse all found AMIs and choose a required image.

    • AMI by ID — Allows you to utilize a shared AMI.

    • AMI by tags — Specify a comma-separated list of AWS tags, for example, Owner=Mike,Project=Glacier,Subnet=Public. If the specified tags point to multiple AMIs, TeamCity will use the most recently created AMI. If no AMIs were found, the image name under the Agents section will be "Image name (no data)" instead of "Image name (ami-xxxxxxxxx)".

      Invalid AMI tags

    3.3. Specify one or multiple instance types.

    • Specify one type (for example, t2.medium) if you need to launch On-Demand or spot instances of this specific type only. You can add multiple images that target the same AMI with different instance type values. By doing so you can set different active instance limits for each type, and manually start instances of a specific type.

    • Set multiple types (for instance, t2.small, t2.medium, and t2.large) if you plan to order spot instances. This approach increases your chances of having a spot instance assigned.

    3.4. Optional: Specify additional image settings.

    • IAM Role — The IAM role that all launched instances will assume. This role specifies the permissions granted to applications running on your EC2 instances. The AWS account used by TeamCity must have iam:ListInstanceProfiles and iam:PassRole permissions to utilize IAM roles.

    • Key pair — Required if you may need to connect to your EC2 instances using SSH.

    • User data — Allows you to specify a script that will be run when an instance launches. Learn more: Windows, Linux.

    • Tags — The list of comma-separated instance tags. For example LaunchedBy=TeamCity,TeamCityCloudProfileName=MyProfile1. Tagging requires the ec2:*Tags permission. See the following section for more information: Tagging.

    3.5. Check Use spot instances if you prefer cheaper spot instances to On-Demand ones. The Max price field lets you to specify your maximum bid price for spot instances (in US dollars). The default On-Demand price will be used if the bid price is not specified.

    TeamCity can automatically choose Regions or Availability Zones in which your spot requests are most likely to succeed based on their spot placement scores. To allow TeamCity to request and utilize these scores, add the ec2:GetSpotPlacementScores IAM permission.

    3.6. Specify networking settings for your EC2 instances.

    This option allows you to add the specific instance to TeamCity. Compared to AMIs that allow TeamCity to start multiple identical instances, this type implies you have a static virtual machine that TeamCity can start and stop.

    3.1. Specify the desired instance type. Since there can be only one active instance, you can select only one type.

    3.2. Choose a Key pair if you may need to connect to your EC2 instance using SSH.

    Amazon Spot Fleet allows you to book a combination of regular (On-Demand) and spot instances based on the given criteria. See the following AWS documentation article for sample requests: Spot Fleet example configurations.

    3.1. In AWS Management Console, Go to Instances | Spot Requests | Request Spot Instances.

    3.2. Specify the required AMI, minimum compute unit, availability zones, and other request details.

    3.3. Click JSON config to download a JSON file with the spot fleet configuration parameters. Note that only fields from the SpotFleetRequestConfigData class are supported.

    Spot fleet config generation button in Amazon

    3.4. Paste the generated JSON configuration file in the Spot Fleet Request field in TeamCity.

    To run the image, TeamCity will launch spot instances matching the allocation strategy specified in the spot fleet configuration.

  4. Enter a integer number ranging from -10000 to 10000 in the Image priority field (the default priority is 0). When TeamCity needs to spin up a new cloud agent, it chooses an image that has the highest priority value.

    TeamCity uses priority values to range images from all existing profiles. For example, newly queued builds can run on cloud agents spawned from profiles A and B. The Profile A has three images with priorities 20, 40, and 60. The Profile B has 10, 30, and 50 priority images. TeamCity will spin up new agents in the following order:

    • Profile A, image priority 60.

    • Profile B, image priority 50.

    • Profile A, image priority 40, and so on.

    An image with a lower priority is used only when an available image with a higher priority reaches its active agents limit.

  5. Set up the maximum number of active cloud agents starting from this image. Note that the total number of agents started from all images added to a profile cannot exceed the limit set on the settings page of this profile.

  6. Specify the agent pool to which newly created instances will belong.

  7. Click Save to exit the image settings page.

After you have configured an image, TeamCity winds up one test agent for this image to test whether it can start and connect to the server. When an agent is connected and authorized, TeamCity saves its parameters to correctly assign builds to compatible agents.

If your image has the Use spot instances setting checked and a test instance cannot be launched at the moment (for example, if there is no available capacity or your bid price is too low), TeamCity will repeat attempts to launch a spot instance once a minute, as long as there are queued builds which can run on this agent.

Manage Cloud Agents

When a build is queued, TeamCity attempts to run queued builds on regular (non-cloud) agents first. If none are currently available, TeamCity finds a compatible cloud image and (if the limit of simultaneously running instances is not yet reached) starts a new instance.

You can manually start and stop cloud agents from the Agents tab. Note that the Start button is disabled if the number of active cloud agents reaches the limit specified in the profile or image settings.

Start and stop cloud agents

The Running instances block shows all agents started from this specific image.

  • Instances that successfully started but have not yet connected to the TeamCity server have AWS instance IDs as names ("i-xxxxxxxxxxxx").

  • Connected and authorized instances have names merged from a parent image name and an AWS instance ID ("Ubuntu-22.04-Large-i-xxxxxxxxxxxx").

  • If a connected agent reports outdated build tools, it will automatically disconnect for an upgrade. Once all required software is updated, the agent will reconnect to the server. TeamCity shows a corresponding warning for outdated agent images.

    Outdated agent image

You can open an interactive terminal to any active agent instance via the Open Terminal button. The terminal allows you to debug and maintain your cloud agent machines.

DSL Configuration

The following Kotlin snippet illustrates a sample DSL configuration for a cloud profile that uses locally stored credentials and includes three images with different settings.

import jetbrains.buildServer.configs.kotlin.* import jetbrains.buildServer.configs.kotlin.amazonEC2CloudImage import jetbrains.buildServer.configs.kotlin.amazonEC2CloudProfile // ... version = "2023.11" project { vcsRoot(MyRoot) buildType(Build) features { // Image 1: On-Demand Instances for a Specific AMI amazonEC2CloudImage { id = "PROJECT_EXT_14" profileId = "amazon-4" name = "Ubuntu-22.04-Large(AMI)" vpcSubnetId = "mySubnet123" iamProfile = "john-doe-ec2-role" keyPairName = "john-doe-u2204-key" instanceType = "t2.large" securityGroups = listOf("security-group-1") instanceTags = mapOf( "LaunchedBy" to "TeamCity" ) maxInstancesCount = 4 source = Source("ami-1234567890") } // Image 2: Spot Instances for an AMI Located by Tags amazonEC2CloudImage { id = "PROJECT_EXT_20" profileId = "amazon-4" name = "Ubuntu-22.04-Medium-Spot(Tags)" vpcSubnetId = "mySubnet123" instanceType = "t3a.medium,t3a.small" securityGroups = listOf("security-group-1,security-group-2") useSpotInstances = true spotInstanceBidPrice = 1.0 instanceTags = mapOf( "LaunchedBy" to "TeamCity" ) maxInstancesCount = 4 source = Source("tags") param("source-tags", "OS=Ubuntu, OSVersion=22.04") } // Image 3: A Single EC2 Instance amazonEC2CloudImage { id = "PROJECT_EXT_22" profileId = "amazon-4" agentPoolId = "-2" name = "Windows-Server2022(Instance)" vpcSubnetId = "mySubnet123" keyPairName = "john-doe-w2022s-keys" instanceType = "m2.xlarge" securityGroups = listOf("security-group-1") source = Source("i-1234567890") } // Parent Profile for All Images amazonEC2CloudProfile { id = "amazon-4" name = "AWS EC2 Profile" description = "A sample Amazon EC2 profile with AMI and Instance images" serverURL = "https://MyTeamCityBuildFarm.dev.gg" terminateIdleMinutes = 30 region = AmazonEC2CloudProfile.Regions.EU_WEST_DUBLIN maxInstancesCount = 20 authType = instanceIAMRole() param("terminate-after-build", "false") param("terminateTimeOut_checkbox", "true") } } } object Build : BuildType({ name = "Build" allowExternalStatus = true vcs { root(MyRoot) } steps { // Build Steps } triggers { vcs {} } features { // Build Features } }) object MyRoot : GitVcsRoot({ name = "https://github.com/...#refs/heads/main" url = "https://github.com/..." branch = "refs/heads/main" branchSpec = "refs/heads/*" authMethod = password { userName = "JohnDoe" password = "..." } param("oauthProviderId", "PROJECT_EXT_8") })

Additional Setup

Required IAM permissions

TeamCity requires the following permissions for Amazon EC2 Resources:

  • ec2:Describe*

  • ec2:StartInstances

  • ec2:StopInstances

  • ec2:TerminateInstances

  • ec2:RebootInstances

  • ec2:RunInstances

  • ec2:ModifyInstanceAttribute

  • ec2:*Tags

To use spot instances, grant the following permissions in addition to those listed above:

  • ec2:RequestSpotInstances

  • ec2:CancelSpotInstanceRequests

  • ec2:GetSpotPlacementScores (optional, allows TeamCity to choose AWS Regions or Availability Zones based on their spot placement scores).

To use Spot Fleets, the following additional permissions are required:

  • ec2:RequestSpotFleet

  • ec2:DescribeSpotFleetRequests

  • ec2:CancelSpotFleetRequests

  • iam:CreateServiceLinkedRole (if you are getting "The provided credentials do not have permission to create the service-linked role for EC2 Spot Fleet" error; you can safely revoke this permission once the service role is created)

To launch an instance under an assumed IAM Role (applicable to instances cloned from AMIs and launch templates), the following additional permissions are required:

  • iam:ListInstanceProfiles

  • iam:PassRole

To use encrypted EBS volumes, the following additional permissions are required:

  • kms:CreateGrant

  • kms:Decrypt

  • kms:DescribeKey

  • kms:GenerateDataKeyWithoutPlainText

  • kms:ReEncrypt

The snippet below illustrates a custom IAM policy definition that allows all EC2 operations initiated by TeamCity from the specific IP address:

{ "Version": "2012-10-17", "Statement": [ { "Sid": "1", "Action": [ "ec2:Describe*", "ec2:StartInstances", "ec2:StopInstances", "ec2:TerminateInstances", "ec2:RebootInstances", "ec2:RunInstances", "ec2:ModifyInstanceAttribute", "ec2:*Tags", "ec2:RequestSpotInstances", "ec2:CancelSpotInstanceRequests", "ec2:GetSpotPlacementScores", "ec2:RequestSpotFleet", "ec2:DescribeSpotFleetRequests", "ec2:CancelSpotFleetRequests" ], "Effect": "Allow", "Condition": { "IpAddress": { "aws:SourceIp": "<TeamCity server IP address>" } }, "Resource": "*" }, { "Sid": "2", "Action": [ "kms:CreateGrant", "kms:Decrypt", "kms:DescribeKey", "kms:GenerateDataKeyWithoutPlainText", "kms:ReEncrypt" ], "Effect": "Allow", "Condition": { "IpAddress": { "aws:SourceIp": "<TeamCity server IP address>" } }, "Resource": "*" }, { "Sid": "3", "Action": [ "iam:CreateServiceLinkedRole", "iam:ListInstanceProfiles", "iam:PassRole" ], "Effect": "Allow", "Condition": { "IpAddress": { "aws:SourceIp": "<TeamCity server IP address>" } }, "Resource": "*" } ] }

See also: Example policies (Linux), Example policies (Windows).

Additional Configuration for Windows Agents

To ensure proper TeamCity agent communication with EC2 API (including access to additional drives) on Windows, add a dependency from the TeamCity Build Agent service on the AmazonSSMAgent or EC2Launch/EC2Config service (the service which ensures the machine is fully initialized in regard to AWS infrastructure use). This can be done, for example, via the Registry or using sc config (for instance, sc config TCBuildAgent depend=EC2Config).
Alternatively, you can use the "Automatic (delayed start)" service starting mode.

Amazon EBS-Optimized Instances

The behavior of EBS optimization in TeamCity is similar to that offered by EC2 console. When configuring an image of the Amazon cloud profile, the optimization can be set using the corresponding box of the Instance Type. Note that:

  • EBS optimization is turned on by default for c4.*, d2.*, and m4.* (non-configurable);

  • EBS optimization is turned off by default for any other instance types and can be turned on for instances that support it (such as c3.xlarge).

Tagging

Tagging Instances Launched by TeamCity

The following requirements must be met for tagging instances launched by TeamCity:

Without tagging permissions, TeamCity can still launch Amazon AMI and EBS images with no tags applied but is unable to launch Amazon EC2 spot instances

TeamCity enables users to get instance launch information by marking the created instances with the teamcity:TeamcityData tag containing <server UUID>:-<cloud profile ID>:-<image reference>. This tag is necessary for TeamCity integration with EC2 and must not be deleted.

Custom tags can be applied to EC2 cloud agent instances: when configuring Cloud profile settings, in the Add Image/Edit Image dialog use the Instance tags: field to specify tags in the format of <key1>=<value1>,<key2>=<value2>. Amazon tag restrictions need to be considered.

When using the equal(=) sign in the tag value, no escaping is needed. For instance, the string extraParam=name=John will be parsed into <key=extraParam> and value <name=John>.

Tagging Instance-Dependent Resources

When launching Amazon EC2 instances, TeamCity tags all the resources (for example, volumes and network adapters) associated with the created instances, which is important when evaluating the overall cost of an instance (taking into account the storage drive type and size, I/O operations (for standard drives), network (transfers out), and so on.

Sharing an EBS Instance Between Multiple TeamCity Servers

As mentioned above, TeamCity tags every instance it launches with the teamcity:TeamcityData tag that stores information about a server, cloud profile, and source (AMI or EBS-instance). So, when several TeamCity servers try to use the same EBS instance, the second one will see the following message "Instance is used by another TeamCity server. Unable to start/stop it". If you are sure that no other TeamCity servers are working with this instance, you can delete the teamcity:TeamcityData tag and the instance will become available for all TeamCity servers again.

Proxy settings

If your TeamCity server needs to use a proxy to connect to AWS API endpoint, configure the following server internal properties to connect to Amazon AWS addresses.

  • teamcity.http.proxy.host.ec2 — proxy server host name

  • teamcity.http.proxy.port.ec2 — proxy server port

For proxy server authentication:

  • teamcity.http.proxy.user.ec2 — proxy access username

  • teamcity.http.proxy.password.ec2 — proxy access user password

For NTLM authentication:

  • teamcity.http.proxy.domain.ec2 — proxy user domain for NTLM authentication

  • teamcity.http.proxy.workstation.ec2 — proxy access workstation for NTLM authentication

Estimating EC2 Costs

Standard Amazon EC2 pricing applies. Amazon charges can depend on the specific configuration implemented to deploy TeamCity. We advise you to regularly check your configuration and Amazon account data to discover and prevent unexpected expenses as soon as possible.

Note that traffic volumes and necessary server and agent machines characteristics depend a big deal on the TeamCity setup and nature of the builds run. See also Estimate Hardware Requirements for TeamCity.

Estimating Traffic

Here are some points to help you estimate TeamCity-related traffic:

  • If TeamCity server is not located within the same EC2 region or availability zone that is configured in TeamCity EC2 settings for agents, traffic between the server and agent is subject to usual Amazon EC2 external traffic charges.

  • When estimating traffic, bear in mind that there are lots types of traffic related to TeamCity (see a non-complete list below).

External connections originated by a server:

  • VCS servers

  • Email servers

  • Maven repositories

Internal connections originated by a server:

  • TeamCity agents (checking status, sending commands, retrieving information like thread dumps, and so on)

External connections originated by a agent:

  • VCS servers (in case of agent-side checkout)

  • Maven repositories

  • Any connections performed from the build process itself

Internal connections originated by an agent:

  • TeamCity server (retrieving build sources in case of server-side checkout or personal builds, downloading artifacts, and so on)

Usual connections served by a server:

  • Web browsers

  • IDE plugins

Uptime Costs

As Amazon rounds machine uptime to the full hour for some configurations (more at How are Amazon EC2 instance hours billed?), adjust timeout setting on the EC2 image setting on TeamCity cloud integration settings according to your usual builds length.

It is also highly recommended to set execution timeout for all your builds so that a build hanging does not cause prolonged instance running with no payload.

Last modified: 15 January 2024