Setting Up TeamCity for Amazon EC2

1. Open the Amazon EC2 console.

2. Create the required instance. Refer to these Amazon tutorials for more information: Linux, Windows, macOS.

   tip

   The exact instance type (for example, t2.small or x1e.xlarge) is irrelevant: you can set the required type in TeamCity settings (see the Add a Cloud Image section).

3. Install an agent on your instance. Depending on the OS type, the required steps may vary. See these articles for more information:

   • Install TeamCity Agent

   • Configure Agent Installation

4. Install any additional software required to run build steps and the TeamCity agent itself: JDKs and JREs, SDKs, runtime frameworks, building tools such as Git and Docker, and so on.

   tip

   When using EC2-hosted agents, you occasionally need to update software installed on corresponding instances and rebuild AMIs. Otherwise, outdated cloud agents that connect to your TeamCity server will instantly disconnect and spend some time upgrading their tools.

   To minimize the number of software that needs regular updates, consider opting for container orchestration solutions instead. This approach allows TeamCity to dynamically create pods that pull the Agent Docker images with the latest building tools.

   TeamCity supports integration with the following solutions:

   • Kubernetes (both self-hosted clusters and hosting services, such as Google Cloud GKE or Amazon EKS)

   • Amazon Container Service (ECS)

5. Run the agent and ensure it successfully connects to your TeamCity server and is compatible with all required configurations.

6. Configure the system to automatically run a TeamCity agent when an instance starts.

7. For Windows instances, follow these steps: Additional configuration for Windows agents.

8. Restart your instance to ensure the build agent starts automatically and can connect to the server.

Skip this section if you want the instance created in the previous step to directly connect to a TeamCity server and serve as a stand-alone agent machine. Otherwise, if you want TeamCity to automatically scale the number of active cloud agents based on the current workload, create an AMI from this instance.

1. Stop the build agent. For Windows instances, stop the agent service but leave its startup type Automatic.

2. Remove all temporary and excess data from your instance: installation wizards, downloaded archives, build logs, and so on.

3. Optional: Delete the <agent_home>/logs and <agent_home>/temp directories.

4. Optional: Delete the <agent_home>/conf/amazon-* file.

5. Remove the following properties from the <agent_home>/conf/buildAgent.properties file:

   • name — TeamCity will automatically assign unique names to your instances.

   • serverUrl — for the EC2 integration plugin, this property is safe to remove since you will provide a server URL for all instances when setting up a cloud profile. Other plugins may require this property to be present and set to a correct value.

   • (optional) authorizationToken — new cloud agents are authorized automatically.

6. Stop the instance.

7. On the overview page of your instance, click Actions | Image and Templates | Create Image.

   

A cloud profile is a collection of general settings for TeamCity to start virtual machines.

1. Navigate to Administration | <Required Project> | Cloud Profiles. If you want cloud agents in this profile to be available globally, choose the <Root project>. Profiles owned by individual projects can be used to spawn agents that can be used only in these projects.

2. Click Create new profile.

3. Set Cloud type to "Amazon EC2".

4. Enter your profile name and optional description.

5. Choose between authentication via access key/secret pair or credentials stored locally on the server machine. Regardless of the selected mode, a user or IAM role used by TeamCity to access AWS resources must have all permissions listed in this section: Required IAM Permissions.

   Access Keys Local Credentials This option lets you specify credentials that TeamCity will use to access your AWS resources. Using static credentials is the least secure approach, we recommend the Use default credential provider chain option instead. 5.1. Go to the AWS Identity and Access Management (IAM) dashboard. 5.2. Switch to the Users tab and find a user whose credentials can be used by TeamCity to access your EC2 instances and AMIs. 5.3. Switch to the Security Credentials tab and scroll to the Access keys section. 5.4. Create a new key. Paste its ID and secret to the related fields in TeamCity UI. The Use default credential provider chain option allows TeamCity to look for AWS credentials stored on the server machine. Typically, the config file with your AWS credentials is located at ~/.aws/config on Linux or macOS, or at C:\Users\USERNAME\.aws\config on Windows. This approach is more stable and secure compared to using static access keys. See the following article for more information about locally stored AWS credentials: Configuration and credential file settings.

6. Choose an AWS region in which your instances are hosted.

7. Set up the agent limit. This number specifies the overall limit for agents created from all cloud images of this profile.

8. Specify the TeamCity server URL. This value will be automatically passed to agents' buildAgent.properties files. If not specified, agents will use the same value as on the Administration | Global Settings page.

9. Specify the set of criteria for winding down active cloud agents. You can choose how long agents can remain idle and (or) how long they can perform actual building routines. The agent will be terminated if any condition is met, but only after that agent finishes the current build.

   

10. Click Apply changes to save the profile and exit the profile settings page.

Cloud profiles specify global settings, such as authorization credentials and instance regions. Each profile can have one or multiple images that store settings related to the specific type of cloud instance that should be started. You can add as many images to a profile as your needs dictate. However, note that the total number of agents started by all images cannot exceed the limit set in the profile settings (and the number of agents permitted by your license).

1. Click the Add image button.

2. Specify the optional image name.

3. Choose the required image type.

   AMI Instance Spot Fleet Request Choose an AMI that TeamCity will use to spawn identical instances. 3.1. Check the Use launch template option if you want TeamCity to import and use a specific launch template. When the default/latest version of the template updates on the server, TeamCity detects these changes and updates the running instances. 3.2. Choose a required AMI. Own AMI — TeamCity scans a collection of AMIs available under credentials specified in the cloud profile settings. You can browse all found AMIs and choose a required image. AMI by ID — Allows you to utilize a shared AMI. AMI by tags — Specify a comma-separated list of AWS tags, for example, Owner=Mike,Project=Glacier,Subnet=Public. If the specified tags point to multiple AMIs, TeamCity will use the most recently created AMI. If no AMIs were found, the image name under the Agents section will be "Image name (no data)" instead of "Image name (ami-xxxxxxxxx)". 3.3. Specify one or multiple instance types. Specify one type (for example, t2.medium) if you need to launch On-Demand or spot instances of this specific type only. You can add multiple images that target the same AMI with different instance type values. By doing so you can set different active instance limits for each type, and manually start instances of a specific type. Set multiple types (for instance, t2.small, t2.medium, and t2.large) if you plan to order spot instances. This approach increases your chances of having a spot instance assigned. note Mac instances support only mac1.metal and mac2.metal types and are available only as bare metal instances on Dedicated Hosts, with a minimum allocation period of 24 hours before you can release the Dedicated Host. Learn more: Amazon EC2 Mac instances. 3.4. Optional: Specify additional image settings. IAM Role — The IAM role that all launched instances will assume. This role specifies the permissions granted to applications running on your EC2 instances. The AWS account used by TeamCity must have iam:ListInstanceProfiles and iam:PassRole permissions to utilize IAM roles. Key pair — Required if you may need to connect to your EC2 instances using SSH. User data — Allows you to specify a script that will be run when an instance launches. Learn more: Windows, Linux. Tags — The list of comma-separated instance tags. For example LaunchedBy=TeamCity,TeamCityCloudProfileName=MyProfile1. Tagging requires the ec2:*Tags permission. See the following section for more information: Tagging. 3.5. Check Use spot instances if you prefer cheaper spot instances to On-Demand ones. The Max price field lets you to specify your maximum bid price for spot instances (in US dollars). The default On-Demand price will be used if the bid price is not specified. TeamCity can automatically choose Regions or Availability Zones in which your spot requests are most likely to succeed based on their spot placement scores. To allow TeamCity to request and utilize these scores, add the ec2:GetSpotPlacementScores IAM permission. tip It is not recommended to use spot instances for production-critical builds due to the possibility of an unexpected spot instance termination by Amazon. If a spot instance is terminated, TeamCity will fail the build with a corresponding build problem and re-add this build to the build queue. 3.6. Specify networking settings for your EC2 instances. VPC — The virtual private cloud to which new instances will belong. Subnets — The IP addresses range for your VPS. Security groups — Specify rules for incoming and outgoing connections to (from) your EC2 instance. This option allows you to add the specific instance to TeamCity. Compared to AMIs that allow TeamCity to start multiple identical instances, this type implies you have a static virtual machine that TeamCity can start and stop. 3.1. Specify the desired instance type. Since there can be only one active instance, you can select only one type. 3.2. Choose a Key pair if you may need to connect to your EC2 instance using SSH. Amazon Spot Fleet allows you to book a combination of regular (On-Demand) and spot instances based on the given criteria. See the following AWS documentation article for sample requests: Spot Fleet example configurations. 3.1. In AWS Management Console, Go to Instances | Spot Requests | Request Spot Instances. 3.2. Specify the required AMI, minimum compute unit, availability zones, and other request details. 3.3. Click JSON config to download a JSON file with the spot fleet configuration parameters. Note that only fields from the SpotFleetRequestConfigData class are supported. 3.4. Paste the generated JSON configuration file in the Spot Fleet Request field in TeamCity. To run the image, TeamCity will launch spot instances matching the allocation strategy specified in the spot fleet configuration. note TeamCity uses own values instead of the following parameters of the JSON config file: TerminateInstancesWithExpiration: false ValidFrom: the time when the image is created ValidUntil: the ValidFrom value plus 5 minutes Type: request warning Currently, LaunchTemplateConfigs are not supported. Use LaunchSpecifications instead to describe required Spot instances.

4. Enter a integer number ranging from -10000 to 10000 in the Image priority field (the default priority is 0). When TeamCity needs to spin up a new cloud agent, it chooses an image that has the highest priority value.

   TeamCity uses priority values to range images from all existing profiles. For example, newly queued builds can run on cloud agents spawned from profiles A and B. The Profile A has three images with priorities 20, 40, and 60. The Profile B has 10, 30, and 50 priority images. TeamCity will spin up new agents in the following order:

   • Profile A, image priority 60.

   • Profile B, image priority 50.

   • Profile A, image priority 40, and so on.

   An image with a lower priority is used only when an available image with a higher priority reaches its active agents limit.

   note

   The value of the Image priority field is assigned to the teamcity.agent.priority property of each agent that originates from this image. You can manually specify this property for any other non-AWS-hosted agent to control which agents TeamCity should prioritize.

5. Set up the maximum number of active cloud agents starting from this image. Note that the total number of agents started from all images added to a profile cannot exceed the limit set on the settings page of this profile.

6. Specify the agent pool to which newly created instances will belong.

7. Click Save to exit the image settings page.

After you have configured an image, TeamCity winds up one test agent for this image to test whether it can start and connect to the server. When an agent is connected and authorized, TeamCity saves its parameters to correctly assign builds to compatible agents.

If your image has the Use spot instances setting checked and a test instance cannot be launched at the moment (for example, if there is no available capacity or your bid price is too low), TeamCity will repeat attempts to launch a spot instance once a minute, as long as there are queued builds which can run on this agent.

TeamCity requires the following permissions for Amazon EC2 Resources:

• ec2:Describe*

• ec2:StartInstances

• ec2:StopInstances

• ec2:TerminateInstances

• ec2:RebootInstances

• ec2:RunInstances

• ec2:ModifyInstanceAttribute

• ec2:*Tags

To use spot instances, grant the following permissions in addition to those listed above:

• ec2:RequestSpotInstances

• ec2:CancelSpotInstanceRequests

• ec2:GetSpotPlacementScores (optional, allows TeamCity to choose AWS Regions or Availability Zones based on their spot placement scores).

To use Spot Fleets, the following additional permissions are required:

• ec2:RequestSpotFleet

• ec2:DescribeSpotFleetRequests

• ec2:CancelSpotFleetRequests

• iam:CreateServiceLinkedRole (if you are getting "The provided credentials do not have permission to create the service-linked role for EC2 Spot Fleet" error; you can safely revoke this permission once the service role is created)

To launch an instance under an assumed IAM Role (applicable to instances cloned from AMIs and launch templates), the following additional permissions are required:

• iam:ListInstanceProfiles

• iam:PassRole

To use encrypted EBS volumes, the following additional permissions are required:

• kms:CreateGrant

• kms:Decrypt

• kms:DescribeKey

• kms:GenerateDataKeyWithoutPlainText

• kms:ReEncrypt*

The snippet below illustrates a custom IAM policy definition that allows all EC2 operations initiated by TeamCity from the specific IP address:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "1",
      "Action": [
        "ec2:Describe*",
        "ec2:StartInstances",
        "ec2:StopInstances",
        "ec2:TerminateInstances",
        "ec2:RebootInstances",
        "ec2:RunInstances",
        "ec2:ModifyInstanceAttribute",
        "ec2:*Tags",
        "ec2:RequestSpotInstances",
        "ec2:CancelSpotInstanceRequests",
        "ec2:GetSpotPlacementScores",
        "ec2:RequestSpotFleet",
        "ec2:DescribeSpotFleetRequests",
        "ec2:CancelSpotFleetRequests"
      ],
      "Effect": "Allow",
      "Condition": {
      	"IpAddress": {
          "aws:SourceIp": "<TeamCity server IP address>"
      	}
      },
      "Resource": "*"
    },
    {
      "Sid": "2",
      "Action": [
        "kms:CreateGrant",
        "kms:Decrypt",
        "kms:DescribeKey",
        "kms:GenerateDataKeyWithoutPlainText",
        "kms:ReEncrypt*"
      ],
      "Effect": "Allow",
      "Condition": {
      	"IpAddress": {
          "aws:SourceIp": "<TeamCity server IP address>"
      	}
      },
      "Resource": "*"
    },
    {
      "Sid": "3",
      "Action": [
        "iam:CreateServiceLinkedRole",
        "iam:ListInstanceProfiles",
        "iam:PassRole"
      ],
      "Effect": "Allow",
      "Condition": {
      	"IpAddress": {
          "aws:SourceIp": "<TeamCity server IP address>"
      	}
      },
      "Resource": "*"
    }
  ]
}

See also: Example policies (Linux), Example policies (Windows).

To ensure proper TeamCity agent communication with EC2 API (including access to additional drives) on Windows, add a dependency from the TeamCity Build Agent service on the AmazonSSMAgent or EC2Launch/EC2Config service (the service which ensures the machine is fully initialized in regard to AWS infrastructure use). This can be done, for example, via the Registry or using sc config (for instance, sc config TCBuildAgent depend=EC2Config).
Alternatively, you can use the "Automatic (delayed start)" service starting mode.

The behavior of EBS optimization in TeamCity is similar to that offered by EC2 console. When configuring an image of the Amazon cloud profile, the optimization can be set using the corresponding box of the Instance Type. Note that:

• EBS optimization is turned on by default for c4.*, d2.*, and m4.* (non-configurable);

• EBS optimization is turned off by default for any other instance types and can be turned on for instances that support it (such as c3.xlarge).

If your TeamCity server needs to use a proxy to connect to AWS API endpoint, configure the following server internal properties to connect to Amazon AWS addresses.

• teamcity.http.proxy.host.ec2 — proxy server host name

• teamcity.http.proxy.port.ec2 — proxy server port

For proxy server authentication:

• teamcity.http.proxy.user.ec2 — proxy access username

• teamcity.http.proxy.password.ec2 — proxy access user password

For NTLM authentication:

• teamcity.http.proxy.domain.ec2 — proxy user domain for NTLM authentication

• teamcity.http.proxy.workstation.ec2 — proxy access workstation for NTLM authentication

Here are some points to help you estimate TeamCity-related traffic:

• If TeamCity server is not located within the same EC2 region or availability zone that is configured in TeamCity EC2 settings for agents, traffic between the server and agent is subject to usual Amazon EC2 external traffic charges.

• When estimating traffic, bear in mind that there are lots types of traffic related to TeamCity (see a non-complete list below).

External connections originated by a server:

• VCS servers

• Email servers

• Maven repositories

Internal connections originated by a server:

• TeamCity agents (checking status, sending commands, retrieving information like thread dumps, and so on)

External connections originated by a agent:

• VCS servers (in case of agent-side checkout)

• Maven repositories

• Any connections performed from the build process itself

Internal connections originated by an agent:

• TeamCity server (retrieving build sources in case of server-side checkout or personal builds, downloading artifacts, and so on)

Usual connections served by a server:

• Web browsers

• IDE plugins

As Amazon rounds machine uptime to the full hour for some configurations (more at How are Amazon EC2 instance hours billed?), adjust timeout setting on the EC2 image setting on TeamCity cloud integration settings according to your usual builds length.

It is also highly recommended to set execution timeout for all your builds so that a build hanging does not cause prolonged instance running with no payload.