Amazon S3 and S3-compatible Storages

1. Navigate to the Administration | <Your_Project> page and switch to the Artifacts Storage tab.

   • Open settings of a <Root project> if you want your new storage to be available for all TeamCity projects.

   • Edit one specific project if your new storage should be available only for this project and its sub-projects.

2. The built-in TeamCity artifacts storage is displayed by default and marked as active. Click Add new storage button to create a new storage.

3. Specify the custom storage name and, if needed, its internally used ID.

4. Set the Type field to "AWS S3".

5. Choose an existing AWS Connection that TeamCity should use to access your Amazon resources. If no suitable AWS connection exists, click the "+" icon to add one.

   tip

   You cannot use AWS Connections that belong to a parent project and have their Available for sub-projects settings disabled.

   However, if a parent project has an S3 storage configured using a non-shared connection, this storage is still available to child projects.

   A user whose credentials the selected AWS Connection uses (or an IAM Role it assumes) to access the S3 buckets should have the following permissions:

   • ListAllMyBuckets

   • GetBucketLocation

   • GetObject

   • ListBucket

   • PutObject

   • DeleteObject

   • GetAccelerateConfiguration (if Transfer Acceleration is enabled)

   tip

   In previous TeamCity versions, the Artifacts Storage dialog allowed you to explicitly specify connection settings: access key credentials, user IAM role, and whether TeamCity should look for credentials in the default AWS locations (the Default provider chain setting).

   Starting with version 2023.11, these settings are exclusive to AWS Connections. If you're migrating from an older version of TeamCity and your existing AWS S3 storages used any of these settings, click the Convert to AWS Connection link. This action transfers AWS-related settings to a new AWS Connection, and selects this new connection as the source connection of your storage.

6. TeamCity uses the selected AWS Connection to retrieve the list of available S3 buckets. Open the Bucket drop-down menu to choose a specific item from the list.

7. (Optional) Specify the path prefix if you want to use the same S3 bucket for all TeamCity projects and configure prefix-based permissions.

8. Amazon S3 buckets support two options to speed up file uploads and downloads:

   • AWS CloudFront — a content delivery network (CDN) that allows TeamCity to transfer arifacts using low-latency CloudFront servers nearby.

   • Transfer Acceleration — a bucket-level feature designed to optimize transfer speeds from across the world into centralized S3 buckets. It enables fast, easy, and secure transfers of files over long distances between your client and an S3 bucket.

   If your bucket is configured to use Transfer Acceleration or CloudFront, choose the corresponding option under the Transfer speed-up section. Otherwise, if you wish TeamCity to transfer files in the regular mode, choose the None type.

   

   note

   Choosing the AWS CloudFront option requires setting up additional fields. See this section for more information: Transferring Artifacts via CloudFront.

   note

   Using Transfer Acceleration applies the following restrictions:

   • The virtual host addressing is always on. Path-style requests are not supported.

   • Transfer Acceleration must be enabled on your S3 bucket.

   • Buckets cannot have dots (.) in their names.

   • User credentials or IAM role the AWS Connection uses to access your bucket must include the GetAccelerateConfiguration permission.

9. To optimize the upload of large files to the storage, you can enable the multipart upload. To do this, tick the Customize threshold and part size setting and set the multipart upload threshold. The minimum allowed value is 5MB. Supported suffixes: KB, MB, GB, TB. If you leave this field empty, multipart upload will be initiated automatically for all files larger than 8 MB (8MB is the default value).

   

   Additionally, you can configure the maximum allowed size of each uploaded file part. The minimum value is 5MB. If left empty, TeamCity will use 8MB as the default value.

   tip

   We recommend that you configure a bucket lifecycle policy to prevent incomplete multipart uploads.

10. Uncheck the Force virtual host addressing option to turn off the corresponding feature (enabled by default). Currently, both hosted-style and path-style requests are supported by TeamCity. Note that Amazon stopped supporting path-style access for new buckets since September 2020.

    note

    This setting cannot be disabled if your storage uses Transfer Acceleration.

11. Tick Verify file integrity after upload to allow TeamCity to perform an additional check-up on uploaded files. If the integrity verification fails, TeamCity writes a corresponding message to the build log.

12. Click Save to save your new storage and return to the list of available storages.

note

If your build agent is deployed behind a proxy, you must also configure the proxy settings on the build agent. See Use Proxy for Outgoing Build Agent Connections.

When viewing a list of storages available for a project, click Make Active to start using the corresponding storage for all new builds of this project. The has N usages link allows you to view which builds used this storage to upload their artifacts.

1. Navigate to the Administration | <Your_Project> page and switch to the Artifacts Storage tab.

   • Open settings of a <Root project> if you want your new storage to be available for all TeamCity projects.

   • Edit one specific project if your new storage should be available only for this project and its sub-projects.

2. The built-in TeamCity artifacts storage is displayed by default and marked as active. Click Add new storage button to create a new storage.

3. Specify the custom storage name and, if needed, its internally used ID.

4. Set the Type field to "Custom S3".

5. Specify the Access key ID and Secret access key values. See documentation for your S3-compatible storage vendor for the information on how to issue access keys.

6. Specify the storage endpoint TeamCity should use to access your bucket.

7. (Optional) Specify the path prefix if you want to use the same S3 bucket for all TeamCity projects and configure prefix-based permissions.

8. To optimize the upload of large files to the storage, you can enable the multipart upload. To do this, tick the Customize threshold and part size setting and set the multipart upload threshold. The minimum allowed value is 5MB. Supported suffixes: KB, MB, GB, TB. If you leave this field empty, multipart upload will be initiated automatically for all files larger than 8 MB (8MB is the default value).

   

   Additionally, you can configure the maximum allowed size of each uploaded file part. The minimum value is 5MB. If left empty, TeamCity will use 8MB as the default value.

9. Uncheck the Force virtual host addressing option to turn off the corresponding feature (enabled by default). Currently, both hosted-style and path-style requests are supported by TeamCity. Note that Amazon stopped supporting path-style access for new buckets since September 2020.

10. Tick Verify file integrity after upload to allow TeamCity to perform an additional check-up on uploaded files. If the integrity verification fails, TeamCity writes a corresponding message to the build log.

11. Click Save to save your new storage and return to the list of available storages.

note

If your build agent is deployed behind a proxy, you must also configure the proxy settings on the build agent. See Use Proxy for Outgoing Build Agent Connections.

When viewing a list of storages available for a project, click Make Active to start using the corresponding storage for all new builds of this project. The has N usages link allows you to view which builds used this storage to upload their artifacts.

Amazon S3 Storage Classes allow you to fine-tune your storage based on its desired performance, as well as availability and resilience of its data.

There are two ways to enable the required storage class:

• On TeamCity side. When uploading artifacts to an S3 bucket, TeamCity adds the x-amz-storage-class header to its PUT requests. The header value depends on the corresponding storage setting in TeamCity (for example, x-amz-storage-class: INTELLIGENT_TIERING). This mode does not require any additional setup on the AWS side.

  Although this approach is not currently supported, we hope to implement this functionality in our future release cycles. Upvote and comment on this YouTrack ticket to support the feature and share your feedback: TW-79992.

• On AWS side. In this mode, TeamCity uploads artifacts in a regular manner and the required storage class is applied by a pre-configured lifecycle rule after the artifacts were uploaded. To set up this rule, do the following:

  1. Open the required S3 storage and switch to the Management tab.

  2. Click Create lifecycle rule.

  3. Check Move current versions of objects between storage classes under the Lifecycle rule actions section.

  4. Choose the required storage class and the delay between the upload and transition dates. Set the Days after object creation to "0" to transition your artifacts as soon as TeamCity uploads them.

     tip

     TeamCity does not support archive storage classes since their files are not immediately available and require additional unpack/warmup actions before they can be fetched.

  5. Enable additional rules for stored artifacts. For example, you can check Expire current versions of objects to label previously uploaded artifacts as expired, and Permanently delete noncurrent versions of objects to periodically clean your storage.

  6. Specify the rule scope to choose whether it should apply to the entire storage or only those artifacts that match the required filter.

  7. Review your rule at the bottom of the page. It may look like the following:

     

  8. Click Create rule to save your lifecycle rule.

See the following AWS help article for more information: Using Amazon S3 storage classes.

Amazon CloudFront is a content delivery network that offers low latency and high transfer speeds. Enabling its support for an S3 storage will allow TeamCity to transfer artifacts through the closest CloudFront server. If your S3 bucket is located in a different region than your TeamCity infrastructure, this could significantly speed up the artifacts' upload/download and reduce expenses.

tip

If you use EC2 build agents located in the same region as the target S3 bucket, these agents will communicate with the bucket directly, omitting CloudFront.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "1",
            "Effect": "Allow",
            "Action": [
                "cloudfront:CreatePublicKey",
                "cloudfront:CreateOriginRequestPolicy",
                "cloudfront:ListCloudFrontOriginAccessIdentities",
                "cloudfront:DeleteKeyGroup",
                "cloudfront:GetPublicKey",
                "cloudfront:ListCachePolicies",
                "cloudfront:CreateDistribution",
                "cloudfront:ListOriginRequestPolicies",
                "cloudfront:DeletePublicKey",
                "cloudfront:CreateCloudFrontOriginAccessIdentity",
                "cloudfront:CreateKeyGroup",
                "cloudfront:CreateCachePolicy",
                "cloudfront:GetDistribution",
                "cloudfront:ListPublicKeys",
                "s3:ListAllMyBuckets",
                "cloudfront:ListKeyGroups",
                "cloudfront:ListDistributions"
            ],
            "Resource": "*"
        },
        {
            "Sid": "2",
            "Effect": "Allow",
            "Action": [
                "s3:PutBucketPolicy",
                "s3:GetBucketPolicy"
            ],
            "Resource": "arn:aws:s3:::<YOUR_BUCKET_NAME>"
        }
    ]
}

{
"Sid": "1",
"Effect": "Allow",
"Principal": {
      "AWS": "arn:aws:iam::cloudfront:user/CloudFront Origin Access Identity <OAI ID"
  },
  "Action": [
      "s3:GetObject",
      "s3:PutObject",
      "s3:DeleteObject"
  ],
  "Resource": "arn:aws:s3:::<S3 bucket name/*"
}

{
"Sid": "PublicRead",
"Effect": "Allow",
"Principal": "*",
  "Action": [
        "s3:GetObject",
        "s3:GetObjectVersion"
    ],
  "Resource": "arn:aws:s3::<BUCKET_NAME>/*"
}

The following sample illustrates how to add an S3 bucket as custom storage for project artifacts, and set this storage as primary (default).

import jetbrains.buildServer.configs.kotlin.*
import jetbrains.buildServer.configs.kotlin.projectFeatures.activeStorage
import jetbrains.buildServer.configs.kotlin.projectFeatures.s3Storage

project {
    // ...
    features {
        activeStorage {
            id = "PROJECT_EXT_37"
            activeStorageID = "PROJECT_EXT_4"
        }
        s3Storage {
            id = "PROJECT_EXT_4"
            awsEnvironment = default {
                awsRegionName = "eu-west-1"
            }
            connectionId = "AwsPrimary"
            storageName = "S3 Transfer Acceleration"
            bucketName = "dk-s3ta"
            enableTransferAcceleration = true
            forceVirtualHostAddressing = true
            verifyIntegrityAfterUpload = true
        }
    }
    // ...
}

tip

To quickly get an ID of an AWS Connection that should be used to retrieve a required bucket, navigate to the required Administration | <Your_Project> | Connections page.

TeamCity server ships with a command-line tool that transfers build artifacts from one storage to another. You can download this tool on the Project Settings | Artifacts Storage page.

Currently, the tool supports the following migration routes:

• From a local directory to an Amazon S3 bucket and vice versa

• From one Amazon S3 bucket to another

We're working on supporting other cloud storage options as well.

Learn more: Artifacts Migration Tool.