Setting Up TeamCity for Kubernetes

Thanks to the Kubernetes Support plugin, TeamCity can run build agents in your Kubernetes cluster.

Requirements

TeamCity integration with Kubernetes does not depend on the kubectl tool and thus does not require installing it in your cluster.

Make sure the TeamCity user is allowed to perform writing operations in the Kubernetes namespace used by TeamCity agents.

You might also require to configure the following privileges for your Kubernetes user role:

Pods: get, create, list, delete.
Deployments: list, get — if you want to create an agent pod using a deployment configuration.
Namespaces: get, list — to allow TeamCity to suggest the namespaces available on your server.

Here is an example of setting up all required permissions via Kubernetes RBAC:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: teamcity:manage-agents
rules:
- apiGroups: [""]
  resources: ["namespaces"]
  verbs: ["list", "get"]
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "create", "list", "delete"]
- apiGroups: ["extensions", "apps"]
  resources: ["deployments"]
  verbs: ["list", "get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: teamcity:manage-agents
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: teamcity:manage-agents
subjects:
  # proper RoleBinding subject depends on your Authentication strategy
  # use one of examples below

  # if you use OIDC/Certificate auth strategies
  - kind: User
    name: teamcity

  # if you use Service account
  - kind: ServiceAccount
    name: teamcity

Kubernetes Cloud Profile Configuration

To establish the integration with Kubernetes, you need to create a dedicated cloud profile in TeamCity. Open the settings of the required project and, under the Cloud Profiles section, click Create new profile.

The table below lists options specific for Kubernetes cloud profiles.

Option	Description
Kubernetes API server URL	Specify the URL of the Kubernetes API server.
Certificate Authority (CA)	Enter the content of the CA certificate for your cluster.
Kubernetes namespace	Specify a required Kubernetes namespace. Leave empty to use the default namespace.
Authentication strategy	Select the required authentication strategy. Depending on the selected strategy, the set of extra options will vary. Refer to the Kubernetes documentation for details on available options. The Token strategy accepts any token types supported by Kubernetes.

Adding Kubernetes Cloud Image

After configuring the general Kubernetes settings, you can proceed with adding a new build agent image.

Click Add image and configure its options:

Option	Description
Pod specification	Select one of the three types described below.
Agent pool	Assign the launched instances to a specific agent pool.
Agent name prefix	(Optional) Show the same prefix for all instances of this image, when displaying them in TeamCity.
Max number of instances	(Optional) Set this number if you want to limit how many instances are launched based on this image.

Use pod template from deployment

Select this option to run a new pod based on a specific deployment configuration created in your Kubernetes cluster. If you are using Kubernetes Dashboard, you can find the list of available deployments under Workloads | Deployments.

You can create a simple deployment from the latest TeamCity agent image as follows:

create deployment simpledep --image="jetbrains/teamcity-agent:latest"
deployment.apps/simpledep created

Run single container

Select this option to run a single container based on any given agent Docker image from Docker Hub. You'll need to specify:

Option	Description
Docker image name	Name of the Docker image, similarly to the one used with the `docker run` command (for example, `jetbrains/teamcity-agent`).
Image pull policy	The policy of updating the image.
Command	(Optional) Set the command run by the container.
Command arguments	(Optional) Set the command arguments.

Use custom pod template

Enter your own pod template in a YAML format. Use this option when you need to specify additional configuration for a launched pod, such as resources' requests/limits or credentials. This option is alternative to the template from deployment specification and allows editing the configuration directly in the TeamCity UI/DSL.

Example of a simple template:

apiVersion: v1
kind: Pod
metadata:
  labels:
    app: teamcity-agent
spec:
  restartPolicy: Never
  containers:
    - name: teamcity-agent
      image: jetbrains/teamcity-agent
      resources:
        limits:
          memory: "2Gi"

Kotlin DSL

To create and setup k8s profiles and images, create KubernetesCloudProfile and KubernetesCloudImage class instances under the project's features group. Set an image profileId property to the profile id value to assign this image to the target profile.

project {
    // ...
    features {
        // ...
        kubernetesCloudProfile {
            id = "kube-1"
            name = "K8S Agents"
            description = "EKS"
            serverURL = "https://myteamcityserver.com"
            terminateIdleMinutes = 30
            apiServerURL = "https://123.gr7.eu-west-1.eks.amazonaws.com"
            caCertData = "abc"
            authStrategy = eks {
                accessId = "aws_access_id"
                secretKey = "aws_key"
                clusterName = "my-k8s-cluster"
            }
        }
        kubernetesCloudImage {
            id = "PROJECT_EXT_10"
            profileId = "kube-1"
            agentPoolId = "21"
            agentNamePrefix = "k8s-singlec"
            maxInstancesCount = 5
            podSpecification = runContainer {
                dockerImage = "jetbrains/teamcity-agent"
            }
        }
        kubernetesCloudImage {
            id = "PROJECT_EXT_11"
            profileId = "kube-1"
            agentPoolId = "21"
            agentNamePrefix = "k8s-pspec"
            maxInstancesCount = 5
            podSpecification = customTemplate {
                customPod = """
                    apiVersion: v1
                    kind: Pod
                    metadata:
                      labels:
                        app: teamcity-agent
                    spec:
                      restartPolicy: Never
                      containers:
                        - name: teamcity-agent
                          image: jetbrains/teamcity-agent
                          resources:
                            limits:
                              memory: "2Gi"
                """.trimIndent()
            }
        }
    }
}

See KubernetesCloudProfile and KubernetesCloudImage for more examples and API descriptions.

Last modified: 18 September 2024