JetBrains Space Help

Self-Hosted Workers

A self-hosted worker is a lightweight agent that you can run on your own machine on Windows, Linux, or macOS. The self-hosted worker connects to Space Automation, gets jobs and the project source code, runs the jobs, and reports results back to Space.

Self-hosted workers let you perform CI/CD workflows that are not possible with containers. For example:

  • Building full .NET Framework applications on Windows.

  • Using specific hardware. For example, building iOS and macOS applications on Apple hardware, running ML tasks on GPUs, and so on.

  • Using more powerful hardware configurations than those provided by Space Cloud.

How does it work

  • The self-hosted worker agent is a Java application that is distributed either as a .zip archive or as a part of a specially-prepared Docker image. Learn more

  • After you register a self-hosted worker in Space, you should run the agent on the host machine. The running agent will regularly send its status to Space.

  • The agent uses pull communication with Space: The agent initiates the communication, and Space only responds to incoming requests. This means that no specific network configuration is required on the agent side. The only requirement is access to your Space instance.

  • To run a job on the self-hosted worker, you should use the host blocks for job steps:

    job("Hello from self-hosted worker") { host("Run echo") { shellScript { content = """ echo "Hello World!" """ } } }

    Important: In order Space could route this job to a self-hosted worker, this job must be assigned to the Self-Hosted Workers worker pool. This could be done either with the help of the Default worker pool parameter or explicitly with the job's requirements block. For example:

    job("Hello from self-hosted worker") { requirements { workerPool = WorkerPools.SELF_HOSTED } host("Run echo") { // ... } }

    Learn more about how to specify job run environment.

  • When a job with at least one host step is triggered, Automation will randomly choose a worker suitable for the job. After the suitable worker is found, Automation initiates job run.

  • The worker agent:

    1. creates a temporary directory on the host machine,

    2. downloads the project source code including the Automation script,

    3. and runs the job.

    Once the job is finished, the agent reports results to Automation and deletes the temporary directory.

  • Self-hosted workers are billed for each minute of work based on your Space subscription plan. Learn more

Requirements to a host machine

The host machine which you want to use as a self-hosted worker must meet the following requirements:

  1. Installed software:

    • Git: any version.

    • Docker, Docker Compose 1.27.0 or later: only if you want to run container steps.

    • JRE 11 or later: only if you want to run Kotlin code using the kotlinScript blocks.

    Important: The paths to Git, Docker, and Java must be specified in the system PATH.

  2. The outbound connections to *.jetbrains.space and download.jetbrains.com must be allowed for the host machine.

Directory structure and working directory

Automation stores project data inside a temporary directory on this worker.

/ //system temp dir, e.g. /tmp on Linux ├─── jetbrains/space/automation/worker/data // step data directory │ └─── steps-{randomId} // auto-generated directory │ └─── step-data-{randomStepId} // auto-generated directory │ ├─── share // file share │ ├─── system // non-user system files │ └─── work │ ├─── {git-repo-name} // cloned git repository (working directory) │ ├─── ... // other project repos (only if specified in the script) │ └─── ... ...

In more detail:

  • {system-temp-dir}/jetbrains/space/automation/worker/data is the default parent directory for step data.

    To change the default path, use the --dataDir agent argument.

    To get this path in a shellScript or kotlinScript, use the JB_SPACE_STEP_DATA_PATH environment variable.

  • .../work/{git-repo-name} is the default location of the project sources. Here {git-repo-name} stands for the project's Git repository name. For details on how to check out additional project repositories, refer to Check out Source Code.

    To get this path in a shellScript or kotlinScript, use the JB_SPACE_WORK_DIR_PATH environment variable.

  • .../share is the external storage used for file sharing.

    To get this path in a shellScript or kotlinScript, use the JB_SPACE_FILE_SHARE_PATH environment variable.

  • After the job is finished, the step data directory is deleted.

What can you run in a self-hosted worker

As well as containers, self-hosted workers let you run shell scripts (shellScript) and arbitrary Kotlin code (kotlinScript).

Note that unlike container, host lets you run both shellScript and kotlinScript in one block. For example:

job("This job works") { host("Run both kotlin and shell scripts") { kotlinScript { api -> // Do smth. } shellScript { content = """ echo Do smth. """ } requirements { type = OSType.Linux } } }

Add self-hosted workers

  1. In Administration, open the Automation page.

  2. Click Add worker.

  3. Specify worker Name and other options:

    • Tags: route jobs to specific workers. Automation will run jobs marked with a specific tag only on workers with the same tag. Learn more

    • Projects:

      • All projects: the worker will be available to all projects in the organization. To choose this option, you must have the Create organization automation workers permission.

      • Specific projects: the worker will be available only to the selected projects. You are able to select only the projects where you have the Project Admin role.

    After you click Add worker, you will be navigated to the worker settings page. The settings include info about the assigned tags, OS, available resources, and an authorization token. Copy the token to the safe place (the token is shown only once right after the worker is created).

  4. Your next task is to run the worker agent on a host machine. Before it happens the worker will be in not initialized state. The instructions on how to do it are provided on the worker settings page under Worker installation instructions: select a worker type and follow the instructions:

    • Docker for Windows / Linux / macOS: the worker runs inside a Docker container running on a host machine.

      For example, to run the worker in a container on a Linux host machine:

      docker run \ --log-opt max-size=10m \ -v /var/run/docker.sock:/var/run/docker.sock \ -v /tmp/jetbrains/space/automation/worker:/tmp/jetbrains/space/automation/worker \ -e SPACE_WORKER_SERVERURL=https://mycompany.jetbrains.space \ -e SPACE_WORKER_TOKEN=abc1234 \ public.registry.jetbrains.space/p/space/containers/space-automation-worker:139318
      Change working directory in a container

      When running inside a container, the worker's working directory is a directory on the host machine mounted inside this container (see the second -v argument):

      docker run \ --log-opt max-size=10m \ -v /var/run/docker.sock:/var/run/docker.sock \ -v /tmp/jetbrains/space/automation/worker:/tmp/jetbrains/space/automation/worker \ -e SPACE_WORKER_SERVERURL=https://mycompany.jetbrains.space \ -e SPACE_WORKER_TOKEN=abc1234 \ public.registry.jetbrains.space/p/space/containers/space-automation-worker:139318

      If you want to change the path to the working directory in the container, you must also use the same path for the mounted directory on the host machine. It's also required to inform the worker about the new path using the SPACE_WORKER_DATADIR environment variable. For example:

      docker run \ --log-opt max-size=10m \ -v /var/run/docker.sock:/var/run/docker.sock \ -v /tmp/newdir/worker:/tmp/newdir/worker \ -e SPACE_WORKER_DATADIR=/tmp/newdir/worker \ -e SPACE_WORKER_SERVERURL=https://mycompany.jetbrains.space \ -e SPACE_WORKER_TOKEN=abc1234 \ public.registry.jetbrains.space/p/space/containers/space-automation-worker:139318

    • Binary for Windows / Linux / macOS: the worker runs on a virtual or hardware machine.

      For example, to run the worker on a Linux host machine:

      unzip ./space-automation-worker.zip && \ chmod +x ./worker.sh && \ ./worker.sh start \ --serverUrl https://mycompany.jetbrains.space \ --token abc1234
      Verify worker binary

      You can verify the authenticity of the downloaded binary by checking a file checksum and a checksum signature.

      Prerequisites

      • gpg (version 2.x) is installed on the machine. You can download the tool from https://gnupg.org/download/

      • The path to gpg must be in the system PATH.

      Instructions

      1. Download the checksum and signature files using the links on the worker's page. Save the files in the same directory with the worker .zip file. The directory must contain the following files:

        • space-automation-worker-windows.zip: the archive with the worker.

        • space-automation-worker-windows.zip.sha256: a SHA256 checksum of the archive file.

        • space-automation-worker-windows.zip.sha256.asc: a signature for the checksum file.

      2. Download the KEYS public key file from https://download.jetbrains.com/KEYS and run:

        gpg --import KEYS

        If you haven't done this before, the output will look similar to this:

        gpg: key 1F7A8F87B9D8F501: 1 signature not checked due to a missing key gpg: /Users/you/.gnupg/trustdb.gpg: trustdb created gpg: key 1F7A8F87B9D8F501: public key "Download <download@jetbrains.com>" imported gpg: Total number processed: 1 gpg: imported: 1 gpg: no ultimately trusted keys found

        Make sure the key ID in the output is 1F7A8F87B9D8F501.

      3. Verify the signature by running:

        gpg --verify space-automation-worker-windows.zip.sha256.asc space-automation-worker-windows.zip.sha256

        Make sure the output contains Good signature from "Download <download@jetbrains.com>"

      4. Check that the ID of the public key is 33FD4BFD33554634053D73C0C2148900BCD3C2AF by running:

        gpg --keyid-format long --list-keys --with-fingerprint 0x33FD4BFD33554634053D73C0C2148900BCD3C2AF

        Make sure the key ID in the output is 1F7A8F87B9D8F501. This guarantees that the downloaded sha256 file is valid.

      5. Verify the checksum of the .zip archive using the sha256 file:

        certUtil -hashfile space-automation-worker-windows.zip SHA256

        The output must look similar to this:

        SHA256 hash of space-automation-worker-windows.zip: cd5b12d9a8c74accade22f808e5ea18b2a018706d4112d649e1adfdd26187c7d CertUtil: -hashfile command completed successfully.

        Make sure the hash value in the output is equal to the value in the space-automation-worker-windows.zip.sha256 file.

      Prerequisites

      sha256sum and gpg (version 2.x) are installed on the machine. To install the tools, run

      • Debian-based distributions: apt install coreutils gpg

      • RedHat-based distributions: yum install coreutils gpg or dnf install coreutils gpg

      Instructions

      1. Download the checksum and signature files using the links on the worker's page. Save the files in the same directory with the worker .zip file. The directory must contain the following files:

        • space-automation-worker-linux.zip: the archive with the worker.

        • space-automation-worker-linux.zip.sha256: a SHA256 checksum of the archive file.

        • space-automation-worker-linux.zip.sha256.asc: a signature for the checksum file.

      2. Download the KEYS public key file from https://download.jetbrains.com/KEYS and run:

        LANG=C gpg --import KEYS

        If you haven't done this before, the output will look similar to this:

        gpg: key 1F7A8F87B9D8F501: 1 signature not checked due to a missing key gpg: /Users/you/.gnupg/trustdb.gpg: trustdb created gpg: key 1F7A8F87B9D8F501: public key "Download <download@jetbrains.com>" imported gpg: Total number processed: 1 gpg: imported: 1 gpg: no ultimately trusted keys found

        Make sure the key ID in the output is 1F7A8F87B9D8F501.

      3. Verify the signature by running:

        LANG=C gpg --verify space-automation-worker-linux.zip.sha256.asc space-automation-worker-linux.zip.sha256

        Make sure the output contains Good signature from "Download <download@jetbrains.com>"

      4. Check that the ID of the public key is 33FD4BFD33554634053D73C0C2148900BCD3C2AF by running:

        gpg --keyid-format long --list-keys --with-fingerprint 0x33FD4BFD33554634053D73C0C2148900BCD3C2AF

        Make sure the key ID in the output is 1F7A8F87B9D8F501. This guarantees that the downloaded sha256 file is valid.

      5. Verify the checksum of the .zip archive using the sha256 file:

        sha256sum -c space-automation-worker-linux.zip.sha256

        The output must look as follows:

        space-automation-worker-linux.zip: OK

      Prerequisites

      sha256sum and gpg (version 2.x) are installed on the machine. You can install the tools, for example, with Homebrew: brew install coreutils gpg

      Instructions

      1. Download the checksum and signature files using the links on the worker's page. Save the files in the same directory with the worker .zip file. The directory must contain the following files:

        • space-automation-worker-mac.zip: the archive with the worker.

        • space-automation-worker-mac.zip.sha256: a SHA256 checksum of the archive file.

        • space-automation-worker-mac.zip.sha256.asc: a signature for the checksum file.

      2. Download the KEYS public key file from https://download.jetbrains.com/KEYS and run:

        LANG=C gpg --import KEYS

        If you haven't done this before, the output will look similar to this:

        gpg: key 1F7A8F87B9D8F501: 1 signature not checked due to a missing key gpg: /Users/you/.gnupg/trustdb.gpg: trustdb created gpg: key 1F7A8F87B9D8F501: public key "Download <download@jetbrains.com>" imported gpg: Total number processed: 1 gpg: imported: 1 gpg: no ultimately trusted keys found

        Make sure the key ID in the output is 1F7A8F87B9D8F501.

      3. Verify the signature by running:

        LANG=C gpg --verify space-automation-worker-mac.zip.sha256.asc space-automation-worker-mac.zip.sha256

        Make sure the output contains Good signature from "Download <download@jetbrains.com>"

      4. Check that the ID of the public key is 33FD4BFD33554634053D73C0C2148900BCD3C2AF by running:

        gpg --keyid-format long --list-keys --with-fingerprint 0x33FD4BFD33554634053D73C0C2148900BCD3C2AF

        Make sure the key ID in the output is 1F7A8F87B9D8F501. This guarantees that the downloaded sha256 file is valid.

      5. Verify the checksum of the .zip archive using the sha256 file:

        sha256sum -c space-automation-worker-mac.zip.sha256

        The output must look as follows:

        space-automation-worker-mac.zip: OK

    Available worker script arguments (note that you can set/get them also as environment variables):

    Argument

    Env variable

    Description

    --serverUrl

    SPACE_WORKER_SERVERURL

    A URL of your Space instance.

    --token

    SPACE_WORKER_TOKEN

    An authorization token issued to the worker. The token is shown only once during the worker registration. If you lose it, you will have to generate a new one.

    --dataDir

    SPACE_WORKER_DATADIR

    Sets the path to the directory where the script will download all required data: project sources, file share, and Automation-specific data.

    --hostname

    SPACE_WORKER_HOSTNAME

    Specifies a host name of the self-hosted worker. It is used mostly for troubleshooting purposes to help you distinguish a worker by its host name.

    --cpu, --mem

    SPACE_WORKER_CPU, SPACE_WORKER_MEM

    Specifies how much CPU and RAM are available on the host machine. These parameters do not anyhow limit how many resources the agent is allowed to consume. They are used only to help Automation select a suitable worker for a job based on the amount of available resources. Learn more

    After you start a worker, its status must change to healthy:

    Healthy

Use self-hosted workers to run steps

In order for Automation to run a job on a worker, the job must contain at least one host step.

job("Build solution") { host("Run msbuild") { shellScript { interpreter = "cmd" content = """ C:\"Program Files (x86)"\MSBuild\14.0\Bin\MsBuild.exe MySolution.sln """ } requirements { type = OSType.Windows } } }

Inside the host step you can use almost the same DSL elements as inside the container step. The main difference is that inside host, you can also:

  • Use the requirements block that lets you specify requirements for the worker where the job must run. Learn more

  • Use a number of shellScript elements. Each of the scripts will be shown separately in the job logs.

  • Use both shellScript and kotlinScript in one host block.

  • Access a number of worker-specific environment variables.

Choose specific workers for a job

You have a number of ways to route specific jobs to specific workers: by tags, by operating system, and by available system resources. To specify the routing condition, use the requirements block inside host. You can specify all three conditions inside requirements.

If Automation cannot find a suitable worker for a job, the job will switch to the Blocked state.

No self-hosted workers

Route jobs by tags

Tagging is one of the ways to route a job to a specific worker: Both the worker and the job must be marked with the same tag. See the details below.

  1. In Administration | Automation, choose the required worker.

  2. Click Edit.

  3. In Tags, specify a unique tag name.

    Self-hosted worker settings

  4. Open the required .space.kts file and specify the same tag in the requirements block of a particular host. For example:

    job("Hello from host") { host("Run echo") { shellScript { content = """ echo Hello World! """ } requirements { workerTags("Pool1") } } }

Route jobs by operating system

The worker agent automatically reports the operating system of its host machine to Space. To choose the suitable worker based on the operating systems, specify the required OS in the script:

requirements { os { // the possible values are // Windows, macOS, Linux type = OSType.Windows } }

Route jobs by available resources

The worker agent automatically reports the system resources (CPU cores and RAM) of its host machine to Space. To choose the suitable worker based on host machine resources, specify the required resources in the script:

requirements { resources { // the job can run on any worker that // has 1 CPU core and 1GB RAM (or more) minCpu = 1.cpu minMemory = 1.gb } }

Note that the worker agent does not guarantee that all these resources will be available to the job: It knows nothing about how the resources are consumed by other processes running on the host machine.

You can manually set the available system resources for a particular worker. This might help, for example, if you don't want to run "heavy" jobs on a particular worker. To set the available resources, run the worker agent with --cpu (set in mCPU) and --mem (set in MB) arguments. For example:

./worker.sh start \ --serverUrl https://mycompany.jetbrains.space \ --token abc1234 \ --cpu 2000 \ --mem 8000

In the example above, regardless of real system resources, the worker will report to Space that the system has 2 CPU cores and 8GB RAM available: if a host in a job requires more resources, the worker will be considered as not suitable for this job. Note that these values do not anyhow prevent the worker agent from consuming more resources on the machine: They just help Space to choose the most suitable worker for a job.

Run container steps in self-hosted workers

Self-hosted workers let you run not only host steps but container steps as well. To be able to run container steps, a worker must have Docker and Docker Compose installed.

Automation will try to run a job with container steps on a self-hosted worker in one of the following cases:

  • A job has both container and host steps.

  • A container step has the requirements section. Note that you can route container steps to particular workers only by using workerTags.

  • A job has only container steps (with no requirements) and the Default worker pool parameter is set to Self-Hosted Workers.

For example:

job("Example") { container(image = "amazoncorretto:17-alpine") { // ... requirements { workerTags("Pool 1") } } }

Learn more

Suspend a worker

You can temporarily suspend a worker: The worker agent keeps running on the host machine, but Automation will not route jobs to the suspended worker. If the worker still has running jobs, it will first finish the jobs and only then switch to the suspended state.

To suspend a worker

  1. Find the required worker in Administration | Automation.

  2. In the worker settings, click Suspend.

To activate the worker back

  1. Find the required worker in Administration | Automation.

  2. In the worker settings, click Activate.

Delete a worker

  1. Find the required worker in Administration | Automation.

  2. In the worker settings, click Delete.

Note that deleting a worker is an instant operation: If the worker still has running jobs, the jobs will be terminated.

View the history of executed steps

All steps executed by a particular self-hosted worker are saved in the worker history.

  1. Find the required worker in Administration | Automation.

  2. In the worker settings, open the Activity tab.

Worker states

State

Description

not initialized

The worker has never connected to Space before: This is the initial worker state after you register it in Space

healthy

The worker is up and running

unhealthy

The worker is low on disk space: Automation will not route jobs to this worker

suspended

The worker was manually suspended

graceful shutdown

The worker is finishing the currently running jobs, then the worker agent process will be terminated. This happens when you terminate the agent process with SIGTERM (for example, when you kill the process)

disconnected

Automation has no connection with the worker. This might happen, for example, if the worker agent is not running on the host machine, the machine is turned off or has no Internet connection

Last modified: 15 December 2023
Service Containers Space Cloud Workers