Share Files Between Steps

Sometimes, you might need to pass files from one step to another within a single job. For example, tests executed in a separate container might require files generated during the build in another container. For this purpose, Automation provides a file share: separate external storage mounted to a host machine. Unlike caching, file sharing doesn't use a permanent file repository – the share exists only during the job lifetime.

If a host machine is a container, by default, the file share is mounted at $mountDir/share. Learn more

If a host machine is a self-hosted worker, by default, the file share is mounted at jetbrains/space/automation/worker/data/{temp-dir}/share. Learn more

To get the path to the file share, use the JB_SPACE_FILE_SHARE_PATH environment variable.

You can work with the file share directory directly using system commands. In the following example, the first container creates file.txt in the file share, and the second container shows the file share contents.

job("Example") {
    container(displayName = "Create file", image = "ubuntu") {
        shellScript {
            content = "touch ${'$'}JB_SPACE_FILE_SHARE_PATH/file.txt"
        }
    }

    container(displayName = "Show file", image = "ubuntu") {
        shellScript {
            content = "ls -la ${'$'}JB_SPACE_FILE_SHARE_PATH"
        }
    }
}

Alternatively, there's a more high-level approach of accessing the file share: If your step runs Kotlin code in the kotlinScript block, you can use the special file-sharing API.

// java.io package provides functions for working with files
import java.io.*

job("Share files") {
    container(displayName = "Create file", image = "amazoncorretto:17-alpine") {
        kotlinScript { api ->
            // create a file in the user's directory
            val path = "~/file.txt"
            val content = "Hello World!"
            val sharedFile = File(path).apply {
                parentFile.mkdirs()
                createNewFile()
                writeText(content)
            }
            // copy the file to the file share
            api.fileShare().put(sharedFile, "file.txt")
        }
    }

    container(displayName = "Show file", image = "amazoncorretto:17-alpine") {
        kotlinScript { api ->
            // Get the file from the file share and print file contents.
            // Because the steps (containers) run sequentially,
            // file.txt is guaranteed to be available in the file share.
            api.fileShare().locate("file.txt")?.let {
                println(it.readText())
            }
        }
    }
}

The maximum capacity of the file share is limited to 2 GB per job. Let's use an example to see how this limit works:

A job consists of two steps (i.e. containers): A and B.
The steps run sequentially one after another.
None of the steps is allowed to allocate more than 2 GB in the file share.
Step A allocates 1.5 GB.
Let's suppose step B allocates 1 GB (the step is allowed to do this as it is less than the 2 GB limit). The size of the file share is 2.5 GB now.
If at the end of step B, the file share size is still 2.5 GB, the job will fail.
If step B cleans the file share so that its size doesn't exceed the 2 GB limit, the job will finish successfully.

You can remove files from the file share either directly (using container commands) or using the API: fileShare().remove().

Automation calculates file share usage twice for each step: First, when a file share volume is mounted to a step, and, for the second time, when the step finishes working (Automation calculates the size of new data allocated by the step). The monthly limit on file share usage is 200 GB for all subscription plans. Automation will warn you once the file share usage limit is reached.

Let's use an example to see how this limit works:

A job consists of three steps (i.e. containers): A, B, and C.
The steps run sequentially one after another.
Step A allocates 1 GB in the file share. Now, the file share size is 1 GB, file share usage is 1 GB.
Automation starts step B and mounts the file share to the step container. Even if step B does nothing with the file share, this is treated as read access to the file share. Now, the file share size is 1 GB, file share usage is 2 GB.
Step B allocates 0.5 GB in addition to the existing data. Now, the file share size is 1.5 GB, file share usage is 2.5 GB.
Step C gets the file share and does not put any new data to it. Now, the file share size is 1.5 GB, file share usage is 5 GB.
Job finishes its work with the resulting 5 GB file share usage. This means that you can run such a job only 40 times a month.

By default, steps in a job run sequentially: the next container starts only after the previous container finishes its work. This prevents conflicts in the file share. But what if two steps (A and B) run in parallel and put the same file in the file share? This behavior is undetermined: the file share will contain the result of either A or B. Consider an example:

job("Example") {
    parallel {
        container(displayName = "Write to file", image = "ubuntu") {
            shellScript {
                content = "echo Hello > /mnt/space/share/file.txt"
            }
        }

        container(displayName = "Write to same file", image = "ubuntu") {
            shellScript {
                content = "echo World > /mnt/space/share/file.txt"
            }
        }
    }
}

Both containers run in parallel: the first container writes Hello to file.txt while the second one writes World to the same file. When the job finishes, file.txt will contain either Hello or World but neither Hello World nor World Hello.

File attributes are preserved when a file is copied to/from the file share.

The fileShare() function provides access to the file sharing API:

Method or Property	Description
`location: Path`	(Read-only) Path to the file share (by default, `/mnt/space/share`).
`put(file: File): Path`	Copies `file` to the file share. Using `relativePath`, you can set a new file name and specify its relative path inside the file share (`/mnt/space/share`). Returns full path to the file.
`put(file: File, relativePath: String): Path`
`locate(relativePath: String): File?`	Locates a file in the file share by its `relativePath`. Returns file path.
`remove(relativePath: String): Boolean`	Deletes a file from the file share by its `relativePath`. Returns `true` if the file was successfully deleted.

Last modified: 15 December 2023

Share Files Between Steps

Access file share directly

Access file share in Kotlin code

File share size limit

Monthly limit on file share usage

Parallel steps and file sharing

File attributes and file sharing

File-sharing API