Cache Files
File caching may improve build times and save resources. It works by storing project dependencies locally in a Space file repository. This way, when the same build steps are run again, the referenced packages can be quickly retrieved from the cache instead of being downloaded from a remote server.
note
This feature must be used only for caching project dependencies. If you want to reuse build artifacts (e.g., compiled binaries), refer to Store File Artifacts.
Projects in Space can have associated default file repositories for storing cached files for the subsequent job runs.
Each project has its own default file repository for cache with corresponding access restrictions (only for project members).
The repository is created automatically when you run the job that references the default repository (e.g., uploads cached files) for the first time.
In a script, you can get the name of the default file repository from the
run:file-caches.default-repository
parameter.You can replace the default file repository with any other file repository in Packages.
The default name of the default repository is default-automation-caches. You can change this name in Jobs | Settings under File Storage | Cache Repository.
By default, if you upload a file that already exists in the default repository, the file is overwritten. Learn more
By default, cache files are deleted if they were not accessed in 7 days. You can change the default retention policy in the repository settings.
To work with file cache, depending on your run environment, use either the job.container.cache
or the job.host.cache
block. For example, to cache dependencies for an npm project:
job("Build frontend") {
container(displayName = "Build npm", image = "node") {
cache {
// To upload to another repo (e.g., 'my-file-repo'), uncomment the next line
// location = CacheLocation.FileRepository(name = "my-file-repo", remoteBasePath = "caches/{{ run:job.repository }}")
// Generate cache file name
// Using a hash of the build file ensures all job runs with the
// same package.json will share the cached dependencies
storeKey = "npm-{{ hashFiles('package.json') }}"
// Fallback option
// If the right cache file is not found, get cache from 'npm-master.tar.gz'
restoreKeys {
+"npm-master"
}
// Local path to the cache file directory
localPath = "node_modules"
}
shellScript {
content = "npm install"
}
}
}
The default cache location
is caches/{{ run:job.repository }}
directory in the default cache repository. Note how the default location is defined on the repository level – by default, all jobs in the same repository reuse the same cache keys. You can override the default location using the location
parameter.
On job run, Automation checks whether the {storeKey}.tar.gz
archive exists in the default location or in the user-specified location
:
If the file exists, Automation:
Downloads the file and extracts it to the
localPath
.Runs the script.
If the file doesn't exist, Automation:
Checks
location
(or the default repository) for archives specified inrestoreKeys
.If a fallback cache archive is found, extracts it to
localPath
and runs the script.If a fallback cache archive is not found, Automation:
Runs the script.
Creates a .tar.gz archive with the directory specified in
localPath
. As a file name, Automation uses thestoreKey
string.Uploads the .tar.gz archive to the default cache repository or to the repository specified in
location
.
If you don't specify location
, Automation uploads the caches .tar.gz archive to the caches/{{ run:job.repository }}
directory of the default default-automation-caches repository.
The cache file name is defined by the storeKey
parameter value: {storeKey}.tar.gz
. The name must be unique and must be unambiguously determined by the required project dependencies. For example, in a Node.js project, dependencies are specified in the packages.json
file, so it makes sense to generate the storeKey
based on the contents of packages.json
. To help you solve this task, Automation provides the hashFiles(files: Collection<String>): String
function. This function generates hash based on the contents of a file or number of files. To call the function, you must use the parameter syntax (double curly braces).
For example: storeKey = "npm-{{ hashFiles('package.json') }}"
or storeKey = "cache_{{ hashFiles('server/build.gradle.kts', 'client/build.gradle.kts') }}"
.
localPath
defines the location of the directory to be cached. You can specify it with:
An absolute path, for example,
/home/root/.m2
.A path relative to the current home directory, for example,
~/.m2
. You can use it only injob.host
steps. Learn moreA path relative to the step working directory (by default, the project root), for example,
node_modules
.
The default cache location is defined on the Git repository level as it uses a relative path which includes the repository name: caches/{{ run:job.repository }}
.
If you want to share caches between Git repositories in the project, use the location
parameter to specify an absolute path to caches. For example:
cache {
location = CacheLocation.FileRepository(name = CacheLocation.DefaultRepositoryName, remoteBasePath = "my-shared-caches")
storeKey = "npm-{{ hashFiles('package.json') }}"
localPath = "node_modules"
}
To cache multiple directories, you can create multiple caching rules. For example:
job("Example") {
container(displayName = "Get caches from two dirs", image = "ubuntu") {
cache {
// all files and dirs from caches/dir1
localPath = "caches/dir1"
// other cache settings
}
cache {
// all files and dirs from caches/dir2
localPath = "caches/dir2"
// other cache settings
}
// here goes the script...
}
}
If you run a step in a self-hosted or a cloud machine using job.host
, you can simply specify the required absolute path to a directory. For example, localPath = "~/.m2/repository"
.
Things are different if you run a step in a container using job.container
. Here you can reference files only under the mount directory, which is, /mnt/space
by default. Therefore, if you want to cache directories outside /mnt/space
, you must change the default mount directory to a particular home directory. As most of the containers used for CI/CD run under the root user, this will be the root
directory. For example:
job("Build backend") {
container(displayName = "Maven build", image = "maven") {
// Change mount directory
// This will overwrite all files in this dir in the image!
mountDir = "/root"
cache {
storeKey = "maven-{{ hashFiles('pom.xml') }}"
// now, we can reference files and dirs
// inside /root/...
localPath = "/root/.m2/repository"
}
shellScript {
content = "mvn install"
}
}
}
If Automation cannot find a {storeKey}.tar.gz
archive, it looks for files specified in the restoreKeys
list. We recommend creating such a fallback cache of project-wide dependencies whose versions change relatively rarely.
restoreKeys {
+"npm-base-frontend"
+"npm-base"
}
File repositories in Space have the Immutable files option. If it's enabled, the repository prohibits uploading a file with the same path more than once. If it's disabled, files can be overwritten. By default, this option is disabled for the default file repository, so files will be updated when changed.
To upload caches to a repository different from the default one, use the cache.location
parameter. Note that this repository must belong to the same project where you run the job.
// to upload cache files to 'my-file-repo'
cache {
location = CacheLocation.FileRepository(name = "my-file-repo", remoteBasePath = CacheLocation.DefaultRemoteBasePath)
storeKey = "npm-{{ dashify('run:git-checkout.ref') }}"
restoreKeys {
+"npm-refs-heads-master"
}
localPath = "node_modules"
}
To upload caches to a file repository created in another project, you should first attach this repository to your current project.
In many scenarios, re-uploading a changed file to the cache is unnecessary. For example, a JavaScript project may change files in the node_modules
directory on each npm install
, even though no dependencies were changed.
To prevent unnecessary cache uploads in such cases, use the cache.reuploadWhenFilesChange
parameter. For example:
cache {
// do not upload changed files to cache
// only newly created files will be uploaded
reuploadWhenFilesChange = false
storeKey = "npm-{{ hashFiles('package.json') }}"
localPath = "node_modules"
}
Open the desired project.
On the sidebar menu, choose Packages.
Open the file repository that contains project caches. In case of the default file repository, that would be default-automation-caches.
Thanks for your feedback!