Configure agents

You can change the default agents configuration by mounting a config file into /opt/datalore/configs/agents-config.yaml inside the Datalore container. This config has the following format:

        docker:
          network: <name-of-agents-network>
          dataloreHost: <host-of-datalore-server>
          instances:
            - <instance1-definition>
            - <instance2-definition>
              ...
            - <instanceN-definition>
    

By default, <name-of-agents-network> is datalore-agents-network, and <host-of-datalore-server is datalore.

Each instance definition has the following fields:

Mandatory fields
`id: basic-agent`	Specifies the unique id for the instance.
`default: true`	Specifies whether the instance is default. Make sure exactly one instance has this parameter set to `true`.
`label: "docker-base"`	Specifies the instance name shown in the UI.
`image: docker.io/jetbrains/datalore-agent:2024.5`	URI of the agent docker image, changed only to specify a custom agent image
Optional fields
`volumes: array of strings`	Specifies the local storages mounted to the agent. Find more details in the chapter below.
`pullPolicy: "IF_ABSENT"`	Specifies when the image will be pulled. Acceptable values are: NEVER, ALWAYS, IF_ABSENT (default).
`cpus: 1.5`	Specifies how much of the available CPU resources a container can use. For example, if the host machine has two CPUs and you set `cpus` to 1.5, the container is guaranteed at most one and a half of the CPUs.
`memory: 4000000000`	Specifies the maximum amount of memory the container can use. The following suffixes are supported: K, M, G, T, Ki, Mi, Gi, Ti. Bytes are assumed when no suffix is specified.
env: - <env1-definition> - <env2-definition> ... - <envN-definition>	Defines environment variables on the agent. Each `env` definition contains a name and a value. Use `DATALORE_SKIP_WARMUP` to skip analysis warmup on agent initialization, which contributes to faster initialization at the cost of slower code completion for some time. Use `DATALORE_MAX_BLOB_SIZE` for every instance to customize output limit, specified in bytes. Example: env: - name: DATALORE_SKIP_WARMUP value: true - name: DATALORE_MAX_BLOB_SIZE value: 10_000_000
`initialPoolSize: 1`	Specifies the number of pre-warmed (pooled) agents. This is the minimum number of agents Datalore will keep started and yet unassigned to any computation. When a notebook is opened, Datalore assigns one of the pooled agents to that notebook and starts another agent to refill the pool. It might be useful to set `initialPoolSize` to a non-zero value if initialization of your agents takes a lot of time.
Options below are only used to adjust UI appearance, not to specify agent properties. To actually set limits for an agent, use the `cpus` and `memory` fields.
`shortName: "S"`	Specifies the short instance name. If it contains more than three letters, only the first three are displayed.
`numCPUs: 2`	Specifies the number of CPUs to display.
`cpuMemoryText: "8 GB"`	Specifies the agent RAM description.
`numGPUs: 1`	Specifies the number of GPUs to display.
`gpuMemoryText: "4 GB"`	Specifies the GPU memory description.
features: - "1 vCPU" - "16 GB RAM" - "1 NVIDIA GPU"	Specifies specific properties of this agent to display in the UI. Only properties containing the "RAM", "CPU", or "GPU"* substrings will be displayed in the agent selection window.*

Configure local storages for agents

You can mount a local server storage to your Datalore agents by using the volumes field for the respective instance definition. See the example below:

            docker:
                instances:
                  - id: docker-base
                    label: "docker-base"
                    description: "docker-base"
                    volumes:
                      - "/tmp/agent:/tmp"
        

The volume field contains an array of strings with the following format:

host-src:container-dest[:options] to bind-mount a host path into the container. Make sure both host-src and container-dest are an absolute path.
volume-name:container-dest[:options] to bind-mount a volume managed by a volume driver into the container. Make sure container-dest is an absolute path.

options is an optional, comma-delimited list of the following:

nocopy disables automatic copying of data from the container path to the volume. The nocopy flag only applies to named volumes.
[ro|rw] mounts a volume read-only or read-write respectively. If omitted or set to rw, volumes are mounted read-write.
[z|Z] applies SELinux labels to allow or deny multiple containers to read and write to the same volume.
- z: a shared content label is applied to the content. This label indicates that multiple containers can share the volume content, for both reading and writing.
- Z: a private unshared label is applied to the content. This label indicates that only the current container can use a private volume. Labeling systems such as SELinux require proper labels to be placed on volume content that is mounted into a container. Without a label, the security system can prevent a container's processes from using the content. By default, the labels set by the host operating system are not modified.
[[r]shared|[r]slave|[r]private] specifies mount propagation behavior. This only applies to bind-mounted volumes, not internal volumes or named volumes. Mount propagation requires the source mount point (the location where the source directory is mounted in the host operating system) to have the correct propagation properties. For shared volumes, the source mount point must be set to shared. For slave volumes, the mount must be set to either shared or slave.

Configure external agents

The procedure above is also applicable for configuring external agents. For more details, refer to this topic.

Last modified: 25 November 2024