Configure agents

Last modified: 28 August 2023

You can change the default agents configuration in the datalore.values.yaml file.

Configure agents

  1. Specify configuration parameters for your agents.

    Under the agentsConfig key, fill in the fields to define agent types (such as Basic and Large machines in the cloud version of Datalore). It has the following schema:

    agentsConfig:
  k8s:
    namespace: <(Optional) namespace identifier>
    instances:
      - id: <Any string used as a unique identifier of this particular instance>
        default: <Whether this instance is used by default>
        label: <Instance name>
        yaml:
          <Kubernetes config of Pod to be used for the instance>
        initialPoolSize: <(Optional) Number of pre-warmed agents to keep>
      - id: <Another type with the same schema as above>
        ...

    If you have Datalore installed in a namespace other than default, you must specify it under the namespace field.

    Each instance definition has the following fields:

    Mandatory fields

    id: basic-agent

    Specifies the unique id for the instance.

    default: true

    Specifies whether the instance is default. Make sure exactly one instance has this parameter set to true.

    label: "k8s-base"

    Specifies the instance name shown in the UI.

    yaml: "yaml code here"

    Valid kubernetes pod YAML describing a Datalore agent. See an example below.

    Optional fields

    initialPoolSize: 1

    Specifies the number of pre-warmed (pooled) agents. This is the minimum number of agents Datalore will keep started and yet unassigned to any computation. When a notebook is opened, Datalore assigns one of the pooled agents to that notebook and starts another agent to refill the pool. It might be useful to set initialPoolSize to a non-zero value if initialization of your agents takes a lot of time.

    note

    Options below are only used to adjust UI appearance, not to specify agent properties. To actually set limits for an agent, use the cpus and memory fields.

    shortName: "S"

    Specifies the short instance name.

    If it contains more than three letters, only the first three are displayed.

    numCPUs: 2

    Specifies the number of CPUs to display.

    cpuMemoryText: "8 GB"

    Specifies the agent RAM description.

    numGPUs: 1

    Specifies the number of GPUs to display.

    gpuMemoryText: "4 GB"

    Specifies the GPU memory description.

    features:
  - "1 vCPU"
  - "16 GB RAM"
  - "1 NVIDIA GPU"

    Specifies specific properties of this agent to display in the UI.

    Only properties containing the "RAM", "CPU", or "GPU" substrings will be displayed in the agent selection window.

    The example below shows what a default Datalore agent YAML looks like.

    agentsConfig:
  k8s:
    instances:
      - id: basic-agent
        ...
        yaml:
          apiVersion: v1
          kind: Pod
          metadata:
            name: k8s-agent
            labels:
              podType: dataloreKubernetesAgent
          spec:
            enableServiceLinks: false
            containers:
              - name: agent
                image: jetbrains/datalore-agent:2023.4
                securityContext:
                  privileged: true
                env:
                  - name: MAX_HEAP_SIZE
                    value: 512m
                volumeMounts:
                  - mountPath: /etc/datalore/logback-config
                    name: logback-config
                resources:
                  limits:
                    cpu: "1"
                    memory: "2048Mi"
                  requests:
                    cpu: "200m"
                    memory: "2048Mi"
            volumes:
              - name: logback-config
                configMap:
                  name: datalore-logback-config

  2. (Optional) Configure collecting logs from Datalore and agents.

    We provide the default logging configuration, which prints requested information to stdout, but you can customize it under the logbackConfig key. Find more information on how to configure Logback in the official documentation.