Upgrade Datalore On-Premises (Helm installation)

Last modified: 11 March 2025

The instructions below explain how to upgrade Datalore On-Premises (Helm installation). The current latest version is 2025.2.

note

Important

  • When planning your upgrade to the latest version of Datalore, the upgrade process must be done in a sequential manner. Datalore upgrade mechanism requires you to step through each intermediate version as opposed to skipping straight to the most recent version. This is crucial to ensure the continuity, stability, and integrity of your system and data. We strongly recommend following the correct path from your current version to the desired version, acknowledging each intermediate version in the process. Failure to adhere to this can result in unforeseen compatibility issues, performance inconsistencies or data loss.

  • If you use a custom agent configuration, update agent images so that agent and server versions match after you upgrade Datalore.

  • Back up Kubernetes volumes before updating Datalore.

tip

If you use Hub

Make sure you upgrade Hub before upgrading Datalore.

Upgrade to version 2025.2

  1. Back up the storage and postgresql-data volumes before upgrading.

  2. Execute the helm repo update command.

  3. Restart Datalore by running the following commands: 

    helm uninstall datalore
helm upgrade --install -f datalore.values.yaml datalore datalore/datalore --version 0.2.28

Upgrade to version 2025.1

  1. Back up the storage and postgresql-data volumes before upgrading.

  2. (Optional) If you're still using secrets and notebook-level SSH keys or Git repositories, make sure to migrate those that you want to keep. Follow this procedure.

  3. Execute the helm repo update command.

  4. Restart Datalore by running the following commands: 

    helm uninstall datalore
helm upgrade --install -f datalore.values.yaml datalore datalore/datalore --version 0.2.26

Upgrade to version 2024.5.1

  1. Back up the storage and postgresql-data volumes before upgrading.

  2. (Optional) Make sure you have migrated from deprecated features.

  3. Execute the helm repo update command.

  4. Restart Datalore by running the following commands: 

    helm uninstall datalore
helm upgrade --install -f datalore.values.yaml datalore datalore/datalore --version 0.2.25

Upgrade to version 2024.5

  1. Back up the storage and postgresql-data volumes before upgrading.

  2. (Optional) Make sure you have migrated from deprecated features.

  3. Execute the helm repo update command.

  4. Restart Datalore by running the following commands: 

    helm uninstall datalore
helm upgrade --install -f datalore.values.yaml datalore datalore/datalore --version 0.2.24

Upgrade to version 2024.4.1

  1. Back up the storage and postgresql-data volumes before upgrading.

  2. Execute the helm repo update command.

  3. Restart Datalore by running the following commands: 

    helm uninstall datalore
helm upgrade --install -f datalore.values.yaml datalore datalore/datalore --version 0.2.23

Upgrade to version 2024.4

warning

This version includes PostgreSQL version migration from Postgres 12 to Postgres 15.

If your Datalore installation uses an external database (not the one shipped with Datalore), we advise to upgrade the database instance using the database provider preferred tooling, or to reach out to your database administrator for further guidance.

  1. Back up the storage and postgresql-data volumes before upgrading.

  2. Execute the helm repo update command.

  3. Restart Datalore by running the following commands: 

    helm uninstall datalore
helm upgrade --install -f datalore.values.yaml datalore datalore/datalore --version 0.2.22

Upgrade to version 2024.3

  1. Back up the storage and postgresql-data volumes before upgrading.

  2. Execute the helm repo update command.

  3. Restart Datalore by running the following commands: 

    helm uninstall datalore
helm upgrade --install -f datalore.values.yaml datalore datalore/datalore --version 0.2.21

Upgrade to version 2024.2.3

  1. Back up the storage and postgresql-data volumes before upgrading.

  2. Execute the helm repo update command.

  3. Restart Datalore by running the following commands: 

    helm uninstall datalore
helm upgrade --install -f datalore.values.yaml datalore datalore/datalore --version 0.2.20

Upgrade to version 2024.2.2

  1. Back up the storage and postgresql-data volumes before upgrading.

  2. Execute the helm repo update command.

  3. Restart Datalore by running the following commands: 

    helm uninstall datalore
helm upgrade --install -f datalore.values.yaml datalore datalore/datalore --version 0.2.19

Upgrade to version 2024.2.1

  1. Back up the storage and postgresql-data volumes before upgrading.

  2. Execute the helm repo update command.

  3. Restart Datalore by running the following commands: 

    helm uninstall datalore
helm upgrade --install -f datalore.values.yaml datalore datalore/datalore --version 0.2.18

Upgrade to version 2024.2

  1. Back up the storage and postgresql-data volumes before upgrading.

  2. Execute the helm repo update command.

  3. Restart Datalore by running the following commands: 

    helm uninstall datalore
helm upgrade --install -f datalore.values.yaml datalore datalore/datalore --version 0.2.17

Upgrade to version 2024.1

  1. Back up the storage and postgresql-data volumes before upgrading.

  2. Execute the helm repo update command.

  3. Restart Datalore by running the following commands: 

    helm uninstall datalore
helm upgrade --install -f datalore.values.yaml datalore datalore/datalore --version 0.2.16

Check content output migration

Upgrade to 2024.1 involves an internal output migration, which we strongly recommend checking by performing the following steps.

  1. When running the updated version (2024.1) for the first time, check Datalore server logs. The logs are expected to contain lines similar to the ones in the example below.

    note

    Timings, thread names and number of files presented in the example will differ from yours.

    12:44:03.931 INFO  [OutputsMigrationEDT] j.d.o.s.i.migration.OutputsMigrationTool Files to migrate: 313

12:44:04.205 INFO  [Thread-3] j.d.o.s.i.m.LoggingOutputMigrationProgressTracker Processed 4/313 (1%) file(s)
12:44:04.261 INFO  [Thread-2] j.d.o.s.i.m.LoggingOutputMigrationProgressTracker Processed 7/313 (2%) file(s)
...
12:44:05.324 INFO  [Thread-3] j.d.o.s.i.m.LoggingOutputMigrationProgressTracker Processed 313/313 (100%) file(s)

  2. Check if there were any errors that occurred during the migration. Such errors will be listed as a short summary below the Processed [n] 100% files line. If no errors are found, no further steps need to be performed.

  3. If there are any errors, save the IDs of notebooks together with the error messages. Scroll up the logs and find and save the respective stacktraces for these errors.

  4. Use the saved IDs to try to open the notebooks with the migration errors. If any of these notebooks fail to open, contact Datalore support providing the error message and stacktrace for each such notebook.

Upgrade to version 2023.6

  1. Back up the storage and postgresql-data volumes before upgrading.

  2. Execute the helm repo update command.

  3. Restart Datalore by running the following commands: 

    helm uninstall datalore
helm upgrade --install -f datalore.values.yaml datalore datalore/datalore --version 0.2.14

Upgrade to version 2023.5.1

  1. Back up the storage and postgresql-data volumes before upgrading.

  2. Execute the helm repo update command.

  3. Remove the following blocks under agentsConfig in the values.yaml file: 

    volumeMounts:
  - mountPath: /etc/datalore/logback-config
    name: logback-config
    volumes:
  - name: logback-config
    configMap:
      name: datalore-logback-config

  4. Restart Datalore by running the following commands: 

    helm uninstall datalore
helm upgrade --install -f datalore.values.yaml datalore datalore/datalore --version 0.2.13

Upgrade to version 2023.5

  1. Back up the storage and postgresql-data volumes before upgrading.

  2. Execute the helm repo update command.

  3. Restart Datalore by running the following commands: 

    helm uninstall datalore
helm upgrade --install -f datalore.values.yaml datalore datalore/datalore --version 0.2.12

Upgrade to version 2023.4

  1. Back up the storage and postgresql-data volumes before upgrading.

  2. Execute the helm repo update command.

  3. Restart Datalore by running the following commands: 

    helm uninstall datalore
helm upgrade --install -f datalore.values.yaml datalore datalore/datalore --version 0.2.11

Upgrade to version 2023.3.1

  1. Back up the storage and postgresql-data volumes before upgrading.

  2. Execute the helm repo update command.

  3. Restart Datalore by running the following commands: 

    helm uninstall datalore
helm upgrade --install -f datalore.values.yaml datalore datalore/datalore --version 0.2.10

note

The updating procedure involves checkpoint migration, which will cause the first startup of the new version to take longer than usual.

Upgrade from 2023.1 to 2023.2

  1. Back up the storage and postgresql-data volumes before upgrading.

  2. Execute the helm repo update command.

  3. Restart Datalore by running the following commands: 

    helm uninstall datalore
helm upgrade --install -f datalore.values.yaml datalore datalore/datalore --version 0.2.8

note

The updating procedure involves checkpoint migration, which will cause the first startup of the new version to take longer than usual.

Upgrade from 2022.3 to 2023.1

  1. Back up the storage and postgresql-data volumes before upgrading.

  2. Execute the helm repo update command.

  3. (Optional) If you built custom images based on 2023.1 agent image, remove the following line from your custom image Dockerfile before rebuilding them: 

    ADD --chown=datalore:datalore https://raw.githubusercontent.com/JetBrains/datalore-configs/2023.1/misc/build_code_insight_data.sh /opt/datalore/build_code_insight_data.sh

  4. Restart Datalore by running the following commands: 

    helm uninstall datalore
helm upgrade --install -f datalore.values.yaml datalore datalore/datalore --version 0.2.7

Upgrade from 2022.2.3 to 2022.3

  1. (Optional) If you have disabled SMTP in your configs, remove the respective parameters.

  2. Back up the storage and postgresql-data volumes before upgrading.

  3. Execute the helm repo update command.

  4. Change the dbRootPassword: XXX parameter in the datalore.values.yaml file to: 

    databaseSecret:
password: XXX

  5. (Optional) Change the following parameter names in the datalore.values.yaml file if you have them set: 

    sqlServerHost > sqlCellsApiHost
sqlServerPort > sqlCellsApiPort

  6. Restart Datalore by running the following commands: 

    helm uninstall datalore
helm upgrade --install -f datalore.values.yaml datalore datalore/datalore --version 0.2.6

    note

    Important

    The procedure assumes that you store your custom values in datalore.values.yaml, which is NOT the same as the earlier location (charts/datalore/values.yaml). See the installation guide for more details.

Upgrade from 2022.2-2022.2.2 to 2022.2.3

  • Execute the helm repo update command.

  • Restart Datalore by running the following commands: 

    helm uninstall <release_name>
helm upgrade --install -f datalore.values.yaml <release_name> datalore/datalore

    note

    Important

    The procedure assumes that you store your custom values in datalore.values.yaml, which is NOT the same as the earlier location (charts/datalore/values.yaml).

Upgrade from 2022.2 to 2022.2.1

  • Update the datalore-configs repository to the 2022.2.1 branch.

  • Restart Datalore by running the following commands: 

    helm uninstall <path_to_datalore-configs>/charts/datalore
helm upgrade --install -f datalore.values.yaml <path_to_datalore-configs>/charts/datalore

Upgrade from versions 2021.3.5-2022.1.4 to 2022.2 (Helm installation) 

  1. Update the datalore-configs repository to the 2022.2 branch.

  2. Change FRONTEND_URL to DATALORE_PUBLIC_URL.

  3. Change the format of dataloreEnv to a helm map.

    Old format:

    dataloreEnv: |
  export FRONTEND_URL="https://datalore.example.com"

  export HUB_PUBLIC_BASE_URL="https://hub.datalore.example.com/hub"
  export HUB_DATALORE_SERVICE_ID="XXX"
  export HUB_DATALORE_SERVICE_SECRET="YYY"
  export HUB_PERM_TOKEN="perm:ZZZ"

    New format:

    dataloreEnv:
  DATALORE_PUBLIC_URL: "https://datalore.example.com"

  HUB_PUBLIC_BASE_URL: "https://hub.datalore.example.com/hub"
  HUB_DATALORE_SERVICE_ID: "XXX"
  HUB_DATALORE_SERVICE_SECRET: "YYY"
  HUB_PERM_TOKEN: "perm:ZZZ"

note

Consider the following:

  • The ENABLE_PLANS variable is unnecessary. Plans are enabled if the plansConfig helm variable is set.

  • DEFAULT_PACKAGE_MANAGER="pip" is a default setting now

  • DEFAULT_BASE_ENV_NAME=minimal is a default setting now.

Data migration when upgrading to 2022.2 or later versions

Starting from version 2022.2, data directories are accessible under new names. When upgrading to version 2022.2 or later, you need to manually specify the new location for the migrated data.

note

Important

Make sure that before the procedure, you have a backup of your data.

  1. To access the Datalore container, run the following command: 

    kubectl exec -it datalore-0 -c datalore -- bash

  2. To rename the data directories in the new version: 

    mv /opt/data/webdav /opt/data/web-dav
mv /opt/data/blobs /opt/data/blob

tip

You only need to specify the new location for one such upgrade. This procedure will not be required when upgrading from version 2022.2 and later.

Earlier versions

Perform the steps in the collapsed section to upgrade from version 0.3.0.

Upgrade to version 2021.3.5

  1. Delete Datalore: 

    kubectl delete -k ./datalore

    tip

    If you upgrade from 0.3.0, ignore the following message:

    Error from server (NotFound): error when deleting "datalore/": services "datalore-on-premise" not found

  2. Clone the datalore-configs repository.

  3. Check out the 2021.1.3.5 branch.

  4. Update Hub:

    1. Copy the spec.template.spec.volumes values from the old hub/statefulSet.yaml file into the new one (in the new branch).

      note

      This is a very important step for preserving your data.

    2. Update statefulset: 

      kubectl apply -k hub/

    3. Get the Hub token and copy it: 

      kubectl logs service/hub

    4. Open Hub at HUB_ROOT_URL, specify the token, and click login and, in the next window, upgrade.

  5. Update Datalore:

    1. Copy datalore/configs/volumes_config.yaml from the old directory to the new one.

      note

      This is a very important step for preserving your data.

    2. Copy datalore/configs/plans_config.yaml to the new directory.

    3. If the agent configuration was changed, copy the datalore/configs/agents_config.yaml to the new directory and change the image: field to the respective tag.

    4. If you used custom environments, copy datalore/configs/envs too.

    5. Copy the ROOT_PASSWORD value from datalore/configs/db_config.yaml to datalore/configs/db_secret_env.txt.

      Pay attention to the difference in format:

      • Old file:

        apiVersion: v1
kind: ConfigMap
metadata:
  name: datalore-on-premise-db-env
data:
  ROOT_PASSWORD: "changeme"

      • New file:

        ROOT_PASSWORD=changeme

    6. Copy the variables from datalore/configs/user_config.yaml to datalore/configs/user_secret_env.sh.

      Pay attention to the difference in format:

      • Old file:

        apiVersion: v1
kind: ConfigMap
metadata:
  name: datalore-on-premise-env
data:
  FRONTEND_URL: "http://datalore.example.com"
  HUB_PUBLIC_BASE_URL: "http://hub.datalore.example.com/hub"
  ...

      • New file:

        export FRONTEND_URL="http://datalore.example.com"
export HUB_PUBLIC_BASE_URL="http://hub.datalore.example.com/hub"
...

      note

      • If you have HUB_INTERNAL_BASE_URL="http://hub:8082/hub" in datalore/configs/user_config.yaml, you can remove the parameter. If it has a different value, copy it to user_secret_env.sh too.

      • Remove PASSWORD_SECRET if the file contains it.

  6. Launch Datalore: 

    kubectl apply -k datalore/

Helm chart and Datalore version mapping

Below is the table that matches Helm chart, Hub, and Datalore versions.

Datalore

Charts

Hub

2025.2

0.2.28

2025.1.62967

2025.1

0.2.26

2024.3.56671

2024.5.1

0.2.25

2024.3.47197

2024.5

0.2.24

2024.3.47197

2024.4.1

0.2.23

2024.2.37269

2024.4

0.2.22

2024.2.37269

2024.3

0.2.21

2023.3.22912

2024.2.3

0.2.20

2023.3.22912

2024.2.2

0.2.19

2023.3.22912

2024.2.1

0.2.18

2023.3.22912

2024.2

0.2.17

2023.3.22912

2024.1

0.2.16

2023.3.22912

2023.6

0.2.14

2023.2.18873

2023.5.1

0.2.13

2023.2.18873

2023.5

0.2.12

2023.2.18873

2023.4

0.2.11

2022.3.15573

2023.3.1

0.2.10

2022.3.15573

2023.3

0.2.9

2022.3.15573

2023.2

0.2.8

2022.2.15039

2023.1

0.2.7

2022.2.15039

2022.3.1

0.2.6

2022.2.15039

2022.3

0.2.5

2022.2.15039

2022.2.3

0.2.4

2022.2.15039

2022.2.2

0.2.3

2021.1.14194

2022.2.1

0.2.2

2021.1.14194

2022.2

0.2.1

2021.1.14194

2022.1.1

0.2.0

2021.1.14194

2021.3

0.1.0

2021.1.13553