Datalore 2024.4 Help

Upgrade Datalore On-Premises (Helm installation)

The instructions below explain how to upgrade Datalore On-Premises (Helm installation). The current latest version is 2024.4.1.

Upgrade to version 2024.4.1

  1. Back up the storage and postgresql-data volumes before upgrading.

  2. Execute the helm repo update command.

  3. Restart Datalore by running the following commands:

    helm uninstall datalore helm upgrade --install -f datalore.values.yaml datalore datalore/datalore --version 0.2.23

Upgrade to version 2024.4

  1. Back up the storage and postgresql-data volumes before upgrading.

  2. Execute the helm repo update command.

  3. Restart Datalore by running the following commands:

    helm uninstall datalore helm upgrade --install -f datalore.values.yaml datalore datalore/datalore --version 0.2.22

Upgrade to version 2024.3

  1. Back up the storage and postgresql-data volumes before upgrading.

  2. Execute the helm repo update command.

  3. Restart Datalore by running the following commands:

    helm uninstall datalore helm upgrade --install -f datalore.values.yaml datalore datalore/datalore --version 0.2.21

Upgrade to version 2024.2.3

  1. Back up the storage and postgresql-data volumes before upgrading.

  2. Execute the helm repo update command.

  3. Restart Datalore by running the following commands:

    helm uninstall datalore helm upgrade --install -f datalore.values.yaml datalore datalore/datalore --version 0.2.20

Upgrade to version 2024.2.2

  1. Back up the storage and postgresql-data volumes before upgrading.

  2. Execute the helm repo update command.

  3. Restart Datalore by running the following commands:

    helm uninstall datalore helm upgrade --install -f datalore.values.yaml datalore datalore/datalore --version 0.2.19

Upgrade to version 2024.2.1

  1. Back up the storage and postgresql-data volumes before upgrading.

  2. Execute the helm repo update command.

  3. Restart Datalore by running the following commands:

    helm uninstall datalore helm upgrade --install -f datalore.values.yaml datalore datalore/datalore --version 0.2.18

Upgrade to version 2024.2

  1. Back up the storage and postgresql-data volumes before upgrading.

  2. Execute the helm repo update command.

  3. Restart Datalore by running the following commands:

    helm uninstall datalore helm upgrade --install -f datalore.values.yaml datalore datalore/datalore --version 0.2.17

Upgrade to version 2024.1

  1. Back up the storage and postgresql-data volumes before upgrading.

  2. Execute the helm repo update command.

  3. Restart Datalore by running the following commands:

    helm uninstall datalore helm upgrade --install -f datalore.values.yaml datalore datalore/datalore --version 0.2.16

Check content output migration

Upgrade to 2024.1 involves an internal output migration, which we strongly recommend checking by performing the following steps.

  1. When running the updated version (2024.1) for the first time, check Datalore server logs. The logs are expected to contain lines similar to the ones in the example below.

    12:44:03.931 INFO [OutputsMigrationEDT] j.d.o.s.i.migration.OutputsMigrationTool Files to migrate: 313
 12:44:04.205 INFO [Thread-3] j.d.o.s.i.m.LoggingOutputMigrationProgressTracker Processed 4/313 (1%) file(s) 12:44:04.261 INFO [Thread-2] j.d.o.s.i.m.LoggingOutputMigrationProgressTracker Processed 7/313 (2%) file(s) ... 12:44:05.324 INFO [Thread-3] j.d.o.s.i.m.LoggingOutputMigrationProgressTracker Processed 313/313 (100%) file(s)

  2. Check if there were any errors that occurred during the migration. Such errors will be listed as a short summary below the Processed [n] 100% files line. If no errors are found, no further steps need to be performed.

  3. If there are any errors, save the IDs of notebooks together with the error messages. Scroll up the logs and find and save the respective stacktraces for these errors.

  4. Use the saved IDs to try to open the notebooks with the migration errors. If any of these notebooks fail to open, contact Datalore support providing the error message and stacktrace for each such notebook.

Upgrade to version 2023.6

  1. Back up the storage and postgresql-data volumes before upgrading.

  2. Execute the helm repo update command.

  3. Restart Datalore by running the following commands:

    helm uninstall datalore helm upgrade --install -f datalore.values.yaml datalore datalore/datalore --version 0.2.14

Upgrade to version 2023.5.1

  1. Back up the storage and postgresql-data volumes before upgrading.

  2. Execute the helm repo update command.

  3. Remove the following blocks under agentsConfig in the values.yaml file:

    volumeMounts: - mountPath: /etc/datalore/logback-config name: logback-config
    volumes: - name: logback-config configMap: name: datalore-logback-config

  4. Restart Datalore by running the following commands:

    helm uninstall datalore helm upgrade --install -f datalore.values.yaml datalore datalore/datalore --version 0.2.13

Upgrade to version 2023.5

  1. Back up the storage and postgresql-data volumes before upgrading.

  2. Execute the helm repo update command.

  3. Restart Datalore by running the following commands:

    helm uninstall datalore helm upgrade --install -f datalore.values.yaml datalore datalore/datalore --version 0.2.12

Upgrade to version 2023.4

  1. Back up the storage and postgresql-data volumes before upgrading.

  2. Execute the helm repo update command.

  3. Restart Datalore by running the following commands:

    helm uninstall datalore helm upgrade --install -f datalore.values.yaml datalore datalore/datalore --version 0.2.11

Upgrade to version 2023.3.1

  1. Back up the storage and postgresql-data volumes before upgrading.

  2. Execute the helm repo update command.

  3. Restart Datalore by running the following commands:

    helm uninstall datalore helm upgrade --install -f datalore.values.yaml datalore datalore/datalore --version 0.2.10

Upgrade from 2023.1 to 2023.2

  1. Back up the storage and postgresql-data volumes before upgrading.

  2. Execute the helm repo update command.

  3. Restart Datalore by running the following commands:

    helm uninstall datalore helm upgrade --install -f datalore.values.yaml datalore datalore/datalore --version 0.2.8

Upgrade from 2022.3 to 2023.1

  1. Back up the storage and postgresql-data volumes before upgrading.

  2. Execute the helm repo update command.

  3. (Optional) If you built custom images based on 2023.1 agent image, remove the following line from your custom image Dockerfile before rebuilding them:

    ADD --chown=datalore:datalore https://raw.githubusercontent.com/JetBrains/datalore-configs/2023.1/misc/build_code_insight_data.sh /opt/datalore/build_code_insight_data.sh

  4. Restart Datalore by running the following commands:

    helm uninstall datalore helm upgrade --install -f datalore.values.yaml datalore datalore/datalore --version 0.2.7

Upgrade from 2022.2.3 to 2022.3

  1. (Optional) If you have disabled SMTP in your configs, remove the respective parameters.

  2. Back up the storage and postgresql-data volumes before upgrading.

  3. Execute the helm repo update command.

  4. Change the dbRootPassword: XXX parameter in the datalore.values.yaml file to:

    databaseSecret: password: XXX

  5. (Optional) Change the following parameter names in the datalore.values.yaml file if you have them set:

    sqlServerHost > sqlCellsApiHost sqlServerPort > sqlCellsApiPort

  6. Restart Datalore by running the following commands:

    helm uninstall datalore helm upgrade --install -f datalore.values.yaml datalore datalore/datalore --version 0.2.6

Upgrade from 2022.2-2022.2.2 to 2022.2.3

  • Execute the helm repo update command.

  • Restart Datalore by running the following commands:

    helm uninstall <release_name> helm upgrade --install -f datalore.values.yaml <release_name> datalore/datalore

Upgrade from 2022.2 to 2022.2.1

  • Update the datalore-configs repository to the 2022.2.1 branch.

  • Restart Datalore by running the following commands:

    helm uninstall <path_to_datalore-configs>/charts/datalore helm upgrade --install -f datalore.values.yaml <path_to_datalore-configs>/charts/datalore

Upgrade from versions 2021.3.5-2022.1.4 to 2022.2 (Helm installation)

  1. Update the datalore-configs repository to the 2022.2 branch.

  2. Change FRONTEND_URL to DATALORE_PUBLIC_URL.

  3. Change the format of dataloreEnv to a helm map.

    Old format:

    dataloreEnv: | export FRONTEND_URL="https://datalore.example.com" export HUB_PUBLIC_BASE_URL="https://hub.datalore.example.com/hub" export HUB_DATALORE_SERVICE_ID="XXX" export HUB_DATALORE_SERVICE_SECRET="YYY" export HUB_PERM_TOKEN="perm:ZZZ"

    New format:

    dataloreEnv: DATALORE_PUBLIC_URL: "https://datalore.example.com" HUB_PUBLIC_BASE_URL: "https://hub.datalore.example.com/hub" HUB_DATALORE_SERVICE_ID: "XXX" HUB_DATALORE_SERVICE_SECRET: "YYY" HUB_PERM_TOKEN: "perm:ZZZ"

Data migration when upgrading to 2022.2 or later versions

Starting from version 2022.2, data directories are accessible under new names. When upgrading to version 2022.2 or later, you need to manually specify the new location for the migrated data.

  1. To access the Datalore container, run the following command:

    kubectl exec -it datalore-0 -c datalore -- bash

  2. To rename the data directories in the new version:

    mv /opt/data/webdav /opt/data/web-dav mv /opt/data/blobs /opt/data/blob

Earlier versions

Perform the steps in the collapsed section to upgrade from version 0.3.0.

Upgrade from version 0.3.0

  1. Execute the following command to stop traffic to Datalore:

    kubectl delete svc datalore-on-premise

  2. Wait for approximately one minute until the running computations are stopped. Then, execute the following command:

    kubectl exec -it datalore-on-premise-0 -c postgres -- psql -ddatalore -Uadmin -c 'delete from core."RunningComputationsData";'

  3. Delete the Docker Hub token:

    kubectl delete secret regcred

  4. Continue the procedure as described below.

Upgrade to version 2021.3.5

  1. Delete Datalore:

    kubectl delete -k ./datalore

  2. Clone the datalore-configs repository.

  3. Check out the 2021.1.3.5 branch.

  4. Update Hub:

    1. Copy the spec.template.spec.volumes values from the old hub/statefulSet.yaml file into the new one (in the new branch).

    2. Update statefulset:

      kubectl apply -k hub/

    3. Get the Hub token and copy it:

      kubectl logs service/hub

    4. Open Hub at HUB_ROOT_URL, specify the token, and click login and, in the next window, upgrade.

  5. Update Datalore:

    1. Copy datalore/configs/volumes_config.yaml from the old directory to the new one.

    2. Copy datalore/configs/plans_config.yaml to the new directory.

    3. If the agent configuration was changed, copy the datalore/configs/agents_config.yaml to the new directory and change the image: field to the respective tag.

    4. If you used custom environments, copy datalore/configs/envs too.

    5. Copy the ROOT_PASSWORD value from datalore/configs/db_config.yaml to datalore/configs/db_secret_env.txt.

      Pay attention to the difference in format:

      • Old file:

        apiVersion: v1 kind: ConfigMap metadata: name: datalore-on-premise-db-env data: ROOT_PASSWORD: "changeme"

      • New file:

        ROOT_PASSWORD=changeme

    6. Copy the variables from datalore/configs/user_config.yaml to datalore/configs/user_secret_env.sh.

      Pay attention to the difference in format:

      • Old file:

        apiVersion: v1 kind: ConfigMap metadata: name: datalore-on-premise-env data: FRONTEND_URL: "http://datalore.example.com" HUB_PUBLIC_BASE_URL: "http://hub.datalore.example.com/hub" ...

      • New file:

        export FRONTEND_URL="http://datalore.example.com" export HUB_PUBLIC_BASE_URL="http://hub.datalore.example.com/hub" ...

  6. Launch Datalore:

    kubectl apply -k datalore/

Helm chart and Datalore version mapping

Below is the table that matches Helm chart, Hub, and Datalore versions.

Datalore

Charts

Hub

2024.4.1

0.2.23

2024.2.37269

2024.4

0.2.22

2024.2.37269

2024.3

0.2.21

2023.3.22912

2024.2.3

0.2.20

2023.3.22912

2024.2.2

0.2.19

2023.3.22912

2024.2.1

0.2.18

2023.3.22912

2024.2

0.2.17

2023.3.22912

2024.1

0.2.16

2023.3.22912

2023.6

0.2.14

2023.2.18873

2023.5.1

0.2.13

2023.2.18873

2023.5

0.2.12

2023.2.18873

2023.4

0.2.11

2022.3.15573

2023.3.1

0.2.10

2022.3.15573

2023.3

0.2.9

2022.3.15573

2023.2

0.2.8

2022.2.15039

2023.1

0.2.7

2022.2.15039

2022.3.1

0.2.6

2022.2.15039

2022.3

0.2.5

2022.2.15039

2022.2.3

0.2.4

2022.2.15039

2022.2.2

0.2.3

2021.1.14194

2022.2.1

0.2.2

2021.1.14194

2022.2

0.2.1

2021.1.14194

2022.1.1

0.2.0

2021.1.14194

2021.3

0.1.0

2021.1.13553

Last modified: 27 September 2024
Install Datalore On-Premises in Docker containers using Docker ComposeUpgrade Datalore On-Premises (Docker installation)