Upgrade Datalore Enterprise (Helm installation)

The instructions below explain how to upgrade Datalore Enterprise (Helm installation). The current latest version is 2023.5.1

When planning your upgrade to the latest version of Datalore, the upgrade process must be done in a sequential manner. Datalore upgrade mechanism requires you to step through each intermediate version as opposed to skipping straight to the most recent version. This is crucial to ensure the continuity, stability, and integrity of your system and data. We strongly recommend following the correct path from your current version to the desired version, acknowledging each intermediate version in the process. Failure to adhere to this can result in unforeseen compatibility issues, performance inconsistencies or data loss.
If you use a custom agent configuration, update agent images so that agent and server versions match after you upgrade Datalore.
Back up Kubernetes volumes before updating Datalore.

Upgrade to version 2023.5.1

Back up the storage and postgresql-data volumes before upgrading.
Execute the helm repo update command.
Remove the following blocks under agentsConfig in the values.yaml file:
volumeMounts: - mountPath: /etc/datalore/logback-config name: logback-config
volumes: - name: logback-config configMap: name: datalore-logback-config
Restart Datalore by running the following commands:
helm uninstall datalore helm upgrade --install -f datalore.values.yaml datalore datalore/datalore --version 0.2.13

Upgrade to version 2023.5

Back up the storage and postgresql-data volumes before upgrading.
Execute the helm repo update command.
Restart Datalore by running the following commands:
helm uninstall datalore helm upgrade --install -f datalore.values.yaml datalore datalore/datalore --version 0.2.12

Upgrade to version 2023.4

Back up the storage and postgresql-data volumes before upgrading.
Execute the helm repo update command.
Restart Datalore by running the following commands:
helm uninstall datalore helm upgrade --install -f datalore.values.yaml datalore datalore/datalore --version 0.2.11

Upgrade to version 2023.3.1

Back up the storage and postgresql-data volumes before upgrading.
Execute the helm repo update command.
Restart Datalore by running the following commands:
helm uninstall datalore helm upgrade --install -f datalore.values.yaml datalore datalore/datalore --version 0.2.10

Upgrade from 2023.1 to 2023.2

Back up the storage and postgresql-data volumes before upgrading.
Execute the helm repo update command.
Restart Datalore by running the following commands:
helm uninstall datalore helm upgrade --install -f datalore.values.yaml datalore datalore/datalore --version 0.2.8

Upgrade from 2022.3 to 2023.1

Back up the storage and postgresql-data volumes before upgrading.
Execute the helm repo update command.
(Optional) If you built custom images based on 2023.1 agent image, remove the following line from your custom image Dockerfile before rebuilding them:
ADD --chown=datalore:datalore https://raw.githubusercontent.com/JetBrains/datalore-configs/2023.1/misc/build_code_insight_data.sh /opt/datalore/build_code_insight_data.sh
Restart Datalore by running the following commands:
helm uninstall datalore helm upgrade --install -f datalore.values.yaml datalore datalore/datalore --version 0.2.7

Upgrade from 2022.2.3 to 2022.3

(Optional) If you have disabled SMTP in your configs, remove the respective parameters.
Back up the storage and postgresql-data volumes before upgrading.
Execute the helm repo update command.
Change the dbRootPassword: XXX parameter in the datalore.values.yaml file to:
databaseSecret: password: XXX
(Optional) Change the following parameter names in the datalore.values.yaml file if you have them set:
sqlServerHost > sqlCellsApiHost sqlServerPort > sqlCellsApiPort
Restart Datalore by running the following commands:
helm uninstall datalore helm upgrade --install -f datalore.values.yaml datalore datalore/datalore --version 0.2.6
The procedure assumes that you store your custom values in datalore.values.yaml, which is NOT the same as the earlier location (charts/datalore/values.yaml). See the installation guide for more details.

Upgrade from 2022.2-2022.2.2 to 2022.2.3

Execute the helm repo update command.
Restart Datalore by running the following commands:
helm uninstall <release_name> helm upgrade --install -f datalore.values.yaml <release_name> datalore/datalore
The procedure assumes that you store your custom values in datalore.values.yaml, which is NOT the same as the earlier location (charts/datalore/values.yaml).

Upgrade from 2022.2 to 2022.2.1

Update the datalore-configs repository to the 2022.2.1 branch.
Restart Datalore by running the following commands:
helm uninstall <path_to_datalore-configs>/charts/datalore helm upgrade --install -f datalore.values.yaml <path_to_datalore-configs>/charts/datalore

Upgrade from versions 2021.3.5-2022.1.4 to 2022.2 (Helm installation)

Update the datalore-configs repository to the 2022.2 branch.
Change FRONTEND_URL to DATALORE_PUBLIC_URL.
Change the format of dataloreEnv to a helm map.
Old format:
dataloreEnv: | export FRONTEND_URL="https://datalore.example.com" export HUB_PUBLIC_BASE_URL="https://hub.datalore.example.com/hub" export HUB_DATALORE_SERVICE_ID="XXX" export HUB_DATALORE_SERVICE_SECRET="YYY" export HUB_PERM_TOKEN="perm:ZZZ"
New format:
dataloreEnv: DATALORE_PUBLIC_URL: "https://datalore.example.com" HUB_PUBLIC_BASE_URL: "https://hub.datalore.example.com/hub" HUB_DATALORE_SERVICE_ID: "XXX" HUB_DATALORE_SERVICE_SECRET: "YYY" HUB_PERM_TOKEN: "perm:ZZZ"

Data migration when upgrading to 2022.2 or later versions

Starting from version 2022.2, data directories are accessible under new names. When upgrading to version 2022.2 or later, you need to manually specify the new location for the migrated data.

To access the Datalore container, run the following command:
kubectl exec -it datalore-0 -c datalore -- bash
To rename the data directories in the new version:
mv /opt/data/webdav /opt/data/web-dav mv /opt/data/blobs /opt/data/blob

Earlier versions

Perform the steps in the collapsed section to upgrade from version 0.3.0.

Upgrade from version 0.3.0

Execute the following command to stop traffic to Datalore:
kubectl delete svc datalore-on-premise
Wait for approximately one minute until the running computations are stopped. Then, execute the following command:
kubectl exec -it datalore-on-premise-0 -c postgres -- psql -ddatalore -Uadmin -c 'delete from core."RunningComputationsData";'
Delete the Docker Hub token:
kubectl delete secret regcred
Continue the procedure as described below.

Upgrade to version 2021.3.5

Delete Datalore:
kubectl delete -k ./datalore
If you upgrade from 0.3.0, ignore the following message:
Error from server (NotFound): error when deleting "datalore/": services "datalore-on-premise" not found
Clone the datalore-configs repository.
Check out the 2021.1.3.5 branch.
Update Hub:
1. Copy the spec.template.spec.volumes values from the old hub/statefulSet.yaml file into the new one (in the new branch).
  This is a very important step for preserving your data.
2. Update statefulset:
  kubectl apply -k hub/
3. Get the Hub token and copy it:
  kubectl logs service/hub
4. Open Hub at HUB_ROOT_URL, specify the token, and click login and, in the next window, upgrade.
Update Datalore:
1. Copy datalore/configs/volumes_config.yaml from the old directory to the new one.
  This is a very important step for preserving your data.
2. Copy datalore/configs/plans_config.yaml to the new directory.
3. If the agent configuration was changed, copy the datalore/configs/agents_config.yaml to the new directory and change the image: field to the respective tag.
4. If you used custom environments, copy datalore/configs/envs too.
5. Copy the ROOT_PASSWORD value from datalore/configs/db_config.yaml to datalore/configs/db_secret_env.txt.
  Pay attention to the difference in format:
  - Old file:
    apiVersion: v1 kind: ConfigMap metadata: name: datalore-on-premise-db-env data: ROOT_PASSWORD: "changeme"
  - New file:
    ROOT_PASSWORD=changeme
6. Copy the variables from datalore/configs/user_config.yaml to datalore/configs/user_secret_env.sh.
  Pay attention to the difference in format:
  - Old file:
    apiVersion: v1 kind: ConfigMap metadata: name: datalore-on-premise-env data: FRONTEND_URL: "http://datalore.example.com" HUB_PUBLIC_BASE_URL: "http://hub.datalore.example.com/hub" ...
  - New file:
    export FRONTEND_URL="http://datalore.example.com" export HUB_PUBLIC_BASE_URL="http://hub.datalore.example.com/hub" ...
  - If you have HUB_INTERNAL_BASE_URL="http://hub:8082/hub" in datalore/configs/user_config.yaml, you can remove the parameter. If it has a different value, copy it to user_secret_env.sh too.
  - Remove PASSWORD_SECRET if the file contains it.
Launch Datalore:
kubectl apply -k datalore/

Helm chart and Datalore version mapping

Below is the table that matches Helm chart, Hub, and Datalore versions.

Datalore	Charts	Hub
2023.5.1	0.2.13	2023.2.18873
2023.5	0.2.12	2023.2.18873
2023.4	0.2.11	2022.3.15573
2023.3.1	0.2.10	2022.3.15573
2023.3	0.2.9	2022.3.15573
2023.2	0.2.8	2022.2.15039
2023.1	0.2.7	2022.2.15039
2022.3.1	0.2.6	2022.2.15039
2022.3	0.2.5	2022.2.15039
2022.2.3	0.2.4	2022.2.15039
2022.2.2	0.2.3	2021.1.14194
2022.2.1	0.2.2	2021.1.14194
2022.2	0.2.1	2021.1.14194
2022.1.1	0.2.0	2021.1.14194
2021.3	0.1.0	2021.1.13553

Last modified: 13 November 2023