Customize or update environment

You can build a custom Docker agent image on top of the default one to customize your environment (install some package from apt to be available in your notebooks, or set up a custom Python environment by pre-installing the required libraries).

Generate indexes to ensure the editor's proper work

After you create a new custom environment or update one of our environments, run the following command in the same file:

RUN /opt/datalore/build_code_insight_data.sh /[environment_path]

where [environment_path] is the path to the environment you created or updated.

If not performed, these indexes will be updated every time the agent starts, which slows down the editor.

Add the created environment file to the main datalore pod

Add the file to following directory:

/opt/datalore/configs/environment_info

Make sure you follow these instructions:

For a conda environment, specify the respective .yml file matching the envrionment name. For example, if you created a conda environment called somename, add an environment_somename.yml file.
For a pip environment, specify the respective .txt file matching the envrionment name. For example, if you created a pip environment called somename, add a requirements_somename.txt file.

Enable arbitrary PyPi server configuration

The procedure sets up a custom PyPI server configuration. This will allow you to explore your custom repositories in the editor's Environment tool and directly install packages from outside the default pypi.org.

Run the following in RUN command when building the custom agent image:
pip config set global.trusted-host "$(pip config get global.trusted-host) <host>" pip config set global.extra-index-url "url>"
For repositories requiring authentication, add the credentials to the <url> part as shown below:
pip config set global.extra-index-url "http(s)://<username>:<password>@<url>/simple/"
To enable it for a specific notebook, add the above commands to the init.sh file.

To make a custom environment available for new notebooks, use the Admin panel as described in this topic.

Examples

Modify the pip/minimal environment

            FROM jetbrains/datalore-agent:2024.5
            COPY my_additional_packages.txt /tmp/my_additional_packages.txt
            RUN /opt/python/envs/minimal/bin/python -m pip install -r /tmp/my_additional_packages.txt
            RUN /opt/datalore/build_code_insight_data.sh /opt/python/envs/minimal
        

Use the additional_packages.txt file as a list of packages and their versions as shown below:

tensorflow==2.3.1
uvicorn==0.12.2
fastapi==0.63.0

Create a new environment

Assuming the name of the new environment is myenv:

                FROM jetbrains/datalore-agent:2024.5
                RUN /opt/python/bin/python -m venv /opt/python/envs/myenv
                RUN /opt/python/envs/myenv/bin/python -m pip install -r /tmp/requirements_myenv.txt
                RUN /opt/datalore/build_code_insight_data.sh /opt/python/envs/myenv
            

Custom Python environment with version 3.10: This custom environment sets Python 3.10 as the default version.
FROM jetbrains/datalore-agent:2024.5 USER root ENV DEBIAN_FRONTEND noninteractive RUN apt-get update && apt-get install -y python3.10-venv && rm -rf /var/lib/apt/lists/* && apt-get clean USER datalore RUN /usr/bin/python3.10 -m venv /opt/python/envs/myenv RUN /opt/python/envs/myenv/bin/python -m pip install ipykernel==5.5.3 ipython==7.31.1 ipython_genutils==0.2.0 jedi==0.17.2 aiohttp==3.8.3 lets-plot pandas RUN /opt/datalore/build_code_insight_data.sh /opt/python/envs/myenv
Custom Python environment with version 3.11: This custom environment sets Python 3.11 as the default version.
FROM jetbrains/datalore-agent:2024.5 USER root RUN apt-get update && apt-get install -y python3.11 python3.11-venv && rm -rf /var/lib/apt/lists/* && apt-get clean USER datalore RUN /usr/bin/python3.11 -m venv /opt/python/envs/myenv RUN /opt/python/envs/myenv/bin/python -m pip install ipykernel==5.5.3 ipython==7.31.1 ipython_genutils==0.2.0 jedi==0.17.2 aiohttp==3.8.3 lets-plot pandas RUN /opt/datalore/build_code_insight_data.sh /opt/python/envs/myenv
Custom R environment without Anaconda: This custom environment sets a non-conda default environment with the R kernel.
At the moment this solution adds this environment to /opt/anaconda/envs. Therefore, this option will be offered as a conda-based environment when creating a notebook. Such a restriction will be removed in the next release.
FROM jetbrains/datalore-agent:2024.5 ENV CUSTOM_ENV_NAME myenv USER root RUN apt-get update && \ DEBIAN_FRONTEND=noninteractive apt-get install -y -q --no-install-recommends libzmq3-dev libcurl4-openssl-dev libssl-dev r-base make g++ libharfbuzz-dev libfribidi-dev libtiff-dev apt-file && \ Rscript -e "install.packages(c('repr', 'IRdisplay', 'IRkernel'), type = 'source')" && \ rm -rf /var/lib/apt/lists/* && apt-get clean RUN sudo chown -R datalore:datalore /home/datalore USER datalore RUN mkdir -p /opt/anaconda3/envs/$CUSTOM_ENV_NAME RUN /opt/python/bin/python -m venv /opt/anaconda3/envs/$CUSTOM_ENV_NAME RUN /opt/anaconda3/envs/$CUSTOM_ENV_NAME/bin/pip install ipykernel==5.5.3 ipython==7.31.1 ipython_genutils==0.2.0 jedi==0.17.2 RUN PATH=/opt/anaconda3/envs/$CUSTOM_ENV_NAME/bin/:$PATH Rscript -e "IRkernel::installspec(sys_prefix=TRUE)" RUN /opt/datalore/build_code_insight_data.sh /opt/anaconda3/envs/$CUSTOM_ENV_NAME

Last modified: 20 August 2024