Set up external agents
With external agents, you can run Datalore computations outside your Datalore installation. For this purpose, you can use Docker or Podman.
Prerequisites
- Basic
Datalore Enterprise installed
One or more machines to run Datalore computations on. Further in the article, referred to as agent machines.
JRE version 11 installed on the agent machines
Docker or Podman installed on the agent machines
- GPU-related
You need the Nvidia container toolkit installed on your agent machines as described in the Nvidia container toolkit installation guide. To ensure that you can access GPU inside the container, run the following command, depending on your container engine:
docker run -it --rm --gpus all jetbrains/datalore-agent:2023.1 nvidia-smiorpodman run -it --rm -e NVIDIA_VISIBLE_DEVICES=all jetbrains/datalore-agent:2023.1 nvidia-smi
Configure the server
Expose ports 4060, 5050, 8081, 8111 on the machine with your Datalore installation so that they are accessible from agent machines. It is assumed later in the procedures that these ports are accessible under your custom hostname (referred to as
DATALORE_INTERNAL
in this article).Create a license.keys file containing the license key for external agents.
Edit the external-server.yaml file:
services: datalore: environment: EXTERNAL_PASSWORD: [any string, equal to EXTERNAL_PASSWORD under external-server] EXTERNAL_SERVER_URL: "http://external-server:8111" COMPUTATION_HOST: [DATALORE_INTERNAL described above] AGENT_MANAGER_HOST: DATALORE_INTERNAL described above] AGENT_DISK_HOST: DATALORE_INTERNAL described above] external-server: ... environment: DB_PASSWORD: [equal to DB_PASSWORD provided in docker-compose.yaml] EXTERNAL_PASSWORD: [any string, equal to EXTERNAL_PASSWORD under datalore] EXTERNAL_PASSWORD: [any string, equal to EXTERNAL_PASSWORD under datalore] volumes: ... - "./license.keys:/opt/bazel/license.keys"Add an external agent description under
agentsConfig
in agents-config.yaml. It's similar to any other agent configuration and has only two specific fields:command
: either "docker" or "podman", depending on what you are using to run the container.additionalOptions
: arbitrary string, will be passed among the other arguments when running the container. May be used to pass some specific env variables, attach volumes, enable GPUs inside containers, etc
Use the snippet below as an example:
external: instances: - id: external-docker-basic label: "External docker basic" description: "External docker basic agent" image: jetbrains/datalore-agent:2023.1 command: "docker" additionalOptions: "-e MY_ENV_VARIABLE=myValue"To enable GPUs, append the following string to the
additionalOptions
, depending on your container engine:for
podman
:-e NVIDIA_VISIBLE_DEVICES=all
for
docker
:--gpus all
Ensure that Datalore is available from the running containers by executing the following command on the agent machines (substitute the proper value for DATALORE_INTERNAL, and also
docker
orpodman
for ENGINE):ENGINE run --rm alpine/curl curl -s http://DATALORE_INTERNAL:8080 -o /dev/null -w "%{http_code}\n"If everything works well, you should get the
200
response code. If the response code differs, it can be for one of the following reasons:Datalore is not running. Make sure that Datalore itself is running and is available from your browser.
Traffic routing is misconfigured. Make sure that your network allows traffic from agent machines to the Datalore server, and vice versa.
Datalore host IP clashes with the IPs reserved for container engine internals. For example, it can happen if the Datalore host resolves to an IP within the
172.17.0.0/16
network for Docker and the10.0.2.0/24
one for Podman. In this case, either change your Datalore installation IP or reconfigure the container engines to use different networks.
When starting the server, pass both docker-compose.yaml and external-server.yaml files:
docker compose -f docker-compose.yaml -f external-server.yaml up
Configure agent machines
Click the avatar in the upper-right corner and select
.Click the Download agent bundle button to download the zip archive with the agent.
Copy this archive to every agent machine and unpack it.
In the unpacked directory, copy conf/buildAgent.dist.properties to conf/buildAgent.properties.
Edit the conf/buildAgent.properties file. For
serverUrl=http://DATALORE_INTERNAL:8111/
, provide theDATALORE_INTERNAL
value from the Configure the server procedure. Set an arbitrary name for the agent (name=SOME_NAME
). It is recommended to provide a unique name for each agent.Run the agent:
bin/agent.sh startWhen the agent starts and connects to the server, it will appear on the list on the External agents tab. Make sure that its name is the same that you provided in the properties file and then click the Authorize button to authenticate the agent.