Set up external agents
With external agents, you can run Datalore computations outside your Datalore installation. For this purpose, you can use Docker or Podman.
Prerequisites
- Basic
Datalore Enterprise installed
One or more machines to run Datalore computations on. Further in the article, referred to as agent machines.
JRE version 11 installed on the agent machines
Docker or Podman installed on the agent machines
- GPU-related
Install the Nvidia container toolkit on your agent machines as described in the Nvidia container toolkit installation guide.
Ensure that you can access GPU inside the container.
Configure the server
Expose ports 8081, 5050, 4060, 8111 of the Datalore StatefulSet so that they are accessible from agent machines. It is assumed later in the procedure that these ports are accessible under your custom hostname (referred to as
DATALORE_INTERNAL
in this article).The recommended way of exposing these ports is through the NodePort services. By default, these ports will be mapped to 30091-30094 on every node of your cluster. After that, you can use the raw IP address of any cluster node, add a load balancer, or route traffic in any other way.
To expose ports through the NodePort service, add the following to your datalore.values.yaml:
nodePorts: enabled: true dataloreEnv: ... AGENT_MANAGER_PORT: "30091" COMPUTATION_PORT: "30092" AGENT_DISK_PORT: "30093" AGENT_MANAGER_LISTEN_PORT: "5050" COMPUTATION_LISTEN_PORT: "4060"Add the following section to your datalore.values.yaml file. Use a random string for the
token
value. For thelicense
value, use the one provided to you by the Datalore team. Make sure it's a dedicated license for external agents and not the license.key used for the on-premises installation.external: enabled: true token: "CHANGEME" license: "LICENSE:license-chars-here"Set the following environment variables under
dataloreEnv
in datalore.values.yaml. ForDATALORE_INTERNAL
, use the respective hostname.dataloreEnv: ... COMPUTATION_HOST: "DATALORE_INTERNAL" AGENTS_MANAGER_HOST: "DATALORE_INTERNAL" AGENTS_DISK_HOST: "DATALORE_INTERNAL"Add a persistent volume for the external agents server. Use `external` name. Depending on your installation, you may either use
volumes
orvolumeClaimTemplates
section inside datalore.values.yaml for it. For more details, refer to this step in the installation procedure for more details.Add an external agent description under
agentsConfig
in datalore.values.yaml. It's similar to any other agent configuration and has only two specific fields:command
: either "docker" or "podman", depending on what you are using to run the container.additionalOptions
: arbitrary string, will be passed among the other arguments when running the container. May be used to pass some specific env variables, attach volumes, enable GPUs inside containers, etc
Use the snippet below as an example:
agentsConfig: ... external: instances: - id: external-docker-basic label: "External docker basic" description: "External docker basic agent" image: jetbrains/datalore-agent:{{ .Values.dataloreVersion | default .Chart.AppVersion }} command: "docker" additionalOptions: "-e MY_ENV_VARIABLE=myValue"Append
--gpus all
to theadditionalOptions
string to enable GPUs.Ensure that Datalore is available from the running containers by executing the following command on the agent machines (substitute the proper value for DATALORE_INTERNAL, and also
docker
orpodman
for ENGINE):ENGINE run --rm alpine/curl curl -s http://DATALORE_INTERNAL:8080 -o /dev/null -w "%{http_code}\n"If everything works well, you should get the
200
response code. If the response code differs, it can be for one of the following reasons:Datalore is not running. Make sure that Datalore itself is running and is available from your browser.
Traffic routing is misconfigured. Make sure that your network allows traffic from agent machines to Datalore server, and vice versa.
Datalore host IP clashes with the IPs reserved for the container engine internals. For example, it can happen if the Datalore host resolves to an IP within the
172.17.0.0/16
network for Docker and the10.0.2.0/24
one for Podman. In this case, either change your Datalore installation IP or reconfigure the container engines to use different networks.
Configure agent machines
Click the avatar in the upper-right corner and select
.Click the Download agent bundle button to download the zip archive with the agent.
Copy this archive to every agent machine and unpack it.
In the unpacked directory, copy conf/buildAgent.dist.properties to conf/buildAgent.properties.
Edit the conf/buildAgent.properties file. For
serverUrl=http://DATALORE_INTERNAL:30094/
, provide theDATALORE_INTERNAL
value from the Configure the server procedure. Set an arbitrary name for the agent (name=SOME_NAME
). It is recommended to provide a unique name for each agent.Run the agent:
bin/agent.sh startWhen the agent starts and connects to the server, it will appear on the list on the External agents tab. Make sure that its name is the same that you provided in the properties file and then click the Authorize button to authenticate the agent.