Datalore 2023.1 Help

Set up external agents

With external agents, you can run Datalore computations outside your Datalore installation. For this purpose, you can use Docker or Podman.

Prerequisites

Basic

  • Datalore Enterprise installed

  • One or more machines to run Datalore computations on. Further in the article, referred to as agent machines.

  • JRE version 11 installed on the agent machines

  • Docker or Podman installed on the agent machines

GPU-related

  1. Install the Nvidia container toolkit on your agent machines as described in the Nvidia container toolkit installation guide.

  2. Ensure that you can access GPU inside the container.

Configure the server

  1. Expose ports 8081, 5050, 4060, 8111 of the Datalore StatefulSet so that they are accessible from agent machines. It is assumed later in the procedure that these ports are accessible under your custom hostname (referred to as DATALORE_INTERNAL in this article).

    The recommended way of exposing these ports is through the NodePort services. By default, these ports will be mapped to 30091-30094 on every node of your cluster. After that, you can use the raw IP address of any cluster node, add a load balancer, or route traffic in any other way.

    To expose ports through the NodePort service, add the following to your datalore.values.yaml:

    nodePorts: enabled: true dataloreEnv: ... AGENT_MANAGER_PORT: "30091" COMPUTATION_PORT: "30092" AGENT_DISK_PORT: "30093" AGENT_MANAGER_LISTEN_PORT: "5050" COMPUTATION_LISTEN_PORT: "4060"

  2. Add the following section to your datalore.values.yaml file. Use a random string for the token value. For the license value, use the one provided to you by the Datalore team. Make sure it's a dedicated license for external agents and not the license.key used for the on-premises installation.

    external: enabled: true token: "CHANGEME" license: "LICENSE:license-chars-here"

  3. Set the following environment variables under dataloreEnv in datalore.values.yaml. For DATALORE_INTERNAL, use the respective hostname.

    dataloreEnv: ... COMPUTATION_HOST: "DATALORE_INTERNAL" AGENTS_MANAGER_HOST: "DATALORE_INTERNAL" AGENTS_DISK_HOST: "DATALORE_INTERNAL"

  4. Add a persistent volume for the external agents server. Use `external` name. Depending on your installation, you may either use volumes or volumeClaimTemplates section inside datalore.values.yaml for it. For more details, refer to this step in the installation procedure for more details.

  5. Add an external agent description under agentsConfig in datalore.values.yaml. It's similar to any other agent configuration and has only two specific fields:

    • command: either "docker" or "podman", depending on what you are using to run the container.

    • additionalOptions: arbitrary string, will be passed among the other arguments when running the container. May be used to pass some specific env variables, attach volumes, enable GPUs inside containers, etc

  6. Use the snippet below as an example:

    agentsConfig: ... external: instances: - id: external-docker-basic label: "External docker basic" description: "External docker basic agent" image: jetbrains/datalore-agent:{{ .Values.dataloreVersion | default .Chart.AppVersion }} command: "docker" additionalOptions: "-e MY_ENV_VARIABLE=myValue"

  7. Append --gpus all to the additionalOptions string to enable GPUs.

  8. Ensure that Datalore is available from the running containers by executing the following command on the agent machines (substitute the proper value for DATALORE_INTERNAL, and also docker or podman for ENGINE):

    ENGINE run --rm alpine/curl curl -s http://DATALORE_INTERNAL:8080 -o /dev/null -w "%{http_code}\n"

    If everything works well, you should get the 200 response code. If the response code differs, it can be for one of the following reasons:

    • Datalore is not running. Make sure that Datalore itself is running and is available from your browser.

    • Traffic routing is misconfigured. Make sure that your network allows traffic from agent machines to Datalore server, and vice versa.

    • Datalore host IP clashes with the IPs reserved for the container engine internals. For example, it can happen if the Datalore host resolves to an IP within the 172.17.0.0/16 network for Docker and the 10.0.2.0/24 one for Podman. In this case, either change your Datalore installation IP or reconfigure the container engines to use different networks.

Configure agent machines

  1. Click the avatar in the upper-right corner and select Admin panel | External agents.

  2. Click the Download agent bundle button to download the zip archive with the agent.

  3. Copy this archive to every agent machine and unpack it.

  4. In the unpacked directory, copy conf/buildAgent.dist.properties to conf/buildAgent.properties.

  5. Edit the conf/buildAgent.properties file. For serverUrl=http://DATALORE_INTERNAL:30094/, provide the DATALORE_INTERNAL value from the Configure the server procedure. Set an arbitrary name for the agent (name=SOME_NAME). It is recommended to provide a unique name for each agent.

  6. Run the agent:

    bin/agent.sh start

  7. When the agent starts and connects to the server, it will appear on the list on the External agents tab. Make sure that its name is the same that you provided in the properties file and then click the Authorize button to authenticate the agent.

Last modified: 20 February 2023
Set up GPU machines Configure plans