Healthcheck & monitoring
Use Datalore's in-built HTTP endpoint (accessible at /health) to verify whether the instance has become online and responsive.
This endpoint and returns OK when no issues are detected.
Use the same endpoint as Kubernetes liveness probe if default Helm charts are used for the deployment.
Datalore has a built-in metrics exporter, which is disabled by default and accessible at the /metrics path when enabled explicitly.
There are two mutually exclusive environment variables of the Datalore server that can be used to enable metrics:
Name | Type | Default value | Description |
|---|---|---|---|
| String | Not defined | Enables the exporter and defines the authentication token required to collect metrics. Mutually exclusive with |
| String | Not defined | Enables the exporter. No authentication will be required to read metrics. Mutually exclusive with |
note
If both of the variables are defined, the authentication will still be enforced.
tip
While Datalore exposes significantly more metrics than listed below, only those included in this list have guaranteed backwards compatibility.
agent_pool_size: shows how many agents the pool currently has.Prometheus query:
sum by (instance_name)(agent_pool_size)
agent_waiting_time_bucket: represents the timespan in which the user waited for an instance startup.Prometheus query:
sum(increase(agent_waiting_time_bucket[10m])) by (le)
agent_in_pool_time_bucket: represents the timespan in which the agent was online and idle before being assigned to a specific notebook.Prometheus query:
sum(increase(agent_in_pool_time_bucket[10m])) by (le)
agents_started_total: shows how many agents were started per minute.Prometheus query:
sum by (instance_name)(rate(agents_started_total[5m])) * 60