OpenTelemetry tracing and metrics
OpenTelemetry is a set of APIs, SDKs, and tools designed to let you create applications that can easily export tracing, metrics, and logs. The OpenTelemetry API provides a single set of APIs for distributed tracing and metrics instrumentation that automatically collects the data and sends it to the backend of your choice.
The OpenTelemetry statistics implemented in PyCharm includes traces and metrics. Tracer creates spans representing activities. Metrics include counters and gauges. For more information, refer to the OpenTelemetry documentation.
You can enable OpenTelemetry using the idea.properties file.
In the IDE, from the main menu, select Help | Edit Custom Properties.
In the file that opens, add the following code:
idea.diagnostic.opentelemetry.otlp=true
Spans
A Span represents a single operation with a trace. Spans can be sent to the endpoint and visualized in the Jaeger UI. Jaeger can be started in Docker. You can follow the link with the description on how to set up Jaeger UI.
Set up Span
Start a Docker container with Jaeger.
Use the following binaries to download Jaeger.
To start the Jaeger container, run the following code in the terminal:
docker run -d --name jaeger \ -e COLLECTOR_ZIPKIN_HTTP_PORT=9411 \ -e COLLECTOR_OTLP_ENABLED=true \ -p 6831:6831/udp \ -p 6832:6832/udp \ -p 5778:5778 \ -p 16686:16686 \ -p 4317:4317 \ -p 4318:4318 \ -p 14250:14250 \ -p 14268:14268 \ -p 14269:14269 \ -p 9411:9411 \ jaegertracing/all-in-one:1.38Go to localhost:16686 to see the spans in Jaeger UI.
The trace endpoint can be defined with property:
idea.diagnostic.opentelemetry.otlpIn case the flag is true, its value is set to default: http://127.0.0.1:4318/
OpenTelemetry Metrics
The OpenTelemetry metrics are stored in the logs folder. For its correct location, refer to the documentation.
open-telemetry-metrics.csv
The metrics reporting path can be defined with idea.diagnostic.opentelemetry.metrics.file.
open-telemetry-connection-metrics.gz
This file is available starting with the PyCharm version 2023.2.
open-telemetry-lux-metrics.gz
Flags for Remote Development OpenTelemetry spans and metrics
rdct.diagnostic.otlp
This is enabled only at the backend side.
The backend spans are accumulated at the backend side and sent to the front-end side. All the spans, both backend’s and frontend’s are sent from the frontend to the defined endpoint.
rdct.connection.metrics.enabled
This should be enabled on both sides.
This is available since the PyCharm version 2023.2.
lux.metrics.enabled
Remote Development metrics
These metrics are recorded one time per second.
They include the following:
rdct.FramedByteChannel.sentFrames.projectName: number of frames sent over TLS (at the backend and frontend)rdct.CountingByteChannel.readBytesRate.projectName: number of bytes received over TLS (at at the backend and frontend sides sides)rdct.CountingByteChannel.writeBytesRate.projectName: number of bytes sent over TLS (at the backend and frontend)rdct.FramedByteChannel.receivedFrames.projectName: number of frames received over TLS (at the backend and frontend)rdct.ThinClientRdPortServerSocketWrapper.sentBytes.projectName: bytes sent from a user application (such as browser) in a thin frontend process via the port forwarding channelrdct.ThinClientRdPortServerSocketWrapper.receivedBytes.projectName: bytes received from a user application (such as browser) in thin frontend process via the port forwarding channel
FramedByteChannel is the byte messages channel over the TLS between the fronten and backend. Bytes are accumulated in frames and sent or received via channel.
rdct.FramedByteChannel.receivedFramesLenRate.projectName: the length of frames received over TLS (at both backend and frontend)rdct.FramedByteChannel.sentFramesLenRate.projectName: the length of frames sent over TLS (at both backend and frontend)rdct.TransportWrapperImpl.connectionState.projectName: the ordinal of the connection state: 0 -> "NOT_CONNECTED", 1 -> "CONNECTED", 2 -> "CLOSED", 3 -> "DECLINED", 4 -> "FAULTED" (at both backend and frontend)rdct.ThinClientDiagnosticsService.pingDirect: thin client ping value (at the frontend)rdct.ThinClientDiagnosticsService.pingUI: the front-end UI thread ping value (at the frontend)
ThinClientDiagnosticsService is per-project front-end side diagnostic service that collects the frontend-backend connection info. Ping metrics are protocol requests processed by the backend process. pingDirect is processed in background thread, pingUI - in EDT. If we do not have a single ping, we assume that either we have problems with the network (perhaps the channel is clogged with some data), or our process is frozen. If we have pingDirect, but no pingUI, then we assume that the UI hangs on the backend. A -1 value is set for the ping property in case response from the backend is not quickly received. rdct.ThinClientDiagnosticsService.connectionState: the ordinal of the frontend connection state: 0 -> WireNotConnected, 1 -> NoPings, 2 -> NoUiThreadPing, 3 -> Connected (at the frontend)
Report connection issues
If you encounter problems with the connection, use the following workflow to collect the data for further investigation of the problems:
Collect Data
In the IDE, from the main menu, select Help | Edit Custom Properties.
Add the following property to the
idea.propertiesfile:rdct.connection.metrics.enabledYou need to enable this property at both the backend and frontend.
The connection metrics will be reported with one second period.
After the issue with connection is reproduced, collect the generated metrics files that will be written in logs folder both at frontend and backend. These files will have
*.gzresolution and file name will look as follows:open-telemetry-connection-metrics.2023-03-29-15-48-40.gzCreate an archive with all the metrics files from the logs folders and attach it to the issue created in the corresponding YouTrack project.