Skip to main content
Version: Current

HTTP tracing with OpenTelemetry

Nussknacker can run with the OpenTelemetry Java agent to trace request-response scenarios: an incoming request, the HTTP calls a scenario makes and their responses. Tracing is optional and experimental; it is off unless you install the agent.

Correlation IDs and trace IDs​

A scenario log carries two Nussknacker identifiers, described in Scenario logs:

  • correlation_id - the value of the request's trace-id header. It is an arbitrary value, not a W3C trace ID.
  • context_id - identifies the processing of one record within an engine run and can repeat after a restart.

OpenTelemetry adds its own trace_id, span_id and trace_flags to the logging context when the agent's Logback MDC instrumentation is enabled. Keep all of them: Nussknacker does not derive OpenTelemetry identifiers from its own IDs.

What the profile enables​

examples/observability/http-tracing.properties enables only the instrumentation an HTTP scenario needs:

  • Pekko HTTP and actors, for incoming requests,
  • the Scala fork-join pool and Java executors, so that the context follows asynchronous continuations,
  • async-http-client, for the HTTP calls of HTTP and OpenAPI components and of the MLflow integration,
  • Logback MDC, which adds the trace and span IDs to log entries,
  • the OpenTelemetry API bridge, so that Nussknacker's HTTP backend sees the agent's active context.

Nussknacker's HTTP backend binds the response callbacks of each call to the context active when the call was sent, so a trace continues after the response is read. Calls made without an active context use the ordinary backend.

The profile disables the agent's log and metric exporters: logs keep going through your logging configuration. If that configuration exports logs over OTLP, map the trace and span IDs to the corresponding OTLP fields there. Enabling the agent's log exporter as well would create a second log stream.

Enabling the agent​

Mount the agent JAR and the profile into the container that runs the scenarios, then append these options to the existing JVM options without replacing the memory settings:

-javaagent:/opt/observability/opentelemetry-javaagent-2.31.1.jar
-Dotel.javaagent.configuration-file=/opt/observability/http-tracing.properties

Set OTEL_EXPORTER_OTLP_TRACES_ENDPOINT to the full HTTP/protobuf URL of your trace receiver, including /v1/traces, and OTEL_EXPORTER_OTLP_TRACES_HEADERS from a secret when the receiver requires authentication.

The agent must run in the JVM that executes the scenario. In the Designer it covers the embedded request-response engine; a separate Lite runtime needs its own agent and profile.

Limits​

  • The profile samples 5% of new root traces and follows the sampling decision of an incoming parent, so traffic that arrives already sampled can exceed 5%.
  • The span queue holds at most 2048 spans, exported in batches of 512 with a 10-second timeout. It is kept in memory, so spans can be dropped under load or when the receiver is unavailable.
  • Tracing covers the HTTP request path. Flink operators, joins, Kafka records and checkpoint replays are not traced: a join of several records needs its own propagation design.
  • The agent adds overhead even to unsampled requests. Measure latency, CPU, allocation and dropped spans on representative scenarios before enabling it in production.

See also: supported libraries, agent configuration, SDK configuration.