OpenTelemetry Distributed Tracing

How OpenTelemetry distributed tracing improves performance visibility and incident response for Fusion deployments

OpenTelemetry (OTel) distributed tracing gives request-level visibility across the Fusion microservice stack. Where Prometheus provides aggregate metrics, OTel traces individual requests from the moment they enter the API Gateway through the Query Service and into Solr, making it possible to pinpoint exactly which service is responsible for a slowdown.

Version and deployment applicability:

• Self-hosted Fusion 5.17 and later: You can configure the OTLP exporter to send traces to a collector you provide. See Configure a customer-provided collector.
• Self-hosted Fusion 5.9.x: OTLP export is not available. Traces are exported using Zipkin instead. See Fusion 5.9.x: sending observability data to your own platform.

The services that support OpenTelemetry are API Gateway, Query, Indexing, Admin, and Apps Manager.

How it works

The following flow shows the default pipeline used in Lucidworks-managed environments, where spans are sent to Grafana Tempo. Self-hosted Fusion 5.17 and later uses the same span instrumentation, but you can redirect the OTel Collector output to a collector you provide instead of Grafana Tempo. See Configure a customer-provided collector.

1. API Gateway creates the root span

   The API Gateway receives the incoming request and creates the root trace span. It propagates trace context downstream using a traceparent header. Only query-path traffic is sampled, which keeps overhead minimal.

2. Query Service creates child spans

   The Query Service picks up the trace context and creates child spans for each integration it calls. The following spans are created for every traced request:

   • Solr span: captures query execution time within Solr.
   • Feign/downstream span: captures HTTP calls made to downstream services.

   The following span is optional and enabled per environment:

   • Kafka span: captures Kafka producer and consumer calls made during query processing.
3. OTel Collector batches the spans

   The OTel Collector receives spans from the Query Service and batches them for export.

4. Grafana Tempo stores the traces

   The OTel Collector forwards the batched spans to Grafana Tempo, which stores them for search and analysis.

5. Grafana Dashboard surfaces the traces

   Traces stored in Grafana Tempo are available in the Grafana Dashboard, where Lucidworks engineers can search, filter, and drill into individual traces to identify performance bottlenecks.

Viewing traces in Grafana

You can use the Grafana Tempo dashboard to search and visualize traces. A trace can be filtered by service, time window, latency threshold, or error status. Drilling into a trace shows the full span tree: API Gateway root span, Query Service child spans, and Solr spans nested within them.

This makes the following workflow possible:

1. Open Grafana and navigate to the Tempo traces dashboard.
2. Select the time window that corresponds to the reported latency spike or other incident.
3. Identify the slowest trace for that window.
4. Drill into the span tree to see which service or integration was the bottleneck.

This full workflow is achievable in under five minutes, without code changes or log searches.

Instrumented integrations

Integration	What it covers	How it’s enabled
Solr	Query execution time within Solr for each request	Configurable
Kafka	Kafka producer/consumer calls made during query processing	Configurable
Feign HTTP client (http)	HTTP calls between Fusion microservices	Configurable
JDBC	Database calls made by Fusion services	Configurable on the Indexing and Apps Manager services only

Solr, Kafka, Feign HTTP client, and JDBC tracing are each controlled by a boolean Helm value, opentelemetry.integrations.<name> (for example, opentelemetry.integrations.solr), set in each service’s Helm chart. Each defaults to false, so set the ones you want to true to enable that integration’s spans. Not every integration is available on every service — see Configure a customer-provided collector for which services support which integrations. Trace volume and overhead can be tuned per integration without touching application code or requiring a Fusion release.

opentelemetry:
  integrations:
    solr: false
    kafka: false
    http: false
    jdbc: false

Configure a customer-provided collector (self-hosted Fusion 5.17 and later)

Self-hosted Fusion 5.17 and later can export OTLP traces directly to a collector you provide, instead of Grafana Tempo. Five services support this:

• API Gateway
• Query
• Indexing
• Admin
• Apps Manager

The Query, Indexing, Admin, and Apps Manager services all share the same configuration structure; only the API Gateway differs, since it’s the entry point that also controls trace propagation to each downstream service.

Both sides of a trace must be configured consistently. If a downstream service is listed as true in the API Gateway’s opentelemetry.services map, that service must also have opentelemetry.enabled: true in its own Helm values — otherwise the trace chain breaks or spans are orphaned.

Gateway services.<name>	Service opentelemetry.enabled	Result
true	true	Full end-to-end trace: gateway and service spans linked
true	false	Gateway exports spans; service does not continue the trace — broken chain
false	true	Service exports spans; gateway does not propagate context — orphaned spans
false	false	No tracing for that service

The only valid states are both enabled or both disabled for a given service.

API Gateway

The API Gateway is the single entry point for external traffic. It creates the root span and, for each service enabled in opentelemetry.services, propagates trace context downstream using a traceparent header. Enabling OTel on the gateway automatically disables Spring Cloud Sleuth.

Value	Description
opentelemetry.enabled	Master toggle. Enables OTLP export and disables Spring Cloud Sleuth.
opentelemetry.endpoint	The OTLP endpoint of your collector. Optional — defaults to a collector on the same node, using the pod’s NODE_IP.
opentelemetry.protocol	The OTLP protocol to use: grpc or http.
opentelemetry.services.query	Propagates trace context to the Query service and exports gateway spans for requests routed to it.
opentelemetry.services.indexing	Propagates trace context to the Indexing service.
opentelemetry.services.admin	Propagates trace context to the Admin service.
opentelemetry.services.appsmanager	Propagates trace context to the Apps Manager service.

opentelemetry:
  enabled: false
  endpoint: "http://$(NODE_IP):4317"
  protocol: grpc
  services:
    query: false
    indexing: false
    admin: false
    appsmanager: false

Downstream services: Query, Indexing, Admin, Apps Manager

These services share the same configuration structure under opentelemetry in their Helm values.

Value	Description
opentelemetry.enabled	Master toggle. Must be true if this service is also enabled in the API Gateway’s services map.
opentelemetry.endpoint	The OTLP endpoint of your collector. Optional — defaults to a collector on the same node, using the pod’s NODE_IP.
opentelemetry.protocol	The OTLP protocol to use: grpc or http.
opentelemetry.serviceVersion	Version tag attached to all spans and logs for this service.
opentelemetry.logging.enabled	Exports application logs to the OTLP collector, independent of tracing.
opentelemetry.tracing.enabled	Enables span export. Can be disabled independently of the master toggle, for example to export logs only.
opentelemetry.tracing.samplingRatio	Fraction of traces to sample, from 0.0 to 1.0.
opentelemetry.integrations.*	Enables span coverage per integration. See Instrumented integrations — not every integration is available on every service.

opentelemetry:
  enabled: false
  endpoint: "http://$(NODE_IP):4317"
  protocol: grpc
  serviceVersion: "1.0.0"
  logging:
    enabled: false
  tracing:
    enabled: true
    samplingRatio: 1.0
  integrations:
    solr: false
    kafka: false
    http: false
    jdbc: false

Point the endpoint at any OTLP-compatible collector. Traces continue to include the same span coverage described in Instrumented integrations; only the export destination changes.

Fusion 5.9.x: sending observability data to your own platform

Self-hosted Fusion 5.9.x does not support OTLP export. Use the following supported paths to send observability data to your own platform instead.

Traces

Fusion 5.9.x exports traces using Spring Cloud Sleuth with Zipkin/Brave, in Zipkin v2 format. Configure the export endpoint with:

Value	Description
zipkin.enabled	Enables trace export.
zipkin.baseUrl	The base URL of your Zipkin-compatible collector.

Most modern observability platforms and OTel Collector builds accept Zipkin v2 format through a Zipkin receiver.

Metrics

Fusion exposes metrics in Prometheus format. A customer-provided OTel Collector can scrape these endpoints using a prometheus receiver. See Prometheus, Grafana, and Loki in Fusion for the available endpoints.

Logs

Fusion writes structured logs to stdout. A customer-provided OTel Collector can ingest these using a filelog receiver. See Grafana Loki for log exploration for the logging format Fusion produces.