axum-tracing-opentelemetry

Middlewares and tools to integrate axum + tracing + opentelemetry.

Read OpenTelemetry header from incoming request
Start a new trace if no trace found in the incoming request
Trace is attached into tracing'span

For examples, you can look at:

the examples folder
davidB/sandboxaxumobservability: Sandbox to experiment axum and observability. This example shows also propagation of the trace between tracing span and service (via reqwest).

```rust //... use axumtracingopentelemetry::opentelemetrytracinglayer;

[tokio::main]

async fn main() -> Result<(), axum::BoxError> { // very opinionated init of tracing, look as is source to make your own axumtracingopentelemetry::tracingsubscriberext::init_subscribers()?;

let app = app();
// run it
let addr = &"0.0.0.0:3000".parse::<SocketAddr>()?;
tracing::warn!("listening on {}", addr);
axum::Server::bind(&addr)
    .serve(app.into_make_service())
    .with_graceful_shutdown(shutdown_signal())
    .await?;
Ok(())

}

fn app() -> Router { // build our application with a route Router::new() .route("/", get(health)) // request processed inside span // opentelemetrytracinglayer setup TraceLayer, that is provided by tower-http so you have to add that as a dependency. .layer(opentelemetrytracinglayer()) .route("/health", get(health)) // request processed without span / trace }

async fn shutdownsignal() { //... opentelemetry::global::shutdowntracer_provider(); } ```

To configure opentelemetry tracer & tracing, you can use function fom axum_tracing_opentelemetry::tracing_subscriber_ext, but they are very opinionated (and WIP to make them more customizable and friendly), so we recommend to make your own composition, but look at the code (to avoid some issue) and share your feedback.

``rust pub fn build_loglevel_filter_layer() -> tracing_subscriber::filter::EnvFilter { // filter what is output on log (fmt) // std::env::set_var("RUST_LOG", "warn,axum_tracing_opentelemetry=info,otel=debug"); std::env::set_var( "RUST_LOG", format!( //axumtracingopentelemetryshould be a level info to emit opentelemetry trace & span //otel::setup` set to debug to log detected resources, configuration read and infered "{},axumtracingopentelemetry=info,otel=debug", std::env::var("RUSTLOG") .orelse(|| std::env::var("OTELLOGLEVEL")) .unwraporelse(|| "info".tostring()) ), ); EnvFilter::fromdefault_env() }

pub fn buildotellayer() -> Result, BoxError> where S: Subscriber + for<'a> LookupSpan<'a>, { use crate::{ initpropagator, //stdio, otlp, resource::DetectResource, }; let otelrsrc = DetectResource::default() //.withfallbackservicename(env!("CARGOPKGNAME")) //.withfallbackserviceversion(env!("CARGOPKGVERSION")) .build(); let oteltracer = otlp::inittracer(otelrsrc, otlp::identity)?; // to not send trace somewhere, but continue to create and propagate,... // then send them to axum_tracing_opentelemetry::stdio::WriteNoWhere::default() // or to std::io::stdout() to print // // let oteltracer = // stdio::inittracer(otelrsrc, stdio::identity, stdio::WriteNoWhere::default())?; initpropagator()?; Ok(tracingopentelemetry::layer().withtracer(oteltracer)) } ```

To retrieve the current trace_id (eg to add it into error message (as header or attributes))

rust let trace_id = axum_tracing_opentelemetry::find_current_trace_id(); json!({ "error" : "xxxxxx", "trace_id": trace_id})

To also inject the trace id into the response (could be useful for debugging) uses the layer response_with_trace_layer

rust // build our application with a route Router::new() ... // include trace context as header into the response .layer(response_with_trace_layer())

Configuration based on environment variable

To ease setup and compliancy with Opentelemetry SDK configuration, the configuration can be done with the following environment variables (see sample init_tracing() above):

OTEL_EXPORTER_OTLP_TRACES_ENDPOINT fallback to OTEL_EXPORTER_OTLP_ENDPOINT for the url of the exporter / collector

OTEL_EXPORTER_OTLP_TRACES_PROTOCOL fallback to OTEL_EXPORTER_OTLP_PROTOCOL, fallback to auto-detection based on ENDPOINT port

OTEL_SERVICE_NAME for the name of the service

OTEL_PROPAGATORS for the configuration of propagator

OTEL_TRACES_SAMPLER & OTEL_TRACES_SAMPLER_ARG for configuration of the sampler

In the context of kubernetes, the above environment variable can be injected by the Opentelemetry operator (via inject-sdk):

yaml apiVersion: apps/v1 kind: Deployment spec: template: metadata: annotations: # to inject environment variables only by opentelemetry-operator instrumentation.opentelemetry.io/inject-sdk: "opentelemetry-operator/instrumentation" instrumentation.opentelemetry.io/container-names: "app" containers: - name: app

Or if you don't setup inject-sdk, you can manually set the environment variable eg

yaml apiVersion: apps/v1 kind: Deployment spec: template: metadata: containers: - name: app env: - name: OTEL_SERVICE_NAME value: "app" - name: OTEL_EXPORTER_OTLP_PROTOCOL value: "grpc" # for otel collector in `deployment` mode, use the name of the service # - name: OTEL_EXPORTER_OTLP_ENDPOINT # value: "http://opentelemetry-collector.opentelemetry-collector:4317" # for otel collector in sidecar mode (imply to deploy a sidecar CR per namespace) - name: OTEL_EXPORTER_OTLP_ENDPOINT value: "http://localhost:4317" # for `daemonset` mode: need to use the local daemonset (value interpolated by k8s: `$(...)`) # - name: OTEL_EXPORTER_OTLP_ENDPOINT # value: "http://$(HOST_IP):4317" # - name: HOST_IP # valueFrom: # fieldRef: # fieldPath: status.hostIP

examples/otlp

In a terminal, run

```sh ❯ cd examples/otlp

or direnv allow

❯ export OTELEXPORTEROTLPTRACESENDPOINT=http://localhost:4317 ❯ export OTELTRACESSAMPLER=alwayson ❯ cargo run warning: axum-tracing-opentelemetry (lib) generated 1 warning Compiling examples-otlp v0.1.0 (/home/david/src/github.com/davidB/axum-tracing-opentelemetry/examples/otlp) Finished dev [unoptimized + debuginfo] target(s) in 2.38s Running /home/david/src/github.com/davidB/axum-tracing-opentelemetry/examples/target/debug/examples-otlp 0.000043962s INFO axumtracingopentelemetry::tools::tracingsubscriberext: init logging & tracing at /home/david/src/github.com/davidB/axum-tracing-opentelemetry/src/tools/tracingsubscriber_ext.rs:82 on main

0.000472423s DEBUG otel::resource: key: service.name, value: unknown_service at /home/david/src/github.com/davidB/axum-tracing-opentelemetry/src/tools/resource.rs:84 on main 0.000213920s INFO examples_otlp: try to call `curl -i http://127.0.0.1:3003/health` (with NO trace) at src/main.rs:72 on main 0.000501468s DEBUG otel::resource: key: os.type, value: linux at /home/david/src/github.com/davidB/axum-tracing-opentelemetry/src/tools/resource.rs:84 on main 0.000549097s DEBUG otel::setup: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT: "http://localhost:4317" at /home/david/src/github.com/davidB/axum-tracing-opentelemetry/src/tools/otlp.rs:22 on main 0.000570727s DEBUG otel::setup: OTEL_EXPORTER_OTLP_TRACES_PROTOCOL: "grpc" at /home/david/src/github.com/davidB/axum-tracing-opentelemetry/src/tools/otlp.rs:23 on main 0.000623135s DEBUG otel::setup: OTEL_TRACES_SAMPLER: "always_on" at /home/david/src/github.com/davidB/axum-tracing-opentelemetry/src/tools/otlp.rs:80 on main 0.000928215s DEBUG otel::setup: OTEL_PROPAGATORS: "tracecontext,baggage" at /home/david/src/github.com/davidB/axum-tracing-opentelemetry/src/tools/mod.rs:94 on main 0.000190306s WARN examples_otlp: listening on 0.0.0.0:3003 at otlp/src/main.rs:15 on main 0.000222566s INFO examples_otlp: try to call `curl -i http://127.0.0.1:3003/` (with trace) at otlp/src/main.rs:16 on main 0.000240970s INFO examples_otlp: try to call `curl -i http://127.0.0.1:3003/heatlh` (with NO trace) at otlp/src/main.rs:17 on main

... ```

Into an other terminal, call the / (endpoint with opentelemetry_tracing_layer and response_with_trace_layer)

```sh ❯ curl -i http://127.0.0.1:3003/ HTTP/1.1 200 OK content-type: application/json content-length: 50 traceparent: 00-b2611246a58fd7ea623d2264c5a1e226-b2c9b811f2f424af-01 tracestate: date: Wed, 28 Dec 2022 17:04:59 GMT

{"mytraceid":"b2611246a58fd7ea623d2264c5a1e226"} ```

call the /health (endpoint with NO layer)

```sh ❯ curl -i http://127.0.0.1:3003/health HTTP/1.1 200 OK content-type: application/json content-length: 15 date: Wed, 28 Dec 2022 17:14:07 GMT

{"status":"UP"} ```

For local dev / demo

To collect and visualize trace on local, one of the simplest solution:

```sh

launch Jaeger with OpenTelemetry, Jaeger, Zipking,... mode.

see https://www.jaegertracing.io/docs/1.41/getting-started/#all-in-one

nerctl or docker or any container runner

nerdctl run --rm --name jaeger \ -e COLLECTORZIPKINHOSTPORT:9411 \ -e COLLECTOROTLP_ENABLED:true \ -p 6831:6831/udp \ -p 6832:6832/udp \ -p 5778:5778 \ -p 16686:16686 \ -p 4317:4317 \ -p 4318:4318 \ -p 14250:14250 \ -p 14268:14268 \ -p 14269:14269 \ -p 9411:9411 \ jaegertracing/all-in-one:1.41

open http://localhost:16686 ```

Then :

setup env variable (or not), (eg see .envrc)

launch your server

send the request

copy trace_id from log (or response header)

paste into Jaeger web UI

Compatibility

| axum | axum-tracing-opentelemetry | |------|----------------------------| | 0.6 | latest - 0.6 | | 0.5 | 0.1 - 0.5 |

Changelog - History

0.10

💥 default configuration for otlp Sampler is no longer hardcoded to always_on, but read environment variables OTEL_TRACES_SAMPLER, OTEL_TRACES_SAMPLER_ARG

✨ provide opinionated tracing_subscriber_ext

✨ log under target otel::setup detected configuration by otel setup tools

✨ add a axum layer for gRPC (#36) (wip)

0.9

add DetectResource builder to help detection for Resource Semantic Conventions | OpenTelemetry

0.8

add init_propagator to configure the global propagator based on content of the env variable OTEL_PROPAGATORS

0.7

add a layerresponse_with_trace_layer to have traceparent injected into response

improve discovery of otlp configuration based on OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_TRACES_PROTOCOL, OTEL_EXPORTER_OTLP_PROTOCOL

0.6

upgrade to axum 0.6

0.5

upgrade to opentelemetry 0.18

breaking change: upgrade opentelemetry-jaeger to 0.17 (switch from PipelineBuiler to AgentPipeline)

0.4

allow customization of tracer

add tracer to export on stdout or stderr

add tracer to export to nowhere (like /dev/null) to allow to have trace_id and the opentelemetry span & metadata on log and http response (without collector)

0.3

Allow customization of exporter pipeline

Fix name of the root span (#6)

0.2

First public release as a crate

0.1

Code originally created at part of axum-extra Add OpenTelemetry middleware by davidpdrsn · Pull Request #769 · tokio-rs/axum

Code copied and modified as part of davidB/sandboxaxumobservability: Sandbox to experiment axum and observability

Published as a standalone crate with OK from original author