Implementing Distributed Tracing with OpenTelemetry

You know the feeling: a request looks fine at the edge, logs are clean, and then users start complaining that “sometimes it’s slow.” By the time you follow the breadcrumb trail across a load balancer, API gateway, frontend service, auth service, queue worker, and downstream database call, the original request is long gone and your context is mostly guesses.

That’s the core problem distributed tracing solves. Once a request crosses service boundaries or asynchronous workflows, logs alone stop being enough. Metrics tell you that something is wrong. Traces help you answer where and why.

OpenTelemetry has become the practical standard for tracing because it avoids vendor lock-in, gives you a consistent instrumentation model, and works across languages and runtimes. If you’re building cloud-native systems in 2026 and you still treat tracing as optional, you’re accepting avoidable downtime and slower incident response.

The real reason tracing matters

Tracing becomes essential when a single user action fans out across multiple components:

• An API request hits a gateway
• The gateway calls an authentication service
• The application service reads from cache and database
• A background job publishes an event
• Another service consumes that event and performs a write
• A third service emits a notification

When latency spikes in this path, the overall number is not enough. You need to know:

• Which hop consumed the time
• Whether the delay was network, queueing, CPU, or I/O
• Whether the issue is isolated or part of a wider regression
• Which dependency is failing under load

That is what a trace gives you: a timeline of the request, enriched with service names, spans, metadata, and timing boundaries.

“If you can’t trace a request across your system, you don’t really operate a distributed system — you operate a collection of guesses.”

— Practical observability rule of thumb

OpenTelemetry fundamentals

OpenTelemetry is a vendor-neutral observability framework that standardizes how you generate, collect, and export telemetry data. It covers traces, metrics, and logs, but in this article we’ll focus on traces.

At a high level, the trace model consists of four key concepts:

Traces and spans

A trace is the whole story. A span is one chapter in that story. Each span has:

• A name, such as GET /checkout or publish_order_created
• Start and end timestamps
• Attributes, like http.status_code=200 or db.system=postgresql
• Parent-child relationships
• Optional events and status codes

Spans form a tree-like structure, although asynchronous systems can make the graph more complex in practice.

Context propagation

Context propagation is the mechanism that allows a trace to continue across process boundaries. Without propagation, each service invents a new trace and you lose the ability to connect the dots.

Most systems use W3C Trace Context headers, commonly:

• traceparent
• tracestate

OpenTelemetry supports this standard, which is one of the reasons it works well across ecosystems.

Exporters and collectors

Instrumented applications do not usually send spans directly to your final observability backend. Instead, they send data to an OpenTelemetry Collector, which can:

• Receive telemetry from applications
• Batch and transform spans
• Sample or filter data
• Enrich spans with metadata
• Forward to storage backends like Grafana Tempo, Jaeger, Datadog, Honeycomb, or vendor-specific systems

This architecture is better than shipping directly from every service to every backend. It centralizes policy and makes instrumentation simpler.

Auto-instrumentation vs manual instrumentation

One of the first decisions you’ll make is whether to rely on auto-instrumentation or add spans manually.

Auto-instrumentation

Auto-instrumentation hooks into popular frameworks and libraries to create spans with minimal code changes.

Pros

• Fastest way to get coverage
• Low engineering effort
• Useful for baseline visibility
• Great for libraries like HTTP clients, server frameworks, and database drivers

Cons

• Can produce noisy spans
• May miss business-level context
• Limited control over naming and attributes
• Sometimes hard to debug when library support is incomplete

Manual instrumentation

Manual instrumentation means you explicitly define spans in the parts of the code that matter most.

Pros

• Better naming and semantic clarity
• Captures business events, not just infrastructure calls
• More control over tags, events, and error handling
• Helps identify “meaningful units of work”

Cons

• More code
• Requires discipline and conventions
• Easy to over-instrument or under-instrument

What I recommend

Use both.

Start with auto-instrumentation to get broad coverage, then manually instrument the parts of the system that define business latency or failure modes. You do not need a span for every function call. You need spans where the answer to “what happened?” matters.

Propagating context across HTTP, gRPC, queues, and background jobs

Propagation is where tracing succeeds or fails. If context breaks at any boundary, the trace becomes fragmented.

HTTP services

HTTP is the easiest case. Most OpenTelemetry SDKs and instrumentations can automatically inject and extract trace headers.

Typical flow:

1. Incoming request arrives with trace headers
2. Server extracts context
3. Application creates spans for the request
4. Outgoing HTTP client injects updated context
5. Downstream service continues the trace

Example in JavaScript/Node.js:

import { context, trace } from '@opentelemetry/api';

const tracer = trace.getTracer('checkout-service');

async function handleCheckout(req, res) {
  const span = tracer.startSpan('POST /checkout');
  try {
    await context.with(trace.setSpan(context.active(), span), async () => {
      await reserveInventory(req.body.items);
      await chargePayment(req.body.payment);
    });
    res.status(200).send({ ok: true });
  } catch (err) {
    span.recordException(err);
    span.setStatus({ code: 2, message: err.message });
    res.status(500).send({ error: 'checkout_failed' });
  } finally {
    span.end();
  }
}

gRPC

gRPC is similar, but the metadata layer is the propagation mechanism. OpenTelemetry libraries typically handle this for you if properly configured.

The main failure mode is custom interceptors that forget to pass context through. If you add middleware for auth or logging, verify that trace context is preserved.

Queues and asynchronous workflows

Queues are where many teams lose traces.

Once a request becomes asynchronous, you need a strategy for continuing the trace across producer and consumer boundaries. That usually means:

• Serialize trace context into message headers or metadata
• Extract context on the consumer side
• Create a new span with the consumed message as child or linked span
• Preserve correlation identifiers where useful

A common pattern is to model the producer span as the parent and the consumer processing span as a continuation or linked span, depending on queue semantics.

sequenceDiagram
    participant Client
    participant API
    participant Queue
    participant Worker
    participant DB

    Client->>API: HTTP request with traceparent
    API->>API: Create server span
    API->>Queue: Publish message with propagated context
    Queue-->>Worker: Deliver message
    Worker->>Worker: Extract context and start consumer span
    Worker->>DB: Write result
    DB-->>Worker: Ack
    Worker-->>API: Emit completion event

Background jobs

Background jobs are tricky because they often lose request context intentionally. That’s fine, but you still want a trace for the job execution itself.

Best practice:

• Attach the originating request ID as an attribute
• Create a new root trace if the job starts much later
• Use links if the job was triggered by multiple upstream events
• Track queue wait time separately from processing time

Backend choices, collectors, sampling, and storage

Instrumentation is only half the story. You also need a pipeline that won’t melt under load.

Collector architecture

The OpenTelemetry Collector often sits between services and your backend. A typical deployment looks like this:

flowchart LR
    A[Service A] -->|spans| C[OpenTelemetry Collector]
    B[Service B] -->|spans| C
    D[Worker] -->|spans| C
    C --> E[Sampling / Batch / Transform]
    E --> F[Trace Backend]
    F --> G[UI / Analysis]

You can run the collector as:

• A sidecar per pod
• A daemonset on Kubernetes
• A centralized gateway
• A hybrid setup

Each option has tradeoffs.

Sampling strategy

Sampling is necessary because collecting every trace from a high-throughput system can be expensive.

Common approaches include:

• Head sampling: decide at trace start whether to keep it
• Tail sampling: decide after seeing the full trace
• Probabilistic sampling: keep a percentage of traces
• Rule-based sampling: retain errors, slow requests, or specific endpoints

Head sampling is simpler and cheaper. Tail sampling is usually more useful for debugging because you can keep the slow or failing traces you actually care about.

Storage backends

Your backend choice depends on query patterns, retention needs, and vendor strategy.

Popular destinations include:

• Grafana Tempo
• Jaeger
• Datadog APM
• Honeycomb
• New Relic
• Elastic Observability

The right answer is not always the one with the fanciest UI. Ask:

• Can it handle our cardinality and ingest volume?
• Can we query by latency, error, service, and route efficiently?
• How long do we need retention?
• Do we need exemplars and metrics correlation?
• What is the cost model at scale?

How to analyze traces in practice

A beautiful trace UI is useless if the team doesn’t know how to use it.

The fastest way to get value is to focus on a few concrete questions:

1. Where is the time spent?
2. Which dependency is slow or failing?
3. Is the issue isolated to one service or systemic?
4. Are we seeing queue buildup, retries, or cascading failure?
5. Is tail latency coming from cold starts, lock contention, or downstream saturation?

Finding bottlenecks

When you inspect a slow trace, look for:

• Long spans with little child activity
• Sequential calls that could be parallelized
• Repeated retries masking an upstream dependency issue
• Database calls with unexpected latency
• Network hops dominating the request

Sometimes the culprit is not where you think. A “slow frontend” bug is often a backend cache miss, a noisy neighbor issue, or an auth service bottleneck.

Dependency failures

Tracing is especially helpful when a service failure cascades.

Examples:

• A payment provider times out, causing retries and queue buildup
• A database pool is exhausted, increasing wait time across unrelated requests
• A slow internal API causes thread starvation
• An unbounded retry policy amplifies load during partial outages

Use spans to confirm whether your retries are actually helping or simply making a bad incident worse.

Tail latency

P50 tells you what is normal. P95 and P99 tell you what users complain about.

A request can look healthy on average while a small fraction of requests are painfully slow. Tracing helps identify the specific pathologies behind these outliers.

In practice, tail latency often comes from:

• Cold starts
• Connection pool contention
• Burst traffic
• Cache stampedes
• GC pauses
• Downstream retries
• Noisy neighbors in shared infrastructure

Common mistakes and how to avoid them

A tracing rollout can fail in predictable ways.

1. Instrumenting everything equally

Not all spans are worth keeping. Focus on user-facing paths, expensive operations, and external dependencies.

2. Ignoring naming conventions

Poor span names make analysis painful. Prefer meaningful names like POST /orders, charge_payment, or fetch_user_profile over generic method names.

3. Losing context at async boundaries

Queues, jobs, and event handlers often drop trace metadata. Build propagation into your message contracts early.

4. Treating traces as logs

Traces are not a dumping ground for every debug detail. Use attributes for structured context, but keep payloads lean.

5. Sampling too aggressively too early

If you drop the very traces you need during incidents, you’ve built an observability system that only works when nothing is wrong.

6. Skipping semantic conventions

OpenTelemetry semantic conventions exist so your data is queryable and consistent. Use them. Inconsistent tags create chaos across teams and services.

7. Not correlating traces with logs and metrics

A trace is strongest when it links to log lines and metric spikes. Add trace IDs to logs and surface exemplars where supported.

A practical rollout strategy

The best tracing rollout is incremental. Don’t try to boil the ocean.

Phase 1: Cover the critical path

Start with:

• Edge gateway or API layer
• One or two key user journeys
• Primary downstream dependencies
• Database and external API calls

This gives you immediate incident value without a huge implementation burden.

Phase 2: Add async propagation

Extend tracing into:

• Message queues
• Event consumers
• Background workers
• Scheduled jobs

At this stage, you’ll start seeing request paths that were previously invisible.

Phase 3: Tune sampling and retention

Once volume grows:

• Keep all errors
• Keep slow traces above a threshold
• Reduce traces for healthy high-volume endpoints
• Set retention based on incident investigation needs

Phase 4: Correlate with metrics and logs

Make traces part of the incident workflow:

• Trace IDs in structured logs
• Alert links to exemplar traces
• Dashboards that surface slow span breakdowns
• Service ownership metadata for faster routing

Phase 5: Standardize across teams

Establish conventions for:

• Span names
• Required attributes
• Error handling
• Sampling defaults
• Collector configuration
• Dashboard and alert ownership

That prevents observability from becoming fragmented across squads.

A reference architecture worth using

If you want a sensible default, this is the architecture I’d recommend for most cloud-native teams:

1. Use OpenTelemetry SDKs in all primary services
2. Enable auto-instrumentation first, then add manual spans for business-critical flows
3. Propagate W3C trace context everywhere possible
4. Deploy the OpenTelemetry Collector as a Kubernetes daemonset or gateway
5. Use tail sampling or hybrid sampling for slow/error traces
6. Export to a backend that supports fast search and long enough retention for incident review
7. Correlate traces with logs and metrics in a single observability workflow

Conclusion

Distributed tracing is not a luxury feature for mature platforms. It is a basic requirement once your request path spans multiple services or asynchronous boundaries. Without it, latency debugging becomes guesswork, incident response slows down, and architectural complexity compounds.

OpenTelemetry gives you a pragmatic path forward: standardize the instrumentation layer, propagate context consistently, and route telemetry through a collector so you can evolve the backend over time. Start small, focus on the flows that matter, and expand systematically.

If you’re rolling this out in an existing platform, resist the urge to instrument every service at once. Pick one painful user journey, wire up traces end to end, and let that success become the template for everything else.

The teams that do tracing well don’t just get better dashboards. They get faster debugging, better architectural decisions, and fewer “we think it might be…” moments during incidents.