Distributed Tracing Done Right: W3C TraceContext, B3, and Custom Header Propagation Explained

by

Your service fails. Somewhere. A request touched six microservices, three queues, and two third-party APIs before dying. Your logs are full of noise. Your metrics tell you something is wrong, not where. You open Jaeger and see… orphaned spans. A trace that stops dead after the first hop because nobody remembered to forward the traceparent header.

That’s the moment you really understand why trace context propagation matters — and why getting it wrong is one of the most expensive invisible bugs in a distributed system.

This article is a ground-up, practical breakdown of how trace context actually travels across service boundaries: the W3C TraceContext standard, the older B3 format that Zipkin popularized, and the cases where you end up rolling custom headers. We’ll look at the wire format, how to configure propagators in OpenTelemetry, and what will silently break your traces if you don’t pay attention.

What "Propagation" Actually Means

A trace is a tree of spans. Span A calls Service B, which calls Service C. For all of that to appear as one coherent trace in your backend, every service in the chain needs to know:

  • Which trace it’s part of (the trace ID)
  • Which span is its immediate parent (the parent span ID)
  • Whether sampling was already decided (the sampling decision)

None of this is magic. It’s just HTTP headers (or message metadata for queues). The upstream service writes them; the downstream service reads them, creates a child span with the right parent, and writes the same headers onward. If anyone in that chain drops the headers — it’s game over for correlation.

There are a few competing wire formats for encoding this information. The two you’ll actually encounter in production are W3C TraceContext and B3.

W3C TraceContext: The Modern Standard

The W3C published the TraceContext specification as a Recommendation in 2020. It’s now the default in OpenTelemetry and broadly supported across cloud providers (Google Cloud Trace, AWS X-Ray via compatibility layers, Azure Monitor, etc.).

It uses two headers.

traceparent

This is the mandatory one. It encodes the essential routing information:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

Breaking it down:

00                               - version (currently always "00")
4bf92f3577b34da6a3ce929d0e0e4736 - trace-id (128-bit, 32 hex chars)
00f067aa0ba902b7                 - parent-span-id (64-bit, 16 hex chars)
01                               - trace-flags (bit field; 01 = sampled)

The trace-flags byte is a bitmask. Right now only bit 0 is defined — the sampled flag. If it’s 01, the trace is being recorded. If it’s 00, the upstream decided to drop it, and you should honor that decision (more on this in Gotchas).

tracestate

Optional but important. It’s a vendor-specific bag of key-value pairs for carrying additional context that doesn’t fit into traceparent:

tracestate: vendor1=opaqueValue1,vendor2=opaqueValue2

AWS uses this for their X-Ray-specific fields. Datadog uses it for propagation continuity. You can add your own vendor key here, scoped to your system. The spec allows up to 32 list-members and the whole header can’t exceed 512 bytes — if you’re tempted to stuff large payloads in here, don’t.

Generating TraceContext in Code (Python + OpenTelemetry)

from opentelemetry import trace
from opentelemetry.propagate import inject, extract
from opentelemetry.propagators.composite import CompositePropagator
from opentelemetry.propagators.b3 import B3MultiFormat
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator

# W3C TraceContext is the default propagator in OTel
tracer = trace.get_tracer("my-service")

with tracer.start_as_current_span("outgoing-call") as span:
    headers = {}
    inject(headers)  # writes traceparent (and tracestate if set)
    # pass headers to your HTTP client
    response = requests.get("http://downstream/api", headers=headers)

On the receiving end:

from opentelemetry.propagate import extract

# In a Flask/FastAPI handler, request.headers is a dict-like object
ctx = extract(request.headers)
with tracer.start_as_current_span("incoming-request", context=ctx):
    # this span is now correctly parented
    pass

That’s the whole loop. Write headers on the way out, read them on the way in, let the SDK handle the formatting.

B3: The Zipkin Format That Refuses to Die

Before W3C TraceContext existed, Zipkin defined its own propagation format called B3 (the name comes from Dapper, Google’s tracing paper — "BigBrotherBird"). It predates the W3C standard by years and is still everywhere: Spring Boot, older Netflix OSS stacks, many legacy Go services, Envoy (configurable), Istio (historically defaulted to B3).

B3 comes in two flavors.

B3 Multi-Header

Spreads the context across multiple headers:

X-B3-TraceId: 4bf92f3577b34da6a3ce929d0e0e4736
X-B3-SpanId:  00f067aa0ba902b7
X-B3-ParentSpanId: aaaaabbbbbbccccc  # absent on root span
X-B3-Sampled: 1                       # "1" or "0"
X-B3-Flags: 1                         # debug flag (forces sampling)

The trace ID can be either 64-bit (16 hex chars) or 128-bit (32 hex chars). If you’re mixing old and new services, you’ll see both. 64-bit IDs will collide eventually on any system doing serious volume — treat them as legacy.

B3 Single Header

A compact form, introduced later to reduce header overhead:

b3: 4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-1-aaaaabbbbbbccccc

Format: {traceId}-{spanId}-{sampling}-{parentSpanId}. The sampling field is 1 (sample), 0 (don’t sample), or d (debug/force sample). Parent span ID is optional and omitted on the root span.

B3 in OpenTelemetry

OTel ships B3 propagators, but you have to opt into them:

# requirements: opentelemetry-propagator-b3

from opentelemetry.propagators.b3 import B3MultiFormat, B3SingleFormat
from opentelemetry.propagate import set_global_textmap
from opentelemetry.propagators.composite import CompositePropagator
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator

# Accept both W3C and B3 multi; emit W3C
set_global_textmap(CompositePropagator([
    TraceContextTextMapPropagator(),
    B3MultiFormat(),
]))

With CompositePropagator, the SDK tries each propagator in order on extraction (first match wins) and runs all of them on injection. That gives you a transition window where you emit both formats and accept either.

In OpenTelemetry Collector config:

# otel-collector-config.yaml
processors:
  # no change needed — propagation happens at the SDK level

extensions:
  health_check:

service:
  extensions: [health_check]
  pipelines:
    traces:
      receivers: [otlp]
      processors: []
      exporters: [jaeger]

For Istio, if your mesh is still defaulting to B3, you can switch globally:

# IstioOperator or MeshConfig
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  meshConfig:
    defaultConfig:
      tracing:
        # Tell the proxy which format to propagate
        custom_tags: {}
    # Envoy-level propagation format
    enableTracing: true
  values:
    pilot:
      traceSampling: 100.0

Istio 1.18+ supports W3C TraceContext natively. If you’re on an older mesh, you might be stuck emitting B3 from Envoy sidecars even if your app code uses W3C — another reason the composite propagator is your friend.

Custom Headers: When and Why

Occasionally you’re in a situation where neither W3C nor B3 applies — either you’re integrating with a proprietary APM vendor that uses its own format, or you’re propagating business-level context (tenant ID, experiment flags, request priority) alongside the tracing headers.

Custom propagation in OTel is straightforward:

from opentelemetry.propagators.textmap import TextMapPropagator, CarrierT, Getter, Setter
from opentelemetry import context
from typing import Optional, Set

# A simple propagator for a hypothetical "X-Tenant-Id" header
_TENANT_KEY = context.create_key("tenant-id")

class TenantPropagator(TextMapPropagator):
    HEADER = "X-Tenant-Id"

    def extract(self, carrier: CarrierT, context: Optional[context.Context] = None, getter: Getter = ...) -> context.Context:
        tenant = getter.get(carrier, self.HEADER)
        if tenant:
            context = context or context.get_current()
            return context.attach(context.set_value(_TENANT_KEY, tenant[0]))
        return context or context.get_current()

    def inject(self, carrier: CarrierT, context: Optional[context.Context] = None, setter: Setter = ...) -> None:
        tenant = context.get_value(_TENANT_KEY, context)
        if tenant:
            setter.set(carrier, self.HEADER, tenant)

    @property
    def fields(self) -> Set[str]:
        return {self.HEADER}

Then compose it:

set_global_textmap(CompositePropagator([
    TraceContextTextMapPropagator(),
    TenantPropagator(),
]))

This pattern is how you bolt business context onto the trace pipeline without coupling it to your application logic. The context value flows automatically with the span — no manual threading.

For queue-based systems (Kafka, RabbitMQ, SQS), the concept is identical but the "carrier" is the message headers/attributes map instead of HTTP headers. OTel’s TextMapPropagator abstraction works the same way — you just implement a custom Getter/Setter against whatever structure your messaging library uses.

Gotchas

Sampling flag mismatch. If the upstream marks a trace as unsampled (flags: 00 in W3C, X-B3-Sampled: 0), you must not start recording spans. Violating this creates phantom data in your backend and breaks tail-based samplers in the Collector. The OTel SDK respects this automatically only if you actually extract the context before creating spans. If you create the span first and extract later, you lose the upstream decision.

64-bit vs 128-bit trace IDs. W3C requires 128-bit. B3 supports 64-bit. If a B3 trace ID comes in as 16 chars and you propagate it as W3C, you need to left-pad with zeros. The OTel B3 propagator does this correctly; hand-rolled parsers often don’t.

Header case sensitivity. HTTP/1.1 headers are case-insensitive by spec. HTTP/2 requires lowercase. Most frameworks normalize to lowercase before giving you the map, but some middleware doesn’t. Traceparent and TRACEPARENT should be treated identically. If you’re writing a custom propagator, always lowercase your lookups.

The tracestate forwarding trap. If you extract a tracestate with vendor-specific keys from an upstream service and then inject it downstream, you’re forwarding state that may not be yours. You should forward unknown entries (the spec requires it for interoperability), but strip your own key before re-inserting if you’re updating it. Getting this wrong causes duplicate vendor keys in the header, which some backends reject entirely.

Clock skew and span ordering. Propagation gets the parent-child relationship right, but if your services have unsynchronized clocks, your trace visualization will show children starting before parents. Use NTP. Seriously — chrony with a reliable source, check offset with chronyc tracking. If you’re on Kubernetes, don’t assume node clocks are synchronized just because they’re VMs.

gRPC metadata. In gRPC, "headers" are called metadata. The OTel gRPC instrumentation handles this, but if you’re using the gRPC interceptors and also doing manual propagation, you can end up with context injected twice or not at all. Use one or the other, not both.

Load balancer header stripping. Some reverse proxies strip or rename unknown headers by default. AWS ALB, nginx with proxy_set_header misconfigurations, and old HAProxy configs are common culprits. Verify that traceparent actually reaches your service by logging raw headers on a test endpoint before trusting your trace graph.

Production-Ready Setup: OpenTelemetry Collector as the Propagation Gateway

If you’re running a polyglot system with a mix of old B3 services and new W3C services, the cleanest production approach is to let the OTel Collector normalize everything at the edge:

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

  # Accept Zipkin/B3 from legacy services
  zipkin:
    endpoint: 0.0.0.0:9411

processors:
  batch:
    timeout: 1s
    send_batch_size: 1024

  # Normalize resource attributes
  resource:
    attributes:
      - key: deployment.environment
        value: production
        action: upsert

exporters:
  # Jaeger via OTLP (Jaeger 1.35+ accepts OTLP natively)
  otlp/jaeger:
    endpoint: jaeger:4317
    tls:
      insecure: true

  # Prometheus for span metrics
  spanmetrics:
    metrics_exporter: prometheus

  prometheus:
    endpoint: 0.0.0.0:8889

service:
  pipelines:
    traces:
      receivers: [otlp, zipkin]
      processors: [batch, resource]
      exporters: [otlp/jaeger]

With this setup, your B3 services send to the Zipkin receiver on port 9411, your new OTLP services send to 4317/4318, and the Collector outputs everything in a unified format to your backend. Your trace IDs are stitched together correctly as long as the propagation headers flow end-to-end — the Collector doesn’t re-propagate; it just collects finished spans.

For the SDK-side configuration in a service that needs to emit both formats during migration:

# otel_setup.py
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.propagate import set_global_textmap
from opentelemetry.propagators.b3 import B3MultiFormat
from opentelemetry.propagators.composite import CompositePropagator
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator

def setup_tracing(service_name: str, collector_endpoint: str) -> None:
    provider = TracerProvider()
    
    exporter = OTLPSpanExporter(endpoint=collector_endpoint, insecure=True)
    provider.add_span_processor(BatchSpanProcessor(exporter))
    
    trace.set_tracer_provider(provider)
    
    # Accept W3C and B3; emit W3C first (downstream services prefer it)
    set_global_textmap(CompositePropagator([
        TraceContextTextMapPropagator(),
        B3MultiFormat(),
    ]))

Call this once at startup, and every OTel-instrumented call in your service will correctly propagate and accept both formats.

Choosing a Format

If you’re starting fresh: W3C TraceContext, no question. It’s the IETF standard, supported everywhere that matters, and OTel defaults to it.

If you have existing Zipkin infrastructure or services that speak B3: run the composite propagator during migration. Don’t do a flag-day cutover — it breaks traces in flight.

If a vendor forces you into a proprietary format: implement a custom TextMapPropagator, wrap it in the composite, and track which services still need it. Vendor-specific propagators are technical debt from day one; get an exit plan in writing.

The actual wire format matters less than consistency. The single most common cause of broken traces is not a wrong format — it’s one service in the chain that was deployed without instrumentation, or with instrumentation that never had its propagator configured. An empty traceparent on an outbound request silently creates a new root span, and your trace splits in two with no visible error.

Audit your service mesh or API gateway logs. If you see traceparent appearing in responses but not requests from any service, that service is a propagation dead-end. Fix those first.

Quick Reference

Header Format Trace ID bits Notes
traceparent W3C 128 IETF standard, OTel default
tracestate W3C — Vendor extensions bag
X-B3-TraceId + friends B3 Multi 64 or 128 Zipkin, legacy Spring/Netflix
b3 B3 Single 64 or 128 Compact B3, Envoy default
Custom Whatever you define Any Business context, proprietary APM

The W3C TraceContext spec and the B3 spec are both short enough to read in an afternoon. Do it once — it removes the mystery from every weird trace gap you’ll ever debug.

👁 Views: 114,796 · Unique visitors: 45,617