You ran your benchmark. The p99 looks solid — 12ms. You deploy to production. Users complain about random freezes. You look at your APM and see 800ms spikes that appear nowhere in your pre-deploy numbers.
You didn’t have a deployment problem. You had a measurement problem — specifically, a problem called coordinated omission, and it’s been quietly lying to you about your tail latencies for years.
Gil Tene (CTO of Azul Systems, creator of HdrHistogram) described this in his now-legendary talk "How NOT to Measure Latency". If you benchmark services and haven’t watched it, stop reading this and go watch it first. The rest of this article is the written version of that insight, with enough practical detail to fix your tooling today.
The Closed-Loop Trap
Most load generators — wrk, ab, hey, naive scripts — operate in a closed-loop model. They send a request, wait for the response, record the latency, then send the next request. Simple, sensible-seeming.
Here’s the problem: the moment your server gets slow, the benchmark slows down with it. The tool stops hammering the server at the rate you asked for. It’s taking a polite break while the server recovers.
This means the benchmark is coordinating its behavior with the system under test. It backs off exactly when things go wrong. And when it backs off, it stops generating the requests that would have exposed the true scope of the problem.
A Concrete Example
Suppose you’re benchmarking an HTTP service at 1000 requests per second. Each request normally takes 1ms. Your benchmark runs for 10 seconds, so you expect ~10,000 measurements.
At second 5, the JVM does a stop-the-world GC pause for 1 full second.
What a closed-loop tool sees:
- 4999 requests at ~1ms
- 1 request that happened to be in-flight during the pause: 1001ms
- 4999 more requests at ~1ms after the recovery
Total: 9999 measurements. One outlier at 1001ms.
p99 = 1ms (the 9900th percentile is well inside the normal cluster).
p99.9 = maybe 1001ms if you’re lucky.
What actually happened:
During that 1-second pause, 1000 requests were supposed to arrive at the server. Each of those requests had to wait at least 1 second before being processed. Their true experienced latency is 1001ms each.
Real p99 = 1001ms. Real p99.9 = 1001ms.
The difference between "12ms" and "1001ms" is not a rounding error. It’s a fundamentally broken measurement.
Why This Happens: The Omission
When the server stalls, a closed-loop tool has no pending requests to record. It’s waiting for the one in-flight request to come back. During that silence, the 1000 requests that would have arrived at the target rate simply don’t exist in the dataset — they were never sent, so they were never recorded as slow.
The tool omitted them. Worse, it did so in coordination with the slowdown — the omission happens precisely when slowdowns occur. Hence the name.
This is not a bug in any specific tool. It’s a fundamental consequence of the closed-loop model. The tool can’t record latency for requests it never sent.
The Open-Loop Model: How to Actually Measure Latency
The correct approach is an open-loop (or Poisson-arrival) model. The load generator maintains a target arrival rate completely independently of whether previous responses have come back. It schedules requests according to a fixed inter-arrival distribution and records each request’s latency against its intended start time, not its actual start time.
If a request was supposed to go out at t=5000ms and the server was frozen, it goes out at t=5001ms (when the server recovers) but its latency is recorded as response_time - 5000ms, not response_time - 5001ms. That 1ms scheduling delay counts.
Even better: in a properly designed open-loop tool, the 999 requests that were queued up behind that stall all have their "start" recorded at their intended dispatch time. The histogram fills up with the real story.
The Tools That Get It Wrong (Most of Them)
wrk — the popular Lua-scriptable HTTP benchmarker — uses a closed-loop model. Fast, great for throughput testing, terrible for accurate latency histograms.
Apache Bench (ab) — closed-loop. Ancient. Don’t use it for latency measurements.
hey — closed-loop by default. Pretty output, wrong p99.
k6 — better than most, but requires careful configuration to avoid coordinated omission. Default scripts are closed-loop.
JMeter — technically supports open-loop patterns but the default thread-group setup gives you coordinated omission. You need to use the "Concurrency Thread Group" plugin or the "Arrivals Thread Group" to get correct behavior.
The Tools That Get It Right
wrk2
The canonical fix. wrk2 (github.com/giltene/wrk2) is Gil Tene’s fork of wrk that adds correct latency measurement via HdrHistogram and an open-loop scheduler.
# Install
git clone https://github.com/giltene/wrk2.git
cd wrk2 && make
# Run at exactly 1000 req/s for 30 seconds, 4 threads, 100 connections
./wrk -t4 -c100 -d30s -R1000 https://cd-linux.club/api/endpoint
The -R1000 flag is the key: it specifies the target rate in requests per second. wrk2 will maintain this rate and correctly record latency for requests that experienced queuing delay.
Output looks like:
Latency Distribution (HdrHistogram)
50.000% 1.02ms
75.000% 1.18ms
90.000% 1.47ms
99.000% 847.32ms ← this is the truth wrk would have hidden
99.900% 1.01s
99.990% 1.01s
99.999% 1.01s
That p99 of 847ms is ugly. It should be — it’s real.
vegeta
github.com/tsenart/vegeta is an open-loop HTTP load testing tool written in Go. It uses a fixed-rate scheduler with coordinated omission correction.
echo "GET https://cd-linux.club/api/endpoint" | \
vegeta attack -rate=1000 -duration=30s | \
vegeta report -type=hdrplot > results.hdr
# Or for a quick summary
echo "GET https://cd-linux.club/api/endpoint" | \
vegeta attack -rate=1000 -duration=30s | \
vegeta report
vegeta also outputs HdrHistogram-compatible data, which you can plot with hdrplot for a proper latency distribution view.
Gatling
Gatling uses a scenario-based model with an injection profile. When you use constantUsersPerSec or rampUsersPerSec, Gatling sends users at the specified rate regardless of system response. It’s not perfectly open-loop by default in all configurations, but it’s significantly better than most alternatives and its reports are excellent for identifying latency distribution shifts over time.
HdrHistogram: The Recording Layer
All the good tools use HdrHistogram (github.com/HdrHistogram/HdrHistogram) for recording. Understanding why matters.
A regular histogram with fixed-size buckets loses precision at high percentiles. If your buckets are 1ms wide and go up to 1s, you have 1000 buckets. Your p99.99 bucket might be "somewhere between 900ms and 901ms" — and at high precision, that’s not good enough.
HdrHistogram is a high-dynamic-range histogram. It maintains precision relative to the value being stored (configurable, default is 3 significant digits) across a very wide range (typically 1 microsecond to ~3600 seconds). It’s compact, lock-free, and can record latency values without losing precision at any point in the distribution.
More importantly, HdrHistogram supports coordinated omission correction. You can provide it with an expectedIntervalBetweenValueSamples value, and it will auto-fill the histogram with intermediate values for any recorded value that exceeds that interval — which is exactly the correction needed for closed-loop tools.
// Java example: correcting a closed-loop recording after the fact
Histogram histogram = new Histogram(3600000000L, 3);
// ... collect measurements with closed-loop tool ...
// Apply coordinated omission correction
Histogram correctedHistogram =
histogram.copyCorrectedForCoordinatedOmission(1_000_000L); // 1ms expected interval
System.out.println("p99 (wrong): " + histogram.getValueAtPercentile(99.0));
System.out.println("p99 (corrected): " + correctedHistogram.getValueAtPercentile(99.0));
This correction is also available in Python (hdrh package), Go (codahale/hdrhistogram), and most other language implementations.
Gotchas
Gotcha #1: "But my test environment doesn’t have GC pauses."
GC pauses are just the most common example. Coordinated omission hides any latency event that affects the whole system: disk flushes, network hiccups, OS scheduler jitter, Linux transparent huge page compaction, iptables rule evaluation under load. In production, these happen constantly. Your test environment doesn’t replicate production load shape, so coordinated omission silently hides the production-representative tail.
Gotcha #2: Interpreting wrk2 output for the first time.
wrk2’s p99 will look shockingly worse than wrk’s p99 on the same service. Don’t panic — your service didn’t get worse. Your measurement got honest. The previous number was wrong.
Gotcha #3: Connection count vs. request rate.
In wrk2, -c sets concurrent connections and -R sets target rate. These interact. If your rate is 1000 req/s and you have 10 connections, each connection handles ~100 req/s. If your service latency is 50ms, each connection can only sustain 20 req/s before queuing. The math has to work out: connections * (1000 / avg_latency_ms) >= target_rate. Under-spec the connections and you’ll hit connection-induced queuing, not service latency. A reasonable starting point: connections = target_rate * p99_latency_in_seconds * 2.
Gotcha #4: The "correct" tool on a broken network path.
Open-loop tools maintain rate by scheduling requests with wall-clock timing. If your test machine’s network is saturated, the requests queue in the kernel. You’re now measuring network + kernel queue latency, not service latency. Run load generators close to the target service (same datacenter, same VPC). Better yet: run multiple instances of your load generator and aggregate the HdrHistogram results.
Gotcha #5: Histogram encoding in dashboards.
If you’re sending metrics to Prometheus via histogram_quantile(), Prometheus uses fixed buckets and linear interpolation between bucket boundaries. This is a lossy representation that can also misrepresent tail latencies, independently of coordinated omission. For serious latency work, consider sending HdrHistogram-encoded data to InfluxDB or using the prometheus-hdrhistogram bridge. At minimum, make sure your Prometheus histogram buckets are dense enough in the tail range you care about.
Production-Ready Latency Testing Workflow
Here’s a setup that gives you defensible numbers:
#!/bin/bash
# benchmark.sh — honest latency measurement
SERVICE_URL="https://cd-linux.club/api/endpoint"
RATE=500 # req/s — must be achievable by the service
DURATION=120s # 2 minutes minimum; short runs miss infrequent GC cycles
THREADS=4
# connections = rate * (expected_p99_latency_s) * 4, minimum 10
CONNECTIONS=50
./wrk2/wrk \
-t${THREADS} \
-c${CONNECTIONS} \
-d${DURATION} \
-R${RATE} \
--latency \
${SERVICE_URL}
A few rules of thumb:
- Run for at least 2 minutes. Most JVM GC strategies have cycles in the 30s–90s range. A 30-second benchmark will miss them.
- Start at 50% of your expected peak throughput. Measure there first. Then step up in 25% increments. Latency distributions at 50% vs 95% capacity tell you where your system starts to degrade.
- Always report the full percentile distribution, not just p50/p99. The shape of the curve from p50 to p99.99 tells you whether you have periodic spikes (bimodal distribution) or gradual degradation (long tail).
- If your p99.9 is more than 10x your p99, dig deeper. That’s almost always a periodic process (GC, cron, buffer flush) rather than load-induced latency.
- Never report a latency number without also reporting the request rate. "p99 = 5ms" is meaningless. "p99 = 5ms at 1000 req/s" is a number.
Why This Matters in Production
The production impact of ignoring coordinated omission is almost always felt in SLOs, not average latencies. Your p50 and p95 will look fine in most situations — the coordinated omission bias hits hardest at the tail. If you have a 99th percentile SLO and you’re measuring with a closed-loop tool, you’re flying blind. You might be violating the SLO under real-world load patterns while your benchmarks report green.
The most dangerous scenario: a service that performs well under synthetic benchmarks but has a stateful operation (cache eviction, compaction, connection pool refresh) that triggers under realistic sustained load. A 30-second wrk run will never trigger it. A 2-minute wrk2 run at realistic rate will.
Get honest about your measurements. Use wrk2 or vegeta. Accept the ugly numbers. They’re the real ones — and they’re the ones your users are experiencing.