Your service’s average response time is 12ms. Your team ships, the PM celebrates, everyone goes home. Then a user on Slack says the app "feels sluggish." You look at the dashboard — still 12ms. You dismiss it.
That user is experiencing your p99.9. And you have no idea what it is.
This is the core problem with how most engineers measure latency. We reach for the mean, sometimes the p95 or p99, treat those numbers as ground truth, and build an entirely false mental model of how our system actually behaves. HDR Histograms exist to fix this, and after you understand them, you’ll never trust a single-number latency metric again.
The Mean Is Useless and the p99 Is Not Enough
Let’s be direct: the arithmetic mean of latency measurements is almost always misleading. If 999 requests complete in 1ms and one request takes 10,000ms, your mean is around 11ms — and your system is broken in a way that affects 0.1% of users. At any meaningful scale, 0.1% is thousands of people.
The reason this happens is straightforward. Latency distributions in real systems are not normal (Gaussian). They are heavily right-skewed with long tails. GC pauses, lock contention, TCP retransmits, cold caches — these events create spikes that are rare but severe, and they are completely invisible in aggregate statistics.
People graduate to percentiles, which is better. But a single percentile is still a single point on a distribution. You know that 99% of requests are faster than X, but you know nothing about how much faster the bulk of them are, and you know nothing about what happens beyond that threshold. The tail above p99 could be a gentle slope or a cliff — you can’t tell.
What you actually need is the full shape of the distribution.
What HDR Histograms Are
HDR stands for High Dynamic Range. The project lives at github.com/HdrHistogram/HdrHistogram and has implementations in Java, C, C++, Go, Python, Rust, and more.
The core insight is elegant: you don’t need uniform precision across the entire range of values you’re measuring. If your latency values span from 100 microseconds to 100 seconds, you don’t need sub-microsecond precision at the top end, and you don’t need second-level precision at the bottom. What you need is proportionally consistent precision — the same relative error across the entire range.
HDR Histograms achieve this with a log-linear bucketing scheme. The value range is divided into sub-buckets, where each "decade" of values gets the same number of buckets. The result: bucket widths scale logarithmically with value magnitude, giving you configurable significant figures of precision across a dynamic range spanning several orders of magnitude.
In practice, you configure the histogram with:
- Lowest trackable value — usually 1 (microseconds)
- Highest trackable value — whatever your worst-case might be (e.g., 3,600,000,000 for an hour in microseconds)
- Number of significant value digits — typically 3, giving you 0.1% relative error
Memory usage stays bounded and predictable regardless of how many values you record. Recording 10 million values takes the same memory as recording 10 values. That’s the other thing that makes HDR Histograms production-viable — they’re designed to live in your hot path.
Reading the Shape
Before diving into code, it’s worth understanding what you’re looking for when you actually look at a histogram.
A healthy, fast service looks like a sharp spike near the low end with a gentle tail. Most requests cluster tightly. The tail might show occasional slow outliers but they don’t extend far.
A service with GC pressure shows bimodal distribution — a cluster of fast requests and a secondary cluster of requests that got caught during a collection. You’ll miss this completely with percentiles alone.
A service under lock contention shows a flat tail that extends to high values. The slowest requests aren’t outliers — they’re a structural feature of how the system serializes work.
Coordinated omission — the sneaky one — makes a system look better than it is. When your load generator backs off under load (because it’s waiting for previous requests to complete before sending new ones), it stops issuing requests during the slow period. The slow requests don’t make it into the histogram because they were never issued. wrk2, which uses HDR Histograms internally and corrects for coordinated omission, was built specifically because this was corrupting almost every latency benchmark in existence.
Practical Usage: Java
The Java implementation is the reference. Add the dependency:
<!-- Maven -->
<dependency>
<groupId>org.hdrhistogram</groupId>
<artifactId>HdrHistogram</artifactId>
<version>2.2.2</version>
</dependency>
Recording values:
import org.HdrHistogram.Histogram;
// Track values from 1 microsecond to 1 hour, 3 significant figures
Histogram histogram = new Histogram(1, 3_600_000_000L, 3);
// In your request handler
long startNs = System.nanoTime();
// ... do work ...
long latencyUs = (System.nanoTime() - startNs) / 1000;
histogram.recordValue(latencyUs);
// Report
System.out.printf("p50: %d µs%n", histogram.getValueAtPercentile(50.0));
System.out.printf("p95: %d µs%n", histogram.getValueAtPercentile(95.0));
System.out.printf("p99: %d µs%n", histogram.getValueAtPercentile(99.0));
System.out.printf("p99.9: %d µs%n", histogram.getValueAtPercentile(99.9));
System.out.printf("max: %d µs%n", histogram.getMaxValue());
For multi-threaded recording, use ConcurrentHistogram or the recorder pattern with SingleWriterRecorder — it avoids lock contention on the hot path:
import org.HdrHistogram.SingleWriterRecorder;
import org.HdrHistogram.Histogram;
// One writer thread
SingleWriterRecorder recorder = new SingleWriterRecorder(1, 3_600_000_000L, 3);
// In the single writer thread
recorder.recordValue(latencyUs);
// In your reporting thread (different thread, periodic)
Histogram intervalHistogram = recorder.intervalHistogram(); // resets the active histogram
// now process intervalHistogram for your metrics window
SingleWriterRecorder uses a double-buffer internally. The recording thread writes to one buffer; when you call intervalHistogram() it flips the buffer and hands you the completed one. Zero locks, zero allocations on the hot path.
Practical Usage: Go
The Go port is solid. Get it:
go get github.com/HdrHistogram/hdrhistogram-go
package main
import (
"fmt"
"time"
hdrhistogram "github.com/HdrHistogram/hdrhistogram-go"
)
func main() {
// 1µs to 1 hour, 3 significant figures
h := hdrhistogram.New(1, 3_600_000_000, 3)
// Simulate recording latencies
for _, latency := range []int64{120, 135, 142, 890, 1200, 45000, 200000} {
h.RecordValue(latency) // microseconds
}
fmt.Printf("p50: %d µs\n", h.ValueAtQuantile(50))
fmt.Printf("p99: %d µs\n", h.ValueAtQuantile(99))
fmt.Printf("p99.9: %d µs\n", h.ValueAtQuantile(99.9))
fmt.Printf("max: %d µs\n", h.Max())
fmt.Printf("mean: %.1f µs\n", h.Mean())
// Print full distribution (useful for debugging)
for _, bar := range h.Distribution() {
if bar.Count > 0 {
fmt.Printf(" [%d - %d µs]: %d\n", bar.From, bar.To, bar.Count)
}
}
}
For production Go services, wire this into your HTTP middleware and expose it via a /metrics endpoint or ship it to your TSDB on a ticker.
Practical Usage: wrk2
If you’re benchmarking HTTP services, stop using wrk and use wrk2. It’s purpose-built for correct latency measurement and outputs HDR Histogram data.
# Install (requires luajit and openssl)
git clone https://github.com/giltene/wrk2.git
cd wrk2 && make
# Run a benchmark at 10k req/s for 60 seconds
./wrk -t4 -c100 -d60s -R10000 --latency https://cd-linux.club/api/health
The -R10000 flag is the key difference from wrk. You’re specifying the constant request rate, not just concurrency. wrk2 uses a coordinated-omission-corrected scheduling model — it issues requests on a schedule regardless of how long previous requests are taking. This means slow requests don’t hide future slow requests. The output includes a full percentile distribution.
Integrating with Prometheus
The standard Prometheus client histogram is a fixed-bucket histogram — you pre-declare bucket boundaries and the library counts observations per bucket. It works but it’s inflexible and you’ll often find your bucket boundaries are wrong for your actual distribution.
The prometheus-metrics-protobuf native histogram format (introduced in Prometheus 2.40) is essentially HDR-style: exponential bucket widths, configurable resolution. Enable it in your Prometheus scrape config:
# prometheus.yml
scrape_configs:
- job_name: 'myservice'
scrape_interval: 15s
# Enable native histograms (experimental in older versions)
native_histogram_min_bucket_factor: 1.1
static_configs:
- targets: ['localhost:9090']
In Go with the official client, opt into native histograms:
import (
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promauto"
)
var requestDuration = promauto.NewHistogram(prometheus.HistogramOpts{
Name: "http_request_duration_seconds",
Help: "HTTP request duration in seconds",
// Native histogram — no bucket boundaries needed
NativeHistogramBucketFactor: 1.1,
NativeHistogramMaxBucketNumber: 100,
NativeHistogramMinResetDuration: 1 * time.Hour,
})
If you’re stuck on classic Prometheus histograms, configure your buckets deliberately based on your actual SLOs, not the default ones. The default .005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10 buckets (in seconds) are useless for a service that should respond in under 50ms.
Storing and Shipping Histogram Data
HDR Histograms have a compact binary serialization format. You can serialize an interval snapshot, ship it to a central store, and deserialize for analysis without losing the distribution shape. This is how tools like hdr-plot and HistogramLogAnalyzer work.
// Serialize to base64 for transport
ByteBuffer buffer = ByteBuffer.allocate(histogram.getNeededByteBufferCapacity());
histogram.encodeIntoByteBuffer(buffer);
String encoded = Base64.getEncoder().encodeToString(buffer.array());
// Deserialize on the other end
byte[] bytes = Base64.getDecoder().decode(encoded);
Histogram decoded = Histogram.decodeFromByteBuffer(ByteBuffer.wrap(bytes), 0);
The Java HistogramLogWriter / HistogramLogReader classes handle writing time-tagged histogram snapshots to a file — useful for post-mortem analysis. Run your load test, capture histogram logs, replay them later with different percentile queries without re-running the test.
Gotchas
Clock resolution matters. On Linux, System.nanoTime() in Java uses clock_gettime(CLOCK_MONOTONIC) which has nanosecond resolution. On some VMs in cloud environments, the resolution can be lower. If you record many values in the sub-microsecond bucket, that’s a signal your clock resolution is lower than your measurement granularity — not that all your requests completed in under 1µs.
Measure the right thing. If you measure latency from the moment your application code starts processing to when it finishes writing the response, you’re excluding the time the request spent queued in the accept backlog, in the NIC ring buffer, or waiting for a thread pool slot. That excluded time is real user-perceived latency. Measure as close to the wire as possible.
Don’t reset histograms too aggressively. If you reset every 10 seconds to get an "interval" metric, a GC pause that takes 500ms is only captured in that 10-second window. Combine interval histograms (for your dashboards) with a cumulative histogram (for understanding the system over time). The SingleWriterRecorder pattern handles this cleanly — you get intervals for free and you can maintain a running total manually.
The max value is not a percentile. When you see histogram.getMaxValue(), that’s the single worst observation in that window. It’s noisy — a single network hiccup can spike it. It’s still worth tracking, but don’t build SLOs on it. p99.9 or p99.99 are more stable and more actionable.
Coordinated omission in your own code. If you have a thread pool handling requests and all threads are busy, new requests queue up. The queuing time is real latency but it happens before your instrumentation fires. Solution: timestamp the request at arrival (when it enters the queue), not when a worker thread picks it up.
HDR Histogram buckets are not evenly spaced. If you’re exporting raw bucket data to Grafana and using a heatmap panel, be aware the visual bucket density is non-uniform by design. This is correct behavior — don’t fight it by interpolating or re-bucketing. Use tools that understand the format natively, like histogram_quantile in Prometheus PromQL on native histograms.
Production-Ready Pattern: Latency SLO Monitoring
Here’s a pattern worth standardizing on. Pick three numbers for every endpoint: p50, p99, and p99.9. Set alerts on the p99 and p99.9. Report p50 in your dashboards as a sanity check, but never make decisions based on it alone.
For SLO compliance, the number you care about is the error budget burn rate — how fast is your p99 exceeding threshold compared to your allowed budget? A brief spike of p99 > 500ms that lasts 2 minutes is very different from a sustained p99 > 500ms for 4 hours. HDR Histograms let you be precise about both magnitude and duration.
# PromQL: fraction of requests exceeding 500ms SLO threshold
# With native histograms, this is exact:
histogram_fraction(0, 0.5, rate(http_request_duration_seconds[5m]))
With classic histograms, you’re interpolating within a bucket — you get an approximation. With native histograms built on HDR-style exponential buckets, the answer is exact to your configured precision.
Tools Worth Knowing
HdrHistogram Plotter — paste your histogram log output at hdrhistogram.github.io/HdrHistogram/plotFiles.html and get a percentile distribution chart instantly. Useful for quick load test analysis.
wrk2 — already mentioned, but worth emphasizing: use it for any HTTP load test where you care about latency accuracy.
HistogramLogAnalyzer — Java GUI tool that reads histogram log files and lets you zoom into time windows, overlay multiple runs, and compare percentiles. Lives in the HdrHistogram GitHub org.
Grafana’s histogram panel — works well with Prometheus native histograms. Configure it to show the heatmap over time so you can see latency distribution shift during incidents.
The fundamental shift HDR Histograms force is moving from "what’s my latency?" to "what does my latency distribution look like?" Those are different questions with different answers and different operational implications. Once you’ve seen a bimodal distribution expose a GC tuning problem that p99 was hiding, or watched the tail of a histogram flatten out as you reduced lock contention, you’ll stop trusting single-number summaries entirely.
The tooling is mature, the libraries are well-maintained, and the cost of instrumenting your service correctly is genuinely low. There’s no good reason to still be flying blind on the tail.