Your load testing tool is probably lying to you. Not maliciously — it just has a design flaw baked in from the start, one that makes your API look faster than it actually is when things go sideways. The flaw has a name: coordinated omission. And the tool that actually fixes it is wrk2.
This isn’t about picking the shiniest benchmark tool. It’s about whether the numbers you’re shipping to your engineering team mean anything at all.
The Problem: Your Benchmark Is Colluding With Your Service
Picture this: you’re running a benchmark at 1000 requests/second. Everything looks fine — p99 latency is 12ms, your service is humming along. You push to production, traffic spikes, and users start complaining about 5-second waits. Your benchmark lied.
Here’s why.
Most load generators — ab, basic wrk, naive JMeter setups — operate like this: send a request, wait for the response, send the next one. Or even with concurrency: "I have 50 connections, each one fires the next request as soon as the previous one finishes." This sounds reasonable. It’s not.
When your service slows down and a request takes 500ms instead of 5ms, the tool is waiting. It’s not generating load during that window. So the slow response is recorded, but the hundred requests that should have been sent during those 500ms are never sent. The system never experiences the backlog that a real-world client would create.
The result: the tool accidentally synchronizes its measurement cadence with the service’s slowdowns. Slow responses are underrepresented in the histogram. The tail looks better than it is. This is coordinated omission — the benchmark coordinates with the service to hide its worst behavior.
Gil Tene (CTO of Azul Systems, the person who coined the term) gives the best analogy: imagine you’re recording subway arrival times, but every time a train is late, you stop your stopwatch and wait. Your "average delay" will be suspiciously low.
How wrk2 Actually Fixes It
wrk2 is Gil Tene’s fork of wrk, available at https://github.com/giltene/wrk2.
The core fix is conceptually simple but mechanically non-trivial: wrk2 uses a constant throughput scheduler. You tell it "I want 500 requests/second" and it will attempt to issue requests at exactly that rate regardless of what the server is doing. If a response is late, the tool doesn’t slow down — it keeps scheduling new requests as planned.
Then, for the requests that were delayed, it doesn’t just record the response time. It records the intended start time vs the actual completion time. So if a request was supposed to go out at T=1000ms but the connection pool was saturated and it couldn’t go until T=1800ms, wrk2 charges that 800ms of waiting to the latency of that request. This is the corrected latency.
Under the hood it uses HdrHistogram (High Dynamic Range Histogram) — a lock-free, allocation-free histogram that can track values from 1 microsecond to hours with no loss of precision. It was also written by Gil Tene. The man clearly has strong opinions about measurement.
The practical effect: wrk2’s p99 and p99.9 numbers tell you what a real user at the tail of the distribution actually experiences, not what the benchmark wanted to experience.
Installation
There are no pre-built binaries. You compile from source. This is a one-time five-minute operation.
# Install build dependencies (Debian/Ubuntu)
sudo apt-get install -y build-essential libssl-dev git
# Clone the repo
git clone https://github.com/giltene/wrk2.git
cd wrk2
# Build
make
# Optional: put it on your PATH
sudo cp wrk /usr/local/bin/wrk2
On RHEL/Fedora:
sudo dnf install -y gcc make openssl-devel git
macOS with Homebrew:
brew install wrk2
Verify it’s working:
wrk2 --version
# wrk 4.0.0 [epoll] Copyright (C) 2012 Will Glozer
The version string still says "wrk" — that’s expected. You’re running wrk2.
Basic Usage
The killer flag is -R — this sets your target request rate (requests per second).
wrk2 -t4 -c100 -d30s -R1000 https://cd-linux.club/api/health
Breaking this down:
-t4— 4 threads (match your CPU cores, not more)-c100— 100 concurrent connections-d30s— run for 30 seconds-R1000— target 1000 requests/second
This is the critical part: without -R, wrk2 behaves like regular wrk — max throughput mode, which defeats the point. Always specify -R.
Sample output:
Running 30s test @ https://cd-linux.club/api/health
4 threads and 100 connections
Thread calibration: mean lat.: 3.245ms, rate sampling interval: 10ms
Thread Stats Avg Stdev Max +/- Stdev
Latency 4.12ms 8.93ms 289.54ms 98.14%
Req/Sec 251.21 47.33 444.00 69.83%
Latency Distribution (HdrHistogram - Recorded Latency)
50.000% 2.72ms
75.000% 3.84ms
90.000% 5.12ms
99.000% 47.23ms
99.900% 201.45ms
99.990% 289.54ms
Detailed Percentile spectrum:
Value Percentile TotalCount 1/(1-Percentile)
1.234 0.000000 1 1.00
...
289.54 1.000000 29987 Infinity
29987 requests in 30.00s, 4.87MB read
Requests/sec: 999.57
Transfer/sec: 166.21KB
The HdrHistogram section is where the real information lives. Ignore the "Avg" line — averages hide everything important about tail behavior.
Reading the Output: What Matters
The percentile breakdown is what you care about:
- p50 — your median user. Half the requests are faster than this.
- p99 — 1 in 100 requests is slower than this. In production at 1000 req/s, that’s 10 users per second having a bad time.
- p99.9 — 1 in 1000. At scale this stops being hypothetical.
- p99.99 — if you’re doing financial transactions or anything where SLAs matter, track this.
The 1/(1-Percentile) column in the detailed spectrum is the return period — how often (statistically) one request hits that latency. A value of 10000 means roughly once every 10,000 requests.
When you see a large gap between p99 and p99.9, that’s the latency cliff — the point where your service hits resource contention and response times explode. Understanding where that cliff is tells you your real capacity ceiling, not the happy-path ceiling.
Lua Scripts for Realistic Load
Benchmarking just GET /health is mostly useless for real capacity planning. wrk2 supports Lua scripts for custom request logic.
POST with JSON body:
-- post_benchmark.lua
-- Benchmarks the /api/orders endpoint with a realistic payload
wrk.method = "POST"
wrk.headers["Content-Type"] = "application/json"
wrk.headers["Authorization"] = "Bearer test-token-12345"
wrk.body = '{"product_id": 42, "quantity": 1, "user_id": 1001}'
function response(status, headers, body)
if status ~= 201 then
-- Track non-2xx responses for debugging
io.write("Unexpected status: " .. status .. "\n")
end
end
Run it:
wrk2 -t4 -c50 -d60s -R500 -s post_benchmark.lua https://cd-linux.club/api/orders
Randomized payload to avoid caching effects:
-- random_ids.lua
-- Prevents CDN/proxy cache from giving you false-positive latency numbers
local ids = {}
for i = 1, 1000 do
ids[i] = math.random(1, 100000)
end
request = function()
local id = ids[math.random(1, #ids)]
local path = "/api/products/" .. id
return wrk.format("GET", path)
end
This matters more than people think. If you hammer the same URL, caches warm up and you’re benchmarking your cache, not your service.
Gotchas
The -R rate is a target, not a guarantee. If you ask for 10,000 req/s from a single machine with 50 connections, you won’t get it. wrk2 will tell you the actual achieved rate. If Requests/sec is significantly below your -R, you’re either connection-limited or your client machine is the bottleneck. Bump -c connections or run wrk2 from multiple machines.
Connections vs rate. A rule of thumb: you need at least rate × p99_latency_seconds connections to sustain your target rate. Targeting 1000 req/s with p99 at 100ms? You need at least 1000 × 0.1 = 100 connections as a floor. Add 20-30% headroom.
Thread calibration at startup. wrk2 spends a brief period calibrating the rate scheduler. Don’t start interpreting results from the first few seconds. Use -d30s at minimum; -d60s is better for anything going to a production sizing document.
Localhost benchmarks are fiction. TCP loopback bypasses network stack overhead, NIC interrupts, and queueing. Always benchmark over a realistic network path — same datacenter, real NIC. The numbers look less impressive and they’re actually true.
Don’t conflate throughput capacity with latency at load. Run separate benchmarks: first find your max throughput (gradually increase -R until you see latency blow up), then characterize latency behavior at 50%, 70%, and 90% of that max. The 90% number is what you put in your runbook as the operational ceiling.
wrk2 doesn’t retry. Dropped connections and 5xx responses are counted but not retried. Make sure you’re tracking Non-2xx or 3xx responses in the output. A benchmark where 5% of requests 500’d but latency looks fine is not a good benchmark.
HTTPS adds overhead. TLS handshakes dominate connection setup cost. When benchmarking HTTPS, use -c high enough that connections are being reused (keep-alive). If you’re seeing disproportionately high latency vs HTTP, that’s usually TLS handshake cost on short-lived connections.
Production-Ready Benchmark Workflow
Here’s the flow I actually use when capacity testing a service before a major release:
#!/bin/bash
# capacity_test.sh
# Usage: ./capacity_test.sh http://service:8080/endpoint
TARGET="${1:-https://cd-linux.club/}"
DURATION="60s"
THREADS=4
CONNECTIONS=200
echo "=== Warmup ==="
wrk2 -t${THREADS} -c${CONNECTIONS} -d10s -R100 ${TARGET} > /dev/null
for RATE in 100 250 500 750 1000 1500 2000; do
echo ""
echo "=== Rate: ${RATE} req/s ==="
wrk2 -t${THREADS} -c${CONNECTIONS} -d${DURATION} -R${RATE} \
--latency ${TARGET} 2>&1 | tee "results_${RATE}.txt"
# Brief cooldown between runs
sleep 5
done
echo ""
echo "=== Summary: p99 by rate ==="
for f in results_*.txt; do
RATE=$(echo $f | grep -o '[0-9]*')
P99=$(grep "99.000%" $f | awk '{print $2}')
echo " ${RATE} req/s → p99: ${P99}"
done
The warmup pass matters — JIT compilation, connection pool initialization, and DNS caches all need to settle before numbers mean anything.
The summary at the end gives you a quick latency-vs-throughput curve. You’re looking for the knee of the curve — the rate where p99 starts climbing non-linearly. That’s your service’s real capacity with margin.
Comparing wrk vs wrk2 on the Same Service
Run both tools against a service that has occasional 200ms GC pauses and compare the p99 outputs. wrk will report something like 15ms p99. wrk2, running at the same nominal throughput, will report 220ms+ p99.
Neither tool made up numbers. But only one of them told you what a real user in the 99th percentile actually waits for. The GC pause causes the request queue to back up, and every request issued during that pause inherits the wait. wrk didn’t send those requests during the pause — it was waiting. wrk2 sent them on schedule and counted the full wait.
This is the entire point.
When wrk2 Is Not the Right Tool
wrk2 is excellent for steady-state throughput characterization. It’s not the right tool for:
- Ramp-up / spike testing — use k6 or Gatling for scenarios with variable load shapes
- Stateful session simulation — logging in, adding to cart, checking out requires more orchestration than Lua scripts reasonably provide
- Distributed load (>~100k req/s) — wrk2 from a single machine tops out. Look at wrk2 instances behind a coordinator, or switch to k6 cloud / Locust
- Protocol testing beyond HTTP — wrk2 is HTTP only
For 80% of backend service capacity tests, wrk2 is exactly what you need. For the other 20%, you’ll know because wrk2 won’t fit the shape of the test.
The Numbers You Should Be Presenting
After a proper wrk2 benchmark run, the output you bring to a capacity review should have:
- Achieved rate (not target rate)
- p50, p95, p99, p99.9 latencies at the intended operating rate
- The rate at which p99 crossed your SLA threshold
- Error rate (non-2xx responses, connection errors)
- The test environment (network path, instance size, service version)
"Our p99 is 8ms" without the rate, environment, and error rate attached is a number with no meaning. wrk2 gives you everything to back it up — don’t leave context on the floor.
Benchmarking is easy to do badly and hard to do right. wrk2 eliminates the most common category of wrong — the kind where your tool quietly collaborates with your service to hide its worst behavior. Install it, always use -R, read the HdrHistogram output, and stop presenting averages to anyone who will make decisions based on them.