Rust gives you control over memory and execution that almost no other language does. That control is wasted if you don’t know where your cycles are going. "It’s fast because it’s Rust" is not a performance strategy — it’s a prayer.
Most Rust developers I’ve talked to either skip profiling entirely or rely on Instant::now() timers scattered across the codebase like breadcrumbs. That works until it doesn’t — until a release binary is 3× slower than expected and you have no idea why, and your timers weren’t measuring the right thing anyway.
This article walks you through the three tools you actually need: perf (the Linux kernel’s sampling profiler), cargo-flamegraph (the fastest path to a visual flame graph), and samply (a newer profiler with a genuinely good UI). We’ll profile real code, read real flame graphs, and talk about where each tool falls short.
The Two Mistakes Everyone Makes First
Before touching any tool, understand the two profiling sins that waste hours:
Sin 1: Profiling a debug build. Rust’s debug builds disable virtually all optimizations. You’ll spend 20 minutes looking at core::fmt dominating your flame graph and wonder if formatting is your bottleneck. It isn’t — you’re measuring unoptimized code. Always profile --release builds, with one important caveat we’ll cover in a moment.
Sin 2: Stripping debug symbols before profiling. Release builds strip symbols by default. Your flame graph becomes a wall of [unknown] frames. You need symbols to see function names; you need optimizations to see realistic performance. The fix is one block in Cargo.toml.
Add this to your project’s Cargo.toml before you do anything else:
[profile.release]
debug = true # embed DWARF symbols without affecting optimization
This keeps all optimizations intact — inlining, loop unrolling, everything — but bakes in symbol information so profilers can map addresses back to function names. The binary gets bigger. Production deployments should strip again (strip = true), but your profiling build needs this.
The Sample Program
We need something worth profiling. Here’s a contrived but realistic piece of code with an obvious hot path (once you find it) and some noise:
// src/main.rs
use std::collections::HashMap;
fn count_words(text: &str) -> HashMap<&str, usize> {
let mut map = HashMap::new();
for word in text.split_whitespace() {
*map.entry(word).or_insert(0) += 1;
}
map
}
fn naive_sort(mut v: Vec<u64>) -> Vec<u64> {
// deliberately bad sort to generate a visible hot path
let n = v.len();
for i in 0..n {
for j in 0..n - 1 {
if v[j] > v[j + 1] {
v.swap(j, j + 1);
}
}
}
v
}
fn generate_data(size: usize) -> Vec<u64> {
(0..size as u64).rev().collect()
}
fn main() {
// word count: lots of short-lived allocations
let corpus: String = std::iter::repeat("the quick brown fox jumps over the lazy dog ")
.take(500_000)
.collect();
let _counts = count_words(&corpus);
// naive sort: pure CPU, branch-heavy
let data = generate_data(8_000);
let _sorted = naive_sort(data);
println!("done");
}
Build it:
cargo build --release
Tool 1: perf
perf is the Linux kernel’s built-in performance counter subsystem. It does hardware counter sampling: every N cycles (or cache misses, or branch mispredictions) it stops execution and records where the instruction pointer was. After enough samples you get a statistical picture of where your program spends time.
Install
# Debian/Ubuntu
sudo apt install linux-perf
# Fedora/RHEL
sudo dnf install perf
Verify it works:
perf stat ls
If you get Permission denied, you need to lower the paranoia level:
# temporary, resets on reboot
echo 1 | sudo tee /proc/sys/kernel/perf_event_paranoid
# or permanent
echo 'kernel.perf_event_paranoid = 1' | sudo tee /etc/sysctl.d/99-perf.conf
sudo sysctl --system
Record and Report
perf record -F 999 -g ./target/release/your_binary
perf report
-F 999 sets the sampling frequency to 999 Hz (just under 1000 to avoid hardware aliasing effects). -g enables call graph collection — without this you get leaf functions but no context on who called them.
perf report opens an interactive TUI. Use arrow keys, press Enter to expand call chains, q to quit. It works but it’s not particularly pleasant. The flame graph is a much better way to consume this data.
Gotcha: Frame Pointers
Modern compilers omit frame pointers by default because they free up a register. This breaks perf‘s call graph unwinding — you’ll see shallow, truncated stacks. Fix it:
RUSTFLAGS="-C force-frame-pointers=yes" cargo build --release
Or add it permanently for the profiling profile:
[profile.release]
debug = true
[profile.release.package."*"]
# note: this doesn't affect your own code's frame pointers
Actually, the correct place is RUSTFLAGS or .cargo/config.toml:
# .cargo/config.toml
[build]
rustflags = ["-C", "force-frame-pointers=yes"]
Don’t do this for production — only for your profiling builds. Alternatively, perf can use DWARF unwinding (--call-graph dwarf) which doesn’t need frame pointers but is heavier:
perf record -F 999 --call-graph dwarf ./target/release/your_binary
Tool 2: cargo-flamegraph
cargo-flamegraph wraps perf (on Linux) or dtrace (on macOS) and produces an SVG flame graph in one command. It’s built on Brendan Gregg’s original flamegraph scripts and is the fastest way to go from "I have a Rust binary" to "I can see what’s happening."
Install
cargo install flamegraph
This also installs the cargo flamegraph subcommand.
Run It
# profile the release binary
cargo flamegraph --bin your_binary
This compiles a release build (with debug = true if you set it), runs perf record internally, and spits out flamegraph.svg in your project root. Open it in any browser.
# if you need to pass arguments to your binary
cargo flamegraph --bin your_binary -- --flag value
# profile tests
cargo flamegraph --test test_name
# profile benchmarks (great with criterion)
cargo flamegraph --bench bench_name
Reading the Flame Graph
The x-axis is not time — it’s stack population. Wider means more samples, which means more CPU time. The y-axis is call depth, with your main function at the bottom and leaf functions at the top.
You’re looking for wide, flat plateaus near the top. Those are your hot paths. Click any frame in the SVG to zoom in. Search with Ctrl+F.
For our sample program, you’ll see naive_sort eating most of the frame, with the bubble sort’s inner loop visible as a near-full-width block. The word count section will show HashMap operations and some allocator activity. That’s exactly where you’d go to optimize.
Gotcha: Root Permissions
perf record usually needs elevated privileges. cargo flamegraph handles this by calling sudo perf automatically. If that fails, either run the whole thing under sudo or set the paranoia level as shown earlier.
Gotcha: Short-Running Binaries
If your binary finishes in under a second, you might not collect enough samples for a meaningful flame graph. Wrap it in a loop, increase the workload, or use a benchmark harness. cargo flamegraph --bench with Criterion is perfect for this — Criterion runs your benchmark for several seconds by design.
Tool 3: samply
samply is a newer profiler by Markus Stange (previously at Mozilla). It uses Linux’s perf_event_open syscall directly and presents results in the Firefox Profiler UI — which is genuinely the best profiler UI I’ve used on any platform.
Where cargo-flamegraph gives you a static SVG, samply gives you a live, interactive timeline. You can zoom in on specific time windows, see per-thread activity, inspect individual samples, and switch between flame graphs, call trees, and source views.
Install
cargo install samply
Run It
samply record ./target/release/your_binary
That’s it. samply records the run, then opens your default browser pointing to the Firefox Profiler UI loaded with your data. No root required on most systems because samply uses user-space sampling by default.
# pass arguments
samply record ./target/release/your_binary -- arg1 arg2
# set sampling frequency (default is 1000 Hz)
samply record -r 4000 ./target/release/your_binary
Why the Firefox Profiler UI is Good
The timeline view shows you when the hotspot happened, not just that it happened. If you have a program that starts fast and gets slow over time (think: a cache that fills up and starts thrashing), the flame graph averaged over the whole run hides that. samply’s timeline makes it obvious.
The "call tree" view is better for reading symbol names in complex codebases. You can right-click a function and jump to source. The "stack chart" gives you a stacked area view of how the stack changed over time, which is useful for spotting phase transitions.
Gotcha: samply and Long-Running Servers
samply attaches to a process for the duration of the run. For servers you don’t want to restart, use samply record --pid <PID> to attach to a running process and detach after a fixed duration:
samply record --pid $(pgrep my_server) --duration 30
Thirty seconds of samples is almost always enough for a production hotspot.
Comparing the Three Tools
| perf | cargo-flamegraph | samply | |
|---|---|---|---|
| Output | TUI / raw data | SVG flame graph | Firefox Profiler UI |
| Root required | Usually yes | Usually yes (via sudo) | No |
| macOS support | No | Yes (dtrace) | Yes |
| Interactive timeline | No | No | Yes |
| Setup friction | Medium | Low | Low |
| Best for | Custom analysis scripts | Quick one-shot flame graphs | Interactive investigation |
My default workflow: samply for initial exploration, cargo flamegraph when I need a shareable artifact (SVG in a PR, Slack message), raw perf when I need hardware counters or I’m writing a script to automate profiling in CI.
Profiling Async Rust
Async code adds a wrinkle. When a tokio task awaits, the stack is stored on the heap as a future state machine. When the profiler samples it, you see the executor’s poll loop, not your business logic stack. This makes flame graphs of async programs hard to read.
A few approaches:
Synchronous hotspots are still visible. CPU-bound work inside async tasks profiles exactly like synchronous code. If you’re doing JSON parsing, crypto, or heavy computation inside an async context, that shows up fine.
tokio-console is a better tool for async-specific problems like task starvation, blocking the runtime, or slow polls. It’s not a CPU profiler but it tells you which tasks are misbehaving. Install it with cargo install tokio-console and add the console-subscriber crate to your project.
Increase the granularity. If you suspect an async bottleneck, isolate the suspected code into a synchronous benchmark and profile that. Criterion + cargo-flamegraph works well here.
CI Integration: Catching Regressions Before Merge
Profiling is most valuable as a regression gate, not a one-off fire drill. Here’s a minimal GitHub Actions job that runs a Criterion benchmark and fails if performance regresses by more than 10%:
# .github/workflows/bench.yml
name: Benchmarks
on:
pull_request:
branches: [main]
jobs:
bench:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Install Rust
uses: dtolnay/rust-toolchain@stable
- name: Run benchmarks
run: |
cargo bench --bench your_bench -- --output-format bencher | tee output.txt
- name: Check for regression
run: |
# compare against baseline stored in branch artifacts
# exact implementation depends on your CI setup
echo "Benchmark output:"
cat output.txt
For a more complete setup, look at cargo-criterion with its JSON output and critcmp for comparing baselines across commits. That combination gives you a table of "before vs. after" numbers on every PR.
Production-Ready Solutions
A few things that separate a real profiling workflow from a one-off investigation:
Pin your profiling binary to a specific CPU. taskset -c 2 ./target/release/your_binary prevents the scheduler from migrating the process, which reduces noise in your measurements. Pick a core that’s isolated from interrupt handling if your machine has many cores.
Disable frequency scaling. CPU frequency governors can cause wildly inconsistent benchmark results. cpupower frequency-set -g performance (needs root) sets all cores to max frequency. Reset with performance → powersave when you’re done.
Use RUSTFLAGS="-C target-cpu=native" when profiling on the machine you’re optimizing for. This enables AVX-512, newer SIMD, and whatever else your CPU supports. The release binary you profile should match the environment where it runs.
Profile with realistic data. Benchmark with a representative sample of your production workload. Hot paths shift dramatically between "process 100 items" and "process 10 million items" — cache behavior, branch prediction patterns, and allocator pressure all change at scale.
Gotchas Summary
- Debug build: always use
--release. Non-negotiable. - No symbols: set
debug = truein the release profile before profiling. - Truncated stacks: add
-C force-frame-pointers=yesor use DWARF unwinding. - Not enough samples: short-running binaries need a loop or a benchmark harness.
- Async noise: the executor dominates the flame graph; isolate CPU-bound work for clean results.
- Inlining hides functions: aggressive inlining can collapse several functions into one frame. Temporarily mark a function
#[inline(never)]to force it to appear separately in the flame graph.
The last point is subtle but useful. If you suspect a small utility function is called in a tight loop, mark it #[inline(never)], profile, verify it’s the culprit, then remove the attribute and optimize its caller.
Where to Go Next
Once you’ve found a hotspot with the tools above, fixing it is a different skill — data structures, cache locality, SIMD, lock contention, allocator pressure. The Rust performance book at https://nnethercote.github.io/perf-book/ is the best single resource for that phase. It’s written by Nicholas Nethercote, who spent years profiling the Firefox JavaScript engine, and it’s full of concrete Rust-specific advice.
The profilers are just the map. Knowing how to read it and where to go once you’ve found the bottleneck — that’s the real work.