A flamegraph is a visualization used to understand where a program spends its time — most commonly for CPU profiling, but also for memory, I/O, or other performance profiling.
A flamegraph is a stack trace visualization where:
- Each box represents a function.
- The width of a box represents how much total time that function or its children spent running.
- The x-axis shows relative time spent in different code paths.
- The y-axis shows the call stack (bottom = root, top = leaf).
- The “flames” grow upward, but the colors don’t mean anything special — they’re just for readability.
Flamegraphs were invented by Brendan Gregg and are extremely common in performance engineering.
They help you quickly answer:
- Which functions use the most CPU?
- What call paths are hot?
- Where should I optimize first?
- Is the program CPU bound, or is time spent blocking?
Problems on using it:
linux kernel 6.14 doesn’t have perf, so I need to use an older version 6.8
use the full path of perf:
/usr/lib/linux-tools-6.8.0-88/perf record -e task-clock -F 99 -g -- ../target/release-with-debug/powdr_openvm prove guest-keccak --input 100 --autoprecompiles 1 --apc-candidates-dir keccak100 --mock --optimistic-precompiles
perf on binary, but the flamegraph doesn’t show the function name, a lot of unknown stuff:
The binary needs to include debug info, add these into the cargo.toml:
[profile.release-with-debug]
inherits = "release"
debug = true
strip = "none"and when build the binary, use flag:
cargo clean
RUSTFLAGS="-C force-frame-pointers=yes" \
cargo build --bin powdr_openvm
a lot of error related to wrong path can happen.
Command
/usr/lib/linux-tools-6.8.0-88/perf record -e task-clock -F 99 -g -- ../target/release-with-debug/powdr_openvm prove guest-keccak --input 100 --autoprecompiles 1 --apc-candidates-dir keccak100 --mock --optimistic-precompiles
/usr/lib/linux-tools-6.14.0-33-generic/perf script > out.perf
/home/shuang/FlameGraph/stackcollapse-perf.pl out.perf > out.folded
/home/shuang/FlameGraph/flamegraph.pl out.folded > flamegraph.svg
open flamegraph.svg
also give perf permission:
sudo sysctl -w kernel.perf_event_paranoid=1
# or 0 if 1 still complains
sudo sysctl -w kernel.kptr_restrict=0