NAME
perf-record - Run a command and record its profile into perf.dataSYNOPSIS
perf record [-e <EVENT> | --event=EVENT] [-a] <command> perf record [-e <EVENT> | --event=EVENT] [-a] -- <command> [<options>]
DESCRIPTION
This command runs a command and gathers a performance counter profile from it, into perf.data - without displaying anything.OPTIONS
<command>...Any command you can specify in a shell.
Select the PMU event. Selection can be:
•a symbolic event name (use perf
list to list all events)
•a raw PMU event in the form of rN
where N is a hexadecimal value that represents the raw register encoding with
the layout of the event control registers as described by entries in
/sys/bus/event_source/devices/cpu/format/*.
•a symbolic or raw PMU event followed
by an optional colon and a list of event modifiers, e.g., cpu-cycles:p. See
the perf-list(1) man page for details on event modifiers.
•a symbolically formed PMU event like
pmu/param1=0x3,param2/ where param1, param2, etc are
defined as formats for the PMU in
/sys/bus/event_source/devices/<pmu>/format/*.
•a symbolically formed event like
pmu/config=M,config1=N,config3=K/
where M, N, K are numbers (in decimal, hex, octal format). Acceptable values for each of 'config', 'config1' and 'config2' are defined by corresponding entries in /sys/bus/event_source/devices/<pmu>/format/* param1 and param2 are defined as formats for the PMU in: /sys/bus/event_source/devices/<pmu>/format/*
There are also some parameters which are not defined in .../<pmu>/format/*. These params can be used to overload default config values per event. Here are some common parameters: - 'period': Set event sampling period - 'freq': Set event sampling frequency - 'time': Disable/enable time stamping. Acceptable values are 1 for enabling time stamping. 0 for disabling time stamping. The default is 1. - 'call-graph': Disable/enable callgraph. Acceptable str are "fp" for FP mode, "dwarf" for DWARF mode, "lbr" for LBR mode and "no" for disable callgraph. - 'stack-size': user stack size for dwarf mode - 'name' : User defined event name. Single quotes (') may be used to escape symbols in the name from parsing by shell and tool like this: name=\'CPU_CLK_UNHALTED.THREAD:cmask=0x1\'. - 'aux-output': Generate AUX records instead of events. This requires that an AUX area event is also provided. - 'aux-sample-size': Set sample size for AUX area sampling. If the '--aux-sample' option has been used, set aux-sample-size=0 to disable AUX area sampling for the event.
See the linkperf:perf-list[1] man page for more parameters.
Note: If user explicitly sets options which conflict with the params, the value set by the parameters will be overridden.
Also not defined in .../<pmu>/format/* are PMU driver specific configuration parameters. Any configuration parameter preceded by the letter '@' is not interpreted in user space and sent down directly to the PMU driver. For example:
perf record -e some_event/@cfg1,@cfg2=config/ ...
will see 'cfg1' and 'cfg2=config' pushed to the PMU driver associated with the event for further processing. There is no restriction on what the configuration parameters are, as long as their semantic is understood and supported by the PMU driver.
•a hardware breakpoint event in the
form of \mem:addr[/len][:access] where addr is the address in memory
you want to break in. Access is the memory access type (read, write, execute)
it can be passed as follows: \mem:addr[:[r][w][x]]. len is the range,
number of bytes from specified addr, which the breakpoint will cover. If you
want to profile read-write accesses in 0x1000, just set mem:0x1000:rw.
If you want to profile write accesses in [0x1000~1008), just set
mem:0x1000/8:w.
•a BPF source file (ending in .c) or a
precompiled object file (ending in .o) selects one or more BPF events. The BPF
program can attach to various perf events based on the ELF section names.
When processing a '.c' file, perf searches an installed LLVM to compile it into an object file first. Optional clang options can be passed via the '--clang-opt' command line option, e.g.:
perf record --clang-opt "-DLINUX_VERSION_CODE=0x50000" \ -e tests/bpf-script-example.c
Note: '--clang-opt' must be placed before '--event/-e'.
•a group of events surrounded by a pair
of brace ("{event1,event2,...}"). Each event is separated by commas
and the group should be quoted to prevent the shell interpretation. You also
need to use --group on "perf report" to view group events
together.
Event filter. This option should follow an event selector (-e) which selects either tracepoint event(s) or a hardware trace PMU (e.g. Intel PT or CoreSight).
•tracepoint filters
In the case of tracepoints, multiple '--filter' options are combined using '&&'.
•address filters
A hardware trace PMU advertises its ability to accept a number of address filters by specifying a non-zero value in /sys/bus/event_source/devices/<pmu>/nr_addr_filters.
Address filters have the format:
filter|start|stop|tracestop <start> [/ <size>] [@<file name>]
Where: - 'filter': defines a region that will be traced. - 'start': defines an address at which tracing will begin. - 'stop': defines an address at which tracing will stop. - 'tracestop': defines a region in which tracing will stop.
<file name> is the name of the object file, <start> is the offset to the code to trace in that file, and <size> is the size of the region to trace. 'start' and 'stop' filters need not specify a <size>.
If no object file is specified then the kernel is assumed, in which case the start address must be a current kernel memory address.
<start> can also be specified by providing the name of a symbol. If the symbol name is not unique, it can be disambiguated by inserting #n where 'n' selects the n'th symbol in address order. Alternately #0, #g or #G select only a global symbol. <size> can also be specified by providing the name of a symbol, in which case the size is calculated to the end of that symbol. For 'filter' and 'tracestop' filters, if <size> is omitted and <start> is a symbol, then the size is calculated to the end of that symbol.
If <size> is omitted and <start> is '*', then the start and size will be calculated from the first and last symbols, i.e. to trace the whole file.
If symbol names (or '*') are provided, they must be surrounded by white space.
The filter passed to the kernel is not necessarily the same as entered. To see the filter that is passed, use the -v option.
The kernel may not be able to configure a trace region if it is not within a single mapping. MMAP events (or /proc/<pid>/maps) can be examined to determine if that is a possibility.
Multiple filters can be separated with space or comma.
Don’t record events issued by perf
itself. This option should follow an event selector (-e) which selects
tracepoint event(s). It adds a filter expression common_pid != $PERFPID
to filters. If other --filter exists, the new filter expression will be
combined with them by &&.
System-wide collection from all CPUs (default
if no target is specified).
Record events on existing process ID (comma
separated list).
Record events on existing thread ID (comma
separated list). This option also disables inheritance by default. Enable it
by adding --inherit.
Record events in threads owned by uid. Name or
number.
Collect data with this RT SCHED_FIFO
priority.
Collect data without buffering.
Event period to sample.
Output file name.
Child tasks do not inherit counters.
Profile at this frequency. Use max to
use the currently maximum allowed frequency, i.e. the value in the
kernel.perf_event_max_sample_rate sysctl. Will throttle down to the currently
maximum allowed frequency. See --strict-freq.
Fail if the specified frequency can’t
be used.
Number of mmap data pages (must be a power of
two) or size specification with appended unit character - B/K/M/G. The size is
rounded up to have nearest pages power of two value. Also, by adding a comma,
the number of mmap pages for AUX area tracing can be specified.
Put all events in a single event group. This
precedes the --event option and remains only for backward compatibility. See
--event.
Enables call-graph (stack chain/backtrace)
recording for both kernel space and user space.
Setup and enable call-graph (stack
chain/backtrace) recording, implies -g. Default is "fp" (for user
space).
The unwinding method used for kernel space is dependent on the unwinder used by the active kernel configuration, i.e CONFIG_UNWINDER_FRAME_POINTER (fp) or CONFIG_UNWINDER_ORC (orc)
Any option specified here controls the method used for user space.
Valid options are "fp" (frame pointer), "dwarf" (DWARF's CFI - Call Frame Information) or "lbr" (Hardware Last Branch Record facility).
In some systems, where binaries are build with gcc --fomit-frame-pointer, using the "fp" method will produce bogus call graphs, using "dwarf", if available (perf tools linked to the libunwind or libdw library) should be used instead. Using the "lbr" method doesn't require any compiler options. It will produce call graphs from the hardware LBR registers. The main limitation is that it is only available on new Intel platforms, such as Haswell. It can only get user call chain. It doesn't work with branch stack sampling at the same time.
When "dwarf" recording is used, perf also records (user) stack dump when sampled. Default size of the stack dump is 8192 (bytes). User can change the size by passing the size after comma like "--call-graph dwarf,4096".
When "fp" recording is used, perf tries to save stack enties up to the number specified in sysctl.kernel.perf_event_max_stack by default. User can change the number by passing it after comma like "--call-graph fp,32".
Don’t print any warnings or messages,
useful for scripting.
Be more verbose (show counter open errors,
etc).
Record per-thread event counts. Use it with
perf report -T to see the values.
Record the sample virtual addresses.
Record the sample physical addresses.
Record the sampled data address data page
size.
Record the sampled code address (ip) page
size
Record the sample timestamps. Use it with
perf report -D to see the timestamps, for instance.
Record the sample period.
Record the sample cpu.
Record the sample identifier i.e.
PERF_SAMPLE_IDENTIFIER bit set in the sample_type member of the struct
perf_event_attr argument to the perf_event_open system call.
Don’t sample.
Collect raw sample records from all opened
counters (default for tracepoint counters).
Collect samples only on the list of CPUs
provided. Multiple CPUs can be provided as a comma-separated list with no
space: 0,1. Ranges of CPUs are specified with -: 0-2. In per-thread mode with
inheritance mode on (default), samples are captured only when the thread
executes on the designated CPUs. Default is to monitor all CPUs.
Do not save the build ids of binaries in the
perf.data files. This skips post processing after recording, which sometimes
makes the final step in the recording process to take a long time, as it needs
to process all events looking for mmap records. The downside is that it can
misresolve symbols if the workload binaries used when recording get locally
rebuilt or upgraded, because the only key available in this case is the
pathname. You can also set the "record.build-id" config variable to
'skip to have this behaviour permanently.
Do not update the buildid cache. This saves
some overhead in situations where the information in the perf.data file (which
includes buildids) is sufficient. You can also set the
"record.build-id" config variable to no-cache to have the
same effect.
monitor only in the container (cgroup) called
"name". This option is available only in per-cpu mode. The cgroup
filesystem must be mounted. All threads belonging to container
"name" are monitored when they run on the monitored CPUs. Multiple
cgroups can be provided. Each cgroup is applied to the corresponding event,
i.e., first cgroup to first event, second cgroup to second event and so on. It
is possible to provide an empty cgroup (monitor all the time) using, e.g., -G
foo,,bar. Cgroups must have corresponding events, i.e., they always refer to
events defined earlier on the command line. If the user wants to track
multiple events for a specific cgroup, the user can use -e e1 -e e2 -G
foo,foo or just use -e e1 -e e2 -G foo.
Enable taken branch stack sampling. Any type
of taken branch may be sampled. This is a shortcut for --branch-filter any.
See --branch-filter for more infos.
Enable taken branch stack sampling. Each
sample captures a series of consecutive taken branches. The number of branches
captured with each sample depends on the underlying hardware, the type of
branches of interest, and the executed code. It is possible to select the
types of branches captured by enabling filters. The following filters are
defined:
The option requires at least one branch type among any, any_call, any_ret,
ind_call, cond. The privilege levels may be omitted, in which case, the
privilege levels of the associated event are applied to the branch filter.
Both kernel (k) and hypervisor (hv) privilege levels are subject to
permissions. When sampling on multiple events, branch stack sampling is
enabled for all the sampling events. The sampled branch type is the same for
all events. The various filters must be specified as a comma separated list:
--branch-filter any_ret,u,k Note that this feature may not be available on all
processors.
•any: any type of branches
•any_call: any function call or system
call
•any_ret: any function return or system
call return
•ind_call: any indirect branch
•call: direct calls, including far
(to/from kernel) calls
•u: only when the branch target is at
the user level
•k: only when the branch target is in
the kernel
•hv: only when the target is at the
hypervisor level
•in_tx: only when the target is in a
hardware transaction
•no_tx: only when the target is not in
a hardware transaction
•abort_tx: only when the target is a
hardware transaction abort
•cond: conditional branches
•save_type: save branch type during
sampling in case binary is not available later For the platforms with Intel
Arch LBR support (12th-Gen+ client or 4th-Gen Xeon+ server), the save branch
type is unconditionally enabled when the taken branch stack sampling is
enabled.
•priv: save privilege state during
sampling in case binary is not available later
Enable weightened sampling. An additional
weight is recorded per sample and can be displayed with the weight and
local_weight sort keys. This currently works for TSX abort events and some
memory events in precise mode on modern Intel CPUs.
Record events of type PERF_RECORD_NAMESPACES.
This enables cgroup_id sort key.
Record events of type PERF_RECORD_CGROUP. This
enables cgroup sort key.
Record transaction flags for transaction
related events.
Use per-thread mmaps. By default per-cpu mmaps
are created. This option overrides that and uses per-thread mmaps. A
side-effect of that is that inheritance is automatically disabled.
--per-thread is ignored with a warning if combined with -a or -C
options.
After starting the program, wait msecs before
measuring (-1: start with events disabled), or enable events only for
specified ranges of msecs (e.g. -D 10-20,30-40 means wait 10 msecs, enable for
10 msecs, wait 10 msecs, enable for 10 msecs, then stop). Note, delaying
enabling of events is useful to filter out the startup phase of the program,
which is often very different.
Capture machine state (registers) at
interrupt, i.e., on counter overflows for each sample. List of captured
registers depends on the architecture. This option is off by default. It is
possible to select the registers to sample using their symbolic names, e.g. on
x86, ax, si. To list the available registers use --intr-regs=\?. To name
registers, pass a comma separated list such as --intr-regs=ax,bx. The list of
register is architecture dependent.
Similar to -I, but capture user registers at
sample time. To list the available user registers use --user-regs=\?.
Record running and enabled time for read
events (:S)
Sets the clock id to use for the various time
fields in the perf_event_type records. See clock_gettime(). In particular
CLOCK_MONOTONIC and CLOCK_MONOTONIC_RAW are supported, some events might also
allow CLOCK_BOOTTIME, CLOCK_REALTIME and CLOCK_TAI.
Select AUX area tracing Snapshot Mode. This
option is valid only with an AUX area tracing event. Optionally, certain
snapshot capturing parameters can be specified in a string that follows this
option: e: take one last snapshot on exit; guarantees that there is at
least one snapshot in the output file; <size>: if the PMU supports this,
specify the desired snapshot size.
Select AUX area sampling. At least one of the
events selected by the -e option must be an AUX area event. Samples on other
events will be created containing data from the AUX area. Optionally sample
size may be specified, otherwise it defaults to 4KiB.
When processing pre-existing threads
/proc/XXX/mmap, it may take a long time, because the file may be huge. A time
out is needed in such cases. This option sets the time out limit. The default
value is 500 ms.
Record context switch events i.e. events of
type PERF_RECORD_SWITCH or PERF_RECORD_SWITCH_CPU_WIDE. In some cases (e.g.
Intel PT, CoreSight or Arm SPE) switch events will be enabled automatically,
which can be suppressed by by the option --no-switch-events.
Path to clang binary to use for compiling BPF
scriptlets. (enabled when BPF support is on)
Options passed to clang when compiling BPF
scriptlets. (enabled when BPF support is on)
Specify vmlinux path which has debuginfo.
(enabled when BPF prologue is on)
Record build-id of all DSOs regardless whether
it’s actually hit or not.
Record build ids in mmap2 events, disables
build id cache (implies --no-buildid).
Use <n> control blocks in asynchronous
(Posix AIO) trace writing mode (default: 1, max: 4). Asynchronous mode is
supported only when linking Perf tool with libc library providing
implementation for Posix AIO API.
Set affinity mask of trace reading thread
according to the policy defined by mode value: node - thread affinity
mask is set to NUMA node cpu mask of the processed mmap buffer cpu - thread
affinity mask is set to cpu of the processed mmap buffer
Specify minimal number of bytes that is
extracted from mmap data pages and processed for output. One can specify the
number using B/K/M/G suffixes.
Produce compressed trace using specified level
n (default: 1 - fastest compression, 22 - smallest trace)
Configure all used events to run in kernel
space.
Configure all used events to run in user
space.
Collect callchains only from kernel space.
I.e. this option sets perf_event_attr.exclude_callchain_user to 1.
Collect callchains only from user space. I.e.
this option sets perf_event_attr.exclude_callchain_kernel to 1.
Record timestamp boundary (time of first/last
samples).
Generate multiple perf.data files, timestamp
prefixed, switching to a new one based on mode value:
"signal" - when receiving a SIGUSR2 (default value) or <size>
- when reaching the size threshold, size is expected to be a number with
appended unit character - B/K/M/G <time> - when reaching the time
threshold, size is expected to be a number with appended unit character -
s/m/h/d
Note: the precision of the size threshold hugely depends on your configuration - the number and size of your ring buffers (-m). It is generally more precise for higher sizes (like >5M), for lower values expect different sizes.
--switch-output --no-no-buildid --no-no-buildid-cache
Events that will cause the switch of the
perf.data file, auto-selecting --switch-output=signal, the results are similar
as internally the side band thread will also send a SIGUSR2 to the main
one.
When rotating perf.data with --switch-output,
only keep N files.
Parse options then exit. --dry-run can be used
to detect errors in cmdline options.
Collect and synthesize given type of events
(comma separated). Note that this option controls the synthesis from the /proc
filesystem which represent task status for pre-existing threads.
Instead of collecting non-sample events (for
example, fork, comm, mmap) at the beginning of record, collect them during
finalizing an output file. The collected non-sample events reflects the status
of the system when record is finished.
Makes all events use an overwritable ring
buffer. An overwritable ring buffer works like a flight recorder: when it gets
full, the kernel will overwrite the oldest records, that thus will never make
it to the perf.data file.
Make a copy of /proc/kcore and place it into a
directory with the perf data file.
Limit the sample data max size, <size>
is expected to be a number with appended unit character - B/K/M/G
The number of threads to run when synthesizing
events for existing processes. By default, the number of threads equals
1.
ctl-fifo / ack-fifo are opened and used as
ctl-fd / ack-fd as follows. Listen on ctl-fd descriptor for command to control
measurement.
'evlist [-v|-g|-F] : display all events -F Show just the sample frequency used for each event. -v Show all fields. -g Show event group information.
#!/bin/bash
ctl_dir=/tmp/
ctl_fifo=${ctl_dir}perf_ctl.fifo test -p ${ctl_fifo} && unlink ${ctl_fifo} mkfifo ${ctl_fifo} exec {ctl_fd}<>${ctl_fifo}
ctl_ack_fifo=${ctl_dir}perf_ctl_ack.fifo test -p ${ctl_ack_fifo} && unlink ${ctl_ack_fifo} mkfifo ${ctl_ack_fifo} exec {ctl_fd_ack}<>${ctl_ack_fifo}
perf record -D -1 -e cpu-cycles -a \ --control fd:${ctl_fd},${ctl_fd_ack} \ -- sleep 30 & perf_pid=$!
sleep 5 && echo 'enable' >&${ctl_fd} && read -u ${ctl_fd_ack} e1 && echo "enabled(${e1})" sleep 10 && echo 'disable' >&${ctl_fd} && read -u ${ctl_fd_ack} d1 && echo "disabled(${d1})"
exec {ctl_fd_ack}>&- unlink ${ctl_ack_fifo}
exec {ctl_fd}>&- unlink ${ctl_fifo}
wait -n ${perf_pid} exit $?
Write collected trace data into several data
files using parallel threads. <spec> value can be user defined list of
masks. Masks separated by colon define CPUs to be monitored by a thread and
affinity mask of that thread is separated by slash:
<cpus mask 1>/<affinity mask 1>:<cpus mask 2>/<affinity mask 2>:...
0,2-4/2-4:1,5-7/5-7
cpu - create new data streaming thread for every monitored cpu core - create new thread to monitor CPUs grouped by a core package - create new thread to monitor CPUs grouped by a package numa - create new threed to monitor CPUs grouped by a NUMA domain
Specify debuginfod URL to be used when
cacheing perf.data binaries, it follows the same syntax as the DEBUGINFOD_URLS
variable, like:
http://192.168.122.174:8002
If the URLs is not specified, the value of DEBUGINFOD_URLS system environment variable is used.
Enable off-cpu profiling with BPF. The BPF
program will collect task scheduling information with (user) stacktrace and
save them as sample data of a software event named "offcpu-time".
The sample period will have the time the task slept in nanoseconds.
Note that BPF can collect stack traces using frame pointer ("fp") only, as of now. So the applications built without the frame pointer might see bogus addresses.
INTEL HYBRID SUPPORT
Support for Intel hybrid events within perf tools.cpu_core/<event name>/ or cpu_atom/<event name>/
perf stat -e cpu_core/cycles/
------------------------------------------------------------ perf_event_attr: size 120 config 0x400000000 sample_type IDENTIFIER read_format TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING disabled 1 inherit 1 exclude_guest 1 ------------------------------------------------------------
------------------------------------------------------------ perf_event_attr: size 120 config 0x800000000 sample_type IDENTIFIER read_format TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING disabled 1 inherit 1 exclude_guest 1 ------------------------------------------------------------
Performance counter stats for 'system wide':
6,744,979 cpu_core/cycles/ 1,965,552 cpu_atom/cycles/
perf_event_attr: size 120 config 0x400000000 sample_type IDENTIFIER read_format TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING disabled 1 inherit 1 enable_on_exec 1 exclude_guest 1
perf_event_attr: size 120 config 0x800000000 sample_type IDENTIFIER read_format TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING disabled 1 inherit 1 enable_on_exec 1 exclude_guest 1
Performance counter stats for 'taskset -c 16 ./triad_loop':
233,066,666 cpu_core/cycles/ (0.43%) 604,097,080 cpu_atom/cycles/ (99.57%)
SEE ALSO
perf-stat(1), perf-list(1), perf-intel-pt(1)2024-06-21 | perf |