31.5. Analyzing the Data

Periodically, the OProfile daemon, oprofiled, collects the samples and writes them to the /var/lib/oprofile/samples/ directory. Before reading the data, make sure all data has been written to this directory by executing the following command as root:

opcontrol --dump

Each sample file name is based on the name of the executable. For example, the samples for the default event on a Pentium III processor for /bin/bash becomes:

\{root\}/bin/bash/\{dep\}/\{root\}/bin/bash/CPU_CLK_UNHALTED.100000

The following tools are available to profile the sample data once it has been collected:

Use these tools, along with the binaries profiled, to generate reports that can be further analyzed.

WarningWarning
 

The executable being profiled must be used with these tools to analyze the data. If it must change after the data is collected, backup the executable used to create the samples as well as the sample files.

Samples for each executable are written to a single sample file. Samples from each dynamically linked library are also written to a single sample file. While OProfile is running, if the executable being monitored changes and a sample file for the executable exists, the existing sample file is automatically deleted. Thus, if the existing sample file is needed, it must be backed up, along with the executable used to create it before replacing the executable with a new version. Refer to Section 31.4 Saving Data for details on how to backup the sample file.

31.5.1. Using opreport

The opreport tool provides an overview of all the executables being profiled.

The following is part of an example output:

Profiling through timer interrupt
          TIMER:0|
  samples|      %|
------------------
    25926 97.5212 no-vmlinux
      359  1.3504 pi
       65  0.2445 Xorg
       62  0.2332 libvte.so.4.4.0
       56  0.2106 libc-2.3.4.so
       34  0.1279 libglib-2.0.so.0.400.7
       19  0.0715 libXft.so.2.1.2
       17  0.0639 bash
        8  0.0301 ld-2.3.4.so
        8  0.0301 libgdk-x11-2.0.so.0.400.13
        6  0.0226 libgobject-2.0.so.0.400.7
        5  0.0188 oprofiled
        4  0.0150 libpthread-2.3.4.so
        4  0.0150 libgtk-x11-2.0.so.0.400.13
        3  0.0113 libXrender.so.1.2.2
        3  0.0113 du
        1  0.0038 libcrypto.so.0.9.7a
        1  0.0038 libpam.so.0.77
        1  0.0038 libtermcap.so.2.0.8
        1  0.0038 libX11.so.6.2
        1  0.0038 libgthread-2.0.so.0.400.7
        1  0.0038 libwnck-1.so.4.9.0

Each executable is listed on its own line. The first column is the number of samples recorded for the executable. The second column is the percentage of samples relative to the total number of samples. The third column is the name of the executable.

Refer to the opreport man page for a list of available command line options, such as the -r option used to sort the output from the executable with the smallest number of samples to the one with the largest number of samples.

31.5.2. Using opreport on a Single Executable

To retrieve more detailed profiled information about a specific executable, use opreport:

opreport <mode> <executable>

<executable> must be the full path to the executable to be analyzed. <mode> must be one of the following:

-l

List sample data by symbols. For example, the following is part of the output from running the command opreport -l /lib/tls/libc-<version>.so:

samples  %        symbol name
12       21.4286  __gconv_transform_utf8_internal
5         8.9286  _int_malloc
4         7.1429  malloc
3         5.3571  __i686.get_pc_thunk.bx
3         5.3571  _dl_mcount_wrapper_check
3         5.3571  mbrtowc
3         5.3571  memcpy
2         3.5714  _int_realloc
2         3.5714  _nl_intern_locale_data
2         3.5714  free
2         3.5714  strcmp
1         1.7857  __ctype_get_mb_cur_max
1         1.7857  __unregister_atfork
1         1.7857  __write_nocancel
1         1.7857  _dl_addr
1         1.7857  _int_free
1         1.7857  _itoa_word
1         1.7857  calc_eclosure_iter
1         1.7857  fopen@@GLIBC_2.1
1         1.7857  getpid
1         1.7857  memmove
1         1.7857  msort_with_tmp
1         1.7857  strcpy
1         1.7857  strlen
1         1.7857  vfprintf
1         1.7857  write

The first column is the starting virtual memory address (vma). The second column is the number of samples for the symbol. The third column is the percentage of samples for this symbol relative to the overall samples for the executable, and the fourth column is the symbol name.

To sort the output from the largest number of samples to the smallest (reverse order), use -r in conjunction with the -l option.

-i <symbol-name>

List sample data specific to a symbol name. For example, the following output is from the command opreport -l -i __gconv_transform_utf8_internal /lib/tls/libc-<version>.so:

samples  %        symbol name
12       100.000  __gconv_transform_utf8_internal

The first line is a summary for the symbol/executable combination.

The first column is the number of samples for the memory symbol. The second column is the percentage of samples for the memory address relative to the total number of samples for the symbol. The third column is the symbol name.

-d

List sample data by symbols with more detail than -l. For example, the following output is from the command opreport -l -d __gconv_transform_utf8_internal /lib/tls/libc-<version>.so:

vma      samples  %        symbol name
00a98640 12       100.000  __gconv_transform_utf8_internal
00a98640 1         8.3333
00a9868c 2        16.6667
00a9869a 1         8.3333
00a986c1 1         8.3333
00a98720 1         8.3333
00a98749 1         8.3333
00a98753 1         8.3333
00a98789 1         8.3333
00a98864 1         8.3333
00a98869 1         8.3333
00a98b08 1         8.3333

The data is the same as the -l option except that for each symbol, each virtual memory address used is shown. For each virtual memory address, the number of samples and percentage of samples relative to the number of samples for the symbol is displayed.

-x <symbol-name>

Exclude the comma-separated list of symbols from the output.

session:<name>

Specify the full path to the session or a directory relative to the /var/lib/oprofile/samples/ directory.

31.5.3. Using opannotate

The opannotate tool tries to match the samples for particular instructions to the corresponding lines in the source code. The resulting files generated should have the samples for the lines at the left. It also puts in a comment at the beginning of each function listing the total samples for the function.

For this utility to work, the executable must be compiled with GCC's -g option. By default, Red Hat Enterprise Linux packages are not compiled with this option.

The general syntax for opannotate is as follows:

opannotate --search-dirs <src-dir> --source <executable>

The directory containing the source code and the executable to be analyzed must be specified. Refer to the opannotate man page for a list of additional command line options.