Oprofile/operf使用小结

这段时间在写zfs并行扫描的工具，被性能问题卡住了，于是又拾起来oprofile，碰到了些问题，记录下来。
1. opcontrol不好使了，比较是被丢弃的，启动的时候发现只有一个NMI counter，而且是已经被预留了的，用不了。

[root@server16 ~]# opcontrol --start --vmlinux=/usr/lib/debug/lib/modules/3.10.0-862.9.1.el7_lustre.2.10.5.x86_64/vmlinux
ATTENTION: Use of opcontrol is discouraged. Please see the man page for operf.
Using default event: CPU_CLK_UNHALTED:100000:0:1:1
Using 2.6+ OProfile kernel interface.
Reading module info.
Failed to open profile device: Device or resource busy
^C
[root@server16 ~]# dmesg | tail -1
[63832.603647] oprofile: counter #0 is already reserved

stackoverflow上有人说清了watchdog好使，然而并非这样，清了还是不行：

[root@server16 ~]# echo "0" | sudo tee /proc/sys/kernel/nmi_watchdog
0
[root@server16 ~]# echo "0" | sudo tee /proc/sys/kernel/watchdog
0
[root@server16 ~]# opcontrol --start --vmlinux=/usr/lib/debug/lib/modules/3.10.0-862.9.1.el7_lustre.2.10.5.x86_64/vmlinux
ATTENTION: Use of opcontrol is discouraged. Please see the man page for operf.
Using default event: CPU_CLK_UNHALTED:100000:0:1:1
Using 2.6+ OProfile kernel interface.
Reading module info.
Failed to open profile device: Device or resource busy

2.切换到operf
既然opcontrol不行，那就用新的operf接口。

[root@server16 ~]# cat operf.man
OPERF(1) General Commands Manual                                                              OPERF(1)



NAME
       operf - Performance profiler tool for Linux


SYNOPSIS
       operf [ options ] [ --system-wide | --pid <pid> | [ command [ args ] ] ]


DESCRIPTION
       Operf  is  an  OProfile  tool  that  can be used in place of opcontrol for profiling. Operf uses the Linux Performance Events Subsystem, and hence, does not
       require the use of the opcontrol daemon -- in fact, operf and opcontrol usage are mutually exclusive.

       By default, operf uses <current_dir>/oprofile_data as the session-dir and stores profiling data there. You can change this by  way of the --session-dir
       option.

       The  usual  post-profiling analysis tools such as opreport(1) and opannotate(1) can be used to generate profile reports. The post-processing analysis tools
       will search for samples in <current_dir>/oprofile_data first. If that directory does not exist, the post-processing tools use the  standard  session-dir of
       /var/lib/oprofile.

       Statistics, such as total samples received and lost samples, are written to the operf.log file that can be found in the <session_dir>/samples directory.


OPTIONS
       command[args]
              The command or application to be profiled. args are the input arguments that the command or application requires. One (and only one) of either com‐
              mand , --pid or --system-wide is required.

       --pid / -p PID
              This option enables operf to profile a running application. PID should be the process ID of the process you wish to profile. When finished  profil‐
              ing (e.g., when the profiled process ends), press Ctrl-c to stop operf. If you run operf --pid as a background job (i.e., with the &), you must stop
              it in a controlled manner in order for it to process the profile data it has collected. Use kill -SIGINT <operf-PID> for this purpose.

       --system-wide / -s
              This option is for performing a system-wide profile. You must have root authority to run operf in this mode. When  finished  profiling, Ctrl-c  to
              stop operf. If you run operf --system-wide as a background job (i.e., with the &), you must stop it in a controlled manner in order for it to process
              the profile data it has collected. Use kill -SIGINT <operf-PID> for this purpose. It is recommended that when running operf with this option, the
              user's  current  working  directory  should be /root or a subdirectory of /root to avoid storing sample data files in locations accessible by regular
              users.

       --vmlinux / k vmlinux_path
              A vmlinux file that matches the running kernel that has symbol and/or debuginfo.  Kernel samples will be attributed to this  binary,  allowing  post-
              processing tools (like opreport) to attribute samples to the appropriate kernel symbols.

       --events / -e event1[,event2[,...]]
              This option is for passing a comma-separated list of event specifications for profiling. Each event spec is of the form:
                 name:count[:unitmask[:kernel[:user]]]
              You  can specify unit mask values using either a numerical value (hex values must begin with "0x") or a symbolic name (if the name= field is
              shown in the ophelp output). For some named unit masks, the hex value is not unique; thus, OProfile tools enforce specifying such unit masks value by
              name.

              Event  names  for  some  IBM  PowerPC systems include a _GRP (group number) suffix. You can pass either the full event name or the base event name
              (i.e., without the suffix) to operf.  If the base event name is passed, operf will automatically choose an appropriate group number  suffix  for  the
              event; thus, OProfile post-processing tools will always show real event names that include the group number suffix.

              When  no event specification is given, the default event for the running processor type will be used for profiling.  Use ophelp to list the available
              events for your processor type.

       --callgraph / -g
              This option enables the callgraph to be saved during profiling. NOTE: The full callchain is recorded, so there is no depth limit.

       --separate-thread / -t
              This option categorizes samples by thread group ID (tgid) and thread ID (tid).  The '--separate-thread' option is useful for seeing  per-thread  sam‐
              ples  in  multi-threaded  applications.   When used in conjunction with the '--system-wide' option, the '--separate-thread' option is also useful for
              seeing per-process (i.e., per-thread group) samples for the case where multiple processes are executing the same program during a profiling run.

       --separate-cpu / -c
              This option categorizes samples by cpu.

       --session-dir / -d path
              This option specifies the session path to hold the sample data. If not specified, the data is saved in the oprofile_data  directory  on  the  current
              path.

       --lazy-conversion / -l
              Use  this  option  to  reduce the overhead of operf during profiling. Normally, profile data received from the kernel is converted to OProfile format
              during profiling time. This is typically not an issue when profiling a single application. But when using the --system-wide option,  this  on-the-fly
              conversion  process  can  cause noticeable overhead, particularly on busy multi-processor systems. The --lazy-conversion option directs operf to wait
              until profiling is completed to do the conversion of profile data.

       --append / -a
              By default, operf moves old profile data from /samples/current to  /samples/previous.   If  a  'previous'  profile  already
              existed,  it will be replaced.  If the --append option is passed, old profile data is left in place and new profile data will be added to it, and the
              'previous' profile (if one existed) will remain untouched.  To access the 'previous' profile, simply add a session specification to the normal  invo‐
              cation of oprofile post-processing tools.  For example:
                 opreport session:previous

       --verbose / -V level
              A  comma-separated  list of debugging control values, used to increase the verbosity of the output.  Valid values are:  debug, record, convert, misc,
              sfile, arcs, or the special value, 'all

看了下manpage，意思是将原来的 opcontrol setup/start/monitor/stop 这一套步骤可以合并到一条命令来完成，如下：

[root@server16 ~]# operf --vmlinux=/usr/lib/debug/lib/modules/3.10.0-862.9.1.el7_lustre.2.10.5.x86_64/vmlinux du -hs /tmp
operf: Profiler started
722M    /tmp

Profiling done.

profile的日志会存到当前目录下的oprofile_data目录里面：

[root@server16 ~]# tree oprofile_data/
oprofile_data/
├── abi
└── samples
    ├── current
    │   ├── {root}
    │   │   └── usr
    │   │       └── bin
    │   │           └── du
    │   │               └── {dep}
    │   │                   ├── {kern}
    │   │                   │   └── ext4
    │   │                   │       └── CPU_CLK_UNHALTED.100000.0.all.all.all
    │   │                   └── {root}
    │   │                       └── usr
    │   │                           ├── bin
    │   │                           │   └── du
    │   │                           │       └── CPU_CLK_UNHALTED.100000.0.all.all.all
    │   │                           └── lib64
    │   │                               ├── ld-2.17.so
    │   │                               │   └── CPU_CLK_UNHALTED.100000.0.all.all.all
    │   │                               └── libc-2.17.so
    │   │                                   └── CPU_CLK_UNHALTED.100000.0.all.all.all
    │   └── stats
    └── operf.log

17 directories, 6 files

报告和展示还是使用opreport命令：

[root@server16 ~]# opreport
Using /root/oprofile_data/samples/ for samples directory.
CPU: Intel Sandy Bridge microarchitecture, speed 2.8e+06 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (No unit mask) count 100000
CPU_CLK_UNHALT...|
  samples| %|
------------------
       50 100.000 du
        CPU_CLK_UNHALT...|
          samples| %|
        ------------------
               18 36.0000 ext4
               16 32.0000 libc-2.17.so
               14 28.0000 du
                2  4.0000 ld-2.17.so

[root@server16 ~]# opreport -l /usr/bin/du
Using /root/oprofile_data/samples/ for samples directory.
warning: /ext4 could not be found.
CPU: Intel Sandy Bridge microarchitecture, speed 2.8e+06 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (No unit mask) count 100000
samples  %        image name symbol name
18       36.0000  ext4                     /ext4
14       28.0000  du                       /usr/bin/du
3         6.0000  libc-2.17.so             __memcpy_ssse3_back
2         4.0000  libc-2.17.so             __close_nocancel
2         4.0000  libc-2.17.so             _int_malloc
2         4.0000  libc-2.17.so             malloc
1         2.0000  ld-2.17.so               _dl_cache_libcmp
1         2.0000  ld-2.17.so               do_lookup_x
1         2.0000  libc-2.17.so             __memset_sse2
1         2.0000  libc-2.17.so             __strlen_sse2_pminub
1         2.0000  libc-2.17.so             closedir
1         2.0000  libc-2.17.so             decode_name
1         2.0000  libc-2.17.so             malloc_consolidate
1         2.0000  libc-2.17.so             memmove
1         2.0000  libc-2.17.so             openat64

总体感觉还是不错的！！！