Chinaunix首页 | 论坛 | 博客
  • 博客访问: 842938
  • 博文数量: 185
  • 博客积分: 495
  • 博客等级: 下士
  • 技术积分: 1417
  • 用 户 组: 普通用户
  • 注册时间: 2012-09-02 15:12
个人简介

治肾虚不含糖,专注内核性能优化二十年。 https://github.com/KnightKu

文章分类

全部博文(185)

文章存档

2019年(1)

2018年(12)

2017年(5)

2016年(23)

2015年(1)

2014年(22)

2013年(82)

2012年(39)

分类: LINUX

2018-09-19 10:07:42

这段时间在写zfs并行扫描的工具,被性能问题卡住了,于是又拾起来oprofile,碰到了些问题,记录下来。
1. opcontrol不好使了,比较是被丢弃的,启动的时候发现只有一个NMI counter,而且是已经被预留了的,用不了。

点击(此处)折叠或打开

  1. [root@server16 ~]# opcontrol --start --vmlinux=/usr/lib/debug/lib/modules/3.10.0-862.9.1.el7_lustre.2.10.5.x86_64/vmlinux
  2. ATTENTION: Use of opcontrol is discouraged. Please see the man page for operf.
  3. Using default event: CPU_CLK_UNHALTED:100000:0:1:1
  4. Using 2.6+ OProfile kernel interface.
  5. Reading module info.
  6. Failed to open profile device: Device or resource busy
  7. ^C
  8. [root@server16 ~]# dmesg | tail -1
  9. [63832.603647] oprofile: counter #0 is already reserved
stackoverflow上有人说清了watchdog好使,然而并非这样,清了还是不行:

  1. [root@server16 ~]# echo "0" | sudo tee /proc/sys/kernel/nmi_watchdog
  2. 0
  3. [root@server16 ~]# echo "0" | sudo tee /proc/sys/kernel/watchdog
  4. 0
  5. [root@server16 ~]# opcontrol --start --vmlinux=/usr/lib/debug/lib/modules/3.10.0-862.9.1.el7_lustre.2.10.5.x86_64/vmlinux
  6. ATTENTION: Use of opcontrol is discouraged. Please see the man page for operf.
  7. Using default event: CPU_CLK_UNHALTED:100000:0:1:1
  8. Using 2.6+ OProfile kernel interface.
  9. Reading module info.
  10. Failed to open profile device: Device or resource busy
2.切换到operf
既然opcontrol不行,那就用新的operf接口。

  1. [root@server16 ~]# cat operf.man
  2. OPERF(1) General Commands Manual OPERF(1)



  3. NAME
  4.        operf - Performance profiler tool for Linux


  5. SYNOPSIS
  6.        operf [ options ] [ --system-wide | --pid <pid> | [ command [ args ] ] ]


  7. DESCRIPTION
  8.        Operf is an OProfile tool that can be used in place of opcontrol for profiling. Operf uses the Linux Performance Events Subsystem, and hence, does not
  9.        require the use of the opcontrol daemon -- in fact, operf and opcontrol usage are mutually exclusive.

  10.        By default, operf uses <current_dir>/oprofile_data as the session-dir and stores profiling data there. You can change this by way of the --session-dir
  11.        option.

  12.        The usual post-profiling analysis tools such as opreport(1) and opannotate(1) can be used to generate profile reports. The post-processing analysis tools
  13.        will search for samples in <current_dir>/oprofile_data first. If that directory does not exist, the post-processing tools use the standard session-dir of
  14.        /var/lib/oprofile.

  15.        Statistics, such as total samples received and lost samples, are written to the operf.log file that can be found in the <session_dir>/samples directory.


  16. OPTIONS
  17.        command[args]
  18.               The command or application to be profiled. args are the input arguments that the command or application requires. One (and only one) of either com‐
  19.               mand , --pid or --system-wide is required.

  20.        --pid / -p PID
  21.               This option enables operf to profile a running application. PID should be the process ID of the process you wish to profile. When finished profil‐
  22.               ing (e.g., when the profiled process ends), press Ctrl-c to stop operf. If you run operf --pid as a background job (i.e., with the &), you must stop
  23.               it in a controlled manner in order for it to process the profile data it has collected. Use kill -SIGINT <operf-PID> for this purpose.

  24.        --system-wide / -s
  25.               This option is for performing a system-wide profile. You must have root authority to run operf in this mode. When finished profiling, Ctrl-c to
  26.               stop operf. If you run operf --system-wide as a background job (i.e., with the &), you must stop it in a controlled manner in order for it to process
  27.               the profile data it has collected. Use kill -SIGINT <operf-PID> for this purpose. It is recommended that when running operf with this option, the
  28.               user's current working directory should be /root or a subdirectory of /root to avoid storing sample data files in locations accessible by regular
  29.               users.

  30.        --vmlinux / k vmlinux_path
  31.               A vmlinux file that matches the running kernel that has symbol and/or debuginfo. Kernel samples will be attributed to this binary, allowing post-
  32.               processing tools (like opreport) to attribute samples to the appropriate kernel symbols.

  33.        --events / -e event1[,event2[,...]]
  34.               This option is for passing a comma-separated list of event specifications for profiling. Each event spec is of the form:
  35.                  name:count[:unitmask[:kernel[:user]]]
  36.               You can specify unit mask values using either a numerical value (hex values must begin with "0x") or a symbolic name (if the name= field is
  37.               shown in the ophelp output). For some named unit masks, the hex value is not unique; thus, OProfile tools enforce specifying such unit masks value by
  38.               name.

  39.               Event names for some IBM PowerPC systems include a _GRP (group number) suffix. You can pass either the full event name or the base event name
  40.               (i.e., without the suffix) to operf. If the base event name is passed, operf will automatically choose an appropriate group number suffix for the
  41.               event; thus, OProfile post-processing tools will always show real event names that include the group number suffix.

  42.               When no event specification is given, the default event for the running processor type will be used for profiling. Use ophelp to list the available
  43.               events for your processor type.

  44.        --callgraph / -g
  45.               This option enables the callgraph to be saved during profiling. NOTE: The full callchain is recorded, so there is no depth limit.

  46.        --separate-thread / -t
  47.               This option categorizes samples by thread group ID (tgid) and thread ID (tid). The '--separate-thread' option is useful for seeing per-thread sam‐
  48.               ples in multi-threaded applications. When used in conjunction with the '--system-wide' option, the '--separate-thread' option is also useful for
  49.               seeing per-process (i.e., per-thread group) samples for the case where multiple processes are executing the same program during a profiling run.

  50.        --separate-cpu / -c
  51.               This option categorizes samples by cpu.

  52.        --session-dir / -d path
  53.               This option specifies the session path to hold the sample data. If not specified, the data is saved in the oprofile_data directory on the current
  54.               path.

  55.        --lazy-conversion / -l
  56.               Use this option to reduce the overhead of operf during profiling. Normally, profile data received from the kernel is converted to OProfile format
  57.               during profiling time. This is typically not an issue when profiling a single application. But when using the --system-wide option, this on-the-fly
  58.               conversion process can cause noticeable overhead, particularly on busy multi-processor systems. The --lazy-conversion option directs operf to wait
  59.               until profiling is completed to do the conversion of profile data.

  60.        --append / -a
  61.               By default, operf moves old profile data from /samples/current to /samples/previous. If a 'previous' profile already
  62.               existed, it will be replaced. If the --append option is passed, old profile data is left in place and new profile data will be added to it, and the
  63.               'previous' profile (if one existed) will remain untouched. To access the 'previous' profile, simply add a session specification to the normal invo‐
  64.               cation of oprofile post-processing tools. For example:
  65.                  opreport session:previous

  66.        --verbose / -V level
  67.               A comma-separated list of debugging control values, used to increase the verbosity of the output. Valid values are: debug, record, convert, misc,
  68.               sfile, arcs, or the special value, 'all

看了下manpage,意思是将原来的 opcontrol setup/start/monitor/stop 这一套步骤可以合并到一条命令来完成,如下:

  1. [root@server16 ~]# operf --vmlinux=/usr/lib/debug/lib/modules/3.10.0-862.9.1.el7_lustre.2.10.5.x86_64/vmlinux du -hs /tmp
  2. operf: Profiler started
  3. 722M /tmp

  4. Profiling done.
profile的日志会存到当前目录下的oprofile_data目录里面:

  1. [root@server16 ~]# tree oprofile_data/
  2. oprofile_data/
  3. ├── abi
  4. └── samples
  5.     ├── current
  6.     │   ├── {root}
  7.     │   │   └── usr
  8.     │   │   └── bin
  9.     │   │   └── du
  10.     │   │   └── {dep}
  11.     │   │   ├── {kern}
  12.     │   │   │   └── ext4
  13.     │   │   │   └── CPU_CLK_UNHALTED.100000.0.all.all.all
  14.     │   │   └── {root}
  15.     │   │   └── usr
  16.     │   │   ├── bin
  17.     │   │   │   └── du
  18.     │   │   │   └── CPU_CLK_UNHALTED.100000.0.all.all.all
  19.     │   │   └── lib64
  20.     │   │   ├── ld-2.17.so
  21.     │   │   │   └── CPU_CLK_UNHALTED.100000.0.all.all.all
  22.     │   │   └── libc-2.17.so
  23.     │   │   └── CPU_CLK_UNHALTED.100000.0.all.all.all
  24.     │   └── stats
  25.     └── operf.log

  26. 17 directories, 6 files
报告和展示还是使用opreport命令:

  1. [root@server16 ~]# opreport
  2. Using /root/oprofile_data/samples/ for samples directory.
  3. CPU: Intel Sandy Bridge microarchitecture, speed 2.8e+06 MHz (estimated)
  4. Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (No unit mask) count 100000
  5. CPU_CLK_UNHALT...|
  6.   samples| %|
  7. ------------------
  8.        50 100.000 du
  9.         CPU_CLK_UNHALT...|
  10.           samples| %|
  11.         ------------------
  12.                18 36.0000 ext4
  13.                16 32.0000 libc-2.17.so
  14.                14 28.0000 du
  15.                 2 4.0000 ld-2.17.so

  16. [root@server16 ~]# opreport -l /usr/bin/du
  17. Using /root/oprofile_data/samples/ for samples directory.
  18. warning: /ext4 could not be found.
  19. CPU: Intel Sandy Bridge microarchitecture, speed 2.8e+06 MHz (estimated)
  20. Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (No unit mask) count 100000
  21. samples % image name symbol name
  22. 18 36.0000 ext4 /ext4
  23. 14 28.0000 du /usr/bin/du
  24. 3 6.0000 libc-2.17.so __memcpy_ssse3_back
  25. 2 4.0000 libc-2.17.so __close_nocancel
  26. 2 4.0000 libc-2.17.so _int_malloc
  27. 2 4.0000 libc-2.17.so malloc
  28. 1 2.0000 ld-2.17.so _dl_cache_libcmp
  29. 1 2.0000 ld-2.17.so do_lookup_x
  30. 1 2.0000 libc-2.17.so __memset_sse2
  31. 1 2.0000 libc-2.17.so __strlen_sse2_pminub
  32. 1 2.0000 libc-2.17.so closedir
  33. 1 2.0000 libc-2.17.so decode_name
  34. 1 2.0000 libc-2.17.so malloc_consolidate
  35. 1 2.0000 libc-2.17.so memmove
  36. 1 2.0000 libc-2.17.so openat64
总体感觉还是不错的!!!


阅读(1280) | 评论(0) | 转发(0) |
给主人留下些什么吧!~~
评论热议
请登录后评论。

登录 注册