性能调试(四)内存性能分析

1:内存管理
2:衡量内存闲忙程度的指标
3:内存资源成为系统性能的瓶颈的征兆
4:什么地方/哪些进程是占用内存资源的大户?
5:利用vmstat命令分析内存的利用率
6:利用ipcs分析消息队列、共享内存和信号量
7:利用GlancePlus分析系统内存资源利用率
8:对内存需求密集型系统的性能调试
内存管理


1)内存管理的主要工作：

跟踪内存的使用和可用内存的情况；
为进程分配内存；
管理磁盘与物理内存之间的换页(paging);
2)什么是虚拟内存(virtual memory)?

virtual memory uses a disk as an extension of RAM so that the effective size of usable memory grows correspondingly. The kernel will write the contents of a currently unused block of memory to the hard disk so that the memory can be used for another purpose. When the original contents are needed again, they are read back into memory. This is all made completely transparent to the user; programs only see the larger amount of memory available and don't notice that parts of them reside on the disk from time to time. Of course, reading and writing the hard disk is slower (on the order of a thousand times slower) than using real memory, so the programs don't run as fast. The part of the hard disk that is used as virtual memory is called the swap space.

物理内存(physical memory)
swap space:
3)paging

A technique by which the contents of a virtual memory address(called pages) are moved from virtual memory(the disk)into and out of the main memory where it is accessed by the CPU.

这个机制是由一个叫vhand的进程来完成。

当可用内存的数量小于LOTSFREE时，例程pageout将被调用来选择什么内存可以释放。它采用two-handed clock algorithm,the reference hand turn off the reference bit of each memory page it references.If the refernce bit is still zero when the second hand gets to it,the page is freed.If the page is clean(unmodified),it is added to the freelist.If the page is dirty,it must be posted to the swap device before being put on the freelist.

lotsfree: based on physical memory (64 MB -> 863) upper bound where paging starts/stops
desfree: based on physical memory (64 MB -> 215)lower bound where paging starts/stops
minfree: based on physical memory (64 MB -> 53) threshold where deactivation starts
4)Page fault

An invalid address error(trap)that occurs when the CPU requests a page from memory that has not been paged in from the disk(virtual memory)to the main memory. The page may also have been paged out from the main memory prior the request.

5)Process Deactivaton

Deactivating a process so its memory pages will be flagged free or paged out rapidly by vhand.

6)Thrashing

A situation in which a process is spending more time paging than processing;a high number of page faults is occuring.

7)Buffer Cache

它是内存的一部分，用于加快文件存取时间；
缓存的大小可以随可用内存动态变化，但也可以通过修改内核参数而改成固定的大小；
缓存可以提高磁盘的读/写性能；
在缓存的内容可以通过sync进程来强制写入磁盘；
从缓存的读和写又称为逻辑读和逻辑写；
8)内存需求

按用途来分，内存可以分成两部分：预留内存和动态内存。

预留内存主要用于存放：

system table
data structures
buffer cache
其中系统表和数据结构占用的数量一般很小，但缓存则可能占到很大一部分。

动态内存主要用于存放：

process text
data stack
share memory segments
其中各进程锁定的内存会影响动态内存的大小。


衡量内存闲忙程度的指标


1)整体内存忙闲指标：

buffer cache size: 缓存区在内存开销中占很大比例；
page in/out rates;
swap in/out rates;
可用内存的大小，或用得到内存的大小(available memory size):
自由内存的大小(free memory size): what is currently available,it should not be confuzed with available memory,which does not change during normal sytem operation;
swap queue length;
2)单个进程的内存衡量指标：

一个进程占用物理内存的大小(resident set size)
一个进程占用虚拟内存的大小(virtual set size)
VM reads and writes: it can show how many physical memory management reads and writes were made to and from the disk during the chosen interval.

内存资源成为系统性能的瓶颈的征兆


当内存资源成为系统性能的瓶颈时，它有一些典型的症状：

很高的换页率(high pageout rate):HP-UX是一个按需调页的操作系统，通常情况下，它只执行调入页面进入内存的操作，以让进程能够运行。只有操作系统觉得系统需要释放一些内存空间时，才会执行从内存调出页面的操作，而过高的调出页面操作说明内存缺乏；
进程进入不活动状态(process deactivation activity):当自由的内存页面数量小于MINFREE时，很多进程将强制进入不活动状态，因为，any deactivation activity represents a condition in which normal paging is inadequate to handle the memory demands.
自由内存的数量很小，但活动的虚拟内存却很大(very small free memory and large active virtual memory)
交换区所有磁盘的活动次数可高(high disk activity on swap devices)
可高的全局系统CPU利用率(high global system CPU utilization):
很长的运行进程队列，但CPU的空闲时间却很多(large run queue with idle CPU)
内存不够出错(out of memory errors)
CPU用于vhand和swapper两中守护进程的时间(CPU time to vhand and swapper)
必须注意的是，有时候我们发现CPU很忙，这似乎是CPU资源成为系统性能的瓶颈，但如果进一步分析，发现vhand和swapper守护进程占用了大量的系统CPU时间，很显然，这时系统性能瓶颈真正所在可能是内存。


什么地方/哪些进程是占用内存资源的大户?


下面是一些典型的占用内存资源的大户：

buffer cache
plocked process：plocked processes are those that are locked in memory and are not elogible to be paged.通常，这些进程都是一些比较重要的进程，不便调出内存。
档案库(archive libraries)：把库放入内存可以加快程序的执行，但它们将占用大量的内存；
共享内存(shared memory)
关系型数据库(relational databases)
X-终端和X-服务器进程(X-terminals and X-servers)：通常，一个X-终端需要额外的2-4兆内存；一个X-服务器需要400KB以上的内存；

利用vmstat命令分析内存的利用率


vmstat-report virtual memory statistics

The vmstat command reports certain statistics kept about process, virtual memory, trap, and CPU activity. It also can clear the accumulators in the kernel sum structure.

它的语法：

vmstat [-dnS] [interval [count]]

vmstat -f | -s | -z

它的选项的说明：

-d: Report disk transfer information as a separate section, in the form of transfers per second.
-n: Provide an output format that is more easily viewed on an 80-column display device. This format separates the default output into two groups: virtual memory information and CPU data. Each group is displayed as a separate line of output. On multiprocessor systems, this display format also provides CPU utilization on a per CPU basis.
-S: Report the number of processes swapped in and out (si and so) instead of page reclaims and address translation faults (re and at)
interval: Display successive lines which are summaries over the last interval seconds. If interval is zero, the output is displayed once only. If the -d option is specified, the column headers are repeated. If -d is omitted, the column headers are not repeated.
count: Repeat the summary statistics count times. If count is omitted or zero, the output is repeated until an interrupt or quit signal is received.
-f: Report on the number of forks and the number of pages of virtual memory involved since boot-up.
-s: Print the total number of several kinds of paging- related events from the kernel sum structure that have occurred since boot-up or since vmstat was last executed with the -z option.
-z: Clear all accumulators in the kernel sum structure. This option is restricted to the super user.
对结果的说明：

在不带参数的vmstat的命令时，我们首先要关注的是avm(active virtual memory)列和free(free list zise)列的值。如果avm的值很大，而free的值却很小，这时，系统可能有内存瓶颈，我们 必须用带-S选项的vmstat命令查看系统是否有deactivation/reactivation活动正在发生。


利用ipcs分析消息队列、共享内存和信号量


ipcs - report status of interprocess communication facilities

ipcs displays certain information about active interprocess communication facilities. With no options, ipcs displays information in short format for the message queues, shared memory segments, and semaphores that are currently active in the system.

它的语法：

ipcs [-mqs] [-abcopt] [-C core] [-N namelist]


利用GlancePlus分析系统内存资源利用率


利用HP的GlancePlus可以对内存使用情况进行分析：

进入GlancePlus；
按?键进入联机帮助界面；
按m键进入内存的详细界面；
通过memory Detail Screen，我们可以知道内存的分布情况，物理内存多少、缓存多少、用户用了多少，系统用了多少、以及换页情况等。


对内存需求密集型系统的性能调试


1)基于硬件的方法：

增加物理内存
使用无盘工作站替代X-terminal
2)基于软件的方法：

减小内核参数maxdsiz的值；
减少内存锁定的使用；
杀死不必要的进程；
识别出需要大量内存的进程；
重新设计应用；
减小内核的大小；
减小系统表的大小；
减小缓存区的大小；
3)利用serialize命令来调度大进程的系统资源需求

/usr/bin/serialize applicaiton
/usr/bin/serialize -p pid