分类： Oracle
2017-05-18 22:22:57
Configuring HugePages for Oracle on Linux (x86-64)
一、介绍
二、配置HugePages
三、强制Oracle使用HugePages (USE_LARGE_PAGES)
四、关闭Transparent HugePages
五、配置1G HugePagesize
一、介绍
对于分配大型SGA大小,HugePages可以给虚拟内存管理的实实在在的利益，没有HugePages,SGA的内存页分将会为4 k的内存页,这必须由Linux内核来进行管理.
使用HugePages,页面大小增加到2 mb(如果支持的硬件配置为1 g),从而减少了总页数由内核来管理,因此减少所需的内存页表在内存中。除了这些变化,与HugePages相关联的内存不能被换出,这迫使SGA保持内存常驻。节省内存和页面管理的努力使HugePages几乎强制Oracle 11 g系统上运行x86 - 64架构。
仅仅因为你有一个大SGA,它并不意味着你会有一个问题,如果你不使用HugePages。它通常是大型SGA和大量数据库连接的组合，从而导致问题。
为了确定当前使用多少内存来支持页表，在服务器处于正常/重负载的情况下运行以下命令
|
# grep PageTables /proc/meminfo PageTables: 1244880 kB # |
自动内存管理(AMM)不兼容Linux HugePages，相反,自动共享内存管理和自动PGA管理应使用与HugePages兼容
二、配置HugePages
运行以下命令来确定当前HugePage使用。默认的HugePage大小是2 mb在Oracle Linux 5.x
? 从下面的输出可以看出,默认情况下没有HugePages定义
|
$ grep Huge /proc/meminfo AnonHugePages: 0 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB $ |
? 创建一个文件叫做hugepages_setting.sh,以下列内容为实际输出
|
#!/bin/bash # # hugepages_settings.sh # # Linux bash script to compute values for the # recommended HugePages/HugeTLB configuration # # Note: This script does calculation for all shared memory # segments available when the script is run, no matter it # is an Oracle RDBMS shared memory segment or not. # Check for the kernel version KERN=`uname -r | awk -F. '{ printf("%d.%d\n",$1,$2); }'` # Find out the HugePage size HPG_SZ=`grep Hugepagesize /proc/meminfo | awk {'print $2'}` # Start from 1 pages to be on the safe side and guarantee 1 free HugePage NUM_PG=1 # Cumulative number of pages required to handle the running shared memory segments for SEG_BYTES in `ipcs -m | awk {'print $5'} | grep "[0-9][0-9]*"` do MIN_PG=`echo "$SEG_BYTES/($HPG_SZ*1024)" | bc -q` if [ $MIN_PG -gt 0 ]; then NUM_PG=`echo "$NUM_PG+$MIN_PG+1" | bc -q` fi done # Finish with results case $KERN in '2.4') HUGETLB_POOL=`echo "$NUM_PG*$HPG_SZ/1024" | bc -q`; echo "Recommended setting: vm.hugetlb_pool = $HUGETLB_POOL" ;; '2.6' | '3.8' | '3.10' | '4.1' ) echo "Recommended setting: vm.nr_hugepages = $NUM_PG" ;; *) echo "Unrecognized kernel version $KERN. Exiting." ;; esac # End |
? 确保文件有执行权限
|
$ chmod u+x hugepages_setting.sh |
确保所有的Oracle服务在服务器上正常运行
? 记录下运行hugepages_setting.sh脚本后推荐的vm.nr_hugepages 值
|
$ ./hugepages_setting.sh Recommended setting: vm.nr_hugepages = 305 $ |
? Root用户下编辑/etc/sysctl.conf文件，根据脚本的输出调整后的内容
|
vm.nr_hugepages=306 |
有些人说在Oracle Linux 6.5上需要设置hugetlb_shm_group，
? 如果需要，使用如下命令获得输出
|
# fgrep dba /etc/group dba:x:54322:oracle # |
? 将结果添加进/etc/sysctl.conf
|
vm.hugetlb_shm_group=54322 |
? 将修改的参数立即生效
|
# sysctl -p |
? 可以看到配置已经生效，但是当前还没被使用
|
$ grep Huge /proc/meminfo AnonHugePages: 0 kB HugePages_Total: 306 HugePages_Free: 306 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB $ |
添加如下内容/etc/security/limits.conf or /etc/security/limits.d/99-grid-oracle-limits.conf
计算HugePages分配的大小KB(HugePages * Hugepagesize)，在这里为306 * 2048 = 2048
|
* soft memlock 626688 * hard memlock 626688 |
检查MEMORY_TARGET参数没有设置为数据库和SGA_TARGET/PGA_AGGREGATE_TARGET参数所使用
|
SQL> show parameter target
NAME TYPE VALUE ------------------------------------ ----------- ------------------------------ archive_lag_target integer 0 db_flashback_retention_target integer 1440 fast_start_io_target integer 0 fast_start_mttr_target integer 0 memory_max_target big integer 0 memory_target big integer 0 parallel_servers_target integer 16 pga_aggregate_target big integer 200M sga_target big integer 600M SQL> |
? 重启OS 和 数据库实例后，再次检查hugepages参数
|
$ grep Huge /proc/meminfo AnonHugePages: 0 kB HugePages_Total: 306 HugePages_Free: 98 HugePages_Rsvd: 93 HugePages_Surp: 0 Hugepagesize: 2048 kB $ |
可以看到，hugepages配置已经生效
注意,如果增加了内存分配或添加新的实例,需要重新测试所需数量的HugePages
三、强制Oracle使用HugePages (USE_LARGE_PAGES)
Sizing the number of HugePages correctly is important because prior to 11.2.0.3, if the whole SGA doesn't fit into the available HugePages, the instance will start up without using any. From 11.2.0.3 onward, the SGA can run partly in HugePages and partly not, so the impact of this issue is not so great. Incorrect sizing may not be obvious to spot. Later releases of the database display a "Large Pages Information" section in the alert log during startup.
****************** Large Pages Information *****************
Total Shared Global Region in Large Pages = 602 MB (100%)
Large Pages used by this instance: 301 (602 MB)
Large Pages unused system wide = 5 (10 MB) (alloc incr 4096 KB)
Large Pages configured system wide = 306 (612 MB)
Large Page size = 2048 KB
***********************************************************
如果正在运行Oracle 11.2.0.2或之后的版本,可以将USE_LARGE_PAGES初始化参数设置为“only”
|
ALTER SYSTEM SET use_large_pages=only SCOPE=SPFILE; SHUTDOWN IMMEDIATE; STARTUP; |
在启动时，警报日志中的“Large Page Information”反映了该参数的使用
****************** Large Pages Information *****************
Parameter use_large_pages = ONLY
Total Shared Global Region in Large Pages = 602 MB (100%)
Large Pages used by this instance: 301 (602 MB)
Large Pages unused system wide = 5 (10 MB) (alloc incr 4096 KB)
Large Pages configured system wide = 306 (612 MB)
Large Page size = 2048 KB
***********************************************************
? 试图启动数据库时没有足够的HugePages持有SGA现在将返回如下错误
|
SQL> STARTUP ORA-27137: unable to allocate large pages to create a shared memory segment Linux-x86_64 Error: 12: Cannot allocate memory SQL> |
警告日志输出的“Large Page Information”部分描述了启动失败和采取的适当行动
|
****************** Large Pages Information ***************** Parameter use_large_pages = ONLY
Large Pages unused system wide = 0 (0 KB) (alloc incr 4096 KB) Large Pages configured system wide = 0 (0 KB) Large Page size = 2048 KB
ERROR: Failed to allocate shared global region with large pages, unix errno = 12. Aborting Instance startup. ORA-27137: unable to allocate Large Pages to create a shared memory segment
ACTION: Total Shared Global Region size is 608 MB. Increase the number of unused large pages to atleast 304 (608 MB) to allocate 100% Shared Global Region with Large Pages. *********************************************************** |
四、关闭Transparent HugePages
从RHEL6 / OL6、Transparent HugePages实现默认启用。他们是为了提高内存管理通过允许HugePages动态分配的“khugepaged”内核线程,而不是在引导时像传统HugePages.
可以使用下面的命令检查当前的设置 enabled=[always]
|
# cat /sys/kernel/mm/transparent_hugepage/enabled [always] madvise never # |
Oracle Linux 6 禁用transparent_hugepage 首选方法是“transparent_hugepage =never”添加到内核引导在/boot/grub/grub.conf文件中
|
title Oracle Linux Server (2.6.39-400.24.1.el6uek.x86_64) root (hd0,0) kernel /vmlinuz-2.6.39-400.24.1.el6uek.x86_64 ro root=/dev/mapper/vg_ol6112-lv_root rd_NO_LUKS KEYBOARDTYPE=pc KEYTABLE=uk LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 rd_NO_DM rd_LVM_LV=vg_ol6112/lv_swap rd_LVM_LV=vg_ol6112/lv_root rhgb quiet numa=off transparent_hugepage=never initrd /initramfs-2.6.39-400.24.1.el6uek.x86_64.img |
Oracle Linux7 的操作是相似的,但使用GRUB2所以需要编辑/boot/grub2/grub.cfg 使用grubby的命令文件”。
|
# grubby --default-kernel /boot/vmlinuz-4.1.12-61.1.6.el7uek.x86_64
# grubby --args="transparent_hugepage=never" --update-kernel /boot/vmlinuz-4.1.12-61.1.6.el7uek.x86_64
# grubby --info /boot/vmlinuz-4.1.12-61.1.6.el7uek.x86_64 index=2 kernel=/boot/vmlinuz-4.1.12-61.1.6.el7uek.x86_64 args="ro vconsole.font=latarcyrheb-sun16 rd.lvm.lv=ol/swap rd.lvm.lv=ol/root crashkernel=auto vconsole.keymap=uk rhgb quiet LANG=en_GB.UTF-8 transparent_hugepage=never" root=/dev/mapper/ol-root initrd=/boot/initramfs-4.1.12-61.1.6.el7uek.x86_64.img title=Oracle Linux Server 7.2, with Unbreakable Enterprise Kernel 4.1.12-61.1.6.el7uek.x86_64 |
服务器必须重新启动才能生效,另外,将以下代码添加到/etc/rc.loca文件并重新启动服务器
|
if test -f /sys/kernel/mm/transparent_hugepage/enabled; then echo never > /sys/kernel/mm/transparent_hugepage/enabled fi if test -f /sys/kernel/mm/transparent_hugepage/defrag; then echo never > /sys/kernel/mm/transparent_hugepage/defrag fi |
? 无论选择哪种方法，请记住在重新启动后检查更改是否有效
|
# cat /sys/kernel/mm/transparent_hugepage/enabled always madvise [never] # |
OL7 / RHEL7还需要考虑tuned profile
下面的脚本展示了如何创建和启用当前活动调优配置文件的修改版本
|
# # Check the active profile # tuned-adm active Current active profile: virtual-guest #
# # Create directory to hold revised profile. # mkdir /etc/tuned/virtual-guest-nothp
# # Create new profile based on the curren active profile.
# cat < [main] include= virtual-guest
[vm] transparent_hugepages=never EOF
# # Make the script executable. # chmod +x /etc/tuned/virtual-guest-nothp/tuned.conf
# # Enable the new profile. # tuned-adm profile virtual-guest-nothp |
五、配置1G HugePagesize
检查如果当前的硬件能够支持Hugepagesize 1G大小
如果下面的命令产生任何输出，它就可以分配1G大小
|
# cat /proc/cpuinfo | grep pdpe1gb |
编辑/etc/grub.conf文件，将以下条目添加到默认grub条目的内核行中。调整“hugepages”条目所需的1 g的页面数量。注意这包括Transparent HugePages的禁用,这不是强制性的,但一个好方法。
|
transparent_hugepage=never hugepagesz=1G hugepages=1 default_hugepagesz=1G |
检查当前的HugePages设置
|
# grep Huge /proc/meminfo HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB # |
重启并再次检查HugePages设置
|
# grep Huge /proc/meminfo HugePages_Total: 1 HugePages_Free: 1 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 1048576 kB # |
