ORA-27102: out of memory 故障

 

  最近的UAT数据库迁移，由于是多个DB需要迁移到同一台机器，一部分完成后，启动后续数据库碰到了ORA-27102错误，提示内存超出，查看系统可用内存，远大于需要启动数据库的sga和pga，究竟是怎么一回事呢？如果你也碰到类似错误，不妨往下看。

 

1、故障现象
oracle@v2013db02u:~> cat /etc/issue

Welcome to SUSE Linux Enterprise Server 10 SP3 (x86_64) - Kernel \r (\l).

oracle@v2013db02u:~> sqlplus / as sysdba

SQL*Plus: Release 10.2.0.3.0 - Production on Sat Mar 30 09:46:57 2013

Copyright (c) 1982, 2006, Oracle.  All Rights Reserved.

Connected to an idle instance.

SQL> startup pfile=/u02/database/MHUAT02/initMHUAT02.ora
ORA-27102: out of memory
Linux-x86_64 Error: 28: No space left on device
SQL> ho oerr ora 27102
27102, 00000, "out of memory"
// *Cause: Out of memory
// *Action: Consult the trace file for details

2、分析与解决
--首先查看了系统实际可用内存
--系统实际可用内存还剩50多个GB呢
SQL> exit
Disconnected
oracle@v2013db02u:/u02/database> free -m
             total       used       free     shared    buffers     cached
Mem:         64435       5968      58467          0          3       4045
-/+ buffers/cache:       1919      62516
Swap:        32773       1232      31540

--MetaLink上给出了关于这个问题的描述 [ID 301830.1]
--是由于kernel.shmall值设置太小引起的

kernel.shmall
      该参数用于配置系统一次能够使用的最大的共享页面数，该参数的值总是ceil(shmmax/PAGE_SIZE)
      Oracle 9i,10g的x86以及x86-64平台推荐以及缺省的大小通常为2097152
      在大多数情况下，该参数是够用的。按照上述计算方式页面内存总大小可以达到8GB(2097152*4096 bytes (shmall*PAGE_SIZE))
      通常情况下，PAGE_SIZE 的大小为4096byte，除非使用了Big Pages 或 Huge Pages
      对于超出8GB系统内存，如共享内存段(shmmax)的最大大小是16G，则所需要共享内存页数(shmall)为16GB/4KB=16777216KB/4KB=4194304(页)
      32GB，64GB可以依上述方式类推。

 

--下面查看本机的设置
v2013db02u:~ # getconf PAGE_SIZE
4096

v2013db02u:~ # /sbin/sysctl -a | grep shm
vm.hugetlb_shm_group = 0
kernel.shmmni = 4096         --定义共享内存段的最大数量，也就是可以开辟多少个共享内存段，一般够用 
kernel.shmall = 2097152     --由于当前server有多个db运行，应该是总页数超出所致
kernel.shmmax = 4294967296  --定义单个共享内存段的最大值，由于当前server没有sga大于4gb的，可以保留原值

--前面我们在使用free时，系统使用内存已经超出了4gb，也表明shmall的超出了最大的共享页面数

 

--Author : Robinson
--Blog   : http://blog.csdn.net/robinson_0612

 

v2013db02u:~ # vi /etc/sysctl.conf  --修改kernel.shmall的值，将其改为16gb能容纳的页面数，使用root身份
v2013db02u:~ # sysctl -p                --使修改立即生效，无需重启server
v2013db02u:~ # /sbin/sysctl -a | grep shm
vm.hugetlb_shm_group = 0
kernel.shmmni = 4096
kernel.shmall = 4194304
kernel.shmmax = 4294967296

 

--再次启动db正常
oracle@v2013db02u:~> export ORACLE_SID=MHUAT02
oracle@v2013db02u:~> sqlplus / as sysdba

SQL*Plus: Release 10.2.0.3.0 - Production on Sat Mar 30 10:29:18 2013

Copyright (c) 1982, 2006, Oracle.  All Rights Reserved.

Connected to an idle instance.

SQL> startup pfile=/u02/database/MHUAT02/initMHUAT02.ora
ORACLE instance started.

Total System Global Area  599785472 bytes
Fixed Size                  2074568 bytes
Variable Size             167774264 bytes
Database Buffers          423624704 bytes
Redo Buffers                6311936 bytes
Database mounted.
Database opened.

 

3、[ID 301830.1]
Upon startup of Linux database get ORA-27102: out of memory Linux-X86_64 Error: 28: No space left on device [ID 301830.1] To Bottom 
--------------------------------------------------------------------------------
Modified:Oct 4, 2012Type:PROBLEMStatus:PUBLISHEDPriority:3 Comments (0)    

In this Document
 Symptoms
  Ask Questions, Get Help, And Share Your Experiences With This Article
 Changes
 Cause
 Solution
 References
--------------------------------------------------------------------------------
Applies to:
Oracle Server - Standard Edition - Version 9.2.0.1 and later
Oracle Server - Enterprise Edition - Version 9.2.0.1 and later
Linux x86-64

Symptoms
      When trying to increase the SGA to approach half available RAM with an Oracle 64-bit version on a Linux 64-bit operating system, even though shmmax is set to match half the amount of  RAM, you get the following error when trying to start the instance:

SQL> startup nomount
ORA-27102: out of memory
Linux-x86_64 Error: 28: No space left on device
Ask Questions, Get Help, And Share Your Experiences With This Article
Would you like to explore this topic further with other Oracle Customers, Oracle Employees, and Industry Experts?

Click here to join the discussion where you can ask questions, get help from others, and share your experiences with this specific article.
Discover discussions about other articles and helpful subjects by clicking here to access the main My Oracle Support Community page for Database Install/Upgrade.

 

Changes
      shmall is too small, most likely is set to the default setting of 2097152

$ cat /proc/sys/kernel/shmall
2097152

 

Cause
      shmall is the total amount of shared memory, in pages, that the system can use at one time.

 

Solution
      Set shmall equal to the sum of all the SGAs on the system, divided by the page size.

The page size can be determined using the following command:

 

$ getconf PAGE_SIZE
4096
For example, if the sum of all the SGAs on the system is 16Gb and the result of  '$ getconf PAGE_SIZE' is 4096 (4Kb) then set shmall to 4194304 pages

As the root user set the shmall to 4194304 in the /etc/sysctl.conf file:

kernel.shmall = 4194304

then run the following command:

$ sysctl -p
$ cat /proc/sys/kernel/shmall
4194304

 

NOTE:

The above command loads the new value and a reboot is not necessary.
Switch back to being the oracle user and retry the startup command.
Modifying /etc/sysctl.conf is a permanent workaround (applies at boot time). If for some reason you DO NOT want to change the system wide configuration, you can do it on the fly by directly changing the kernel pseudo FS AKA procfs.

e.g. echo "4194304" > /proc/sys/kernel/shmall

Using HUGEPAGES does not alter the calculation for configuring shmall.