Oracle on Solaris 10 : Fixing the 'ORA-27102: out

Oracle on Solaris 10 : Fixing the 'ORA-27102: out of memory' Error

(Crossposting the 2+ year old blog entry from my other blog hosted on blogger. Source URL:
http://technopark02.blogspot.com/2006/09/solaris-10oracle-fixing-ora-27102-out.html)

Symptom:

As part of a database tuning effort you increase the SGA/PGA sizes; and Oracle greets with an ORA-27102: out of memory error message. The system had enough free memory to serve the needs of Oracle.

SQL> startup
ORA-27102: out of memory
SVR4 Error: 22: Invalid argument

Diagnosis

$ oerr ORA 27102
27102, 00000, "out of memory"
// *Cause: Out of memory
// *Action: Consult the trace file for details

Not so helpful. Let's look the alert log for some clues.

% tail -2 alert.log
WARNING: EINVAL creating segment of size 0x000000028a006000
fix shm parameters in /etc/system or equivalent

Oracle is trying to create a 10G shared memory segment (depends on SGA/PGA sizes), but operating system (Solaris in this example) responded with an invalid argument (EINVAL) error message. There is a little hint about setting shm parameters in /etc/system.

Prior to Solaris 10, shmsys:shminfo_shmmax parameter has to be set in /etc/system with maximum memory segment value that can be created. 8M is the default value on Solaris 9 and prior versions; where as 1/4th of the physical memory is the default on Solaris 10 and later. On a Solaris 10 (or later) system, it can be verified as shown below:

% prtconf | grep Mem
Memory size: 32760 Megabytes

% id -p
uid=59008(oracle) gid=10001(dba) projid=3(default)

% prctl -n project.max-shm-memory -i project 3
project: 3: default
NAME    PRIVILEGE       VALUE    FLAG   ACTION                       RECIPIENT
project.max-shm-memory
        privileged      7.84GB      -   deny                                 -
        system          16.0EB    max   deny                                 -

Now it is clear that the system is using the default value of 8G in this scenario, where as the application (Oracle) is trying to create a memory segment (10G) larger than 8G. Hence the failure.

So, the solution is to configure the system with a value large enough for the shared segment being created, so Oracle succeeds in starting up the database instance.

On Solaris 9 and prior releases, it can be done by adding the following line to /etc/system, followed by a reboot for the system to pick up the new value.

set shminfo_shmmax = 0x000000028a006000

However shminfo_shmmax parameter was obsoleted with the release of Solaris 10; and Sun doesn't recommend setting this parameter in /etc/system even though it works as expected.

---

1. solaris10开始，，已删除了以下参数：
      ■ semsys:seminfo_semmaem
      ■ semsys:seminfo_semmap
      ■ semsys:seminfo_semmns
      ■ semsys:seminfo_semmnu
      ■ semsys:seminfo_semvmx
      ■ semsys:seminfo_semume
      ■ semsys:seminfo_semusz
      ■ shmsys:shminfo_shmmin
      ■ shmsys:shminfo_shmseg
      以下参数已过时：
      ■ shmsys:shminfo_shmmni
      ■ shmsys:shminfo_shmmax
   也就是说，不再需要修改/etc/system文件并reboot，而是通过prctl 命令来进行实时修改，再用projmod命令可以将此保存，重启后仍然生效。projmod命令是 将参数保存在 /etc/project文件中。

2. solaris10如果采用默认方式安装Oracle10g,只需要修改max-shm-memory这一个参数即可。
在ORACLE的官方文档中，将max-shm-memory和shmsys:shminfo_shmmax这l两个参数对应起来，并加了一个推荐值4294967295，但实际上这两个参数意义并不相同。shmsys:shminfo_shmmax是一个共享内存段的最大值，而project.max-shm-memory是属于同一个project的用户所能够创建的共享内存总和最大值，在数值上，project.max-shm-memory=shmsys:shminfo_shmmax*shmsys:shminfo_shmmni 。
   例如，在/etc/system 文件中配置如下：
    set semsys:seminfo_semmni=100
    set semsys:seminfo_semmns=1024
    set semsys:seminfo_semmsl=256
    set semsys:seminfo_semvmx=32767
    set shmsys:shminfo_shmmax=4294967295
    set shmsys:shminfo_shmmni=100
可以查到的project.max-shm-memory是400G

3. 如果project.max-shm-memory为4G的话，在不考虑其他用户使用共享内存的情况下，则oracle用户下所有数据库的SGA和不能超过4G，如果只有一个数据库，则这个数据库的SGA不能大过4G。
    安装过程中会报out of memory 错误，应该是由于Oracle创建shared memory segment 时，其大小超过了设置的max-shm-memory值，而Oracle创建shared memory segment 是根据SGA/PGA 大小的，所以由此得到两种解决办法：一是调小SGA，另一个是将max-shm-memory改大（也不是越大越好，一般为物理内存的一半即可，具体根据实际情况修改）。

---

On Solaris 10 and later, this value can be changed dynamically on a per project basis with the help of resource control facilities . This is how we do it on Solaris 10 and later:

% prctl -n project.max-shm-memory -r -v 10G -i project 3

% prctl -n project.max-shm-memory -i project 3
project: 3: default
NAME    PRIVILEGE       VALUE    FLAG   ACTION                       RECIPIENT
project.max-shm-memory
        privileged      10.0GB      -   deny                                 -
        system          16.0EB    max   deny                                 -

Note that changes done with prctl command on a running system are temporary, and will be lost when the system is rebooted. To make the changes permanent, create a project with projadd command as shown below:

% projadd -p 3  -c 'eBS benchmark' -U oracle -G dba  -K 'project.max-shm-memory=(privileged,10G,deny)' OASB

Finally make sure the project is created with projects -l or cat /etc/project commands.

% projects -l
...
...
OASB
        projid : 3
        comment: "eBS benchmark"
        users  : oracle
        groups : dba
        attribs: project.max-shm-memory=(privileged,10737418240,deny)

% cat /etc/project
...
...
OASB:3:eBS benchmark:oracle:dba:project.max-shm-memory=(privileged,10737418240,deny)

With these changes, Oracle would start the database up normally.

SQL> startup
ORACLE instance started.

Total System Global Area 1.0905E+10 bytes
Fixed Size                  1316080 bytes
Variable Size            4429966096 bytes
Database Buffers         6442450944 bytes
Redo Buffers               31457280 bytes
Database mounted.
Database opened.

Related information:

4. Man page of
5. Man page of

---

Addendum : Oracle RAC settings

Anonymous Bob suggested the following settings for Oracle RAC in the form of a comment for the benefit of others who run into similar issue(s) when running Oracle RAC. I'm pasting the comment as is (Disclaimer: I have not verified these settings):

Thanks for a great explanation, I would like to add one comment that will help those with an Oracle RAC installation. Modifying the default project covers oracle processes great and is all that is needed for a single instance DB. In RAC however, the CRS process starts the DB and it is a root owned process and root does not use the default project. To fix ORA-27102 issue for RAC I added the following lines to an init script that runs before the init.crs script fires.

# Recommended Oracle RAC system params
ndd -set /dev/udp udp_xmit_hiwat 65536
ndd -set /dev/udp udp_recv_hiwat 65536

# For root processes like crsd
prctl -n project.max-shm-memory -r -v 8G -i project system
prctl -n project.max-shm-ids -r -v 512 -i project system

# For oracle processes like sqlplus
prctl -n project.max-shm-memory -r -v 8G -i project default
prctl -n project.max-shm-ids -r -v 512 -i project default

So simple yet it took me a week working with Oracle and SUN to come up with that answer...Hope that helps someone out.

Bob
# posted by Blogger Bob : 6:48 AM, April 25, 2008

Posted at 01:10AM Nov 22, 2008 by Giri Mandalika in Solaris  |  Comments[12]

Comments:

Good.This doc helped me.

I had one more question.

I have a 3 node cluster , which is already configured,

When I do a ps -ef |grep oracle withy Project details , I see it running with user.oracle instead of the default system project.

How can I setup it up that way?

Regards,

vijaykumar Kammar

Posted by vijay on December 23, 2008 at 07:00 AM PST #

Hi,

Thanks a lot for this blog...helped me solve a critical issue!!

~Kaushik

Posted by Kaushik Sethna on March 06, 2009 at 10:49 AM PST #

Thanks a lot for this blog...helped me solve a critical issue!!

Posted by on March 09, 2009 at 11:35 AM PDT #

tnx

Posted by on March 19, 2009 at 05:41 AM PDT #

We were struggling with the same problem for two days.
This article is written in very precise manner and we could solve our problem.

Thanks for such a nice article.

Posted by Tanweer Khan on April 07, 2009 at 11:33 PM PDT #

Thank you for this great blog. I was trying to install an SAP - Oracle central instance and it wasn't going well. Very informative and easy to follow. Thanks much!

Posted by Rick on April 21, 2009 at 11:11 AM PDT #

Thanks for such a good and concise article

Regards
Lanto

Posted by Lanto RAZAKALALAO on April 28, 2009 at 04:49 AM PDT #

good doc

Posted by Babu on June 07, 2009 at 09:42 PM PDT #

Thanks this blog.
but I found there have some issue when set default project:

bash-3.00# prctl -n project.max-shm-memory -r -v 8G -i project default
prctl: default: No controllable process found in task, project, or zone.

but I edited the file /etc/project as following, it worked.

zone1 oracle $ cat /etc/project
system:0::::
user.root:1::::
noproject:2::::
default:3::::project.max-shm-memory=(privileged,2147483648,deny)

Posted by Jason Wang on July 20, 2009 at 08:10 PM PDT #

So Sun, in its infinite wisdom, took a process where all one had to do was edit one file and replaced it with this debacle?

No wonder the company died.
How does a change like this even get approved?

Lets take a simple, one step process and replace it with projects, multiple settings and even more commands with untold flags that one has to know.

Brilliant. Thanks.

Posted by Ian on August 12, 2009 at 11:13 AM PDT #

So Sun, in its infinite wisdom, took a process where all one had to do was edit one file and replaced it with this debacle?

No wonder the company died.

Posted by on November 18, 2009 at 11:11 PM PST #

I'm running into trouble using the projmod commands to set max-shm-memory for a group above 16GB. My system memory is 64GB. Is there a hard-cap on project memory of which I should be aware?