RAC环境在ASM上创建表空间出错ORA-569

在一个的测试数据库上，创建表空间时出现了这个错误。

 

 

由于数据库环境比较复杂，简单描述一下。

这个测试环境安装的是Oracle 1106 for Solaris 10 sparc 64bit的RAC环境，搭建了ASM实例用于存放共享数据文件。

在RAC环境的其中一个节点上，又建立了一个单实例的数据库，并把这个数据库的数据文件也放到了ASM实例上。

结果尝试在这个实例上添加新的表空间时报错：

SQL> select file_name from dba_data_files;

FILE_NAME
------------------------------------------------------------------------------
+DATA/test/datafile/system.533.668281219
+DATA/test/datafile/sysaux.534.668281227
+DATA/test/datafile/undotbs1.535.668281229
+DATA/test/datafile/users.537.668281241

SQL> create tablespace test datafile '+DATA/test/datafile/test01.dbf' size 4096m;
create tablespace test datafile '+DATA/test/datafile/test01.dbf' size 4096m
*
第 1 行出现错误:
ORA-01119: 创建数据库文件 '+DATA/test/datafile/test01.dbf' 时出错
ORA-17502: ksfdcre: 4 未能创建文件 +DATA/test/datafile/test01.dbf
ORA-00569: Failed to acquire global enqueue.
ORA-00569: Failed to acquire global enqueue.

这个错误似乎很少见，查看了一下Oracle的官方错误文档描述：

ORA-00569: Failed to acquire global enqueue.
Cause: A prior error occurred on one of the instances in the cluster. Typically errors are caused by shared pool resource contention.
Action: Check for and resolve prior errors on all instances in the cluster. If there is shared pool resource contention, increase the SHARED_POOL_SIZE, DML_ LOCKS, PROCESSES, TRANSACTIONS, CLUSTER_DATABASE_INSTANCES and PARALLEL_MAX_SERVERS initialization parameters.

虽然对问题进行了描述，不过从错误看不出导致问题的真正原因。

查询了一下METALINK，找到了一些错误说明，不过没有和当前错误相似度很高的，大部分出现这个错误的同时，都会伴随ORA-600错误和ORA-4031错误。

不过现在有一个简单的方法来确定到底是数据库产生的问题还是ASM实例导致的问题，现在只需要登陆RAC实例，执行类似的添加表空间的操作，检查是否会出现相同的问题就可以了：

bash-3.00$ export ORACLE_SID=ractest1
bash-3.00$ sqlplus "/ as sysdba"

SQL*Plus: Release 11.1.0.6.0 - Production on 星期三 2月 18 17:12:42 2009

Copyright (c) 1982, 2007, Oracle.  All rights reserved.

已连接到空闲例程。

SQL> startup
ORACLE 例程已经启动。

Total System Global Area 1603887104 bytes
Fixed Size                  2095208 bytes
Variable Size             741722008 bytes
Database Buffers          855638016 bytes
Redo Buffers                4431872 bytes
数据库装载完毕。
数据库已经打开。
SQL> CREATE TABLESPACE TEST DATAFILE '+DATA/ractest/datafile/test01.dbf' SIZE 4096M;
CREATE TABLESPACE TEST DATAFILE '+DATA/ractest/datafile/test01.dbf' SIZE 4096M
*
第 1 行出现错误:
ORA-01119: 创建数据库文件 '+DATA/ractest/datafile/test01.dbf' 时出错
ORA-17502: ksfdcre: 4 未能创建文件 +DATA/ractest/datafile/test01.dbf
ORA-00569: Failed to acquire global enqueue.
ORA-00569: Failed to acquire global enqueue.

可以看到，相同的错误产生了，看来问题可能和ASM实例的状态有关系，登陆ASM实例，进行简单的检查：

bash-3.00$ export ORACLE_SID=+ASM1
bash-3.00$ sqlplus "/ as sysdba"

SQL*Plus: Release 11.1.0.6.0 - Production on 星期三 2月 18 17:33:12 2009

Copyright (c) 1982, 2007, Oracle.  All rights reserved.

连接到:
Oracle Database 11g Enterprise Edition Release 11.1.0.6.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options

SQL> set pages 100 lines 120
SQL> select instance_name, status from v$instance;

INSTANCE_NAME                    STATUS
-------------------------------- ------------------------
+ASM1                            STARTED

由于ASM实例可以用来检查的动态视图太少，从现有的视图也看不到特别的地方，看来只能重启数据库和ASM实例，再次检查问题：

bash-3.00$ export ORACLE_SID=test
bash-3.00$ sqlplus "/ as sysdba"

SQL*Plus: Release 11.1.0.6.0 - Production on 星期三 2月 18 17:41:40 2009

Copyright (c) 1982, 2007, Oracle.  All rights reserved.

连接到:
Oracle Database 11g Enterprise Edition Release 11.1.0.6.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options

SQL> shutdown immediate
数据库已经关闭。
已经卸载数据库。
ORACLE 例程已经关闭。
SQL> exit
bash-3.00$ export ORACLE_SID=ractest1
bash-3.00$ sqlplus "/ as sysdba"

SQL*Plus: Release 11.1.0.6.0 - Production on 星期三 2月 18 17:43:21 2009

Copyright (c) 1982, 2007, Oracle.  All rights reserved.

连接到:
Oracle Database 11g Enterprise Edition Release 11.1.0.6.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options

SQL> shutdown immediate
数据库已经关闭。
已经卸载数据库。
ORACLE 例程已经关闭。
SQL> exit
从 Oracle Database 11g Enterprise Edition Release 11.1.0.6.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options 断开
bash-3.00$ export ORACLE_SID=+ASM1  
bash-3.00$ sqlplus "/ as sysdba"

SQL*Plus: Release 11.1.0.6.0 - Production on 星期三 2月 18 17:44:05 2009

Copyright (c) 1982, 2007, Oracle.  All rights reserved.

连接到:
Oracle Database 11g Enterprise Edition Release 11.1.0.6.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options

SQL> shutdown immediate
^CORA-01013: user requested cancel of current operation

SQL> CONN / AS SYSDBA
已连接。
SQL> shutdown abort
ASM 实例已关闭
SQL> startup
ASM 实例已启动

Total System Global Area  284008448 bytes
Fixed Size                  2087944 bytes
Variable Size             256754680 bytes
ASM Cache                  25165824 bytes
ASM diskgroups mounted
SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.1.0.6.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options
bash-3.00$ export ORACLE_SID=test
bash-3.00$ sqlplus "/ as sysdba"

SQL*Plus: Release 11.1.0.6.0 - Production on 星期三 2月 18 17:47:22 2009

Copyright (c) 1982, 2007, Oracle.  All rights reserved.

已连接到空闲例程。

SQL> startup
ORACLE 例程已经启动。

Total System Global Area  806133760 bytes
Fixed Size                  2099064 bytes
Variable Size             455669896 bytes
Database Buffers          343932928 bytes
Redo Buffers                4431872 bytes
数据库装载完毕。
数据库已经打开。
SQL> create tablespace test datafile '+DATA/test/datafile/test01.dbf' size 4096m;
create tablespace test datafile '+DATA/test/datafile/test01.dbf' size 4096m
*
第 1 行出现错误:
ORA-01119: 创建数据库文件 '+DATA/test/datafile/test01.dbf' 时出错
ORA-17502: ksfdcre: 4 未能创建文件 +DATA/test/datafile/test01.dbf
ORA-00569: Failed to acquire global enqueue.
ORA-00569: Failed to acquire global enqueue.

可以看到，重启ASM实例，问题仍然出现。不过ASM实例也是在两个节点上同时运行的，莫非是另一个节点的ASM实例出现了问题：

bash-3.00$ export ORACLE_SID=+ASM2
bash-3.00$ sqlplus "/ as sysdba"

SQL*Plus: Release 11.1.0.6.0 - Production on 星期四 2月 19 16:38:38 2009

Copyright (c) 1982, 2007, Oracle.  All rights reserved.

连接到:
Oracle Database 11g Enterprise Edition Release 11.1.0.6.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options

SQL> set pages 100 lines 120
SQL> select instance_name, status from v$instance;

INSTANCE_NAME                    STATUS
-------------------------------- ------------------------
+ASM2                            STARTED

SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.1.0.6.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options

检查ASM实例未发现异常，尝试重启ASM实例：

bash-3.00$ srvctl stop instance -d ractest -i ractest2
bash-3.00$ srvctl stop asm -n ser2
bash-3.00$ srvctl start asm -n ser2

再次登陆test数据库，执行CREATE TABLESPACE语句：

bash-3.00$ export ORACLE_SID=test
bash-3.00$ sqlplus "/as sysdba"

SQL*Plus: Release 11.1.0.6.0 - Production on 星期四 2月 19 16:41:09 2009

Copyright (c) 1982, 2007, Oracle.  All rights reserved.

连接到:
Oracle Database 11g Enterprise Edition Release 11.1.0.6.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options

SQL> set pages 100 lines 120
SQL> create tablespace test datafile '+DATA/test/datafile/test01.dbf' size 4096m;

表空间已创建。

看来问题果然和ASM实例状态不正常有关。

检查asm实例2的alert文件，发现在运行CREATE TABLESPACE语句对应的时间点，出现了ORA-4031错误：

Wed Feb 18 15:50:04 2009
Errors in file /data/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_lmd0_3099.trc  (incident=2412):
ORA-04031: unable to allocate 3512 bytes of shared memory ("shared pool","unknown object","sga heap(1,0)","ges enqueues")
Incident details in: /data/oracle/diag/asm/+asm/+ASM2/incident/incdir_2412/+ASM2_lmd0_3099_i2412.trc
Wed Feb 18 15:50:05 2009
Trace dumping is performing id=[cdmp_20090218155005]
WARNING: ran out of shared pool for GES enqueue object.
Errors in file /data/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_lmd0_3099.trc  (incident=2413):
ORA-04031: unable to allocate 3512 bytes of shared memory ("shared pool","unknown object","sga heap(1,0)","ges enqueues")
Incident details in: /data/oracle/diag/asm/+asm/+ASM2/incident/incdir_2413/+ASM2_lmd0_3099_i2413.trc
Trace dumping is performing id=[cdmp_20090218155013]
Wed Feb 18 17:17:23 2009
Errors in file /data/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_lmd0_3099.trc  (incident=2414):
ORA-04031: unable to allocate 3512 bytes of shared memory ("shared pool","unknown object","sga heap(1,0)","ges enqueues")
Incident details in: /data/oracle/diag/asm/+asm/+ASM2/incident/incdir_2414/+ASM2_lmd0_3099_i2414.trc
WARNING: ran out of shared pool for GES enqueue object.
Wed Feb 18 17:17:24 2009
Trace dumping is performing id=[cdmp_20090218171724]
Errors in file /data/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_lmd0_3099.trc  (incident=2415):
ORA-04031: unable to allocate 3512 bytes of shared memory ("shared pool","unknown object","sga heap(1,0)","ges enqueues")
Incident details in: /data/oracle/diag/asm/+asm/+ASM2/incident/incdir_2415/+ASM2_lmd0_3099_i2415.trc
Trace dumping is performing id=[cdmp_20090218171732]
Wed Feb 18 17:18:02 2009
Errors in file /data/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_lmd0_3099.trc  (incident=2416):
ORA-04031: unable to allocate 3512 bytes of shared memory ("shared pool","unknown object","sga heap(1,0)","ges enqueues")
Incident details in: /data/oracle/diag/asm/+asm/+ASM2/incident/incdir_2416/+ASM2_lmd0_3099_i2416.trc
WARNING: ran out of shared pool for GES enqueue object.
Wed Feb 18 17:18:03 2009
Trace dumping is performing id=[cdmp_20090218171803]
Errors in file /data/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_lmd0_3099.trc  (incident=2417):
ORA-04031: unable to allocate 3512 bytes of shared memory ("shared pool","unknown object","sga heap(1,0)","ges enqueues")
Incident details in: /data/oracle/diag/asm/+asm/+ASM2/incident/incdir_2417/+ASM2_lmd0_3099_i2417.trc
Trace dumping is performing id=[cdmp_20090218171811]

这个ORA-4031错误已经和Oracle的metalink文章相符了，和Oracle错误文档上对这个错误的描述也是一致的。

而且这个ORA-4031错误信息也很明显，在分配全局对了资源的时候出现的错误。

检查ASM实例的sga，发现：

SQL> show sga

Total System Global Area  284008448 bytes
Fixed Size                  2087944 bytes
Variable Size             256754680 bytes
ASM Cache                  25165824 bytes

对于跑了多个RAC环境的ASM实例而言，200M的SGA显然太小了，和大部分Oracle默认参数一样，默认的ASM实例参数也是偏小的。

以前也碰到过一次由于ASM实例PROCESS参数太小，导致ASM实例无法登陆的问题。因此，如果选择ASM作为产品库的存储方式，那么ASM实例要重新设置，默认的参数很可能无法满足需要。