Subject: crs_stat command failed with CRS-0184 and crsctl failed with PROC-4
Doc ID: 371378.1 Type: PROBLEM
Modified Date: 30-APR-2007 Status: MODERATED
In this Document
Symptoms
Cause
Solution
References
--------------------------------------------------------------------------------
This document is being delivered to you via Oracle Support's Rapid Visibility (RaV) process, and therefore has not been subject to an independent technical review.
Applies to:
Oracle Server - Enterprise Edition - Version: 10.2.0.2
This problem can occur on any platform.
Symptoms
Before manually starting the database using 'srvctl instance -d ocs -i delphi-1", the "crs_stat -t" command was working fine on all nodes, problem started after manually starting the database using "srvctl" command.
$ crs_stat -t
CRS-0184: Cannot communicate with the CRS daemon.
$ crsctl check boot
Failure 4 at procr_open_key with status 4.
PROC-4: The cluster registry key to be operated on does not exist.
$ crsctl check crs
Failure 1 contacting CSS daemon
Cannot communicate with CRS
Cannot communicate with EVM
crsd.log
/opt/crs/oracle/product/10.2.0/crs/crs/init/delphi-1.pid
[ clsdmt][52]Listening to (ADDRESS=(PROTOCOL=ipc)(KEY=delphi-1DBG_CRSD))
2006-05-16 18:08:57.021: [ CRSOCR][1]32OCR api procr_open_key failed for key CRS.CUR. OCR
error
code = 4 OCR error msg: PROC-4: The cluster registry key to be operated on does not exist.
2006-05-16 18:08:57.192: [ CRSMAIN][1]32Using Authorizer location:
/opt/crs/oracle/product/10.2.0/crs/crs/auth/
2006-05-16 18:08:57.220: [ OCRSRV][35]th_select_handler: Failed to retrieve procctx from
ht.
constr = [5988488] retval lht [-27] Signal CV.
Cause
OCR got corrupted.
Solution
To resolve this issue need to restore OCR from backup.
Oracle automatically takes backups of the OCR every 4 hours. Oracle also keeps the last 3
backups, up to 4 hours old, one day old, and one week old available. Here are the steps for restoring the OCR.
1. Find out what time the problem that the inconsistency in the OCR occurred.
2. Find an OCR backup from a time prior to when the inconsistency occurred.
To do this cd to $ORA_CRS_HOME/cdata/
[or]
Run $ORA_CRS_HOME/bin/ocrconfig -showbacku p (In this case you may not get any result)
Example:
# pwd
/t02/app/oracle/product/crs/cdata/crs_opcbsol
# ls -ltr
total 46560
-rw-r----- 1 root root 3960832 Apr 12 19:53 week.ocr
-rw-r----- 1 root root 3960832 Apr 13 03:53 day.ocr
-rw-r----- 1 root root 3960832 Apr 14 03:54 backup02.ocr
-rw-r----- 1 root root 3960832 Apr 14 03:54 day_.ocr
-rw-r----- 1 root root 3960832 Apr 14 07:54 backup01.ocr
-rw-r----- 1 root root 3960832 Apr 14 11:54 backup00.ocr
3. If you have a backup of the OCR from prior to the time of the inconsistency,
reboot the nodes in single user mode or runlevel 1. If you are unable to
reboot into single user mode for some reason, you can disable CRS with:
Sun or Linux:
/etc/init.d/init.crs disable
/etc/init.d/init.crs stop
HP-UX or HP Tru64::
/sbin/init.d/init.crs disable
/sbin/init.d/init.crs stop
IBM AIX:
/etc/init.crs disable
/etc/init.crs stop
4. After all nodes are rebooted in single user mode and/or you have verified that
CRS is not running (ps -ef | grep crs), restore the OCR with ocrconfig.
Example:
cd $ORA_CRS_HOME/bin
./ocrconfig -restore /t02/app/oracle/product/crs/cdata/crs_opcbsol/week.ocr
5. Re-enable CRS if it was disabled. Example:
Sun or Linux:
/etc/init.d/init.crs enable
HP-UX or HP Tru64::
/sbin/init.d/init.crs enable
IBM AIX:
/etc/init.crs enable
6. Reboot the nodes.
