Troubleshooting 10g and 11.1 Clusterware Reboots

处理RAC问题必看的一篇文档

Troubleshooting 10g and 11.1 Clusterware Reboots [ID 265769.1]
		Modified 20-OCT-2010     Type TROUBLESHOOTING     Status PUBLISHED

In this Document
  
  
  
  
     
     
     
     
     
     
     
     
     
     
     
  

Applies to: Oracle Server - Enterprise Edition
Information in this document applies to any platform.
Purpose
This document is intended for DBA's and support analysts experiencing 10g or 11.1 Clusterware Reboots.  For 11.2 and above, see Last Review Date December 3, 2008 Instructions for the Reader

A Troubleshooting Guide is provided to assist in debugging a specific issue. When possible, diagnostic tools are included in the document to assist in troubleshooting.

Troubleshooting Details 10g or 11.1 RAC: TROUBLESHOOTING CRS REBOOTS
-- For 11.2 and above, see
-------------------------------------

If there is a ocssd.bin problem/failure, the oprocd daemon detected a scheduling
problem, or some other fatal problem, a node will reboot in a RAC cluster. This
functionality is used for I/O fencing to ensure that writes from I/O capable clients
can be cleared avoiding potential corruption scenarios in the event of a network
split, node hang, or some other fatal event.

1.0 - PROCESS ROLES FOR REBOOTS
OCSSD (aka CSS daemon) - This process is spawned in init.cssd. It runs in both
vendor clusterware and non-vendor clusterware environments and is armed with a
node kill via the init script. OCSSD's primary job is internode health monitoring
and RDBMS instance endpoint discovery. It runs as the Oracle user.

PS Output:
oracle 686 0.0 0.23207216608 ? S 11:42:42 0:12 /oracle/10g/crs/bin/ocssd.bin

INIT.CSSD - In a normal environment, init spawns init.cssd, which in turn spawns
OCSSD as a child. If ocssd dies or is killed, the node kill functionality of the
init script will kill the node. If the script is killed, its ocssd survives and
continues operating. However init has been instructed to respawn init.cssd via
inittab. When it does so, the second init.cssd will attempt to start its own ocssd.
That ocssd starts up, finds that its endpoint is owned by the first ocssd, fails,
and then the 2nd init.cssd kills the node.

PS Output:
root 635 0.0 0.0 1120 840 ? S 11:41:41 0:00 /bin/sh /etc/init.d/init.cssd fatal

OPROCD - This process is spawned in any non-vendor clusterware environment, except
on Windows where Oracle uses a kernel driver to perform the same actions and Linux
prior to version 10.2.0.4. If oprocd detects problems, it will kill a node via C
code. It is spawned in init.cssd and runs as root. This daemon is used to detect
hardware and driver freezes on the machine. If a machine were frozen for long enough
that the other nodes evicted it from the cluster, it needs to kill itself to prevent
any IO from getting reissued to the disk after the rest of the cluster has remastered
locks."

PS Output:
root 684 0.0 0.0 2240 968 ? S 11:42:42 0:00 /oracle/10g/crs/bin/oprocd start -t 1000 -m 50

OCLSOMON (10.2.0.2 and above) - This process monitors the CSS daemon for hangs or
scheduling issues and can reboot a node if there is a perceived hang.


2.0 - DETERMINING WHICH PROCESS IS RESPONSIBLE FOR A REBOOT

* Messages file locations:
Sun: /var/adm/messages
HP-UX: /var/adm/syslog/syslog.log
Tru64: /var/adm/messages
Linux: /var/log/messages
IBM: /bin/errpt -a > messages.out

** CSS log locations:
11.1 and 10.2: /log//cssd
10.1: /css/log

*** Oprocd log locations:
In /etc/oracle/oprocd or /var/opt/oracle/oprocd depending on version/platform.

Note that oprocd only runs when no vendor clusterware is running or on Linux > 10.2.0.4

3.0 - TROUBLESHOOTING OCSSD REBOOTS
If you have encountered an OCSSD reboot, review common causes in section 3.1 below.
If the problem cannot be determined by reviewing the common causes, review and
collect the data from section 3.3.

3.1 - COMMON CAUSES OF OCSSD REBOOTS
- Network failure or latency between nodes. It would take at least 30 consecutive
missed checkins to cause a reboot, where heartbeats are issued once per second.

Example of missed checkins in the CSS log:

WARNING: clssnmPollingThread: node (1) at 50% heartbeat fatal, eviction in 29.100 seconds
WARNING: clssnmPollingThread: node (1) at 75% heartbeat fatal, eviction in 14.960 seconds
WARNING: clssnmPollingThread: node (1) at 75% heartbeat fatal, eviction in 13.950 seconds

The first thing to do is find out if the missed checkins ARE the problem or are a
result of the node going down due to other reasons. Check the messages file to see
what exact time the node went down and compare it to the time of the missed checkins.

- If the messages file reboot time < missed checkin time then the node eviction was
likely not due to these missed checkins.

- If the messages file reboot time > missed checkin time then the node eviction was
likely a result of the missed checkins.


- Problems writing to or reading from the CSS voting disk.

Example of a voting disk problem in the CSS log:

ERROR: clssnmDiskPingMonitorThread: voting device access hanging (160008 miliseconds)

- Lack of CPU resources. There are some situations which will appear to be missed
heartbeat issues, however turn out to be caused by a user running a high
sustained load average. When a machine gets too heavily loaded, the scheduling
reliability can be bad. This could cause CSS to not get scheduled in time and
thus CSS cannot get its work done. If this happens, the node is declared
not-viable for cluster work and is evicted.

- A problem with the executables (for example, removing CRS Home files)

- Misconfiguration of CRS. Possible misconfigurations:

- Wrong network selected as the private network for CRS (confirm with CSS log,
/etc/hosts, and ifconfig output). Make sure it is not the public or VIP
address. Look in the CSS log for strings like...
clsc_listen: (*) Listening on
(ADDRESS=(PROTOCOL=tcp)(HOST=dlsun2046)(PORT=61196))

- Putting the CSS vote file on a Netapp that's shared over some kind of public
network or otherwise excessively loaded/unreliable network. If this is the
case, you are likely to see the following message in the CSS logfile:

ERROR: clssnmDiskPingThread(): Large disk IO timeout * seconds.

If you ever see this error, then it's important to investigate why the disk
subsystem is unresponsive.

See section 3.2 for information on how to correct common misconfiguration
problems.

- Killing the "init.cssd fatal" process or "ocssd" process.

- An unexpected failure of the OCSSD process, this can be caused by any of the
above issues.

- An Oracle bug. Known bugs that can cause CSS reboots:


- CSS Fails to Flush Writes After Installing 10.1.0.2 CRS on Linux with OCFS
Workaround: Put OCR and CSS Voting files on raw devices
Fixed in OCFS 1.0.11 and above.

- A deadlock can occur between 2 threads of the CSS daemon process.
Fixed in 10.1.0.4 and above.




SOLARIS ONLY: See these bugids that fixed the problem (in Solaris 9; the fixes were
backported to Solaris 8 Update 6):
cond_timedwait(), sigtimedwait(), poll() and /proc time out too soon
Fix for BugID 4308370 causes timeout failures when system time is reset



3.2 - FILES TO REVIEW AND GATHER FOR OCSSD REBOOTS
If logging a service request, please provide ALL of the following files to Oracle
Support if possible:

- All the files in the following directories from all nodes.

For 10.2 and above, all files under:

/log

Recommended method for gathering these for each node would be to run the
diagcollection.pl script.

For 10.1:

/crs/log
/crs/init
/css/log
/css/init
/evm/log
/evm/init
/srvm/log

Recommended method for gathering these for each node:

cd
tar cf crs.tar crs/init crs/log css/init css/log evm/init evm/log srvm/log

- Messages or Syslog from all nodes from the time of the problem:

Sun: /var/adm/messages
HP-UX: /var/adm/syslog/syslog.log
Tru64: /var/adm/messages
Linux: /var/log/messages
IBM: /bin/errpt -a > messages.out

- If a core files was written it would be useful to obtain a stack trace of the
core file using "TECH Getting a Stack Trace from a CORE file".
Core files are usually writtin in one of the following directories:

/crs/init
/css/init
/evm/init

You should also check all threads of the core file and get a stack trace for each.
has information on gathering multiple threads.

- OCR dump file - To get this cd to /bin as the root user and issue
"ocrdump ". This will generate two files (ocrdump.log and the
a dump file with the name given for it).

- 'opatch lsinventory -detail' output for the CRS home

- Back up the scls_scr directory, inittab, and hosts file for analysis with:

Sun, HP-UX, HP Tru64:

cd /
tar cf /var/backup/oraclecrs.tar var/opt/oracle/scls_scr etc/hosts etc/inittab

Linux, IBM-AIX:

cd /
tar cf /var/backup/oraclecrs.tar etc/oracle/scls_scr etc/hosts etc/inittab


- Ifconfig output from each node (ifconfig -a on unix platforms).
@ These can be useful in determining whether the user has set up the private
@ interconnect names to resolve to the same IP on all nodes. They can be useful
@ in determining whether someone is using their public IP as their private
@ interconnect address. In some cases it may also be useful to get the output
@ of nslookup for each private interconnect name. If some layer is attempting to
@ connect across the private interconnect and there is a possibility of getting
@ different results from /etc/hosts and from the nameserver, it could cause quite
@ a bit of confusion.

@ If it looks like there are network issues, asking for a detailed map of
@ their network wiring is pretty appropriate, and can be handy.

- It would also be useful to get the following from each node leading up to the time
of the reboot:

- netstat -is (or equivelant)
- iostat -x (or equivelant)
- vmstat (or equivelant)
- ping -s (or equivelant) output of the private network

There is a tool called "OS Watcher" that helps gather this information. This tool
will dump netstat, vmstat, iostat, and other output at an inverval and save x number
of hours of archived data. For more information about this tool see .

4.0 - TROUBLESHOOTING OPROCD REBOOTS
If you have encountered an OPROCD reboot, review common causes in section 4.1 below.
If the problem cannot be determined by reviewing the common causes, review and
collect the data from section 4.2.

4.1 - COMMON CAUSES OF OPROCD REBOOTS
- A problem detected by the OPROCD process. This can be caused by 4 things:

1) An OS scheduler problem.
2) The OS is getting locked up in a driver or hardware.
3) Excessive amounts of load on the machine, thus preventing the scheduler from
behaving reasonably.
4) An Oracle bug.

OPROCD Bugs Known to Cause Reboots:


- OPROCD may reboot the node whenever the system date is moved
backwards.
Fixed in 10.2.0.3+

- Oprocd is prone to time regression due to current API used (AIX only)
Fixed in 10.1.0.3 + One off patch for Bug 4206159.


Diagnostic Fixes (VERY NECESSARY IN MOST CASES):

- Oprocd logfile is cleared after a reboot
Fixed in 10.2.0.4+

- Increase the warning levels if a reboot is approaching
Fixed in 10.2.0.3+

4.2 - FILES TO REVIEW AND GATHER FOR OPROCD REBOOTS
If logging a service request, please provide ALL of the following files to Oracle
Support if possible:

- Oprocd logs in /etc/oracle/oprocd or /var/opt/oracle/oprocd depending on version/platform.

- All the files in the following directories from all nodes.

For 10.2 and above, all files under:

/log

Recommended method for gathering these for each node would be to run the
diagcollection.pl script.

For 10.1:

/crs/log
/crs/init
/css/log
/css/init
/evm/log
/evm/init
/srvm/log

Recommended method for gathering these for each node:

cd
tar cf crs.tar crs/init crs/log css/init css/log evm/init evm/log srvm/log

- Messages or Syslog from all nodes from the time of the problem:

Sun: /var/adm/messages
HP-UX: /var/adm/syslog/syslog.log
Tru64: /var/adm/messages
Linux: /var/log/messages
IBM: /bin/errpt -a > messages.out

- 'opatch lsinventory -detail' output for the CRS home

- It would also be useful to get the following from each node leading up to the time
of the reboot:

- netstat -is (or equivelant)
- iostat -x (or equivelant)
- vmstat (or equivelant)

There is a tool called "OS Watcher" that helps gather this information. This tool
will dump netstat, vmstat, iostat, and other output at an inverval and save x number
of hours of archived data. For more information about this tool see .



5.0 - TROUBLESHOOTING OCLSOMON REBOOTS
If you have encountered an OCLSOMON reboot, review common causes in section 5.1 below.
If the problem cannot be determined by reviewing the common causes, review and
collect the data from section 5.2.

5.1 - COMMON CAUSES OF OCLSOMON REBOOTS
- A problem detected by the OCLSOMON process. This can be caused by 4 things:

1) A thread(s) within the CSS daemon hung.
2) An OS scheduler problem.
3) Excessive amounts of load on the machine, thus preventing the scheduler from
behaving reasonably.
4) An Oracle bug.


5.2 - FILES TO REVIEW AND GATHER FOR OCLSOMON REBOOTS
If logging a service request, please provide ALL of the following files to Oracle
Support if possible:

- All the files in the following directories from all nodes. For a description of
these directories, see :

For 10.2, all files under:

/log

Recommended method for gathering these for each node would be to run the
diagcollection.pl script.

- Messages or Syslog from all nodes from the time of the problem:

Sun: /var/adm/messages
HP-UX: /var/adm/syslog/syslog.log
Tru64: /var/adm/messages
Linux: /var/log/messages
IBM: /bin/errpt -a > messages.out

- 'opatch lsinventory -detail' output for the CRS home

- It would also be useful to get the following from each node leading up to the time
of the reboot:

- netstat -is (or equivelant)
- iostat -x (or equivelant)
- vmstat (or equivelant)

There is a tool called "OS Watcher" that helps gather this information. This tool
will dump netstat, vmstat, iostat, and other output at an inverval and save x number
of hours of archived data. For more information about this tool see .


References - CRS and 10g Real Application Clusters
- Troubleshooting 11.2 Clusterware Node Evictions (Reboots)