oracle RAC 故障处理一例

环境AIX6.1.7，oracle11gR2.ocssd文件中一直报错误信息：

2012-11-22 09:57:14.701: [    CSSD][2577]clssnmvDHBValidateNCopy: node 1, itsmdb1, has a disk HB, but no network HB, DHB has rcfg 248973026, wrtcnt, 66143, LATS 3532331338, lastSeqNo 66142, uniqueness 1353548671, timestamp 1353549434/3630123152
2012-11-22 09:57:15.074: [    CSSD][1029]clssscSelect: cookie accept request 110976300
2012-11-22 09:57:15.074: [    CSSD][1029]clssgmAllocProc: (111518e90) allocated
2012-11-22 09:57:15.075: [    CSSD][1029]clssgmClientConnectMsg: properties of cmProc 111518e90 - 0,1,2,3,4
2012-11-22 09:57:15.075: [    CSSD][1029]clssgmClientConnectMsg: Connect from con(6dec) proc(111518e90) pid(4718720) version 11:2:1:4, properties: 0,1,2,3,4
2012-11-22 09:57:15.075: [    CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-11-22 09:57:15.076: [    CSSD][1029]clssscSelect: cookie accept request 111518e90
2012-11-22 09:57:15.076: [    CSSD][1029]clssscevtypSHRCON: getting client with cmproc 111518e90
2012-11-22 09:57:15.076: [    CSSD][1029]clssgmRegisterClient: proc(4/111518e90), client(1/11125d9f0)
2012-11-22 09:57:15.076: [    CSSD][1029]clssgmJoinGrock: global grock CRF- new client 11125d9f0 with con 6e18, requested num -1, flags 0x4000e00
2012-11-22 09:57:15.076: [    CSSD][1029]clssgmJoinGrock: ignoring grock join for client not requiring fencing until group information has been received from the master; group name CRF-, member number -1, flags 0x4000e00
2012-11-22 09:57:15.077: [    CSSD][1029]clssgmDiscEndpcl: gipcDestroy 6e18
2012-11-22 09:57:15.077: [    CSSD][1029]clssgmDeadProc: proc 111518e90
2012-11-22 09:57:15.077: [    CSSD][1029]clssgmDestroyProc: cleaning up proc(111518e90) con(6dec) skgpid  ospid 4718720 with 0 clients, refcount 0
2012-11-22 09:57:15.077: [    CSSD][1029]clssgmDiscEndpcl: gipcDestroy 6dec
2012-11-22 09:57:15.588: [    CSSD][3348]clssgmWaitOnEventValue: after CmInfo State  val 3, eval 1 waited 0
2012-11-22 09:57:15.705: [    CSSD][2577]clssnmvDHBValidateNCopy: node 1, itsmdb1, has a disk HB, but no network HB, DHB has rcfg 248973026, wrtcnt, 66144, LATS 3532332342, lastSeqNo 66143, uniqueness 1353548671, timestamp 1353549435/3630124162
2012-11-22 09:57:16.589: [    CSSD][3348]clssgmWaitOnEventValue: after CmInfo State  val 3, eval 1 waited 0
2012-11-22 09:57:16.706: [    CSSD][2577]clssnmvDHBValidateNCopy: node 1, itsmdb1, has a disk HB, but no network HB, DHB has rcfg 248973026, wrtcnt, 66145, LATS 3532333342, lastSeqNo 66144, uniqueness 1353548671, timestamp 1353549436/3630125169
2012-11-22 09:57:17.595: [    CSSD][3348]clssgmWaitOnEventValue: after CmInfo State  val 3, eval 1 waited 0
2012-11-22 09:57:17.710: [    CSSD][2577]clssnmvDHBValidateNCopy: node 1, itsmdb1, has a disk HB, but no network HB, DHB has rcfg 248973026, wrtcnt, 66146, LATS 3532334347, lastSeqNo 66145, uniqueness 1353548671, timestamp 1353549437/3630126173
2012-11-22 09:57:18.049: [    CSSD][3862]clssnmSendingThread: sending join msg to all nodes
2012-11-22 09:57:18.049: [    CSSD][3862]clssnmSendingThread: sent 4 join msgs to all nodes
2012-11-22 09:57:18.605: [    CSSD][3348]clssgmWaitOnEventValue: after CmInfo State  val 3, eval 1 waited 0
2012-11-22 09:57:18.716: [    CSSD][2577]clssnmvDHBValidateNCopy: node 1, itsmdb1, has a disk HB, but no network HB, DHB has rcfg 248973026, wrtcnt, 66147, LATS 3532335352, lastSeqNo 66146, uniqueness 1353548671, timestamp 1353549438/3630127177
2012-11-22 09:57:19.224: [    CSSD][1029]clssscSelect: cookie accept request 110976300
2012-11-22 09:57:19.224: [    CSSD][1029]clssgmAllocProc: (111518e90) allocated
2012-11-22 09:57:19.224: [    CSSD][1029]clssgmClientConnectMsg: properties of cmProc 111518e90 - 0,1,2,3,4
2012-11-22 09:57:19.224: [    CSSD][1029]clssgmClientConnectMsg: Connect from con(71a8) proc(111518e90) pid(4587546) version 11:2:1:4, properties: 0,1,2,3,4
2012-11-22 09:57:19.224: [    CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-11-22 09:57:19.226: [    CSSD][1029]clssscSelect: cookie accept request 111518e90
2012-11-22 09:57:19.226: [    CSSD][1029]clssscevtypSHRCON: getting client with cmproc 111518e90
2012-11-22 09:57:19.226: [    CSSD][1029]clssgmRegisterClient: proc(4/111518e90), client(1/11125d9f0)
2012-11-22 09:57:19.226: [    CSSD][1029]clssgmJoinGrock: global grock CRF- new client 11125d9f0 with con 71d4, requested num -1, flags 0x4000e00
2012-11-22 09:57:19.226: [    CSSD][1029]clssgmJoinGrock: ignoring grock join for client not requiring fencing until group information has been received from the master; group name CRF-, member number -1, flags 0x4000e00
2012-11-22 09:57:19.226: [    CSSD][1029]clssgmDiscEndpcl: gipcDestroy 71d4
2012-11-22 09:57:19.227: [    CSSD][1029]clssgmDeadProc: proc 111518e90
2012-11-22 09:57:19.227: [    CSSD][1029]clssgmDestroyProc: cleaning up proc(111518e90) con(71a8) skgpid  ospid 4587546 with 0 clients, refcount 0
2012-11-22 09:57:19.227: [    CSSD][1029]clssgmDiscEndpcl: gipcDestroy 71a8
2012-11-22 09:57:19.523: [GIPCHALO][1543] gipchaLowerProcessNode: no valid interfaces found to node for 3532336160 ms, node 11125bcb0 { host 'itsmdb1', haName 'CSS_itsmdb-scan', srcLuid 5e44e1cc-679ae7a0, dstLuid 00000000-00000000 numInf 0, contigSeq 0, lastAck 0, lastValidAck 0, sendSeq [110 : 110], createTime 3532225979, sentRegister 1, localMonitor 1, flags 0x4 }
2012-11-22 09:57:19.605: [    CSSD][3348]clssgmWaitOnEventValue: after CmInfo State  val 3, eval 1 waited 0

通过登录grid用户。操作：

itsmdb1-> perl mcasttest.pl -n itsmdb1,itsmdb2 -i en7
###########  Setup for node itsmdb1  ##########
Checking node access 'itsmdb1'
Checking node login 'itsmdb1'
Checking/Creating Directory /tmp/mcasttest for binary on node 'itsmdb1'
Distributing mcast2 binary to node 'itsmdb1'
###########  Setup for node itsmdb2  ##########
Checking node access 'itsmdb2'
Checking node login 'itsmdb2'
Checking/Creating Directory /tmp/mcasttest for binary on node 'itsmdb2'
Distributing mcast2 binary to node 'itsmdb2'
###########  testing Multicast on all nodes  ##########

Test for Multicast address 230.0.1.0

Nov 22 11:17:47 | Multicast Succeeded for en7 using address 230.0.1.0:42000

Test for Multicast address 224.0.0.251

Nov 22 11:17:49 | Multicast Succeeded for en7 using address 224.0.0.251:42001

测试显示通信正常。

但是，仍然报错。

后来，修改网卡绑定参数。

然后继续通过perl mcasttest.pl -n itsmdb1,itsmdb2 -i en7测试依然正常。

重新启动服务器。

重启后恢复正常。