处理listener和ASM状态为UNKNOWN，并且无法使用srvctl start 启动的问题

========phanx.com========
Author:   phanx
Updated: 2013-7-7
转载请保留作者信息
=========================

使用srvctl start启动listener和asm均报错
# srvctl start listener -n coredb1
CRS-1028: Dependency analysis failed because of: 'Resource in UNKNOWN state: ora.coredb1.LISTENER_COREDB1.lsnr '
CRS-0223: Resource ' ora.coredb1.LISTENER_COREDB1.lsnr ' has placement error

查看状态，发现状态为UNKNOWN

# crs_stat -t
Name           Type           Target    State     Host       
------------------------------------------------------------
ora....SM1.asm application    ONLINE    UNKNOWN   coredb1    
ora....B1.lsnr application    ONLINE    UNKNOWN   coredb1    
ora....db1.gsd application    ONLINE    ONLINE    coredb1    
ora....db1.ons application    ONLINE    ONLINE    coredb1    
ora....db1.vip application    ONLINE    ONLINE    coredb1    
ora....SM2.asm application    ONLINE    ONLINE    coredb2    
ora....B2.lsnr application    ONLINE    ONLINE    coredb2    
ora....db2.gsd application    ONLINE    ONLINE    coredb2    
ora....db2.ons application    ONLINE    ONLINE    coredb2    
ora....db2.vip application    ONLINE    ONLINE    coredb2    

用export SRVM_TRACE=TRUE 开启相关Debug信息

export SRVM_TRACE=TRUE

# srvctl start listener -n coredb1            
/u01/app/oracle/crs/jdk/jre/bin/java -classpath /u01/app/oracle/crs/jlib/netcfg.jar:/u01/app/oracle/crs/jdk/jre/lib/rt.jar:/u01/app/oracle/crs/jdk/jre/lib/i18n.jar:/u01/app/oracle/crs/jlib/srvm.jar:/u01/app/oracle/crs/jlib/srvmhas.jar:/u01/app/oracle/crs/jlib/srvmasm.jar:/u01/app/oracle/crs/srvm/jlib/srvctl.jar -DTRACING.ENABLED=true -DTRACING.LEVEL=2 oracle.ops.opsctl.OPSCTLDriver start listener -n coredb1
[main] [14:12:28:761] [OPSCTLDriver.setInternalDebugLevel:165]  tracing is true at level 2 to file null
[main] [14:12:28:873] [OPSCTLDriver.:95]  Security manager is set
[main] [14:12:28:880] [CommandLineParser.parse:178]  parsing cmdline args
[main] [14:12:28:880] [CommandLineParser.parse2WordCommandOptions:948]  parsing 2-word cmdline
[main] [14:12:28:934] [HASContext.getInstance:200]  Module init : 16
[main] [14:12:28:934] [HASContext.getInstance:223]  Local Module init : 19
[main] [14:12:28:934] [HASContext.:93]  moduleInit = 19
[main] [14:12:28:944] [Library.getInstance:97]  Created instance of Library.
[main] [14:12:28:944] [Library.load:197]  Loading libsrvmhas10.a...
[main] [14:12:28:944] [Library.load:203]  oracleHome null
[main] [14:12:28:944] [Library.load:239]  Loading  library srvmhas10 from path=
/u01/app/oracle/crs/jdk/jre/bin:/u01/app/oracle/crs/jdk/jre/bin/classic:/u01/app/oracle/crs/jdk/jre/bin:/u01/app/oracle/crs/lib32:/u01/app/oracle/crs/srvm/lib32:/usr/lib
[main] [14:12:28:949] [Library.load:247]  Loaded library srvmhas10 from path=
/u01/app/oracle/crs/jdk/jre/bin:/u01/app/oracle/crs/jdk/jre/bin/classic:/u01/app/oracle/crs/jdk/jre/bin:/u01/app/oracle/crs/lib32:/u01/app/oracle/crs/srvm/lib32:/usr/lib
[main] [14:12:28:957] [has.HASContextNative.Native]  prsr_trace: no lsf ctx, line=Native: allocHASContext

[main] [14:12:28:957] [has.HASContextNative.Native] allocHASContext: Came in
[main] [14:12:28:958] [has.HASContextNative.Native] allocHASContext: module_init = 19
[main] [14:12:28:958] [has.HASContextNative.Native] allocHASContext: META context [1]
[main] [14:12:28:959] [has.HASContextNative.Native] allocHASContext: LSF context [1]
[main] [14:12:28:959] [has.HASContextNative.Native] prsr_trace: Native: prsr_initCLSS

[main] [14:12:29:8] [has.HASContextNative.Native] allocHASContext: CLSS context [1]
[main] [14:12:29:8] [has.HASContextNative.Native] allocHASContext: retval [1]
[main] [14:12:29:8] [HASContext.getInstance:258]  HAS Context Allocated: 1 to
[main] [14:12:29:8] [ClusterUtil.:55]  ClusterUtil Instance created
[main] [14:12:29:8] [ClusterUtil.:58]  ClusterUtil m_bSKGXN=true
[main] [14:12:29:9] [has.ClusterUtilNative.Native]  prsr_trace: Native: getHostName

[main] [14:12:29:9] [has.ClusterUtilNative.Native]  prsr_trace: Native: getHostName:hostname length: 256

[main] [14:12:29:64] [has.ClusterUtilNative.Native]  prsr_trace: Native: getHostName:hostname: coredb1

[main] [14:12:29:64] [GetActiveNodes.getHostName:476]  hostname = coredb1
[main] [14:12:29:64] [Cluster.getHostName:1022]  hostname = coredb1
[main] [14:12:29:64] [CommandLineParser.isValidNodeName:1578]  nodename=coredb1 hostname: coredb1
[main] [14:12:29:65] [OPSCTLDriver.execute:174]  executing srvctl command
[main] [14:12:29:65] [OPSCTLDriver.execute:199]  executing 2-word command verb=3 noun=107
[main] [14:12:29:79] [NodeApps.:117]  in constructor with coredb1
[main] [14:12:29:79] [Listener.:193]  Creating listener with node name as coredb1 and resource name as ora.coredb1.LISTENER_COREDB1.lsnr and listener name as LISTENER_COREDB1
[main] [14:12:29:79] [Listener.isRunning:236]  Trying to find the status of ora.coredb1.LISTENER_COREDB1.lsnr on node coredb1
[main] [14:12:29:80] [Operation.:63]  Setting to be a synchronized operation
[main] [14:12:29:81] [HASContext.getInstance:200]  Module init : 6
[main] [14:12:29:81] [HASContext.getInstance:223]  Local Module init : 4
[main] [14:12:29:81] [has.HASContextNative.Native]  prsr_trace: Native: allocHASContext

[main] [14:12:29:81] [has.HASContextNative.Native]  allocHASContext: Came in
[main] [14:12:29:81] [has.HASContextNative.Native]  allocHASContext: module_init = 4
[main] [14:12:29:82] [has.HASContextNative.Native]  prsr_trace: Native: prsr_initSCLS

[main] [14:12:29:82] [has.HASContextNative.Native]  allocHASContext: SCLS context [1]
[main] [14:12:29:82] [has.HASContextNative.Native]  allocHASContext: retval [1]
[main] [14:12:29:83] [HASContext.getInstance:258]  HAS Context Allocated: 2 to
[main] [14:12:29:83] [Util.:90]  Util Instance created.
[main] [14:12:29:84] [has.UtilNative.Native]  prsr_trace: Native: getCRSHome

[main] [14:12:29:84] [has.UtilNative.Native]  prsr_trace: Native: getCRSHome crs_home=/u01/app/oracle/crs(**)

[main] [14:12:29:84] [HASContext.getCRSHome:479]  /u01/app/oracle/crs
[main] [14:12:29:85] [Util.getCRSHome:508]  getCRSHome: ret=/u01/app/oracle/crs
[main] [14:12:29:85] [HAOperation.setCRSHome:149]  CRS_HOME: /u01/app/oracle/crs
[main] [14:12:29:85] [HAOperation.checkCommandVerb:212]  Checking an existence of CRS command
[main] [14:12:29:85] [NodeApps.nodeRunning:371]  Finding the status of ora.coredb1.LISTENER_COREDB1.lsnr on coredb1
[main] [14:12:29:86] [LocalCommand.execute:54]  LocalCommand.execute: Calling run method
[main] [14:12:29:88] [HAOperationImpl.runCommand:1238]  CRS cmd is: /u01/app/oracle/crs/bin/crs_stat -u ora.coredb1.LISTENER_COREDB1.lsnr
[main] [14:12:29:89] [RuntimeExec.runCommand:74]  Calling Runtime.exec() with the command
[main] [14:12:29:89] [RuntimeExec.runCommand:76]  /u01/app/oracle/crs/bin/crs_stat
[main] [14:12:29:89] [RuntimeExec.runCommand:76]  -u
[main] [14:12:29:89] [RuntimeExec.runCommand:76]  ora.coredb1.LISTENER_COREDB1.lsnr
[main] [14:12:29:104] [RuntimeExec.runCommand:131]  runCommand: Waiting for the process
[Thread-0] [14:12:29:105] [StreamReader.run:61]  In StreamReader.run
[Thread-1] [14:12:29:105] [StreamReader.run:61]  In StreamReader.run
[Thread-0] [14:12:29:127] [StreamReader.run:65]  OUTPUT>NAME=ora.coredb1.LISTENER_COREDB1.lsnr
[Thread-0] [14:12:29:127] [StreamReader.run:65]  OUTPUT>TYPE=application
[Thread-0] [14:12:29:127] [StreamReader.run:65]  OUTPUT>TARGET=OFFLINE
[Thread-0] [14:12:29:127] [StreamReader.run:65]  OUTPUT>STATE=OFFLINE
[main] [14:12:29:129] [RuntimeExec.runCommand:133]  runCommand: process returns 0
[Thread-0] [14:12:29:129] [StreamReader.run:65]  OUTPUT>
[main] [14:12:29:129] [RuntimeExec.runCommand:147]  RunTimeExec: output>
[main] [14:12:29:129] [RuntimeExec.runCommand:150]  NAME=ora.coredb1.LISTENER_COREDB1.lsnr
[main] [14:12:29:130] [RuntimeExec.runCommand:150]  TYPE=application
[main] [14:12:29:130] [RuntimeExec.runCommand:150]  TARGET=OFFLINE
[main] [14:12:29:131] [RuntimeExec.runCommand:150]  STATE=OFFLINE
[main] [14:12:29:143] [RuntimeExec.runCommand:150] 
[main] [14:12:29:143] [RuntimeExec.runCommand:155]  RunTimeExec: error>
[main] [14:12:29:143] [RuntimeExec.runCommand:175]  Returning from RunTimeExec.runCommand
[main] [14:12:29:143] [HAOperationImpl.runCommand:1247]  signed exit value = 0
[main] [14:12:29:143] [HAOperationImpl.runCommand:1269]  set status OK
[main] [14:12:29:143] [HAStatusOperation.run:132]  Returned from executing the HA Operation
[main] [14:12:29:144] [HAStatusOperation.run:137]  OUTPUT> NAME=ora.coredb1.LISTENER_COREDB1.lsnr
[main] [14:12:29:144] [HAStatusOperation.run:137]  OUTPUT> TYPE=application
[main] [14:12:29:144] [HAStatusOperation.run:137]  OUTPUT> TARGET=OFFLINE
[main] [14:12:29:144] [HAStatusOperation.run:137]  OUTPUT> STATE=OFFLINE
[main] [14:12:29:144] [HAStatusOperation.run:137]  OUTPUT>
[main] [14:12:29:144] [HAStatusOperation.run:143]  ERROR>
[main] [14:12:29:144] [LocalCommand.execute:56]  LocalCommand.execute: Returned from run method
[main] [14:12:29:145] [HAResourceStatus.parseHAStatusOutput:293]  got status: resource=ora.coredb1.LISTENER_COREDB1.lsnrtarget=OFFLINE status=OFFLINE host=null
[main] [14:12:29:145] [Listener.isRunning:250]  online status of ora.coredb1.LISTENER_COREDB1.lsnr on coredb1 is false
[main] [14:12:29:146] [Listener.start:266]  Running new start ..
[main] [14:12:29:146] [Operation.:63]  Setting to be a synchronized operation
[main] [14:12:29:146] [HASContext.getInstance:200]  Module init : 6
[main] [14:12:29:146] [HASContext.getInstance:223]  Local Module init : 0
[main] [14:12:29:146] [HASContext.getInstance:258]  HAS Context Allocated: 3 to
[main] [14:12:29:146] [Util.:90]  Util Instance created.
[main] [14:12:29:146] [has.UtilNative.Native]  prsr_trace: Native: getCRSHome

[main] [14:12:29:147] [has.UtilNative.Native]  prsr_trace: Native: getCRSHome crs_home=/u01/app/oracle/crs(**)

[main] [14:12:29:147] [HASContext.getCRSHome:479]  /u01/app/oracle/crs
[main] [14:12:29:147] [Util.getCRSHome:508]  getCRSHome: ret=/u01/app/oracle/crs
[main] [14:12:29:148] [HAOperation.setCRSHome:149]  CRS_HOME: /u01/app/oracle/crs
[main] [14:12:29:148] [HAOperation.checkCommandVerb:212]  Checking an existence of CRS command
[main] [14:12:29:148] [NodeApps.start:422]  Starting service for ora.coredb1.LISTENER_COREDB1.lsnr on coredb1
[main] [14:12:29:148] [LocalCommand.execute:54]  LocalCommand.execute: Calling run method
[main] [14:12:29:149] [HAStartOperation.run:81]  Executing the HA operation /u01/app/oracle/crs/bin/crs_startora.coredb1.LISTENER_COREDB1.lsnr
[main] [14:12:29:149] [HAOperationImpl.runCommand:1238]  CRS cmd is: /u01/app/oracle/crs/bin/crs_start ora.coredb1.LISTENER_COREDB1.lsnr -f
[main] [14:12:29:149] [RuntimeExec.runCommand:74]  Calling Runtime.exec() with the command
[main] [14:12:29:149] [RuntimeExec.runCommand:76]  /u01/app/oracle/crs/bin/crs_start
[main] [14:12:29:149] [RuntimeExec.runCommand:76]  ora.coredb1.LISTENER_COREDB1.lsnr
[main] [14:12:29:150] [RuntimeExec.runCommand:76]  -f
[main] [14:12:29:157] [RuntimeExec.runCommand:131]  runCommand: Waiting for the process
[Thread-2] [14:12:29:162] [StreamReader.run:61]  In StreamReader.run
[Thread-3] [14:12:29:162] [StreamReader.run:61]  In StreamReader.run
[Thread-2] [14:12:29:284] [StreamReader.run:65]  OUTPUT>Attempting to start `ora.coredb1.LISTENER_COREDB1.lsnr` on member `coredb1`
[Thread-2] [14:12:30:369] [StreamReader.run:65]  OUTPUT>`ora.coredb1.LISTENER_COREDB1.lsnr` on member `coredb1` has experienced an unrecoverable failure.
[Thread-2] [14:12:30:370] [StreamReader.run:65]  OUTPUT>Human intervention required to resume its availability.
[Thread-2] [14:12:30:388] [StreamReader.run:65]  OUTPUT>coredb1:ora.coredb1.LISTENER_COREDB1.lsnr:/u01/app/oracle/product/10.2.0/asm/bin/racgwrap[64]: %ORACLE_HOME%/bin/racgmain:  not found.
[Thread-3] [14:12:30:389] [StreamReader.run:65]  ERROR>CRS-0215: Could not start resource 'ora.coredb1.LISTENER_COREDB1.lsnr'.
[Thread-3] [14:12:30:389] [StreamReader.run:65]  ERROR>
[main] [14:12:30:394] [RuntimeExec.runCommand:133]  runCommand: process returns 115
[main] [14:12:30:394] [RuntimeExec.runCommand:147]  RunTimeExec: output>
[main] [14:12:30:394] [RuntimeExec.runCommand:150]  Attempting to start `ora.coredb1.LISTENER_COREDB1.lsnr` on member `coredb1`
[main] [14:12:30:395] [RuntimeExec.runCommand:150]  `ora.coredb1.LISTENER_COREDB1.lsnr` on member `coredb1` has experienced an unrecoverable failure.
[main] [14:12:30:395] [RuntimeExec.runCommand:150]  Human intervention required to resume its availability.
[main] [14:12:30:395] [RuntimeExec.runCommand:150]  coredb1:ora.coredb1.LISTENER_COREDB1.lsnr:/u01/app/oracle/product/10.2.0/asm/bin/racgwrap[64]: %ORACLE_HOME%/bin/racgmain:  not found.
[main] [14:12:30:395] [RuntimeExec.runCommand:155]  RunTimeExec: error>
[main] [14:12:30:395] [RuntimeExec.runCommand:158]  CRS-0215: Could not start resource 'ora.coredb1.LISTENER_COREDB1.lsnr'.
[main] [14:12:30:395] [RuntimeExec.runCommand:158] 
[main] [14:12:30:395] [RuntimeExec.runCommand:175]  Returning from RunTimeExec.runCommand
[main] [14:12:30:395] [HAOperationImpl.runCommand:1247]  signed exit value = 115
[main] [14:12:30:396] [HAOperationImpl.runCommand:1287]  set status FAILED
[main] [14:12:30:396] [HAStartOperation.run:84]  Returned from executing the HA Operation
[main] [14:12:30:396] [HAStartOperation.run:89]  OUTPUT> Attempting to start `ora.coredb1.LISTENER_COREDB1.lsnr` on member `coredb1`
[main] [14:12:30:396] [HAStartOperation.run:89]  OUTPUT> `ora.coredb1.LISTENER_COREDB1.lsnr` on member `coredb1` has experienced an unrecoverable failure.
[main] [14:12:30:396] [HAStartOperation.run:89]  OUTPUT> Human intervention required to resume its availability.
[main] [14:12:30:396] [HAStartOperation.run:89]  OUTPUT> coredb1:ora.coredb1.LISTENER_COREDB1.lsnr:/u01/app/oracle/product/10.2.0/asm/bin/racgwrap[64]: %ORACLE_HOME%/bin/racgmain:  not found.
[main] [14:12:30:396] [HAStartOperation.run:95]  ERROR> CRS-0215: Could not start resource 'ora.coredb1.LISTENER_COREDB1.lsnr'.
[main] [14:12:30:397] [LocalCommand.execute:56]  LocalCommand.execute: Returned from run method
[main] [14:12:30:397] [HAOperationResult.getOutputAll:114]  outLine is [coredb1:ora.coredb1.LISTENER_COREDB1.lsnr:/u01/app/oracle/product/10.2.0/asm/bin/racgwrap[64]: %ORACLE_HOME%/bin/racgmain:  not found.]
[main] [14:12:30:398] [HAOperationResult.getOutputAll:115]  errLine is [CRS-0215: Could not start resource 'ora.coredb1.LISTENER_COREDB1.lsnr'.]
oracle.ops.mgmt.database.ListenerException: coredb1:ora.coredb1.LISTENER_COREDB1.lsnr:/u01/app/oracle/product/10.2.0/asm/bin/racgwrap[64]: %ORACLE_HOME%/bin/racgmain:  not found.
CRS-0215: Could not start resource 'ora.coredb1.LISTENER_COREDB1.lsnr'.
        at oracle.ops.mgmt.nodeapps.Listener.start(Listener.java:274)
        at oracle.ops.opsctl.StartAction.executeListener(StartAction.java:404)
        at oracle.ops.opsctl.Action.execute(Action.java:126)
        at oracle.ops.opsctl.OPSCTLDriver.execute(OPSCTLDriver.java:209)
        at oracle.ops.opsctl.OPSCTLDriver.main(OPSCTLDriver.java:129)
coredb1:ora.coredb1.LISTENER_COREDB1.lsnr:/u01/app/oracle/product/10.2.0/asm/bin/racgwrap[64]: %ORACLE_HOME%/bin/racgmain:  not found.
CRS-0215: Could not start resource 'ora.coredb1.LISTENER_COREDB1.lsnr'.

从错误信息中可以看到应该是由于ORACLE_HOME这个变量信息不正确从而引起racgmain等相关程序无法找到的问题。

再查看相关信息
# crs_stat -p ora.coredb1.LISTENER_COREDB1.lsnr  
NAME=ora.coredb1.LISTENER_COREDB1.lsnr
TYPE=application
ACTION_SCRIPT=/u01/app/oracle/product/10.2.0/asm/bin/racgwrap
ACTIVE_PLACEMENT=0
AUTO_START=1
CHECK_INTERVAL=600
DESCRIPTION=CRS application for listener on node
FAILOVER_DELAY=0
FAILURE_INTERVAL=0
FAILURE_THRESHOLD=0
HOSTING_MEMBERS=coredb1
OPTIONAL_RESOURCES=
PLACEMENT=restricted
REQUIRED_RESOURCES=ora.coredb1.vip
RESTART_ATTEMPTS=5
SCRIPT_TIMEOUT=600
START_TIMEOUT=0
STOP_TIMEOUT=0
UPTIME_THRESHOLD=7d
USR_ORA_ALERT_NAME=
USR_ORA_CHECK_TIMEOUT=0
USR_ORA_CONNECT_STR=/ as sysdba
USR_ORA_DEBUG=0
USR_ORA_DISCONNECT=false
USR_ORA_FLAGS=
USR_ORA_IF=
USR_ORA_INST_NOT_SHUTDOWN=
USR_ORA_LANG=
USR_ORA_NETMASK=
USR_ORA_OPEN_MODE=
USR_ORA_OPI=false
USR_ORA_PFILE=
USR_ORA_PRECONNECT=none
USR_ORA_SRV=
USR_ORA_START_TIMEOUT=0
USR_ORA_STOP_MODE=immediate
USR_ORA_STOP_TIMEOUT=0
USR_ORA_VIP=

检查racgwrap
# more /u01/app/oracle/product/10.2.0/asm/bin/racgwrap
#!/bin/sh
#
# Copyright (c) 2001, 2010, Oracle and/or its affiliates. All rights reserved.
#
# racgmain Wrapper Script
#

# Do not change the line below for ORACLE_HOME setting
ORACLE_HOME=%ORACLE_HOME%
export ORACLE_HOME

 

与coredb2上对应的racgwrap不一致， 修改ORACLE_HOME=,例如 ORACLE_HOME=/u01/app/oracle/production/10.2.0/asm。
然后使用crs_stop -f 强行停止相关资源，将资源状态置为OFFLINE。

# crs_stop ora.coredb1.LISTENER_COREDB1.lsnr -f
Attempting to stop `ora.coredb1.LISTENER_COREDB1.lsnr` on member `coredb1`
Stop of `ora.coredb1.LISTENER_COREDB1.lsnr` on member `coredb1` succeeded.
# crs_stop ora.coredb1.ASM1.asm -f
Attempting to stop `ora.coredb1.ASM1.asm` on member `coredb1`
Stop of `ora.coredb1.ASM1.asm` on member `coredb1` succeeded.

在启动，可正常启动

# srvctl start listener -n coredb1
# srvctl start asm -n coredb1


在My Support上也有一片文档专门对这个问题进行了说明 CRS-215 Srvctl unable to start ASM, Listener, RDBMS Resources 文档 ID: 740319.1
造成这种情况的原因可能是在对RDBMS_HOME或者ASM_HOME应用CRS PSU的时候，没有正确运行pre或者post 脚本。从而导致%ORACLE_HOME%没有被替换成正确的路径。