PSU实施异常解决

--RAC实施
1：问题
[root@andy_node1 OPatch]# ./opatch auto /data1/media/20130820/psu07 -ocmrf /data1/oracle/app/11.2.0/grid/OPatch/ocm/bin/ocm.rsp
patch /data1/media/20130820/psu07/16619898/custom/server/16619898  apply successful for home  /data1/oracle/app/oracle/product/11.2.0/dbhome_1
patch /data1/media/20130820/psu07/16619892  apply failed  for home  /data1/oracle/app/oracle/product/11.2.0/dbhome_1
patch /data1/media/20130820/psu07/16619898  apply successful for home  /data1/oracle/app/11.2.0/grid
patch /data1/media/20130820/psu07/16619892  apply successful for home  /data1/oracle/app/11.2.0/grid

2：查看详细日志
[root@andy_node1 OPatch]# vi /data1/oracle/app/11.2.0/grid/cfgtoollogs/opatchauto2013-08-25_10-41-06.log
: Opening file /etc/oracle/ocr.loc
2013-08-25 10:42:39: Value (FALSE) is set for key=local_only
oralce:
$opatch prereq CheckComponents -ph /data1/media/20130820/psu07/16619898/custom/server/16619898 -oh /data1/oracle/app/oracle/product/11.2.0/dbhome_1
$opatch prereq CheckComponents -ph /data1/media/20130820/psu07/16619892 -oh /data1/oracle/app/oracle/product/11.2.0/dbhome_1
$opatch prereq CheckConflictAgainstOH -ph /data1/media/20130820/psu07/16619898/custom/server/16619898 -oh /data1/oracle/app/oracle/product/11.2.0/dbhome_1
$opatch prereq CheckConflictAgainstOH -ph /data1/media/20130820/psu07/16619892 -oh /data1/oracle/app/oracle/product/11.2.0/dbhome_1
grid:
$opatch prereq CheckComponents -ph /data1/media/20130820/psu07/16619898 -oh /data1/oracle/app/11.2.0/grid
$opatch prereq CheckComponents -ph /data1/media/20130820/psu07/16619892 -oh /data1/oracle/app/11.2.0/grid
$opatch prereq CheckConflictAgainstOH -ph /data1/media/20130820/psu07/16619898 -oh /data1/oracle/app/11.2.0/grid
$opatch prereq CheckConflictAgainstOH -ph /data1/media/20130820/psu07/16619892 -oh /data1/oracle/app/11.2.0/grid
oracle:
$opatch prereq CheckApplicable -ph /data1/media/20130820/psu07/16619898/custom/server/16619898 -oh /data1/oracle/app/oracle/product/11.2.0/dbhome_1
$opatch prereq CheckApplicable -ph /data1/media/20130820/psu07/16619892 -oh /data1/oracle/app/oracle/product/11.2.0/dbhome_1

srvctl stop home -o /data1/oracle/app/oracle/product/11.2.0/dbhome_1 -s /data1/oracle/app/oracle/product/11.2.0/dbhome_1/srvm/admin/stophome.txt -n andy_node1
/sbin/fuser -k /data1/oracle/app/oracle/product/11.2.0/dbhome_1/bin/oracle
/bin/su oracle -c ' /data1/media/20130820/psu07/16619898/custom/server/16619898/custom/scripts/prepatch.sh -dbhome /data1/oracle/app/oracle/product/11.2.0/dbhome_1 '
/bin/su oracle -c ' /data1/oracle/app/oracle/product/11.2.0/dbhome_1/OPatch/opatch napply /data1/media/20130820/psu07/16619898/custom/server/16619898
-local -silent -ocmrf /data1/oracle/app/11.2.0/grid/OPatch/ocm/bin/ocm.rsp -oh /data1/oracle/app/oracle/product/11.2.0/dbhome_1 -invPtrLoc /data1/oracle/app/oracle/product/11.2.0/dbhome_1/oraInst.loc '
/bin/su oracle -c ' /data1/oracle/app/oracle/product/11.2.0/dbhome_1/OPatch/opatch napply /data1/media/20130820/psu07/16619892
-local -silent -ocmrf /data1/oracle/app/11.2.0/grid/OPatch/ocm/bin/ocm.rsp -oh /data1/oracle/app/oracle/product/11.2.0/dbhome_1 -invPtrLoc /data1/oracle/app/oracle/product/11.2.0/dbhome_1/oraInst.loc '
/bin/su exited with rc=73
 Verifying environment and performing prerequisite checks...
 Prerequisite check "CheckActiveFilesAndExecutables" failed.
 The details are:
 Following executables are active :
 /data1/oracle/app/oracle/product/11.2.0/dbhome_1/lib/libclntsh.so.11.1
 UtilSession failed: Prerequisite check "CheckActiveFilesAndExecutables" failed.
 Log file location: /data1/oracle/app/oracle/product/11.2.0/dbhome_1/cfgtoollogs/opatch/opatch2013-08-25_10-43-47AM_1.log
 OPatch failed with error code 73
2013-08-25 10:43:53: patch /data1/media/20130820/psu07/16619892  apply failed  for home  /data1/oracle/app/oracle/product/11.2.0/dbhome_1

[root@andy_node1 ~]# vi /data1/oracle/app/oracle/product/11.2.0/dbhome_1/cfgtoollogs/opatch/opatch2013-08-25_10-43-47AM_1.log
OPatch invoked as follows: 'napply /data1/media/20130820/psu07/16619892
-local -silent -ocmrf /data1/oracle/app/11.2.0/grid/OPatch/ocm/bin/ocm.rsp -oh /data1/oracle/app/oracle/product/11.2.0/dbhome_1 -invPtrLoc /data1/oracle/app/oracle/product/11.2.0/dbhome_1/oraInst.loc '
Start fuser command /sbin/fuser /data1/oracle/app/oracle/product/11.2.0/dbhome_1/bin/oracle at Sun Aug 25 10:43:53 CST 2013
Finish fuser command /sbin/fuser /data1/oracle/app/oracle/product/11.2.0/dbhome_1/bin/oracle at Sun Aug 25 10:43:53 CST 2013
Start fuser command /sbin/fuser /data1/oracle/app/oracle/product/11.2.0/dbhome_1/bin/renamedg at Sun Aug 25 10:43:53 CST 2013
Finish fuser command /sbin/fuser /data1/oracle/app/oracle/product/11.2.0/dbhome_1/bin/renamedg at Sun Aug 25 10:43:53 CST 2013
Start fuser command /sbin/fuser /data1/oracle/app/oracle/product/11.2.0/dbhome_1/bin/nid at Sun Aug 25 10:43:53 CST 2013
Finish fuser command /sbin/fuser /data1/oracle/app/oracle/product/11.2.0/dbhome_1/bin/nid at Sun Aug 25 10:43:53 CST 2013
Start fuser command /sbin/fuser /data1/oracle/app/oracle/product/11.2.0/dbhome_1/bin/tnslsnr at Sun Aug 25 10:43:53 CST 2013
Finish fuser command /sbin/fuser /data1/oracle/app/oracle/product/11.2.0/dbhome_1/bin/tnslsnr at Sun Aug 25 10:43:53 CST 2013
Start fuser command /sbin/fuser /data1/oracle/app/oracle/product/11.2.0/dbhome_1/bin/proc at Sun Aug 25 10:43:53 CST 2013
Finish fuser command /sbin/fuser /data1/oracle/app/oracle/product/11.2.0/dbhome_1/bin/proc at Sun Aug 25 10:43:53 CST 2013
Start fuser command /sbin/fuser /data1/oracle/app/oracle/product/11.2.0/dbhome_1/bin/rman at Sun Aug 25 10:43:53 CST 2013
Finish fuser command /sbin/fuser /data1/oracle/app/oracle/product/11.2.0/dbhome_1/bin/rman at Sun Aug 25 10:43:53 CST 2013
Start fuser command /sbin/fuser /data1/oracle/app/oracle/product/11.2.0/dbhome_1/bin/wrap at Sun Aug 25 10:43:53 CST 2013
Finish fuser command /sbin/fuser /data1/oracle/app/oracle/product/11.2.0/dbhome_1/bin/wrap at Sun Aug 25 10:43:53 CST 2013
Start fuser command /sbin/fuser /data1/oracle/app/oracle/product/11.2.0/dbhome_1/bin/plshprof at Sun Aug 25 10:43:53 CST 2013
Finish fuser command /sbin/fuser /data1/oracle/app/oracle/product/11.2.0/dbhome_1/bin/plshprof at Sun Aug 25 10:43:53 CST 2013
Start fuser command /sbin/fuser /data1/oracle/app/oracle/product/11.2.0/dbhome_1/lib/libclntsh.so.11.1 at Sun Aug 25 10:43:53 CST 2013
Finish fuser command /sbin/fuser /data1/oracle/app/oracle/product/11.2.0/dbhome_1/lib/libclntsh.so.11.1 at Sun Aug 25 10:43:53 CST 2013
Following executables are active :
/data1/oracle/app/oracle/product/11.2.0/dbhome_1/lib/libclntsh.so.11.1
Prerequisite check "CheckActiveFilesAndExecutables" failed.
The details are:
Following executables are active :
/data1/oracle/app/oracle/product/11.2.0/dbhome_1/lib/libclntsh.so.11.1
Start fuser command /sbin/fuser /data1/oracle/app/oracle/product/11.2.0/dbhome_1/bin/amdu at Sun Aug 25 10:43:53 CST 2013
Finish fuser command /sbin/fuser /data1/oracle/app/oracle/product/11.2.0/dbhome_1/bin/amdu at Sun Aug 25 10:43:53 CST 2013
Start fuser command /sbin/fuser /data1/oracle/app/oracle/product/11.2.0/dbhome_1/bin/kfed at Sun Aug 25 10:43:53 CST 2013
Finish fuser command /sbin/fuser /data1/oracle/app/oracle/product/11.2.0/dbhome_1/bin/kfed at Sun Aug 25 10:43:53 CST 2013
OUI-67073:UtilSession failed: Prerequisite check "CheckActiveFilesAndExecutables" failed.
Finishing UtilSession at Sun Aug 25 10:43:53 CST 2013
Log file location: /data1/oracle/app/oracle/product/11.2.0/dbhome_1/cfgtoollogs/opatch/opatch2013-08-25_10-43-47AM_1.log
Stack Description: java.lang.RuntimeException: Prerequisite check "CheckActiveFilesAndExecutables" failed.
StackTrace: oracle.opatch.OPatchSessionHelper.runApplyPrereqs(OPatchSessionHelper.java:6420)
StackTrace: oracle.opatch.opatchutil.NApply.process(NApply.java:1587)
StackTrace: oracle.opatch.opatchutil.OUSession.napply(OUSession.java:1136)
StackTrace: sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
StackTrace: sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
StackTrace: sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
StackTrace: java.lang.reflect.Method.invoke(Method.java:592)
StackTrace: oracle.opatch.UtilSession.process(UtilSession.java:322)
StackTrace: oracle.opatch.OPatchSession.main(OPatchSession.java:2346)
StackTrace: oracle.opatch.OPatch.main(OPatch.java:613)

3：日志里就有解决办法
[oracle@andy_node1 lib]$ ls -l libclntsh.so
lrwxrwxrwx 1 oracle oinstall 17 May  7  2012 libclntsh.so -> libclntsh.so.11.1

[root@andy_node1 ~]# /sbin/fuser /data1/oracle/app/oracle/product/11.2.0/dbhome_1/lib/libclntsh.so.11.1
/data1/oracle/app/oracle/product/11.2.0/dbhome_1/lib/libclntsh.so.11.1: 26679m

[root@andy_node1 ~]# ps -ef | grep 26679
root     12370 11255  0 20:36 pts/0    00:00:00 grep 26679
patrol   26679  8362  0 12:17 ?        00:00:03 /data1/oracle/app/oracle/product/11.2.0/dbhome_1/bin/sqlplus -s
oracle   26682 26679  0 12:17 ?        00:02:08 oracleandydb1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))

[root@andy_node1 ~]# ps -ef | grep sqlplus
root     12396 11255  0 20:36 pts/0    00:00:00 grep sqlplus
patrol   26679  8362  0 12:17 ?        00:00:03 /data1/oracle/app/oracle/product/11.2.0/dbhome_1/bin/sqlplus -s
[root@andy_node1 ~]#

4：不停的去kill -9 sqlplus进程，再次执行成功
[root@andy_node1 OPatch]# ./opatch auto /data1/media/20130820/psu07 -ocmrf /data1/oracle/app/11.2.0/grid/OPatch/ocm/bin/ocm.rsp
patch /data1/media/20130820/psu07/16619898/custom/server/16619898  apply successful for home  /data1/oracle/app/oracle/product/11.2.0/dbhome_1
patch /data1/media/20130820/psu07/16619892  apply successful for home  /data1/oracle/app/oracle/product/11.2.0/dbhome_1
patch /data1/media/20130820/psu07/16619898  apply successful for home  /data1/oracle/app/11.2.0/grid
patch /data1/media/20130820/psu07/16619892  apply successful for home  /data1/oracle/app/11.2.0/grid

--单机physical dataguard实施
1：dataguard是rman恢复来的，没有加到crs里，实际上加进去，自动open也是open read only with apply模式的。
考虑打rac时crs会自动起停，本次想停掉has和db来打，并想看看能否遇到GI和DB分开打的情况。
[grid@andy_phydg ~]$ crsctl status res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS      
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA_DG.dg
               ONLINE  ONLINE       andy_phydg                                 
ora.LISTENER.lsnr
               ONLINE  ONLINE       andy_phydg                                 
ora.OCR_DG.dg
               ONLINE  ONLINE       andy_phydg                                 
ora.asm
               ONLINE  ONLINE       andy_phydg              Started            
ora.ons
               OFFLINE OFFLINE      andy_phydg                                 
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
      1        ONLINE  ONLINE       andy_phydg                                 
ora.diskmon
      1        OFFLINE OFFLINE                                                  
ora.evmd
      1        ONLINE  ONLINE       andy_phydg                                 
[grid@andy_phydg ~]$

2：问题
[root@andy_phydg OPatch]# ./opatch auto /data1/media/20130820/psu07 -ocmrf /data1/grid/crs/OPatch/ocm/bin/ocm.rsp
Executing /data1/grid/crs/perl/bin/perl ./crs/patch11203.pl -patchdir /data1/media/20130820 -patchn psu07 -ocmrf /data1/grid/crs/OPatch/ocm/bin/ocm.rsp -paramfile /data1/grid/crs/crs/install/crsconfig_params
/data1/grid/crs/crs/install/crsconfig_params
/data1/grid/crs/crs/install/s_crsconfig_defs

This is the main log file: /data1/grid/crs/cfgtoollogs/opatchauto2013-08-25_13-00-37.log
This file will show your detected configuration and all the steps that opatchauto attempted to do on your system: /data1/grid/crs/cfgtoollogs/opatchauto2013-08-25_13-00-37.report.log

2013-08-25 13:00:37: Starting Oracle Restart Patch Setup
Using configuration parameter file: /data1/grid/crs/crs/install/crsconfig_params
Not able to retreive database home information
[root@andy_phydg OPatch]#

[oracle@andy_phydg 17067783]$ vi /data1/grid/crs/cfgtoollogs/opatchauto2013-08-25_13-00-37.log
/bin/su grid -c ' /data1/grid/crs/OPatch/opatch util saveconfigurationsnapshot -configFile /data1/grid/crs/cfgtoollogs/opatchauto2013-08-25_13-00-37.cfg.log '
2013-08-25 13:00:43: /bin/su exited with rc=73
 Invoking utility "saveconfigurationsnapshot"
 UtilSession failed: UtilSession failed to generate configuration xml file [ PRKH-1010 : Unable to communicate with CRS services.
 PRKH-3003 : An attempt to communicate with the CSS daemon failed ]

 OPatch failed with error code 73

2013-08-25 13:00:43: Looking for configured databases on node andy_phydg
2013-08-25 13:00:44: Databases configured on node andy_phydg are: PRCR-1119 : Failed to look up CRS resources of database type PRCR-1115 : Failed to find entities of type resource that match filters (TYPE == ora.database.type) and contain attributes DB_UNIQUE_NAME,ORACLE_HOME,VERSION Cannot communicate with crsd
2013-08-25 13:00:44: Determining ORACLE_HOME paths for configured databases
2013-08-25 13:00:44: Executing cmd: /data1/grid/crs/bin/srvctl config database -d PRCR-1119 : Failed to look up CRS resources of database type
2013-08-25 13:00:44: Command output:
>  PRKO-2002 : Invalid command line option: :
>End Command output
2013-08-25 13:00:44: try with srvctl config database -v
2013-08-25 13:00:44: Executing cmd: /data1/grid/crs/bin/srvctl config database -v
2013-08-25 13:00:45: Command output:
>  PRCR-1119 : Failed to look up CRS resources of database type
>  PRCR-1115 : Failed to find entities of type resource that match filters (TYPE == ora.database.type) and contain attributes DB_UNIQUE_NAME,ORACLE_HOME,VERSION
>  Cannot communicate with crsd
>End Command output
2013-08-25 13:00:45: Not able to retreive database home information

3：尝试
[root@andy_phydg OPatch]# ./opatch auto /data1/media/20130820/psu07 -oh /data1/grid/crs -ocmrf /data1/grid/crs/OPatch/ocm/bin/ocm.rsp
Executing /data1/grid/crs/perl/bin/perl ./crs/patch11203.pl -patchdir /data1/media/20130820 -patchn psu07 -oh /data1/grid/crs -ocmrf /data1/grid/crs/OPatch/ocm/bin/ocm.rsp -paramfile /data1/grid/crs/crs/install/crsconfig_params
/data1/grid/crs/crs/install/crsconfig_params
/data1/grid/crs/crs/install/s_crsconfig_defs

This is the main log file: /data1/grid/crs/cfgtoollogs/opatchauto2013-08-25_13-06-05.log
This file will show your detected configuration and all the steps that opatchauto attempted to do on your system: /data1/grid/crs/cfgtoollogs/opatchauto2013-08-25_13-06-05.report.log

2013-08-25 13:06:05: Starting Oracle Restart Patch Setup
Using configuration parameter file: /data1/grid/crs/crs/install/crsconfig_params
Clusterware is either  not running or not configured. You have the following 2 options
1. Configure and Start the Clusterware on this node and re-run the tool
2. or Run the tool with the -och option and then invoke  tool with -oh to patch the RDBMS home
[root@andy_phydg OPatch]#

4：解决
[root@andy_phydg OPatch]# ./opatch auto /data1/media/20130820/psu07 -och /data1/grid/crs -ocmrf /data1/grid/crs/OPatch/ocm/bin/ocm.rsp
patch /data1/media/20130820/psu07/16619898  apply successful for home  /data1/grid/crs
patch /data1/media/20130820/psu07/16619892  apply successful for home  /data1/grid/crs
CRS-4123: Oracle High Availability Services has been started.
[root@andy_phydg OPatch]#

停掉has,会Checking the status of ohasd失败
[root@andy_phydg OPatch]# ./opatch auto /data1/media/20130820/psu07 -oh /data1/andydb/db -ocmrf /data1/grid/crs/OPatch/ocm/bin/ocm.rsp
Executing /data1/grid/crs/perl/bin/perl ./crs/patch11203.pl -patchdir /data1/media/20130820 -patchn psu07 -oh /data1/andydb/db -ocmrf /data1/grid/crs/OPatch/ocm/bin/ocm.rsp -paramfile /data1/grid/crs/crs/install/crsconfig_params
/data1/grid/crs/crs/install/crsconfig_params
/data1/grid/crs/crs/install/s_crsconfig_defs

This is the main log file: /data1/grid/crs/cfgtoollogs/opatchauto2013-08-25_13-45-23.log
This file will show your detected configuration and all the steps that opatchauto attempted to do on your system: /data1/grid/crs/cfgtoollogs/opatchauto2013-08-25_13-45-23.report.log

2013-08-25 13:45:23: Starting Oracle Restart Patch Setup
Using configuration parameter file: /data1/grid/crs/crs/install/crsconfig_params
Clusterware is either  not running or not configured. You have the following 2 options
1. Configure and Start the Clusterware on this node and re-run the tool
2. or Run the tool with the -och option and then invoke  tool with -oh to patch the RDBMS home
[root@andy_phydg OPatch]#

启动has
[root@andy_phydg OPatch]# ./opatch auto /data1/media/20130820/psu07 -oh /data1/andydb/db -ocmrf /data1/grid/crs/OPatch/ocm/bin/ocm.rsp
patch /data1/media/20130820/psu07/16619898/custom/server/16619898  apply successful for home  /data1/andydb/db
patch /data1/media/20130820/psu07/16619892  apply successful for home  /data1/andydb/db
[root@andy_phydg OPatch]#

DB不在crs里，需要sqlplus到mount standby database去开启mrp。