Applying PSU Patch 11.2.0.1.2 To A Two Node RAC

NOTE : This article is aimed at showing the issues and resolution while applying the PSU patch 11.2.0.1.2 (9655006) to a specific environment and not generalized. Please refer the readme of the PSU patch for detailed procedure to apply PSU patch.

Video Demos are also uploaded to show you the errors and successfull installation of the PSU to 2 node RAC.

.

1. Record Pre Patch Information.

1. Login to each node in RAC as grid user and execute the following command.
   

   $GRID_ORACLE_HOME/OPatch/opatch lsinventory
   

   $GRID_ORACLE_HOME/OPatch/opatch lsinventory -bugs_fixed | grep -i ‘GI PSU’
   

2. Login to each node in RAC as oracle user and execute the following command.
   

   $ORACLE_HOME/OPatch/opatch lsinventory
   

   $ORACLE_HOME/OPatch/opatch lsinventory -bugs_fixed | grep -i ‘DATABASE PSU’
   

3. Connect to each instance and record registry information.
   

   SQL> select comp_name,version,status from dba_registry;
   

2. OPatch Utility Information.

$ORACLE_HOME/OPatch/opatch version -h /u01/home/oracle/product/11.2.0/db_1

$GRID_ORACLE_HOME/OPatch/opatch version -h /u01/home/11.2.0/grid

3. OCM Configuration.

Create ocm response file using the following command and provide appropriate values for the prompts.

$GRID_ORACLE_HOME/OPatch/ocm/bin/emocmrsp

Verify the created file using,

$GRID_ORACLE_HOME/OPatch/ocm/bin/emocmrsp –verbose ocm.rsp

NOTE: The Opatch utility will prompt for your OCM (Oracle Configuration Manager) response file when it is run. Without which we cant proceed further.

4. Validation of Oracle Inventory.

$GRID_ORACLE_HOME/OPatch/opatch lsinventory -detail -oh /u01/home/11.2.0/grid

$ORACLE_HOME/OPatch/opatch lsinventory -detail –oh /u01/home/oracle/product/11.2.0/db_1

5. One-off Patch Conflict Detection and Resolution.

NA

6. Download and Unzip the PSU Patch 9655006

$cd /u01/home/oracle/product/11.2.0/db_1/patches/psupatch

$unzip p9655006_11201_Linux.zip

$chmow -R 777 *

7. Patching GI Home

NOTE: If the GI home is shared, then make sure to shut down the GI stack on all remote nodes. Keep the GI stack up and running on the local node.

NOTE: If the GI home is not shared, then make sure the GI stack is running on all nodes in the cluster.

Our Grid Home is not shared, So don’t shutdown any services.

$su – ( Login to root user )

#/u01/home/11.2.0/grid/OPatch/opatch auto /u01/home/oracle/product/11.2.0/db_1/patches/psupatch -oh /u01/home/11.2.0/grid

Monitor the logfile created in $GRID_ORACLE_HOME/cfgtoollogs/

Execute the above opatch command on each RAC node as root user.

** Please refer the Issue & Resolutions secion in the same document for any issues.

8. Patching RAC Database Homes

All Oracle processes and applications (such as emconsole and emagent) that are running from the database home and that are not managed by clusterware should be stopped manually before you apply the patch using the opatch auto command.

$su – ( Login to root user )

# /u01/home/oracle/product/11.2.0/db_1/OPatch/opatch auto /u01/home/oracle/product/11.2.0/db_1/patches/psupatch -oh /u01/home/oracle/product/11.2.0/db_1

Monitor the logfile created in $ORACLE_HOME/cfgtoollogs/

Execute the above opatch command on each RAC node as root user.

** Please refer the Issue & Resolutions secion in the same document for any issues.

9. Loading Modified SQL Files into the Database.

For each database instance running on the Oracle home being patched, connect to the database using SQL*Plus. Connect as SYSDBA and run the catbundle.sql script as follows:

cd $ORACLE_HOME/rdbms/admin

sqlplus /nolog

SQL> CONNECT / AS SYSDBA

SQL> @catbundle.sql psu apply

SQL> QUIT

Check the log files in $ORACLE_HOME/cfgtoollogs/catbundle for any errors

10. Patch Successful Verification Steps.

1. Login to each node in RAC as grid user and execute the following command.
   

   $GRID_ORACLE_HOME/OPatch/opatch lsinventory -bugs_fixed | grep -i ‘GI PSU’
   

2. Login to each node in RAC as oracle user and execute the following command.
   

   $ORACLE_HOME/OPatch/opatch lsinventory -bugs_fixed | grep -i ‘DATABASE PSU’
   

3. Connect to each instance and record registry information.
   

   SQL> select comp_name,version,status from dba_registry;
   

   SQL> select * from dba_registry_history;
   

11. Issues & Resolutions.

Issue 1:

When applying the patch, you may get the following error

The opatch minimum version check for patch /u01/home/oracle/product/11.2.0/db_1/patches/9655006/custom failed for /u01/home/11.2.0/grid

The opatch minimum version check for patch /u01/home/oracle/product/11.2.0/db_1/patches/9655006/etc failed for /u01/home/11.2.0/grid

The opatch minimum version check for patch /u01/home/oracle/product/11.2.0/db_1/patches/9655006/files failed for /u01/home/11.2.0/grid

Opatch version check failed for oracle home /u01/home/11.2.0/grid

Opatch version check failed

update the opatch version for the failed homes and retry

Solution:

Ref Note : 1308858.1

We need to provide the Patch unzipped base directory, not the directory including patch number

Ex : /u01/home/oracle/product/11.2.0/db_1/patches/psupatch/9655006 ( Wrong )

/u01/home/oracle/product/11.2.0/db_1/patches/psupatch ( correct )

Issue 2

Patch may exit with the following error messages

Unable to determine if /u01/home/11.2.0/grid is shared oracle home

Enter ‘yes’ if this is not a shared home or if the prerequiste actions are performed to patch this shared home (yes/no):yes

You must kill crs processes or reboot the system to properly

cleanup the processes started by Oracle clusterware

The Oracle Clusterware stack failed to stop.

You should stop the stack with ‘crsctl stop crs’ and rerun the command

The opatch Applicable check failed for /u01/home/11.2.0/grid. The patch is not applicable for /u01/home/11.2.0/grid

patch ././9655006 apply failed for home /u01/home/11.2.0/grid

Solution :

This error may be specific to this environment though want to specify it here. This error is due to the reason that there are some cluster resources available referring the 10g database installed earlier. Delete those resources from clusterware.

# crsctl delete resource -f

Issue 3 :

PSU patch for GRID home on node 2 failed with the following error

Unable to determine if /u01/home/11.2.0/grid is shared oracle home

Enter ‘yes’ if this is not a shared home or if the prerequiste actions are performed to patch this shared home (yes/no):yes

Successfully unlock /u01/home/11.2.0/grid

patch ././9655006 apply failed for home /u01/home/11.2.0/grid

Verified the detailed log file locate in /u01/home/11.2.0/grid/cfgtoollogs/ and found the permission issue on some files.

The following actions have failed:

Copy failed from ‘/u01/home/oracle/product/11.2.0/db_1/patches/psupatch/9655006/files/bin/crsctl.bin’ to ‘/u01/home/11.2.0/grid/bin/crsctl.bin’…

Copy failed from ‘/u01/home/oracle/product/11.2.0/db_1/patches/psupatch/9655006/files/bin/oifcfg.bin’ to ‘/u01/home/11.2.0/grid/bin/oifcfg.bin’…

Solution:

Give 777 permission to these files crsctl.bin and oifcfg.bin

Issue 4 :

PSU Patch for RDBMS Home on node 1 failed with the following error

Unable to determine if /u01/home/oracle/product/11.2.0/db_1 is shared oracle home

Enter ‘yes’ if this is not a shared home or if the prerequiste actions are performed to patch this shared home (yes/no):yes

patch ././9655006/custom/server/9655006 apply failed for home /u01/home/oracle/product/11.2.0/db_1

Verified the detailed log file locate in /u01/home/oracle/product/11.2.0/db_1/cfgtoollogs and found the platform issue.

Running prerequisite checks…

Prerequisite check “CheckPatchApplicableOnCurrentPlatform” failed.

The details are:

Patch ( 9655006 ) is not applicable on current platform.

Platform ID needed is : 46

Platform IDs supported by patch are: 226

UtilSession failed: Prerequisite check “CheckPatchApplicableOnCurrentPlatform” failed.

OPatch failed with error code 73

Solution:

export OPATCH_PLATFORM_ID=226 and execute the psu patch command again.

Issue 5:

While applying Grid Infrastructure PSU (patch 9343627) to $GRID_HOME, opatch prerequisite check CheckSystemSpace failed:

Running prerequisite checks…

 Prerequisite check “CheckSystemSpace” failed.

 The details are:

 Required amount of space(4373569440) is not available.

 UtilSession failed: Prerequisite check “CheckSystemSpace” failed.

 

 OPatch failed with error code 73

Solution:

Follow the note ID 1088455.1 and make sure you have enough space on disk in GRID HOME mount point.

Reference NOTE IDs : 1082394.1 , 1308858.1 , 1169036.1 , 1290354.1, 1088455.1 and 1210964.1