aix5300-07 hacmp oracle 10g

Oracle Server - Enterprise Edition - Version: 10.1 to 10.2
AIX5L Based Systems (64-bit)
STATUS of HACMP 5.3 & 5.4 Certification with Oracle Clusterware 10g (R1 & R2)
What is being announced?
STATUS of IBM HACMP 5.3, 5.4 Certification with Oracle RAC 10g
IBM and Oracle have developed a joint plan to enhance the 10gR2 RAC and HACMP 5.3 & 5.4, RSCT and AIX 5L 5.3 offerings so this certification can complete. The HACMP 5.3 certification is projected complete by end of September 2007, the HACMP 5.4 certification is projected to complete by the end of 2007.

In the 10g product range, IBM and Oracle plan to certify only 10gR2 RAC with HACMP 5.3 & 5.4, and this certified combination will require updated versions AIX 5L 5.3 and RSCT. Consistent with life cycle service planning, IBM and Oracle will not certify any new combinations of the following:

· 10gR1 with any new versions of HACMP or AIX 5L
· 10gR2 on AIX 5L 5.2 with HACMP 5.3 or 5.4

Customers must develop upgrade plans consistent with documented life cycle plans for the currently supported versions of these older combinations. The required enhancements will be distributed through each company’s service process. Installation of the service from both IBM and Oracle will be required for use of 10gR2 RAC with HACMP 5.3 and 5.4 deployments. IBM is providing special support terms for customers that must remain on HACMP 5.2 during the completion of this certification, details are provided below.
When the certifications are complete, the enhancements will continue to provide Oracle RAC customers with a capability that is optimized to provide the highly dependable deployments they require.


What do you need to do?
Review the complete list of currently certified combinations. Please consult Oracle’s Certify web site – General notes for RAC for UNIX on IBM AIX based systems. Insure their current deployment is consistent with currently certified combinations of software.

Confirm that their cluster is implemented in a manner consistent with HACMP best practices. A particular focus on redundancy for the networks and adapters which provide the routes by which HACMP transfers heart beats between nodes is important. This is an area where more is better; the most reliable clusters have three or more networks to ensure that no simple combinations of hardware or software failures prevent heart beats from flowing. The full text of the HACMP best practices document can be found at:

Prepare for upgrades to 10g R2, AIX 5L 5.3, RSCT and HACMP 5.3.

Those using 9i R2 RAC may continue to use HACMP 5.2 or 5.3; no changes are required for this version of Oracle. Those planning to upgrade to 10g should review recommendations for those versions, and plan to move to 10gR2.
Those using 10g R1 RAC and HACMP should remain on HACMP 5.2, and establish plans to upgrade to 10gR2 RAC and AIX 5L 5.3, which will be requirements for a supported deployment with HACMP 5.3 and above. Suggested planning horizon for these upgrades is 3 to 9 months.
Those using 10gR2 RAC with HACMP should remain on HACMP 5.2 for now, which is currently certified for use on AIX 5.2 and 5.3. Plans should be made to upgrade to AIX 5L 5.3, which will be a required for this certified deployment. Suggested planning horizon for upgrades to AIX 5L 5.3 are 3 to 9 months, to HACMP 5.3 are 6 to 12 months.

Continue to monitor this Metalink note for updates on delivery and certification of the enhanced versions of 10gR2, AIX 5L 5.3, RSCT and HACMP 5.4.
Contact IBM to confirm your need for continued HACMP 5.2 support.

For further details on HACMP use customers should contact their IBM sales representative, or for technical question email .


Configuring HACMP 5.3, 10gR2 with Multi-Node Disk Heartbeat (MNDHB)
Introduction
This section documents the required software levels and new configuration requirements for using Oracle 10g RAC Database with HACMP on AIX 5L. In order to ensure high availability and data integrity in the event of system failures the proper levels of Oracle 10g RAC Clusterware, AIX and HACMP software must be installed and the system must be properly configured, including the new HACMP 5.3 capability for Multi-Node Disk Heartbeat (MNDHB). The certified solution requires one MNDHB network to be defined for each RAC Voting Disk. Each MNDHB and Voting Disk pair must be collocated on a single hdisk, separate from the other pairs. Users must also configure MNDHB so that the node is halted if access is lost to a quorum of the MNDHB networks in the enhanced concurrent volume group. The following provides general guidance on how to do this.

The changes for Multi-Node Disk Heartbeat for use with HACMP v5.3 may be obtained from APAR IZ01809, which delivers the support in ifix format which can be applied to a system with HACMP v5.3 and PTF5 (IY94307). Future service updates will contain the same support in PTF format.

The MNDHB network works in conjunction with Oracle CSS changes in patch Bug 5497221, to maintain database integrity in the event of a cluster partition. The MNDHB network must be configuredBug 5497221, to maintain database integrity in the event of a cluster partition. The MNDHB network must be configured and collocated on hdisks with the Oracle voting disks, to properly fence partitioned nodes. Previous recommendations for configuring multiple HACMP heartbeat paths over both IP and non-IP networks still apply.

The intent of this paper is to document the configuration of the new MNDHB network required to use Oracle 10gR2 RAC. Please refer to section 4 for additional cluster configuration documentation. The HACMP APAR IZ01809 also contains a README document with extensive planning, configuration and usage information.
2) Software Requirements
AIX:
       AIX 5.3 TL06 or newer (specifically bos.rte.lvm must be at least 5.3.0.60)

HACMP:
       HACMP V5.3 with PTF5 (APAR: IY94307) and cluster.es.clvm installed
       HACMP APAR: IZ01809

RSCT (component of AIX):
       RSCT (rsct.basic.rte) version 2.4.7.3
       RSCT APAR: IZ01838

Oracle 10gR2:
       Oracle 10gR2 patchset 10.2.0.3 plus patch for Bug 5497221

2.1) Obtaining the required IBM software
Obtain the latest IBM software at “Quick links for AIX fixes”:
If the required APAR is not available on the website, contact IBM support for the interim fix (ifix).

2.2) Obtaining the required Oracle software
The required Oracle patch can be downloaded from , under "Patches & Updates” search for patch Bug 5497221 for AIX 5L platform.

2.3) Planning
When upgrading an existing Oracle RAC cluster downtime must be taken into account. Installation of the AIX and Oracle patch sets will require shutting down all nodes in an existing cluster, and rebooting each node several times. A rolling upgrade is not possible with this patch set.

3) Configuration
In order to reduce the likelihood of a cluster partition, HACMP recommends multiple IP networks and at least one non-IP network. The non-IP networks can be implemented using RS232 or disk heart beating. For systems using Oracle RAC 10g and HACMP enhanced concurrent resources (enhanced concurrent logical volumes) for database storage, Multi-node Disk Heartbeat (MNDHB) networks must be configured.
3.1) Configuring HACMP and MNDHB networks for u with RAC 10g
Install, configure and have HACMP running prior to installing RAC. For an Oracle database 10g RAC configuration do not use HACMP for IP failovers on the RAC network interfaces (public, VIP or private). These network interfaces should not be configured to use HACMP IP failover. VIP failovers are managed by RAC CRS. The RAC network interfaces are bound to individual nodes and RAC instances. Problems can occur with the CRS if HACMP reconfigures IP addresses over different interfaces or fails over addresses across nodes. You can use HACMP for fail over of IP address on the RAC node if these addresses are not used by RAC.
3.2) Sample configuration steps
Prior to beginning the configuration steps please make sure the HACMP clcomdES daemon is running.
# lssrc -s clcomdES
If the daemon is not active start it:
# startsrc –s clcomdES
Also setup the following configuration file:

· /usr/es/sbin/cluster/etc/rhosts

Add the HACMP cluster node IP names to this file. In HACMP release 5.1 and higher, the Cluster Communications daemon uses the trusted /usr/es/sbin/cluster/etc/rhosts file, and removes reliance on an /.rhosts file. In HACMP 5.2 and up, the daemon provides support for message authentication and encryption.
Follow these steps to create the HACMP cluster:

Create HACMP cluster and add RAC nodes
# smitty cm_add_change_show_an_hacmp_cluster.dialog
* Cluster Name []
Create HACMP cluster nodes; repeat for each RAC node.
# smitty cm_add_a_node_to_the_hacmp_cluster_dialog
* Node Name []
Communication Path to Node []
Create HACMP ethernet heartbeat networks. The HACMP configuration requires network definitions, however we will select NO for the IP address takeover for these networks, since they are used by RAC. Create at least TWO network definitions, one for the Oracle public interface and a second one for the Oracle private (cluster interconnect) network. Additional ethernet heartbeat networks can be added if desired.
# smitty cm_add_a_network_to_the_hacmp_cluster_select
- select ether network
* Network Name []
* Network Type ether
* Netmask []
* Enable IP Address Takeover via IP Aliases [No]
IP Address Offset for Heart beating over IP Aliases []
For each of the networks added in step 3, define all of the IP names for each RAC node associated with that network. For example, for the Oracle private network, define the Oracle private IP name for each RAC node.
# smitty cm_add_communication_interfaces_devices.select
- select: Add Pre-defined Communication Interfaces and Devices / Communication Interfaces / desired network
* IP Label/Address []
* Network Type ether
* Network Name some_network_name
* Node Name []
Network Interface []
Create an HACMP resource group for the enhanced concurrent volume group resource with the following options.
# smitty config_resource_group.dialog.custom
* Resource Group Name []
* Participating Nodes (Default Node Priority) [Startup Policy Online On All Available Nodes
Fallover Policy Bring Offline (On Error Node Only)
Fallback Policy Never Fallback
Create an AIX enhanced concurrent volume group (Big VG, or Scalable VG) from ‘smitty mkvg’ or command line. The VG must contain at least one hdisk for each voting disk (three voting disks are recommended).
# smitty _mksvg
VOLUME GROUP name [] PP SIZE in MB
* PHYSICAL VOLUME names []
Force the creation of a volume group? no
Activate volume group AUTOMATICALLY no at system restart?
Volume Group MAJOR NUMBER []
Create VG Concurrent Capable? enhanced concurrent
Max PPs per VG in kilobytes
Max Logical Volumes
Under “Change/Show Resources for a Resource Group (standard)” ADD the concurrent volume group to the resource group added in above steps
# smitty cm_change_show_resources_std_resource_group_menu_dmn.select
- select resource group created in step 5
Resource Group Name shared_storage
Participating Nodes (Default Node Priority)
Startup Policy Online On All Available Nodes
Fallover Policy Bring Offline (On Error Node Only)
Fallback Policy Never Fallback
Concurrent Volume Groups [Use forced varyon of volume groups, if necessary false
Application Servers []
One MNDHB network is defined for each RAC Voting Disk. Each MNDHB and Voting Disk pair, must be collocated on a single hdisk, separate from the other pairs. The MNDHB network and Voting Disks exist on shared logical volumes in an enhanced concurrent logical volume managed by HACMP as an enhanced concurrent resource. Create a MNDHB LV on each of the hdisks in the VG created in step 6, for the hdisks which will also have a voting disk LV.
# smitty cl_add_mndhb_lv
- select resource group defined in step 5
* Physical Volume name
Logical Volume Name []
Logical Volume Label []
Volume Group name ccvg
Resource Group Name shared_storage
Network Name [net_diskhbmulti_01]
Note: when you define the LVs for the RAC voting disks, they should be defined on the same disks, one per disk, as used in this step for the MNDHB LVs.
Configure MNDHB so that the node is halted if access is lost to a quorum of the MNDHB networks in the enhanced concurrent volume group.
# smitty cl_set_mndhb_response
- select the VG created in step 6
On loss of access Halt the node
Optional notification method []
Volume Group ccvg
Verify and Synchronize HACMP configuration.
# smitty cm_initialization_and_standard_config_menu_dmn
- select “Verify and Synchronize HACMP Configuration”
Enter Yes if prompted: “Would you like to import shared VG: ccvg, in resource group: onto node: to node: racha702 [Yes / No]:”
3.3) Sample configuration output


# /usr/es/sbin/cluster/utilities/cltopinfo
Cluster Name: rachacmp7
Cluster Connection Authentication Mode: Standard
Cluster Message Authentication Mode: None
Cluster Message Encryption: None
Use Persistent Labels for Communication: No
There are 4 node(s) and 5 network(s) defined
NODE racha701:
           Network net_diskhbmulti_01
                 racha701_1 /dev/lv01
           Network net_diskhbmulti_02
                 racha701_2 /dev/lv02
            Network net_diskhbmulti_03
                racha701_3 /dev/lv03
            Network net_ether_01
                racha701 XXX.XX.XXX.XX
            Network net_ether_02
                racha701p 192.168.10.71
NODE racha702:
            Network net_diskhbmulti_01
                racha702_2 /dev/lv01
            Network net_diskhbmulti_02
                racha702_3 /dev/lv02
            Network net_diskhbmulti_03
                racha702_4 /dev/lv03
            Network net_ether_01
                racha702 XXX.XX.XXX.XX
            Network net_ether_02
                racha702p 192.168.10.72
NODE racha703:
            Network net_diskhbmulti_01
                 racha703_3 /dev/lv01
            Network net_diskhbmulti_02
                 racha703_4 /dev/lv02
            Network net_diskhbmulti_03
                 racha703_5 /dev/lv03
            Network net_ether_01
                 racha703 XXX.XX.XXX.XX
            Network net_ether_02
                 racha703p 192.168.10.73
NODE racha704:
             Network net_diskhbmulti_01
                  racha704_4 /dev/lv01
             Network net_diskhbmulti_02
                  racha704_5 /dev/lv02
             Network net_diskhbmulti_03
                  racha704_6 /dev/lv03
             Network net_ether_01
                   racha704 XXX.XX.XXX.XX
             Network net_ether_02
                  racha704p 192.168.10.74

Resource Group shared_storage
              Startup Policy Online On All Available Nodes
              Fallover Policy Bring Offline (On Error Node Only)
              Fallback Policy Never Fallback
              Participating Nodes racha701 racha702 racha703 racha704

# /usr/es/sbin/cluster/sbin/cl_show_mndhb_vg
The following network is configured for Multi-Node Disk Heartbeat:

Network net_diskhbmulti_01
               Network net_diskhbmulti_01 uses Logical Volume [/dev/lv01] in
               Volume Group [ccvg] for heartbeat.
               Volume Group [ccvg] is managed by Resource Group [shared_storage]
               The following nodes participate in this network:
                          racha701
                          racha702
                          racha703
                          racha704
              On loss of access, HACMP will:
               Halt the node

The following network is configured for Multi-Node Disk Heartbeat:

Network net_diskhbmulti_02
               Network net_diskhbmulti_02 uses Logical Volume [/dev/lv02] in
               Volume Group [ccvg] for heartbeat.
               Volume Group [ccvg] is managed by Resource Group [shared_storage]
               The following nodes participate in this network:
                         racha701
                  bsp; racha702
                         racha703
                         racha704
                On loss of access, HACMP will:
                Halt the node

The following network is configured for Multi-Node Disk Heartbeat:

Network net_diskhbmulti_03
               Network net_diskhbmulti_03 uses Logical Volume [/dev/lv03] in
               Volume Group [ccvg] for heartbeat.
               Volume Group [ccvg] is managed by Resource Group [shared_storage]
               The following nodes participate in this network:
                         racha701
                         racha702
                         racha703
                         racha704
                 On loss of access, HACMP will:
                 Halt the node
Note : there are important considerations for installing future HACMP updates to a cluster with multi-node disk heart beat configured. The README file included with IZ01809 describes these considerations and the correct procedures to be used for future updates.
3.4) Upgrading an existing HACMP/RAC installation
Upgrading to HACMP 5.3 / Oracle RAC 10gr2 + patch can be accomplished by following these basic steps. NOTE: It is highly recommended that all databases and OCR be backed up prior to doing an upgrade.
1. Shut down all Oracle RAC Databases, all nodeapps and CRS on all nodes.
2. Disable CRS from starting on reboot:

#crsctl disable crs
3. Shut down HACMP on all nodes.
4. Install HACMP APAR IZ01809 Following the directions in the README included with that APAR.
5. Identify if the existing Voting Disk LVs are already on separate hdisks and if these each have space (at least 32MB) for the MNDHB LVs. If so, create a MNDHB LV on each of the hdisks. If not, then create new MNDHB LVs and new Voting Disk LVs, located on hdisks as described in section 3.2 step 8.
6. Verify and Synchronize HACMP configuration.
7. Start HACMP on all nodes.
8. Install Oracle RAC patch Bug 5497221 on each node by following the README included with the patch.
9. If new LVs for voting disks were added in step 5, then replace each of the existing voting disks with the new ones. Refer to metalink Note 452486.1 for how to change the voting disks.
10. Re-enable CRS using

#crsctl enable CRS
11. Start the CRS on all nodes and verify all resources start correctly.

4) Additional Documentation

README from HACMP APAR: IZ01809
README from Oracle patch 5497221  
HIGH AVAILABILITY CLUSTER MULTIPROCESSING BEST PRACTICES
HACMP 5.3 documentation: