Add a new node into the cluster!

Add a new node into the cluster!

分类： ORACLE11G 2011-06-22 11:13 98人阅读 评论(0) 举报

Prepare the account info:

mkdir /export/home/grid
chown -R grid:oinstall /export/home
/usr/sbin/useradd -g oinstall -G asmadmin,asmdba,asmoper -d /export/home/grid -m grid

/usr/sbin/useradd -g oinstall -G dba,asmdba -d /export/home/oracle -m oracle
set passwords
Set .profile for oracle:
bash-3.00$ more .profile
ORACLE_BASE=/export/oracle
ORACLE_HOME=/export/oracle/product/11.2.0/dbhome_1
ORACLE_SID=MCSPRD_1
#NLS_LANG=AMERICAN_AMERICA.ZHS16GKB
PATH=$ORACLE_HOME/bin:/bin:/sbin:/usr/ccs/bin:/usr/local/bin
TMPDIR=/tmp
LD_LIBRARY_PATH=$ORACLE_HOME/lib:$ORACLE_HOME/network/lib:/lib:/lib64:/usr/local/lib
export DISPLAY ORACLE_BASE ORACLE_HOME ORACLE_SID NLS_LANG PATH TMPDIR LD_LIBRARY_PATH
umask 022

Oracle11gR2 RAC: Adding Database Nodes to an Existing Cluster

Environment:

• Oracle RAC Database 11.2.0.1
• Oracle Grid Infrastructure 11.2.0.1
• Non-GNS

Adding Nodes to a RAC Administrator-Managed Database

Cloning to Extend an Oracle RAC Database (~20 minutes or less depending)

Phase I - Extending Oracle Clusterware to a new cluster node

1. Make physical connections, and install the OS.

2. Create Oracle accounts, and setup SSH among the new node and the existing cluster nodes.

3. Verify the requirements for cluster node addition using the Cluster Verification Utility (CVU). From an existing cluster node (as ‘grid’ user):

bash-3.00$ ./cluvfy stage -post hwos -n perf-5240-1,perf-5240-2,perf-5240-4-verbose

Performing post-checks for hardware and operating system setup

Checking node reachability...
^Cbash-3.00$ ./cluvfy stage -post hwos -n perf-5240-1,perf-5240-2,perf-5240-4 -verbose

Performing post-checks for hardware and operating system setup

Checking node reachability...

Check: Node reachability from node "perf-5240-1"
Destination Node Reachable?
------------------------------------ ------------------------
perf-5240-1 yes
perf-5240-2 yes
perf-5240-4 yes
Result: Node reachability check passed from node "perf-5240-1"


Checking user equivalence...

Check: User equivalence for user "grid"
Node Name Comment
------------------------------------ ------------------------
perf-5240-4 passed
perf-5240-2 passed
perf-5240-1 passed
Result: User equivalence check passed for user "grid"

Checking node connectivity...

Checking hosts config file...
Node Name Status Comment
------------ ------------------------ ------------------------
perf-5240-4 passed
perf-5240-2 passed
perf-5240-1 passed

Verification of the hosts config file successful


Interface information for node "perf-5240-4"
Name IP Address Subnet Gateway Def. Gateway HW Address MTU
------ --------------- --------------- --------------- --------------- ----------------- ------
nxge0 10.80.189.126 10.80.189.0 10.80.189.126 10.80.189.254 00:21:28:26:A5:9C 1500
nxge0 10.80.189.132 10.80.189.0 10.80.189.132 10.80.189.254 00:21:28:26:A5:9C 1500
nxge3 172.16.1.4 172.16.0.0 172.16.1.4 10.80.189.254 00:21:28:26:A5:9F 1500


Interface information for node "perf-5240-2"
Name IP Address Subnet Gateway Def. Gateway HW Address MTU
------ --------------- --------------- --------------- --------------- ----------------- ------
nxge0 10.80.189.29 10.80.189.0 10.80.189.29 10.80.189.254 00:14:4F:F3:97:94 1500
nxge0 10.80.189.177 10.80.189.0 10.80.189.29 10.80.189.254 00:14:4F:F3:97:94 1500
nxge0 10.80.189.30 10.80.189.0 10.80.189.29 10.80.189.254 00:14:4F:F3:97:94 1500
nxge3 172.16.1.2 172.16.0.0 172.16.1.2 10.80.189.254 00:14:4F:F3:97:97 1500


Interface information for node "perf-5240-1"
Name IP Address Subnet Gateway Def. Gateway HW Address MTU
------ --------------- --------------- --------------- --------------- ----------------- ------
nxge0 10.80.189.25 10.80.189.0 10.80.189.25 10.80.189.254 00:21:28:26:6D:70 1500
nxge0 10.80.189.26 10.80.189.0 10.80.189.25 10.80.189.254 00:21:28:26:6D:70 1500
nxge0 10.80.189.175 10.80.189.0 10.80.189.25 10.80.189.254 00:21:28:26:6D:70 1500
nxge0 10.80.189.176 10.80.189.0 10.80.189.25 10.80.189.254 00:21:28:26:6D:70 1500
nxge3 172.16.1.1 172.16.0.0 172.16.1.1 10.80.189.254 00:21:28:26:6D:73 1500


Check: Node connectivity of subnet "10.80.189.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
perf-5240-4:nxge0 perf-5240-4:nxge0 yes
perf-5240-4:nxge0 perf-5240-2:nxge0 yes
perf-5240-4:nxge0 perf-5240-2:nxge0 yes
perf-5240-4:nxge0 perf-5240-2:nxge0 yes
perf-5240-4:nxge0 perf-5240-1:nxge0 yes
perf-5240-4:nxge0 perf-5240-1:nxge0 yes
perf-5240-4:nxge0 perf-5240-1:nxge0 yes
perf-5240-4:nxge0 perf-5240-1:nxge0 yes
perf-5240-4:nxge0 perf-5240-2:nxge0 yes
perf-5240-4:nxge0 perf-5240-2:nxge0 yes
perf-5240-4:nxge0 perf-5240-2:nxge0 yes
perf-5240-4:nxge0 perf-5240-1:nxge0 yes
perf-5240-4:nxge0 perf-5240-1:nxge0 yes
perf-5240-4:nxge0 perf-5240-1:nxge0 yes
perf-5240-4:nxge0 perf-5240-1:nxge0 yes
perf-5240-2:nxge0 perf-5240-2:nxge0 yes
perf-5240-2:nxge0 perf-5240-2:nxge0 yes
perf-5240-2:nxge0 perf-5240-1:nxge0 yes
perf-5240-2:nxge0 perf-5240-1:nxge0 yes
perf-5240-2:nxge0 perf-5240-1:nxge0 yes
perf-5240-2:nxge0 perf-5240-1:nxge0 yes
perf-5240-2:nxge0 perf-5240-2:nxge0 yes
perf-5240-2:nxge0 perf-5240-1:nxge0 yes
perf-5240-2:nxge0 perf-5240-1:nxge0 yes
perf-5240-2:nxge0 perf-5240-1:nxge0 yes
perf-5240-2:nxge0 perf-5240-1:nxge0 yes
perf-5240-2:nxge0 perf-5240-1:nxge0 yes
perf-5240-2:nxge0 perf-5240-1:nxge0 yes
perf-5240-2:nxge0 perf-5240-1:nxge0 yes
perf-5240-2:nxge0 perf-5240-1:nxge0 yes
perf-5240-1:nxge0 perf-5240-1:nxge0 yes
perf-5240-1:nxge0 perf-5240-1:nxge0 yes
perf-5240-1:nxge0 perf-5240-1:nxge0 yes
perf-5240-1:nxge0 perf-5240-1:nxge0 yes
perf-5240-1:nxge0 perf-5240-1:nxge0 yes
perf-5240-1:nxge0 perf-5240-1:nxge0 yes
Result: Node connectivity passed for subnet "10.80.189.0" with node(s) perf-5240-4,perf-5240-2,perf-5240-1


Check: TCP connectivity of subnet "10.80.189.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
perf-5240-1:10.80.189.25 perf-5240-4:10.80.189.126 passed
perf-5240-1:10.80.189.25 perf-5240-4:10.80.189.132 passed
perf-5240-1:10.80.189.25 perf-5240-2:10.80.189.29 passed
perf-5240-1:10.80.189.25 perf-5240-2:10.80.189.177 passed
perf-5240-1:10.80.189.25 perf-5240-2:10.80.189.30 passed
perf-5240-1:10.80.189.25 perf-5240-1:10.80.189.26 passed
perf-5240-1:10.80.189.25 perf-5240-1:10.80.189.175 passed
perf-5240-1:10.80.189.25 perf-5240-1:10.80.189.176 passed
Result: TCP connectivity check passed for subnet "10.80.189.0"


Check: Node connectivity of subnet "172.16.0.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
perf-5240-4:nxge3 perf-5240-2:nxge3 yes
perf-5240-4:nxge3 perf-5240-1:nxge3 yes
perf-5240-2:nxge3 perf-5240-1:nxge3 yes
Result: Node connectivity passed for subnet "172.16.0.0" with node(s) perf-5240-4,perf-5240-2,perf-5240-1


Check: TCP connectivity of subnet "172.16.0.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
perf-5240-1:172.16.1.1 perf-5240-4:172.16.1.4 passed
perf-5240-1:172.16.1.1 perf-5240-2:172.16.1.2 passed
Result: TCP connectivity check passed for subnet "172.16.0.0"


Interfaces found on subnet "10.80.189.0" that are likely candidates for VIP are:
perf-5240-4 nxge0:10.80.189.126 nxge0:10.80.189.132
perf-5240-2 nxge0:10.80.189.29 nxge0:10.80.189.177 nxge0:10.80.189.30
perf-5240-1 nxge0:10.80.189.25 nxge0:10.80.189.26 nxge0:10.80.189.175 nxge0:10.80.189.176

Interfaces found on subnet "172.16.0.0" that are likely candidates for a private interconnect are:
perf-5240-4 nxge3:172.16.1.4
perf-5240-2 nxge3:172.16.1.2
perf-5240-1 nxge3:172.16.1.1

Result: Node connectivity check passed


Checking for multiple users with UID value 0
PRVF-4132 : Multiple users "root,droot" with UID "0" exist on "perf-5240-4".
PRVF-4132 : Multiple users "root,droot" with UID "0" exist on "perf-5240-2".
PRVF-4132 : Multiple users "root,droot" with UID "0" exist on "perf-5240-1".
Result: Check for multiple users with UID value 0 failed
## We can ignore this!
Post-check for hardware and operating system setup was unsuccessful on all the nodes.
4. Compare an existing node (reference node) with the new node(s) to be added (as ‘grid’ user):

./cluvfy comp peer -n perf-5240-1,perf-5240-2,perf-5240-4 -r 11gR2 -osdba dba -verbose

Verifying peer compatibility

Checking peer compatibility...

Peer comparison: Physical memory
Node Name Status
------------------------------------ ------------------------
perf-5240-4 63.7812GB (6.6879488E7KB)
perf-5240-2 63.7812GB (6.6879488E7KB)
perf-5240-1 63.7812GB (6.6879488E7KB)
COMMENT: Physical memory comparison completed

Peer comparison: Available memory
Node Name Status
------------------------------------ ------------------------
perf-5240-4 61.2963GB (6.4273832E7KB)
perf-5240-2 51.8987GB (5.4419728E7KB)
perf-5240-1 51.6288GB (5.4136704E7KB)
COMMENT: Available memory comparison completed

Peer comparison: Swap space
Node Name Status
------------------------------------ ------------------------
perf-5240-4 16.0038GB (1.678124E7KB)
perf-5240-2 16.0038GB (1.678124E7KB)
perf-5240-1 16.0038GB (1.678124E7KB)
COMMENT: Swap space comparison completed

Peer comparison: Free disk space for "/var/tmp/"
Node Name Status
------------------------------------ ------------------------
perf-5240-4 12.6794GB (1.3295304E7KB)
perf-5240-2 11.1506GB (1.1692214E7KB)
perf-5240-1 12.4865GB (1.3093035E7KB)
COMMENT: Free disk space comparison completed for "/var/tmp/"

Peer comparison: User existence for "grid"
Node Name Status
------------------------------------ ------------------------
perf-5240-4 grid(245150)
perf-5240-2 grid(244361)
perf-5240-1 grid(244361)
COMMENT: User existence for "grid" comparison completed

Peer comparison: Group existence for "oinstall"
Node Name Status
------------------------------------ ------------------------
perf-5240-4 oinstall(65000)
perf-5240-2 oinstall(65000)
perf-5240-1 oinstall(65000)
COMMENT: Group existence for "oinstall" comparison completed

Peer comparison: Group existence for "dba"
Node Name Status
------------------------------------ ------------------------
perf-5240-4 dba(65001)
perf-5240-2 dba(300)
perf-5240-1 dba(300)
COMMENT: Group existence for "dba" comparison completed

Peer comparison: Group membership for "grid" in "oinstall (Primary)"
Node Name Status
------------------------------------ ------------------------
perf-5240-4 yes
perf-5240-2 yes
perf-5240-1 yes
COMMENT: Group membership for "grid" in "oinstall (Primary)" comparison completed

Peer comparison: Group membership for "grid" in "dba"
Node Name Status
------------------------------------ ------------------------
perf-5240-4 no
perf-5240-2 yes
perf-5240-1 yes
COMMENT: Group membership for "grid" in "dba" comparison completed

Peer comparison: Run level
Node Name Status
------------------------------------ ------------------------
perf-5240-4 3
perf-5240-2 3
perf-5240-1 3
COMMENT: Run level comparison completed

Peer comparison: System architecture
Node Name Status
------------------------------------ ------------------------
perf-5240-4 64-bit sparcv9 kernel modules
perf-5240-2 64-bit sparcv9 kernel modules
perf-5240-1 64-bit sparcv9 kernel modules
COMMENT: System architecture comparison completed

Peer comparison: Kernel version
Node Name Status
------------------------------------ ------------------------
perf-5240-4 5.10-2010.08.11.10.56
perf-5240-2 5.10-2010.08.11.10.56
perf-5240-1 5.10-2010.08.11.10.56
COMMENT: Kernel version comparison completed

Peer comparison: Kernel param "project.max-sem-ids"
Node Name Status
------------------------------------ ------------------------
perf-5240-4 128
perf-5240-2 128
perf-5240-1 128
COMMENT: Kernel param "project.max-sem-ids" comparison completed

Peer comparison: Kernel param "process.max-sem-nsems"
Node Name Status
------------------------------------ ------------------------
perf-5240-4 512
perf-5240-2 512
perf-5240-1 512
COMMENT: Kernel param "process.max-sem-nsems" comparison completed

Peer comparison: Kernel param "project.max-shm-memory"
Node Name Status
------------------------------------ ------------------------
perf-5240-4 16732571648
perf-5240-2 16732559360
perf-5240-1 429496729600
COMMENT: Kernel param "project.max-shm-memory" comparison completed

Peer comparison: Kernel param "project.max-shm-ids"
Node Name Status
------------------------------------ ------------------------
perf-5240-4 128
perf-5240-2 128
perf-5240-1 128
COMMENT: Kernel param "project.max-shm-ids" comparison completed

Peer comparison: Package existence for "SUNWarc"
Node Name Status
------------------------------------ ------------------------
perf-5240-4 SUNWarc-11.10.0-2005.01.21.15.53
perf-5240-2 SUNWarc-11.10.0-2005.01.21.15.53
perf-5240-1 SUNWarc-11.10.0-2005.01.21.15.53
COMMENT: Package existence for "SUNWarc" comparison completed

Peer comparison: Package existence for "SUNWbtool"
Node Name Status
------------------------------------ ------------------------
perf-5240-4 SUNWbtool-11.10.0-2005.01.21.15.53
perf-5240-2 SUNWbtool-11.10.0-2005.01.21.15.53
perf-5240-1 SUNWbtool-11.10.0-2005.01.21.15.53
COMMENT: Package existence for "SUNWbtool" comparison completed

Peer comparison: Package existence for "SUNWhea"
Node Name Status
------------------------------------ ------------------------
perf-5240-4 SUNWhea-11.10.0-2005.01.21.15.53
perf-5240-2 SUNWhea-11.10.0-2005.01.21.15.53
perf-5240-1 SUNWhea-11.10.0-2005.01.21.15.53
COMMENT: Package existence for "SUNWhea" comparison completed

Peer comparison: Package existence for "SUNWlibm"
Node Name Status
------------------------------------ ------------------------
perf-5240-4 SUNWlibm-5.10-2004.12.18
perf-5240-2 SUNWlibm-5.10-2004.12.18
perf-5240-1 SUNWlibm-5.10-2004.12.18
COMMENT: Package existence for "SUNWlibm" comparison completed

Peer comparison: Package existence for "SUNWlibms"
Node Name Status
------------------------------------ ------------------------
perf-5240-4 SUNWlibms-5.10-2004.11.23
perf-5240-2 SUNWlibms-5.10-2004.11.23
perf-5240-1 SUNWlibms-5.10-2004.11.23
COMMENT: Package existence for "SUNWlibms" comparison completed

Peer comparison: Package existence for "SUNWsprot"
Node Name Status
------------------------------------ ------------------------
perf-5240-4 SUNWsprot-5.10-2004.11.23
perf-5240-2 SUNWsprot-5.10-2004.11.23
perf-5240-1 SUNWsprot-5.10-2004.11.23
COMMENT: Package existence for "SUNWsprot" comparison completed

Peer comparison: Package existence for "SUNWtoo"
Node Name Status
------------------------------------ ------------------------
perf-5240-4 SUNWtoo-11.10.0-2005.01.21.15.53
perf-5240-2 SUNWtoo-11.10.0-2005.01.21.15.53
perf-5240-1 SUNWtoo-11.10.0-2005.01.21.15.53
COMMENT: Package existence for "SUNWtoo" comparison completed

Peer comparison: Package existence for "SUNWi1of"
Node Name Status
------------------------------------ ------------------------
perf-5240-4 SUNWi1of-6.6.2.7400-0.2004.12.15
perf-5240-2 SUNWi1of-6.6.2.7400-0.2004.12.15
perf-5240-1 SUNWi1of-6.6.2.7400-0.2004.12.15
COMMENT: Package existence for "SUNWi1of" comparison completed

Peer comparison: Package existence for "SUNWi1cs"
Node Name Status
------------------------------------ ------------------------
perf-5240-4 SUNWi1cs-2.0-2004.08.12.10.14
perf-5240-2 SUNWi1cs-2.0-2004.08.12.10.14
perf-5240-1 SUNWi1cs-2.0-2004.08.12.10.14
COMMENT: Package existence for "SUNWi1cs" comparison completed

Peer comparison: Package existence for "SUNWi15cs"
Node Name Status
------------------------------------ ------------------------
perf-5240-4 SUNWi15cs-2.0-2004.08.23.14.43
perf-5240-2 SUNWi15cs-2.0-2004.08.23.14.43
perf-5240-1 SUNWi15cs-2.0-2004.08.23.14.43
COMMENT: Package existence for "SUNWi15cs" comparison completed

Peer comparison: Package existence for "SUNWxwfnt"
Node Name Status
------------------------------------ ------------------------
perf-5240-4 SUNWxwfnt-6.6.2.7400-0.2004.12.15
perf-5240-2 SUNWxwfnt-6.6.2.7400-0.2004.12.15
perf-5240-1 SUNWxwfnt-6.6.2.7400-0.2004.12.15
COMMENT: Package existence for "SUNWxwfnt" comparison completed

Peer comparison: Package existence for "SUNWlibC"
Node Name Status
------------------------------------ ------------------------
perf-5240-4 SUNWlibC-5.10-2004.12.22
perf-5240-2 SUNWlibC-5.10-2004.12.22
perf-5240-1 SUNWlibC-5.10-2004.12.22
COMMENT: Package existence for "SUNWlibC" comparison completed

Peer comparison: Package existence for "SUNWcsl"
Node Name Status
------------------------------------ ------------------------
perf-5240-4 SUNWcsl-11.10.0-2005.01.21.15.53
perf-5240-2 SUNWcsl-11.10.0-2005.01.21.15.53
perf-5240-1 SUNWcsl-11.10.0-2005.01.21.15.53
COMMENT: Package existence for "SUNWcsl" comparison completed

Verification of peer compatibility was successful.
5. Verify the integrity of the cluster and new node by running from an existing cluster node (as ‘grid’ user):

./cluvfy stage -pre nodeadd -n perf-5240-1,perf-5240-2,perf-5240-4 -fixup -verbose >>pre.txt

6. Add the new node by running the following from an existing cluster node (as ‘grid’ user):

Not using GNS

GI_HOME/oui/bin/addNode.sh “CLUSTER_NEW_NODES={perf-5240-4.central.sun.com}” “CLUSTER_NEW_VIRTUAL_HOSTNAMES={perf4-vip.central.sun.com}“
 then run the root.sh on the new node to start up the crs.