分类： Oracle
2008-04-22 17:05:35
前提：新添加节点与已有节点的os及patch号及Oracle 所需的所有补丁一致，私有及公有网络均相同，共享同一存储,新节点能和已存在的节点访问同一共享磁盘。
环境：已有lxsfrac05,lxsfrac06构成rac cluster环境，要添加一个lxsfrac08节点到cluster中。
总的步骤可分为两部分：
一）按照sfrac文档在新节点上安装sfrac，并配置好网络
二）按照oracle文档在新节点上完成添加节点操作
一）按照sfrac文档（实际上是vcs文档）在新节点上安装sfrac，并配置好网络
1）在新节点上安装sfrac软件
# installsfrac -installonly –rsh (只装软件)
2) 在新节点上启动VXVM
root@lxsfrac08 # vxdisk list
VxVM vxdisk ERROR V-5-1-684 IPC failure: Configuration daemon is not accessible
root@lxsfrac08 # vxinstall
Please consult VxVM Release Notes to determine if a license key is necessary
for your system. If it is, please use the information below to request a
key. Installation will not be successful if a license key is required and
not present.
VxVM uses license keys to control access. If you have a SPARCstorage
Array (SSA) controller or a Sun Enterprise Network Array (SENA) controller
attached to your system, then VxVM will grant you a limited use license
automatically. The SSA and/or SENA license grants you unrestricted use
of disks attached to an SSA or SENA controller, but disallows striping,
RAID-5, and DMP on non-SSA and non-SENA disks. If you are not running an
SSA or SENA controller, then you must obtain a license key to operate.
Licensing information:
System host ID: 16571a1e
Host type: i86pc
SPARCstorage Array or Sun Enterprise Network Array: No arrays found
Some licenses are already installed. Do you wish to review them
[y,n,q,?] (default: y) n
Do you wish to enter another license key [y,n,q,?] (default: n) n
Do you want to use enclosure based names for all disks ?
[y,n,q,?] (default: n) y
Starting the relocation daemon, vxrelocd.
Starting the auto site reattach daemon, vxsited.
Starting the cache deamon, vxcached.
Starting the diskgroup config backup daemon, vxconfigbackupd.
Starting the dg monitoring daemon for rlinks with STORAGE protocol, vxvvrsecdgd.
Do you want to setup a system wide default disk group?
[y,n,q,?] (default: y) n
The installation is successfully completed.
root@lxsfrac08 # vxdisk list
DEVICE TYPE DISK GROUP STATUS
AMS_WMS0_0 auto:cdsdisk - (oracle_dg_lxsfrac06_lxsfrac05 online shared
AMS_WMS0_1 auto:cdsdisk - (vxfencoorddg) online
AMS_WMS0_2 auto:cdsdisk - (vxfencoorddg) online
AMS_WMS0_3 auto:cdsdisk - - online
AMS_WMS0_4 auto:cdsdisk - (oracledg) online
AMS_WMS0_5 auto:cdsdisk - - online
AMS_WMS0_6 auto:cdsdisk - (oracle_dg_lxsfrac06_lxsfrac05 online shared
AMS_WMS0_7 auto:cdsdisk - (vxfencoorddg) online
AMS_WMS0_8 auto - - error
AMS_WMS0_9 auto:cdsdisk - - online
AMS_WMS0_10 auto:cdsdisk - (oracle_dg_lxsfrac06_lxsfrac05 online shared
AMS_WMS0_11 auto:cdsdisk - - online
Disk_0 auto:none - - online invalid
3)配置LLT,GAB,VCSMM和VXFEN
3.1)修改/etc/system
添加安装oracle rac需要的参数(生效需要reboot)
3.2)修改一个已存在的节点的/etc/hosts并将其copy到其他的节点（包含新添加的节点）
root@lxsfrac08 # more /etc/hosts
::1 localhost
127.0.0.1 localhost
10.198.90.36 lxsfrac06
10.198.90.35 lxsfrac05
10.198.90.73 lxsfrac08 loghost
#Priv
192.168.2.35 lxsfrac05-priv
192.168.2.36 lxsfrac06-priv
192.168.2.73 lxsfrac08-priv
#VIP
10.198.94.142 lxsfrac05-vip
10.198.94.143 lxsfrac06-vip
10.198.94.145 lxsfrac08-vip
3.3)修改/etc/llthosts
root@lxsfrac05 # vi /etc/llthosts
0 lxsfrac06
1 lxsfrac05
2 lxsfrac08
3.4)在新节点修改/etc/llttab文件
root@lxsfrac08 # vi /etc/llttab
set-node lxsfrac08
set-cluster 99
link e1000g0 /dev/e1000g:0 - ether - -
link e1000g1 /dev/e1000g:1 - ether - -
3.5）在新节点上创建/etc/gabtab，并修改已存在节点的
/sbin/gabconfig -c –n3
N后面的数字表示节点数（要算上新节点）
注意： 如果是在单个节点的cluster里添加新节点，则必须启用fencing
3.6）往新节点里复制 /etc/vcsmmtab和 /etc/vxfendg root@lxsfrac08 # # scp lxsfrac05:/etc/vcsmmtab /etc
root@lxsfrac08 # # scp lxsfrac05:/etc/vxfendg /etc
----未启用vxfencing不需要做这一步
3.7） 在新节点上启动LLT 和GAB
root@lxsfrac08 # /etc/init.d/llt start
root@lxsfrac08 # /etc/init.d/gab start
3.8）往新节点里复制 /etc/vxfenmode
# scp lxsfrac05: /etc/vxfenmode /etc
3.9 ） 在新节点上启动 ODM
root@lxsfrac08 # /etc/init.d/odm start
此时新节点上应该能启动a,b,d端口。
3．10）在新节点上启动 vcsmm (o端口)
root@lxsfrac08 # /etc/init.d/vcsmm start
3.11 ） 在新节点上检查已启动的端口，确保a,b,d,o已启动
root@lxsfrac08 # /sbin/gabconfig -a
为安装oracle在新节点上的准备
1 ）在新节点上准备2个ip ,一个公有，一个私有
2 ）在新节点上创建oracle用户，组，重设口令
3 ）在新节点上创建ocr,voting disk的挂载点，设置权限
4 ）在老节点上执行如下命令以认出新节点
# /opt/VRTS/bin/vxclustadm -m vcs -t gab reinit
5 ）重起新节点vcs会将master的配置信息传到新节点
Reboot 后会多出H端口
root@lxsfrac01 # gabconfig -a
======================================================
Port a gen 465507 membership 012
Port b gen 465506 membership 012
Port d gen 465505 membership 012
Port h gen 465512 membership 012
Port o gen 465504 membership 012
6 ）在master节点上关闭原有集群所有的vcs,修改配置文件
root@lxsfrac04 # hastop -all
root@lxsfrac04 # vi main.cf
"main.cf" 102 lines, 2395 characters
Q
include "types.cf"
include "CFSTypes.cf"
include "CVMTypes.cf"
include "MultiPrivNIC.cf"
include "OracleTypes.cf"
include "PrivNIC.cf"
cluster testrac (
UserNames = { admin = ImnFmhMjnInnLvnHmk }
Administrators = { admin }
HacliUserLevel = COMMANDROOT
)
system lxsfrac03 (
)
system lxsfrac04 (
)
system lxsfrac01 (
)
group cvm (
SystemList = { lxsfrac03 = 0, lxsfrac04 = 1,lxsfrac01 = 2 }
AutoFailOver = 0
Parallel = 1
AutoStartList = { lxsfrac03, lxsfrac04, lxsfrac01 }
)
CFSMount oracle_cfs (
Critical = 0
MountPoint @lxsfrac03 = "/oracle"
MountPoint @lxsfrac04 = "/oracle"
MountPoint @lxsfrac01 = "/oracle"
BlockDevice @lxsfrac03 = "/dev/vx/dsk/oracle_dg_lxsfrac04_lxsfrac03/oracle_vol"
BlockDevice @lxsfrac04 = "/dev/vx/dsk/oracle_dg_lxsfrac04_lxsfrac03/oracle_vol"
BlockDevice @lxsfrac01 = "/dev/vx/dsk/oracle_dg_lxsfrac04_lxsfrac03/oracle_vol"
)
CFSfsckd vxfsckd (
)
CVMCluster cvm_clus (
CVMClustName = testrac
CVMNodeId = { lxsfrac01 = 2, lxsfrac03 = 1, lxsfrac04 = 0 }
CVMTransport = gab
CVMTimeout = 200
)
CVMVolDg oracle_vol (
Critical = 0
CVMDiskGroup @lxsfrac03 = oracle_dg_lxsfrac04_lxsfrac03
CVMDiskGroup @lxsfrac04 = oracle_dg_lxsfrac04_lxsfrac03
CVMDiskGroup @lxsfrac01 = oracle_dg_lxsfrac04_lxsfrac03
CVMVolume @lxsfrac03 = { oracle_vol }
CVMVolume @lxsfrac04 = { oracle_vol }
CVMVolume @lxsfrac01 = { oracle_vol }
CVMActivation @lxsfrac03 = sw
CVMActivation @lxsfrac04 = sw
CVMActivation @lxsfrac01 = sw
)
CVMVxconfigd cvm_vxconfigd (
Critical = 0
CVMVxconfigdArgs = { syslog }
)
PrivNIC ora_priv (
Critical = 0
Device = { e1000g1 = 0, e1000g3 = 1 }
Address @lxsfrac03 = "192.168.0.138"
Address @lxsfrac04 = "192.168.0.137"
Address @lxsfrac01 = "192.168.0.122"
NetMask = "255.255.255.0"
)
cvm_clus requires cvm_vxconfigd
oracle_cfs requires oracle_vol
oracle_vol requires vxfsckd
vxfsckd requires cvm_clus
// resource dependency tree
//
// group cvm
// {
// PrivNIC ora_priv
// CFSMount oracle_cfs
// {
// CVMVolDg oracle_vol
// {
// CFSfsckd vxfsckd
// {
// CVMCluster cvm_clus
// {
// CVMVxconfigd cvm_vxconfigd
// }
// }
// }
// }
// }
检查配置文件
root@lxsfrac04 # hscf –verify .
将lxsfrac04的配置文件复制到lxsfrac03,lxsfrac01
root@lxsfrac04 # rcp main.cf lxsfrac03:/etc/VRTSvcs/conf/config/
root@lxsfrac04 # rcp main.cf lxsfrac01:/etc/VRTSvcs/conf/config/
7）启动lxsfrac04的vcs
root@lxsfrac04 # hastart
8）检查状态
root@lxsfrac04 # hastatus
9） 在lxsfrac04上执行如下命令以确保能认出新节点
root@lxsfrac04 # /etc/vx/bin/vxclustadm -m vcs -t gab reinit
root@lxsfrac04 # /etc/vx/bin/vxclustadm nidmap
Name CVM Nid CM Nid State
lxsfrac03 1 1 Out of Cluster
lxsfrac04 0 0 Joined: Master
lxsfrac01 2 2 Out of Cluster
10）在老节点lxsfrac03上也运行7-9步
root@lxsfrac03 # hastart
root@lxsfrac03 # hastatus
root@lxsfrac03 # /etc/vx/bin/vxclustadm -m vcs -t gab reinit
root@lxsfrac03 # /etc/vx/bin/vxclustadm nidmap
Name CVM Nid CM Nid State
lxsfrac03 1 1 Joined: Slave
lxsfrac04 0 0 Joined: Master
lxsfrac01 2 2 Out of Cluster
11） 在新节点上检查gab端口
Lxsfrac01# gabconfig -a
启动vcs
1)在启动vcs时如果此时f,u,v,w端口已启动，启动主机来重起vcs。否则直接执行hastart来启动。
12 ) 检查此时vcs状态
# hastatus -sum
root@lxsfrac01#/etc/vx/bin/vxclustadm nidmap
Name CVM Nid CM Nid State
lxsfrac03 1 1 Joined: Slave
lxsfrac04 0 0 Joined: Master
lxsfrac01 2 2 Joined: Slave
root@lxsfrac03#/etc/vx/bin/vxclustadm nidmap
Name CVM Nid CM Nid State
lxsfrac03 1 1 Joined: Slave
lxsfrac04 0 0 Joined: Master
lxsfrac01 2 2 Out of Cluster
root@lxsfrac04 #/etc/vx/bin/vxclustadm nidmap
Name CVM Nid CM Nid State
lxsfrac03 1 1 Joined: Slave
lxsfrac04 0 0 Joined: Master
lxsfrac01 2 2 Joined: Slave
二）按照oracle文档完成添加新节点操作，主要包括
1. 在新节点上配置os及硬件
2. 将新节点添加到 Oracle 集群中
3. 在新节点上安装rac软件
4. 为新节点重配listener
5. 在新节点上用dbca创建新实例
2.1) 在已存在的节点(lxsfrac05或lxsfrac06) 上以oracle用户执行$crs_home/oui/bin/addNode.sh
oracle@lxsfrac05 # /oracle/crs/oui/bin/addNode.sh
下面的新加节点没出现就手工输入,会提示执行下面脚本
root@lxsfrac05 # /oracle/crs/install/rootaddnode.sh
clscfg: EXISTING configuration version 3 detected.
clscfg: version 3 is 10G Release 2.
Attempting to add 1 new nodes to the configuration
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node
node 2: lxsfrac08 lxsfrac08-priv lxsfrac08
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
/oracle/crs/bin/srvctl add nodeapps -n lxsfrac08 -A lxsfrac08-vip/255.255.248.0/bge0 -o /oracle/crs
shell-init: error retrieving current directory: getcwd: cannot access parent directories: Permission denied
Error occurred during initialization of VM
java.lang.Error: Properties init: Could not determine current working directory.
出错原因是因为当前目录不对。
root@lxsfrac05 # pwd
/etc/VRTSvcs/conf/config
root@lxsfrac05 # cd /
root@lxsfrac05 # /oracle/crs/install/rootaddnode.sh
clscfg: EXISTING configuration version 3 detected.
clscfg: version 3 is 10G Release 2.
Node lxsfrac08 is already assigned nodenum 2.
Aborting: No configuration data has been changed.
clscfg -add -nn nameA,numA,nameB,numB,... -pn privA,numA,privB,numB,...
[-hn hostA,numA,hostB,numB,...] [-t p1,p2,p3,p4]
-nn specifies nodenames in the same fashion as -nn in -install mode
-pn specifies private interconnect names as -pn in -install mode
-hn specifies hostnames in the same fashion as -hn in -install mode
-t specifies port numbers to be used by CRS daemons on the new node(s)
default ports: 49895,49896,49897,49898
WARNING: Using this tool may corrupt your cluster configuration. Do not
use unless you positively know what you are doing.
/oracle/crs/bin/srvctl add nodeapps -n lxsfrac08 -A lxsfrac08-vip/255.255.248.0/bge0 -o /oracle/crs
root@lxsfrac08 # /oracle/crs/root.sh
WARNING: directory '/oracle' is not owned by root
Checking to see if Oracle CRS stack is already configured
Checking to see if any 9i GSD is up
OCR LOCATIONS = /oracle/ocrcfg1
Setting the permissions on OCR backup directory
Setting up NS directories
Oracle Cluster Registry configuration upgraded successfully
WARNING: directory '/oracle' is not owned by root
clscfg: EXISTING configuration version 3 detected.
clscfg: version 3 is 10G Release 2.
Successfully accumulated necessary OCR keys.
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node
node 0: lxsfrac06 lxsfrac06-priv lxsfrac06
node 1: lxsfrac05 lxsfrac05-priv lxsfrac05
clscfg: Arguments check out successfully.
NO KEYS WERE WRITTEN. Supply -force parameter to override.
-force is destructive and will destroy any previous cluster
configuration.
Oracle Cluster Registry for cluster has already been initialized
Startup will be queued to init within 30 seconds.
Adding daemons to inittab
Expecting the CRS daemons to be up within 600 seconds.
CSS is active on these nodes.
lxsfrac06
lxsfrac05
lxsfrac08
CSS is active on all nodes.
Waiting for the Oracle CRSD and EVMD to start
Oracle CRS stack installed and running under init(1M)
Running vipca(silent) for configuring nodeapps
Creating VIP application resource on (0) nodes.
Creating GSD application resource on (0) nodes.
Creating ONS application resource on (0) nodes.
Starting VIP application resource on (2) nodes...
Starting GSD application resource on (2) nodes...
Starting ONS application resource on (2) nodes...
执行完后，看资源会添加红色的部分
root@lxsfrac06 # crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora....05.lsnr application ONLINE ONLINE lxsfrac05
ora....c05.gsd application ONLINE ONLINE lxsfrac05
ora....c05.ons application ONLINE ONLINE lxsfrac05
ora....c05.vip application ONLINE ONLINE lxsfrac05
ora....06.lsnr application ONLINE ONLINE lxsfrac06
ora....c06.gsd application ONLINE ONLINE lxsfrac06
ora....c06.ons application ONLINE ONLINE lxsfrac06
ora....c06.vip application ONLINE ONLINE lxsfrac06
ora....c08.gsd application ONLINE ONLINE lxsfrac08
ora....c08.ons application ONLINE ONLINE lxsfrac08
ora....c08.vip application ONLINE ONLINE lxsfrac08
2.2) 在已存在的节点(lxsfrac05或lxsfrac06) 上以oracle用户执行$oracle_home/oui/bin/addNode.sh
[oracle@lxsfrac05] $/oracle/10g/oui/bin/addNode.sh
Starting Oracle Universal Installer...
Checking installer requirements...
Checking operating system version: must be 5.10.Actual5.10 Passed
Checking Temp space: must be greater than 250 MB.Actual 4971MB Passed
Checking swap space: must be greater than 500 MB.Actual 6950 MB Passed
Checking monitor: must be configured to display at least 256 colors. Actual 65536 Passed
All installer requirements met.
Oracle Universal Installer, Version 10.2.0.1.0 Production
Copyright (C) 1999, 2005, Oracle. All rights reserved.
Warning: Cannot convert string "-monotype-arial-regular-r-normal--*-140-*-*-p-*-iso8859-1" to type FontStruct
按照提示在新节点上执行脚本
root@lxsfrac08 # /oracle/10g/root.sh
Running Oracle10 root.sh script...
The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /oracle/10g
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The file "dbhome" already exists in /usr/local/bin. Overwrite it? (y/n) [n]: y
Copying dbhome to /usr/local/bin ...
The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n) [n]: y
Copying oraenv to /usr/local/bin ...
The file "coraenv" already exists in /usr/local/bin. Overwrite it? (y/n) [n]: y
Copying coraenv to /usr/local/bin ...
Entries will be added to the /var/opt/oracle/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
root@lxsfrac05 # export DISPLAY=10.198.90.55:5
root@lxsfrac05 # /oracle/crs/bin/vipca -nodelist lxsfrac05,lxsfrac06,lxsfrac08
Warning: Cannot convert string "-monotype-arial-regular-r-normal--*-140-*-*-p-*-iso8859-1" to type FontStruct
2．3为新节点重配listener
[oracle@lxsfrac08] $./netca
Oracle Net Services Configuration:
Warning: Cannot convert string "-monotype-arial-regular-r-normal--*-140-*-*-p-*-iso8859-1" to type FontStruct
Configuring Listener:LISTENER
lxsfrac06...
lxsfrac05...
lxsfrac08...netca
Listener configuration complete.
Oracle Net Services configuration terminated by user.The exit code is -1
会发现新加了一个新节点的listener资源。
root@lxsfrac06 # crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora....05.lsnr application ONLINE ONLINE lxsfrac05
ora....c05.gsd application ONLINE ONLINE lxsfrac05
ora....c05.ons application ONLINE ONLINE lxsfrac05
ora....c05.vip application ONLINE ONLINE lxsfrac05
ora....06.lsnr application ONLINE ONLINE lxsfrac06
ora....c06.gsd application ONLINE ONLINE lxsfrac06
ora....c06.ons application ONLINE ONLINE lxsfrac06
ora....c06.vip application ONLINE ONLINE lxsfrac06
ora....08.lsnr application ONLINE ONLINE lxsfrac08
ora....c08.gsd application ONLINE ONLINE lxsfrac08
ora....c08.ons application ONLINE ONLINE lxsfrac08
ora....c08.vip application ONLINE ONLINE lxsfrac08
如果新添嘉的listener是offline，启动之
root@lxsfrac08#/oracle/crs/bin/srvctl start nodeapps -n lxsfrac08
lxsfrac08:ora.lxsfrac08.LISTENER_LXSFRAC08.lsnr:
lxsfrac08:ora.lxsfrac08.LISTENER_LXSFRAC08.lsnr:LSNRCTL for Solaris: Version 10.2.0.1.0 - Production on 10-APR-2008 15:12:42
lxsfrac08:ora.lxsfrac08.LISTENER_LXSFRAC08.lsnr:
lxsfrac08:ora.lxsfrac08.LISTENER_LXSFRAC08.lsnr:Copyright (c) 1991, 2005, Oracle. All rights reserved.
lxsfrac08:ora.lxsfrac08.LISTENER_LXSFRAC08.lsnr:
lxsfrac08:ora.lxsfrac08.LISTENER_LXSFRAC08.lsnr:Starting /oracle/10g/bin/tnslsnr: please wait...
lxsfrac08:ora.lxsfrac08.LISTENER_LXSFRAC08.lsnr:
lxsfrac08:ora.lxsfrac08.LISTENER_LXSFRAC08.lsnr:TNSLSNR for Solaris: Version 10.2.0.1.0 - Production
lxsfrac08:ora.lxsfrac08.LISTENER_LXSFRAC08.lsnr:System parameter file is /oracle/10g/network/admin/listener.ora
lxsfrac08:ora.lxsfrac08.LISTENER_LXSFRAC08.lsnr:Log messages written to /oracle/10g/network/log/listener_lxsfrac08.log
lxsfrac08:ora.lxsfrac08.LISTENER_LXSFRAC08.lsnr:TNS-01151: Missing listener name, LISTENER_LXSFRAC08, in LISTENER.ORA
lxsfrac08:ora.lxsfrac08.LISTENER_LXSFRAC08.lsnr:
lxsfrac08:ora.lxsfrac08.LISTENER_LXSFRAC08.lsnr:Listener failed to start. See the error message(s) above...
lxsfrac08:ora.lxsfrac08.LISTENER_LXSFRAC08.lsnr:
lxsfrac08:ora.lxsfrac08.LISTENER_LXSFRAC08.lsnr:
lxsfrac08:ora.lxsfrac08.LISTENER_LXSFRAC08.lsnr:LSNRCTL for Solaris: Version 10.2.0.1.0 - Production on 10-APR-2008 15:12:43
lxsfrac08:ora.lxsfrac08.LISTENER_LXSFRAC08.lsnr:
lxsfrac08:ora.lxsfrac08.LISTENER_LXSFRAC08.lsnr:Copyright (c) 1991, 2005, Oracle. All rights reserved.
lxsfrac08:ora.lxsfrac08.LISTENER_LXSFRAC08.lsnr:
lxsfrac08:ora.lxsfrac08.LISTENER_LXSFRAC08.lsnr:TNS-01101: Could not find service name LISTENER_LXSFRAC08
CRS-0215: Could not start resource 'ora.lxsfrac08.LISTENER_LXSFRAC08.lsnr'.
2.4在新节点上用dbca创建新实例
Dbca前要确保
/oracle/oraInventory/ContentsXML/inventory.xml和
/oracle/10g/inventory/ContentsXML/oraclehomeproperties.xml
都有各节点的名字,如果没有的话就手工添加，否则让你选择在主机上新添加实例时无法找到新节点。
