对Oracle数据库进行容灾(2)

Veritas Volume Replicator VVR的配置
1. 两个节点上都需要做：
修改两个节点的/etc/hosts
10.198.90.122 lxsfrac01
10.198.90.139 lxsfrac02
192.168.0.122 lxsfrac01-priv
192.168.0.139 lxsfrac02-priv

分别创建私有网络(专有)，建议数据复制使用专用网络。

root@lxsfrac01 # ifconfig e1000g1 plumb 192.168.0.122 netmask 255.255.255.0 up
root@lxsfrac02 # ifconfig e1000g1 plumb 192.168.0.139 netmask 255.255.255.0 up
root@lxsfrac01 # ping 192.168.0.139
192.168.0.139 is alive

创建相同名称的磁盘组（含有最少两个磁盘），相同名称数据卷oravol，创建复制日志卷(SRL)，
分别在本地安装Oracle 10gR2.(安装Oracle过程略。)
root@lxsfrac01 # vxdg init oradg AMS_WMS0_2 AMS_WMS0_3
root@lxsfrac01 # vxassist -g oradg make oravol 10g
root@lxsfrac01 # vxedit -g oradg set user=oracle group=dba mode=644 oravol
root@lxsfrac01 # vxassist -g oradg make srl 800m

2．在备用节点上修改配置文件/etc/vx/vras/.rdg，让其包含主节点上的disk group ID
（这里是1193985774.38.lxsfrac01）。当创建备用RVG时，这个文件用来对主节点的授权。
可以通过vxprint –l来得到disk group ID
root@lxsfrac02 # vi /etc/vx/vras/.rdg
………
1193985774.38.lxsfrac01

"/etc/vx/vras/.rdg" 43 lines, 1544 characters

3．在主节点上：
创建vxfs文件系统
mount -F vxfs -o largefiles /dev/vx/dsk/oradg/oravol /oracle
chown oracle:dba /oracle
切换到oracle用户使用dbca建库，并且所有数据文件选择/oracle目录
创建Primary RVG：
root@lxsfrac01 # vradmin -g oradg createpri ora_rvg oravol srl
其中的oravol是要复制的数据卷列表，当多于一个时，用逗号隔开。

创建Secondary RVG：
root@lxsfrac01 # vradmin -g oradg addsec ora_rvg lxsfrac01-priv lxsfrac02-priv

设置复制方式为异步：
root@lxsfrac01 # vradmin -g oradg set ora_rvg lxsfrac02-priv synchronous=off

开始复制：
root@lxsfrac01 # vradmin -g oradg -a startrep ora_rvg
Message from Primary:
VxVM VVR vxrlink WARNING V-5-1-3359 Attaching rlink to non-empty rvg. Autosync will be performed.
VxVM VVR vxrlink INFO V-5-1-3614 Secondary data volumes detected with rvg ora_rvg as parent:
VxVM VVR vxrlink INFO V-5-1-6183 oravol: len=20971520 primary_datavol=oravol
VxVM VVR vxrlink INFO V-5-1-3365 Autosync operation has started
root@lxsfrac01 #
root@lxsfrac01 # vradmin -g oradg -f startrep ora_rvg
VxVM VVR vradmin INFO V-5-52-1203 Replication to Secondary host lxsfrac02-priv already started.


检查相关信息和状态，显示数据复制过程，可以通过使用-i参数指定interval
root@lxsfrac01 # vxrvg -g oradg getrlinks ora_rvg
rlk_lxsfrac02-priv_ora_rvg

root@lxsfrac01 # vxrlink -g oradg status rlk_lxsfrac02-priv_ora_rvg

Fri Nov 02 17:01:22 2007
VxVM VVR vxrlink INFO V-5-1-4464 Rlink rlk_lxsfrac02-priv_ora_rvg is in AUTOSYNC. 9888256 Kbytes remaining.
root@lxsfrac01 # vxrlink -g oradg status rlk_lxsfrac02-priv_ora_rvg

Fri Nov 02 17:02:08 2007
VxVM VVR vxrlink INFO V-5-1-4464 Rlink rlk_lxsfrac02-priv_ora_rvg is in AUTOSYNC. 9375488 Kbytes remaining.
root@lxsfrac01 # vxrlink -g oradg status rlk_lxsfrac02-priv_ora_rvg

Fri Nov 02 17:13:26 2007
VxVM VVR vxrlink INFO V-5-1-4464 Rlink rlk_lxsfrac02-priv_ora_rvg is in AUTOSYNC. 1901824 Kbytes remaining.
root@lxsfrac01 # vxrlink -g oradg status rlk_lxsfrac02-priv_ora_rvg

Fri Nov 02 17:19:16 2007
VxVM VVR vxrlink INFO V-5-1-4467 Rlink rlk_lxsfrac02-priv_ora_rvg is up to date

可以手工进行gracefully切换
root@lxsfrac01 # umount /oracle
root@lxsfrac01 # vradmin -g oradg migrate ora_rvg lxsfrac02-priv
VxVM VVR vradmin WARNING V-5-52-93 Make sure applications using Primary data volumes are stopped.
vradmin: Continue with migrate (y/n)? y
root@lxsfrac01 #

1下面我们就在上面搭建好的VVR环境中对容灾数据进行切换测试：

注：在下面测试中，主节点为lxsfrac02，备用节点为lxsfrac01

====================================== 显示主备节点状态 =================================

root@lxsfrac02 # vradmin -g oradg -l repstatus ora_rvg
Replicated Data Set: ora_rvg
Primary:
Host name: lxsfrac02-priv
RVG name: ora_rvg
DG name: oradg
RVG state: enabled for I/O
Data volumes: 1
VSets: 0
SRL name: srl
SRL size: 800.00 M
Total secondaries: 1

Secondary:
Host name: lxsfrac01-priv
RVG name: ora_rvg
DG name: oradg
Rlink from Primary: rlk_lxsfrac01-priv_ora_rvg
Rlink to Primary: rlk_lxsfrac02-priv_ora_rvg
Configured mode: asynchronous
Latency protection: off
SRL protection: autodcm
Data status: inconsistent
Replication status: resync in progress (autosync)
Current mode: asynchronous
Logging to: DCM (contains 5181696 Kbytes) (autosync)
Timestamp Information: N/A
Bandwidth Limit: N/A

==================================== 显示replication link状态 =================================

root@lxsfrac02 # vxrlink -g oradg status rlk_lxsfrac01-priv_ora_rvg

Mon Nov 05 16:52:51 2007
VxVM VVR vxrlink INFO V-5-1-4467 Rlink rlk_lxsfrac01-priv_ora_rvg is up to date

==================================== 在主节点插入试验数据 16:55 ================================

oracle@lxsfrac02 $ sysdba

SQL*Plus: Release 10.2.0.1.0 - Production on Mon Nov 5 16:55:12 2007

Copyright (c) 1982, 2005, Oracle. All rights reserved.


Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - Production
With the Partitioning, OLAP and Data Mining options

SQL> create table test (a varchar(20) ) tablespace users ;

Table created.

SQL> insert into test values ('1') ;

1 row created.

SQL> commit ;

Commit complete.

SQL> insert into test values ('2' ) ;

1 row created.

SQL>
SQL>
SQL> commit ;

Commit complete.

====================================== 立即异常关闭oracle 和 操作系统 ============================

root@lxsfrac02 # ps -ef|egrep "ora_|asm_"|grep -v grep |grep -v crs|awk '{print $2}' |xargs kill -9
root@lxsfrac02 #
root@lxsfrac02 # ps -ef|grep ora
root 1930 1 0 16:08:13 ? 0:02 /opt/VRTSobc/pal33/bin/vxpal -a StorageAgent -x
oracle 19420 19419 0 16:55:12 ? 0:00 oracleracg1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
oracle 19414 19410 0 16:55:11 pts/3 0:00 -bash
oracle 19419 19414 0 16:55:12 pts/3 0:00 sqlplus /as sysdba
root 19872 3725 0 16:56:49 pts/2 0:00 grep ora
root@lxsfrac02 #
root@lxsfrac02 # poweroff

从系统日志/var/tmp/messages可以查看到相关信息：

Nov 5 16:57:26 lxsfrac02 poweroff: [ID 662345 auth.crit] poweroffed by root

======================================= 在备用节点进行接管 =======================================

root@lxsfrac01 # vradmin -g oradg migrate ora_rvg lxsfrac01-priv
VxVM VVR vradmin WARNING V-5-52-93 Make sure applications using Primary data volumes are stopped.
vradmin: Continue with migrate (y/n)? y
Message from Host lxsfrac01-priv:
VxVM VVR vradmin ERROR V-5-52-449 Secondary ora_rvg does not have an active Primary.
root@lxsfrac01 #
root@lxsfrac01 # vradmin -g oradg takeover ora_rvg
root@lxsfrac01 # date
Mon Nov 5 17:05:13 CST 2007
root@lxsfrac01 # vxprint -htqg oradg
dg oradg default default 10000 1193985774.38.lxsfrac01

dm AMS_WMS0_2 AMS_WMS0_2 auto 65536 4116224 -
dm AMS_WMS0_3 AMS_WMS0_3 auto 65536 146687968 -

rv ora_rvg 1 ENABLED ACTIVE primary 1 srl
rl rlk_lxsfrac02-priv_ora_rvg ora_rvg ENABLED ACTIVE lxsfrac02-priv oradg rlk_lxsfrac01-priv_ora_rvg
v oravol ora_rvg ENABLED ACTIVE 20971520 SELECT - fsgen
pl oravol-01 oravol ENABLED ACTIVE 20971520 CONCAT - RW
sd AMS_WMS0_3-01 oravol-01 AMS_WMS0_3 0 20971520 0 AMS_WMS0_3 ENA
pl oravol-02 oravol ENABLED ACTIVE LOGONLY CONCAT - RW
sd AMS_WMS0_2-01 oravol-02 AMS_WMS0_2 0 352 LOG AMS_WMS0_2 ENA
pl oravol-03 oravol ENABLED ACTIVE LOGONLY CONCAT - RW
sd AMS_WMS0_3-03 oravol-03 AMS_WMS0_3 22609920 352 LOG AMS_WMS0_3 ENA
v srl ora_rvg ENABLED ACTIVE 1638400 SELECT - SRL
pl srl-01 srl ENABLED ACTIVE 1638400 CONCAT - RW
sd AMS_WMS0_3-02 srl-01 AMS_WMS0_3 20971520 1638400 0 AMS_WMS0_3 ENA
root@lxsfrac01 #

================================ 启动备用节点Oracle数据库，并检查数据完整 ========================

root@lxsfrac01 # mount -F vxfs /dev/vx/dsk/oradg/oravol /oracle
root@lxsfrac01 #
root@lxsfrac01 # su - oracle
Sun Microsystems Inc. SunOS 5.10 Generic January 2005
oracle@lxsfrac01 $ sysdba

SQL*Plus: Release 10.2.0.1.0 - Production on Mon Nov 5 17:05:37 2007

Copyright (c) 1982, 2005, Oracle. All rights reserved.

Connected to an idle instance.

SQL> startup
ORACLE instance started.

Total System Global Area 2147483648 bytes
Fixed Size 2120448 bytes
Variable Size 496870656 bytes
Database Buffers 1644167168 bytes
Redo Buffers 4325376 bytes
Database mounted.
Database opened.
SQL> select * from test ;

A
--------------------
1
2

================================ 显示数据正常 ===========================
SQL> !date
Mon Nov 5 17:06:58 CST 2007

SQL> exit
Disconnected from Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - Production
With the Partitioning, OLAP and Data Mining options


附：创建配置VVR过程中部分错误诊断

1．错误V-5-1-3370，备份异常中断，复制不能继续
root@lxsfrac01 # vxrlink -g oradg recover rlk_lxsfrac02-priv_ora_rvg
VxVM VVR vxrlink ERROR V-5-1-3370 Can not recover rlk_lxsfrac02-priv_ora_rvg until ora_rvg is recovered
root@lxsfrac01 # vxrecover -g oradg -s
root@lxsfrac01 # vxprint
………………显示正常

2．错误V-5-52-449，需要检查主、备节点磁盘组状态为import，并且volume/rlink/srl状态为enabled和active
root@lxsfrac01 # vradmin -g oradg -a startrep ora_rvg
Message from Host lxsfrac01-priv:
VxVM VVR vradmin ERROR V-5-52-449 Secondary ora_rvg does not have an active Primary.

在第二个节点vxdg import oradg
然后recover
直到两个节点vxprint -htqg oradg都正常：ENABLED/ACTIVE

3．错误V-5-1-434，我遇到了两次，一次是在createpri，一次在addsec，原因是主备的DCM默认都需要mirror，所以所在的磁盘组都最少包括两块磁盘(或LUN)，通过vxdg -g oradg adddisk another_disk解决。

root@lxsfrac01 # vradmin -g oradg createpri ora_rvg \
> system,sysaux,temp,users,example,ctlfile1,ctlfile2,ctlfile3,redo101,redo102,redo103,undo11,spfile \
> srl
Message from Primary:
VxVM vxassist ERROR V-5-1-434 Cannot allocate space for 32 block log

root@lxsfrac01 # vradmin -g oradg addsec ora_rvg lxsfrac01-priv lxsfrac02-priv
Message from Host lxsfrac02-priv:
VxVM vxassist ERROR V-5-1-434 Cannot allocate space for 352 block log

4．检查vvr后台进程
root@lxsfrac02 # ps -ef|grep vrad|grep -v grep
root 9808 1 0 16:24:47 ? 0:00 /usr/sbin/vradmind
如果进程状态不对，使用下面语句来手工启动：
/etc/init.d/vras-vradmind.sh start

可以看到vradmin是随系统自启动的：
root@lxsfrac02 # find /etc/rc* -name '*vr*' -print
/etc/rc0.d/K01vras-vradmind
/etc/rc1.d/K01vras-vradmind
/etc/rc2.d/S96vradmind
/etc/rcS.d/K01vras-vradmind

1