Oracle 11g R1 RAC with ASM零停机时间存储迁移具体方案

这是我为一个朋友做的零停机时间存储迁移的测试，他的需求简单如下：需要做一次存储的更换，2-nodes Oracle DATABASE 11G R1 RAC ON SORARIS WITH ASM，计划更换存储，原存储划分9个LUN，其中2个用于OCR，3个用于表决盘，剩余的用于ASM磁盘组（只有一个），数据库具体版本号为11.1.0.7。

我当时提供了几种方案，包括非停机迁移和停机迁移：
一、零宕机时间
1）保证新存储和RAC当前节点间的可用性；
2）新存储划分LUN，可以重新规划存储方案；
3）迁移OCR和表决盘
4）给现有ASM磁盘组添加ASM磁盘(新存储分配的)，充分利用ASM REBALANCE技术
5）删除原存储的ASM磁盘
6）观察期

 二、可停机情况
1）保证新存储和RAC当前节点间的可用性；
2）新存储划分LUN，可以重新规划存储方案；
3）迁移OCR和表决盘
4）创建新ASM磁盘组，进行磁盘组的迁移
5）删除原存储的ASM磁盘
6）观察期

或使用终极方法：
1）RMAN全库备份
2）重新部署RAC，with新存储
3）RMAN恢复

因为我对这种案例比较感兴趣，所以决定自己测试并提供具体操作方案。

4月9日上午：
首先，我需要准备一个环境，我很少使用11G R1，更何况是RAC，至今都没有部署过这个版本的。但这也让我感到新鲜，花了一个上午的时间从网上获得了介质。
备注：我这里选择了linux平台，当然和朋友的Soraris Sparc平台是有区别的，但此番我做的事情只是数据库层面的，因此平台问题先忽略。
都如下：
     – 11.1.0.6.0
     linux.x64_11gR1_clusterware.zip
     linux.x64_11gR1_database_1013.zip
     – 升级包11.1.0.7.0
     p6890831_111070_Linux-x86-64.zip
     – PSU
     p14739378_111070_Linux-x86-64.zip
     p11724936_111070_Linux-x86-64.zip
     p11724953_111070_Linux-x86-64.zip
     – Opatch
     p6880880_111000_Linux-x86-64.zip
     其他ASMLIb包等已经拥有。

4月9日下午：
我的部署是从下午开始的，列出我所做的事情：
1）VMware Workstation 8中克隆出两个OEL 5 64 bit OS（我有一个通用的模板机）；
2）配置两个OS，比如修改hostname、修改IP、添加共享磁盘等等；
3）做好RAC安装前准备，比如安装所需包，创建目录、用户，修改参数，分区，SSH用户等效性，NTP等等
4）安装Clusterware
5）安装RDBMS SOFTWARE
6）升级到11.1.0.7.0这个版本（分别升级Clusterware和RDBMS）
7）APPLY 3个PSU，UPDATE到11.1.0.7.14
8）创建监听器、创建库
9）最后检查

主要做的是以上9个内容，这让我花费了不少时间，粗略计算应该有5、6小时。我是在SSD上做的操作，所以速度是比较给力的，但我中间遇到了几个问题，导致我花费更多的时间。
1）提供OCR和表决盘的裸设备大小过小，11gR1 clusterware安装的时候有提示，需要提供256M以上大小空间的裸设备，但我之前全部分配200M，这导致让我重分配磁盘，耗费不少时间；
2）通过UDEV配置裸设备，这是一个非常奇怪的问题，有个配置出来的raw6，在安装clusterware的时候，提供于存放ocr，老报错，比如：权限不足等，但我反复检查也没找到原因，最后是废弃了RAW6 而替换成RAW11(往后排了)，问题解决；
3）DBCA建库时，ASM磁盘组创建报错，报ORA-15032、ORA-15063，也就是我的第二个节点无法MOUNT磁盘组，这个问题我试图手工创建磁盘组来解决，最后在第二个节点create diskgroup成功，第一个节点成功Mount上（反过来报错），但最后建库的时候依然报错，而且无法强制执行下去，报错后第一个节点的磁盘组状态变成dismounted。这让我折腾了好一会儿，也搜索过很多资料，但都没帮助我解决问题，最后我想过放弃使用，替换用Openfiler来模拟SAN，以解决问题。但我突然想到vmware虚拟机的配置文件里一个参数，这个参数我以前测试的时候都加过，而这次没加，没加原因是我参考了一份网上的部署文档。该参数为scsi1.sharedBus = "virtual"，加上这个参数之后，问题彻底解决了，DBCA成功过去。

除了这3个问题，当然还有一些小问题，但基本都很快搞定了。

这里再谈实验过程中的经验：大家在自己的测试环境里操作的时候，也要注意回退这个注意事项，没有一个操作永远不失败的人，某些失败的后果是，只能重头再来，如果大家想到了这个问题的严重性，那肯定会多注意，回退技术有多种，其中通过虚拟机测试的时候，务必利用好快照技术，  宁愿多次shutdown并snapshot，也不愿重头再来！我几乎每个阶段进行快照。

show下结果：

SQL> select * from dba_registry_history; ACTION_TIME                    ACTION   NAMESPAC VERSION            ID BUNDL COMMENTS ------------------------------ -------- -------- ---------- ---------- ----- ---------------------------------------- 09-APR-13 05.22.59.816394 PM   APPLY    SERVER 11.1.0.7 14 PSU   PSU 11.1.0.7.14 [root@khm11gR1rac1 ~]# crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora.khm.db     application    ONLINE    ONLINE    khm1...rac2 
ora....m1.inst application    ONLINE    ONLINE    khm1...rac1 
ora....m2.inst application    ONLINE    ONLINE    khm1...rac2 
ora....SM1.asm application    ONLINE    ONLINE    khm1...rac1 
ora....C1.lsnr application    ONLINE    ONLINE    khm1...rac1 
ora....ac1.gsd application    ONLINE    ONLINE    khm1...rac1 
ora....ac1.ons application    ONLINE    ONLINE    khm1...rac1 
ora....ac1.vip application    ONLINE    ONLINE    khm1...rac1 
ora....SM2.asm application    ONLINE    ONLINE    khm1...rac2 
ora....C2.lsnr application    ONLINE    ONLINE    khm1...rac2 
ora....ac2.gsd application    ONLINE    ONLINE    khm1...rac2 
ora....ac2.ons application    ONLINE    ONLINE    khm1...rac2 
ora....ac2.vip application    ONLINE    ONLINE    khm1...rac2 

通过一天的时间，环境已准备好，剩下需要做迁移存储的测试，另外还有时间的话，我可能还会做upgrade到11g R2的实验。

4月10日晚上：
因为上下午忙着处理公司一些事情，所以存储迁移测试直到下班之后才进行的。
这次我做的实验是零停机时间存储迁移。
这个实验必须要滚动方式进行，不然修改存储都可能会导致节点重启。

节点1： 先停节点一的实例，包括ASM实例 [root@khm11gR1rac1 ~]# srvctl stop instance -d khm -i khm [root@khm11gR1rac1 ~]# srvctl stop asm -n khm11gR1rac1 [root@khm11gR1rac1 ~]# crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora.khm.db     application    ONLINE    ONLINE    khm1...rac1 
ora....m1.inst application    OFFLINE   OFFLINE               
ora....m2.inst application    ONLINE    ONLINE    khm1...rac2 
ora....SM1.asm application    OFFLINE   OFFLINE               
ora....C1.lsnr application    ONLINE    ONLINE    khm1...rac1 
ora....ac1.gsd application    ONLINE    ONLINE    khm1...rac1 
ora....ac1.ons application    ONLINE    ONLINE    khm1...rac1 
ora....ac1.vip application    ONLINE    ONLINE    khm1...rac1 
ora....SM2.asm application    ONLINE    ONLINE    khm1...rac2 
ora....C2.lsnr application    ONLINE    ONLINE    khm1...rac2 
ora....ac2.gsd application    ONLINE    ONLINE    khm1...rac2 
ora....ac2.ons application    ONLINE    ONLINE    khm1...rac2 
ora....ac2.vip application    ONLINE    ONLINE    khm1...rac2 连接新存储，我这里使用openfiler模拟SAN并使用iscsi方式连接，此操作略。 重启iscsi服务，这样将识别新挂接过来的LUN [root@khm11gR1rac1 ~]# service iscsi restart 使用fdisk工具分区格式化，操作略。 修改rawdevices文件好让系统启动时挂载裸设备 [root@khm11gR1rac1 ~]# vi /etc/sysconfig/rawdevices  -- 添加如下内容 /dev/raw/raw1 /dev/sdb1 /dev/raw/raw2 /dev/sdc1 /dev/raw/raw3 /dev/sdd1 /dev/raw/raw4 /dev/sde1 /dev/raw/raw5 /dev/sdf1 裸设备映射： [root@khm11gR1rac1 ~]# vi /etc/udev/rules.d/60-raw.rules  -- 添加如下内容 ACTION=="add", KERNEL=="/dev/sdb1",RUN+="/bin/raw /dev/raw/raw1 %N" ACTION=="add", ENV{MAJOR}=="8",ENV{MINOR}=="17",RUN+="/bin/raw /dev/raw/raw1 %M %m" ACTION=="add", KERNEL=="/dev/sdc1",RUN+="/bin/raw /dev/raw/raw2 %N" ACTION=="add", ENV{MAJOR}=="8",ENV{MINOR}=="33",RUN+="/bin/raw /dev/raw/raw2 %M %m" ACTION=="add", KERNEL=="/dev/sdd1",RUN+="/bin/raw /dev/raw/raw3 %N" ACTION=="add", ENV{MAJOR}=="8",ENV{MINOR}=="49",RUN+="/bin/raw /dev/raw/raw3 %M %m" ACTION=="add", KERNEL=="/dev/sde1",RUN+="/bin/raw /dev/raw/raw4 %N" ACTION=="add", ENV{MAJOR}=="8",ENV{MINOR}=="65",RUN+="/bin/raw /dev/raw/raw4 %M %m" ACTION=="add", KERNEL=="/dev/sdf1",RUN+="/bin/raw /dev/raw/raw5 %N" ACTION=="add", ENV{MAJOR}=="8",ENV{MINOR}=="81",RUN+="/bin/raw /dev/raw/raw5 %M %m" KERNEL=="raw[1-5]", OWNER="oracle", GROUP="oinstall", MODE="640" 重启udev： [root@khm11gR1rac1 ~]# start_udev Starting udev: [ OK ] 重启rawdevices服务： [root@khm11gR1rac1 ~]# service rawdevices restart Assigning devices: /dev/raw/raw1 --> /dev/sdb1 /dev/raw/raw1: bound to major 8, minor 17 /dev/raw/raw2 --> /dev/sdc1 /dev/raw/raw2: bound to major 8, minor 33 /dev/raw/raw3 --> /dev/sdd1 /dev/raw/raw3: bound to major 8, minor 49 /dev/raw/raw4 --> /dev/sde1 /dev/raw/raw4: bound to major 8, minor 65 /dev/raw/raw5 --> /dev/sdf1 /dev/raw/raw5: bound to major 8, minor 81 done 下面查看结果： [root@khm11gR1rac1 ~]# ls -l /dev/raw total 0 crw-r----- 1 oracle oinstall 162, 1 Apr 11 01:38 raw1
crw-r----- 1 oracle oinstall 162, 10 Apr 11 01:38 raw10
crw-r----- 1 oracle oinstall 162, 11 Apr 11 01:38 raw11
crw-r----- 1 oracle oinstall 162, 2 Apr 11 01:38 raw2
crw-r----- 1 oracle oinstall 162, 3 Apr 11 01:38 raw3
crw-r----- 1 oracle oinstall 162, 4 Apr 11 01:38 raw4
crw-r----- 1 oracle oinstall 162, 5 Apr 11 01:38 raw5
crw-r----- 1 oracle oinstall 162, 6 Apr 11 01:38 raw6
crw-r----- 1 oracle oinstall 162, 7 Apr 11 01:38 raw7
crw-r----- 1 oracle oinstall 162, 8 Apr 11 01:38 raw8
crw-r----- 1 oracle oinstall 162, 9 Apr 11 01:38 raw9 创建ASM磁盘： [root@khm11gR1rac1 ~]# /etc/init.d/oracleasm createdisk khmasm /dev/sdo1 Marking disk "khmasm" as an ASM disk: [ OK ] [root@khm11gR1rac1 ~]# oracleasm listdisks ADISK
ASMDISK
KHMASM 查看当前资源启动状态： [root@khm11gR1rac1 ~]# crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora.khm.db     application    ONLINE    ONLINE    khm1...rac1 
ora....m1.inst application    OFFLINE   OFFLINE               
ora....m2.inst application    ONLINE    ONLINE    khm1...rac2 
ora....SM1.asm application    OFFLINE   OFFLINE               
ora....C1.lsnr application    ONLINE    OFFLINE               
ora....ac1.gsd application    ONLINE    ONLINE    khm1...rac1 
ora....ac1.ons application    ONLINE    ONLINE    khm1...rac1 
ora....ac1.vip application    ONLINE    ONLINE    khm1...rac2 
ora....SM2.asm application    ONLINE    ONLINE    khm1...rac2 
ora....C2.lsnr application    ONLINE    ONLINE    khm1...rac2 
ora....ac2.gsd application    ONLINE    ONLINE    khm1...rac2 
ora....ac2.ons application    ONLINE    ONLINE    khm1...rac2 
ora....ac2.vip application    ONLINE    ONLINE    khm1...rac2 启动监听器、asm实例和数据库实例： [root@khm11gR1rac1 ~]# srvctl start listener -n khm11gR1rac1 [root@khm11gR1rac1 ~]# srvctl start asm -n khm11gR1rac1 [root@khm11gR1rac1 ~]# srvctl start instance -d khm -i khm1 [root@khm11gR1rac1 ~]# crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora.khm.db     application    ONLINE    ONLINE    khm1...rac1 
ora....m1.inst application    OFFLINE   ONLINE    khm1...rac1 
ora....m2.inst application    ONLINE    ONLINE    khm1...rac2 
ora....SM1.asm application    ONLINE    ONLINE    khm1...rac1 
ora....C1.lsnr application    ONLINE    ONLINE    khm1...rac1 
ora....ac1.gsd application    ONLINE    ONLINE    khm1...rac1 
ora....ac1.ons application    ONLINE    ONLINE    khm1...rac1 
ora....ac1.vip application    ONLINE    ONLINE    khm1...rac1 
ora....SM2.asm application    ONLINE    ONLINE    khm1...rac2 
ora....C2.lsnr application    ONLINE    ONLINE    khm1...rac2 
ora....ac2.gsd application    ONLINE    ONLINE    khm1...rac2 
ora....ac2.ons application    ONLINE    ONLINE    khm1...rac2 
ora....ac2.vip application    ONLINE    ONLINE    khm1...rac2 节点2： 节点2上操作与节点1大致相同： [root@khm11gR1rac2 ~]# srvctl stop instance -d khm -i khm2 [root@khm11gR1rac2 ~]# srvctl stop asm -n khm11gR1rac2 [root@khm11gR1rac2 ~]# crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora.khm.db     application    ONLINE    ONLINE    khm1...rac1 
ora....m1.inst application    ONLINE    ONLINE    khm1...rac1 
ora....m2.inst application    OFFLINE   OFFLINE               
ora....SM1.asm application    ONLINE    ONLINE    khm1...rac1 
ora....C1.lsnr application    ONLINE    ONLINE    khm1...rac1 
ora....ac1.gsd application    ONLINE    ONLINE    khm1...rac1 
ora....ac1.ons application    ONLINE    ONLINE    khm1...rac1 
ora....ac1.vip application    ONLINE    ONLINE    khm1...rac1 
ora....SM2.asm application    OFFLINE   OFFLINE               
ora....C2.lsnr application    ONLINE    ONLINE    khm1...rac2 
ora....ac2.gsd application    ONLINE    ONLINE    khm1...rac2 
ora....ac2.ons application    ONLINE    ONLINE    khm1...rac2 
ora....ac2.vip application    ONLINE    ONLINE    khm1...rac2 [root@khm11gR1rac2 ~]# vi /etc/sysconfig/rawdevices  -- 添加内容 /dev/raw/raw1 /dev/sdb1 /dev/raw/raw2 /dev/sdc1 /dev/raw/raw3 /dev/sdd1 /dev/raw/raw4 /dev/sde1 /dev/raw/raw5 /dev/sdf1 [root@khm11gR1rac2 ~]# vi /etc/udev/rules.d/60-raw.rules  -- 添加内容 ACTION=="add", KERNEL=="/dev/sdb1",RUN+="/bin/raw /dev/raw/raw1 %N" ACTION=="add", ENV{MAJOR}=="8",ENV{MINOR}=="17",RUN+="/bin/raw /dev/raw/raw1 %M %m" ACTION=="add", KERNEL=="/dev/sdc1",RUN+="/bin/raw /dev/raw/raw2 %N" ACTION=="add", ENV{MAJOR}=="8",ENV{MINOR}=="33",RUN+="/bin/raw /dev/raw/raw2 %M %m" ACTION=="add", KERNEL=="/dev/sdd1",RUN+="/bin/raw /dev/raw/raw3 %N" ACTION=="add", ENV{MAJOR}=="8",ENV{MINOR}=="49",RUN+="/bin/raw /dev/raw/raw3 %M %m" ACTION=="add", KERNEL=="/dev/sde1",RUN+="/bin/raw /dev/raw/raw4 %N" ACTION=="add", ENV{MAJOR}=="8",ENV{MINOR}=="65",RUN+="/bin/raw /dev/raw/raw4 %M %m" ACTION=="add", KERNEL=="/dev/sdf1",RUN+="/bin/raw /dev/raw/raw5 %N" ACTION=="add", ENV{MAJOR}=="8",ENV{MINOR}=="81",RUN+="/bin/raw /dev/raw/raw5 %M %m" KERNEL=="raw[1-5]", OWNER="oracle", GROUP="oinstall", MODE="640" [root@khm11gR1rac2 ~]# start_udev Starting udev: [ OK ] [root@khm11gR1rac2 ~]# service rawdevices restart Assigning devices: /dev/raw/raw1 --> /dev/sdb1 /dev/raw/raw1: bound to major 8, minor 17 /dev/raw/raw2 --> /dev/sdc1 /dev/raw/raw2: bound to major 8, minor 33 /dev/raw/raw3 --> /dev/sdd1 /dev/raw/raw3: bound to major 8, minor 49 /dev/raw/raw4 --> /dev/sde1 /dev/raw/raw4: bound to major 8, minor 65 /dev/raw/raw5 --> /dev/sdf1 /dev/raw/raw5: bound to major 8, minor 81 done [root@khm11gR1rac2 ~]# ls -l /dev/raw total 0 crw-r----- 1 oracle oinstall 162, 1 Apr 11 01:38 raw1
crw-r----- 1 oracle oinstall 162, 10 Apr 11 01:38 raw10
crw-r----- 1 oracle oinstall 162, 11 Apr 11 01:38 raw11
crw-r----- 1 oracle oinstall 162, 2 Apr 11 01:38 raw2
crw-r----- 1 oracle oinstall 162, 3 Apr 11 01:38 raw3
crw-r----- 1 oracle oinstall 162, 4 Apr 11 01:38 raw4
crw-r----- 1 oracle oinstall 162, 5 Apr 11 01:38 raw5
crw-r----- 1 oracle oinstall 162, 6 Apr 11 01:38 raw6
crw-r----- 1 oracle oinstall 162, 7 Apr 11 01:38 raw7
crw-r----- 1 oracle oinstall 162, 8 Apr 11 01:38 raw8
crw-r----- 1 oracle oinstall 162, 9 Apr 11 01:38 raw9 节点1上已创建asm磁盘，节点2只要扫描即可： [root@khm11gR1rac2 ~]# oracleasm scandisks Reloading disk partitions: done Cleaning any stale ASM disks... Scanning system for ASM disks... Instantiating disk "KHMASM" [root@khm11gR1rac2 ~]# oracleasm listdisks ADISK
ASMDISK
KHMASM [root@khm11gR1rac2 ~]# srvctl start listener -n khm11gR1rac2 [root@khm11gR1rac2 ~]# srvctl start asm -n khm11gR1rac2 [root@khm11gR1rac2 ~]# srvctl start instance -d khm -i khm2 [root@khm11gR1rac2 ~]# crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora.khm.db     application    ONLINE    ONLINE    khm1...rac1 
ora....m1.inst application    OFFLINE   ONLINE    khm1...rac1 
ora....m2.inst application    ONLINE    ONLINE    khm1...rac2 
ora....SM1.asm application    ONLINE    ONLINE    khm1...rac1 
ora....C1.lsnr application    ONLINE    ONLINE    khm1...rac1 
ora....ac1.gsd application    ONLINE    ONLINE    khm1...rac1 
ora....ac1.ons application    ONLINE    ONLINE    khm1...rac1 
ora....ac1.vip application    ONLINE    ONLINE    khm1...rac1 
ora....SM2.asm application    ONLINE    ONLINE    khm1...rac2 
ora....C2.lsnr application    ONLINE    ONLINE    khm1...rac2 
ora....ac2.gsd application    ONLINE    ONLINE    khm1...rac2 
ora....ac2.ons application    ONLINE    ONLINE    khm1...rac2 
ora....ac2.vip application    ONLINE    ONLINE    khm1...rac2

通过上面的操作，已经将新的存储成功连接并识别到划分好的LUN，这期间采用滚动方式，所以业务是不会断的。
下面要进行OCR和表决盘的迁移。

迁移OCR

通过ocrcheck命令可查看ocr存储位置以及其他信息: [root@khm11gR1rac1 ~]# ocrcheck Status of Oracle Cluster Registry is as follows : Version : 2 Total space (kbytes) : 304920 Used space (kbytes) : 3852 Available space (kbytes) : 301068 ID : 1963893554 Device/File Name : /dev/raw/raw11 Device/File integrity check succeeded Device/File Name : /dev/raw/raw10 Device/File integrity check succeeded Cluster registry integrity check succeeded Logical corruption check succeeded 另外也可以查看/etc/oracle/ocr.loc： [root@khm11gR1rac1 ~]# cat /etc/oracle/ocr.loc  ocrconfig_loc=/dev/raw/raw11
ocrmirrorconfig_loc=/dev/raw/raw10
local_only=FALSE -- 从上面可以看出/dev/raw/raw10上是ocr的镜像 确保crs健康运行： [root@khm11gR1rac1 ~]# crsctl check crs Cluster Synchronization Services appears healthy Cluster Ready Services appears healthy Event Manager appears healthy 通过ocrconfig命令可以在线迁移ocr，注意如下方式： [root@khm11gR1rac1 ~]# ocrconfig -replace ocr /dev/raw/raw1 [root@khm11gR1rac1 ~]# ocrconfig -replace ocrmirror /dev/raw/raw2 -- 一个是ocr，另一个是ocrmirror，分别对应/etc/oracle/ocr.loc的ocrconfig_loc和ocrmirrorconfig_loc所指的位置。 再次查看，已成功迁移： [root@khm11gR1rac1 ~]# ocrcheck Status of Oracle Cluster Registry is as follows : Version : 2 Total space (kbytes) : 304920 Used space (kbytes) : 3852 Available space (kbytes) : 301068 ID : 1963893554 Device/File Name : /dev/raw/raw1 Device/File integrity check succeeded Device/File Name : /dev/raw/raw2 Device/File integrity check succeeded Cluster registry integrity check succeeded Logical corruption check succeeded

扩展：如果是Oracle 11g R2版本，ocrconfig命令增强了一些，更改ocr位置的命令也有所改变，如下：
ocrconfig -replace -replacement

迁移表决盘

查看表决磁盘当前存放路径： [root@khm11gR1rac1 ~]# crsctl query css votedisk 0. 0 /dev/raw/raw7 1. 0 /dev/raw/raw8 2. 0 /dev/raw/raw9 Located 3 voting disk(s). 在Oracle 11g R2之前版本呢，只能采用添加删除的方式进行迁移。首先删除一个，因为当前有3个存放路径，所以删除一个并不产生影响： [root@khm11gR1rac1 ~]# crsctl delete css votedisk /dev/raw/raw9 Successful deletion of voting disk /dev/raw/raw9. 添加表决磁盘到新的路径： [root@khm11gR1rac1 ~]# crsctl add css votedisk /dev/raw/raw5 Now formatting voting disk: /dev/raw/raw5. Successful addition of voting disk /dev/raw/raw5. [root@khm11gR1rac1 ~]# crsctl delete css votedisk /dev/raw/raw8 Successful deletion of voting disk /dev/raw/raw8. [root@khm11gR1rac1 ~]# crsctl add css votedisk /dev/raw/raw4 Now formatting voting disk: /dev/raw/raw4. Successful addition of voting disk /dev/raw/raw4. 当试图删除表决磁盘首个存放路径的时候会收到报错： [root@khm11gR1rac1 ~]# crsctl delete css votedisk /dev/raw/raw7 Failure 8 with Cluster Synchronization Services while deleting voting disk. 添加force选项强制删除： [root@khm11gR1rac1 ~]# crsctl delete css votedisk /dev/raw/raw7 -force Successful deletion of voting disk /dev/raw/raw7. [root@khm11gR1rac1 ~]# crsctl query css votedisk 1. 0 /dev/raw/raw4 2. 0 /dev/raw/raw5 Located 2 voting disk(s). [root@khm11gR1rac1 ~]# crsctl add css votedisk /dev/raw/raw3 Now formatting voting disk: /dev/raw/raw3. Successful addition of voting disk /dev/raw/raw3. 最后查看： [root@khm11gR1rac1 ~]# crsctl query css votedisk 0. 0 /dev/raw/raw3 1. 0 /dev/raw/raw4 2. 0 /dev/raw/raw5 Located 3 voting disk(s).

扩展：Oracle 11g R2中crsctl命令增强很多，其中针对votedisk的操作也增多了replace功能：
crsctl replace   – replaces the location of voting files
可见，在这个版本上DBA大可以在线替换存储位置。

由此，OCR和表决磁盘在线迁移成功。
下面开始添加asm磁盘，这里使用ASM的REBALANCE技术来解决在线迁移数据。

登录到ASM实例，查看当前磁盘组的信息： [root@khm11gR1rac1 ~]# su - oracle [oracle@khm11gR1rac1 ~]$ export ORACLE_SID=+ASM1 [oracle@khm11gR1rac1 ~]$ sqlplus / as sysasm

SQL*Plus: Release 11.1.0.7.0 - Production on Thu Apr 11 02:51:11 2013 Copyright (c) 1982, 2008, Oracle. All rights reserved. Connected to: Oracle Database 11g Enterprise Edition Release 11.1.0.7.0 - 64bit Production With the Partitioning, Real Application Clusters, OLAP, Data Mining and Real Application Testing options

SQL> col NAME for a10
SQL> set line 150 SQL> select NAME, ALLOCATION_UNIT_SIZE, STATE, TOTAL_MB, FREE_MB USABLE_FILE_MB from gv$asm_diskgroup; NAME       ALLOCATION_UNIT_SIZE STATE                    TOTAL_MB USABLE_FILE_MB ---------- -------------------- ---------------------- ---------- -------------- KHM_DATA 1048576 MOUNTED 15319 13407 KHM_DATA 1048576 MOUNTED 15319 13407 SQL> select failgroup, name from v$asm_disk where group_number=(select group_number from v$asm_diskgroup where name = 'KHM_DATA'); FAILGROUP                                                    NAME ------------------------------------------------------------ ---------- ASMDISK                                                      ASMDISK 现往该磁盘组里添加一个新创建的ASM磁盘，指定REBALANCE的速度： SQL> alter diskgroup khm_data add disk 'ORCL:KHMASM' name khm_asm REBALANCE POWER 10; Diskgroup altered. 备注：这里只测试添加一个ASM磁盘，在真实环境中可能会增加多个，那么采用如下方式： alter diskgroup khm_data add disk 'ORCL:ASM1', 'ORCL:ASM2', 'ORCL:ASM3' REBALANCE POWER n; 当然，先添加磁盘后手动进行REBALANCE操作也可以。详细的操作请查阅官方文档。 注意：REBALANCE过程根据数据量的大小不同完成时间也不一，可以针对当前系统情况指定合理的POWER值。 REBALANCE过程可以通过v$asm_operation视图来查看： SQL> set line 150 SQL> col ERROR_CODE for a5
SQL> select * from v$asm_operation; GROUP_NUMBER OPERATION  STATE         POWER     ACTUAL      SOFAR   EST_WORK   EST_RATE EST_MINUTES ERROR ------------ ---------- -------- ---------- ---------- ---------- ---------- ---------- ----------- ----- 2 REBAL      REAP 10 1 597 597 503 0 REBALANCE完成后记录将消失： SQL> select * from v$asm_operation; no rows selected 添加ASM磁盘后的状态： SQL> select NAME, ALLOCATION_UNIT_SIZE, STATE, TOTAL_MB, FREE_MB USABLE_FILE_MB from gv$asm_diskgroup; NAME       ALLOCATION_UNIT_SIZE STATE                    TOTAL_MB USABLE_FILE_MB ---------- -------------------- ---------------------- ---------- -------------- KHM_DATA 1048576 MOUNTED 30638 28714 KHM_DATA 1048576 MOUNTED 30638 28714 SQL> select failgroup, name, path from v$asm_disk where group_number=(select group_number from v$asm_diskgroup where name = 'KHM_DATA'); FAILGROUP                                                    NAME ------------------------------------------------------------ ---------- ASMDISK                                                      ASMDISK
KHM_ASM                                                      KHM_ASM 好，现在可以删除旧的ASM磁盘了。 SQL> col FAILGROUP for a20
SQL> col NAME for a20
SQL> col PATH for a20
SQL> select failgroup, name, path from v$asm_disk where group_number=(select group_number from v$asm_diskgroup where name = 'KHM_DATA'); FAILGROUP            NAME                 PATH -------------------- -------------------- -------------------- ASMDISK              ASMDISK              ORCL:ASMDISK
KHM_ASM              KHM_ASM              ORCL:KHMASM 删除的时候也可以指定rebalance速度： SQL> alter diskgroup khm_data drop disk ASMDISK rebalance power 5; Diskgroup altered. 观察rebalance情况： SQL> select * from v$asm_operation; GROUP_NUMBER OPERATION  STATE         POWER     ACTUAL      SOFAR   EST_WORK   EST_RATE EST_MINUTES ERROR ------------ ---------- -------- ---------- ---------- ---------- ---------- ---------- ----------- ----- 2 REBAL      RUN 5 5 6 970 167 5 SQL> select * from v$asm_operation; GROUP_NUMBER OPERATION  STATE         POWER     ACTUAL      SOFAR   EST_WORK   EST_RATE EST_MINUTES ERROR ------------ ---------- -------- ---------- ---------- ---------- ---------- ---------- ----------- ----- 2 REBAL      RUN 5 5 67 796 201 3 SQL> select * from v$asm_operation; GROUP_NUMBER OPERATION  STATE         POWER     ACTUAL      SOFAR   EST_WORK   EST_RATE EST_MINUTES ERROR ------------ ---------- -------- ---------- ---------- ---------- ---------- ---------- ----------- ----- 2 REBAL      RUN 5 5 682 840 228 0 SQL> select * from v$asm_operation; no rows selected

rebalance完成，查看数据保证无误： SQL> select count(*) from luocs.t1; COUNT(*) ---------- 68728

现在系统里用到的全部为新存储上的空间，这样就可以卸载旧存储了。

首先清理环境，同样滚动方式卸载旧磁盘： 节点1： [root@khm11gR1rac1 ~]# srvctl stop instance -d khm -i khm1 [root@khm11gR1rac1 ~]# srvctl stop asm -n khm11gR1rac1 [root@khm11gR1rac1 ~]# crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora.khm.db     application    ONLINE    ONLINE    khm1...rac1 
ora....m1.inst application    OFFLINE   OFFLINE               
ora....m2.inst application    ONLINE    ONLINE    khm1...rac2 
ora....SM1.asm application    OFFLINE   OFFLINE               
ora....C1.lsnr application    ONLINE    ONLINE    khm1...rac1 
ora....ac1.gsd application    ONLINE    ONLINE    khm1...rac1 
ora....ac1.ons application    ONLINE    ONLINE    khm1...rac1 
ora....ac1.vip application    ONLINE    ONLINE    khm1...rac1 
ora....SM2.asm application    ONLINE    ONLINE    khm1...rac2 
ora....C2.lsnr application    ONLINE    ONLINE    khm1...rac2 
ora....ac2.gsd application    ONLINE    ONLINE    khm1...rac2 
ora....ac2.ons application    ONLINE    ONLINE    khm1...rac2 
ora....ac2.vip application    ONLINE    ONLINE    khm1...rac2 [root@khm11gR1rac1 ~]# oracleasm listdisks ADISK
ASMDISK
KHMASM 删除创建的ASM磁盘： [root@khm11gR1rac1 ~]# /etc/init.d/oracleasm deletedisk asmdisk Removing ASM disk "asmdisk": [ OK ] 修改/etc/sysconfig/rawdevices和/etc/udev/rules.d/60-raw.rules，清除旧磁盘信息，并重启udev和rawdevices服务，操作略。 查看crs状态： [root@khm11gR1rac1 ~]# crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora.khm.db     application    ONLINE    ONLINE    khm1...rac1 
ora....m1.inst application    OFFLINE   OFFLINE               
ora....m2.inst application    ONLINE    ONLINE    khm1...rac2 
ora....SM1.asm application    OFFLINE   OFFLINE               
ora....C1.lsnr application    ONLINE    OFFLINE               
ora....ac1.gsd application    ONLINE    ONLINE    khm1...rac1 
ora....ac1.ons application    ONLINE    ONLINE    khm1...rac1 
ora....ac1.vip application    ONLINE    ONLINE    khm1...rac2 
ora....SM2.asm application    ONLINE    ONLINE    khm1...rac2 
ora....C2.lsnr application    ONLINE    ONLINE    khm1...rac2 
ora....ac2.gsd application    ONLINE    ONLINE    khm1...rac2 
ora....ac2.ons application    ONLINE    ONLINE    khm1...rac2 
ora....ac2.vip application    ONLINE    ONLINE    khm1...rac2 [root@khm11gR1rac1 ~]# srvctl start listener -n khm11gR1rac1 [root@khm11gR1rac1 ~]# srvctl start asm -n khm11gR1rac1 [root@khm11gR1rac1 ~]# srvctl start instance -d khm -i khm1 [root@khm11gR1rac1 ~]# crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora.khm.db     application    ONLINE    ONLINE    khm1...rac1 
ora....m1.inst application    OFFLINE   ONLINE    khm1...rac1 
ora....m2.inst application    ONLINE    ONLINE    khm1...rac2 
ora....SM1.asm application    ONLINE    ONLINE    khm1...rac1 
ora....C1.lsnr application    ONLINE    ONLINE    khm1...rac1 
ora....ac1.gsd application    ONLINE    ONLINE    khm1...rac1 
ora....ac1.ons application    ONLINE    ONLINE    khm1...rac1 
ora....ac1.vip application    ONLINE    ONLINE    khm1...rac1 
ora....SM2.asm application    ONLINE    ONLINE    khm1...rac2 
ora....C2.lsnr application    ONLINE    ONLINE    khm1...rac2 
ora....ac2.gsd application    ONLINE    ONLINE    khm1...rac2 
ora....ac2.ons application    ONLINE    ONLINE    khm1...rac2 
ora....ac2.vip application    ONLINE    ONLINE    khm1...rac2 节点2： [root@khm11gR1rac2 ~]# srvctl stop instance -d khm -i khm2 [root@khm11gR1rac2 ~]# srvctl stop asm -n khm11gR1rac2 [root@khm11gR1rac2 ~]# crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora.khm.db     application    ONLINE    ONLINE    khm1...rac1 
ora....m1.inst application    ONLINE    ONLINE    khm1...rac1 
ora....m2.inst application    OFFLINE   OFFLINE               
ora....SM1.asm application    ONLINE    ONLINE    khm1...rac1 
ora....C1.lsnr application    ONLINE    ONLINE    khm1...rac1 
ora....ac1.gsd application    ONLINE    ONLINE    khm1...rac1 
ora....ac1.ons application    ONLINE    ONLINE    khm1...rac1 
ora....ac1.vip application    ONLINE    ONLINE    khm1...rac1 
ora....SM2.asm application    OFFLINE   OFFLINE               
ora....C2.lsnr application    ONLINE    ONLINE    khm1...rac2 
ora....ac2.gsd application    ONLINE    ONLINE    khm1...rac2 
ora....ac2.ons application    ONLINE    ONLINE    khm1...rac2 
ora....ac2.vip application    ONLINE    ONLINE    khm1...rac2 [root@khm11gR1rac2 ~]#  同样修改/etc/sysconfig/rawdevices和/etc/udev/rules.d/60-raw.rules，清除旧磁盘信息，并重启udev和rawdevices服务，操作略。 因为节点1上已删除asm磁盘，所以这里只要扫描即可。 [root@khm11gR1rac2 ~]# oracleasm scandisks Reloading disk partitions: done Cleaning any stale ASM disks... Cleaning disk "ASMDISK" Scanning system for ASM disks... [root@khm11gR1rac2 ~]# oracleasm listdisks ADISK
KHMASM [root@khm11gR1rac2 ~]# srvctl start listener -n khm11gR1rac2 [root@khm11gR1rac2 ~]# srvctl start asm -n khm11gR1rac2 [root@khm11gR1rac2 ~]# srvctl start instance -d khm -i khm2 [root@khm11gR1rac2 ~]# crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora.khm.db     application    ONLINE    ONLINE    khm1...rac1 
ora....m1.inst application    OFFLINE   ONLINE    khm1...rac1 
ora....m2.inst application    ONLINE    ONLINE    khm1...rac2 
ora....SM1.asm application    ONLINE    ONLINE    khm1...rac1 
ora....C1.lsnr application    ONLINE    ONLINE    khm1...rac1 
ora....ac1.gsd application    ONLINE    ONLINE    khm1...rac1 
ora....ac1.ons application    ONLINE    ONLINE    khm1...rac1 
ora....ac1.vip application    ONLINE    ONLINE    khm1...rac1 
ora....SM2.asm application    ONLINE    ONLINE    khm1...rac2 
ora....C2.lsnr application    ONLINE    ONLINE    khm1...rac2 
ora....ac2.gsd application    ONLINE    ONLINE    khm1...rac2 
ora....ac2.ons application    ONLINE    ONLINE    khm1...rac2 
ora....ac2.vip application    ONLINE    ONLINE    khm1...rac2

为了确认系统可用性，最后我还手动重启服务器，当然，这在生产环境中是不太容易实现的。

[root@khm11gR1rac1 ~]# srvctl stop database -d khm [root@khm11gR1rac1 ~]# srvctl stop asm -n khm11gR1rac1 [root@khm11gR1rac1 ~]# srvctl stop asm -n khm11gR1rac2 [root@khm11gR1rac1 ~]#  srvctl stop nodeapps -n khm11gR1rac1 [root@khm11gR1rac1 ~]#  srvctl stop nodeapps -n khm11gR1rac2 [root@khm11gR1rac1 ~]#  crsctl stop crs [root@khm11gR1rac1 ~]# shutdown -r now [root@khm11gR1rac2 ~]#  crsctl stop crs [root@khm11gR1rac2 ~]# shutdown -r now 重启之后： [root@khm11gR1rac1 ~]# raw -qa /dev/raw/raw1: bound to major 8, minor 17 /dev/raw/raw2: bound to major 8, minor 33 /dev/raw/raw3: bound to major 8, minor 49 /dev/raw/raw4: bound to major 8, minor 65 /dev/raw/raw5: bound to major 8, minor 81 [root@khm11gR1rac1 ~]# ocrcheck Status of Oracle Cluster Registry is as follows : Version : 2 Total space (kbytes) : 304920 Used space (kbytes) : 3856 Available space (kbytes) : 301064 ID : 1963893554 Device/File Name : /dev/raw/raw1 Device/File integrity check succeeded Device/File Name : /dev/raw/raw2 Device/File integrity check succeeded Cluster registry integrity check succeeded Logical corruption check succeeded [root@khm11gR1rac1 ~]# crsctl query css votedisk 0. 0 /dev/raw/raw3 1. 0 /dev/raw/raw4 2. 0 /dev/raw/raw5 Located 3 voting disk(s). SQL> select name from v$datafile; NAME -------------------------------------------------------------------------------- +KHM_DATA/khm/datafile/system.260.812339753 +KHM_DATA/khm/datafile/sysaux.261.812339755 +KHM_DATA/khm/datafile/undotbs1.272.812339757 +KHM_DATA/khm/datafile/users.266.812339757 +KHM_DATA/khm/datafile/undotbs2.265.812340387 +KHM_DATA/khm/datafile/khm.267.812341203 6 rows selected. SQL> select count(*) from luocs.t1; COUNT(*) ---------- 68728 SQL> select count(*) from luocs.hm; COUNT(*) ---------- 1

实验做到晚上9点半，发现公司里只剩自己了，赶紧收拾准备走人！途中给朋友发信息告知结果，朋友貌似非常高兴又非常期待的样子。

另外停机时间存储迁移的具体方案，难度不是很大，如果有时间，我再进行测试并分享结果。