IBM DS4000/5000系列的存储是一款销量不错的中端存储产品,这是一款OEM LSI公司的产品,不过该存储所属的LSI Engenio 中端存储在2011年3月已经被Netapp收购了(IT界也挺乱的^_^)。光纤存储一般都用于UNIX环境,在Linux上的使用,尤其是需要结合多路径软件,相对来说就比较少,参考资料总是很有限。我将自己在RHEL 5.3系统上安装使用多路径软件RDAC的过程记录了下来,希望有一定的参考价值。
1. 使用 uname -r 命令查看当前Linux系统的内核版本
- [root@test01 ~]# uname -r
-
2.6.18-128.el5
2. 下载RDAC驱动,登陆
,,找到与当前Linux系统内核版本一致的RDAC驱动。
根据我实验中的版本,选择下载了Version 09.03.0C05.0331(rdac-LINUX-09.03.0C05.0331-source.tar.gz)
3. 将rdac的安装包上传至~目录(root用户根目录)
- [root@test01 ~]# pwd
-
/root
-
[root@test01 ~]# ls -l|grep rdac
-
-rw-r--r-- 1 root root 399735 Jun 20 19:33 rdac-LINUX-09.03.0C05.0331-source.tar.gz
4. 解压源码包
- [root@test01 ~]# tar -zxvf rdac-LINUX-09.03.0C05.0331-source.tar.gz
-
[root@test01 ~]# ls -l|grep rdac
-
drwxr-xr-x 5 root root 4096 Apr 15 2010 linuxrdac-09.03.0C05.0331
-
-rw-r--r-- 1 root root 399735 Jun 20 19:33 rdac-LINUX-09.03.0C05.0331-source.tar.gz
5. 参考解压出来的目录中的Readme.txt文件,编译并安装RDAC软件
Linux一定要安装下面的包(此段内容未经我验证):
gcc
glibc-devel
kernel-headers
glibc-headers
libgomp
kernel-devel 或 kernel-xen-devel(如果使用启用 Xen 的内核的话)
- [root@test01 linuxrdac-09.03.0C05.0331]# make
-
make[1]: Entering directory `/usr/src/kernels/2.6.18-128.el5-x86_64'
-
CC [M] /root/linuxrdac-09.03.0C05.0331/MPP_hba.o
-
CC [M] /root/linuxrdac-09.03.0C05.0331/mppLnx26p_upper.o
-
CC [M] /root/linuxrdac-09.03.0C05.0331/mppLnx26p_sysdep.o
-
CC [M] /root/linuxrdac-09.03.0C05.0331/mppCmn_s2tos3.o
-
CC [M] /root/linuxrdac-09.03.0C05.0331/mppCmn_SysInterface.o
-
CC [M] /root/linuxrdac-09.03.0C05.0331/mppLnx26p_vhbamisc.o
-
CC [M] /root/linuxrdac-09.03.0C05.0331/mppLnx26p_vhbatask.o
-
CC [M] /root/linuxrdac-09.03.0C05.0331/mppLnx26p_vhba.o
-
CC [M] /root/linuxrdac-09.03.0C05.0331/mppLnx26p_vhbaproc.o
-
CC [M] /root/linuxrdac-09.03.0C05.0331/mppLnx26p_vhbalib.o
-
CC [M] /root/linuxrdac-09.03.0C05.0331/mppLnx26p_vhbaio.o
-
LD [M] /root/linuxrdac-09.03.0C05.0331/mppUpper.o
-
LD [M] /root/linuxrdac-09.03.0C05.0331/mppVhba.o
-
Building modules, stage 2.
-
MODPOST
-
CC /root/linuxrdac-09.03.0C05.0331/mppUpper.mod.o
-
LD [M] /root/linuxrdac-09.03.0C05.0331/mppUpper.ko
-
CC /root/linuxrdac-09.03.0C05.0331/mppVhba.mod.o
-
LD [M] /root/linuxrdac-09.03.0C05.0331/mppVhba.ko
-
make[1]: Leaving directory `/usr/src/kernels/2.6.18-128.el5-x86_64'
-
make[1]: Entering directory `/usr/src/kernels/2.6.18-128.el5-x86_64'
-
Building modules, stage 2.
-
MODPOST
-
make[1]: Leaving directory `/usr/src/kernels/2.6.18-128.el5-x86_64'
-
gcc -I/root/linuxrdac-09.03.0C05.0331 -I/root/linuxrdac-09.03.0C05.0331/mpp_linux_headers/ -I/root/linuxrdac-09.03.0C05.0331/mpp_linux_sys_headers/ -c ./utility/mppUtil.c -o mppUtil.o
-
/bin/bash ./genfileattributes bld
-
gcc -I/root/linuxrdac-09.03.0C05.0331 -I/root/linuxrdac-09.03.0C05.0331/mpp_linux_headers/ -I/root/linuxrdac-09.03.0C05.0331/mpp_linux_sys_headers/ -c ./utility/mppUtil26p_sysdep.c -o mppUtilSysdep.o
-
gcc mppUtil.o mppUtilSysdep.o -o mppUtil
-
gcc -o genuniqueid genuniqueid.c
-
-
[root@test01 linuxrdac-09.03.0C05.0331]# make install
-
make[1]: Entering directory `/usr/src/kernels/2.6.18-128.el5-x86_64'
-
Building modules, stage 2.
-
MODPOST
-
make[1]: Leaving directory `/usr/src/kernels/2.6.18-128.el5-x86_64'
-
make[1]: Entering directory `/usr/src/kernels/2.6.18-128.el5-x86_64'
-
Building modules, stage 2.
-
MODPOST
-
make[1]: Leaving directory `/usr/src/kernels/2.6.18-128.el5-x86_64'
-
/bin/bash ./genfileattributes bld
-
gcc -I/root/linuxrdac-09.03.0C05.0331 -I/root/linuxrdac-09.03.0C05.0331/mpp_linux_headers/ -I/root/linuxrdac-09.03.0C05.0331/mpp_linux_sys_headers/ -c ./utility/mppUtil26p_sysdep.c -o mppUtilSysdep.o
-
gcc mppUtil.o mppUtilSysdep.o -o mppUtil
-
Checking Host Adapter Configuration...
-
Detected 2 Emulex Host Adapter Port(s) on the system
-
Detected 2 QLogic Host Adapter Port(s) on the system
-
Host Adapters from different supported vendors co-exists on your system.
-
Please wait while we modify the system configuration files.
-
Your kernel version is 2.6.18-128.el5
-
Preparing to install MPP driver against this kernel version...
-
Generating module dependencies...
-
Creating new MPP initrd image...
-
You must now edit your boot loader configuration file, /boot/grub/menu.lst, to
-
add a new boot menu, which uses mpp-2.6.18-128.el5.img as the initrd image.
-
Now Reboot the system for MPP to take effect.
-
The new boot menu entry should look something like this (note that it may
-
vary with different system configuration):
-
-
...
-
-
title Red Hat Linux (2.6.18-128.el5) with MPP support
-
root (hd0,5)
-
kernel /vmlinuz-2.6.18-128.el5 ro root=LABEL=RH9
-
initrd /mpp-2.6.18-128.el5.img
-
...
-
MPP driver package has been successfully installed on your system.
请注意观察安装完成后的最后几行内容,意思是在/boot目录下会产生一个mpp-
.img 的文件,并提示必须修改启动列表的文件"You must now edit your boot loader configuration file, /boot/grub/menu.lst…."。
6. 编辑启动项菜单,/boot/grub/menu.lst,将mpp-.img加入启动项,为了避免因错误无法启动,可以在下面新增一个title。红色字体为修改或增加的部分。
- # grub.conf generated by anaconda
-
#
-
# Note that you do not have to rerun grub after making changes to this file
-
# NOTICE: You have a /boot partition. This means that
-
# all kernel and initrd paths are relative to /boot/, eg.
-
# root (hd0,0)
-
# kernel /vmlinuz-version ro root=/dev/rootlv/rootvg
-
# initrd /initrd-version.img
-
#boot=/dev/cciss/c0d0
-
default=1
-
timeout=5
-
splashimage=(hd0,0)/grub/splash.xpm.gz
-
hiddenmenu
-
title Red Hat Enterprise Linux Server (2.6.18-128.el5)
-
root (hd0,0)
-
kernel /vmlinuz-2.6.18-128.el5 ro root=/dev/rootvg/rootlv rhgb quiet
-
initrd /initrd-2.6.18-128.el5.img
-
title Red Hat Enterprise Linux Server (2.6.18-128.el5) with RDAC
-
root (hd0,0)
-
kernel /vmlinuz-2.6.18-128.el5 ro root=/dev/rootvg/rootlv rhgb quiet
-
initrd /initrd-2.6.18-128.el5.img
-
initrd /mpp-2.6.18-128.el5.img
7. reboot Linux系统,并使用新的mpp启动选项
8. 检查确认相关Module entries是否安装,红色部分是必须要有的
- [root@test01 ~]# lsmod
-
Module Size Used by
-
autofs4 57033 2
-
hidp 83521 2
-
rfcomm 104809 0
-
l2cap 89281 10 hidp,rfcomm
-
bluetooth 118597 5 hidp,rfcomm,l2cap
-
sunrpc 197897 1
-
dm_multipath 55257 0
-
scsi_dh 41665 1 dm_multipath
-
video 53197 0
-
hwmon 36553 0
-
backlight 39873 1 video
-
sbs 49921 0
-
i2c_ec 38593 1 sbs
-
i2c_core 56129 1 i2c_ec
-
button 40545 0
-
battery 43849 0
-
asus_acpi 50917 0
-
acpi_memhotplug 40133 0
-
ac 38729 0
-
ipv6 424609 78
-
xfrm_nalgo 43333 1 ipv6
-
crypto_api 42945 1 xfrm_nalgo
-
parport_pc 62312 0
-
lp 47121 0
-
parport 73165 2 parport_pc,lp
-
joydev 43969 0
-
sr_mod 50789 0
-
cdrom 68713 1 sr_mod
-
shpchp 70765 0
-
serio_raw 40517 0
-
pcspkr 36289 0
-
bnx2 210249 0
-
dm_raid45 99025 0
-
dm_message 36161 1 dm_raid45
-
dm_region_hash 46145 1 dm_raid45
-
dm_mem_cache 38977 1 dm_raid45
-
dm_snapshot 51465 0
-
dm_zero 35265 0
-
dm_mirror 53065 0
-
dm_log 44865 3 dm_raid45,dm_region_hash,dm_mirror
-
dm_mod 100369 11 dm_multipath,dm_raid45,dm_snapshot,dm_zero,dm_mirror,dm_log
-
mppVhba 165152 0
-
qla2xxx 1107173 0
-
lpfc 352909 0
-
scsi_transport_fc 73801 2 qla2xxx,lpfc
-
ata_piix 56901 0
-
libata 208721 1 ata_piix
-
cciss 99401 5
-
mppUpper 163708 1 mppVhba
-
sg 69865 0
-
sd_mod 56385 0
-
scsi_mod 196569 11 scsi_dh,sr_mod,mppVhba,qla2xxx,lpfc,scsi_transport_fc,libata,cciss,mppUpper,sg,sd_mod
-
ext3 168017 2
-
jbd 94257 1 ext3
-
uhci_hcd 57433 0
-
ohci_hcd 55925 0
-
ehci_hcd 65741 0
9. 查看MPP版本
- [root@test01 ~]# mppUtil -V
-
Linux MPP Driver Version: 09.03.0C05.0331
10. 到这里为止,RDAC算是安装完成了,接下去要做的就是在DS4000中划盘和mapping,我在这里就不详细描述了,只是需要强调一点:host type要选择LNXCL或者是LNXCLVMWARE,而不要选择Linux,因为需要使得ADT/AVT是Disable的,让失效切换发生在控制器级别,而不是逻辑卷级别。
11. 在存储中完成LUN mapping后,就可以在Linux中重新扫描和发现LUN
RHEL中查看光纤卡的WWN的方法
-
[root@test01 ~]# cat /sys/class/fc_host/host2/port_name
-
0x10000000c9884666
-
[root@test01 ~]# cat /sys/class/fc_host/host3/port_name
-
0x10000000c9884667
-
[root@test01 ~]# cat /sys/class/fc_host/host4/port_name
-
0x5001438004c3f330
-
[root@test01 ~]# cat /sys/class/fc_host/host5/port_name
-
0x5001438004c3f331
- [root@test01 ~]# hot_add或者mppBusRescan
12. 使用相关MPP(RDAC)命令
- [root@test01 ~]# mppUtil –a
-
Hostname = test01
-
Domainname = (none)
-
Time = GMT 06/21/2011 00:45:06
-
-------------------------------------------------------------------------------------------------------
-
Info of Array Module's seen by this Host.
-
-------------------------------------------------------------------------------------------------------
-
ID WWN Type Name
-
-------------------------------------------------------------------------------------------------------
-
0 600a0b8000564856000000004cbf5555 FC test_DS4700
[root@test01 ~]#
mppUtil -a test_DS4700
Hostname = test01
Domainname = (none)
Time = GMT 06/21/2011 00:45:06
MPP Information:
----------------
ModuleName: test01_DS4700 SingleController: N
VirtualTargetID: 0x000 ScanTriggered: N
ObjectCount: 0x000 AVTEnabled: N
WWN: 600a0b8000564856000000004cbf5555 RestoreCfg: N
ModuleHandle: none Page2CSubPage: Y
FirmwareVersion: 7.60.28.xx FailoverMethod: C
ScanTaskState: 0x00000000
LBPolicy: LeastQueueDepth
Controller 'A' Status:
-----------------------
ControllerHandle: none ControllerPresent: Y
UTMLunExists: N Failed: N
NumberOfPaths: 1 FailoverInProg: N
ServiceMode: N
Path #1
---------
DirectoryVertex: present Present: Y
PathState: OPTIMAL
PathId: 77020000 (hostId: 2, channelId: 0, targetId: 0)
Controller 'B' Status:
-----------------------
ControllerHandle: none ControllerPresent: Y
UTMLunExists: N Failed: N
NumberOfPaths: 1 FailoverInProg: N
ServiceMode: N
Path #1
---------
DirectoryVertex: present Present: Y
PathState: OPTIMAL
PathId: 77040000 (hostId: 4, channelId: 0, targetId: 0)
Lun Information
---------------
Lun #0 - WWN: 600a0b8000513fdc000009934dfb0c4f
----------------
LunObject: present CurrentOwningPath: A
RemoveEligible: N BootOwningPath: A
NotConfigured: N PreferredPath: A
DevState: OPTIMAL ReportedPresent: Y
ReportedMissing: N
NeedsReservationCheck: N
TASBitSet: Y
NotReady: N
Busy: N
Quiescent: N
VD_Ownership_Transfer_Attempt_Count: 0
Controller 'A' Path
--------------------
NumLunObjects: 1 RoundRobinIndex: 0
Path #1: LunPathDevice: present
DevState: OPTIMAL
RemoveState: 0x0 StartState: 0x1 PowerState: 0x0
Controller 'B' Path
--------------------
NumLunObjects: 1 RoundRobinIndex: 0
Path #1: LunPathDevice: present
DevState: OPTIMAL
RemoveState: 0x0 StartState: 0x1 PowerState: 0x0
Lun #1 - WWN: 600a0b80005140100000094d4dfb0d80
----------------
LunObject: present CurrentOwningPath: B
RemoveEligible: N BootOwningPath: B
NotConfigured: N PreferredPath: B
DevState: OPTIMAL ReportedPresent: Y
ReportedMissing: N
NeedsReservationCheck: N
TASBitSet: Y
NotReady: N
Busy: N
Quiescent: N
VD_Ownership_Transfer_Attempt_Count: 0
Controller 'A' Path
--------------------
NumLunObjects: 1 RoundRobinIndex: 0
Path #1: LunPathDevice: present
DevState: OPTIMAL
RemoveState: 0x0 StartState: 0x1 PowerState: 0x0
Controller 'B' Path
--------------------
NumLunObjects: 1 RoundRobinIndex: 0
Path #1: LunPathDevice: present
DevState: OPTIMAL
RemoveState: 0x0 StartState: 0x1 PowerState: 0x0
Lun #2 - WWN: 600a0b8000514010000009474dfb0d25
----------------
LunObject: present CurrentOwningPath: B
RemoveEligible: N BootOwningPath: B
NotConfigured: N PreferredPath: B
DevState: OPTIMAL ReportedPresent: Y
ReportedMissing: N
NeedsReservationCheck: N
TASBitSet: Y
NotReady: N
Busy: N
Quiescent: N
VD_Ownership_Transfer_Attempt_Count: 0
Controller 'A' Path
--------------------
NumLunObjects: 1 RoundRobinIndex: 0
Path #1: LunPathDevice: present
DevState: OPTIMAL
RemoveState: 0x0 StartState: 0x1 PowerState: 0x0
Controller 'B' Path
--------------------
NumLunObjects: 1 RoundRobinIndex: 0
Path #1: LunPathDevice: present
DevState: OPTIMAL
RemoveState: 0x0 StartState: 0x1 PowerState: 0x0
Lun #3 - WWN: 600a0b8000514010000009494dfb0d35
----------------
LunObject: present CurrentOwningPath: B
RemoveEligible: N BootOwningPath: B
NotConfigured: N PreferredPath: B
DevState: OPTIMAL ReportedPresent: Y
ReportedMissing: N
NeedsReservationCheck: N
TASBitSet: Y
NotReady: N
Busy: N
Quiescent: N
VD_Ownership_Transfer_Attempt_Count: 0
Controller 'A' Path
--------------------
NumLunObjects: 1 RoundRobinIndex: 0
Path #1: LunPathDevice: present
DevState: OPTIMAL
RemoveState: 0x0 StartState: 0x1 PowerState: 0x0
Controller 'B' Path
--------------------
NumLunObjects: 1 RoundRobinIndex: 0
Path #1: LunPathDevice: present
DevState: OPTIMAL
RemoveState: 0x0 StartState: 0x1 PowerState: 0x0
Lun #4 - WWN: 600a0b80005140100000094b4dfb0d5f
----------------
LunObject: present CurrentOwningPath: B
RemoveEligible: N BootOwningPath: B
NotConfigured: N PreferredPath: B
DevState: OPTIMAL ReportedPresent: Y
ReportedMissing: N
NeedsReservationCheck: N
TASBitSet: Y
NotReady: N
Busy: N
Quiescent: N
VD_Ownership_Transfer_Attempt_Count: 0
Controller 'A' Path
--------------------
NumLunObjects: 1 RoundRobinIndex: 0
Path #1: LunPathDevice: present
DevState: OPTIMAL
RemoveState: 0x0 StartState: 0x1 PowerState: 0x0
Controller 'B' Path
--------------------
NumLunObjects: 1 RoundRobinIndex: 0
Path #1: LunPathDevice: present
DevState: OPTIMAL
RemoveState: 0x0 StartState: 0x1 PowerState: 0x0
Lun #5 - WWN: 600a0b8000513fdc000009954dfb0c79
----------------
LunObject: present CurrentOwningPath: A
RemoveEligible: N BootOwningPath: A
NotConfigured: N PreferredPath: A
DevState: OPTIMAL ReportedPresent: Y
ReportedMissing: N
NeedsReservationCheck: N
TASBitSet: Y
NotReady: N
Busy: N
Quiescent: N
VD_Ownership_Transfer_Attempt_Count: 0
Controller 'A' Path
--------------------
NumLunObjects: 1 RoundRobinIndex: 0
Path #1: LunPathDevice: present
DevState: OPTIMAL
RemoveState: 0x0 StartState: 0x1 PowerState: 0x0
Controller 'B' Path
--------------------
NumLunObjects: 1 RoundRobinIndex: 0
Path #1: LunPathDevice: present
DevState: OPTIMAL
RemoveState: 0x0 StartState: 0x1 PowerState: 0x0
- [root@test01 ~]# /opt/mpp/lsvdev
-
Array Name Lun sd device
-
-------------------------------------
-
test_DS4700 0 -> /dev/sda
-
test_DS4700 1 -> /dev/sdb
-
test_DS4700 2 -> /dev/sdc
-
test_DS4700 3 -> /dev/sdd
-
test_DS4700 4 -> /dev/sde
-
test_DS4700 5 -> /dev/sdf
-
-
[root@test01 ~]# ls -lR /proc/mpp
-
/proc/mpp:
-
total 0
-
dr-xr-xr-x 4 root root 0 Jun 21 08:46 test_DS4700
-
-
/proc/mpp/test_DS4700:
-
total 0
-
dr-xr-xr-x 3 root root 0 Jun 21 08:46 controllerA
-
dr-xr-xr-x 3 root root 0 Jun 21 08:46 controllerB
-
-rw-r--r-- 1 root root 0 Jun 21 08:46 virtualLun0
-
-rw-r--r-- 1 root root 0 Jun 21 08:46 virtualLun1
-
-rw-r--r-- 1 root root 0 Jun 21 08:46 virtualLun2
-
-rw-r--r-- 1 root root 0 Jun 21 08:46 virtualLun3
-
-rw-r--r-- 1 root root 0 Jun 21 08:46 virtualLun4
-
-rw-r--r-- 1 root root 0 Jun 21 08:46 virtualLun5
-
-
/proc/mpp/test_DS4700/controllerA:
-
total 0
-
dr-xr-xr-x 2 root root 0 Jun 21 08:46 lpfc_h2c0t0
-
-
/proc/mpp/test_DS4700/controllerA/lpfc_h2c0t0:
-
total 0
-
-rw-r--r-- 1 root root 0 Jun 21 08:46 LUN0
-
-rw-r--r-- 1 root root 0 Jun 21 08:46 LUN1
-
-rw-r--r-- 1 root root 0 Jun 21 08:46 LUN2
-
-rw-r--r-- 1 root root 0 Jun 21 08:46 LUN3
-
-rw-r--r-- 1 root root 0 Jun 21 08:46 LUN4
-
-rw-r--r-- 1 root root 0 Jun 21 08:46 LUN5
-
-
/proc/mpp/test_DS4700/controllerB:
-
total 0
-
dr-xr-xr-x 2 root root 0 Jun 21 08:46 qla2xxx_h4c0t0
-
-
/proc/mpp/test_DS4700/controllerB/qla2xxx_h4c0t0:
-
total 0
-
-rw-r--r-- 1 root root 0 Jun 21 08:46 LUN0
-
-rw-r--r-- 1 root root 0 Jun 21 08:46 LUN1
-
-rw-r--r-- 1 root root 0 Jun 21 08:46 LUN2
-
-rw-r--r-- 1 root root 0 Jun 21 08:46 LUN3
-
-rw-r--r-- 1 root root 0 Jun 21 08:46 LUN4
-
-rw-r--r-- 1 root root 0 Jun 21 08:46 LUN5
阅读(6041) | 评论(0) | 转发(0) |