Chinaunix首页 | 论坛 | 博客
  • 博客访问: 396962
  • 博文数量: 36
  • 博客积分: 1998
  • 博客等级: 上尉
  • 技术积分: 486
  • 用 户 组: 普通用户
  • 注册时间: 2006-08-23 13:48
文章分类

全部博文(36)

文章存档

2012年(11)

2011年(13)

2010年(12)

分类: 服务器与存储

2011-06-21 09:16:08

    IBM DS4000/5000系列的存储是一款销量不错的中端存储产品,这是一款OEM LSI公司的产品,不过该存储所属的LSI Engenio 中端存储在2011年3月已经被Netapp收购了(IT界也挺乱的^_^)。光纤存储一般都用于UNIX环境,在Linux上的使用,尤其是需要结合多路径软件,相对来说就比较少,参考资料总是很有限。我将自己在RHEL 5.3系统上安装使用多路径软件RDAC的过程记录了下来,希望有一定的参考价值。

1. 使用 uname -r 命令查看当前Linux系统的内核版本

  1. [root@test01 ~]# uname -r
  2. 2.6.18-128.el5
2. 下载RDAC驱动,登陆
,,找到与当前Linux系统内核版本一致的RDAC驱动。
根据我实验中的版本,选择下载了Version 09.03.0C05.0331(rdac-LINUX-09.03.0C05.0331-source.tar.gz)

3. 将rdac的安装包上传至~目录(root用户根目录)

  1. [root@test01 ~]# pwd
  2. /root
  3. [root@test01 ~]# ls -l|grep rdac
  4. -rw-r--r-- 1 root root 399735 Jun 20 19:33 rdac-LINUX-09.03.0C05.0331-source.tar.gz
4. 解压源码包

  1. [root@test01 ~]# tar -zxvf rdac-LINUX-09.03.0C05.0331-source.tar.gz
  2. [root@test01 ~]# ls -l|grep rdac
  3. drwxr-xr-x 5 root root 4096 Apr 15 2010 linuxrdac-09.03.0C05.0331
  4. -rw-r--r-- 1 root root 399735 Jun 20 19:33 rdac-LINUX-09.03.0C05.0331-source.tar.gz
5. 参考解压出来的目录中的Readme.txt文件,编译并安装RDAC软件
Linux一定要安装下面的包(此段内容未经我验证):
gcc
glibc-devel
kernel-headers
glibc-headers
libgomp
kernel-devel 或 kernel-xen-devel(如果使用启用 Xen 的内核的话)

  1. [root@test01 linuxrdac-09.03.0C05.0331]# make
  2. make[1]: Entering directory `/usr/src/kernels/2.6.18-128.el5-x86_64'
  3. CC [M] /root/linuxrdac-09.03.0C05.0331/MPP_hba.o
  4. CC [M] /root/linuxrdac-09.03.0C05.0331/mppLnx26p_upper.o
  5. CC [M] /root/linuxrdac-09.03.0C05.0331/mppLnx26p_sysdep.o
  6. CC [M] /root/linuxrdac-09.03.0C05.0331/mppCmn_s2tos3.o
  7. CC [M] /root/linuxrdac-09.03.0C05.0331/mppCmn_SysInterface.o
  8. CC [M] /root/linuxrdac-09.03.0C05.0331/mppLnx26p_vhbamisc.o
  9. CC [M] /root/linuxrdac-09.03.0C05.0331/mppLnx26p_vhbatask.o
  10. CC [M] /root/linuxrdac-09.03.0C05.0331/mppLnx26p_vhba.o
  11. CC [M] /root/linuxrdac-09.03.0C05.0331/mppLnx26p_vhbaproc.o
  12. CC [M] /root/linuxrdac-09.03.0C05.0331/mppLnx26p_vhbalib.o
  13. CC [M] /root/linuxrdac-09.03.0C05.0331/mppLnx26p_vhbaio.o
  14. LD [M] /root/linuxrdac-09.03.0C05.0331/mppUpper.o
  15. LD [M] /root/linuxrdac-09.03.0C05.0331/mppVhba.o
  16. Building modules, stage 2.
  17. MODPOST
  18. CC /root/linuxrdac-09.03.0C05.0331/mppUpper.mod.o
  19. LD [M] /root/linuxrdac-09.03.0C05.0331/mppUpper.ko
  20. CC /root/linuxrdac-09.03.0C05.0331/mppVhba.mod.o
  21. LD [M] /root/linuxrdac-09.03.0C05.0331/mppVhba.ko
  22. make[1]: Leaving directory `/usr/src/kernels/2.6.18-128.el5-x86_64'
  23. make[1]: Entering directory `/usr/src/kernels/2.6.18-128.el5-x86_64'
  24. Building modules, stage 2.
  25. MODPOST
  26. make[1]: Leaving directory `/usr/src/kernels/2.6.18-128.el5-x86_64'
  27. gcc -I/root/linuxrdac-09.03.0C05.0331 -I/root/linuxrdac-09.03.0C05.0331/mpp_linux_headers/ -I/root/linuxrdac-09.03.0C05.0331/mpp_linux_sys_headers/ -c ./utility/mppUtil.c -o mppUtil.o
  28. /bin/bash ./genfileattributes bld
  29. gcc -I/root/linuxrdac-09.03.0C05.0331 -I/root/linuxrdac-09.03.0C05.0331/mpp_linux_headers/ -I/root/linuxrdac-09.03.0C05.0331/mpp_linux_sys_headers/ -c ./utility/mppUtil26p_sysdep.c -o mppUtilSysdep.o
  30. gcc mppUtil.o mppUtilSysdep.o -o mppUtil
  31. gcc -o genuniqueid genuniqueid.c

  32. [root@test01 linuxrdac-09.03.0C05.0331]# make install
  33. make[1]: Entering directory `/usr/src/kernels/2.6.18-128.el5-x86_64'
  34. Building modules, stage 2.
  35. MODPOST
  36. make[1]: Leaving directory `/usr/src/kernels/2.6.18-128.el5-x86_64'
  37. make[1]: Entering directory `/usr/src/kernels/2.6.18-128.el5-x86_64'
  38. Building modules, stage 2.
  39. MODPOST
  40. make[1]: Leaving directory `/usr/src/kernels/2.6.18-128.el5-x86_64'
  41. /bin/bash ./genfileattributes bld
  42. gcc -I/root/linuxrdac-09.03.0C05.0331 -I/root/linuxrdac-09.03.0C05.0331/mpp_linux_headers/ -I/root/linuxrdac-09.03.0C05.0331/mpp_linux_sys_headers/ -c ./utility/mppUtil26p_sysdep.c -o mppUtilSysdep.o
  43. gcc mppUtil.o mppUtilSysdep.o -o mppUtil
  44. Checking Host Adapter Configuration...
  45. Detected 2 Emulex Host Adapter Port(s) on the system
  46. Detected 2 QLogic Host Adapter Port(s) on the system
  47. Host Adapters from different supported vendors co-exists on your system.
  48. Please wait while we modify the system configuration files.
  49. Your kernel version is 2.6.18-128.el5
  50. Preparing to install MPP driver against this kernel version...
  51. Generating module dependencies...
  52. Creating new MPP initrd image...
  53. You must now edit your boot loader configuration file, /boot/grub/menu.lst, to
  54. add a new boot menu, which uses mpp-2.6.18-128.el5.img as the initrd image.
  55. Now Reboot the system for MPP to take effect.
  56. The new boot menu entry should look something like this (note that it may
  57. vary with different system configuration):

  58. ...

  59. title Red Hat Linux (2.6.18-128.el5) with MPP support
  60. root (hd0,5)
  61. kernel /vmlinuz-2.6.18-128.el5 ro root=LABEL=RH9
  62. initrd /mpp-2.6.18-128.el5.img
  63. ...
  64. MPP driver package has been successfully installed on your system.
请注意观察安装完成后的最后几行内容,意思是在/boot目录下会产生一个mpp-.img 的文件,并提示必须修改启动列表的文件"You must now edit your boot loader configuration file, /boot/grub/menu.lst…."。

6. 编辑启动项菜单,/boot/grub/menu.lst,将mpp-.img加入启动项,为了避免因错误无法启动,可以在下面新增一个title。红色字体为修改或增加的部分。

  1. # grub.conf generated by anaconda
  2. #
  3. # Note that you do not have to rerun grub after making changes to this file
  4. # NOTICE: You have a /boot partition. This means that
  5. # all kernel and initrd paths are relative to /boot/, eg.
  6. # root (hd0,0)
  7. # kernel /vmlinuz-version ro root=/dev/rootlv/rootvg
  8. # initrd /initrd-version.img
  9. #boot=/dev/cciss/c0d0
  10. default=1
  11. timeout=5
  12. splashimage=(hd0,0)/grub/splash.xpm.gz
  13. hiddenmenu
  14. title Red Hat Enterprise Linux Server (2.6.18-128.el5)
  15.         root (hd0,0)
  16.         kernel /vmlinuz-2.6.18-128.el5 ro root=/dev/rootvg/rootlv rhgb quiet
  17.         initrd /initrd-2.6.18-128.el5.img
  18. title Red Hat Enterprise Linux Server (2.6.18-128.el5) with RDAC
  19.         root (hd0,0)
  20.         kernel /vmlinuz-2.6.18-128.el5 ro root=/dev/rootvg/rootlv rhgb quiet
  21.         initrd /initrd-2.6.18-128.el5.img
  22.         initrd /mpp-2.6.18-128.el5.img
7. reboot Linux系统,并使用新的mpp启动选项

8. 检查确认相关Module entries是否安装,红色部分是必须要有的

  1. [root@test01 ~]# lsmod
  2. Module Size Used by
  3. autofs4 57033 2
  4. hidp 83521 2
  5. rfcomm 104809 0
  6. l2cap 89281 10 hidp,rfcomm
  7. bluetooth 118597 5 hidp,rfcomm,l2cap
  8. sunrpc 197897 1
  9. dm_multipath 55257 0
  10. scsi_dh 41665 1 dm_multipath
  11. video 53197 0
  12. hwmon 36553 0
  13. backlight 39873 1 video
  14. sbs 49921 0
  15. i2c_ec 38593 1 sbs
  16. i2c_core 56129 1 i2c_ec
  17. button 40545 0
  18. battery 43849 0
  19. asus_acpi 50917 0
  20. acpi_memhotplug 40133 0
  21. ac 38729 0
  22. ipv6 424609 78
  23. xfrm_nalgo 43333 1 ipv6
  24. crypto_api 42945 1 xfrm_nalgo
  25. parport_pc 62312 0
  26. lp 47121 0
  27. parport 73165 2 parport_pc,lp
  28. joydev 43969 0
  29. sr_mod 50789 0
  30. cdrom 68713 1 sr_mod
  31. shpchp 70765 0
  32. serio_raw 40517 0
  33. pcspkr 36289 0
  34. bnx2 210249 0
  35. dm_raid45 99025 0
  36. dm_message 36161 1 dm_raid45
  37. dm_region_hash 46145 1 dm_raid45
  38. dm_mem_cache 38977 1 dm_raid45
  39. dm_snapshot 51465 0
  40. dm_zero 35265 0
  41. dm_mirror 53065 0
  42. dm_log 44865 3 dm_raid45,dm_region_hash,dm_mirror
  43. dm_mod 100369 11 dm_multipath,dm_raid45,dm_snapshot,dm_zero,dm_mirror,dm_log
  44. mppVhba 165152 0
  45. qla2xxx 1107173 0
  46. lpfc 352909 0
  47. scsi_transport_fc 73801 2 qla2xxx,lpfc
  48. ata_piix 56901 0
  49. libata 208721 1 ata_piix
  50. cciss 99401 5
  51. mppUpper 163708 1 mppVhba
  52. sg 69865 0
  53. sd_mod 56385 0
  54. scsi_mod 196569 11 scsi_dh,sr_mod,mppVhba,qla2xxx,lpfc,scsi_transport_fc,libata,cciss,mppUpper,sg,sd_mod
  55. ext3 168017 2
  56. jbd 94257 1 ext3
  57. uhci_hcd 57433 0
  58. ohci_hcd 55925 0
  59. ehci_hcd 65741 0
 9. 查看MPP版本

  1. [root@test01 ~]# mppUtil -V
  2. Linux MPP Driver Version: 09.03.0C05.0331
10. 到这里为止,RDAC算是安装完成了,接下去要做的就是在DS4000中划盘和mapping,我在这里就不详细描述了,只是需要强调一点:host type要选择LNXCL或者是LNXCLVMWARE,而不要选择Linux,因为需要使得ADT/AVT是Disable的,让失效切换发生在控制器级别,而不是逻辑卷级别。

11. 在存储中完成LUN mapping后,就可以在Linux中重新扫描和发现LUN

RHEL中查看光纤卡的WWN的方法

  1. [root@test01 ~]# cat /sys/class/fc_host/host2/port_name
  2. 0x10000000c9884666
  3. [root@test01 ~]# cat /sys/class/fc_host/host3/port_name
  4. 0x10000000c9884667
  5. [root@test01 ~]# cat /sys/class/fc_host/host4/port_name
  6. 0x5001438004c3f330
  7. [root@test01 ~]# cat /sys/class/fc_host/host5/port_name
  8. 0x5001438004c3f331
  1. [root@test01 ~]# hot_add或者mppBusRescan
12. 使用相关MPP(RDAC)命令
  1. [root@test01 ~]# mppUtil –a
  2. Hostname = test01
  3. Domainname = (none)
  4. Time = GMT 06/21/2011 00:45:06
  5. -------------------------------------------------------------------------------------------------------
  6. Info of Array Module's seen by this Host.
  7. -------------------------------------------------------------------------------------------------------
  8. ID WWN Type Name
  9. -------------------------------------------------------------------------------------------------------
  10. 0 600a0b8000564856000000004cbf5555 FC test_DS4700
 [root@test01 ~]# mppUtil -a test_DS4700
Hostname    = test01
Domainname  = (none)
Time        = GMT 06/21/2011 00:45:06

MPP Information:
----------------
      ModuleName: test01_DS4700                             SingleController: N
 VirtualTargetID: 0x000                                       ScanTriggered: N
     ObjectCount: 0x000                                          AVTEnabled: N
             WWN: 600a0b8000564856000000004cbf5555               RestoreCfg: N
    ModuleHandle: none                                        Page2CSubPage: Y
 FirmwareVersion: 7.60.28.xx                                 FailoverMethod: C
   ScanTaskState: 0x00000000
        LBPolicy: LeastQueueDepth


Controller 'A' Status:
-----------------------
ControllerHandle: none                                    ControllerPresent: Y
    UTMLunExists: N                                                  Failed: N
   NumberOfPaths: 1                                          FailoverInProg: N
                                                                ServiceMode: N

    Path #1
    ---------
 DirectoryVertex: present                                           Present: Y
       PathState: OPTIMAL             
          PathId: 77020000 (hostId: 2, channelId: 0, targetId: 0)


Controller 'B' Status:
-----------------------
ControllerHandle: none                                    ControllerPresent: Y
    UTMLunExists: N                                                  Failed: N
   NumberOfPaths: 1                                          FailoverInProg: N
                                                                ServiceMode: N

    Path #1
    ---------
 DirectoryVertex: present                                           Present: Y
       PathState: OPTIMAL             
          PathId: 77040000 (hostId: 4, channelId: 0, targetId: 0)



Lun Information
---------------
    Lun #0 - WWN: 600a0b8000513fdc000009934dfb0c4f
    ----------------
       LunObject: present                                 CurrentOwningPath: A
  RemoveEligible: N                                          BootOwningPath: A
   NotConfigured: N                                           PreferredPath: A
        DevState: OPTIMAL                                   ReportedPresent: Y
                                                            ReportedMissing: N
                                                      NeedsReservationCheck: N
                                                                  TASBitSet: Y
                                                                   NotReady: N
                                                                       Busy: N
                                                                  Quiescent: N
                                        VD_Ownership_Transfer_Attempt_Count: 0

    Controller 'A' Path
    --------------------
   NumLunObjects: 1                                         RoundRobinIndex: 0
         Path #1: LunPathDevice: present          
                       DevState: OPTIMAL             
                    RemoveState: 0x0  StartState: 0x1  PowerState: 0x0

    Controller 'B' Path
    --------------------
   NumLunObjects: 1                                         RoundRobinIndex: 0
         Path #1: LunPathDevice: present          
                       DevState: OPTIMAL             
                    RemoveState: 0x0  StartState: 0x1  PowerState: 0x0


    Lun #1 - WWN: 600a0b80005140100000094d4dfb0d80
    ----------------
       LunObject: present                                 CurrentOwningPath: B
  RemoveEligible: N                                          BootOwningPath: B
   NotConfigured: N                                           PreferredPath: B
        DevState: OPTIMAL                                   ReportedPresent: Y
                                                            ReportedMissing: N
                                                      NeedsReservationCheck: N
                                                                  TASBitSet: Y
                                                                   NotReady: N
                                                                       Busy: N
                                                                  Quiescent: N
                                        VD_Ownership_Transfer_Attempt_Count: 0

    Controller 'A' Path
    --------------------
   NumLunObjects: 1                                         RoundRobinIndex: 0
         Path #1: LunPathDevice: present          
                       DevState: OPTIMAL             
                    RemoveState: 0x0  StartState: 0x1  PowerState: 0x0

    Controller 'B' Path
    --------------------
   NumLunObjects: 1                                         RoundRobinIndex: 0
         Path #1: LunPathDevice: present          
                       DevState: OPTIMAL             
                    RemoveState: 0x0  StartState: 0x1  PowerState: 0x0


    Lun #2 - WWN: 600a0b8000514010000009474dfb0d25
    ----------------
       LunObject: present                                 CurrentOwningPath: B
  RemoveEligible: N                                          BootOwningPath: B
   NotConfigured: N                                           PreferredPath: B
        DevState: OPTIMAL                                   ReportedPresent: Y
                                                            ReportedMissing: N
                                                      NeedsReservationCheck: N
                                                                  TASBitSet: Y
                                                                   NotReady: N
                                                                       Busy: N
                                                                  Quiescent: N
                                        VD_Ownership_Transfer_Attempt_Count: 0

    Controller 'A' Path
    --------------------
   NumLunObjects: 1                                         RoundRobinIndex: 0
         Path #1: LunPathDevice: present          
                       DevState: OPTIMAL             
                    RemoveState: 0x0  StartState: 0x1  PowerState: 0x0

    Controller 'B' Path
    --------------------
   NumLunObjects: 1                                         RoundRobinIndex: 0
         Path #1: LunPathDevice: present          
                       DevState: OPTIMAL             
                    RemoveState: 0x0  StartState: 0x1  PowerState: 0x0


    Lun #3 - WWN: 600a0b8000514010000009494dfb0d35
    ----------------
       LunObject: present                                 CurrentOwningPath: B
  RemoveEligible: N                                          BootOwningPath: B
   NotConfigured: N                                           PreferredPath: B
        DevState: OPTIMAL                                   ReportedPresent: Y
                                                            ReportedMissing: N
                                                      NeedsReservationCheck: N
                                                                  TASBitSet: Y
                                                                   NotReady: N
                                                                       Busy: N
                                                                  Quiescent: N
                                        VD_Ownership_Transfer_Attempt_Count: 0

    Controller 'A' Path
    --------------------
   NumLunObjects: 1                                         RoundRobinIndex: 0
         Path #1: LunPathDevice: present          
                       DevState: OPTIMAL             
                    RemoveState: 0x0  StartState: 0x1  PowerState: 0x0

    Controller 'B' Path
    --------------------
   NumLunObjects: 1                                         RoundRobinIndex: 0
         Path #1: LunPathDevice: present          
                       DevState: OPTIMAL             
                    RemoveState: 0x0  StartState: 0x1  PowerState: 0x0


    Lun #4 - WWN: 600a0b80005140100000094b4dfb0d5f
    ----------------
       LunObject: present                                 CurrentOwningPath: B
  RemoveEligible: N                                          BootOwningPath: B
   NotConfigured: N                                           PreferredPath: B
        DevState: OPTIMAL                                   ReportedPresent: Y
                                                            ReportedMissing: N
                                                      NeedsReservationCheck: N
                                                                  TASBitSet: Y
                                                                   NotReady: N
                                                                       Busy: N
                                                                  Quiescent: N
                                        VD_Ownership_Transfer_Attempt_Count: 0

    Controller 'A' Path
    --------------------
   NumLunObjects: 1                                         RoundRobinIndex: 0
         Path #1: LunPathDevice: present          
                       DevState: OPTIMAL             
                    RemoveState: 0x0  StartState: 0x1  PowerState: 0x0

    Controller 'B' Path
    --------------------
   NumLunObjects: 1                                         RoundRobinIndex: 0
         Path #1: LunPathDevice: present          
                       DevState: OPTIMAL             
                    RemoveState: 0x0  StartState: 0x1  PowerState: 0x0


    Lun #5 - WWN: 600a0b8000513fdc000009954dfb0c79
    ----------------
       LunObject: present                                 CurrentOwningPath: A
  RemoveEligible: N                                          BootOwningPath: A
   NotConfigured: N                                           PreferredPath: A
        DevState: OPTIMAL                                   ReportedPresent: Y
                                                            ReportedMissing: N
                                                      NeedsReservationCheck: N
                                                                  TASBitSet: Y
                                                                   NotReady: N
                                                                       Busy: N
                                                                  Quiescent: N
                                        VD_Ownership_Transfer_Attempt_Count: 0

    Controller 'A' Path
    --------------------
   NumLunObjects: 1                                         RoundRobinIndex: 0
         Path #1: LunPathDevice: present          
                       DevState: OPTIMAL             
                    RemoveState: 0x0  StartState: 0x1  PowerState: 0x0

    Controller 'B' Path
    --------------------
   NumLunObjects: 1                                         RoundRobinIndex: 0
         Path #1: LunPathDevice: present          
                       DevState: OPTIMAL             
                    RemoveState: 0x0  StartState: 0x1  PowerState: 0x0


  1. [root@test01 ~]# /opt/mpp/lsvdev
  2. Array Name Lun sd device
  3. -------------------------------------
  4. test_DS4700 0 -> /dev/sda
  5. test_DS4700 1 -> /dev/sdb
  6. test_DS4700 2 -> /dev/sdc
  7. test_DS4700 3 -> /dev/sdd
  8. test_DS4700 4 -> /dev/sde
  9. test_DS4700 5 -> /dev/sdf

  10. [root@test01 ~]# ls -lR /proc/mpp
  11. /proc/mpp:
  12. total 0
  13. dr-xr-xr-x 4 root root 0 Jun 21 08:46 test_DS4700

  14. /proc/mpp/test_DS4700:
  15. total 0
  16. dr-xr-xr-x 3 root root 0 Jun 21 08:46 controllerA
  17. dr-xr-xr-x 3 root root 0 Jun 21 08:46 controllerB
  18. -rw-r--r-- 1 root root 0 Jun 21 08:46 virtualLun0
  19. -rw-r--r-- 1 root root 0 Jun 21 08:46 virtualLun1
  20. -rw-r--r-- 1 root root 0 Jun 21 08:46 virtualLun2
  21. -rw-r--r-- 1 root root 0 Jun 21 08:46 virtualLun3
  22. -rw-r--r-- 1 root root 0 Jun 21 08:46 virtualLun4
  23. -rw-r--r-- 1 root root 0 Jun 21 08:46 virtualLun5

  24. /proc/mpp/test_DS4700/controllerA:
  25. total 0
  26. dr-xr-xr-x 2 root root 0 Jun 21 08:46 lpfc_h2c0t0

  27. /proc/mpp/test_DS4700/controllerA/lpfc_h2c0t0:
  28. total 0
  29. -rw-r--r-- 1 root root 0 Jun 21 08:46 LUN0
  30. -rw-r--r-- 1 root root 0 Jun 21 08:46 LUN1
  31. -rw-r--r-- 1 root root 0 Jun 21 08:46 LUN2
  32. -rw-r--r-- 1 root root 0 Jun 21 08:46 LUN3
  33. -rw-r--r-- 1 root root 0 Jun 21 08:46 LUN4
  34. -rw-r--r-- 1 root root 0 Jun 21 08:46 LUN5

  35. /proc/mpp/test_DS4700/controllerB:
  36. total 0
  37. dr-xr-xr-x 2 root root 0 Jun 21 08:46 qla2xxx_h4c0t0

  38. /proc/mpp/test_DS4700/controllerB/qla2xxx_h4c0t0:
  39. total 0
  40. -rw-r--r-- 1 root root 0 Jun 21 08:46 LUN0
  41. -rw-r--r-- 1 root root 0 Jun 21 08:46 LUN1
  42. -rw-r--r-- 1 root root 0 Jun 21 08:46 LUN2
  43. -rw-r--r-- 1 root root 0 Jun 21 08:46 LUN3
  44. -rw-r--r-- 1 root root 0 Jun 21 08:46 LUN4
  45. -rw-r--r-- 1 root root 0 Jun 21 08:46 LUN5





阅读(6041) | 评论(0) | 转发(0) |
给主人留下些什么吧!~~