建立软RAID

建立软RAID

以下大部分信息为实验过程中系统的反馈，可以乎略，主要为留个查找的念性。

实验环境：虚拟机，操作系统CentOS 5.2
用虚拟机加入三块硬盘，大小均为6GB
 
# fdisk -l
 
Disk /dev/sda: 6442 MB, 6442450944 bytes
255 heads, 63 sectors/track, 783 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
 
   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          13      104391   83  Linux
/dev/sda2              14         783     6185025   8e  Linux LVM
 
Disk /dev/sdb: 6442 MB, 6442450944 bytes
255 heads, 63 sectors/track, 783 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
 
Disk /dev/sdb doesn't contain a valid partition table
 
Disk /dev/sdc: 6442 MB, 6442450944 bytes
255 heads, 63 sectors/track, 783 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
 
Disk /dev/sdc doesn't contain a valid partition table
 
Disk /dev/sdd: 6442 MB, 6442450944 bytes
255 heads, 63 sectors/track, 783 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
 
Disk /dev/sdd doesn't contain a valid partition table
 
接下来将第二块硬盘格式化。
# fdisk /dev/sdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.
 
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
 
Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-783, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-783, default 783):
Using default value 783
 
Command (m for help): t
Selected partition 1
Hex code (type L to list codes): l
 
 0  Empty           1e  Hidden W95 FAT1 80  Old Minix       be  Solaris boot
 1  FAT12           24  NEC DOS         81  Minix / old Lin bf  Solaris
 2  XENIX root      39  Plan 9          82  Linux swap / So c1  DRDOS/sec (FAT-
 3  XENIX usr       3c  PartitionMagic  83  Linux           c4  DRDOS/sec (FAT-
 4  FAT16 <32M      40  Venix 80286     84  OS/2 hidden C:  c6  DRDOS/sec (FAT-
 5  Extended        41  PPC PReP Boot   85  Linux extended  c7  Syrinx
 6  FAT16           42  SFS             86  NTFS volume set da  Non-FS data
 7  HPFS/NTFS       4d  QNX4.x          87  NTFS volume set db  CP/M / CTOS / .
 8  AIX             4e  QNX4.x 2nd part 88  Linux plaintext de  Dell Utility
 9  AIX bootable    4f  QNX4.x 3rd part 8e  Linux LVM       df  BootIt
 a  OS/2 Boot Manag 50  OnTrack DM      93  Amoeba          e1  DOS access
 b  W95 FAT32       51  OnTrack DM6 Aux 94  Amoeba BBT      e3  DOS R/O
 c  W95 FAT32 (LBA) 52  CP/M            9f  BSD/OS          e4  SpeedStor
 e  W95 FAT16 (LBA) 53  OnTrack DM6 Aux a0  IBM Thinkpad hi eb  BeOS fs
 f  W95 Ext'd (LBA) 54  OnTrackDM6      a5  FreeBSD         ee  EFI GPT
10  OPUS            55  EZ-Drive        a6  OpenBSD         ef  EFI (FAT-12/16/
11  Hidden FAT12    56  Golden Bow      a7  NeXTSTEP        f0  Linux/PA-RISC b
12  Compaq diagnost 5c  Priam Edisk     a8  Darwin UFS      f1  SpeedStor
14  Hidden FAT16 <3 61  SpeedStor       a9  NetBSD          f4  SpeedStor
16  Hidden FAT16    63  GNU HURD or Sys ab  Darwin boot     f2  DOS secondary
17  Hidden HPFS/NTF 64  Novell Netware  b7  BSDI fs         fd  Linux raid auto
18  AST SmartSleep  65  Novell Netware  b8  BSDI swap       fe  LANstep
1b  Hidden W95 FAT3 70  DiskSecure Mult bb  Boot Wizard hid ff  BBT
1c  Hidden W95 FAT3 75  PC/IX
Hex code (type L to list codes): fd
Changed system type of partition 1 to fd (Linux raid autodetect)
 
Command (m for help): w
The partition table has been altered!
 
Calling ioctl() to re-read partition table.
Syncing disks.
 
用相同的方法把另外两块硬盘格式化。
都格式化完后查看一下硬盘状态。
 
# fdisk -l
Disk /dev/sda: 6442 MB, 6442450944 bytes
255 heads, 63 sectors/track, 783 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
 
   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          13      104391   83  Linux
/dev/sda2              14         783     6185025   8e  Linux LVM
 
Disk /dev/sdb: 6442 MB, 6442450944 bytes
255 heads, 63 sectors/track, 783 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
 
   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1         783     6289416   fd  Linux raid autodetect
 
Disk /dev/sdc: 6442 MB, 6442450944 bytes
255 heads, 63 sectors/track, 783 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
 
   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1               1         783     6289416   fd  Linux raid autodetect
 
Disk /dev/sdd: 6442 MB, 6442450944 bytes
255 heads, 63 sectors/track, 783 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
 
   Device Boot      Start         End      Blocks   Id  System
/dev/sdd1               1         783     6289416   fd  Linux raid autodetect
 
建立RAID 5
 
# mdadm --create /dev/md0 --level=5 --raid-device=3 /dev/sdb1 /dev/sdc1 /dev/sdd1
mdadm: array /dev/md0 started.
 
查看创建过程
# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdd1[3] sdc1[1] sdb1[0]
      12578688 blocks level 5, 64k chunk, algorithm 2 [3/2] [UU_]
      [===>.................]  recovery = 17.6% (1112836/6289344) finish=4.4min speed=19414K/sec
 
unused devices:
[root@changzi ~]# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdd1[3] sdc1[1] sdb1[0]
      12578688 blocks level 5, 64k chunk, algorithm 2 [3/2] [UU_]
      [=======>.............]  recovery = 37.9% (2384784/6289344) finish=3.4min speed=18910K/sec
 
unused devices:
[root@changzi ~]# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdd1[3] sdc1[1] sdb1[0]
      12578688 blocks level 5, 64k chunk, algorithm 2 [3/2] [UU_]
      [=================>...]  recovery = 86.2% (5422580/6289344) finish=0.7min speed=19344K/sec
 
unused devices:
[root@changzi ~]# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdd1[2] sdc1[1] sdb1[0]
      12578688 blocks level 5, 64k chunk, algorithm 2 [3/3] [UUU]
 
unused devices:
 
在/var/log/message中会有通知的消息
 
# tail /var/log/messages
Nov 28 19:43:56 changzi kernel: RAID5 conf printout:
Nov 28 19:43:56 changzi kernel:  --- rd:3 wd:3 fd:0
Nov 28 19:43:56 changzi kernel:  disk 0, o:1, dev:sdb1
Nov 28 19:43:56 changzi kernel:  disk 1, o:1, dev:sdc1
Nov 28 19:43:56 changzi kernel:  disk 2, o:1, dev:sdd1
 
把当前的RAID设置存储在一个配置文件里。
# echo DEVICE /dev/sdb1 /dev/sdc1 /dev/sdd1 >/etc/mdadm.conf
# mdadm --detail --scan >> /etc/mdadm.conf
# cat /etc/mdadm.conf
DEVICE /dev/sdb1 /dev/sdc1 /dev/sdd1
ARRAY /dev/md0 level=raid5 num-devices=3 UUID=c0128fb4:db22196a:91a1d8cb:51da19ee
 
查看当前物理卷
# pvdisplay
  /dev/hdc: open failed: No medium found
  --- Physical volume ---
  PV Name               /dev/sda2
  VG Name               VolGroup00
  PV Size               5.90 GB / not usable 24.06 MB
  Allocatable           yes (but full)
  PE Size (KByte)       32768
  Total PE              188
  Free PE               0
  Allocated PE          188
  PV UUID               LHGiiL-rF1f-HXXw-dDIA-Qc3t-c8W2-OH9JU3
 
查看当前卷组
# vgdisplay
  /dev/hdc: open failed: No medium found
  --- Volume group ---
  VG Name               VolGroup00
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  3
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                2
  Open LV               2
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               5.88 GB
  PE Size               32.00 MB
  Total PE              188
  Alloc PE / Size       188 / 5.88 GB
  Free  PE / Size       0 / 0
  VG UUID               nRXmQF-puwD-09WF-Urkn-a2c3-1rGt-VYbDdH
 
查看当前逻辑卷
# lvdisplay
  /dev/hdc: open failed: No medium found
  --- Logical volume ---
  LV Name                /dev/VolGroup00/LogVolRoot
  VG Name                VolGroup00
  LV UUID                y7UrC3-gmlk-COcU-loS5-pm1o-deFO-9a4wUS
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                5.38 GB
  Current LE             172
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:0
 
  --- Logical volume ---
  LV Name                /dev/VolGroup00/LogVolSwap
  VG Name                VolGroup00
  LV UUID                GPBNdC-1f6c-E7Ag-01Lr-M1Sp-JlwP-mtoDBk
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                512.00 MB
  Current LE             16
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:1
 
建立LVM
# pvcreate /dev/md0
  Physical volume "/dev/md0" successfully created
 
将物理卷加入卷组
# vgcreate VolGroup01 /dev/md0
  /dev/hdc: open failed: No medium found
  Volume group "VolGroup01" successfully created
 
查看当前卷组信息
# vgdisplay
  --- Volume group ---
  VG Name               VolGroup00
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  3
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                2
  Open LV               2
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               5.88 GB
  PE Size               32.00 MB
  Total PE              188
  Alloc PE / Size       188 / 5.88 GB
  Free  PE / Size       0 / 0
  VG UUID               nRXmQF-puwD-09WF-Urkn-a2c3-1rGt-VYbDdH
 
  --- Volume group ---
  VG Name               VolGroup01
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  1
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                0
  Open LV               0
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               11.99 GB
  PE Size               4.00 MB
  Total PE              3070
  Alloc PE / Size       0 / 0
  Free  PE / Size       3070 / 11.99 GB
  VG UUID               xrLh2B-LgCX-TJOR-voYH-QdJS-DhrM-crWvA0
 
在卷组上创建逻辑卷
# lvcreate -L 8G -n LogVolWEB1 VolGroup01
  Logical volume "LogVolWEB1" created
 
看一下当前卷组的信息
# vgdisplay VolGroup01
  --- Volume group ---
  VG Name               VolGroup01
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  2
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                1
  Open LV               0
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               11.99 GB
  PE Size               4.00 MB
  Total PE              3070
  Alloc PE / Size       2048 / 8.00 GB
  Free  PE / Size       1022 / 3.99 GB
  VG UUID               xrLh2B-LgCX-TJOR-voYH-QdJS-DhrM-crWvA0
 
对逻辑卷进行分区
# mkfs -t ext3 /dev/VolGroup01/LogVolWEB1
mke2fs 1.39 (29-May-2006)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
1048576 inodes, 2097152 blocks
104857 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=2147483648
64 block groups
32768 blocks per group, 32768 fragments per group
16384 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632
 
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
 
This filesystem will be automatically checked every 22 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.
 
编辑/etc/fstab，在文件尾加下如下行，然后重新挂载
# vim /etc/fstab
/dev/VolGroup01/LogVolWEB1      /mnt/web        ext3    defaults        0 0
 
# mkdir /mnt/web
# mount -a
# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVolRoot
                      5.3G  2.1G  2.9G  42% /
/dev/sda1              99M   12M   83M  13% /boot
tmpfs                 125M     0  125M   0% /dev/shm
/dev/mapper/VolGroup01-LogVolWEB1
                      7.9G  147M  7.4G   2% /mnt/web
 
将剩余部分也给分成逻辑卷
# lvcreate -l 1022 -n LogVolWEB2 VolGroup01
  Logical volume "LogVolWEB2" created
 
此处的1022来自上面显示的Free  PE / Size       1022 / 3.99 GB
 
此时卷组状态
# vgdisplay VolGroup01
  --- Volume group ---
  VG Name               VolGroup01
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  5
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                2
  Open LV               1
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               11.99 GB
  PE Size               4.00 MB
  Total PE              3070
  Alloc PE / Size       3070 / 11.99 GB
  Free  PE / Size       0 / 0
  VG UUID               xrLh2B-LgCX-TJOR-voYH-QdJS-DhrM-crWvA0
 
卷组全部用完：Free  PE / Size       0 / 0
 
此时分挂载点的状态
# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVolRoot
                      5.3G  2.1G  2.9G  42% /
/dev/sda1              99M   12M   83M  13% /boot
tmpfs                 125M     0  125M   0% /dev/shm
/dev/mapper/VolGroup01-LogVolWEB1
                      7.9G  147M  7.4G   2% /mnt/web
/dev/mapper/VolGroup01-LogVolWEB2
                      4.0G   73M  3.7G   2% /mnt/web2
 
接下来进行故障模拟
mdadm命令提供了模拟故障的方式。
# mdadm /dev/md0 -f /dev/sdc1
mdadm: set /dev/sdc1 faulty in /dev/md0
 
查看日志
# tail /var/log/messages
Nov 28 20:20:57 changzi kernel: raid5: Disk failure on sdc1, disabling device. Operation continuing on 2 devices
Nov 28 20:20:57 changzi kernel: RAID5 conf printout:
Nov 28 20:20:57 changzi kernel:  --- rd:3 wd:2 fd:1
Nov 28 20:20:57 changzi kernel:  disk 0, o:1, dev:sdb1
Nov 28 20:20:57 changzi kernel:  disk 1, o:0, dev:sdc1
Nov 28 20:20:57 changzi kernel:  disk 2, o:1, dev:sdd1
Nov 28 20:20:57 changzi kernel: RAID5 conf printout:
Nov 28 20:20:57 changzi kernel:  --- rd:3 wd:2 fd:1
Nov 28 20:20:57 changzi kernel:  disk 0, o:1, dev:sdb1
Nov 28 20:20:57 changzi kernel:  disk 2, o:1, dev:sdd1
或者查看状态
# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdd1[2] sdc1[3](F) sdb1[0]
      12578688 blocks level 5, 64k chunk, algorithm 2 [3/2] [U_U]
 
unused devices:
可以看到sdc1的后面有个F
 
从RAID设备里删除设备
# mdadm /dev/md0 -r /dev/sdc1
mdadm: hot removed /dev/sdc1
 
更换硬盘，加回盘阵
# mdadm /dev/md0 -a /dev/sdc1
mdadm: re-added /dev/sdc1
 
加回时可以在/proc/mdstat中看进度
# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdc1[1] sdd1[2] sdb1[0]
      12578688 blocks level 5, 64k chunk, algorithm 2 [3/2] [U_U]
      [=>...................]  recovery =  9.0% (567912/6289344) finish=13.4min speed=7064K/sec
 
unused devices:
 
# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdc1[1] sdd1[2] sdb1[0]
      12578688 blocks level 5, 64k chunk, algorithm 2 [3/3] [UUU]
 
unused devices: