全部博文（408）
分类： LINUX
2006-04-21 16:21:51
This document should help you get your Hardware running on Software RAID 1. This was born out of two week's worth of frustration and lack of documentation on this specific combination. I will write this as a step-by-step guide, and point out issues that arise and reasons as we go.
I recommend that you read and re-read the and before going any further. I found that these two documents explained enough to get me started. However, using them alone I repeatedly got kernel panics as I tried to move to the new RAID1 root filesystem right after the initrd image ran (actually, while it is trying to mount the root is where the problem was for me, see ROOT= below).
My aim was to use standard packages, with no recompliation, and having all packages upgradeable by standard (apt) means.
I am going to discuss booting on i386 hardware, because that is what I used. Some of that I discuss may be relevant to other architectures, but I don't know.
I was using a 1 GHz PIII machine, with two 80 GB IDE Hard Disk drives, on a Soltek SL-65KV2 motherboard, with 512 MB of RAM, and a floppy drive, in a reasonable case, plus an Intel Etherpro 100 NIC, and a snazzy Adaptec 29160 SCSI card. You don't need the SCSI card, but a network card for which there is already a Linux driver for is of use. The important bit is that I had two identical Hard Drives.
Create a set of boot disks: the files you will need are available on your closest Debian mirror. At this point in time, you need the Rescue Disk (boot.bin), Root File System (root.bin), and the four (4) driver disks. All these images are located in $debian/dists/testing/disks-i386/. Refer to the documentation on using "dd" or "rawrite2" for putting these onto your media.
Everything plugs together like normal, except your Hard Drives are attached to each of your IDE Controllers. Most mother boards support up to four (4) Hard Drives: two on the first controller, and two on the second. Each controller supports two drives: one as a master, and one as a slave. There are generally "jumpers" on hard drives to set them as master or slave: both the drives you have should be set to master. Since they are equivalent, it doesn't matter which one is plugged into the 1st IDE controller, and which is on the second.
To get started, you need to get a base system set up on one disk. Our plan is to have everything installed on one partition, then create our RAID devices, boot on to it, and then create our other partitions and migrate to it as needed. We aim to end up with one partition for /boot and the root filesystem, and others as needed. We can do the whole thing on one RAID partition, but thats your choice. We will also use DevFS, and ReiserFS.
Boot from the rescue disk, then the root, then follow the installation instructions. When you get prompted to partition, think about how bug your disk is, and how you would like it carved up in the long run. You can change this later, but starting out with a plan can be simpler for you.
For example, I had 80 GB to play with (two 80 GB drives under RAID 1 gives 80 GB of space at the end). I chose to have 4 GB for my root partition, and divide the rest up later on for /home, /usr, /var, and /usr/local (plus a little bit for swap).
Don't worry about running dselect or installing any "task-" packages. We just want a simple system set up on vanilla ext2 filesystem. We should end up with a self booting system on ext2, running whatever standard kernel was chosen.
Now we need to prepare for running a RAID setup. Our packages need an update. Use apt, because it rocks, and install the following:
Edit /etc/modules and add the following modules:
Edit /etc/mkinitrd/modules, and add the same modules to this list. Your initrd image needs to be able to read and write to your RAID array, before your filesystem is mounted. Initrd is the trick here. You probably also want to see if you need to edit /etc/mkinitrd/mkinitrd.cfg and set the variable ROOT=probe to be ROOT=/dev/md0, or possibly, if using DevFS, ROOT=/dev/md/0.
Regenerate your initrd image for your new kernel with mkinitrd -o /tmp/initrd-new /lib/modules/2.4.x-... . If all is good, move this to /boot/initrd-2.4.x-... and edit your /etc/lilo.conf to add initrd=/boot/initrd against the "Linux" kernel entry. Run lilo, and you should see an asterisk next to the boot image "Linux".
Reboot into your new kernel
You now have a system that can use RAID on root. Fire up cfdisk, and partition your disk that will be one half of your RAID 1 array. Make sure you set the partition type to fd on all of them, and the first one should be marked as boot-able.
Create your raidtab. Copy /usr/share/doc/raidtools2/examples/raid1.conf or similar, and define your RAID devices. /dev/md0 will be our first one; you can leave the rest for the moment. Define it as having two devices, one being /dev/hdc1, and /dev/hda1, in that order. Be sure to mark /dev/hda1 as a failed-disk! You are still using this disk, and don't want it trampled on.
Now make the md0 device: mkraid /dev/md0. Now format the device: mkfs.reiserfs /dev/md0. Mount it somewhere: mount /dev/md0 /mnt. Copy your current system to it: cd /; find . -xdev | cpio -p /mnt. Edit the new systems /etc/fstab, which will currently be located at /mnt/etc/fstab: change the root partition to /dev/md0, and the partition type to reiserfs.
Edit your lilo.conf (on both systems) and update the definition for your Linux mount with: append="md=0,/dev/hdc1,/dev/hda1", and root=/dev/md0. Run lilo again to update it.
Reboot now onto your RAID 1 on reiserfs on root partition. Check that it worked with df.
Partition the rest of hdc now if you hadn't already. You may want swap on RAID 1, but you'll be far better off purchasing a bit of extra memory in its place. Thats up to you.
Now time to bring hda back into the fold. You can duplicate your hdc partition table to hda using sfdisk -d /dev/hdc | sfdisk /dev/hda.
You can bring hda online now by editing your /etc/raidtab and replacing the failed-disk against hda1 with raid-disk, and then running raidhotadd /dev/md0 /dev/hda1. The drive should start to sync up: you can watch /proc/mdstat for details on how this is going. You can continue while it does this.
So, now you can move other partitions to RAID 1. You don't need to muck around with failed-disk directives any more. You already have the partitions you need, so all you need to do is edit /etc/raidtab and define them, then mkraid /dev/mdi, format it as above, mount it somewhere, and copy your current filesystem tree to it. The a quick way of activating it is, using /usr as an example: mv /usr /usr2; mkdir /usr; mount /dev/mdi /usr, and if all has gone well, then remove /usr2. Later, rinse, repeat.
You have a modern filesystem. Now, I recommend that you use a filesystem that is in the standard kernel tree, and available as a package. You entire system can be updated to newer kernels automatically: your new initrds will be generated with the required modules and ROOT definition. Just apt-get your new kernel, and all *should* be well. However, I make no promises.
You may wish to have your Hard Disks set up with hdparm to use multi-byte transfers. Install the hdparm package, and read the man page. I edited /etc/init.d/raid2, and against the "start" option, added hdparm -m 16 /dev/hda; hdparm -m 16 /dev/hdc: your options to -m may vary: read the manual page.
I ran into problems using a boot parmater of "devfs=mount". I'm not sure, but DevFS may be moving the traditional /dev/md0 to /dev/md/0 too early, and since DevFS daemon is not running at this point, there is no symbolic link to follow. Hence, devfs=mount may require that your kernel initrd image have ROOT=/dev/md/0 in place of ROOT=/dev/md0. This should have been fixed from version 0.1.12 of the initrd-tools package.
phobe:~> bonnie -s 1000
Writing with putc()...done
Writing intelligently...done
Rewriting...done
Reading with getc()...done
Reading intelligently...done
start 'em...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version 1.02 ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
phobe 1000M 10898 90 29458 63 9559 13 10947 77 31891 14 317.4 1
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 9265 70 +++++ +++ 15273 99 12506 99 +++++ +++ 12694 99
phobe,1000M,10898,90,29458,63,9559,13,10947,77,31891,14,317.4,1,16,9265,70,+++++,+++,15273,99,12506,99,+++++,+++,12694,99
phobe:~> df -k Filesystem 1k-blocks Used Available Use% Mounted on /dev/md0 4003520 91092 3912428 3% / /dev/md1 4003584 304068 3699516 8% /usr /dev/md2 4003584 232564 3771020 6% /var /dev/md3 4003520 1071384 2932136 27% /home /dev/md5 62035860 11357460 50678400 19% /usr/local
phobe:~> cat /proc/mdstat Personalities : [raid1] read_ahead 1024 sectors md5 : active raid1 ide/host0/bus1/target0/lun0/part7[1] ide/host0/bus0/target0/lun0/part7[0] 62037760 blocks [2/2] [UU] md4 : active raid1 ide/host0/bus1/target0/lun0/part6[1] ide/host0/bus0/target0/lun0/part6[0] 97664 blocks [2/2] [UU] md3 : active raid1 ide/host0/bus1/target0/lun0/part5[1] ide/host0/bus0/target0/lun0/part5[0] 4003648 blocks [2/2] [UU] md2 : active raid1 ide/host0/bus1/target0/lun0/part3[1] ide/host0/bus0/target0/lun0/part3[0] 4003712 blocks [2/2] [UU] md1 : active raid1 ide/host0/bus1/target0/lun0/part2[1] ide/host0/bus0/target0/lun0/part2[0] 4003712 blocks [2/2] [UU] md0 : active raid1 ide/host0/bus0/target0/lun0/part1[1] ide/host0/bus1/target0/lun0/part1[0] 4003648 blocks [2/2] [UU] unused devices:
Remember, you need to have this in /etc/raid/raidtab, and a symlink from /etc/raidtab.
raiddev /dev/md/0 raid-level 1 nr-raid-disks 2 nr-spare-disks 0 chunk-size 4 device /dev/hdc1 raid-disk 0 device /dev/hda1 #failed-disk 1 raid-disk 1 #/usr raiddev /dev/md1 raid-level 1 nr-raid-disks 2 nr-spare-disks 0 chunk-size 4 device /dev/hda2 raid-disk 0 device /dev/hdc2 raid-disk 1 #/var raiddev /dev/md2 raid-level 1 nr-raid-disks 2 nr-spare-disks 0 chunk-size 4 device /dev/hda3 raid-disk 0 device /dev/hdc3 raid-disk 1 #/home raiddev /dev/md3 raid-level 1 nr-raid-disks 2 nr-spare-disks 0 chunk-size 4 device /dev/hda5 raid-disk 0 device /dev/hdc5 raid-disk 1 # swap raiddev /dev/md4 raid-level 1 nr-raid-disks 2 nr-spare-disks 0 chunk-size 4 device /dev/hda6 raid-disk 0 device /dev/hdc6 raid-disk 1 # usr/local raiddev /dev/md5 raid-level 1 nr-raid-disks 2 nr-spare-disks 0 chunk-size 4 device /dev/hda7 raid-disk 0 device /dev/hdc7 raid-disk 1
# Support LBA for large hard disks. lba32 # Specifies the boot device. This is where Lilo installs its boot # block. It can be either a partition, or the raw device, in which # case it installs in the MBR, and will overwrite the current MBR. boot=/dev/md0 raid-extra-boot="/dev/hda,/dev/hdc" # Specifies the device that should be mounted as root. (`/') root=/dev/md0 # Installs the specified file as the new boot sector install=/boot/boot.b # Specifies the location of the map file map=/boot/map # Specifies the number of deciseconds (0.1 seconds) LILO should # wait before booting the first image. delay=20 vga=normal default=LinuxNoDevFS image=/vmlinuz label=Linux root=/dev/md0 append="md=0,/dev/hda1,/dev/hdc1 devfs=mount" read-only initrd=/initrd.img image=/vmlinuz label=LinuxNoDevFS root=/dev/md0 append="md=0,/dev/hda1,/dev/hdc1" read-only initrd=/initrd.img image=/vmlinuz.old label=LinuxOLD root=/dev/md0 append="md=0,/dev/hda1,/dev/hdc1 devfs=mount" read-only initrd=/initrd.img.old image=/vmlinuz.old label=LinuxOLDNoDevFS read-only append="md=0,/dev/hda1,/dev/hdc1" optional root=/dev/hda1 initrd=/initrd.img.old
So what to do if you can't get your root RAID1 filesystem to boot? Here is a straightforward way to get to your md0:
Just to make sure everyone is convinced this works: the system I installed this on is still working fine today, 8th May 2003 (touch wood). Over this time I have had only one failed disk, and many peacful nights of sleep knowing that one disk failing at any time isnt a huge job.
while [ true ]; do cat /proc/memory ;sleep 1; done;