Background:
When using Volume Manager (VxVM) with a certain subset of storage arrays it is necessary to install 'add-on' RPMs which may include an array support library (ASL) and array policy module (APM). APM RPMs commonly contain array specific kernel modules which when loaded are used by the dynamic multi pathing driver (DMP) and indicate special functionality offered by the array which can be used by DMP, or provide specific instructions which the array expects the DMP driver to send when performing operations such as path fail over. As such if the APM module is not loaded then certain functionality offered by the array may not be used by DMP, or the DMP driver may become unable to interact correctly with the array leading to unexpected failures during operation.
The Linux kernel expects all available kernel modules to be located under the '/lib/modules/`uname -r`' directory tree. As such, Volume Manager modules are commonly located in the '/lib/modules/`uname -r`/veritas/vxvm' directory. The Volume Manager product contains a number of features to ensure that modules are correctly available in this location when the system boots, for example:
Volume Manager RPMs: When a Volume Manager RPM is installed (core product or APM RPM) the postinstall script run during RPM installation examines the version of kernel which is currently running then compares this with the modules shipped inside the corresponding RPM. Using this information a 'best fit' kernel module is selected and copied or symbolically linked to the '/lib/modules/`uname -r`/veritas/vxvm' directory before being loaded.
Volume Manager start up scripts: /etc/vx/vxvm-startup is commonly run during system boot. One of the the purposes of this script is to load Volume Manager kernel modules such that the product is ready for operation. The steps taken to do this are to:
- Attempt to load the vxmp.ko module which should be located in '/lib/modules/`uname -r`/veritas/vxvm'
- If the load of vxdmp.ko fails the script will remove and recreate the '/lib/modules/`uname -r`/veritas/vxvm' directory
- The script then uses procedures defined in /etc/vx/modinst-vxvm and a list of required modules from /etc/vx/vxvm-startup to locate all 'best fit' versions of these modules from the /etc/vx/kernel directory
- The 'best fit' versions of these modules are then symbolically linked into the '/lib/modules/`uname -r`/veritas/vxvm' directory before being loaded once more
Issue:
Following on from the above, when a kernel upgrade takes place on a system running Volume Manager, we would expect the following to happen:
- The system is booted on the existing kernel with Volume Manager modules either copied or symbolically linked to the '/lib/modules/`uname -r`/veritas/vxvm' directory
- The updated kernel RPM is installed which (amongst other activities) creates a '/lib/modules/`uname -r`' directory for the new kernel
- The system is rebooted using the new kernel
- During startup the system runs /etc/vx/vxvm-startup which attempts to load required Volume Manager modules however at this stage the '/lib/modules/`uname -r`/veritas/vxvm' does not exist hence the modules fail to load
- /etc/vx/vxvm-startup sources /etc/vx/modinst-vxvm and uses code within this script to locate the 'best fit' version of each required module before creating a symbolic link to the discovered module in the '/lib/modules/`uname -r`/veritas/vxvm' directory
- Once complete required modules can be loaded and Volume Manager starts as normal
The above sequence of events ensures that core Volume Manager kernel modules are able to be loaded after a kernel upgrade, however the vxvm-startup script does not ensure that any other kernel modules, such as those installed by 'add on' APM RPMs, are available in the correct directory for the new kernel. As such Volume Manager will likely start without issue however 'add on' APM modules may not be loaded and arrays may exhibit unexpected issues during operation.
Workaround:
Note that with the current design of Volume Manager and the way in which APMs are handled, the product only guarantees that 'add on' APM modules are copied to the correct directory for the kernel in use when the RPM is initially installed. There is currently no provision for automatic copying of modules when a new kernel is installed at a later date. As such it is important that users manually check that expected 'add on' APM modules are placed in the new kernels module directory after a kernel upgrade is performed. Various methods for doing this are discussed below:
1. Remove and reinstall 'add on' APM RPMs after the kernel upgrade and subsequent reboot. When reinstalled the APM postinstall script ensures that the correct kernel modules are copied to the new kernels '/lib/modules' directory tree
2. Manually copy required 'add on' APM kernel modules into place within the new kernels '/lib/modules' directory tree after kernel upgrade and subsequent reboot. For example:
After kernel upgrade and initial reboot the /etc/vx/vxvm-startup script creates the '/lib/modules/`uname -r`/veritas/vxvm' directory and populates it with symbolic links to required modules:
[root@rdgv20z08 ~]# ls /lib/modules/2.6.18-194.el5/veritas/vxvm
dmpaaa.ko dmpaa.ko dmpalua.ko dmpapf.ko dmpapg.ko dmpap.ko dmpCLARiiON.ko dmpjbod.ko vxdmp.ko vxio.ko vxspec.ko
Note however that all add on APM kernel modules are missing from this directory.
To identify any APM modules which need to be copied, list files in the old kernels '/lib/modules/`uname -r`/veritas/vxvm' directory and look for any files which are NOT a symbolic link (as these files will likely have been added by 'add on' RPMs such as APMs). Note that in this instance the old kernel version being used was 2.6.18-164.el5:
[root@rdgv20z07 vx]# ls -al /lib/modules/2.6.18-164.el5/veritas/vxvm
total 612
drwxr-xr-x 2 root root 4096 Apr 16 09:30 .
drwxr-xr-x 4 root root 4096 Apr 16 09:17 ..
lrwxrwxrwx 3 root root 39 Apr 16 09:17 dmpaaa.ko -> /etc/vx/kernel/dmpaaa.ko.2.6.18-8.el5.1
lrwxrwxrwx 3 root root 38 Apr 16 09:17 dmpaa.ko -> /etc/vx/kernel/dmpaa.ko.2.6.18-8.el5.1
lrwxrwxrwx 3 root root 40 Apr 16 09:17 dmpalua.ko -> /etc/vx/kernel/dmpalua.ko.2.6.18-8.el5.1
lrwxrwxrwx 3 root root 39 Apr 16 09:17 dmpapf.ko -> /etc/vx/kernel/dmpapf.ko.2.6.18-8.el5.1
lrwxrwxrwx 3 root root 39 Apr 16 09:17 dmpapg.ko -> /etc/vx/kernel/dmpapg.ko.2.6.18-8.el5.1
lrwxrwxrwx 3 root root 38 Apr 16 09:17 dmpap.ko -> /etc/vx/kernel/dmpap.ko.2.6.18-8.el5.1
lrwxrwxrwx 3 root root 44 Apr 16 09:17 dmpCLARiiON.ko -> /etc/vx/kernel/dmpCLARiiON.ko.2.6.18-8.el5.1
lrwxrwxrwx 3 root root 40 Apr 16 09:17 dmpjbod.ko -> /etc/vx/kernel/dmpjbod.ko.2.6.18-8.el5.1
-rwxr-xr-x 1 root root 610385 Apr 16 09:30 dmpnetapp.ko <=== NOTE
lrwxrwxrwx 1 root root 36 Apr 16 09:17 vxdmp.ko -> /etc/vx/kernel/vxdmp.ko.2.6.18-8.el5
lrwxrwxrwx 1 root root 35 Apr 16 09:17 vxio.ko -> /etc/vx/kernel/vxio.ko.2.6.18-8.el5
lrwxrwxrwx 1 root root 37 Apr 16 09:17 vxspec.ko -> /etc/vx/kernel/vxspec.ko.2.6.18-8.el5
Note that in the above output we see the dmpnetapp.ko kernel module. Any such modules should be copied to the new kernels /lib/modules directory tree, i.e.:
[root@rdgv20z07 vx]# cp /lib/modules/2.6.18-164.el5/veritas/vxvm/dmpnetapp.ko /lib/modules/`uname -r`/veritas/vxvm
Once complete modules.dep should be checked for correctness by running 'depmod -aq':
[root@rdgv20z07 ~]# depmod -aq
WARNING: Module /lib/modules/2.6.18-194.el5/veritas/vxvm/vxdmp.ko is not an elf object
WARNING: Module /lib/modules/2.6.18-194.el5/veritas/vxvm/dmpCLARiiON.ko is not an elf object
WARNING: Module /lib/modules/2.6.18-194.el5/veritas/vxvm/dmpap.ko is not an elf object
WARNING: Module /lib/modules/2.6.18-194.el5/veritas/vxvm/dmpaaa.ko is not an elf object
WARNING: Module /lib/modules/2.6.18-194.el5/veritas/vxvm/dmpaa.ko is not an elf object
WARNING: Module /lib/modules/2.6.18-194.el5/veritas/vxvm/dmpapf.ko is not an elf object
WARNING: Module /lib/modules/2.6.18-194.el5/veritas/vxvm/vxio.ko is not an elf object
WARNING: Module /lib/modules/2.6.18-194.el5/veritas/vxvm/dmpapg.ko is not an elf object
WARNING: Module /lib/modules/2.6.18-194.el5/veritas/vxvm/vxspec.ko is not an elf object
WARNING: Module /lib/modules/2.6.18-194.el5/veritas/vxvm/dmpjbod.ko is not an elf object
WARNING: Module /lib/modules/2.6.18-194.el5/veritas/vxvm/dmpalua.ko is not an elf object
Finally the system should be rebooted to allow clean start up of Volume Manager now that required APM kernel modules are in place and can be loaded.
Note that this procedure assumes that a minor upgrade is being performed (i.e. here we are staying on the 2.6.18 kernel release but are simply installing a later build of the kernel). If, however, kernel minor versions are changing (for example if moving from a 2.6.18 kernel to a 2.6.31 kernel), the ASL/APM RPM will need to be reinstalled after upgrade as binary compatibility may be lost between these kernels and as such an entirely different module may be required on the later kernel.
It is expected that handling of kernel modules added by 'add on' RPMs will be improved in future versions of the Volume Manager product. For more information on this please contact Symantec Technical support and quote Etrack incident number 'e2025487'
