Problem Description:
As a result of running vxassist to shrink a volume size,
# vxassist -g testdg -f shrinkto u100 36g or
# vxassist -g testdg -f shrinkby u100 -9g
There are occasion attempt to mount or fsck the filesystem will fail.
Cause:
The issue occurs when a volume has been forcefully reduced in size leaving the underlying filesystem with size larger than the volume.
Error Code / Message:
# mount -F vxfs /dev/vx/dsk/testdg/u100 /c1mnt
UX:vxfs mount: ERROR: V-3-24706: /dev/vx/dsk/testdg/u100 no such device or filesystem on it missing one or more devices .....
# fsck -F vxfs /dev/vx/rdsk/testdg/u100
UX:vxfs fsck: ERROR: V-3-26248: could not read from block offset devid/blknum 0/xxxxxx . Device containing meta data may be missing in vset or device too big to be read on a 32 bit system.
UX:vxfs fsck: ERROR: V-3-20694: cannot initialize aggregate file system check failure, aborting .
Diagnostic Steps:
1. Check the current status and size of filesystem and volume
# fstyp -v /dev/vx/dsk/testdg/u100
vxfs
magic a501fcf5 version 6 ctime Sat Jan 28 01:27:37 2006
logstart 0 logend 0
bsize 1024 size 47185920 dsize 0 ninode 47185920 nau 0...........
...
where
size of filesystem = (bsize*size)/512 sectors
=1024*47185920/512
= 94371840 sectors (=45Gb)
# vxprint -g testdg -htv u100
v u100 - ENABLED ACTIVE 75497472 SELECT - fsgen
pl u100-01 u100 ENABLED ACTIVE 75497472 CONCAT - RW
sd testdg01-01 u100-01 testdg01 0 16777216 0 c1t2d0 ENA
sd testdg01-08 u100-01 testdg01 167772160 16777216 16777216 c1t2d0 ENA
sd testdg01-12 u100-01 testdg01 146800640 20971520 33554432 c1t2d0 ENA
sd testdg01-13 u100-01 testdg01 234881024 20971520 54525952 c1t2d0 ENA
where: filesystem = 94371840 sectors (=45Gb)
volume size = 75497472 sectors, (=36Gb)
Difference is volume size = 18874368 sectors (=9Gb)
So to resolve the mount issue we need to grow the volume by 9Gb and back to the original size =45Gb
Optional: If the original volume is mirrored, it is recommended to break the mirror plex and keep this as backup copy
# vxplex -g dis
2. Use "vxdg free" first is to check what disk are available to be used
# vxdg -g testdg free
GROUP DISK DEVICE TAG OFFSET LENGTH FLAGS
testdg testdg03 c1t4d0s2 c1t4d0 285212672 18874368 -
testdg testdg04 c1t5d0s2 c1t5d0 285826704 869616 -
3. Check available data to determine the configuration of the volume prior to running the vxassist to shrink the volume.
(i) Copy of an old vxprint -g testdg -ht
(ii) Use backup directory: /etc/vx/cbg/bk/testdg.1224220109.123.xxx to recreate vxprint file by selecting the *cfrec. file created before the shrink command.
# cd /etc/vx/cbg/bk/testdg.1224220109.123.xxx
# cat 1224220109.123.xxx.cfgrec. | vxprint -D - -ht
(iii) Use backup log : /var/adm/vx/cmdlog.* to check what commands where used
# cd /var/adm/vx
# cat cmdlog | grep shrinkto
# 4474, 23451, Mon Apr 6 18:30:31 2009
/usr/sbin/vxassist -ttaskid_112 -f -g testdg shrinkto u100 75497472 alloc=testdg03
The log indicates testdg03 was used to shrink the volume.
Solution:
As per the diagnostic steps above, we need to grow the volume by 9Gb to the original size=45Gb
using # vxdg free output to determine the free space available. There are 3 possible options:
1) Selection is limited to only one sub-disk
# vxdg -g testdg free
GROUP DISK DEVICE TAG OFFSET LENGTH FLAGS
testdg testdg03 c1t4d0s2 c1t4d0 285212672 18874368 -
testdg testdg04 c1t5d0s2 c1t5d0 285826704 869616 -
In this cases the value under the "length" field = 18874368 sectors=9Gb indicate there is only one sub-disk which match the missing sectors and hence it only choice to be used as the sub-disk to re-grow the volume.
# vxassist -g dg growby 18874368 u100 testdg03
# fsck -F vxfs /dev/vx/rdsk/dg/u100
# mount -F vxfs /dev/vx/dsk/testdg/u100 /u100
Note1: At this stage, if you managed to relocate the correct disk, there will be no missing data.
2) No likely candidate for the missing sub-disk and you required to reallocated the missing space from alternative source.
# vxdg -g testdg free
GROUP DISK DEVICE TAG OFFSET LENGTH FLAGS
testdg testdg03 c1t4d0s2 c1t4d0 285212672 144368 -
testdg testdg04 c1t5d0s2 c1t5d0 285826704 3416 -
There is not enough available disk to create the missing sub-disk, the sub-disk may have been reallocated to other volume (u1yy). To recover shrink an existing volume in the disk group (if possible choice a volume using the same the device) to free up the required space.
# vxresize -g testdg shrinkby u1yy 18874368 testdg03 ( free space from testdg03 disk)
# vxassist -g dg growto 18874368 u100 testdg03
# fsck -F vxfs -o full /dev/vx/rdsk/dg/u100
# mount -F vxfs /dev/vx/dsk/testdg/u100 /u100
Alternatively, add a new disk
# vxdisksetup - i. cxtydz
# vxdg -g testdg adddisk testdgxx=cxtydz
# vxassist -g dg growto 94371840 u100 testdgxx
# fsck -F vxfs -o full /dev/vx/rdsk/dg/u100
# mount -F vxfs /dev/vx/dsk/testdg/u100 /u100
Note 2; Since you have reallocated or added a new disk to recreate the missing sub-disk, running fsck - o full will determine if you have lost any data as a result of this operation. Since it may be possible there was no "data" on the sub-disk in the first place.
3) Mutilple sub-disks are available to concatenated to the required length of the missing subdisk
#vxdg -g testdg free
GROUP DISK DEVICE TAG OFFSET LENGTH FLAGS
testdg testdg01 c1t2d0s2 c1t2d0 50331648 16777216 -
testdg testdg02 c1t3d0s2 c1t3d0 236978176 6291456 -
testdg testdg03 c1t4d0s2 c1t4d0 50331648 16777216 -
testdg testdg03 c1t4d0s2 c1t4d0 236978176 6291456 -
testdg testdg03 c1t4d0s2 c1t4d0 285212672 2097152 -
testdg testdg04 c1t5d0s2 c1t5d0 0 16777216 -
testdg testdg04 c1t5d0s2 c1t5d0 163577856 12582912 -
testdg testdg04 c1t5d0s2 c1t5d0 285826704 869616 -
Not easy to guess which of the free sub-disks can be used to make the total =18874368 sectors, as it could be any combination of 2 or more sub-disks. Using an old vxprint or equivalent output we can use vxmake command to recreate the volume bottom-up.
From old vxprint we have :
v u100 - ENABLED ACTIVE 94371840 SELECT - fsgen
pl u100-01 u100 ENABLED ACTIVE 94371840 CONCAT - RW
sd testdg01-01 u100-01 testdg01 0 16777216 0 c1t2d0 ENA
sd testdg01-08 u100-01 testdg01 167772160 16777216 16777216 c1t2d0 ENA
sd testdg01-12 u100-01 testdg01 146800640 20971520 33554432 c1t2d0 ENA
sd testdg01-13 u100-01 testdg01 234881024 20971520 54525952 c1t2d0 ENA
sd testdg03-06 u100-01 testdg03 50331648 16777216 75497472 c1t4d0 ENA
sd testdg03-09 u100-01 testdg03 285212672 2097152 92274688 c1t4d0 ENA
So we need to recreate volume as follows :
(i) recreate the 2 missing subdisks
# vxmake -g testdg sd testdg03-06 disk=testdg03 offset=50331648 length=16777216
# vxmake -g testdg sd testdg03-09 disk=testdg03 offset=285212672 length=2097152
(ii) add subdisks to volume
# vxsd -g testdg assoc u100-01 testdg03-06
# vxsd -g testdg assoc u100-01 testdg03-09
(iii) check status
# vxprint -g testdg -ht
v u100 - ENABLED ACTIVE 75497472SELECT - fsgen
pl u100-01 u100 ENABLED ACTIVE 94371840 CONCAT - RW
sd testdg01-01 u100-01 testdg01 0 16777216 0 c1t2d0 ENA
sd testdg01-08 u100-01 testdg01 167772160 16777216 16777216 c1t2d0 ENA
sd testdg01-12 u100-01 testdg01 146800640 20971520 33554432 c1t2d0 ENA
sd testdg01-13 u100-01 testdg01 234881024 20971520 54525952 c1t2d0 ENA
sd testdg03-06 u100-01 testdg03 50331648 16777216 75497472 c1t4d0 ENA
sd testdg03-09 u100-01 testdg03 285212672 2097152 92274688 c1t4d0 ENA
(iv) grow volume to match and then fsck and mount
# vxresize -g testdg growto u100 94371840
# fsck -F vxfs /dev/vx/rdsk/testdg/u100
# mount -F vxfs /dev/vx/dsk/testdg/u100 /u100
Note 3: Again after replacing the missing sub-disks based on an old configuration, running fsck - o full will determine if you have lost any data as a result of this operation.
