Dealing With Duplicate PVIDs for rootvg

Editor’s note: This is the first article in a two-part series on addressing Logical Volume Manager (LVM) issues. Part two will focus on determining volume group (VG) types in AIX.

Recently, I came across an LVM issue on AIX involving duplicate physical volume identifiers (PVIDs) for rootvg. In this article, I’ll explain how I dealt with this challenge.

At a customer site, I was given a new AIX system to build. The OS had already been installed, so all I had to do was update it, configure the NIM and a few other UNIX services. But as soon as I logged on, I immediately found a problem. The output from the lspv command indicated two disks were assigned to rootvg. This was unexpected, as I’d asked for two disks: one for rootvg and another for nimvg. But that was OK; I could always reduce rootvg and take out one of the disks. However, upon closer inspection, I discovered both disks had the same PVID. This was a problem.

# lspv
hdisk8 00f6050a2cd79ef8 rootvg          active
hdisk7 00f6050a2cd79ef8 rootvg          active

To use one of the disks for nimvg, I would need to remove the duplicate PVID from one of the disks, but which one? I could clear a PVID from an hdisk using the chdev command. I attempted to do this on hdisk7 first. It reported that the disk was “busy.” This meant it was active in rootvg.

# chdev -a pv=clear -l hdisk7
Method error (/etc/methods/chgdisk):
        0514-062 Cannot perform the requested function because the
                 specified device is busy.
     pv

So hdisk8 must be the disk with the duplicate PVID. I executed the chdev command, and it successfully cleared the PVID from this disk.

# chdev -a pv=clear -l hdisk8
hdisk8 changed

To ensure the object data manager (ODM) was consistent with the on-disk information, I then ran the synclvodm command. This would synchronize and rebuild the logical volume control block (LVCB), the ODM and the volume group descriptor area (VGDA).

# synclvodm -v rootvg
synclvodm: Physical volume data updated.
synclvodm: Logical volume hd5 updated.
synclvodm: Logical volume hd6 updated.
synclvodm: Logical volume hd8 updated.
synclvodm: Logical volume hd4 updated.
synclvodm: Logical volume hd2 updated.
synclvodm: Logical volume hd9var updated.
synclvodm: Logical volume hd3 updated.
synclvodm: Logical volume hd1 updated.
synclvodm: Logical volume hd10opt updated.
synclvodm: Logical volume hd11admin updated.
synclvodm: Logical volume livedump updated.
synclvodm: Logical volume isolv updated.
synclvodm: Logical volume mylv updated.
synclvodm: Logical volume lv2 updated.

Now I was in a much better position, one in which I could create a new volume group using hdisk8. The disk was currently without a PVID, as evident by this lspv output:

# lspv
hdisk8 none None
hdisk7          00f6050a2cd79ef8                    rootvg          active

To guarantee hdisk8 would be allocated a new, unique PVID on VG creation, I ran the chdev command to allocate one. The disk was assigned a new PVID, different from that of hdisk7.

# chdev -a pv=yes -l hdisk8
hdisk8 changed

# lspv
hdisk8 00f604889bd16959 None
hdisk7          00f6050a2cd79ef8                    rootvg          active

From this point, I could create a new VG using hdisk8.

# mkvg -f -y nimvg hdisk8
nimvg

# lspv
hdisk8          00f604889bd16959                    nimvg           active
hdisk7          00f6050a2cd79ef8                    rootvg          active

You may ask, how did we end up with duplicate PVIDs in the first place? This also crossed my mind. If I’d had more time to investigate, my first stop would have been to the storage administrator’s desk. Perhaps the way he was replicating or provisioning the SAN logical unit numbers (LUNs) was causing the duplicate PVID issue?

My next stop would have been to the UNIX administrator who installed AIX. Perhaps he was doing something strange when installing/configuring the OS or, more likely, something odd with the way in which he was assigning the disk from the virtual I/O server to the AIX client (using cpdvi or dd, for example).

My best guess was they were replicating rootvg (using some form of copy services functionality on their storage subsystem). And every time they deployed a new AIX system, they were assigning a “new” replicated LUN (with AIX preinstalled), rather than installing AIX from media each time. Therefore, the disk replication process was duplicating the PVID.