LVM Recovery

A friend of mine made a grieve mistake – partition a disk containing Linux LVM directly on it, without any partition table. Oops.

When dealing with multi-Tera sized disks, one gets to encounter limitations not known on smaller scales – the 2TB limitation. Normal partition table can contain only around 2TB mapping, meaning that to create larger partitions, or even smaller partitions which exceed that specific limit, you have to take one of two actions:

  • Use GPT partition tables, which is meant for large disks, and partition the disk to the size limits you desire
  • Define LVM PV directly on the block device (the command would look like ‘pvcreate /dev/sdb -> see? No partitions)

“Surprisingly” and for no good reason, it appears that the disk which was used completely for the LVM PV suddenly had a single GPT partition on it. Hmmmm.

This is/was a single disk in a two-PV VG continging a single LV spanned all over the VG space. Following the “mysterious” actions, the VG refused to start, claiming that it could not find PV with PVID <some UID>.

This is a step where one should stop and call a professional if he doesn’t know for sure how to continue. These following actions are very risky to your data, and could result in you either recovering from tapes (if exist) or seeking a new job, if this is/was some mission-critical data.

First – go to /etc/lvm/archive and find the latest file named after the VG which has been destroyed. Look into it – you should see the PV is in there. Search the PV based on the UID reported not to repond on the logs.

Second – you need to remove the GPT partition from the disk. The PV will be recreated exactly as it was suppoed to be before. Replace /dev/some_disk with your own device file.

fdisk /dev/some_disk

d

w

Third – Reread the VG archive file, to be on the safe side. Verify again that the PV you are about to recreate is the one you are to. When done, run the following command

pvcreate -u <UID> /dev/some_disk

Again – the name of the device file has been changed in this example to prevent copy-paste incidents from happening.

Fourth – Run vgcfgrestore with the name of the VG as parameter. This command would restore your meta information into the PV and VG.

vgcfgrestore VG_TEST

Fifth – Activate the VG:

vgchange -ay VG_TEST

Now the volumes should be up, and you have the ability to attempt to mount these volumes.

Notice that the data might be corrupted in some way. Running fsck is recommended, although time-consuming.

Good luck!

Tags: , , , ,

3 Responses to “LVM Recovery”

  1. Matt Simmons Says:

    Cool entry, thanks for posting this. Most of us don’t get to deal with >2TB LUNs on a daily basis, but this is going to become more and more common. Nice to see people out there taking the arrows for the rest of us 🙂

    Thanks again!

  2. ez-aton Says:

    The text was altered with some minor corrections. I was so tired yesterday when I wrote this that the final two lines were trailing off.

    The multi TB sized systems is becoming more and more common with the ever-dropping price of storage. I don’t really think that there is any better way of managing these huge amounts of data but to use LVM, but this tool is powerful and dangerous (like all disk-tools) if you don’t know what you’re doing.

    Fixing LVM volume which got abuse is not a trivial action. In this specific case, I was able to save about 9TB of data from human error. Can’t promise it will always be the case 🙂

    Ez

  3. Florin Sfetea Says:

    Many many thanks, this helped a lot !
    Even if I did not had the old archive(backup) file I used one from a similar system. My first try was with a system with a different partitioning so data looked like corrupted.
    Then I found a second system with partitions the same where and I manage to get all my data back untouched.
    The pvcreate on my live fedora needs the restorefile same for vgcfgrestore.

Leave a Reply