Posts Tagged ‘volume group’

Relocating LVs with snapshots

Monday, February 2nd, 2009

Linux LVM is a wonderful thing. It is scalable, flexible, and truly, almost enterprise-class in every details. It lacks, of course, at IO performance for LVM snapshots, but this can be worked-around in several creative ways (if I haven’t shown here before, I will sometime).

What it can’t do is dealing with a mixture of Stripes, Mirrors and Snapshots in a single logical volume. It cannot allow you to mirror a stripped LV (even if you can follow the requirementes), it cannot allow you to snapshot a mirrored or a stripped volume. You get the idea. A volume you can protect, you cannot snapshot. A volume with snapshots cannot be mirrored or altered.

For the normal user, what you get is usually enough. For storage management per-se, this is just not enough. When I wanted to reduce a VG – remove a disk from an existing volume group,  I had to evacuate it from any existing logical volume. The command to perform this actions is ‘pvmove‘ which is capable of relocating data from within a PV to other PVs. This is done through mirroring each logical volume and then removing the origin.

Mirroring, however, cannot be performed on LVs with snapshots, or on an already mirrored LV, so these require different handling.

We can detect which LVs reside on our physical volume by issuing the following command

pvdisplay -m /dev/sdf1

/dev/sdf1 was only an example. You will see the contents of this PV. So next, performing

pvmove /dev/sdf1

would attempt to relocate every existing LV from this specific PV to any other available PV. We can use this command to change the disk balance and allocations on multi-disk volume groups. This will be discussed on a later post.

Following a ‘pvmove‘ command, all linear volumes are relocated, if space permits, to another PVs. The remaining LVs are either mirrored or LVs with snapshots.

To relocate a mirrored LV, you need to un-mirror it first. To do so, first detect using ‘pvdisplay‘ which LV is belongs to (the name should be easy to follow) and then change it to non-mirrored.

lvconvert -m0 /dev/VolGroup00/test-mirror

This will convert it to be a linear volume instead of a mirror, so you could move it, if it still resides on the PV you are to remove.

Snapshot volumes are more complicated, due to their nature. Since all my snapshots are of a filesystem, I could allow myself to use tar to perform the action.

The steps are as follow:

  1. tar the contents of the snapshot source to nowhere, but save an incremental file
  2. Copy the source incremental file to a new name, and tar the contents of a snapshot according to this copy.
  3. Repeat the previous step for each snapshot.
  4. Remove all snapshots
  5. Relocate the snapshot source using ‘pvmove
  6. Build the snapshots and then recover the data into them

This is a script to do steps 1 to 3. It will not remove LVs, for obvious reasons. This script was not tested, but should work, of course 🙂

None of the LVs should be mounted for it to function. It’s better to have harder requirements than to destroy data by double-mounting it, or accessing it while it is being changed.

# Get: VG Base-LV, snapshot name, snapshot name, snapshot name...
# Example:
# ./backup VolGroup00 base snap1 snap2 snap3
# Written by Ez-Aton

if [ "[email protected]" -le 3 ]
   echo "Parameters: $0 VG base snap snap snap snap"
   exit 1
shift 2

function check_not_mounted () {
   # Check if partition is mounted
   if mount | grep /dev/mapper/${VG}-${1}
      return 0
      return 1

function create_base_diff () {
   # This function will create the diff file for the base
   mount /dev/${VG}/${BASE} $MNT
   if [ $? -ne 0 ]
      echo "Failed to mount base"
      exit 1
   cd $MNT
   tar -g $TARGET/${BASE}.tar.gz.diff -czf - . > /dev/null
   cd -
   umount $MNT

function create_snap_diff () {
   mount /dev/${VG}/${1} $MNT
   if [ $? -ne 0 ]
      echo "Failed to mount base"
      exit 1
   cp $TARGET/${BASE}.tar.gz.diff $TARGET/$1.tar.gz.diff
   cd $MNT
   tar -g $TARGET/${1}.tar.gz.diff -czf $TARGET/${1}.tar.gz .
   cd -
   umount $MNT

function create_mount () {
   # Creates a temporary mount point
   if [ ! -d /mnt/$$ ]
      mkdir /mnt/$$

if check_not_mounted $BASE
   echo "$BASE is mounted. Exiting now"
   exit 1
for i in [email protected]
   if check_not_mounted $i
      create_snap_diff $i
      echo "$i is mounted! I will not touch it!"

The remaining steps should be rather easy – just mount the newly created snapshots and restore the tar file on them.

Aquiring and exporting external disk software RAID and LVM

Wednesday, August 22nd, 2007

I had one of my computers die a short while ago. I wanted to get the data inside its disk into another computer.

Using the magical and rather cheap USB2SATA I was able to connect the disk, however, the disk was part of a software mirror (md device) and had LVM on it. Gets a little complicated? Not really:

(connect the device to the system)

Now we need to query which device it is:


It is quite easy. In my case it was /dev/sdk (don’t ask). It shown something like this:

usb 1-6: new high speed USB device using address 2
Initializing USB Mass Storage driver…
scsi5 : SCSI emulation for USB Mass Storage devices
Vendor: WDC WD80 Model: WD-WMAM92757594 Rev: 1C05
Type: Direct-Access ANSI SCSI revision: 02
SCSI device sdk: 156250000 512-byte hdwr sectors (80000 MB)
sdk: assuming drive cache: write through
SCSI device sdk: 156250000 512-byte hdwr sectors (80000 MB)
sdk: assuming drive cache: write through
sdk: sdk1 sdk2 sdk3
Attached scsi disk sdk at scsi5, channel 0, id 0, lun 0

This is good. The original system was RH4, so the standard structure is /boot on the first partition, swap and then one large md device containing LVM (at least – my standard).

Lets list the partitions, just to be sure:

# fdisk -l /dev/sdk

Disk /dev/sdk: 80.0 GB, 80000000000 bytes
255 heads, 63 sectors/track, 9726 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sdk1 * 1 13 104391 fd Linux raid autodetect
/dev/sdk2 14 144 1052257+ 82 Linux swap
/dev/sdk3 145 9726 76967415 fd Linux raid autodetect

Good. As expected. Let’s activate the md device:

# mdadm –assemble /dev/md2 /dev/sdk3
mdadm: /dev/md2 has been started with 1 drive (out of 2).

It’s going well. Now we have the md device active, and we can try to scan for LVM:

# pvscan

PV /dev/md2 VG SVNVG lvm2 [73.38 GB / 55.53 GB free]

Activating the VG is a desired action. Notice the name – SVNVG (a note at the bottom):

# vgchange -a y /dev/SVNVG
3 logical volume(s) in volume group “SVNVG” now active

Now we can list the LVs and mount them on our desired location:

]# lvs
LV VG Attr LSize Origin Snap% Move Log Copy%
LogVol00 SVNVG -wi-a- 2.94G
LogVol01 SVNVG -wi-a- 4.91G
VarVol SVNVG -wi-a- 10.00G


mount /dev/SVNVG/VarVol /mnt/

and it’s all ours.

To remove this connected the disk, we need to reverse the above process.

First, we will umount the volume:

umount /mnt

Now we need to disable the Volume Group:

# vgchange -a n /dev/SVNVG
0 logical volume(s) in volume group “SVNVG” now active

0 logical volumes active means we were able to disable the whole VG.

Disable the MD device:

# mdadm –manage -S /dev/md2

Now we can disconnect the physical disk (actually, the USB) and continue with out life.

A note: RedHat systems name their logical volumes using a default name VolGroup00. You cannot have two VGs with the same name! If you activate a VG which originated from RH system and used a default name, and your current system uses the same defaults, you need to connect the disk to an external system (non RH would do fine) and change the VG name using vgrename before you can proceed.