Posts Tagged ‘XenServer’

XenServer and its damn too small system disks

Thursday, December 26th, 2013

I love XenServer. I love the product, I believe it to be a very good answer for SMBs, and enterprises. It lacks on external support, true, but the price tag for many of the ‘external capabilities’ on VMware, for instance, are very high, so many SMBs, especially, learn to live without them. XenServer gives a nice pack of features, at a very reasonable price.

One of the missing features is the management packs of hardware vendors, such as HP, Dell and IBM. Well, HP does have something, and its installation is always some sort of a challenge, but they do, so scratch that. Others, however, do not supply management packs. The bright side is that with Domain0 being a full featured i386 Centos 5 distribution, I can install the Centos/RHEL management packs, and have a ball. This brings us to another challenge there – the size of the system disk (root partition) by default is too small – 4GB, and while it works quite well without any external components, it tends to get filled very fast with external packages installed, like Dell tools, etc. Not only that, but on a system with many patches the patches backups take their toll, and consume valuable space. While my solution will not work for those who aim at the smallest possible space, such as SD or Disk-on-Key for the XenServer OS, it aims for the most of us all, where the system resides on several tenths of gigabytes at least, and is capable of sustaining the ‘loss’ of additional 4GB. This process modifies the install.img file, and authors the CD as a new one, your own privately-modified instance of XenServer installation. Mind you that this change will be effective only for new installations. I have not tested this as the upgrade path for existing systems, although I believe no harm will be done to those who upgrade. Also – it was performed and tested on XenServer 6.2, and not 6.2 SP1, or prior versions, although I believe that the process should look pretty similar in nature.

You will need a Linux machine to perform this operation, end to end. You could probably use some Windows applications on the way, but I have no idea as to which or what.

Step one: Open the ISO, and copy it to somewhere useful (assume /tmp is useful):

mkdir /tmp/ISO
mkdir /tmp/RW
mount -o loop /path/to/XenServer-6.2.0-install-cd.iso /tmp/ISO
cd /tmp/ISOtar cf – . | ( cd /tmp/RW ; tar xf – )

Step two: Extract the contents of the install.img file in the root of the CDROM:

mkdir /tmp/install
cd /tmp/install
cat /tmp/RW/install.img | gzip -dc | cpio -id

Step three: Edit the contents of the definitions file:

vi opt/xensource/installer/

Change the value of ‘root_size’ to something to your taste. Mind you that with 4GB it was tight, but still usable, even with additional 3rd party tools, so don’t become greedy. I defined it to be 6GB (6144)

Step four: Wrap it up:

cd /tmp/install ; find . | cpio -o -H newc | gzip -9 > /tmp/RW/install.img

Step five: Author the CD, and prepare it to be burned:

cd /tmp/RW
mkisofs -J -T -o /share/temp/XenServer-6.2-modified.iso -V “XenServer 6.2” -volset “XenServer 6.2” -A “XenServer 6.2”
-b boot/isolinux/isolinux.bin -no-emul-boot -boot-load-size 4 -boot-info-table -R -m TRANS.TBL .

You now have a file called ‘XenServer-6.2-modified.iso’ in your /tmp, which will install your XenServer with the disk partition size you have set it to install. Cheers.

BTW, and to make it entirely clear – I cannot be held responsible to any damage caused to any system you tweaked using this (or for that matter – any other) guide I published.

Enjoy your XenServer’s new apartment!

XenServer – increase LVM over iSCSI LUN size – online

Wednesday, September 4th, 2013

The following procedure was tested by me, and was found to be working. The version of the XenServer I am using in this particular case is 6.1, however, I belive that this method is generic enough so that it could work for every version of XS, assuming you're using iSCSI and LVM (aka - not NetApp, CSLG, NFS and the likes). It might act as a general guideline for fiber channel communication, but this was not tested by me, and thus - I have no idea how it will work. It should work with some modifications when using Multipath, however, regarding multipath, you can find in this particular blog some notes on increasing multipath disks. Check the comments too - they might offer some better and simplified way of doing it.

So - let's begin.

First - increase the size of the LUN through the storage. For NetApp, it involves something like:

lun resize /vol/XenServer/luns/SR1.lun +1t

You should always make sure your storage volume, aggregate, raid group, pool or whatever is capable of holding the data, or - if using thin provisioning - that a well tested monitoring system is available to alert you when running low on storage disk space.

Now, we should identify the LUN. From now on - every action should be performed on all XS pool nodes, one after the other.

cat /proc/partitions

We should keep the output of this command somewhere. We will use it later on to identify the expanded LUN.

Now - let's scan for storage changes:

iscsiadm -m node -R

Now, running the previous command again will have a slightly different output. We can not identify the modified LUN

cat /proc/partitions

We should increase it in size. XenServer uses LVM, so we should harness it to our needs. Let's assume that the modified disk is /dev/sdd.

pvresize /dev/sdd

After completing this task on all pool hosts, we should run sr-scan command. Either by CLI, or through the GUI. When the scan operation completes, the new size would show.

Hope it helps!

XenServer 6.2 is now Open Source!

Tuesday, June 25th, 2013

It is an amazing news to me. I really love XenServer. I think that Citrix were able to make a good use of Linux mechanisms for the purposes of virtualization, without abusing the OS layer (like some of the other virtualization solutions did). The file locations are decent (for example – most parts are located in /opt, which is the right place for it to be at), and in general, it always felt to me as if Citrix developers (and the original XenSource developers) had respect for the OS. I liked it, and still do.

The product was not perfect. There were ups and downs, there were times when I cursed it, and times when I was full of joy by its behavior – which can happen from time to time, if you really like and care about a software product.

So, today Citrix announced that XenServer 6.2, the shiny new release, will become fully open-sourced. That the entire feature-set of XenServer previous versions can be yours for free. However, for support and some minor administrative tasks, you will want to purchase the licensed version. As far as I understood, these are the differences. Almost nothing more. Wow. Kudos!

Grab it while it’s hot here!

Recovery of a StorageRepository (SR) in XenServer, part one

Wednesday, February 6th, 2013

In this part I will discuss a possible solution to a problem I encountered several times already – failure to understand XenServer use of LVM, but first – a little explanation of the topic.

XenServer makes extensive use of LVM technology in order to support the storage requirements of virtual disks. It is being utilized in two methods – LVMoISCSI/LVMoHBA and ext. In both cases, XenServer defines the initial layout as a LVM framework. The LVM, except for the system disk, is positioned directly on the disk in whole, and not on the first partition. I imagine that the desire to avoid dealing with GPT/Basic/Other partitioning schemes is the root of this notion. While it does solve the disk partitioning method problem, it creates a different problem – PEBKC problem (Problem Exists Between Keyboard and Chair). Lack of understanding that there is no partition on the disk, but the data is structured directly on it, is the cause of relatively frequent deletion of the LVM structure as it being replaced by a partitioning layout. The cause of it can be one of two common problems – the first is that the LUN/disk is exposed directly to a Windows machine, which asks joyfully if one would like to ‘sign the partition’. If one does so, a basic partitioning structure is created, and the LVM data structure is overwritten by it. The second problem is a little less common, and involves lack of understanding of the LVM structure as employed by XenServer, when performing disk tasks as the root user on the XenServer host directly. In this case, the user will not be aware of the data structure, and might be tempted to partition, and God forbid – even format the created partition. The result would be a total loss of the SR.

This was about how data is structured and how it is erased or damaged.

I was surprised to discover the ‘easy’ method of recovery from a partitioning table layer over the LVM metadata. I assume that no one has attempted to format the resulting partition(s), but stopped only at creating the partition layout and attempting to understand why it doesn’t work anymore in XenServer.

The easy way, which will be discussed here, is the first of two articles I intend on writing about LVM recovery. If this ‘easy’ method works for you – no need to try your luck with the more complex one.

So, to work. In case someone has created a partition layout, overwriting, as explained earlier, the LVM metadata structure, the symptoms would be that a disk will have (a) partition(s). For example, the results of ‘cat /proc/partitions’ would look like that (snipping the irrelevant parts)

8         16        156290904 sdb
8        17        156288321 sdb1

As clearly visible – the bold line should not be there. The output of ‘fdisk -l /dev/sdb’ showed (again – snipping the irrelevant parts):

/dev/sdb1                                1                   19457                 156288321       83  Linux

It proves someone has manually attempted to partition the disk. Had a mount command worked (example: ‘mount /dev/sdb1 /mnt’) my response e-mail message would go like this: “Sorry. The data was overwritten. Can’t do anything about it”, however, this was not the case. Not this time.

The magic trick I used was to remove the partition entirely, freeing the disk to be identified as LVM, if it could – I wasn’t sure it would – and then take some recovery actions.

First – fdisk to remove the partition:

fdisk /dev/sdb << EOF

Now, a pvscan operation could take place. The following command returned the correct value – a PV ID which wasn’t there before, meaning that the PV information was still intact:


Now, a simple ‘SR Repair’ operation could take place.

My next article in this series will show a more complex method of recovery to employ when this ‘easy’ one doesn’t work.

XenServer – Setting virtual disks names based on the VM names

Wednesday, January 2nd, 2013

One of the worst things you can have in XenServer, is some wize-guy performing a ‘forget storage’ on a storage device still holding virtual disks related to VMs. As XenServer database is internal (for the whole pool) and not per-VM, all references to this virtual disks disappear, and you remain with bunch of VMs without disks, and later on, when the recovered from the shock and restored the SR, with a bunch of virtual disks you have no clue as to where they belong. Why? Because we are lazy, and we tend to skip the part where you can (or is it – should?) define a custom name for your virtual disks so you would know later on (for example – in the case specified above) where they belong(ed).

To solve this annoying issue, and to save time for Citrix XenServer admins, I have created a script which resets the VDI (virtual disk object) names to the name of the VM+ the logical position of the virtual disk (example: xvda, hdb, etc), related to the VM. That way, it will become very easy to identify the disks in case of such annoying micro-catastrophy (micro because no data is lost, just where it belongs…).

The script can be called manually, and since we’re lazy people, and we will forget to handle it manually every said interval, and will accumulate virtual machines with “Template of XYZ” virtual disks, it can be called from cron. When called manually, it asks the user to proceed by pressing ‘Enter’. If called from cron, it just runs.



# This script will reset the names of the virtual disks used for each VM to a standard name, based on the VM name
# and the disk position
# It is meant to solve problems where due to 'forget storage' operations or the likes
# virtual disk associations disappear, and you face many disks having the same name
# Written by Ez-Aton:

if [ -t 1 ]
        echo "This script will reset *all* VM disks to a name constructed of the VM and the disk name (xvda, hdb, etc)"
        echo "This operation is not reversible, however, it can be called repeatedly"
        echo "If you want this script to skip a said virtual disk, make sure its name includes the name of the VM"
        echo "For example 'vm1 the real important data disk' for a disk used by vm1."
        echo "Note that the name is case sensitive, and it is very important that to match the name using upper/lower case letters as needed"
        echo "To abort, press Ctrl+C"
        echo "To proceed, press Enter"
        read abc

VM_LIST=`xe vm-list is-control-domain=false --minimal | tr , ' '`

for i in $VM_LIST
        # Resetting several parameters, so we have a clean start
        # We iterate through all existing VMs, to get both their names, and their disks
        VM_NAME="`xe vm-param-get uuid=$i param-name=name-label`"
        if [ -z "$VM_NAME" ]
                # We have a problem with empty VM names, so we will use the VMs uuid
        VBD_LIST=`xe vbd-list vm-uuid=$i --minimal | tr , ' '`
        for j in $VBD_LIST
                # Resetting several parameters, so we have a clean start
                # We iterate through all existing VBDs to reset the VDI nane
                VDI_UUID=`xe vbd-param-get uuid=$j param-name=vdi-uuid`
                if [ "$VDI_UUID" == "" ]
                        # This is a virtual CDROM
                DEV_NAME=`xe vbd-param-get uuid=$j param-name=device`
                VDI_NAME=`xe vbd-param-get uuid=$j param-name=vdi-name-label`

                # Test if the name was reset in the past or manually
                TGT_NAME="$VM_NAME $DEV_NAME"
                if [[ "$TGT_NAME" = "$VDI_NAME" ]]
                        # There is nothing to do
                        echo "Name already includes VM name, so nothing to do"
                        # Here we reset the VDI name
                        echo xe vdi-param-set uuid=$VDI_UUID name-label="$TGT_NAME"
                        xe vdi-param-set uuid=$VDI_UUID name-label="$TGT_NAME"