Running Systems

The simple scripts in /etc/xen/scripts which manage networking are fine for most usages, however, when your server is using bonding together with VLAN tagging (802.11q) you should consider an alternative.

A PDF document written by Mark Nielsen, GPS Senior Consultant, Red Hat, Inc (I lost the original link, sorry) named “BOND/VLAN/Xen Network Configuration” as a service to the community, game me few insights on the subject. Following one of its references, I saw a bit more elegant method of doing a bridging setup under RedHat, which takes managing the bridges away from xend, and leaves it at the system level. Lets see how it’s done on RedHat style Linux distribution.

Manage your normal networking configurations

If you’re using VLAN tagging over bonding, than you should have to setup a bonding device (be it bond0) which has definitions such as this:

/etc/sysconfig/network-scripts/ifcfg-eth0 and /etc/sysconfig/network-scripts/ifcfg-eth1

DEVICE=eth0
BOOTPROTO=none
ONBOOT=yes
MASTER=bond0
SLAVE=yes
ISALIAS=no

/etc/sysconfig/network-scripts/ifcfg-bond0

DEVICE=bond0
BOOTPROTO=none
BONDING_OPTS=”mode=1 miimon=100″
ONBOOT=yes

This is rather stright-forward, and should be rather a default for such a setup. Now comes the more interesting part. Originally, the next configuration part would be bond0.2 and bond0.3 (in my example). The original configuration would have looked like this (this is in bold because I tend to fast-read myself, and tend to miss things too often). This is not how it should look when we’re done!

/etc/sysconfig/network-scripts/ifcfg-bond0.2 (same applies to ifcfg-bond0.3)

DEVICE=bond0.2
BOOTPROTO=static
IPADDR=192.168.0.2
NETMASK=255.255.255.0
ONBOOT=yes
VLAN=yes

Configure bridging

To setup a bridge device for bond0.2, replace the mentioned above ifcfg-bond0.2 with this new /etc/sysconfig/network-scripts/ifcfg-bond0.2

DEVICE=bond0.2
BOOTPROTO=static
ONBOOT=yes
VLAN=yes
BRIDGE=xenbr0

Now, create a new file /etc/sysconfig/network-scripts/ifcfg-xenbr0

DEVICE=xenbr0
BOOTPROTO=static
IPADDR=192.168.0.2
NETMASK=255.255.255.0
ONBOOT=yes
TYPE=bridge

Now, on network restart, the bridge will be brought up, holding the right IP address – all done by initscripts, with no Xen intervention. You will want to repeat the last the “Configure bridge” part for any additional bridge you want to be enabled for Xen machines.

Don’t let Xen bring any bridges up

This is the last part of our drill, and it is very important. If you don’t do it, you’ll get a nice networking mess. As said before, Xen (community), by default, can’t handle bondings or vlan tags, so it will attempt to create or modify bridges to eth0 or the likes. Edit /etc/xen/xend-config.sxp and remark any line containing a directive containing starting with “network-script“. Such a directive would be, for example

(network-script network-bridge)

Restart xend and restart networking. You should now be able to configure VMs to use xenbr0 and xenbr1, etc (according to your own personal settings).

Unlike VMware Server, Xen’s HyperVisor does not allow an easy collection of performance information. The management machine, called “Domain-0″ is actually a privileged virtual machine, and thus – get its own small share of CPUs and RAM. Collecting performance information on it will lead to, well, collecting performance information for a single VM, and not the whole bunch.

Local tools, such as “xentop” allows collection of information, however, combining this with Cacti, or any other SNMP-based collection tool is a bit tricky.

A great solution is provided by Ian P. Christian in his blog post about Xen monitoring. He has created a Perl script to collect information. I have taken the liberty to fix several minor things with his permission. The modified scripts are presented below. Name the script (according to your version of Xen) “/usr/local/bin/xen_stats.pl” and set it to be executable:

For Xen 3.1

#!/usr/bin/perl -w
 
use strict;
 
# declare...
sub trim($);
#<a href="/blog/files/xen_cloud.tar.gz" title="xen_cloud.tar.gz" target="_blank">xen_cloud.tar.gz</a>
# we need to run 2 iterations because CPU stats show 0% on the first, and I'm putting .1 second betwen them to speed it up
my @result = split(/\n/, `xentop -b -i 2 -d.1`);
 
# remove the first line
shift(@result);
 
shift(@result) while @result && $result[0] !~ /^xentop - /;
 
# the next 3 lines are headings..
shift(@result);
shift(@result);
shift(@result);
shift(@result);
 
foreach my $line (@result)
{
  my @xenInfo = split(/[\t ]+/, trim($line));
  printf("name: %s, cpu_sec: %d, cpu_percent: %.2f, vbd_rd: %d, vbd_wr: %d\n",
    $xenInfo[0],
    $xenInfo[2],
    $xenInfo[3],
    $xenInfo[14],
    $xenInfo[15]
    );
}
 
# trims leading and trailing whitespace
sub trim($)
{
  my $string = shift;
  $string =~ s/^\s+//;
  $string =~ s/\s+$//;
  return $string;
}

For Xen 3.2 and Xen 3.3

#!/usr/bin/perl -w
 
use strict;
 
# declare…
sub trim($);
 
# we need to run 2 iterations because CPU stats show 0% on the first, and I’m putting .1 second between them to speed it up
my @result = split(/\n/, `/usr/sbin/xentop -b -i 2 -d.1`);
 
# remove the first line
shift(@result);
shift(@result) while @result && $result[0] !~ /^[\t ]+NAME/;
shift(@result);
 
foreach my $line (@result)
{
        my @xenInfo = split(/[\t ]+/, trim($line));
        printf(“name: %s, cpu_sec: %d, cpu_percent: %.2f, vbd_rd: %d, vbd_wr: %d\n“,
        $xenInfo[0],
        $xenInfo[2],
        $xenInfo[3],
        $xenInfo[14],
        $xenInfo[15]
        );
}
# trims leading and trailing whitespace
sub trim($)
{
        my $string = shift;
        $string =~ s/^\s+//;
        $string =~ s/\s+$//;
        return $string;
}

Cron settings for Domain-0

Create a file “/etc/cron.d/xenstat” with the following contents:

# This will run xen_stats.pl every minute
*/1 * * * * root /usr/local/bin/xen_stats.pl > /tmp/xen-stats.new && cat /tmp/xen-stats.new > /var/run/xen-stats

SNMP settings for Domain-0

Add the line below to “/etc/snmp/snmpd.conf” and then restart the snmpd service

extend xen-stats   /bin/cat /var/run/xen-stats

Cacti

I reduced Ian Cacti script to be based on a per-server setup, meaning this script gets the host (dom-0) name from Cacti, but cannot support live migrations. I will try to deal with combining live migrations with Cacti in the future.

Download and extract my modified xen_cloud.tar.gz file. Extract it, place the script and config in its relevant location, and import the template into Cacti. It should work like charm.

A note – the PHP script will work only on PHP5 and above. Works flawlessly on Centos5.2 for me.

Linux works perfectly well with multiple storage links using dm-multipath. Not only that, but HP has released their own spawn of dm-multipath, which is optimized (or so claimed, but, anyhow, well configured) to work with EVA and MSA storage devices.

This is great, however, what do you do when mapping volume snapshots through dm-multipath? For each new snapshot, you enjoy a new WWID, which will remap to a new “mpath” name, or raw wwid (if “user_friendly_name” is not set). This can, and will set chaos to remote scripts. On each reboot, the new and shiny snapshot will aquire a new name, thus making scripting a hellish experience.

For the time being I have not tested ext3 labels. I suspect that using labels will fail, as the dm-multipath over layer device does not hide the under layered sd devices, and thus – the system might detect the same label more than once – once for each under layered device, and once for the dm-multipath over layer.

A solution which is both elegant and useful is to fixate the snapshots’ WWID through a small alteration to SSSU command. Append a string such as this to the snap create command:

WORLD_WIDE_LUN_Name="6300-0000-0000-0000-0010-0000"

Don’t use the numbers supplied here. “invent” your own

Mind you that you must use dashes, else the command will fail.

Doing so will allow you to always use the same WWID for the snapshots, and thus – save tons of hassle after system reboot when accessing snapshots through dm-multipath.

I was required to write a small listener. This listener doesn’t have to do anything fancy – it has to process input from a network source (over TCP) and shove it into a file with random name.

The quick and dirty solution is to use netcat (nc). A short command such as this would do the trick easily:

nc -l -p 10025 > /tmp/file.$$

This, however, will not allow multiple sessions, and, netcat tends to close when the session ends, which will require me to hack some weird workaround. Not the best method.

Well, then, to inetd we aim. I have decided to aim at inetd and not the more modern xinetd, since the previous is somewhat quicker to setup, and above all, it doesn’t really matter. You are most invited to transform the line supplied here to any other supersever you like

The script is located inside /usr/local/bin and it is called tcp_script . It is a very simple script:

#!/bin/bash
dd status=noxfer of=/tmp/out.$$

I have used ‘dd‘ because it worked well. I am sure other solutions are available as well.

inetd will redirect the standard input to the script, which will redirect it, in turn, to the file /tmp/out.$$

10025 stream tcp nowait root /usr/local/bin/tcp_script tcp_script

Reload (HUP, whatever) inetd, and you’re set.

It’s either me, or NetworkManager sux. This is the most incomplete and incompetent piece of software I have ever seen. The only reason I actually use it is that there is no better alternative which does whatever NetworkManager manages, somehow, to do.

My initial thought was to setup NetworkManager as the dial-up component. What should be more obvious than that? Well, it just won’t connect, and its logs are horrific at best. So back to gksudo script for us, then…

You can put this script cell.sh in /usr/local/sbin/ and change its permissions to eXecutable (by root, of course), and then add a nice menu item called “Cellular Internet” in your Gnome Menu. It should run ‘gksudo /usr/local/sbin/cell.sh‘

That’s all.

To get your newly purchased Sierra AirCard 880E on your Ubuntu Hardy, you should follow these simple steps:

1. Make sure you have GCC on your computer: ‘sudo apt-get install gcc‘
2. Make sure you have kernel sources installed on your computer: ‘sudo apt-get install linux-headers-2.6.24-19-generic‘
3. Download the driver from Sierra Wireless site
4. Extract and compile (‘make‘, ‘sudo make install‘)
5. Add to /etc/modprobe.d/blacklist-modem the line: ‘blacklist ehci_hcd’
6. Insert the device into the laptop. When everything is OK, you should see that a USB device is registered using ‘dmesg‘ shortly afterwards, or ‘lsusb‘
7. Configure /etc/wvdial.conf using the attached config file wvdial.conf
8. Add to /etc/ppp/peers the file wvdial

Done. To connect, you need to run ‘sudo wvdial &‘

I will add in the future a nice gksudo with a nice-looking connect/disconnect script

This is an interesting bug I have encountered:

The output of an sssu command should look like this:

EVA> DELETE STORAGE “\Virtual Disks\Linux\oracle\SNAP_ORACLE”

EVA>

It still leaves the snapshot (SNAP_ORACLE in this case) visible, until the web interface is used to press on “Ok”.

This happened to me on HP EVA with HP StorageWorks Command View EVA 7.0 build 17.

When sequential delete command is given, it looks like this:

EVA> DELETE STORAGE “\Virtual Disks\Linux\oracle\SNAP_ORACLE”

Error: Error cannot get object properties. [ Deletion completed]

EVA>

When this command is given for a non-existing snapshot, it looks like this:

EVA> DELETE STORAGE “\Virtual Disks\Linux\oracle\SNAP_ORACLE”

Error: \Virtual Disks\Linux\oracle\SNAP_ORACLE not found

So I run the removal command twice (scripted) on an sssu session without “halt_on_errors”. This removes the snapshots correctly.

We are about to have a stand in a show in Israel. To pull some attention, I have searched for a method to automate a random rotation of the famous Beryl/Compiz cube.

An extension of the method provided in here (using macros) is demonstrated below, using a script.

This is a bit more complicated, as I have used the motions of the mouse to achieve a “show” out of it (just changing desktops isn’t enough nowadays…)

Check out below for the full script.

#!/bin/bash
# This script will rotate the cube one click on each direction each predefined
# timer.
# Written by ezaton at tournament.org.il
# Check out my technical blog “Running Systems” at http://www.tournmament.org.il/run

# Set timer (seconds)
TIMER=10
VERT_TIMER=1

# Possible directions? 4 (up, down, left, right. will be marked from 0 to 3)
# Addition – set it to give higher priority to side-stepping. So max is 10, and only 0&1 represent
# up/down
# Added 10 to represent 2xleft and 11 to represent 2xright
POSS=12

# Temp command file
TMP_FILE=/tmp/rotate.macro

function create_file {
# This function will create and secure the macro file
\rm $TMP_FILE
if [ -f $TMP_FILE ]; then
echo “$TMP_FILE still exists”
exit 1
fi
echo “” > $TMP_FILE
chmod 700 $TMP_FILE
}

function run_macro {
# Run the actual macro
cat $TMP_FILE | xmacroplay $DISPLAY &>/dev/null
}

function left {
# This function will build the macro file for the “left” command
echo > $TMP_FILE
echo “MotionNotify 100 380″ >> $TMP_FILE
echo “KeyStrPress Alt_L” >> $TMP_FILE
echo “KeyStrPress Control_L” >> $TMP_FILE
echo “ButtonPress 1″ >> $TMP_FILE
echo “MotionNotify 100 380″ >> $TMP_FILE
echo “MotionNotify 130 380″ >> $TMP_FILE
echo “MotionNotify 150 380″ >> $TMP_FILE
echo “MotionNotify 170 380″ >> $TMP_FILE
echo “MotionNotify 190 380″ >> $TMP_FILE
echo “MotionNotify 210 380″ >> $TMP_FILE
echo “MotionNotify 230 380″ >> $TMP_FILE
echo “MotionNotify 250 380″ >> $TMP_FILE
echo “MotionNotify 270 380″ >> $TMP_FILE
echo “MotionNotify 290 380″ >> $TMP_FILE
echo “MotionNotify 310 380″ >> $TMP_FILE
echo “MotionNotify 330 380″ >> $TMP_FILE
echo “ButtonRelease 1″ >> $TMP_FILE
echo “KeyStrRelease Control_L” >> $TMP_FILE
echo “KeyStrRelease Alt_L” >> $TMP_FILE
run_macro
}

function up {
# This function will build the macro file for the “up” command
echo > $TMP_FILE
echo “MotionNotify 100 100″ >> $TMP_FILE
echo “KeyStrPress Alt_L” >> $TMP_FILE
echo “KeyStrPress Control_L” >> $TMP_FILE
echo “ButtonPress 1″ >> $TMP_FILE
echo “MotionNotify 100 100″ >> $TMP_FILE
echo “MotionNotify 100 120″ >> $TMP_FILE
echo “MotionNotify 100 140″ >> $TMP_FILE
echo “MotionNotify 100 160″ >> $TMP_FILE
echo “MotionNotify 100 180″ >> $TMP_FILE
echo “MotionNotify 100 200″ >> $TMP_FILE
echo “MotionNotify 100 220″ >> $TMP_FILE
echo “MotionNotify 100 240″ >> $TMP_FILE
echo “MotionNotify 100 260″ >> $TMP_FILE
echo “MotionNotify 100 280″ >> $TMP_FILE
echo “MotionNotify 100 300″ >> $TMP_FILE
echo “MotionNotify 100 320″ >> $TMP_FILE
echo “ButtonRelease 1″ >> $TMP_FILE
echo “KeyStrRelease Control_L” >> $TMP_FILE
echo “KeyStrRelease Alt_L” >> $TMP_FILE
run_macro
}

function right {
# This function will build the macro file for the “right” command
echo > $TMP_FILE
echo “MotionNotify 340 380″ >> $TMP_FILE
echo “KeyStrPress Alt_L” >> $TMP_FILE
echo “KeyStrPress Control_L” >> $TMP_FILE
echo “ButtonPress 1″ >> $TMP_FILE
echo “MotionNotify 340 380″ >> $TMP_FILE
echo “MotionNotify 320 380″ >> $TMP_FILE
echo “MotionNotify 300 380″ >> $TMP_FILE
echo “MotionNotify 280 380″ >> $TMP_FILE
echo “MotionNotify 260 380″ >> $TMP_FILE
echo “MotionNotify 240 380″ >> $TMP_FILE
echo “MotionNotify 220 380″ >> $TMP_FILE
echo “MotionNotify 200 380″ >> $TMP_FILE
echo “MotionNotify 180 380″ >> $TMP_FILE
echo “MotionNotify 160 380″ >> $TMP_FILE
echo “MotionNotify 140 380″ >> $TMP_FILE
echo “MotionNotify 120 380″ >> $TMP_FILE
echo “ButtonRelease 1″ >> $TMP_FILE
echo “KeyStrRelease Control_L” >> $TMP_FILE
echo “KeyStrRelease Alt_L” >> $TMP_FILE
run_macro
}

function down {
# This function will build the macro file for the “down” command
echo > $TMP_FILE
echo “MotionNotify 100 330″ >> $TMP_FILE
echo “KeyStrPress Alt_L” >> $TMP_FILE
echo “KeyStrPress Control_L” >> $TMP_FILE
echo “ButtonPress 1″ >> $TMP_FILE
echo “MotionNotify 100 330″ >> $TMP_FILE
echo “MotionNotify 100 310″ >> $TMP_FILE
echo “MotionNotify 100 290″ >> $TMP_FILE
echo “MotionNotify 100 270″ >> $TMP_FILE
echo “MotionNotify 100 250″ >> $TMP_FILE
echo “MotionNotify 100 230″ >> $TMP_FILE
echo “MotionNotify 100 210″ >> $TMP_FILE
echo “MotionNotify 100 190″ >> $TMP_FILE
echo “MotionNotify 100 170″ >> $TMP_FILE
echo “MotionNotify 100 150″ >> $TMP_FILE
echo “MotionNotify 100 130″ >> $TMP_FILE
echo “MotionNotify 100 110″ >> $TMP_FILE
echo “ButtonRelease 1″ >> $TMP_FILE
echo “KeyStrRelease Control_L” >> $TMP_FILE
echo “KeyStrRelease Alt_L” >> $TMP_FILE
run_macro
}

function fix_vert {
# Fixes a case of vertical extention (non-viewable screen)
sleep $VERT_TIMER
case “$1″ in
1) down
;;
0) up
;;
esac
}

# Verify we have xmacroplay
which xmacroplay
if [ "$?" -ne "0" ]; then
echo “Missing xmacroplay. Install it”
echo “Use apt get install xmacro”
exit 1
fi

# Do we use X and have a defined display? Won’t work otherwise…
if [ -z "$DISPLAY" ]; then
echo “DISPLAY is not defined. Exiting”
exit 1
fi

create_file

# We start where all is viewable

while true; do
# Select direction
DIRECTION=$RANDOM
let “DIRECTION %= $POSS”
# Debug:        echo “*** $DIRECTION ***”
case “$DIRECTION” in
0) up
fix_vert 1
;;
1) down
fix_vert 0
;;
[2-5]) left
;;
[6-9]) right
;;
10)     left
left
;;
11)     right
right
;;
esac
sleep $TIMER
done

exit 0

When one wants to achieve fast provisioning of virtual machines, some solutions might come into account. The one I prefer uses Linux LVM snapshot capabilities to duplicate one working machine into few.

This can happen, of course, only if the host running VMware-Server is Linux.

LVM snapshots have one vast disadvantage – performance. When a block on the source of the snapshot is being changed for the first time, the original block is being replicated to each and every snapshot COOW space. It means that a creation of a 1GB file on a volume having ten snapshots means a total copy of 10GB of data across your disks. You cannot ignore this performance impact.

LVM2 has support for read/write snapshots. I have come up with a nice way of utilizing this capability to my benefit. An R/W snapshot which is being changed does not replicate its changes to any other snapshot. All changes are considered local to this snapshot, and are being maintained only in its COOW space. So adding a 1GB file to a snapshot has zero impact on the rest of the snapshots or volumes.

The idea is quite simple, and it works like this:

1. Create adequate logical volume with a given size (I used 9GB for my own purposes). The name of the LV in my case will be /dev/VGVM3/centos-base

2. Mount this LV on a directory, and create a VM inside it. In my case, it’s in /vmware/centos-base

3. Install the VM as the baseline for all your future VMs. If you might not want Apache on some of them, don’t install it on the baseline.

4. Install vmware-tools on the baseline.

5. Disable the service “kudzu”

6. Update as required

7. In my case I always use DHCP. You can set it to obtain its IP once from a given location, or whatever you feel like.

8. Shut down the VM.

9. In the VM’s .vmx file add a line like this:

uuid.action = “create”

I have added below (expand to read) two scripts which will create the snapshot, mount it and register it, including new MAC and UUID.

Press below for the scripts I have used to create and destroy VMs

create-replica.sh:

#!/bin/sh
# This script will replicate vms from a given (predefined) source to a new system
# Written by Ez-Aton, http://www.tournament.org.il/run
# Arguments: name

# FUNCITONS BE HERE
test_can_do () {
# To be able to snapshot, we need a set of things to happen
if [ -d $DIR/$TARGET ] ; then
echo “Directory already exists. You don’t want to do it…”
exit 1
fi
if [ -f $VG/$TARGET ] ; then
echo “Target snapshot exists”
exit 1
fi
if [ `vmrun list | grep -c $DIR/$SRC/$SRC.vmx` -gt "0" ] ; then
echo “Source VM is still running. Shut it down before proceeding”
exit 1
fi
if [ `vmware-cmd -l | grep -c $DIR/$TARGET/$SRC.vmx` -ne "0" ] ; then
echo “VM already registered. Unregister first”
exit 1
fi
}

do_snapshot () {
# Take the snapshot
lvcreate -s -n $TARGET -L $SNAPSIZE $VG/$SRC
RET=$?
if [ "$RET" -ne "0" ]; then
echo “Failed to create snapshot”
exit 1
fi
}

mount_snapshot () {
# This function creates the required directories and mounts the snapshot there
mkdir $DIR/$TARGET
mount $VG/$TARGET $DIR/$TARGET
RET=$?
if [ "$RET" -ne "0" ]; then
echo “Failed to mount snapshot”
exit 1
fi
}

alter_snap_vmx () {
# This function will alter the name in the VMX and make it the $TARGET name
cat $DIR/$TARGET/$SRC.vmx | grep -v “displayName” > $DIR/$TARGET/$TARGET.vmx
echo “displayName = \”$TARGET\”" >> $DIR/$TARGET/$TARGET.vmx
cat $DIR/$TARGET/$TARGET.vmx > $DIR/$TARGET/$SRC.vmx
\rm $DIR/$TARGET/$TARGET.vmx
}

register_vm () {
# This function will register the VM to VMWARE
vmware-cmd -s register $DIR/$TARGET/$SRC.vmx
}

# MAIN
if [ -z "$1" ]; then
echo “Arguments: The target name”
exit 1
fi

# Parameters:
SRC=centos-base         #The name of the source image, and the source dir
PREFIX=centos             #All targets will be created in the name centos-$NAME
DIR=/vmware               #My VMware VMs default dir
SNAPSIZE=6G              #My COOW space
VG=/dev/VGVM3           #The name of the VG
TARGET=”$PREFIX-$1″

test_can_do
do_snapshot
mount_snapshot
alter_snap_vmx
register_vm
exit 0

remove-replica.sh:

#!/bin/sh
# This script will remove a snapshot machine
# Written by Ez-Aton, http://www.tournament.org.il/run
# Arguments: machine name

#FUNCTIONS
does_it_exist () {
# Check if the described VM exists
if [ `vmware-cmd -l | grep -c $DIR/$TARGET/$SRC.vmx` -eq "0" ]; then
echo “No such VM”
exit 1
fi
if [ ! -e $VG/$TARGET ]; then
echo “There is no matching snapshot volume”
exit 1
fi
if [ `lvs $VG/$TARGET | awk '{print $5}' | grep -c $SRC` -eq "0" ]; then
echo “This is not a snapshot, or a snapshot of the wrong LV”
exit 1
fi
}

ask_a_thousand_times () {
# This function verifies that the right thing is actually done
echo “You are about to remove a virtual machine and an LVM. Details:”
echo “Machine name: $TARGET”
echo “Logical Volume: $VG/$TARGET”
echo -n “Are you sure? (y/N): ”
read RES
if [ "$RES" != "Y" ]&&[ "$RES" != "y" ]; then
echo “Decided not to do it”
exit 0
fi
echo “”
echo “You have asked to remove this machine”
echo -n “Again: Are you sure? (y/N): ”
read RES
if [ "$RES" != "Y" ]&&[ "$RES" != "y" ]; then
echo “Decided not to do it”
exit 0
fi
echo “Removing VM and snapshot”
}

shut_down_vm () {
# Shut down the VM and unregister it
vmware-cmd $DIR/$TARGET/$SRC.vmx stop hard
vmware-cmd -s unregister $DIR/$TARGET/$SRC.vmx
}

remove_snapshot () {
# Umount and remove the snapshot
umount $DIR/$TARGET
RET=$?
if [ "$RET" -ne "0" ]; then
echo “Cannot umount $DIR/$TARGET”
exit 1
fi
lvremove -f $VG/$TARGET
RET=$?
if [ "$RET" -ne "0" ]; then
echo “Cannot remove snapshot LV”
exit 1
fi
}

remove_dir () {
# Removes the mount point
rmdir $DIR/$TARGET
}

#MAIN
if [ -z "$1" ]; then
echo “No machine name. Exiting”
exit 1
fi

#PARAMETERS:
DIR=/vmware                #VMware default VMs location
VG=/dev/VGVM3            #The name of the VG
PREFIX=centos              #Prefix to the name. All these VMs will be called centos-$NAME
TARGET=”$PREFIX-$1″
SRC=centos-base           #The name of the baseline image, LVM, etc. All are the same

does_it_exist
ask_a_thousand_times
shut_down_vm
remove_snapshot
remove_dir

exit 0

Pros:

1. Very fast provisioning. It takes almost five seconds, and that’s because my server is somewhat loaded.

2. Dependable: KISS at its marvel.

3. Conservative on space

4. Conservative on I/O load (unlike the traditional use of LVM snapshot, as explained in the beginning of this section).

Cons:

1. Cannot streamline the contents of snapshot into the main image (LVM team will implement it in the future, I think)

2. Cannot take a snapshot of a snapshot (same as above)

3. If the COOW space of any of the snapshots is full (viewable through the command ‘lvs‘) then on boot, the source LV might not become active (confirmed RH4 bug, and this is the system I have used)

4. My script does not edit/alter /etc/fstab (I have decided it to be rather risky, and it was not worth the effort at this time)

5. My script does not check if there is enough available space in the VG. Not required, as it will fail if creation of LV will fail

You are most welcome to contribute any further changes done to this script. Please maintain my URL in the script if you decide to use it.

Thanks!

Many of us use tar (many times with gzip or bzip2) for archiving purposes. When performing such an action, a large file, usually, too large, remains. To extract from it, or to split it becomes an effort.

This post will show an example of a small script to split an archive and later on, to directly extract the data out of the slices.

Let’s assume we have a directory called ./Data . To archive it using tar+gzip, we can perform the following action:

tar czf /tmp/Data.tar.gz Data

For verbose display (although it’s could slow down things a bit), add the flag ‘v’.

Now we have a file called /tmp/Data.tar.gz

Lets split it to slices sized 10 MB each:

cd /tmp
mkdir slices
i=1 # Our counter
skip=0 # This is the offset. Will be used later
chunk=10 # Slice size in MB
let size=$chunk * 1024 # And in kbytes
file=Data.tar.gz # Name of the tar.gz file we slice
while true ; do
# Deal with numbers lower than 10
if [ $i -lt "10" ]; then
j=0${i}
else
j=${i}
fi
dd if=${fie} of=slices/${file}.slice${j} bs=1M count=${chunk} skip=${skip}
# Just to view the files with out own eyes
ls -s slices/${file}.slice${j}
if [ `ls -s slices/${file}.slice${j} | awk '{print $1}'` -lt "${size}" ]; then
echo “Done”
break
fi
let i=$i+1
let skip=$skip+$chunk
done

This will break the tar.gz file to a files with running numbers added to their names. It assumes that the number of slices would not exceed 99. You can extend the script to deal with three digits numbers. The sequence is important for later. Stay tuned

Ok, so we have a list of files with a numerical suffix, which, combined, include our data. We can test their combined integrity:

cd /tmp/slices
i=1
file=Data.tar.gz
for i in `ls`; do
cat ${file}.slice${i} >> ../Data1.tar.gz
done

This will allow us to compare /tmp/Data.tar.gz and /tmp/Data1.tar.gz. I tend to use md5sum for such tasks:

md5sum Data.tar.gz
d74ba284a454301d85149ec353e45bb7 Data.tar.gz
md5sum Data1.tar.gz
d74ba284a454301d85149ec353e45bb7 Data1.tar.gz

They are similar. Great. We can remove Data1.tar.gz. We don’t need it anymore.

To recover the contents of the slices, without actually wasting space by combining them before extracting their contents (which requires time, and disk space), we can run a script such as this:

cd /tmp/slices
file=Data.tar.gz
(for i in `ls ${file}.slice*`; do
cat $i
done ) | tar xzvf -

This will extract the contents of the joined archive to the current directory.

This is all for today. Happy moving of data