Running Systems

XenServer is a wonderful tool. One of the better parts of it is its powerful scripting language, powered by the ‘xe’ command.

In order to capture a mass of snapshots, you can either do it manually from the GUI, or scripted. The script supplied below will include shell functions to capture Quiesce snapshots, and it that fails, normal snapshots of every running VM on the system.

Reason: NetApp SnapMirror, or other backup (maybe for later export) scheduled actions.

#!/bin/bash
# This script will supply functions for snapshotting and snapshot destroy including disks
# Written by Ez-Aton
# Visit my web blog for more stuff, at http://run.tournament.org.il
 
# Global variables:
UUID_LIST_FILE=/tmp/SNAP_UUIDS.txt
 
# Function
function assign_all_uuids () {
	# Construct artificial non-indexed list with name (removing annoying characters) and UUID
	LIST=""
	for UUID in `xe vm-list power-state=running is-control-domain=false | grep uuid | awk '{print $NF}'`
	do
		NAME=`xe vm-param-get param-name=name-label uuid=$UUID | tr ' ' _ | tr -d '(' | tr -d ')'`
		LIST="$LIST $NAME:$UUID"
	done
	echo $LIST
}
 
function take_snap_quiesce () {
	# We attempt to take a snapshot with quench
	# Arguments: $1 name ; $2 uuid
	# We attempt to snapshot the machine and set the value of snap_uuid to the snapshot uuid, if successful.
	# Return 1 if failed
 
	if SNAP_UUID=`xe vm-snapshot-with-quiesce vm=$2 new-name-label=${1}_snapshot`
	then
		# echo "Snapshot-with-quiesce for $1 successful"
		return 0
	else
		echo "Snapshot-with-quiesce for $1 failed"
		return 1
	fi
}
 
function take_snap () {
	# We attempt to take a snapshot
	# Arguments: $1 name ; $2 uuid
	# We attempt to snapshot the machine and set the value of snap_uuid to the snapshot uuid, if successful.
	# Return 1 if failed
 
	if SNAP_UUID=`xe vm-snapshot vm=$2 new-name-label=${1}_snapshot`
	then
		#echo "Snapshot for $1 successful"
		echo $SNAP_UUID
		return 0
	else
		echo "Snapshot-with-quiesce for $1 failed"
		return 1
	fi
}
 
function stop_ha_template () {
	# Templates inherit their settings from the origin
	# We need to turn off HA
	# $1 : Template UUID
	if [ -z "$1" ]
	then
		echo "Missing template UUID"
		return 1
	fi
	xe template-param-set ha-always-run=false uuid=$1
}
 
function get_vdi () {
	# This function will get a space delimited list of VDI UUIDs of a given snapshot/template UUID
	# Arguments: $1 template UUID
	# It will also verify that each VBD is an actual snapshot
	if [ -z "$1" ]
	then
		echo "No arguments? We need the template UUID"
		return 1
	fi
	VDIS=""
	for VBD in `xe vbd-list vm-uuid=$1 | grep ^uuid | awk '{print $NF}'`
	do
		echo "VBD: $VBD"
		if [ ! `xe vbd-param-get param-name=type uuid=$VBD` = "CD" ]
		then
			CUR_VDI=`xe vdi-list vbd-uuids=$VBD | grep ^uuid | awk '{print $NF}'`
			if `xe vdi-param-get uuid=$CUR_VDI param-name=is-a-snapshot`
			then
				VDIS="$VDIS $CUR_VDI"
			else
				echo "VDI is not a snapshot!"
				return 1
			fi
			CUR_VDI=""
		fi
	done
	echo $VDIS
}
 
function remove_vdi () {
	# This function will get a list of VDIs and remove them
	# Carefull!
	for VDI in $@
	do
		if xe vdi-destroy uuid=$VDI
		then
			echo "Success in removing VDI $VDI"
		else
			echo "Failure in removing VDI $VDI"
			return 1
		fi
	done
}
 
function remove_template () {
	# This funciton will remove a template
	# $1 template UUID
	if [ -z "$1" ]
	then
		echo "Required UUID"
		return 1
	fi
	xe template-param-set is-a-template=false uuid=$1
	if ! xe vm-uninstall force=true uuid=$1
	then
		echo "Failure to remove VM/Template"
		return 1
	fi
}
 
function remove_all_template () {
	# This function will completely remove a template
	# The steps are as follow:
	# $1 is the UUID of the template
	# Calculate its VDIs
	# Remove the template
	# Remove the VDIs
	if [ -z "$1" ]
	then
		echo "No Template UUID was supplied"
		return 1
	fi
	# We now collect the value of $VDIS
	get_vdi $1
	if [ "$?" -ne "0" ]
	then
		echo "Failed to get VDIs for Template $1"
		return 1
	fi
	if ! remove_template $1
	then
		echo "Failure to remove template $1"
		return 1
	fi
	if ! remove_vdi $VDIS
	then
		return 1
	fi
}
 
function create_all_snapshots () {
	# In this function we will run all over $LIST and create snapshots of each machine, keeping the UUID of it inside a file
	# $@ - list of machines in the $LIST format
	if [ -f $UUID_LIST_FILE ]
	then
		mv $UUID_LIST_FILE $UUID_LIST_FILE.$$
	fi
	for i in $@
	do
		SNAP_UUID=`take_snap_quiesce ${i%%:*} ${i##*:}`
		if [ "$?" -ne "0" ]
		then
			echo "Problem taking snapshot with quiesce for ${i%%:*}"
			echo "Attempting normal snapshot"
			SNAP_UUID=`take_snap ${i%%:*} ${i##*:}`
			if [ "$?" -ne "0" ]
                	then
                        	echo "Problem taking snapshot for ${i%%:*}"
				SNAP_UUID=""
			fi
		fi
		stop_ha_template $SNAP_UUID
		echo $SNAP_UUID >> $UUID_LIST_FILE
	done
}

Possible use will be like this:

. /usr/local/bin/xen_functions.sh

create_all_snapshots `assign_all_uuids` &> /tmp/snap_create.log

Most documentation on the net is about how to run a cluster-in-a-box under Vmware. Very few seem to care about protecting Vmware guests under real RedHat cluster with a shared storage.

This article is just about it. While I would not recommend using Vmware in such a setup, it has been the case, and that Vmware guest actually resides on the shared storage. To relocate it is out of the question, so migrating it together with other resources is the only valid option.

To do so, I have created a simple script which will accept start/stop/status arguments. The Vmware guest VMX is hard-coded into the script, but in an easy-to-change format. This script will attempt to freeze the Vmware guest, and only if it fails, to shut it down. Mind you that the blog’s HTML formatting might alter quotation marks into UTF-8 marks which will not be understood well by shell.

#!/bin/bash
# This script will start/stop/status VMware machine
# Written by Ez-Aton
# http://www.tournament.org.il/run
 
# Hardcoded. Change to match your own settings!
VMWARE="/export/vmware/hosts/Windows_XP_Professional/Windows XP Professional.vmx"
VMRUN="/usr/bin/vmrun"
TIMEOUT=60
 
function status () {
  # This function will return success if the VM is up
  $VMRUN list | grep "$VMWARE" &>/dev/null
  if [[ "$?" -eq "0" ]]
  then
    echo "VM is up"
    return 0
  else
    echo "VM is down"
    return 1
  fi
}
 
function start () {
  # This function will start the VM
  $VMRUN start "$VMWARE"
  if [[ "$?" -eq "0" ]]
  then
    echo "VM is starting"
    return 0
  else
    echo "VM failed"
    return 1
  fi
}
 
function stop () {
  # This function will stop the VM
  $VMRUN suspend "$VMWARE"
  for i in `seq 1 $TIMEOUT`
  do
    if status
    then
      echo
    else
      echo "VM Stopped"
      return 0
    fi
    sleep 1
  done
  $VMRUN stop "$VMWARE" soft
}
 
case "$1" in
start)     start
        ;;
stop)      stop
        ;;
status)   status
        ;;
esac
RET=$?
 
exit $RET

Since the formatting is killed by the blog, you can find the script here: vmware1

I intend on building a “real” RedHat Cluster agent script, but this should do for the time being.

Enjoy!

Many of us use tar (many times with gzip or bzip2) for archiving purposes. When performing such an action, a large file, usually, too large, remains. To extract from it, or to split it becomes an effort.

This post will show an example of a small script to split an archive and later on, to directly extract the data out of the slices.

Let’s assume we have a directory called ./Data . To archive it using tar+gzip, we can perform the following action:

tar czf /tmp/Data.tar.gz Data

For verbose display (although it’s could slow down things a bit), add the flag ‘v’.

Now we have a file called /tmp/Data.tar.gz

Lets split it to slices sized 10 MB each:

cd /tmp
mkdir slices
i=1 # Our counter
skip=0 # This is the offset. Will be used later
chunk=10 # Slice size in MB
let size=$chunk * 1024 # And in kbytes
file=Data.tar.gz # Name of the tar.gz file we slice
while true ; do
# Deal with numbers lower than 10
if [ $i -lt "10" ]; then
j=0${i}
else
j=${i}
fi
dd if=${fie} of=slices/${file}.slice${j} bs=1M count=${chunk} skip=${skip}
# Just to view the files with out own eyes
ls -s slices/${file}.slice${j}
if [ `ls -s slices/${file}.slice${j} | awk '{print $1}'` -lt "${size}" ]; then
echo “Done”
break
fi
let i=$i+1
let skip=$skip+$chunk
done

This will break the tar.gz file to a files with running numbers added to their names. It assumes that the number of slices would not exceed 99. You can extend the script to deal with three digits numbers. The sequence is important for later. Stay tuned

Ok, so we have a list of files with a numerical suffix, which, combined, include our data. We can test their combined integrity:

cd /tmp/slices
i=1
file=Data.tar.gz
for i in `ls`; do
cat ${file}.slice${i} >> ../Data1.tar.gz
done

This will allow us to compare /tmp/Data.tar.gz and /tmp/Data1.tar.gz. I tend to use md5sum for such tasks:

md5sum Data.tar.gz
d74ba284a454301d85149ec353e45bb7 Data.tar.gz
md5sum Data1.tar.gz
d74ba284a454301d85149ec353e45bb7 Data1.tar.gz

They are similar. Great. We can remove Data1.tar.gz. We don’t need it anymore.

To recover the contents of the slices, without actually wasting space by combining them before extracting their contents (which requires time, and disk space), we can run a script such as this:

cd /tmp/slices
file=Data.tar.gz
(for i in `ls ${file}.slice*`; do
cat $i
done ) | tar xzvf -

This will extract the contents of the joined archive to the current directory.

This is all for today. Happy moving of data

Kickstart is a great method of hands-free installation of RHEL/Centos (and other derived systems). Its power is in its easy interface and rather powerful %post scripting directives. Its weakness is in its lack of flexibility where it comes to package selection and various custom actions.

On some cases, companies use web interface (usually home-made) which builds kickstart config files on-demand. On some cases, the administrator is required to build several kickstart config files for pre-anticipated setups.

I was looking for something which will give me the power to maintain a fixed configuration on one hand, and will allow me some tweaks and variants, when I want them. I could have used the %post scripting sections, but this gets quite complicated, especially when you want to add only one package (but with its dependencies), or you want to force full update of the system before it goes online, or even select its hostname, assuming it is not yet defined in the DNS.

I base my system on a simple DHCP/BootP + tftp server which answers to all bootp requests and offers a simple menu (just type a number and press on Enter). The original schema was quite simple: type 4 for Centos4.3, and then add -min if you wanted it to use a kickstart file with a minimum configuration. Then I wanted to add the option to update the system in an early stage, so I have added -update, which would have looked in the menu like “4-min-update” option. Quite readable, however, it generated lots of work when maintaining the pxelinux.cfg/default file and the ks themselves. Too many variations tend to require lots of care.

Adding parameters to the boot menu is possible, and would result in them existing in /proc/cmdline for later parsing.

I have decided to parse a set of predefined parameters supplied during boot time, and to change the kickstart config file according to them. It actually works quite well. This is a less-sophisticated and more of a stand-alone system compared to this system. Also, it doesn’t require me to alter the system’s boot process.

This is my ks.cfg file, which includes the flexibility additions:

# Kickstart file generated by Ez-Aton

install
nfs –server=install-server –dir=/mnt/samba/Centos
lang en_US.UTF-8
langsupport –default=en_US.UTF-8 en_US.UTF-8
keyboard us
skipx
network –device eth0 –bootproto dhcp
rootpw –iscrypted RpUKzjDc9k2gU
firewall –disabled
selinux –disabled
authconfig –enableshadow –enablemd5
timezone Asia/Jerusalem
bootloader –location=mbr

%packages
e2fsprogs
grub
lvm2
kernel
net-snmp
net-snmp-utils
kernel-devel
kernel-smp-devel
gcc

%pre
# By Ez-Aton http://www.tournament.org.il/run
for i in `cat /proc/cmdline`; do
echo $i >> /tmp/vars.tmp
done
grep “=” /tmp/vars.tmp > /tmp/vars
KS=/tmp/ks.cfg
update=”"
name=”"
pkg=”"
. /tmp/vars
if [ ! -z "$update" ]; then
echo “yum update -y” >> $KS
fi
if [ ! -z "$name" ]; then
value=”dhcp –hostname $name”
cat $KS | sed s/dhcp/”$value”/ > $KS.tmp
cat $KS.tmp > $KS
fi
if [ ! -z "$pkg" ]; then
pkg_line=`grep -n ^%packages $KS | cut -f 1 -d \:`
max_line=`wc -l $KS | awk ‘{print $1}’`
head -n $pkg_line $KS > $KS.tmp
for i in `echo $pkg | sed s/,/\ /g`; do
echo $i >> $KS.tmp
done
let tail_line=$max_line-$pkg_line
tail -n $tail_line $KS >> $KS.tmp
cat $KS.tmp > $KS
fi

%post

So, as you can see, I take the following parameters:

update=yes (it can be update=anything)

name=hostname (in case it cannot be retrieved from the DHCP server)

pkg=pkg1,pkg2,{pkg3,…} (To add specific packages to the installation)

It was tested to work on Centos4.3 system, and will probably work on RHEL and Centos versions 4.x all along. I didn’t test it on RHEL5/Centos5 yet.

If you use the script, please leave my name and blog URL in it. Also, if you modify it for your needs, I would be glad to get back the modifications you have made, to include them.

Enjoy.

For some reason (probably a typo) I’ve missed an important character in an example I gave here, but I have just recently fixed it. Anyhow, to clarify this, here is the extended description of the correction.

The $IFS Bash system variable defines what is the default separator between strings. Changing it can help when dealing with, for example, file names with spaces in them, variables which should be considered one unit, but are separated by semicolon, etc.

To change the default string separator from "space or tab or new-line" to new-line only. you need to set, in Bash the following parameter:

IFS=$’\n’

First and unrelated – this is my birthday. It reminds me that another year passed, and generally speaking, I do not take this too well…

Due to massive SPAM attacks, my commenting system is turned off for a while now, and I need to see how I can re-enable it safely.

Bash – here we go.

When you want a single script to spawn several commands in parallel, the best way is to use the ampersand at the end of each command, example:

/usr/bin/find / -name 123 &

/bin/grep -r abc / &

etc.

If you do not want the output from these commands to mix together, you would probably wish to redirect it to a file, for example (redirecting all outputs):

/usr/bin/find / -name 123 &>/tmp/find.out &

/bin/grep -r abc / &>/tmp/grep.out &

You can later “cat” the two files in your own desired order.

This adds two interesting issues – the first is about how you can tell that both commands finished. There are several methods, such as collecting their PIDs, and looping with “sleep” until they are no longer there. Alternate, and more elegant method is by using “wait“. This command will wait for both commands (in our example. As many commands as you have forked to the background) to finish, and only then continue. So we can add, in our example, the following lines:

wait

cat /tmp/find.out

cat /tmp/grep.out

This will insure that both outputs are not mixed together, and are readable.

The second issue caused by the output redirection we’ve added earlier is the handling of killing these commands. Let’s assume that our script is time-limited, and if it exceeds its given time limits, it gets killed. In this case, this script will be killed, however, its children will not die, and will become owned by init, PID 1. This will keep these commands running. Try to assume, for that matter, that every 10 minutes we run the main script, and that it is limited to these ten minutes. We might kill the system’s I/O performance since we might reach a case where several “find” commands are running in parallel – each invoked by our main script at a different time.

To handle such case, we can use the command “trap“. It allows us to handle signals in a method we desire. notice that if you capture SIGTERM (kill -15 – the default kill) and misuse it, the only method of stopping the main script will be by invoking SIGKILL (kill -9) on it, which bypasses all trap directives.

In our example, let’s add this (assume we are aware of each PID)

trap “kill $PID1 $PID2 ; exit 0″ SIGTERM

So we can sum up our example script to be like this:

/usr/bin/find / -name 123 &>/tmp/find.out &

PID1=$!

/bin/grep -r abc / &>/tmp/grep.out &

PID2=$!

trap “kill $PID1 $PID2 ; exit 0″ SIGTERM

wait

cat /tmp/find.out

cat /tmp/grep.out

This wraps it up. Hope it helps.

This was a tricky action. Assume I have a list of variables, obtained by an external source:

var1=a

var2=b

var3=c

I cannot use loop and in it the phrase ${var$i} (where i is the integer counter). It just doesn’t work. I used this instead to assign the values to an array:

var[$i]=$(eval echo "\${var${i}}")

That way, I was able to loop through these values later easily.

So… we can use assigned var names inside a var if we do it right: $(eval echo "\${var${i}}")

Let’s assume we have a file containing lines such as this:

first last

one two

three four five

If we write a simple script to deal with each line in a turn, we would write something like this:

for i in `cat file`; do echo $i; done

This would echo, however, each word at its turn. If we want the whole line in this echo, we need to set BASH special variable: IFS.

Example:

export IFS=$’\n’ ; for i in `cat file`; do echo $i; done

This would do the trick. We define the Internal Field Separator to be newline only, and not newline, spaces and tabs, as the default goes.

The credit for this piece of information I can easily give to this site. Thanks, guys.

Here’s a little script I’ve written which deletes older than AGE days files, and has an exclude list, just in case. It’s meant to be run by cron on a daily basis:

#!/bin/sh

# Source of all evil
DIR=/ftp
# Age of file in days
AGE=10
# Exclude list – Use pipe (|) seperated values. Example:
# EXCLUDE=”me|tal” for excluding both “me” and both “tal”. Use the longest
# possible expression, for accurate match. For example:
# EXCLUDE=”/ftp/me|/ftp/tal”. Below is the default minimal exclude list.
EXCLUDE=”lost\+found|incoming”

echo -n \” > /tmp/del-list.txt

find $DIR/*/* -mtime +$AGE -print | grep -vE “$EXCLUDE” | tr ‘\n’ “\”\n\”" >> /tmp/del-list.txt

for i in `cat /tmp/del-list.txt` ; do
echo $i >> /var/log/del-ftp.log
done

cat /tmp/del-list.txt | xargs \rm -Rf

\rm /tmp/del-list.txt

It seems to work. So far, I have delete 2nd level directories when old enough (10 days by default), and I can handle files with spaces in their names (scheduled delete of filenames with spaces – for the sake of those searching for a solution. At least, I’ve used this expression and didn’t find a solution online).