Archive for the ‘Scripting/Programing’ Category

XenServer create snapshots for all machines

Friday, August 7th, 2009

XenServer is a wonderful tool. One of the better parts of it is its powerful scripting language, powered by the ‘xe’ command.

In order to capture a mass of snapshots, you can either do it manually from the GUI, or scripted. The script supplied below will include shell functions to capture Quiesce snapshots, and it that fails, normal snapshots of every running VM on the system.

Reason: NetApp SnapMirror, or other backup (maybe for later export) scheduled actions.

#!/bin/bash
# This script will supply functions for snapshotting and snapshot destroy including disks
# Written by Ez-Aton
# Visit my web blog for more stuff, at http://run.tournament.org.il
 
# Global variables:
UUID_LIST_FILE=/tmp/SNAP_UUIDS.txt
 
# Function
function assign_all_uuids () {
	# Construct artificial non-indexed list with name (removing annoying characters) and UUID
	LIST=""
	for UUID in `xe vm-list power-state=running is-control-domain=false | grep uuid | awk '{print $NF}'`
	do
		NAME=`xe vm-param-get param-name=name-label uuid=$UUID | tr ' ' _ | tr -d '(' | tr -d ')'`
		LIST="$LIST $NAME:$UUID"
	done
	echo $LIST
}
 
function take_snap_quiesce () {
	# We attempt to take a snapshot with quench
	# Arguments: $1 name ; $2 uuid
	# We attempt to snapshot the machine and set the value of snap_uuid to the snapshot uuid, if successful.
	# Return 1 if failed
 
	if SNAP_UUID=`xe vm-snapshot-with-quiesce vm=$2 new-name-label=${1}_snapshot`
	then
		# echo "Snapshot-with-quiesce for $1 successful"
		return 0
	else
		echo "Snapshot-with-quiesce for $1 failed"
		return 1
	fi
}
 
function take_snap () {
	# We attempt to take a snapshot
	# Arguments: $1 name ; $2 uuid
	# We attempt to snapshot the machine and set the value of snap_uuid to the snapshot uuid, if successful.
	# Return 1 if failed
 
	if SNAP_UUID=`xe vm-snapshot vm=$2 new-name-label=${1}_snapshot`
	then
		#echo "Snapshot for $1 successful"
		echo $SNAP_UUID
		return 0
	else
		echo "Snapshot-with-quiesce for $1 failed"
		return 1
	fi
}
 
function stop_ha_template () {
	# Templates inherit their settings from the origin
	# We need to turn off HA
	# $1 : Template UUID
	if [ -z "$1" ]
	then
		echo "Missing template UUID"
		return 1
	fi
	xe template-param-set ha-always-run=false uuid=$1
}
 
function get_vdi () {
	# This function will get a space delimited list of VDI UUIDs of a given snapshot/template UUID
	# Arguments: $1 template UUID
	# It will also verify that each VBD is an actual snapshot
	if [ -z "$1" ]
	then
		echo "No arguments? We need the template UUID"
		return 1
	fi
	VDIS=""
	for VBD in `xe vbd-list vm-uuid=$1 | grep ^uuid | awk '{print $NF}'`
	do
		echo "VBD: $VBD"
		if [ ! `xe vbd-param-get param-name=type uuid=$VBD` = "CD" ]
		then
			CUR_VDI=`xe vdi-list vbd-uuids=$VBD | grep ^uuid | awk '{print $NF}'`
			if `xe vdi-param-get uuid=$CUR_VDI param-name=is-a-snapshot`
			then
				VDIS="$VDIS $CUR_VDI"
			else
				echo "VDI is not a snapshot!"
				return 1
			fi
			CUR_VDI=""
		fi
	done
	echo $VDIS
}
 
function remove_vdi () {
	# This function will get a list of VDIs and remove them
	# Carefull!
	for VDI in $@
	do
		if xe vdi-destroy uuid=$VDI
		then
			echo "Success in removing VDI $VDI"
		else
			echo "Failure in removing VDI $VDI"
			return 1
		fi
	done
}
 
function remove_template () {
	# This funciton will remove a template
	# $1 template UUID
	if [ -z "$1" ]
	then
		echo "Required UUID"
		return 1
	fi
	xe template-param-set is-a-template=false uuid=$1
	if ! xe vm-uninstall force=true uuid=$1
	then
		echo "Failure to remove VM/Template"
		return 1
	fi
}
 
function remove_all_template () {
	# This function will completely remove a template
	# The steps are as follow:
	# $1 is the UUID of the template
	# Calculate its VDIs
	# Remove the template
	# Remove the VDIs
	if [ -z "$1" ]
	then
		echo "No Template UUID was supplied"
		return 1
	fi
	# We now collect the value of $VDIS
	get_vdi $1
	if [ "$?" -ne "0" ]
	then
		echo "Failed to get VDIs for Template $1"
		return 1
	fi
	if ! remove_template $1
	then
		echo "Failure to remove template $1"
		return 1
	fi
	if ! remove_vdi $VDIS
	then
		return 1
	fi
}
 
function create_all_snapshots () {
	# In this function we will run all over $LIST and create snapshots of each machine, keeping the UUID of it inside a file
	# $@ - list of machines in the $LIST format
	if [ -f $UUID_LIST_FILE ]
	then
		mv $UUID_LIST_FILE $UUID_LIST_FILE.$$
	fi
	for i in $@
	do
		SNAP_UUID=`take_snap_quiesce ${i%%:*} ${i##*:}`
		if [ "$?" -ne "0" ]
		then
			echo "Problem taking snapshot with quiesce for ${i%%:*}"
			echo "Attempting normal snapshot"
			SNAP_UUID=`take_snap ${i%%:*} ${i##*:}`
			if [ "$?" -ne "0" ]
                	then
                        	echo "Problem taking snapshot for ${i%%:*}"
				SNAP_UUID=""
			fi
		fi
		stop_ha_template $SNAP_UUID
		echo $SNAP_UUID >> $UUID_LIST_FILE
	done
}

Possible use will be like this:

. /usr/local/bin/xen_functions.sh

create_all_snapshots `assign_all_uuids` &> /tmp/snap_create.log

Ad-hoc remote backups to tape

Sunday, July 19th, 2009

I have a nice SCSI tape connected to a single server. This allows for on-demand backups, with the hope (and seldom, with the established knowledge) that I can recover the data I have there.

Old computers, decommissioned computers and systems I wish to erase and reuse are seldom backed-up, just because of the effort in doing it. I will need to manually run something or the other, and who wants this chore?

I know that there are many full-featured backup systems out there, OSS and all, with the capability of doing what I want to do, however, these commonly use backup agents, tape formats and what’s more, just to make a simple one-time backup (which is what I want) – it looked too bloated for my needs.

Again – my needs are: take this machine, run a simple script which can be obtained from an NFS share, wait for X minutes doing something else, and be assured your system is backed up.

I have written the script below to satisfy these requirements. Hope it helps others. Notice the single SSH leading connection and its functionality. It leaves a raw text file on tape with a simple description of the backup process, and the next tracks are the contents of each mount point.

I was a bit spartan with comments, but in general, this script should be quite self-explanatory:

#!/bin/bash
# This script will backup local disk to remote tape
# Written by Ez-Aton - http://run.tournament.org.il/
 
SERVER=kruvi # The name of the server with the direct attached tape
SRV_USER=root
TAPE=/dev/nst0 # Non-rewinding tape. We need to be able to add more tracks and not overwrite our own track
SSH="ssh -o StrictHostKeyChecking=no -o ControlMaster=auto -o ControlPath=~/.ssh/socket-%r@%h:%p"
WORK_FILE=/tmp/work.$$
TAR_LOG=/tmp/backup.log
TAR_ARG="czf - --one-file-system"
 
MOUNTS=`df -TlP | grep -v tmpfs | tail -n +2 | awk '{print $7}'`
# Assume nobody is stupid enough to use white spaces in mount paths
NUM_MOUNTS=`echo $MOUNTS | wc -w`
SUM_FILE=/tmp/summery.txt
 
clean_log () {
        : > $TAR_LOG
}
 
first_disk () {
        # Assume first disk is the first entry in /proc/partitions
        DISK="/dev/`cat /proc/partitions | head -n 3 | tail -n 1 | awk '{print $4}'`"
}
 
create_sum () {
        echo "Creating summery"
        # Collect information and place it in the file. It will be the first track of the tape
        echo "Hostname: `hostname`" > $SUM_FILE
        echo >> $SUM_FILE
        date >> $SUM_FILE
        echo >> $SUM_FILE
        for i in $MOUNTS; do df -h $i | tail -n +2 >> $SUM_FILE ; done
        echo >> $SUM_FILE
        echo "There will be $(($NUM_MOUNTS + 1)) tracks in addition to the first one" >> $SUM_FILE
}
 
create_leading_ssh () {
        # Use a nice trick for giving password only once:
        $SSH -f $SRV_USER@$SERVER 'while true; do sleep 100; done'
        echo "post leading"
}
 
monitor_proc () {
        # Monitor SSH process
        # Run in the background
        touch $WORK_FILE
        PID=`ps aux | grep "$SSH" | grep -v grep | awk '{print $2}'`
        if [ -z "$PID" ]
        then
                echo "Done so soon?"
                return 1
        fi
        while [ -f $WORK_FILE ]
        do
                sleep 10
        done
        kill $PID
}
 
test_tape_cmd () {
        CMD="mt -f $TAPE status"
}
 
remote_tape_append () {
        CMD="cat > $TAPE"
}
 
test_tape () {
        test_tape_cmd
        if ! $SSH $SRV_USER@$SERVER $CMD
        then
                echo "Tape on $SERVER is not ready"
                exit 1
        fi
}
 
backup_mount () {
        # Backup the actual mount
        # $1 - the path of the mount
        remote_tape_append
        if [ -z "$1" ]
        then
                echo "Mount path is empty?"
                exit 1
        fi
        echo "Backing up $1"
        cd "$1"
        tar $TAR_ARG . | $SSH $SRV_USER@$SERVER "$CMD" > $TAR_LOG 2>&1
}
 
append_header () {
        remote_tape_append
        cat $SUM_FILE | $SSH $SRV_USER@$SERVER "$CMD"
}
 
add_mbr () {
        remote_tape_append
        first_disk
        if [ -z "$DISK" ]
        then
                echo "Can't decide on the first boot disk. Exiting now"
                echo "No MBR backup exists"
                exit 0
        fi
        echo "Backing MBR"
        dd if=$DISK bs=1M count=1 | $SSH $SRV_USER@$SERVER "$CMD"
}
 
create_sum
create_leading_ssh
monitor_proc &
test_tape
append_header
for i in $MOUNTS
do
        backup_mount $i
done
add_mbr
rm $WORK_FILE

Cables connection in Israel for Linux

Thursday, May 14th, 2009

Update to 0.2. Links remain the same. At the moment I cannot host many versions (it’s mostly uncomfortable), but this might change in the future.

I have created a GUI cables installer and configurator for L2TP on Linux.

I have noticed that there is no GUI solution, so, after this has been brought up, I have done it (!!!)

I have uploaded these files here, and you are welcome to use them.

Remember – they are designed for a blank Ubuntu (currently. More distros will be supported in the future, upon request) with not much of junk installed. Also – they are designed for the simple user. Double-click and run. That’s it.

Quoting my readme file:

L2TP Cables connection in Israel (and across the world, where relevant) by Ez-Aton

—About:
This is an installer and configurator for L2TP over cables in Israel
With some luck, by running this installer, you will be able to connect
to the Internet with a dialer!

The system assumes you have little technical knowledge of Linux and you
are not expected to have any. Follow the defaults, and you should be fine.

This configuration will be cross distro in the future, meaning it will work
both on your Ubuntu, your RHEL, your Centos, Mandrake, etc. In order for me
to be able to do so, please assist by sending information on systems I am
not familiar with yet, per the appendix at the bottom.
Also, you can feel free to send me info in case the system did not work for
you (and let me know what are the differences from a default installation),
or, as always, send me money.

Visit my technical blog for updates and all kind of other technical stuff, at

http://run.tournament.org.il

OSS work is meant to be based on others work, and that I have done. I would
like to thank (and mention below) the resources for without this would not
have happened.

I hope you enjoy this dialer!

Ez

—How to use
Simply double-click on the “cables” icon on your desktop, and the system will
get you connected.
For CLI utilization: Run /usr/local/bin/cables

—Tools and resources used:
To create this package I have used the following tools and resources
makeself http://megastep.org/makeself/
xl2tpd by http://www.xelerance.com/software/xl2tpd/
xl2tpd guide for Israel Cables http://stuff.pulkes.org/l2tp/
ISP LNS list http://www.cables.org.il/cable-vpn/vpn.html
My connect/disconnect scripts from http://run.tournament.org.il

—License
This package contents are under GNUv2 license, meaning you have full permission
to modify the contents of this package, except for the binary packages included
with it, where you are binded by their respective licenses.

—My Distro/ISP is not supported!
Well, these things happen. Over 300 distros our there, and I can’t have them all.
However – you have your own distro, right? For me to add it to this package
(assuming you don’t want to do this yourself) you will have to supply me with the
following info:
* What distro, kernel and version, and how you get the distro name
(for example – on Redhat – /etc/redhat-release. On Ubuntu – /etc/lsb-release)
* The file containing the version inforamtion (see above)
* The versions available from your repositories of xl2tpd or l2tpd for older
releases, and where you can get them
* Your ISP, your ISPs LNS names/addresses
* Your country
* All other info you think relevant

—Change log
0.2 – Added ability to enter manual LNS address. Added Orange LNS. Fixed fixroute to allow both IP and hostname without problems. Fixed cables connection script to run fixroute anyhow.
0.1 – Initial release

Download it here: cables_connect.sh

If you want the scripts and sources (not for the simple user!), you can get there here: l2tp-cables

Centreon and batch-adding hosts

Monday, April 27th, 2009

Centreon is a nice GUI wrapper for Nagios. It is using MySQL as its configuration engine, and it functions quite well. One thing Cacti can do but Centreon can’t is mass automatic addition of servers. I have had a new site with an installed Centreon, and I wanted to add about 40 servers to be monitored. This is a tedious work, and I was searching for some semi-automatic method of doing it.
This is not perfect, but it worked for me.
In this case I do not replicate service-group relationship, but only add a mass of servers.

First – create a text file containing a list of servers and IPs. It should look like this:
serv1:1.2.3.4
serv2:10.2.1.3
new_srv:2.3.4.1

I have placed in in /tmp/machines

Second – find the last host entry. In my case the DB name is Centreon, so I run the following command:

mysql -u root -p centreon -e’select host_id from host’

This should return a colum with numbers. Find the largest one and increment it by one. In my example the last one was 19, so my initial host_id will be 20.

You should now find the host_template_model_html_id you are to use. There are few methods for that, but the easiest way is to find another host information which matches to some level your desired information. In my case it was called “DB1″, so this looks like this:

mysql -u root -p centreon -e”select host_template_model_htm_id from host where host_name=’DB1′”

Please note that my blog formatting might change the quote character. You might not want to copy/paste it, but type it yourselves.

The result of the above query should give us a template ID. In my case it was “2″, which is fine by me.

If you want a better reference for the values entered, you can do a whole select for a single host to verify your values match mine:

mysql -u root -p centreon -e”select * from host where host_name=’DB1′\G”

This should give you long listing and information of the host, as a reference.

My script goes like this, based on the assumptions made above:

#!/bin/bash
HOST=20
for i in `cat /tmp/machines`
do
   NAME=`echo $i | cut -f1 -d:`
   IP=`echo $2 | cut -f2 -d:`
   echo "insert into host values ('$HOST',2,NULL,NULL,1,NULL,NULL,NULL,NULL,'$NAME','$NAME','$IP',NULL,NULL,'2','2','2','2','2',NULL,'2',NULL,NULL,'2','2','2','2',NULL,NULL,'2',NULL,NULL,'0',NULL,'1','1');" >> /tmp/insert_sql.sql
   echo "insert into extended_host_information values('',$HOST,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL);" >> /tmp/insert_sql.sql
   let HOST++
done

This should create a file called /tmp/insert_sql.sql which then should be first reviewed, and then inserted into your database.

Needless to say – back up your database first, just in case:

mysqldump -u root -p –opt -B centreon > /tmp/centreon_backup.sql

and then insert the newly created data:

mysql -u root -p centreon < /tmp/insert_sql.sql

Notice – at this point, no service relationship is created. I think it is quite a chore only to create the nodes. Adding the service relationships complicates things a bit, and I did not want to go there at this specific stage. However, for few tenths of monitored hosts, this is quite a lifesaver.

Notice that this is only Centreon configuration, and you will be required to apply it (through the GUI) to Nagios.

RedHat Cluster custom Oracle “Agent”/script V1.0

Friday, April 24th, 2009

Working with RH Cluster quite a lot, I have decided to create an online store of customer agents/scripts.

I have not, so far, invested the effort of making these agents accept settings from the cluster.conf file, but this might happen.

Let the library be!

Oracle DB script/agent:

Although I discovered (a bit late) that RH Cluster for Oracle Ent. Linux 5.2 does include oracle DB agent, this script should be good enough for RHEL4 RH Cluster versions as well.

This script only checks that the ’smon’ process is up. Nothing fancy. This script can include, in the future, the ability to check that Oracle responses to SQL queries (meaning – actually working).

?Download oracle.sh
#!/bin/bash
#Service script for Oracle DB under RH Cluster
#Written by Ez-Aton
#http://run.tournament.org.il
 
# Global variables
ORACLE_USER=oracle
HOMEDIR=/home/$ORACLE_USER
OVERRIDE_FILE=/var/tmp/oracle_override
REC_LIST="user@domain.com"
 
function override () {
	if [ -f $OVERRIDE_FILE ]
	then
		exit 0
	fi
}
 
function start () {
	su - $ORACLE_USER -c ". $HOMEDIR/.bash_profile ; sqlplus / as sysdba << EOF
startup
EOF
"
	status
}
 
function stop () {
	su - $ORACLE_USER -c ". $HOMEDIR/.bash_profile ; sqlplus / as sysdba << EOF
shutdown immediate
EOF
"
	status && return 1 || return 0
}
 
function status () {
	ps -afu $ORACLE_USER | grep -v grep | grep smon
	return $?
}
 
function notify () {
	mail -s "$1 oracle on `hostname`" $REC_LIST < /dev/null
}
 
override
case "$1" in
start)	start
	notify $1
	;;
stop)	stop
#	notify $1
	;;
status)	status
	;;
*)	echo "Usage: $0 start|stop|status"
	;;
esac

I usually place this script (with execution permissions, of course) in /usr/local/sbin and call it as a “script” from the cluster configuration. You will probably be required to alter the first few variable lines to match to your environment.

Listener Agent/script:

The tnslsnr should be started/stopped as well, if we want the $ORACLE_HOME to migrate as well. This is its agent/script:

?Download lsnr.sh
#!/bin/bash
#Service script for Oracle DB under RH Cluster
#Written by Ez-Aton
#http://run.tournament.org.il
 
ORACLE_USER=oracle
HOMEDIR=/home/$ORACLE_USER
OVERRIDE_FILE=/var/tmp/oracle_override
 
function override () {
if [ -f $OVERRIDE_FILE ]
then
exit 0
fi
}
 
function start () {
su - $ORACLE_USER -c ". $HOMEDIR/.bash_profile ; lsnrctl start"
status
}
 
function stop () {
su - $ORACLE_USER -c ". $HOMEDIR/.bash_profile ; lsnrctl stop"
status && return 1 || return 0
}
 
function status () {
su - $ORACLE_USER -c ". $HOMEDIR/.bash_profile ; lsnrctl status"
}
 
override
case "$1" in
start)    start
;;
stop)    stop
;;
status)    status
;;
*)    echo "Usage: $0 start|stop|status"
;;
esac

Again – place it in /usr/local/sbin and call it from the cluster configuration file as type “script”.

I will add more agents and more resources for RedHat Cluster in the future.

Wordpress Mu 2.7.1 and Multi-Site Manager

Friday, April 24th, 2009

I have neglected this blog due to routine work lately. I have done some things, but not much to write home about, and being swamped with work, I just lacked the initiative to add new stuff, or to dwell into new technologies and problems. Routine is routine, and upgrade to Wordpress Mu sometimes introduces new issues.

A very useful plugin called “Multi-Site Manager” is available for, well, managing easily multi-site configuration of a single WPMU from the GUI.

I have been using it for a while, and I was happy with it, however, after the latest Mu upgrade (2.7.1), a single problem appeared – you cannot manage sites from a newly created domain. Already existing domains functionality was not changed, but only new ones.

I through myself knee deep into its MySQL data, and got to the following workaround. This should be incorporated into the code, but me being such a lame PHP coder, maybe it’s not the best idea that I do it…

Problem:

You have created a new site, but from within its admin you cannot view the site admin panel. Manually entering the URL results in a “Permission Denied” error message.

Solution:

Login to your MySQL DB.

Use the MU database.

Find your new site ID:

select * from wp_site;

Write down the site number. Also, write down another site number – a small site would be best.

Get the correct entry for site_admins from some other blog

Select * from wp_sitemeta where meta_key=’site_admins’;

Response should look something like this:

+———+———+————-+——————————————+
| meta_id | site_id | meta_key | meta_value |
+———+———+————-+——————————————+
| 8 | 1 | site_admins | a:1:{i:0;s:5:”adm”;} |
| 467 | 21 | site_admins | a:1:{i:0;s:5:”adm”;} |
| 245 | 13 | site_admins | a:1:{i:0;s:5:”adm”;} |
| 45 | 3 | site_admins | a:1:{i:0;s:5:”adm”;} |
| 28 | 2 | site_admins | a:1:{i:0;s:5:”adm”;} |
| 62 | 5 | site_admins | a:1:{i:0;s:5:”adm”;} |
| 126 | 8 | site_admins | a:1:{i:0;s:5:”adm”;} |
+———+———+————-+——————————————+

Create a site_admin entry for your new site:

insert into wp_sitemeta (site_id,meta_key,meta_value) values (‘22′,’site_admins’,'a:1:{i:0;s:5:”adm”;}’);

Im my example, 22 is the site ID, obtained from the first query.

Following that, refresh your dashboard GUI and you should be able to find your “Site Admin” panel there. From here you can define your desired settings for the site as you see fit.

Relocating LVs with snapshots

Monday, February 2nd, 2009

Linux LVM is a wonderful thing. It is scalable, flexible, and truly, almost enterprise-class in every details. It lacks, of course, at IO performance for LVM snapshots, but this can be worked-around in several creative ways (if I haven’t shown here before, I will sometime).

What it can’t do is dealing with a mixture of Stripes, Mirrors and Snapshots in a single logical volume. It cannot allow you to mirror a stripped LV (even if you can follow the requirementes), it cannot allow you to snapshot a mirrored or a stripped volume. You get the idea. A volume you can protect, you cannot snapshot. A volume with snapshots cannot be mirrored or altered.

For the normal user, what you get is usually enough. For storage management per-se, this is just not enough. When I wanted to reduce a VG – remove a disk from an existing volume group,  I had to evacuate it from any existing logical volume. The command to perform this actions is ‘pvmove‘ which is capable of relocating data from within a PV to other PVs. This is done through mirroring each logical volume and then removing the origin.

Mirroring, however, cannot be performed on LVs with snapshots, or on an already mirrored LV, so these require different handling.

We can detect which LVs reside on our physical volume by issuing the following command

pvdisplay -m /dev/sdf1

/dev/sdf1 was only an example. You will see the contents of this PV. So next, performing

pvmove /dev/sdf1

would attempt to relocate every existing LV from this specific PV to any other available PV. We can use this command to change the disk balance and allocations on multi-disk volume groups. This will be discussed on a later post.

Following a ‘pvmove‘ command, all linear volumes are relocated, if space permits, to another PVs. The remaining LVs are either mirrored or LVs with snapshots.

To relocate a mirrored LV, you need to un-mirror it first. To do so, first detect using ‘pvdisplay‘ which LV is belongs to (the name should be easy to follow) and then change it to non-mirrored.

lvconvert -m0 /dev/VolGroup00/test-mirror

This will convert it to be a linear volume instead of a mirror, so you could move it, if it still resides on the PV you are to remove.

Snapshot volumes are more complicated, due to their nature. Since all my snapshots are of a filesystem, I could allow myself to use tar to perform the action.

The steps are as follow:

  1. tar the contents of the snapshot source to nowhere, but save an incremental file
  2. Copy the source incremental file to a new name, and tar the contents of a snapshot according to this copy.
  3. Repeat the previous step for each snapshot.
  4. Remove all snapshots
  5. Relocate the snapshot source using ‘pvmove
  6. Build the snapshots and then recover the data into them

This is a script to do steps 1 to 3. It will not remove LVs, for obvious reasons. This script was not tested, but should work, of course :-)

None of the LVs should be mounted for it to function. It’s better to have harder requirements than to destroy data by double-mounting it, or accessing it while it is being changed.

#!/bin/bash
# Get: VG Base-LV, snapshot name, snapshot name, snapshot name...
# Example:
# ./backup VolGroup00 base snap1 snap2 snap3
# Written by Ez-Aton
 
TARGET=/tmp
if [ "$@" -le 3 ]
then
   echo "Parameters: $0 VG base snap snap snap snap"
   exit 1
fi
VG=$1
BASE=$2
shift 2
 
function check_not_mounted () {
   # Check if partition is mounted
   if mount | grep /dev/mapper/${VG}-${1}
   then
      return 0
   else
      return 1
   fi
}
 
function create_base_diff () {
   # This function will create the diff file for the base
   mount /dev/${VG}/${BASE} $MNT
   if [ $? -ne 0 ]
   then
      echo "Failed to mount base"
      exit 1
   fi
   cd $MNT
   tar -g $TARGET/${BASE}.tar.gz.diff -czf - . &gt; /dev/null
   cd -
   umount $MNT
}
 
function create_snap_diff () {
   mount /dev/${VG}/${1} $MNT
   if [ $? -ne 0 ]
   then
      echo "Failed to mount base"
      exit 1
   fi
   cp $TARGET/${BASE}.tar.gz.diff $TARGET/$1.tar.gz.diff
   cd $MNT
   tar -g $TARGET/${1}.tar.gz.diff -czf $TARGET/${1}.tar.gz .
   cd -
   umount $MNT
}
 
function create_mount () {
   # Creates a temporary mount point
   if [ ! -d /mnt/$$ ]
   then
      mkdir /mnt/$$
   fi
   MNT=/mnt/$$
}
 
create_mount
if check_not_mounted $BASE
then
   create_base_diff
else
   echo "$BASE is mounted. Exiting now"
   exit 1
fi
for i in $@
do
   if check_not_mounted $i
   then
      create_snap_diff $i
   else
      echo "$i is mounted! I will not touch it!"
   fi
done

The remaining steps should be rather easy – just mount the newly created snapshots and restore the tar file on them.

How to create self-contained Solaris 10 x86 Jumpstart kit

Saturday, December 27th, 2008

I was required to create a self-contained, single DVD to automate the installation of Solaris 10 on x86_64. I could not find any up-to-date straight forward guide which can explain how to do it, so I do it here. This is not an explanation for dummies, so you must know (to some degree, of course) what you’re doing.

I will describe the procedure in whole, and will explain in greater details below, if I see fit. A section which will be explained later will be marked with (*) at the end of the line.

  • Install Solaris 10 x86 on a machine. Many actions will happen on this little server…
  • Setup your Solaris installation according to your likings. Make sure you have your beloved users, your passwords, your configurations. Don’t mind much about networking configurations (IP, Netmask, etc) – as they will be unconfigured for the image.
  • Create a Flash Image (flar) of the system (*)
  • Copy the contents of the installation DVD to a directory inside your system. Let’s call it /tmp/dvd
  • Remove /tmp/dvd/Solaris_10/Product directory. You will not need it.
  • Extract the contents of /tmp/dvd/x86.miniroot to /tmp/miniroot (*)
  • Perform several actions with the extracted miniroot (*)
  • Re-archive the contents of the x86.miniroot and place them instead of /tmp/dvd/boot/x86.miniroot
  • Place the flar file inside /tmp/dvd/flash
  • Edit your jumpstart files inside /tmp/dvd/.install_config (*)
  • Edit /tmp/dvd/boot/grub/menu.lst boot loader to add an entry for your installation (*)
  • Create an ISO from the DVD directory (*)
  • Burn the DVD and try to use it

And now for the drill-down

Creating a flash image

Use the command flarcreate to create your own flash image:

flarcreate -n sol10_automation -c -x /tmp /tmp/sol10_auto.flar

This should do the work. Remember – /tmp will not be persistent across reboots! Make sure your files are not there before you reboot the system!

Extracting/Archiving the x86.miniroot

To do so, you need to run the command /boot/solaris/bin/root_archive

Extracting the image can be done like this:

/boot/solaris/boot/root_archive unpack /tmp/dvd/boot/x86.miniroot /tmp/miniroot

Archiving the image can be done like this:

/boot/solaris/boot/root_archive pack /tmp/dvd/boot/x86.miniroot /tmp/miniroot

Actions to perform on the extracted miniroot

Three actions are to be performed on the extracted miniroot. In our example, it resides on /mnt/miniroot.

First, you need to remove the default sysidcfg (which is a symbolic link)

rm /mnt/miniroot/etc/sysidcfg

Now, you have to place your custom sysidcfg in there, instead.

This is an example of my own sysidcfg file:

name_service=NONE
network_interface=nge0 {primary hostname=sol10
ip_address=10.10.10.10
netmask=255.0.0.0
default_route=NONE
protocol_ipv6=no }
nfs4_domain=dynamic
service_profile=open
root_password=12wR2rF34t
security_policy=NONE
system_locale=en_US.UTF-8
timezone=GMT
timeserver=localhost
keyboard=US-English
terminal=xterm

The root password is encrypted. Take it from your own /etc/shadow file. For more information about sysidcfg file, check out Sun site.

Following that, you need to edit a specific file in the miniroot. Edit /tmp/miniroot/usr/sbin/install.d/profind and search for the cdrom() function. Search the line

if [ -f /tmp/.preinstall ]; then

and hash (remark) it. Don’t forget to remark the closing “fi” below.

Jumpstart contents

This has to be inside /tmp/dvd/.install_config . Edit the file /tmp/dvd/.install_config/rules and make sure it has only one line (in our example. If you know what you’re doing with Jumpstart, go ahead!)

any -   x86-begin any_machine  x86-end

This line will match any hardware, run x86-begin script (from that same directory) on it prior to running the installation itself, and run x86-end script on it after the installation phase. It allows up further customisation during installs (verify what type of RAID, check memory, whatever). The installation profile itself is the file any_machine.

You will need to run “check” on the file to build the rules.ok file

cd /tmp/dvd/.install_conf

/tmp/dvd/Solaris_10/Misc/jumpstart_sample/check

Lets look at my any_machine file:

install_type    flash_install
archive_location local_file /cdrom/flash/sol10_auto.flar
partitioning    explicit
filesys         any 8196 swap
filesys         any 10240 /
filesys         any free /storage

Notice that the installation type is “flash_install” and that the location of the file is local, inside /cdrom (where the bootable dvd will be mounted) inside a directory called flash. Partitioning is defined here, explicitly.

For more information about Jumpstart, search in Sun site. They have plenty of information.

Edit Grub

Add the following entry to your /tmp/dvd/boot/grub/menu.lst file

title Solaris10 Jumpstart
kernel /boot/multiboot kernel/unix – install -B \
install_media=cdrom
module /boot/x86.miniroot

Make sure it is the default option for grub.

Creating DVD ISO from the directory

We’re almost done. To create a DVD iso file from the directory, perform the following actions:

cd /tmp/dvd

mkisofs -b boot/grub/stage2_eltorito -c .catalog -no-emul-boot -boot-load-size 4 -boot-info-table -relaxed-filenames -l -ldots -r -N -d -D -V SOL_10_1008_X86 -o /tmp/sodvd.iso .

Don’t ignore the “.” at the end!

(This specific line was tested on Linux, but there is no reason for it not to work on any modern Solaris system)

Appendix

You would like to keep your /tmp/dvd directory somewhere else, or you will lose it on your next reboot.

This sums it up. Let me know if the procedure is broken somehow.

MySQL permissions for LVM Snapshots

Thursday, October 23rd, 2008

aking LVM snapshots as a mean of backing up MySQL is rather simple, as can be described here. However, if you are into security, you would strive to grant minimal permissions for the action to the MySQL user. Per MySQL Documentation, the required privileges is “RELOAD”. That should be enough, granted on *.*, of course.

Xen VMs performance collection

Saturday, October 18th, 2008

Unlike VMware Server, Xen’s HyperVisor does not allow an easy collection of performance information. The management machine, called “Domain-0″ is actually a privileged virtual machine, and thus – get its own small share of CPUs and RAM. Collecting performance information on it will lead to, well, collecting performance information for a single VM, and not the whole bunch.

Local tools, such as “xentop” allows collection of information, however, combining this with Cacti, or any other SNMP-based collection tool is a bit tricky.

A great solution is provided by Ian P. Christian in his blog post about Xen monitoring. He has created a Perl script to collect information. I have taken the liberty to fix several minor things with his permission. The modified scripts are presented below. Name the script (according to your version of Xen) “/usr/local/bin/xen_stats.pl” and set it to be executable:

For Xen 3.1

?Download xen_stats.pl
#!/usr/bin/perl -w
 
use strict;
 
# declare...
sub trim($);
#<a href="/blog/files/xen_cloud.tar.gz" title="xen_cloud.tar.gz" target="_blank">xen_cloud.tar.gz</a>
# we need to run 2 iterations because CPU stats show 0% on the first, and I'm putting .1 second betwen them to speed it up
my @result = split(/\n/, `xentop -b -i 2 -d.1`);
 
# remove the first line
shift(@result);
 
shift(@result) while @result &amp;&amp; $result[0] !~ /^xentop - /;
 
# the next 3 lines are headings..
shift(@result);
shift(@result);
shift(@result);
shift(@result);
 
foreach my $line (@result)
{
  my @xenInfo = split(/[\t ]+/, trim($line));
  printf("name: %s, cpu_sec: %d, cpu_percent: %.2f, vbd_rd: %d, vbd_wr: %d\n",
    $xenInfo[0],
    $xenInfo[2],
    $xenInfo[3],
    $xenInfo[14],
    $xenInfo[15]
    );
}
 
# trims leading and trailing whitespace
sub trim($)
{
  my $string = shift;
  $string =~ s/^\s+//;
  $string =~ s/\s+$//;
  return $string;
}

For Xen 3.2 and Xen 3.3

?Download xen_stats.pl
#!/usr/bin/perl -w
 
use strict;
 
# declare…
sub trim($);
 
# we need to run 2 iterations because CPU stats show 0% on the first, and I’m putting .1 second between them to speed it up
my @result = split(/\n/, `/usr/sbin/xentop -b -i 2 -d.1`);
 
# remove the first line
shift(@result);
shift(@result) while @result &amp;&amp; $result[0] !~ /^[\t ]+NAME/;
shift(@result);
 
foreach my $line (@result)
{
        my @xenInfo = split(/[\t ]+/, trim($line));
        printf(“name: %s, cpu_sec: %d, cpu_percent: %.2f, vbd_rd: %d, vbd_wr: %d\n,
        $xenInfo[0],
        $xenInfo[2],
        $xenInfo[3],
        $xenInfo[14],
        $xenInfo[15]
        );
}
# trims leading and trailing whitespace
sub trim($)
{
        my $string = shift;
        $string =~ s/^\s+//;
        $string =~ s/\s+$//;
        return $string;
}

Cron settings for Domain-0

Create a file “/etc/cron.d/xenstat” with the following contents:

# This will run xen_stats.pl every minute
*/1 * * * * root /usr/local/bin/xen_stats.pl > /tmp/xen-stats.new && cat /tmp/xen-stats.new > /var/run/xen-stats

SNMP settings for Domain-0

Add the line below to “/etc/snmp/snmpd.conf” and then restart the snmpd service

extend xen-stats   /bin/cat /var/run/xen-stats

Cacti

I reduced Ian Cacti script to be based on a per-server setup, meaning this script gets the host (dom-0) name from Cacti, but cannot support live migrations. I will try to deal with combining live migrations with Cacti in the future.

Download and extract my modified xen_cloud.tar.gz file. Extract it, place the script and config in its relevant location, and import the template into Cacti. It should work like charm.

A note – the PHP script will work only on PHP5 and above. Works flawlessly on Centos5.2 for me.