Running Systems

On most cases, when preparing a recovery server, you can just ‘tar’ the entire server’s contents and just move it, along with a short recipe on how to rebuild the original partition layout (software Raid? LVM? flat partition tables?), how to mount volumes in-place, how to extract the tar files into the right locations, and how to install your favorite boot loader, either Lilo or Grub. Also, beforehand, you deal with taking a nice snapshot (or capturing the system in a single-user phase), and life is good, yada yada yada.

Most of us, however, never prepare for a rainy day. It’s not that we don’t want to, it’s not that we don’t plan, it’s just that we never seem to get to it, and after all, the hardware is rather new, and there should be no reason for failure. I can guess some of you heard this before – maybe in their own voices.

So, backup is a tiring job, and I will not deal with the things you need to do to maintain a replica of your data, but I will deal with how to prepare, quickly and easily, a system-recovery server (or a postmortem server) with just a little thought beforehand. This might be a bit too late for you if you’re reading it now, but, well, for the next time…

This little trick worked (as part of a large-scale process) when I migrated a server from 64bit server to a 32bit server (yeah, I know – the other way around).

It assumes you use RPM as your tool to install applications, and that if you do not, you have a method of knowing which piece of software you installed from source, and which package was installed from an external source (not your day-to-day RPM repository).

On the source server, run ‘rpm -qa > /tmp/rpmlist.long‘. Keep this file. It is important. Also, try to keep your yum.repos.d directory, or at least know which rpm repositories you use (I always use rpmforge, so I see no problem with that).

Install your target server – Same version as the source, minimal package selection. Copy the file rpmlist.long to /tmp. Make sure yum is configured (I will deal here with YUM, but you can replace it with any other repository client of your choice). Run the two following lines:

cat /tmp/rpmlist.long | sed s/-[0-9].*$/”/g > /tmp/rpmlist-short

for i in `cat /tmp/rpmlist-short`; do

yum install -y $i

done

This will add the missing RPMs with their dependencies, and will bring your system to a similar status. At least, this is a good place to start recovering.

On future chapters:

- Fully migrating from 64 to 32 bit and vice versa

- Using LVM snapshots for a smart backup, and for a smart recovery

Many of us use tar (many times with gzip or bzip2) for archiving purposes. When performing such an action, a large file, usually, too large, remains. To extract from it, or to split it becomes an effort.

This post will show an example of a small script to split an archive and later on, to directly extract the data out of the slices.

Let’s assume we have a directory called ./Data . To archive it using tar+gzip, we can perform the following action:

tar czf /tmp/Data.tar.gz Data

For verbose display (although it’s could slow down things a bit), add the flag ‘v’.

Now we have a file called /tmp/Data.tar.gz

Lets split it to slices sized 10 MB each:

cd /tmp
mkdir slices
i=1 # Our counter
skip=0 # This is the offset. Will be used later
chunk=10 # Slice size in MB
let size=$chunk * 1024 # And in kbytes
file=Data.tar.gz # Name of the tar.gz file we slice
while true ; do
# Deal with numbers lower than 10
if [ $i -lt "10" ]; then
j=0${i}
else
j=${i}
fi
dd if=${fie} of=slices/${file}.slice${j} bs=1M count=${chunk} skip=${skip}
# Just to view the files with out own eyes
ls -s slices/${file}.slice${j}
if [ `ls -s slices/${file}.slice${j} | awk '{print $1}'` -lt "${size}" ]; then
echo “Done”
break
fi
let i=$i+1
let skip=$skip+$chunk
done

This will break the tar.gz file to a files with running numbers added to their names. It assumes that the number of slices would not exceed 99. You can extend the script to deal with three digits numbers. The sequence is important for later. Stay tuned

Ok, so we have a list of files with a numerical suffix, which, combined, include our data. We can test their combined integrity:

cd /tmp/slices
i=1
file=Data.tar.gz
for i in `ls`; do
cat ${file}.slice${i} >> ../Data1.tar.gz
done

This will allow us to compare /tmp/Data.tar.gz and /tmp/Data1.tar.gz. I tend to use md5sum for such tasks:

md5sum Data.tar.gz
d74ba284a454301d85149ec353e45bb7 Data.tar.gz
md5sum Data1.tar.gz
d74ba284a454301d85149ec353e45bb7 Data1.tar.gz

They are similar. Great. We can remove Data1.tar.gz. We don’t need it anymore.

To recover the contents of the slices, without actually wasting space by combining them before extracting their contents (which requires time, and disk space), we can run a script such as this:

cd /tmp/slices
file=Data.tar.gz
(for i in `ls ${file}.slice*`; do
cat $i
done ) | tar xzvf -

This will extract the contents of the joined archive to the current directory.

This is all for today. Happy moving of data

Update to the version posted here, the new shiny version includes several interesting additions. From the ChangeLog:

NabRSS Version 0.3 Jul 15th 07:
- DB – Extended feed_list table to include selected voice
- DB – Inserted all voices into DB
- add_url.php – Enhanced to allow for (optional) selection of voice to an RSS feed
- add_url.php – Now checks input for duplicates and prevents them
- list.php – Enhanced to support voice definitions
- get.php – Changed stracture to somewhat more natural (to me, that is)
- get.php – Added support for voice selection. If not selected, randomize the voice
- get.php – Updated the DB scheme comment
- header.php – fixed problem with comments. They were incorrect, and now they are ok.

Also easy (or at least, easier) upgrade from past versions.

Available here: nabrss-0.3.tar.gz

Enjoy!

Again, on a short notice, the package introduced here and updated here has been updated.

From the ChangeLog:

NabRSS Version 0.21 Jul 14th 07:
- Started moving functions into functions.php
- Added support for RSS feeds with login
- Hiding passwords for login-based RSS feeds
- Minor updates to README.txt

Enjoy

NabRSS has been updated from the version posted here.

NabRSS is a set of PHP scripts meant to assist anyone with a PHP/MySQL/Apache server available to gather feeds from RSS sources and send them via Nabaztag API to his/her Nabaztag.

Available here: nabrss-0.2.tar.gz

Quoting from the changelog:

NabRSS Version 0.2 Jul 14th 07:
- get.php: Fixed an incorrect condition in function debug_set
- get.php: Added function set_ear_pos to randomize ear position on every post
- get.php: Added function set_voice to randomize each RSS post voice
- list.php: Added a “Reset” button to allow reseting the last post saved in the date field inside the DB

Todo (short range):
- DB: Extend the scheme and move voices into DB as a seperated table
- add_url.php: Select a voice for each RSS feed. Possiblity to select “random”
- list.php: Show selected voice for each RSS feed
- get.php: Insert a function to deal with voices per post

I have written an RSS to Nabaztag (using API) php scripts. Special thanks to magpierss, for doing the complicated work of converting RSS feeds to PHP-usable arrays.

This system requires MySQL server, Apache server and PHP version 5 (haven’t tested it on PHP version 4).

The scripts are supplied under GPL license, and are free for use and distribution. Please keep the header information containing my details on every distribution of the scripts. Also, please help me make the scripts better, by leaving feedback and sending me fixes.

You can find the archive here: nabrss-0.1.tar.gz

Enjoy!

For the latest (and currently whole) ks.cfg I use, check this link. I have extended the logic there, and got the following out of it. Showing only the %pre section:

%pre
# By Ez-Aton http://www.tournament.org.il/run
for i in `cat /proc/cmdline`; do
echo $i >> /tmp/vars.tmp
done
grep “=” /tmp/vars.tmp > /tmp/vars
# Parse command line. Using only vars with type var=value (doesn’t matter
# what the actual value is)
KS=/tmp/ks.cfg
update=”"
name=”"
pkg=”"
. /tmp/vars
# Shall we update the system during the %post section?
if [ ! -z "$update" ]; then
echo “yum update -y” >> $KS
fi
# Shall we reboot the system after the installation?
if [ ! -z "$reboot" ]; then
echo “reboot” > $KS.tmp
cat $KS >> $KS.tmp
cat $KS.tmp > $KS
fi
# What is the machine’s hostname?
if [ ! -z "$name" ]; then
value=”dhcp –hostname $name”
cat $KS | sed s/dhcp/”$value”/ > $KS.tmp
cat $KS.tmp > $KS
fi
# Shall we add another package to the installation preset?
if [ ! -z "$pkg" ]; then
pkg_line=`grep -n ^%packages $KS | cut -f 1 -d \:`
max_line=`wc -l $KS | awk ‘{print $1}’`
head -n $pkg_line $KS > $KS.tmp
for i in `echo $pkg | sed s/,/\ /g`; do
echo $i >> $KS.tmp
done
let tail_line=$max_line-$pkg_line
tail -n $tail_line $KS >> $KS.tmp
cat $KS.tmp > $KS
fi
# Is it a virtual machine running on VMWare? If so, we’ll install vmware-tools
if [ ! -z "$vmware" ]; then
# We need vmhttp value for server. It can be the name and path
# of the web server
if [ -z "$vmhttp" ]; then
# my defaults
vmhttp=”centos4-01″
fi
# The name of the rpm is always vmware.rpm
echo “wget http://$vmhttp/vmware.rpm” >> $KS
echo “rpm -i vmware.rpm” >> $KS
fi

RHEL4 tends to change the names of network cards when there are more than one. If you had a NIC called eth0 during install time, it doesn’t mean that it will maintain that name after the first reboot. It could switch names with its friend, and be called now eth1, while the previous eth1 name is now eth0.

A solution using udev was posted in HPs forums, and can be reached directly through here. I will quote it:

Device persistence can also be enabled to ensure that the NICs identifying themselves as eth1, eth2, etc… always remain on the same hardware ports in case of a failure of a single NIC port. You don’t want your eth names to shift.

Upgrading to udev-095 from udev-039 that ships with RHEL4 is the smoothest solution, but that wasn’t an option for me. Using names other than eth0 – eth3 also wasn’t an option for me. Here is what we ended up using to get around udev-039’s inability to re-use eth0-ethx names.

Create a udev rule using YOUR MACS
Create an /etc/mactab file using YOUR MACS
Modify /etc/init.d/network to run nameif

/etc/udev/rules.d/20-net.rules
——————————–
KERNEL=”eth*”, SYSFS{address}=”00:0b:cd:69:c3:66″, NAME=”NIC1″
KERNEL=”eth*”, SYSFS{address}=”00:0b:cd:69:c3:65″, NAME=”NIC2″
KERNEL=”eth*”, SYSFS{address}=”00:11:0a:17:66:26″, NAME=”NIC3″
KERNEL=”eth*”, SYSFS{address}=”00:11:0a:17:66:27″, NAME=”NIC4″

/etc/mactab
————-
eth0 00:0b:cd:69:c3:66
eth1 00:0b:cd:69:c3:65
eth2 00:11:0a:17:66:26
eth3 00:11:0a:17:66:27

/etc/init.d/network
————————
(add right after
>>
# Check that networking is up.
[ "${NETWORKING}" = "no" ] && exit 0
<<)

# RDD: add ‘nameif’ usage; uses /etc/mactab
nameif || echo “nameif: reports error”

While DL140 G3 is quite a new piece of hardware, RHEL4, even with the later updates, is rather old.

When you decide to install RHEL4 on a DL140 G3 server, my first recommendation is this: if you decide to use the embedded SATA-II RAID controller – don’t. This is a driver-based RAID, much like the past win-modem devices. Some major parts of its operations are based on calculations done through the driver, directly on the host CPU. It has no advantages comparing to software RAID, and its major disadvantage is its immobile state – unlike software based mirror, this array cannot “migrate” to another server, unless this server is of the same type of hardware. Not sure about it, but it might also require close enough version of firmware as well.

It happened that your boss believes in win-RAID devices (despite the note above), or for some other reason you decide to use this win-RAID, here are the steps to install the system.

1. Download the latest driver disk image for RHEL4 from HP site.

2. If you have the privilage of having an NFS server, uncompress the image and put it on it, where it can be accessed through network.

3. Test that you can mount it from another server. Verify you can reach the image file. Debugging incorrect NFS issues can waste lots of time.

4. If you don’t have the privilege, I hope you have a USB floppy. Put the image on a floppy disk:

gzip -dc /path/to/compressed/image/file.gz | dd of=/dev/fd0

(/dev/fd0 assuming this is not /dev/sdX, as it tends to be with USB floppies)

5. Boot the server with the first RHEL4 CD in the drive, or with PXE, or whatever is your favorite method. In the initial boot prompt type:

linux text dd=nfs:server:/path/to/nfs/disk/disk_image.dd

This assuming that the name of the file (including its full path) is /path/to/nfs/disk/disk_image.dd. For floppy users, type dd=floppy instead.

6. RHEL will boot, loading “ahci” module (which is bad) during its startup. It will ask you to select through which network card the system is to reach the NFS server. I assume you have a working DHCP in your site.

7. As soon as you are able to use the virtual terminal (Ctrl+F2) switch to it.

8. Run the following commands:

cd /tmp
mkdir temp
cd temp
gzip -S .cgz -dc /tmp/ramfs/DD-0/modules.czf | cpio -id

cd to the modules directory, and look at the modules. Know which is the module which fits your running kernel. You can do this by using ‘uname’ command.

9. Run the following commands

rmmod ahci
rmmod adpahci
insmod KERNEL_VER/ARCH/adpahci.ko

Replace KERNEL_VER with your running(!!!) single-CPU kernel version, and replace ARCH with your architecture, either i386 or x86_64

10. Using Ctrl+F1 return to your running installer. Continue installation until the end but do not reboot the system when done.

11. When installation is done, before the reboot, return to the virtual console using Ctrl+F2.

12. Run the following commands to prepare your system for a happy reboot:

cp /tmp/temp/KERNEL_VER/ARCH/adpachi.ko /mnt/sysimage/lib/modules/KERNEL_VER/kernel/drivers/scsi/

cp /tmp/temp/KERNEL_VERsmp/ARCH/adpachi.ko /mnt/sysimage/lib/modules/KERNEL_VERsmp/kernel/drivers/scsi

Notice that we’ve copied both the single CPU (UP) and the SMP versions.

13. Edit modprobe.conf of the system-to-be and remove the line containing “alias scsi_hostadapter ahci” from the file.

14. Chroot into the system-to-be, and build your initrd:

chroot /mnt/sysimage
cd /boot
mv initrd-KERNEL_VER.img initrd-KERNEL_VER.img.orig
mv initrd-KERNEL_VERsmp.img initrd-KERNEL_VERsmp.img
mkinitrd /boot/initrd-KERNEL_VER.img KERNEL_VER
mkinitrd /boot/initrd-KERNEL_VERsmp.img KERNEL_VERsmp
exit

If things went fine so far, you are now ready to reboot. Use Ctrl+F1 to return to the installation (anaconda) console, and reboot the system.

Notes:

1. You need to download the “Driver Diskette” from HP site.

2. The latest Driver Diskette will support only Update3 and Update4 based systems. At this time, Update5 has no modules by HP yet. You can compile your own, but this is not in our scope.

3. Avoid using floppies at all cost.

4. Do not install the system in full GUI mode. In the model I have installed the VGA (Matrox device) had a bug and did not allow to reach the virtual text consoles. It disconnected the VGA. If you use GUI installation, you will be required to reboot the system into rescue mode and do steps 7 to 14 then.

5. Underlining the word smp is meant to help you not forget it. This is the more important one.

6. On the system itself, using Xorg, I was able to reach max resolution of 640×480 even with the display drivers supplied by HP. I was able to reach 1024×768 only when using 256 colors.

We all know this, right? In my new system, I have forgotten to activate the DVD-R’s DMA. I tried burning a DVD, and it went slow – real slow.

I have checked the DMA status:

$ hdparm /dev/hda

/dev/hda:
IO_support = 0 (default 16-bit)
unmaskirq = 1 (on)
using_dma = 0 (off)
keepsettings = 0 (off)
readonly = 0 (off)
readahead = 256 (on)
HDIO_GETGEO failed: Inappropriate ioctl for device

I have rather expected that. After a quick glance, I have activated the DMA, while the DVD was still burning, against all common sense.

# hdparm -d1 /dev/hda

/dev/hda:
setting using_dma to 1 (on)
using_dma = 1 (on)

DMA seemed to be on, and I was happy. I noticed that:

a. the burning process did not fail

b. burning speed increased dramatically

I thought I was able to pull this trick off. However, the DVD was unreadable later on, and I noticed an interesting pattern on it, as can be described only by my poor attempt to capture it in a picture:

Notice the “layers”. You can see the transparent plastic (and my own hand holding the DVD) and the black no-data layer (seems green from here, but it is black), and then the regular marks of burned CD, and about one third of the way (sector-wise, it is about fifth or sixth of the way) a change in the pattern of the marks. This goes until the last circle of unburned CD.

Check out this other attempt to capture the texture:

Here the change in the texture is very visible.

A note – this is not a rewritable DVD, and the whole operation went in one go.

Interesting.