Archive for July, 2007

Preperation of recovery server for RPM based systems

Monday, July 23rd, 2007

On most cases, when preparing a recovery server, you can just ‘tar’ the entire server’s contents and just move it, along with a short recipe on how to rebuild the original partition layout (software Raid? LVM? flat partition tables?), how to mount volumes in-place, how to extract the tar files into the right locations, and how to install your favorite boot loader, either Lilo or Grub. Also, beforehand, you deal with taking a nice snapshot (or capturing the system in a single-user phase), and life is good, yada yada yada.

Most of us, however, never prepare for a rainy day. It’s not that we don’t want to, it’s not that we don’t plan, it’s just that we never seem to get to it, and after all, the hardware is rather new, and there should be no reason for failure. I can guess some of you heard this before – maybe in their own voices.

So, backup is a tiring job, and I will not deal with the things you need to do to maintain a replica of your data, but I will deal with how to prepare, quickly and easily, a system-recovery server (or a postmortem server) with just a little thought beforehand. This might be a bit too late for you if you’re reading it now, but, well, for the next time…

This little trick worked (as part of a large-scale process) when I migrated a server from 64bit server to a 32bit server (yeah, I know – the other way around).

It assumes you use RPM as your tool to install applications, and that if you do not, you have a method of knowing which piece of software you installed from source, and which package was installed from an external source (not your day-to-day RPM repository).

On the source server, run ‘rpm -qa > /tmp/rpmlist.long‘. Keep this file. It is important. Also, try to keep your yum.repos.d directory, or at least know which rpm repositories you use (I always use rpmforge, so I see no problem with that).

Install your target server – Same version as the source, minimal package selection. Copy the file rpmlist.long to /tmp. Make sure yum is configured (I will deal here with YUM, but you can replace it with any other repository client of your choice). Run the two following lines:

cat /tmp/rpmlist.long | sed s/-[0-9].*$/”/g > /tmp/rpmlist-short

for i in `cat /tmp/rpmlist-short`; do

yum install -y $i

done

This will add the missing RPMs with their dependencies, and will bring your system to a similar status. At least, this is a good place to start recovering.

On future chapters:

– Fully migrating from 64 to 32 bit and vice versa

– Using LVM snapshots for a smart backup, and for a smart recovery

Splitting archive and combining later on the fly

Wednesday, July 18th, 2007

Many of us use tar (many times with gzip or bzip2) for archiving purposes. When performing such an action, a large file, usually, too large, remains. To extract from it, or to split it becomes an effort.

This post will show an example of a small script to split an archive and later on, to directly extract the data out of the slices.

Let’s assume we have a directory called ./Data . To archive it using tar+gzip, we can perform the following action:

tar czf /tmp/Data.tar.gz Data

For verbose display (although it’s could slow down things a bit), add the flag ‘v’.

Now we have a file called /tmp/Data.tar.gz

Lets split it to slices sized 10 MB each:

cd /tmp
mkdir slices
i=1 # Our counter
skip=0 # This is the offset. Will be used later
chunk=10 # Slice size in MB
let size=$chunk * 1024 # And in kbytes
file=Data.tar.gz # Name of the tar.gz file we slice
while true ; do
# Deal with numbers lower than 10
if [ $i -lt “10” ]; then
j=0${i}
else
j=${i}
fi
dd if=${fie} of=slices/${file}.slice${j} bs=1M count=${chunk} skip=${skip}
# Just to view the files with out own eyes
ls -s slices/${file}.slice${j}
if [ `ls -s slices/${file}.slice${j} | awk ‘{print $1}’` -lt “${size}” ]; then
echo “Done”
break
fi
let i=$i+1
let skip=$skip+$chunk
done

This will break the tar.gz file to a files with running numbers added to their names. It assumes that the number of slices would not exceed 99. You can extend the script to deal with three digits numbers. The sequence is important for later. Stay tuned 🙂

Ok, so we have a list of files with a numerical suffix, which, combined, include our data. We can test their combined integrity:

cd /tmp/slices
i=1
file=Data.tar.gz
for i in `ls`; do
cat ${file}.slice${i} >> ../Data1.tar.gz
done

This will allow us to compare /tmp/Data.tar.gz and /tmp/Data1.tar.gz. I tend to use md5sum for such tasks:

md5sum Data.tar.gz
d74ba284a454301d85149ec353e45bb7 Data.tar.gz
md5sum Data1.tar.gz
d74ba284a454301d85149ec353e45bb7 Data1.tar.gz

They are similar. Great. We can remove Data1.tar.gz. We don’t need it anymore.

To recover the contents of the slices, without actually wasting space by combining them before extracting their contents (which requires time, and disk space), we can run a script such as this:

cd /tmp/slices
file=Data.tar.gz
(for i in `ls ${file}.slice*`; do
cat $i
done ) | tar xzvf –

This will extract the contents of the joined archive to the current directory.

This is all for today. Happy moving of data 🙂

I have had time this weekend. NabRSS version 0.3 has been released

Sunday, July 15th, 2007

Update to the version posted here, the new shiny version includes several interesting additions. From the ChangeLog:

NabRSS Version 0.3 Jul 15th 07:
– DB – Extended feed_list table to include selected voice
– DB – Inserted all voices into DB
– add_url.php – Enhanced to allow for (optional) selection of voice to an RSS feed
– add_url.php – Now checks input for duplicates and prevents them
– list.php – Enhanced to support voice definitions
– get.php – Changed stracture to somewhat more natural (to me, that is)
– get.php – Added support for voice selection. If not selected, randomize the voice
– get.php – Updated the DB scheme comment
– header.php – fixed problem with comments. They were incorrect, and now they are ok.

Also easy (or at least, easier) upgrade from past versions.

Available here: nabrss-0.3.tar.gz

Enjoy!

NabRSS updated to version 0.21

Saturday, July 14th, 2007

Again, on a short notice, the package introduced here and updated here has been updated.

From the ChangeLog:

NabRSS Version 0.21 Jul 14th 07:
– Started moving functions into functions.php
– Added support for RSS feeds with login
– Hiding passwords for login-based RSS feeds
– Minor updates to README.txt

Enjoy

NabRSS updated to version 0.2

Saturday, July 14th, 2007

NabRSS has been updated from the version posted here.

NabRSS is a set of PHP scripts meant to assist anyone with a PHP/MySQL/Apache server available to gather feeds from RSS sources and send them via Nabaztag API to his/her Nabaztag.

Available here: nabrss-0.2.tar.gz

Quoting from the changelog:

NabRSS Version 0.2 Jul 14th 07:
– get.php: Fixed an incorrect condition in function debug_set
– get.php: Added function set_ear_pos to randomize ear position on every post
– get.php: Added function set_voice to randomize each RSS post voice
– list.php: Added a “Reset” button to allow reseting the last post saved in the date field inside the DB

Todo (short range):
– DB: Extend the scheme and move voices into DB as a seperated table
– add_url.php: Select a voice for each RSS feed. Possiblity to select “random”
– list.php: Show selected voice for each RSS feed
– get.php: Insert a function to deal with voices per post