Posts Tagged ‘backup’

Splitting archive and combining later on the fly

Wednesday, July 18th, 2007

Many of us use tar (many times with gzip or bzip2) for archiving purposes. When performing such an action, a large file, usually, too large, remains. To extract from it, or to split it becomes an effort.

This post will show an example of a small script to split an archive and later on, to directly extract the data out of the slices.

Let’s assume we have a directory called ./Data . To archive it using tar+gzip, we can perform the following action:

tar czf /tmp/Data.tar.gz Data

For verbose display (although it’s could slow down things a bit), add the flag ‘v’.

Now we have a file called /tmp/Data.tar.gz

Lets split it to slices sized 10 MB each:

cd /tmp
mkdir slices
i=1 # Our counter
skip=0 # This is the offset. Will be used later
chunk=10 # Slice size in MB
let size=$chunk * 1024 # And in kbytes
file=Data.tar.gz # Name of the tar.gz file we slice
while true ; do
# Deal with numbers lower than 10
if [ $i -lt “10” ]; then
dd if=${fie} of=slices/${file}.slice${j} bs=1M count=${chunk} skip=${skip}
# Just to view the files with out own eyes
ls -s slices/${file}.slice${j}
if [ `ls -s slices/${file}.slice${j} | awk ‘{print $1}’` -lt “${size}” ]; then
echo “Done”
let i=$i+1
let skip=$skip+$chunk

This will break the tar.gz file to a files with running numbers added to their names. It assumes that the number of slices would not exceed 99. You can extend the script to deal with three digits numbers. The sequence is important for later. Stay tuned 🙂

Ok, so we have a list of files with a numerical suffix, which, combined, include our data. We can test their combined integrity:

cd /tmp/slices
for i in `ls`; do
cat ${file}.slice${i} >> ../Data1.tar.gz

This will allow us to compare /tmp/Data.tar.gz and /tmp/Data1.tar.gz. I tend to use md5sum for such tasks:

md5sum Data.tar.gz
d74ba284a454301d85149ec353e45bb7 Data.tar.gz
md5sum Data1.tar.gz
d74ba284a454301d85149ec353e45bb7 Data1.tar.gz

They are similar. Great. We can remove Data1.tar.gz. We don’t need it anymore.

To recover the contents of the slices, without actually wasting space by combining them before extracting their contents (which requires time, and disk space), we can run a script such as this:

cd /tmp/slices
(for i in `ls ${file}.slice*`; do
cat $i
done ) | tar xzvf –

This will extract the contents of the joined archive to the current directory.

This is all for today. Happy moving of data 🙂

HP-UX, Oracle 8i, DataProtector,

Tuesday, July 11th, 2006

Imagine Omniback 4.x to Omniback 5.5 upgrade on HP-UX. Imagine you assume all existing backup procedure (and you were told there is license only for disk agent) is based on filesystem backup. Assume you know there’s Oracle installed on this server, but no relevant agent (again, filesystem only, no DB agents…)

You are cautious. You move the current /opt/omni to /opt/omni.old directory. You hash the line containing the relevant entry in /etc/inetd.conf. You are prepared.

You install the newer version by running the installer script with the flag -install da, so you would install only Disk Agent (after all, this is nothing but a client to this whole backup procedure).

You check everything, and it all seems to work correctly as far as you care or know. Suddenly, someone notices that Oracle Listener (TNS) does not listen anymore. Trying to bring it back up results in a message which seems like this:

/usr/lib/pa20_64/ Unable to find library ‘’.

It doesn’t look good. You are in a little crisis. Restart to the Oracle Engine itself results in a shutdown, but it never starts back again. It doesn’t look good.

It appears that there was an Oracle Agent for Omniback installed there previously, and that you removed it uncleanly by your Disk-Agent-Only upgrade.

The solution could have been to install the Oracle Agent. It can be also related to recreating the required links, say from /opt/omni.old/lib/ or from $ORACLE_HOME/lib64/ (if there were any…) to $ORACLE_HOME/lib64/, the first is based on the older Omniback, the later is based on the assumption a backup was made.

The quickest solution was to install the newer agent, with the Oracle8i agent (called “oracle8”) and link: “ln -s /opt/omni/lib/ ~oracle/lib64/” (assuming we’re running Oracle 8i on 64bit HP-UX PA-RISC).

I never quite remember it – extracting a specific file from tar archive

Monday, June 26th, 2006

I always forget, and this blog is meant to help me remember.

Found in Tar’s manual, are these simple directives:

Have a file called file.tar, do:

tar -xf file.tar full/path/without/trailing/slashes

It can be a file or directory. You can have a list of files, such:

tar -xf file.tar home/me home/you home/us/important-file

Should do the trick.

A bug in restore in Centos4.1 and probably RHEL 4 update 1

Sunday, February 26th, 2006

I’ve been to Hostopia today. The land of hosting servers. I’ve had an emergency job on one Linux server, due to a mistake I’ve made. It appears that the performance hindrance of using raid0 instead of raid1 (Centos/RH default raid setup is raid0 and not raid1, which led me to this mistake) for the root partition is terrible.

I tend to setup servers in the following way:

Small (100MB) raid1 partition (/dev/sda1 and /dev/sdb1, usually) for /boot.

Two separated partitions for swap (/dev/sda2 and /dev/sdb2), each just half the required total swap.

One large raid1 (/dev/sda3 and /dev/sdb3) containing LVM, which, in turn, holds the “/” and the rest of the data partitions, if required.

In this specific case, I’ve made a mistake and was not aware of it on time. I’ve setup the large LVM over a stripe (raid0) by mistake. I’ve had degraded performance on the server, and all disk access were slow. Very slow. Since it is impossible to break such a raid array without loosing data, I’ve had to backup the data currently there, and make sure I would be able to restore it. It’s an old habit of mine to use dump and restore. Both ends of the procedure worked so far perfectly, on all *nix operating systems I’ve had experience with. I’ve dumped the data, using one of the swap partitions as a container (formatted as ext3, of course), and was ready to continue.

I’ve reached the server farm, where all hosting servers stood in long rows (I’m so sorry I did not take a picture. Some of those so called “servers” had color leds in their fans!), and got busy on this specific server. Had to backup all from the start, as it failed to complete before (and this time, I’ve done so to my laptop through the 2nd NIC), and then I’ve booted into rescue mode, destroyed the LVM, destroyed the raid (md device), and recreated them. It went fine, except that restore failed to work. The claim was “. is not the root” or something similar. Checking restore via my laptop worked fine, but the server itself failed to work. Eventually, after long waste of time, I’ve installed minimal Centos4.1 setup on the server, and tried to restore through overwrite from within a chroot environment. It failed as well. Same error message. I’ve suddenly decided to check if I’ve had an update to the dump package, and there was. Installing it solved the issue. I was able to restore the volume (using the “u” flag, to overwrite files), and all was fine.

I’ve wasted over an hour over this stupid bug. Pity.

Keeping the static copy of the up-to-date restore binary. Now I will not have these problems again. I hope 🙂