Posts Tagged ‘real-time replica’

Poor Man’s DRP + Snapshots – Linux only

Friday, October 6th, 2006

When you own a data storage, one of your major considerations is how to backup your data. Several solutions exist to answer this question.

When your data grows to a certain size, you encounter an additional issues – How to backup the data with minimum performance impact.

It is quite obvious that backup devices has a specific speed and performance. It is quite obvious that is you have more data than you can stream into your tape deviced during night, your backup would probably continue during working hours.

Several solutions exist to deal with this problem, amongst you can find the solution of faster backup tapes, broader bandwidth between your storage container and your backup devices. The issue I will demonstrate has to do with a third option – create a real-time replica on another server, and backup the replica only.

When it comes to Linux, I’ve always felt that the backup/restore software companies were rather slow to supply solutions fit for Linux, especailly compared to the widening usage of Linux-based systems in the market.

One of the more intriging solutions which grew in the OpenSource community is called DRBD – Distributed Redundant Block Device. It allows the creation of a logical block device which overlayes two physical block devices – one local and one remotely accessible via network. It can be easilly described as network Raid-1 solution.

The wonders of real-time volume replica between two servers should not be discussed here. The advantages are well known, as are the disadvantages, of which the largest one is the heavy performance toll on such a system.

The wonders of snapshots are also well known. NetApp gains its main capital based on their sophisticated snapshot technology (WAFL, etc). Other storage vendors have added the abilities to take snapshots with higher or lower effeciancy, however, one of the newer players in this under-the-spotlight area is the OpenSource LVM2 for Linux, with its snapshot capabilities. Although still not perfect, it does show a promise I will soon demonstrate, combined with DRBD, described above.

The combined wonders of volume replication together with scheduled snapshots can offer the ability to execute backup of consistant snapshot data, the ability to get back to a desired volume’s point-in-time and the power to reduce the load of backing up on mission-critical datacenters. All these, at the price of internet connection which will allow you to download the latest DRBD software.

I have tested it on a home-made setup – Two Virtual Linux server running on a single VMware-Server machine.

The host is Pentium4 1.8GHz, with 1GB RDRAM, and three IDE harddrives, running Centos 4.4

The guests are two Centos 4.4 machines, with 160MB RAM each, two virtual NICs – one public and one private, minimal installation, and Dag Wieers‘ YUM repositories added to them.

The guest will be called DRBD-test1 and DRBD-test2. The first will act as the mission-critical server, and the second will be the replica (target) server.

Both guests were updated to the latest updates available at this time. Both are running kernel version 2.6.9-42.0.2.EL, DRBD version 0.7.21-1.c4, and kernel-module-drbd-2.6.9-42.EL-0.7.21-1.c4

Installing the kernel-module package put the drbd.ko modules in /lib/modules/2.6.9-42.EL instead of my running kernel (2.6.9-42.0.2.EL), so after verifying that the modules were able to load into my running kernel, I have moved them to the kernel/drivers/block directory inside the modules tree, and run ‘depmod -a‘.

I decided to use a consistant configuraion, and defined the storage to replicate in a similar manner:

On /dev/sdb I’ve created PV (pvcreate /dev/sdb). Assigned this PV to VG named vg00, and created two LVs on it: meta (256MB) and source (2GB) on the guest acting as the mission critical server, and meta (256MB) and target (2GB) on the one acting as replica.

I have created the device /dev/drbd0, per DRBD’s Howto, built the configuration file drbd.conf, and loaded the modules.

Forced the Source guest to act as the primary, and replication began.

When replication has finished, I have created a snapshot of the LV target and mounted it correctly: "lvcreate -L 200M -s -n snap /dev/vg00/target && mount /dev/vg00/snap /mnt"

I was able to access the data inside the volume, without changing the Primary/Secondary order of the servers. I have created a script which used DD to stress the I/O of the DRBD volume on the source server, and created a script which took scheduled (every minute) snapshots of the target volume. I have learned the following:

1. It works, but

2. The size limitaion forced on snapshot (200MB in my case) should never be filled up. When running DD on the source volume (creating 50MB empty files), the space consumed by the snapshots increases, and if/when a snapshot exceeds the 100% utilization, it is inaccessible anymore. To view the current usage of a snapshot, run "lvdisplay /dev/vg00/snap" (in my example).

During that evaluation, one of my virtual server crushed, due to LVM2 snapshot problem. LVM2 is not yet perfect on RH based systems…

Performance on another time. I wan’t too happy with it, however on this experiment my goal was to find out if such a setup can be built rather than to measure the performance impact.

Generally speaking – I was rather happy with the results – It showed that this setup can actually work. It proved to me again that OSS innovations elevate Linux to the enterprise.

Now that I know that such a setup can be done, all left to do is to fine-tune it to minimum performance impact, and test again to see if it can actually be a well-suited solution for the questions I’ve started with.

Microsoft Exchange, data replication

Sunday, November 27th, 2005

Here’s a little issue. If you were to replicate MS Exchange DB from one machine to another, how/what would you have done?

The scenario goes as follows: You have your own domain, and you use, for your own core services AD and MS Exchange for the whole organization. While AD supports some built-in replication, so you could hold a second
site and join a Windows server machine into this domain, and assuming you have some sort of link, you would have a replica of your AD on the secondary site. In case something happened to your primary site, your secondary site would have a complete and correct replica of your primary site.

Exchange doesn’t act that nice. Replicating MS Exchange is not simple, even assuming you have a method of transferring the data from one site to the other.

Two methods I can think of – The first would be to somehow proxy the information getting into it, and transport it (maybe rewrite the headers to point to a different recipient) to another site as well. In this case, you are actually agnostic to the store. You don’t care if you maintain your mailboxes on MS Exchange, Windows, Unix, or whatever system. You have two copies, and you’re fine with it. You should find some logical method to make your primary site’s mail server transport all mail, even internal mail, through this proxy mechanism.

The second method is a bit more interesting. If we could have had a real-time replica of the primary site’s Exchange DB, we would have been able to “mount” it on the secondary site. It is not trivial, but it can be done using some
not-so-common software or hardware based solutions. However, the secondary site would require few things, per this list:

1) Similarity:

I would strive to similarity between both sites. MS recommend similarity (when dealing with cold-recover), per this site, especially with the Exchange patch level, and somewhat with Windows patch level. The more similar, the better chances we’ll have. So:

a. Similarity in patch level.

b. Similarity in Storage group settings, especially transaction logs’ location.

c. Similarity in the structure of each store under each storage group

d. Similarity in the absolute path of each store’s edb and stm location between source and target

2) Recoverability:

To make things recoverable, you should make sure each store has, under the “Database” tab, under properties, the option which will allow restore to overwrite the DB.

3) Flexibility:

You need to find some flexible script which will be able to change, as fast as possible, the pointers in your AD, to point to the new Exchange server, in the users’ Exchange attributes. I have such a script, but I cannot disclose it here.

Armed with these three, you can copy, transfer, replicate, or whatever, your DBs from one location to the other. Make sure you replicate the logs as well, else it’ll get messy, and will require tons of time for DB recovery.

To recover, you should only mount the stores. Assuming you have followed the prerequisites, you would be able to mount the stores in no time. Otherwise, you would need to run eseutil on the stores. It might get messy.

Afterwards, only one thing to do – mass change the attributes of the users in question to point to the alternate Exchange server, and the alternate Exchange store. Should work like a charm.