Posts Tagged ‘redhat4’

Installing RHEL4 on HP DL140 G3 with the embedded RAID enabled

Friday, July 6th, 2007

While DL140 G3 is quite a new piece of hardware, RHEL4, even with the later updates, is rather old.

When you decide to install RHEL4 on a DL140 G3 server, my first recommendation is this: if you decide to use the embedded SATA-II RAID controller – don’t. This is a driver-based RAID, much like the past win-modem devices. Some major parts of its operations are based on calculations done through the driver, directly on the host CPU. It has no advantages comparing to software RAID, and its major disadvantage is its immobile state – unlike software based mirror, this array cannot “migrate” to another server, unless this server is of the same type of hardware. Not sure about it, but it might also require close enough version of firmware as well.

It happened that your boss believes in win-RAID devices (despite the note above), or for some other reason you decide to use this win-RAID, here are the steps to install the system.

1. Download the latest driver disk image for RHEL4 from HP site.

2. If you have the privilage of having an NFS server, uncompress the image and put it on it, where it can be accessed through network.

3. Test that you can mount it from another server. Verify you can reach the image file. Debugging incorrect NFS issues can waste lots of time.

4. If you don’t have the privilege, I hope you have a USB floppy. Put the image on a floppy disk:

gzip -dc /path/to/compressed/image/file.gz | dd of=/dev/fd0

(/dev/fd0 assuming this is not /dev/sdX, as it tends to be with USB floppies)

5. Boot the server with the first RHEL4 CD in the drive, or with PXE, or whatever is your favorite method. In the initial boot prompt type:

linux text dd=nfs:server:/path/to/nfs/disk/disk_image.dd

This assuming that the name of the file (including its full path) is /path/to/nfs/disk/disk_image.dd. For floppy users, type dd=floppy instead.

6. RHEL will boot, loading “ahci” module (which is bad) during its startup. It will ask you to select through which network card the system is to reach the NFS server. I assume you have a working DHCP in your site.

7. As soon as you are able to use the virtual terminal (Ctrl+F2) switch to it.

8. Run the following commands:

cd /tmp
mkdir temp
cd temp
gzip -S .cgz -dc /tmp/ramfs/DD-0/modules.czf | cpio -id

cd to the modules directory, and look at the modules. Know which is the module which fits your running kernel. You can do this by using ‘uname’ command.

9. Run the following commands

rmmod ahci
rmmod adpahci
insmod KERNEL_VER/ARCH/adpahci.ko

Replace KERNEL_VER with your running(!!!) single-CPU kernel version, and replace ARCH with your architecture, either i386 or x86_64

10. Using Ctrl+F1 return to your running installer. Continue installation until the end but do not reboot the system when done.

11. When installation is done, before the reboot, return to the virtual console using Ctrl+F2.

12. Run the following commands to prepare your system for a happy reboot:

cp /tmp/temp/KERNEL_VER/ARCH/adpachi.ko /mnt/sysimage/lib/modules/KERNEL_VER/kernel/drivers/scsi/

cp /tmp/temp/KERNEL_VERsmp/ARCH/adpachi.ko /mnt/sysimage/lib/modules/KERNEL_VERsmp/kernel/drivers/scsi

Notice that we’ve copied both the single CPU (UP) and the SMP versions.

13. Edit modprobe.conf of the system-to-be and remove the line containing “alias scsi_hostadapter ahci” from the file.

14. Chroot into the system-to-be, and build your initrd:

chroot /mnt/sysimage
cd /boot
mv initrd-KERNEL_VER.img initrd-KERNEL_VER.img.orig
mv initrd-KERNEL_VERsmp.img initrd-KERNEL_VERsmp.img
mkinitrd /boot/initrd-KERNEL_VER.img KERNEL_VER
mkinitrd /boot/initrd-KERNEL_VERsmp.img KERNEL_VERsmp
exit

If things went fine so far, you are now ready to reboot. Use Ctrl+F1 to return to the installation (anaconda) console, and reboot the system.

Notes:

1. You need to download the “Driver Diskette” from HP site.

2. The latest Driver Diskette will support only Update3 and Update4 based systems. At this time, Update5 has no modules by HP yet. You can compile your own, but this is not in our scope.

3. Avoid using floppies at all cost.

4. Do not install the system in full GUI mode. In the model I have installed the VGA (Matrox device) had a bug and did not allow to reach the virtual text consoles. It disconnected the VGA. If you use GUI installation, you will be required to reboot the system into rescue mode and do steps 7 to 14 then.

5. Underlining the word smp is meant to help you not forget it. This is the more important one.

6. On the system itself, using Xorg, I was able to reach max resolution of 640×480 even with the display drivers supplied by HP. I was able to reach 1024×768 only when using 256 colors.

RedHat / Centos Kickstart tweaks

Sunday, July 1st, 2007

Kickstart is a great method of hands-free installation of RHEL/Centos (and other derived systems). Its power is in its easy interface and rather powerful %post scripting directives. Its weakness is in its lack of flexibility where it comes to package selection and various custom actions.

On some cases, companies use web interface (usually home-made) which builds kickstart config files on-demand. On some cases, the administrator is required to build several kickstart config files for pre-anticipated setups.

I was looking for something which will give me the power to maintain a fixed configuration on one hand, and will allow me some tweaks and variants, when I want them. I could have used the %post scripting sections, but this gets quite complicated, especially when you want to add only one package (but with its dependencies), or you want to force full update of the system before it goes online, or even select its hostname, assuming it is not yet defined in the DNS.

I base my system on a simple DHCP/BootP + tftp server which answers to all bootp requests and offers a simple menu (just type a number and press on Enter). The original schema was quite simple: type 4 for Centos4.3, and then add -min if you wanted it to use a kickstart file with a minimum configuration. Then I wanted to add the option to update the system in an early stage, so I have added -update, which would have looked in the menu like “4-min-update” option. Quite readable, however, it generated lots of work when maintaining the pxelinux.cfg/default file and the ks themselves. Too many variations tend to require lots of care.

Adding parameters to the boot menu is possible, and would result in them existing in /proc/cmdline for later parsing.

I have decided to parse a set of predefined parameters supplied during boot time, and to change the kickstart config file according to them. It actually works quite well. This is a less-sophisticated and more of a stand-alone system compared to this system. Also, it doesn’t require me to alter the system’s boot process.

This is my ks.cfg file, which includes the flexibility additions:

# Kickstart file generated by Ez-Aton

install
nfs –server=install-server –dir=/mnt/samba/Centos
lang en_US.UTF-8
langsupport –default=en_US.UTF-8 en_US.UTF-8
keyboard us
skipx
network –device eth0 –bootproto dhcp
rootpw –iscrypted RpUKzjDc9k2gU
firewall –disabled
selinux –disabled
authconfig –enableshadow –enablemd5
timezone Asia/Jerusalem
bootloader –location=mbr

%packages
e2fsprogs
grub
lvm2
kernel
net-snmp
net-snmp-utils
kernel-devel
kernel-smp-devel
gcc

%pre
# By Ez-Aton http://www.tournament.org.il/run
for i in `cat /proc/cmdline`; do
echo $i >> /tmp/vars.tmp
done
grep “=” /tmp/vars.tmp > /tmp/vars
KS=/tmp/ks.cfg
update=””
name=””
pkg=””
. /tmp/vars
if [ ! -z “$update” ]; then
echo “yum update -y” >> $KS
fi
if [ ! -z “$name” ]; then
value=”dhcp –hostname $name”
cat $KS | sed s/dhcp/”$value”/ > $KS.tmp
cat $KS.tmp > $KS
fi
if [ ! -z “$pkg” ]; then
pkg_line=`grep -n ^%packages $KS | cut -f 1 -d :`
max_line=`wc -l $KS | awk ‘{print $1}’`
head -n $pkg_line $KS > $KS.tmp
for i in `echo $pkg | sed s/,/ /g`; do
echo $i >> $KS.tmp
done
let tail_line=$max_line-$pkg_line
tail -n $tail_line $KS >> $KS.tmp
cat $KS.tmp > $KS
fi

%post

So, as you can see, I take the following parameters:

update=yes (it can be update=anything)

name=hostname (in case it cannot be retrieved from the DHCP server)

pkg=pkg1,pkg2,{pkg3,…} (To add specific packages to the installation)

It was tested to work on Centos4.3 system, and will probably work on RHEL and Centos versions 4.x all along. I didn’t test it on RHEL5/Centos5 yet.

If you use the script, please leave my name and blog URL in it. Also, if you modify it for your needs, I would be glad to get back the modifications you have made, to include them.

Enjoy.

RHEL 4 32bit on Tyan Thunder K8QE

Saturday, September 9th, 2006

It’s sort of a relationship between myself and this Tyan. Same server, changing demands. This time, we’ve had to install on this server RHEL4 32bit.

Net-installing RHEL4 Update 3 was impossible. See here, so we’ve installed it from CDs (didn’t have PCI-X network card handy).

When booting the server, it showed us only a single CPU. Since this machine has four dual-core CPUs, this number – one – was rather unexpected.

We’ve tried booting the system with additional parameters. We’ve tried, per RH tips, to allow “apic=bigsmp” parameter, however, when booting the server with these parameters, the server entered into reset loop. We’ve tried playing with other parameters, such as “apic acpi=noirq iommu=memaper=3 swiotlb=65536” (obtained for Google cache for a forum site which I cannot find again now), but for no avail. Finally, we’ve decided to upgrade the kernel to the one in Update4.

Doing so had special effects. When using it without any special parameters, the server entered a reboot loop, however, when we’ve added the parameter “apic=bigsmp” the server booted correcly and showed all eight CPUs.

Update – Netboot on RHEL x86 (32bit) with Broadcom (tg3) – no network

Sunday, August 27th, 2006

In my post just below, I have defined a set of tests to verify the possible cause of the tg3 problem. It had nothing to do with autoneg, and was fixed in RHEL 4 Update 4. That 32bit installer works correctly.

One last thing to test – rebuild the installer initrd, and replace tg3 module by one built from source (for example, HP’s tg3 drivers from the Proliant Support Pack) for this kernel. I wonder if it will work.

RedHat Cluster, and some more

Sunday, February 12th, 2006

It’s been a long while since I’ve written. I get to have, once a while, a period of time dedicated for laziness. I’ve had just one of these for the last few weeks, in which I’ve been almost completely idle. Usually, waking up from such idle time is a time dedicated to self studies and hard work, so I don’t fight my idle periods too hard. This time, I’ve had the pleasure of testing and playing, for personal reasons, both with VMWare GSX, in a “Cluster-In-a-Box” setup, based on recommendation regarding MSCS, altered for Linux (and later, Veritas Cluster Service) and both with RedHat Cluster Server, with the notion of playing with RedHat’s GFS, but, regrettably, without the last.

First, VMware. In their latest rivalty with Microsoft over the issue of Virtualization of servers and desktops, MS has gained an advantage lately. Due to the lower prices of “Virtual Server 2005”, comparing with “VMware GSX Server”, and due to their excellent marketing system (from which we should all learn, if I may say!), Not a few servers and virtual server farms, especially the ones running Windows/Windows setups, had moved to this MS solution, which is as capable as VMware GSX Server. Judging by the history of such rivalries, MS would have won. They always have. However, VMware, in an excellent move, has announced that the next generation of their GSX, simply called “Server”, would be for free. Free for everyone. In this they probably mean to invest more in their more robust ESX server, and give the GSX as a taste of their abilities. While MS do not have any more advanced product than their Virtual Server, it could mean a death blow to their effort in this direction. It could even mean they will just give away their product! While this will happen, we, the customers, will earn a selection of free, advanced and reliable products designed for virtualization. Could it be any better than that?

One more advantage of this “Virtualization for the People” is that community based virtual images, of even the most complicated to install setups can and would be widely available. Meaning to shorten installation time, and allow for a quick working system for everyone. It will require, however, better knowledge and understanding of the products themselves, as merely installing them will not be enough. To survive the future market, you won’t be able to just sell an installation of a product, but should be able to support an out-of-the-box setup of it. That’s for the freelances, and the partially freelances of us…

So, I’ve reinstalled my GSX, and started playing with it. The original goal was to actually run a working setup of RHEL, VCS and Oracle 10g. Unfortunately, VCS supports only RH3 (update 2?), and not RH4, which was a shame. At that point, I’ve considered using RH Cluster Server for the task at hand. It grew to the task of learning this cluster server, and nothing more, which I did, and I can and would share my concepts about it here.

First – Names – I’ve had the pleasure of working with numerous cluster solutions. I’m thrilled each time I get to play with another cluster solution the naming conventions, and name changes vendors do, just to keep themselves unique. I hate it. So here’s a little explanation:
All clusters contain a group of resources (Resource Group, as most vendors call them). This group contains a set of resources, and in some cases, relations (order of startup, dependencies, etc). Each resource could be any single element required for an application. Example – Resource could be an IP address, which without you won’t be able to contact the application. Resource could be a disk device, containing the application’s data. It could be an application start/stop script, and it could be a sub-application – an application required for the whole group to be up, such as a DB for DB driven web server. The order you would ask them to start would be IP, disk, DB, web server (in our case). You’d ask the IP to be brought up first because some of the cluster servers can trick an IP based clients into some delay, so the client hardly feels the short downtime of application failover. But this is for later. So, in a resource group, we have resources. If we can separate resources into different groups, if they have no required dependency between them, it is always better to do so. In our previous example, lets say our web server uses the DB, but it contacts it using IP address, or using hostname. In this case, we won’t need the DB to run on the same physical machine the web server is using, and in such a case, assuming the physical disk holding the DB and the one holding the rest of the web application are not the same disk, we could separate them.

The idea, if I can try to sum it up, is to split your application to the smallest self-maintained structures. Each structure will be called resource group, and each component in this structure is a resource. On some cluster servers, one could group and set dependencies between resource groups, which allows for even more scalability, but that is not our case.

So we had resource groups containing resources. Each computer, a member in the cluster, is called a node. Now, let’s assume our cluster containing three nodes, but we want our application (our resource group) to be able to run on only two specific? In this case, we need to define, for our resource group, which nodes are to be associated with it. In RH Cluster Server, a thing called “Domain” is designed for it. This Domain containes a list of nodes. This Domain can be associated with Resource Group, and thus set priority of failover, and set the group of nodes allowed to deal with the resource group.

All clusters have a single point of error (unlike failure). The whole purpose of the cluster is to allow for non-cluster-aware application the high-availability you could expect for a (relatively) low price. We’re great – we know how to bring an application up, we know how to bring it down. We can assume when the other node(s) is/are down. We cannot be sure of it. We try. We demand few means of communication, so that one link failure won’t cause us to corrupt our shared volumes (by trying multiple access into them). We set a whole system of logic, a heartbit, just name it, to avoid, at almost all cost, a status of split-head – two cluster nodes believing they are the only ones up. You can guess what it means, right?

In RH, there is a heartbit, sure. However, it is based on bonding, in the event of more than one NIC, and not on separated infrastructures. It is a simple network-based HB, with nothing special about it. In case of loss of connection, it would have reset the inactive node, if it saw fit, using a mechanism they call “Fence”. A “Fence” is a system by which the cluster can *know* for sure (or almost for sure) a node has been down, or the cluster can physically take a node down (poweroff if needs), such as control of the UPS the node is connected to, or its power switch, or alternate monitoring infrastructure, such as the Fibre Channel Switch, etc. In such an event, the cluster can know for sure, or can assume, at least, that the hung node has been reset, or it can force it to reset, to release some hung application.

Naming – Resource group is called Service. Resource remains resource, but an application resource *must* be defined by an rc-like script, which accepts start/stop (/restart?). Nothing complicated to it, really. The service contains all required resources.

I was not happy with the cluster, if I can sum up my issues with it. Monitoring machines (nodes) it did correctly, but in the simple enough example I’ve chosen to setup, using apache as a resource (only afterwards I’ve noticed it to be the example RedHat used in their documentation) it failed miserably to take the correct action when an application failed (unlike a failure of a node). I’ve defined my “Service” to contain the following three items:

1) IP Address – Unique for my testing purposes.

2) Shared partition (in my case, and thanks to VMware, /dev/sdb1, mounted at /var/www/html)

3) The Apache application – “/etc/init.d/httpd”

All in all, it was brought up correctly, and switch-over went just fine, including in a case of correct and incorrect reset of the active/passive node, however, when I’ve killed my apache (killall httpd), the cluster detected failure in the application, but was helpless with it. I was unable to bring down the “Service”, as it failed to turn off Apache (duh!), so it did not release neither the IP address, nor the shared volume. In so doing, I’ve had to restart the service rgmanager on both nodes, after manual removal of the remains of the “Service”. I didn’t like it. I expect the cluster to notice failure in the application, which it did, but I expect it to either try to restart the application (/etc/init.d/httpd stop && /etc/init.d/httpd start) before it fails completely, or to set a flag saying it is down, remove the remains of the “Service” from the node in question (release the IP address and the shared storage), and try to bring it up on the other node(s). It did nothing of the likes. It just failed, completely, and required manual intervention.

I expect HA-Cluster to be able to react to an application or resource failure, and not just to a node failure. Since HA-Clusters are meant for the non-ideal world, a place where computers crash, where hardware failures occure, and where applications just die, while servers remain working, I expect the Cluster Server to be able to handle the full variety of problems, but maybe i was expecting too much. I believe it to be better in their future versions, and I believe it could have been done quite easily right now, as long as detection of the failed application occurred, which it has, but it’s not for me to define the cluster’s abilities. This cluster is not mature enough for real-life production sites, if and only because of its failure to react correctly to a resource failure, without demanding manual intervention. A year from now, I’ll probably recommend it as a cheap and reliable solution for most common HA-related tasks, but not today.

That leaves me with VCS and Oracle, which I’ll deal with in the future, wether I like it or not 🙂