Posts Tagged ‘redhat4’

Persistent raw devices for Oracle RAC with iSCSI

Saturday, December 6th, 2008

If you’re into Oracle RAC over iSCSI, you should be rather content – this configuration is a simple and supported. However, working with some iSCSI target devices, order and naming is not consistent between both Oracle nodes.

The simple solutions are by using OCFS2 labels, or by using ASM, however, if you decide to place your voting disks and cluster registry disks on raw devices, you are to face a problem.


There are few guides, but the simple method is this:

  1. Configure mapping in your iSCSI target device
  2. Start the iscsid and iscsi services on your Linux
    • service iscsi start
    • service iscsid start
    • chkconfig iscsi on
    • chkconfig iscsid on
  3. Run “iscsiadm -m discovery -t st -p target-IP
  4. Run “iscsiadm -m node -L all
  5. Edit /etc/iscsi/send_targets and add to it the IP address of the target for automatic login on restart

You need to configure partitioning according to the requirements.

If you are to setup OCFS2 volumes for the voting and for the cluster registry, there should not be a problem as long as you use labels, however, if you require raw volumes, you need to change udev to create your raw devices for you.

On a system with persistent disk naming, follow this process, however, on a system with changing disk names (every reboot names are different), the process can become a bit more complex.

First, detect your scsi_id for each device. While names might change upon reboots, scsi_ids do not.

scsi_id -g -u -s /block/sdc

Replace sda with the device name you are looking for. Notice that /block/sda is a reference to /sys/block/sdc

Use the scsi_id generated by that to create the raw devices. Edit /etc/udev/rules.d/50-udev.rules and find line 298. Add a line below with the following contents:

KERNEL==”sd*[0-9]”, ENV{ID_SERIAL}==”14f70656e66696c000000000004000000010f00000e000000″, SYMLINK+=”disk/by-id/$env{ID_BUS}-$env{ID_SERIAL}-part%n” ACTION==”add” RUN+=”/bin/raw /dev/raw/raw%n %N”

Things to notice:

  1. The ENV{ID_SERIAL} is the same scsi_id obtained earlier
  2. This line will create a raw device in the name of raw and number in /dev/raw for each partition
  3. If you want to differtiate between two (or more) disks, change the name from raw to an aduqate name, like “crsa”, “crsb”, etc, for example:

KERNEL==”sd*[0-9]”, ENV{ID_SERIAL}==”14f70656e66696c000000000005000000010f00000e000000″, SYMLINK+=”disk/by-id/$env{ID_BUS}-$env{ID_SERIAL}-part%n” ACTION==”add” RUN+=”/bin/raw /dev/raw/crs%n %N”

Following these changes, run “udevtrigger” to reload the rules. Be advised that “udevtrigger” might reset network connection.

Oracle RAC with EMC iSCSI Storage Panics

Tuesday, October 14th, 2008

I have had a system panicking when running the mentioned below configuration:

  • RedHat RHEL 4 Update 6 (4.6) 64bit (x86_64)
  • Dell PowerEdge servers
  • Oracle RAC 11g with Clusterware 11g
  • EMC iSCSI storage
  • EMC PowerPate
  • Vote and Registry LUNs are accessible as raw devices
  • Data files are accessible through ASM with libASM

During reboots or shutdowns, the system used to panic almost before the actual power cycle. Unfortunately, I do not have a screen capture of the panic…

Tracing the problem, it seems that iSCSI, PowerIscsi (EMC PowerPath for iSCSI) and networking services are being brought down before “killall” service stops the CRS.

The service file was never to be executed with a “stop” flag by the start-stop of services, as it never left a lock file (for example, in /var/lock/subsys), and thus, its existence in /etc/rc.d/rc6.d and /etc/rc.d/rc0.d is merely a fake.

I have solved it by changing /etc/init.d/ script a bit:

  • On “Start” action, touch a file called /var/lock/subsys/
  • On “Stop” action, remove a file called /var/lock/subsys/

Also, although I’m not sure about its necessity, I have changed script SYSV execution order in /etc/rc.d/rc0.d and /etc/rc.d/rc6.d from wherever it was (K96 in one case and K76 on another) to K01, so it would be executed with the “stop” parameter early during shutdown or reboot cycle.

It solved the problem, although future upgrades to Oracle ClusterWare will require being aware of this change.

Bonding in RedHat RHEL4

Sunday, July 20th, 2008

This is a rather common knowledge now that on RHEL4 you need to state inside /etc/modprobe.conf the following line, when you want more than one bonding interfaces:

options bonding max_bonds=2

Then you attempt to use a trick to address different bonding devices with their name (aka, bond0 and bond1, and maybe bond2, etc), using an option as follows in your /etc/modprobe.conf:

options bond1 -o bond1 miimon=100

It works perfectly fine, until you try to set different parameters for your bonding devices, such as in this example (again, from /etc/modprobe.conf):

options bond0 -o bond0 mode=1 miimon=100
options bond1 -o bond1 mode=1 arp_validate=1 arp_ip_target= arp_interval=1000

These different options will not work. The 2nd (and all next) bonding devices will use bond0’s settings.

A note about this can be found in /usr/share/doc/kernel-doc-2.6.9/Documentation/networking/bonding.txt (requires the package “kernel-doc”):

NOTE: It has been observed that some Red Hat supplied kernels are apparently unable to rename modules at load time (the “-o bond1” part).  Attempts to pass that option to modprobe will produce an “Operation not permitted” error.  This has been reported on some Fedora Core kernels, and has been seen on RHEL 4 as well.  On kernels exhibiting this problem, it will be impossible to configure multiple bonds with differing parameters.

Without the ability to rename modules, we are unable to set, through /etc/modprobe.conf any bond-specific options.

An option which cannot be found in /usr/share/doc/initscripts-7.93.31.EL/sysconfig.txt (part of the “initscripts” package) is to remove any bond-specific parameters from /etc/modprobe.conf and to add to /etc/sysconfig/network-scripts/ifcfg-bondX a line as follows:

BONDING_MODULE_OPTS=’miimon=100 primary=eth0′

Here you can state your bonding options, and when you will restart your networking (provided you actually unload the “bonding” module during that process), your bonds will behave as you expect them to.

A small thing I need to confirm yet is the behavior of the bonding device if settings are changed without unloading the “bonding” module between the ifdown and ifup commands.

Vlan Tagging with bonding network interface on RHEL4

Saturday, May 17th, 2008

This is not a simple task, as there are few things which should actually happen for it to work.

First – the switch port should support vlan tagging (of course, right?)

I have used vlan2 for “external” network, and vlan3 for “internal” network.

My configuration looks like this:











I hope it helps anyone who is into vlan tagging over bonding interfaces.

RHEL4 tends to change network interfaces names

Friday, July 6th, 2007

RHEL4 tends to change the names of network cards when there are more than one. If you had a NIC called eth0 during install time, it doesn’t mean that it will maintain that name after the first reboot. It could switch names with its friend, and be called now eth1, while the previous eth1 name is now eth0.

A solution using udev was posted in HPs forums, and can be reached directly through here. I will quote it:

Device persistence can also be enabled to ensure that the NICs identifying themselves as eth1, eth2, etc… always remain on the same hardware ports in case of a failure of a single NIC port. You don’t want your eth names to shift.

Upgrading to udev-095 from udev-039 that ships with RHEL4 is the smoothest solution, but that wasn’t an option for me. Using names other than eth0 – eth3 also wasn’t an option for me. Here is what we ended up using to get around udev-039’s inability to re-use eth0-ethx names.

Create a udev rule using YOUR MACS
Create an /etc/mactab file using YOUR MACS
Modify /etc/init.d/network to run nameif

KERNEL=”eth*”, SYSFS{address}=”00:0b:cd:69:c3:66″, NAME=”NIC1″
KERNEL=”eth*”, SYSFS{address}=”00:0b:cd:69:c3:65″, NAME=”NIC2″
KERNEL=”eth*”, SYSFS{address}=”00:11:0a:17:66:26″, NAME=”NIC3″
KERNEL=”eth*”, SYSFS{address}=”00:11:0a:17:66:27″, NAME=”NIC4″

eth0 00:0b:cd:69:c3:66
eth1 00:0b:cd:69:c3:65
eth2 00:11:0a:17:66:26
eth3 00:11:0a:17:66:27

(add right after
# Check that networking is up.
[ “${NETWORKING}” = “no” ] && exit 0

# RDD: add ‘nameif’ usage; uses /etc/mactab
nameif || echo “nameif: reports error”