Running Systems

There is a major confusion among DBAs regarding how to setup raw devices for Oracle RAC or Oracle Clusterware. This confusion is caused by the turn RedHat took in how to define raw devices.

Raw devices are actually a manifestation of character devices pointing to block devices. Character devices are non-buffered, so they act as FIFO, and have no OS cache, which is why Oracle likes them so much for Clusterware CRS and voting.

On other Unix types, commonly there are two invocations for each disk device – a block device (i.e /dev/dsk/c0d0t0s1) and a character device (i.e. /dev/rdsk/c0d0t0s1). This is not the case for Linux, and thus, a special “raw”, aka character, device is to be defined for each partition we want to participate in the cluster, either as CRS or voting disk.

On RHEL4, raw devices were setup easily using the simple and coherent file /etc/sysconfig/rawdevices, which included an internal example. On RHEL5 this is not the case, and customizing in a rather less documented method the udev subsystem is required.

Check out the source of this information, at this entry about raw devices. I will add it here, anyhow, with a slight explanation:

1. Add to /etc/udev/rules.d/60-raw.rules:

ACTION==”add”, KERNEL==”sdb1″, RUN+=”/bin/raw /dev/raw/raw1 %N”

2. To set permission (optional, but required for Oracle RAC!), create a new /etc/udev/rules.d/99-raw-perms.rules containing lines such as:

KERNEL==”raw[1-2]“, MODE=”0640″, GROUP=”oinstall”, OWNER=”oracle”

Notice this:

1. The raw-perms.rules file name has to begin with the number 99, which defines its order during rules apply, so that it will be used after all other rules take place. Using lower numbers might cause permissions to be incorrect.
2. The following permissions have to apply:

OCR Device(s): root:oinstall , mode 0640

Voting device(s): oracle:oinstall, mode 0666

You don’t have to use raw devices for ASM volumes on Linux, as the ASMLib library is very effective and easier to manage.

Installation of minimal selection of Centos 5 – only base, core and dialup selections, will leave a small-sized system, with the minimum required.

This is a good setup to start setting up a firewall, and it lets you add any required package later using ‘yum’.

This said, you cannot forward X over SSH at this state. A minimal setup is a minimal setup, after all. I was searching for the package required to allow X forwarding over SSH, and found it – xorg-x11-xauth

Install this one, and X will be forwarded over your SSH connection (you still need to connect with the ‘-X’ flag from the client).

I was playing a bit with iSCSI initiator (client) and decided to see how complicated it is to setup a shared storage (for my purposes) through iSCSI. This proves to be quite easy…

On the server:

1. Download iSCSI Enterprise Target from here, or you can install scsi-target-utils from Centos5 repository

2. Compile (if required) and install on your server. Notice – you will need kernel-devel packages

3. Create a test Logical Volume:

lvcreate -L 1G -n iscsi1 /dev/VolGroup00

4. Edit your /etc/ietd.conf file to look something like this:

Target iqn.2001-04.il.org.tournament:diskserv.disk1
Lun 0 Path=/dev/VolGroup00/iscsi1,Type=fileio
InitialR2T Yes
ImmediateData No
MaxRecvDataSegmentLength 8192
MaxXmitDataSegmentLength 8192
MaxBurstLength 262144
FirstBurstLength 65536
DefaultTime2Wait 2
DefaultTime2Retain 20
MaxOutstandingR2T 8
DataPDUInOrder Yes
DataSequenceInOrder Yes
ErrorRecoveryLevel 0
HeaderDigest CRC32C,None
DataDigest CRC32C,None
# various target parameters
Wthreads 8

5. Start iscsi-target service:

/etc/init.d/iscsi-target start

On the client:

1. Install open-iscsi package. It will be called iscsi-initiator-utils for RHEL5 and Centos5

2. Run detection command:

iscsiadm -m discovery -t sendtargets -p <server IP address>

3. You should get a nice reply. Something like this. <IP> refers to the server’s IP

<IP>:3260,1 iqn.2001-04.il.org.tournament:diskserv.disk1

4. Login to the devices using the following command:

iscsiadm -m node -T iqn.2001-04.il.org.tournament:diskserv.disk1 -p <IP>:3260,1 -l

5. Run fdisk to view your new disk

fdisk -l

6. To disconnect the iSCSI device, run the following command:

iscsiadm -m node -T iqn.2001-04.il.org.tournament:diskserv.disk1 -p <IP>:3260,1 -u

This will not allow you to set the iSCSI initiator during boot time. You will have to google your own distro and its bolts and nuts, but this will allow you a proof of concept of a working iSCSI

Good luck!

Recently I have installed several Redhat systems on IA64 platforms. Since it required only slight adjustments, and since there were two sets of systems, RHEL3 Update2 and RHEL4 Update3, I have decided to use Kickstart for both, each with his own ks.cfg file.

For lack of any other explanation at the moment, I can only say I feel I have encountered a bug with RHEL3 on IA64 platform and ks handling.

Steps:

1. Bring up a dedicated installation server. Install on it DHCP Server, Name Server, TFTP Service (activated from xinetd), NFS Service.

2. Setup DHCP for a dedicated network card. Address pool 192.168.0.x. Server IP: 192.168.0.1

3. Verify it’s working.

4. Extract RH images to the NFS root directory, under the distro name. Example – /install/rhel3.2-ia64

5. Add elilo PXE image for IA64 in /tftpboot. Add a file elilo.conf (elilo.conf)

6. Install both servers – RHEL3 and RHEL4

7. Take anaconda-ks.cfg and use it (with slight modifications) to fit my needs. Really minor changes.

8. Boot the next nodes based on these ks files. (RHEL3 ks file: ks.cfg)

While RHEL4 works fine and uses my ks.cfg, RHEL3 does not. It seems to start using it, and then go on to asking me all these annoying questions (Welcome to RedHat 3 installation!)

I have even tried building ks.cfg using redhat-config-kickstart tool, but same results.

Since installation is done using serial console, I cannot access other virtual consoles and debug the problem on-the-fly.

***UPDATE***

Per a suggestion in a forum, I have looked again into the elilo.conf file, and noticed that the ks path was different. Matter of paying attention. This is probably the problem, and I will verify it soon.

I’ve had a belief for quite a while now that Linux, unlike other types of systems, was unable to produce any I/O SNMP information. I only recently found out that it was partially true – all production-level distros, such as RedHat (and Centos, for that matter) were unable to produce any output for any SNMP DISKIO queries.

I had found a bugzilla entry about it, so I raise the glove in a request to any of the maintainers of an RH-compatible repositories to recompile (and maintain, of course) an alternate net-snmp package which supports diskio.

Meanwhile, I have found this blog post, which offers an alternate (and quite clumsy, yet working) solution to the disk performance measurement issue in Linux. I haven’t tried it yet, but I will, rather soon.

—Update—

I have used the script from the blog post mentioned above, and it works.

Speed could be an issue. Comparing two servers the speed differential was amazing.

Both servers are connected on the same switch as the server running the query is connected. Server1 has a P2 233MHz CPU, while Server2 has a dual 2.8GHz Xion CPU.

~$ time snmpwalk -c COMMUNITY -v2c Server1 1.3.6.1.4.1.2021.13.15 > /dev/null

real 0m0.311s
user 0m0.024s
sys 0m0.020s

~$ time snmpwalk -c COMMUNITY -v2c Server2 1.3.6.1.4.1.2021.13.15 > /dev/null

real 0m8.303s
user 0m0.044s
sys 0m0.012s

Looks like a huge difference. However, I believe it’s currently good enough for me.