Running Systems

I was required to create a self-contained, single DVD to automate the installation of Solaris 10 on x86_64. I could not find any up-to-date straight forward guide which can explain how to do it, so I do it here. This is not an explanation for dummies, so you must know (to some degree, of course) what you’re doing.

I will describe the procedure in whole, and will explain in greater details below, if I see fit. A section which will be explained later will be marked with (*) at the end of the line.

• Install Solaris 10 x86 on a machine. Many actions will happen on this little server…
• Setup your Solaris installation according to your likings. Make sure you have your beloved users, your passwords, your configurations. Don’t mind much about networking configurations (IP, Netmask, etc) – as they will be unconfigured for the image.
• Create a Flash Image (flar) of the system (*)
• Copy the contents of the installation DVD to a directory inside your system. Let’s call it /tmp/dvd
• Remove /tmp/dvd/Solaris_10/Product directory. You will not need it.
• Extract the contents of /tmp/dvd/x86.miniroot to /tmp/miniroot (*)
• Perform several actions with the extracted miniroot (*)
• Re-archive the contents of the x86.miniroot and place them instead of /tmp/dvd/boot/x86.miniroot
• Place the flar file inside /tmp/dvd/flash
• Edit your jumpstart files inside /tmp/dvd/.install_config (*)
• Edit /tmp/dvd/boot/grub/menu.lst boot loader to add an entry for your installation (*)
• Create an ISO from the DVD directory (*)
• Burn the DVD and try to use it

And now for the drill-down

Creating a flash image

Use the command flarcreate to create your own flash image:

flarcreate -n sol10_automation -c -x /tmp /tmp/sol10_auto.flar

This should do the work. Remember – /tmp will not be persistent across reboots! Make sure your files are not there before you reboot the system!

Extracting/Archiving the x86.miniroot

To do so, you need to run the command /boot/solaris/bin/root_archive

Extracting the image can be done like this:

/boot/solaris/boot/root_archive unpack /tmp/dvd/boot/x86.miniroot /tmp/miniroot

Archiving the image can be done like this:

/boot/solaris/boot/root_archive pack /tmp/dvd/boot/x86.miniroot /tmp/miniroot

Actions to perform on the extracted miniroot

Three actions are to be performed on the extracted miniroot. In our example, it resides on /mnt/miniroot.

First, you need to remove the default sysidcfg (which is a symbolic link)

rm /mnt/miniroot/etc/sysidcfg

Now, you have to place your custom sysidcfg in there, instead.

This is an example of my own sysidcfg file:

name_service=NONE
network_interface=nge0 {primary hostname=sol10
ip_address=10.10.10.10
netmask=255.0.0.0
default_route=NONE
protocol_ipv6=no }
nfs4_domain=dynamic
service_profile=open
root_password=12wR2rF34t
security_policy=NONE
system_locale=en_US.UTF-8
timezone=GMT
timeserver=localhost
keyboard=US-English
terminal=xterm

The root password is encrypted. Take it from your own /etc/shadow file. For more information about sysidcfg file, check out Sun site.

Following that, you need to edit a specific file in the miniroot. Edit /tmp/miniroot/usr/sbin/install.d/profind and search for the cdrom() function. Search the line

if [ -f /tmp/.preinstall ]; then

and hash (remark) it. Don’t forget to remark the closing “fi” below.

Jumpstart contents

This has to be inside /tmp/dvd/.install_config . Edit the file /tmp/dvd/.install_config/rules and make sure it has only one line (in our example. If you know what you’re doing with Jumpstart, go ahead!)

any –   x86-begin any_machine  x86-end

This line will match any hardware, run x86-begin script (from that same directory) on it prior to running the installation itself, and run x86-end script on it after the installation phase. It allows up further customisation during installs (verify what type of RAID, check memory, whatever). The installation profile itself is the file any_machine.

You will need to run “check” on the file to build the rules.ok file

cd /tmp/dvd/.install_conf

/tmp/dvd/Solaris_10/Misc/jumpstart_sample/check

Lets look at my any_machine file:

install_type    flash_install
archive_location local_file /cdrom/flash/sol10_auto.flar
partitioning    explicit
filesys         any 8196 swap
filesys         any 10240 /
filesys         any free /storage

Notice that the installation type is “flash_install” and that the location of the file is local, inside /cdrom (where the bootable dvd will be mounted) inside a directory called flash. Partitioning is defined here, explicitly.

For more information about Jumpstart, search in Sun site. They have plenty of information.

Edit Grub

Add the following entry to your /tmp/dvd/boot/grub/menu.lst file

title Solaris10 Jumpstart
kernel /boot/multiboot kernel/unix – install -B
install_media=cdrom
module /boot/x86.miniroot

Make sure it is the default option for grub.

Creating DVD ISO from the directory

We’re almost done. To create a DVD iso file from the directory, perform the following actions:

cd /tmp/dvd

mkisofs -b boot/grub/stage2_eltorito -c .catalog -no-emul-boot -boot-load-size 4 -boot-info-table -relaxed-filenames -l -ldots -r -N -d -D -V SOL_10_1008_X86 -o /tmp/sodvd.iso .

Don’t ignore the “.” at the end!

(This specific line was tested on Linux, but there is no reason for it not to work on any modern Solaris system)

Appendix

You would like to keep your /tmp/dvd directory somewhere else, or you will lose it on your next reboot.

This sums it up. Let me know if the procedure is broken somehow.

I was requested to setup a “poor man’s” load balancing. The server accesses various HTTP servers with a GET command, and the customer fears that the server’s IP will get blocked. To work around this, or at least – to minimize the problem, the customer has purchased three IP addresses.

I have assigned all three addresses to the server, and was into smart routing as a solution. I did not want to capture all outbound communication, but only HTTP (port 80). Also – the system is Centos, which means there are only few available iptbles modules, so “random” module is not an option. Compiling a new kernel for a server across an ocean didn’t sound like the best idea, so I have attempted to work with the available tools.

With net.ipv4.conf.default.rp_filter set to 0 in /etc/sysctl.conf , and with real internet IPs (won’t work above NAT), I added the following rules to the mangle table:

iptables -t mangle -N CUST_OUTPUT
iptables -t mangle -A CUST_OUTPUT -o ! lo -p tcp -m recent –remove -j MARK –set-mark 1
iptables -t mangle -A OUTPUT -o ! lo -p tcp –dport 80 -m state –state NEW -m recent –update –seconds 60 –hitcount 6 -j  CUST_OUTPUT
iptables -t mangle -A OUTPUT -o ! lo -p tcp –dport 80 -m state –state NEW -m recent –update –seconds 60 –hitcount 4 -j MARK –set-mark 3
iptables -t mangle -A OUTPUT -m state –state NEW -m mark –mark 3 -j ACCEPT
iptables -t mangle -A OUTPUT -o ! lo -p tcp –dport 80 -m state –state NEW -m recent –update –seconds 60 –hitcount 2 -j MARK –set-mark 2
iptables -t mangle -A OUTPUT -m state –state NEW -m mark –mark 2 -j ACCEPT
iptables -t mangle -A OUTPUT -o ! lo -p tcp –dport 80 -m state –state NEW -m recent –set

To the nat table, I have added these following rules:

iptables -t nat -A POSTROUTING -m mark –mark 2 -j SNAT –to 1.1.1.2
iptables -t nat -A POSTROUTING -m mark –mark 3 -j SNAT –to 1.1.1.3
iptables -t nat -A POSTROUTING -m mark –mark 1 -j SNAT –to 2.3.4.5

(replaced the real customer’s IPs with the junk above). Of course – all three IP addresses are accessible from the Internet and fully routable.

After a short run, I saw the following lines when running iptables -t nat -L -v

Chain POSTROUTING (policy ACCEPT 39055 packets, 2495K bytes)
pkts bytes target     prot opt in     out     source               destination
143  8580 SNAT       all  —  any    any     anywhere             anywhere            MARK match 0x2 to:1.1.1.2
144  8640 SNAT       all  —  any    any     anywhere             anywhere            MARK match 0x3 to:1.1.1.3
72  4320 SNAT       all  —  any    any     anywhere             anywhere             MARK match 0x1 to:2.3.4.5

Statistically, this is quite ok. Mark 0x1 forces the same route as Mark 0x0, so this is rather balanced.

Works like a charm, for now 🙂

If you’re into Oracle RAC over iSCSI, you should be rather content – this configuration is a simple and supported. However, working with some iSCSI target devices, order and naming is not consistent between both Oracle nodes.

The simple solutions are by using OCFS2 labels, or by using ASM, however, if you decide to place your voting disks and cluster registry disks on raw devices, you are to face a problem.

iSCSI on RHEL5:

There are few guides, but the simple method is this:

1. Configure mapping in your iSCSI target device
2. Start the iscsid and iscsi services on your Linux
   • service iscsi start
   • service iscsid start
   • chkconfig iscsi on
   • chkconfig iscsid on
3. Run “iscsiadm -m discovery -t st -p target-IP“
4. Run “iscsiadm -m node -L all“
5. Edit /etc/iscsi/send_targets and add to it the IP address of the target for automatic login on restart

You need to configure partitioning according to the requirements.

If you are to setup OCFS2 volumes for the voting and for the cluster registry, there should not be a problem as long as you use labels, however, if you require raw volumes, you need to change udev to create your raw devices for you.

On a system with persistent disk naming, follow this process, however, on a system with changing disk names (every reboot names are different), the process can become a bit more complex.

First, detect your scsi_id for each device. While names might change upon reboots, scsi_ids do not.

scsi_id -g -u -s /block/sdc

Replace sda with the device name you are looking for. Notice that /block/sda is a reference to /sys/block/sdc

Use the scsi_id generated by that to create the raw devices. Edit /etc/udev/rules.d/50-udev.rules and find line 298. Add a line below with the following contents:

KERNEL==”sd*[0-9]”, ENV{ID_SERIAL}==”14f70656e66696c000000000004000000010f00000e000000″, SYMLINK+=”disk/by-id/$env{ID_BUS}-$env{ID_SERIAL}-part%n” ACTION==”add” RUN+=”/bin/raw /dev/raw/raw%n %N”

Things to notice:

1. The ENV{ID_SERIAL} is the same scsi_id obtained earlier
2. This line will create a raw device in the name of raw and number in /dev/raw for each partition
3. If you want to differtiate between two (or more) disks, change the name from raw to an aduqate name, like “crsa”, “crsb”, etc, for example:

KERNEL==”sd*[0-9]”, ENV{ID_SERIAL}==”14f70656e66696c000000000005000000010f00000e000000″, SYMLINK+=”disk/by-id/$env{ID_BUS}-$env{ID_SERIAL}-part%n” ACTION==”add” RUN+=”/bin/raw /dev/raw/crs%n %N”

Following these changes, run “udevtrigger” to reload the rules. Be advised that “udevtrigger” might reset network connection.