Posts Tagged ‘Linux system’

Hot adding Qlogic LUNs – the new method

Friday, August 8th, 2008

I have demonstrated how to hot-add LUNs to a Linux system with Qlogic HBA. This has become irrelevant with the newer method, available for RHEL4 Update 3 and above.

The new method is as follow:

echo 1 > /sys/class/fc_host/host<ID>/issue_lip
echo “—” > /sys/class/scsi_host/host<ID>/scan

Replace “<ID>” with your relevant HBA ID.

Notice – due to the blog formatting, the 2nd line might appear incorrect – these are three dashes, and not some Unicode specialy formatted dash.

Graphing on-demand Linux system performance parameters

Tuesday, May 20th, 2008

Current servers are way more powerful than we could have imagined before. With quad-core CPUs, even the simple dual-socket servers contain lots of horse-power. Remember our attitude towards CPU power five years ago, and see that we’re way beyond our needs.

When modern servers are equipped with at least eight cores, other, non-CPU related issues become noticeable. Storage, as always, remains a common bottleneck, and, as an increase in expectations always accommodate increase in abilities, memory and other elements can be the cause for performance degradation.

sar‘ is a known tool for Linux and other Unix flavors, however, understanding the contexts within is not trivial, and while the data is there, figuring what is relevant for the issue at hand becomes, with more disk devices, and more CPUs, even more complicated.

kSar is a simple java utility which makes this whole mess into a simple, readable graphs, capable of being exported to PDF for the pleasure of the customers (where applicable). It parses existing sar files, or the extracted contents of ‘sa’ files (from, by default, /var/log/sa/). It is a useful tool, and I recommend it with all my heart.

Alas, when it comes to parsing ‘sa’ files, you will need, in most cases, either to export the file into text on the source machine, or use a similar version of sysstat tools, as changes in versions reflect changes in the binary format used by sar.

You can obtain the sysstat utils from here, and compile it for your needs. You will need only ‘sar’ on your own machine.

An important note – you will not be able to compile sysstat utils using GCC 4.x. Only 3.x will do it. The error would look like:

warning: ‘packed’ attribute ignored for field of type `unsigned char…

followed by compilation errors. Using GCC version 3.x will work just fine.

DSL (Damn Small Linux) Diskless boot

Friday, August 31st, 2007

I have come across a requirement to boot a thin client on a very cheap hardware into Linux. Due to the tight hardware requirements, and the tight budget, I have decided to focus on diskless systems, which can be easily modified and purchased to our needs.

Not only that, but due to the hardware configuration (Via 333MHz, 128MB RAM, etc) I have decided to focus on a miniature Linux system.

I dislike re-doing what someone else has done, unless I can do it noticeably better. I have decided to use DSL (Damn Small Linux) as my system of choice, with only minor changes to fit my needs:

Out of the “box”, I was unable to find network-boot DSL. Quickly searching their site, the version which seemed to fit was the initrd-only system. I downloaded it from this mirror, but you can find it as the dsl-x.x.x-initrd.iso file.

Extracting the initrd from the ISO file is quite simple:

mkdir /mnt/iso
mount -o loop dsl-x.x.x-initrd.iso /mnt/iso

And from here you can just copy the contents of the directory /mnt/iso/boot/isolinux/ selectively to your tftpboot directory.

So I got 50MB initrd which worked just fine. Changing this was quite a procedure, because in addition to the steps per the wiki hacking guide, I was required to extract the KNOPPIX file outside of the initrd, and repackage it when done. Quite messy, however, stand-alone as soon as the system has been able to boot.

An alternate I have decided to investigate into was of booting into nfs mount, aka, accessing the KNOPPIX iso disk through NFS and not through CDROM.

I was able to find some leads in DSL forums at this page, which lead to this guide. I was able to download pxe boot image from Knoppix themselves, however, it was based on an old kernel (2.4.20-XFS) which was part of Knoppix 3.3 (cannot find it anymore) and although reached the level of actually booting my nfs, didn’t include enough network drivers (I wanted pcnet32 to be able to “play” with VMware for the task), and was incompatible with my existing DSL.

I had opened the supplied Knoppix initrd, and replaced the modules version to the one supplied with DSL – 2.4.24, per the rest of the system. In addition, I have added my required modules, etc, and was able to boot successfully both on VMware and on the thin client hardware.

To replace the modules, one needs to follow these general-only guidelines (these are not exactly step-by-step instructions):

Mount through loop the DSL KNOPPIX image, for example, in /mnt/dsl
Uncompress the Knoppix PXE initrd
Mount through loop the uncompressed Knoppix PXE initrd, for example, in /mnt/initrd
cd to /mnt/initrd/modules
Replace all modules in the current tree with the ones supplied by DSL, obtainable from /mnt/dsl/lib/modules/2.4.26 directory tree, including the cloop.o module
Umount the initrd image
Compress the initrd image
Boot using DSL linux and the new initrd image.

In order to boot successfully, you need to supply the pxe boot these two instructions:

nfsdir=nfs-server:/path/to/KNOPPIX directory

(since I was quite unsure about the letter case required, I have created a symlink from lower-case to upper case, so I had a link /mnt/KNOPPIX to a directory /mnt/knoppix, and inside this directory, a file called knoppix and a symbolic link to this file KNOPPIX. In my case, the exported path was /mnt/ only. Notice this one!).

BOOT_IMAGE=KNOPPIX – but you can have different KNOPPIX images for different purposes.

Finally it has worked correctly. Changes can be done only to the KNOPPIX iso image, per the hacking guide.

This is my PXE-enabled initrd, based on the text above, which fits DSL-3.4.1: minirt24.gz

DL140 Generation 1 (G1) and LILO do not work

Wednesday, February 14th, 2007

I add to this blog all pieces of information which I might think could help other people.

One of the things I have encountered short while ago had to do with DL140 G1 system cannot boot Linux.

It’s Linux system (RedHat 4 32bit) was deployed by an external system and the system could not boot. However, when installed from CD, the system booted just fine. The symptom for this was that after booting, the screen showed:

LILO: Loading Linux…..

and that’s all. The system could have booted (and did so once) after several days, however, this is not really a desired status.

It seems to be an issue of Lilo and this specific hardware. Other systems (non DL140) were able to boot just fine using Lilo, and this same kernel version was bootable on that system through other means.

Replacing Lilo with GRUB during installation/deployment solved the isse. FYI.

VMware experiance – lots of it

Monday, January 16th, 2006

During the past few days/weeks, I’ve had the pleasure (and will have in the future as well) of playing with VMware ESX (2.5.2) and GSX (3.2.1), as well as Workstation in my long forgotten past, and here I try to describe my own personal impressions of the product.

First – it is a good product. I enjoyed working with it. It is not too complicated, however, it is not documented enough, and finding some solutions for specific problems were not easy and were not made easy by their docuemntations online and their web site.

The GSX I will start with. It is a modern, easilly usable product. It allows to run virtual systems on a running Windows or Linux system, and it allows for remote management of such systems. Good remote GUI (VMware Console), which allows some cool stunts such as installing a guest (virtual, but we’ll keep to VMware’s lingo here) OS directly from your own CDROM, on your own personal desktop. If you don’t get it – Install a Windows server, call it Server1. Install VMware GSX on it, and then run on your desktop the VMware-Console software. Using this software you can define a whole guest system on Server1, control it, and view its "physically attached" keyboard, mouse, screen. So, you can map your own desktop’s CDROM to a guest system on Server1, and install the guest from there. It’s a stunt which allow you never to leave your own chair! It doesn’t exist on the more expensive and advanced ESX, and it’s a pitty.

You can define, using the VMware-Console, or even using a web-based management interface a larger variaty of hardware on a guest system using the GSX than you can using the ESX. The ESX’s console and web interface did not allow for serial ports on a guest. It did not allow for sound, or for USB. So it appears that although the ESX version is more advanced, it is limited comparing to the lesser GSX.

I’ve discovered, during such an effort, that I could manually define a serial port on ESX guest system. I believe other devices can be defined as well, but I wouldn’t want to try that, nor would I be able to do so without a good example of a GSX system’s guest configuration file as an example. I’ve come to a resolution here, and it was working, for the time being.

The ESX version is more like a mainframe style system – it allows for an embedded system slicing and partitioning for consolidation of numerous virtual machines. Lots of buzz-words, but all they mean is that you can have one stronger PC hardware running few virtual configurations (guests), easier to manage, and with better utilization of your actual resources, as physical servers tend to lay idle noticable part of the day in most cases.

It adds in, however, few, more complicated considerations into the soup – if I had 3 servers doing nothing most of the day, but at 4 AM, all of them start to index local files, I couldn’t care less. However, on such consolidated setup, I would care – for better utilization, I would measure the amount of time, or estimated amount of time each require for its own task, and try to spread it better around the clock – this one will start a bit earlier, and that one will start a bit later, so I would not get to hog my system. It brings us to the major problem of such a setup – I/O. Each computers system ever built had problems with its I/O. I/O, and especially disk access, is the slowest mechanism in a computer. You can calculate millions and tens of millions instructions per second, but you would need few minutes to put the results on the disk. You could say that the I/O problem can be identified at two levels:

1) General disk access – Reading and writing to disks is rather slow.

2) Small files – Most files on the average system are small. Very small. Disk layout, as hard as any FS might try, results in random and spread layout, which leads to high seek-time when reading and writing small files, which is, actually, the main occupation of any OS I/O subsystem.

Virtual and consolidated solutions are no different than that. Each virtual OS requires its own share with the physical hardware’s disk I/O, which might lead, in some cases, to poor performance of all guest OSes, just because of disk hog, which, by the way, is the harderst to measure and detect. Moreover, it is the harderst to solve. You can always pour in some more hard-drives, but the host (Container) I/O subsystem remains the same single system, and the load generated by large amounts of small, random reads and writes remains the same. So, unless you use some QoS mechanism, you can get a single machine to hog your entire virtual construction. This is one of the biggest downsides of such consolidation solutions.

With P-Series, by the way, they can allow consolidation of the hardware into few I/O seperated virtual machines (Logical Partitions, or LPAR, as IBM call them. They call everything "Partitions"). VMware ESX supports such a setup as well, but I wonder how well, since it is not really hardware-bound setup (as LPAR is), they manage to prevent negative effects and degrade of performance of one I/O channel on others.

I guess that for low-I/O systems, or for lab usage, ESX could do the trick. You can run a full OS cluster (Windows or Linux) on it, and it will work correctly, and nicely. Unless you’re up to disconnecting physical (or virtual) disks from guest servers, it is a good solution for you.

So, to sum things up, I can say that I enjoy "playing" with VMware products. I enjoy them because they’re innovative, sophisticated, and they look sexy, but I am well aware of the way the market chooses its current solutions, and I am aware of the fact many utilize VMware products for the sake of consolidation and ease of management, without propper consideration or understanding of the well expected performance loss which can be part of it (but does not have to be, if you calculate things correctly). A friend has told me about ESX setup he has encountered, where the had quad-CPU system, with 16GB RAM, running 16 guest OSes, of which MS Exchange, MSSQL2005, MS-SMS, and more, using a single shelve of raid5 based storage, connected via two 2Gb/s fibre connections, setup as failback (only one active link at a time). It was over loaded, and was performing badly. Nice server, though 🙂

One last thing about ESX is that it would not install on purely IDE systems. It requires SCSI (and maybe SATA?) for the space holding the guests virtual hard drives.

So, enough about VMware today. I wonder if there’s some easy matrix for "tell me what servers will do, and we’ll calculate I/O, CPU and memory for your future server", instead of the poor way of "I’ve discovered my server is too weak for the task, half a year after deployment", which we see too much of today.