Multiple iSCSI interface on the same subnet

As far as routing goes, it is a very bad idea to place multiple network interfaces with IPs of a single network (subnet). The routing table, which decides which interface to route the data through, reads the table line by line, thus – all traffic goes through a single interface of the batch (-> of the interfaces “living” on the same network). A common way of working around it is by using network teaming (Linux = bonding) to handle round-robin or active/passive of multiple interfaces on a single network, and that means that the interfaces are “bonded” into a single logical interface, which, then, has no routing issues whatsoever.

Having multiple network interfaces with IPs of the same network does not help with throughput, but does it increase redundancy? The simple answer is “no” – It does not. In Linux, when an interface is disconnected, it does not go down automatically (unlike Windows, for example), meaning that the routing table does not get updated with remaining interfaces. The traffic would still be targeted at that “dead” interface. Moreover – even if Linux did do that, this is a solution for a single type of a problem. What if someone changed the switch to have a different network VLAN on that particular (one of two or more) port? Link remains up, but communication doesn’t get anywhere.

These problems are more noticeable when the single network is an iSCSI network, and the system needs both multiple path access to the iSCSI storage, and redundancy becomes an issue, because iSCSI network is commonly – a critical network.

It is common to connect multiple iSCSI networks and not multiple ports on a single network, maintaining fabric-like separation of networks, and devices, where possible. However – this article will point at what to do if the network layout is not under your control and you need to place multiple network interfaces on the same physical network.

As mentioned before, the routing table will make sure that all your communication to that particular network go by the first routed network interface. However – the iSCSI daemon (open-iscsi) has the capability to bind to specific interfaces. To do so, you need to define an interface to iSCSI. Do so by running the command:

iscsiadm -m iface -I NIC_NAME –op=new

The name is user defined. I recommend using the same names used by the OS, like ‘eth5’ or ‘ib0’ or ‘eno1’ – to keep it simple. This is a declarative definition only. To actually bind the ‘iface’ to the real interface device, you need to run the following command:

iscsiadm -m iface -I NIC_NAME  –op=update -n iface.hwaddress -v 00:AA:BB:CC:DD:EE

Use the real interface MAC address. Then you can discover and login to targets with the defined interfaces only:

iscsiadm -m discovery -t st -p TARGET_IP -I NIC1 -I NIC2

iscsiadm -m node -L all -I NIC1 -I NIC2

According to the documentations, this will require setting the sysctl net.ipv4.conf.default.rp_filter to 0.

This should do the trick, so now multipathing should show the right amount of paths.

A new version to kSar

kSar is one of the best tools to display an easy chart based on your system’s SAR metrics collection. kSar has not been developed since sometime in 2013, and it lacks the ability to parse modern Linux sysstat charts, leaving us with some missing info. However – some good soul has taken kSar and forked in into development. This tool can be found here, and it’s working nicely with RHEL6 metrics, as far as I could see. Grab it if you haven’t already!

Smartmontools 6.5 for RHEL 6

I have created an RPM package, and SRPM package, which I will share here, for smartmontools version 6.5 on RHEL 6. Note that the official version is 5.43 which is clueless with many modern SSD disks. I have yet to test it correctly, and in general – use at your own risk.

smartmontools-6-5-1-el6-src

XenServer – Map all VMs disks to Storage Repositories

Read the rest of this entry »

Dell Server admin tools (srvadmin) on Centos 5/XenServer

Recently, with the release of a new 8.x version of Dell SrvAdmin tools, the Centos/RHEL5 (and XenServer, by the way) repositories disappeared. It appears that Dell will not support the RHEL5.x brands anymore.

The proper solution is to install the last 7 SrvAdmin tools, as can be shown in this site.

This solves the problem for XenServer as well.