Running Systems

(This article is the essence of a post from this Redhat Archive and it goes as follows:

Problem: You need to detect what deletes files on your Linux

Solution: Using auditd, with the right flags, you could get a lot of information.

In Practice:

• If the mount point/directory is /oracle, then:
• (as root:) auditctl –w /oracle -k whodeletedit -p w
  (Explanation: Monitor the directory /oracle, and log everything under the label “whodeleteit”. Monitor write operations)
• To see, later, who deleted files, run (as root): ausearch -i -k whodeletedit -x /bin/rm
• You would want to stop the logging as soon as you found the culprit, by running (as root):  auditctl –W /oracle -k whodeletedit -p w

I hope it helps you just as it helps me.

Assume a server has two network interfaces as follows:

• eth0 : 192.168.0.1/24
• eth1 : 192.168.10.1/24

Let’s assume these interfaces reside on the different VLANs. Lets assume they were connected incorrectly, in such a way that eth0 is connected to VLAN 10, which servers 192.168.10.0/24 and eth1 is connected to VLAN 2, which serves 192.168.0.0/24.

You would expect that queries by other hosts on VLAN 2 (which is connected to eth1, but serves 192.168.0.0/24!) would not get responses from the server. You are wrong.

Linux will answer who-is queries on VLAN 2, replying with eth1’s MAC address to queries for 192.168.0.1 IP address.

This example is a simple example, but it can get ugly if your eth0 mimics a different network, and you want the server to be disconnected. I have had to “forge” a network setup on a different VLAN, mimicking the original network and subnet. However – a “backdoor” I have opened (on an additional NIC) between the mimicking server and the original server on a different IP class (a private one) resulted in the mimicking server answering to ARP queries, causing the clients to attempt connecting to the mimicking server instead of the production server. The clients could not complete the TCP handcheck because the mimicking server attempted to contact them via eth0, which was on the fake network, and did not actually reach anywhere.

This was a more complex example, however – the result is the same – the response on the “wrong” network interface to ARP who-is queries might hijack data which should be delivered elsewhere.

There is a solution! You need to setup the sysctl parameter arp_ignore  to either of the following values. The parameter is hidden in /proc/sys/net/ipv4/conf/<NIC>/arp_ignore

The parameters documentation is as follows:

arp_ignore – INTEGER
Define different modes for sending replies in response to received ARP requests that resolve local target IP addresses:
0 – (default): reply for any local target IP address, configured on any interface
1 – reply only if the target IP address is local address configured on the incoming interface
2 – reply only if the target IP address is local address configured on the incoming interface and both with the sender’s IP address are part from same subnet on this interface
3 – do not reply for local addresses configured with scope host, only resolutions for global and link addresses are replied

The value “1” or “2” would do the trick in such cases.

As far as routing goes, it is a very bad idea to place multiple network interfaces with IPs of a single network (subnet). The routing table, which decides which interface to route the data through, reads the table line by line, thus – all traffic goes through a single interface of the batch (-> of the interfaces “living” on the same network). A common way of working around it is by using network teaming (Linux = bonding) to handle round-robin or active/passive of multiple interfaces on a single network, and that means that the interfaces are “bonded” into a single logical interface, which, then, has no routing issues whatsoever.

Having multiple network interfaces with IPs of the same network does not help with throughput, but does it increase redundancy? The simple answer is “no” – It does not. In Linux, when an interface is disconnected, it does not go down automatically (unlike Windows, for example), meaning that the routing table does not get updated with remaining interfaces. The traffic would still be targeted at that “dead” interface. Moreover – even if Linux did do that, this is a solution for a single type of a problem. What if someone changed the switch to have a different network VLAN on that particular (one of two or more) port? Link remains up, but communication doesn’t get anywhere.

These problems are more noticeable when the single network is an iSCSI network, and the system needs both multiple path access to the iSCSI storage, and redundancy becomes an issue, because iSCSI network is commonly – a critical network.

It is common to connect multiple iSCSI networks and not multiple ports on a single network, maintaining fabric-like separation of networks, and devices, where possible. However – this article will point at what to do if the network layout is not under your control and you need to place multiple network interfaces on the same physical network.

As mentioned before, the routing table will make sure that all your communication to that particular network go by the first routed network interface. However – the iSCSI daemon (open-iscsi) has the capability to bind to specific interfaces. To do so, you need to define an interface to iSCSI. Do so by running the command:

iscsiadm -m iface -I NIC_NAME –op=new

The name is user defined. I recommend using the same names used by the OS, like ‘eth5’ or ‘ib0’ or ‘eno1’ – to keep it simple. This is a declarative definition only. To actually bind the ‘iface’ to the real interface device, you need to run the following command:

iscsiadm -m iface -I NIC_NAME  –op=update -n iface.hwaddress -v 00:AA:BB:CC:DD:EE

Use the real interface MAC address. Then you can discover and login to targets with the defined interfaces only:

iscsiadm -m discovery -t st -p TARGET_IP -I NIC1 -I NIC2

iscsiadm -m node -L all -I NIC1 -I NIC2

According to the documentations, this will require setting the sysctl net.ipv4.conf.default.rp_filter to 0.

This should do the trick, so now multipathing should show the right amount of paths.

kSar is one of the best tools to display an easy chart based on your system’s SAR metrics collection. kSar has not been developed since sometime in 2013, and it lacks the ability to parse modern Linux sysstat charts, leaving us with some missing info. However – some good soul has taken kSar and forked in into development. This tool can be found here, and it’s working nicely with RHEL6 metrics, as far as I could see. Grab it if you haven’t already!

I have created an RPM package, and SRPM package, which I will share here, for smartmontools version 6.5 on RHEL 6. Note that the official version is 5.43 which is clueless with many modern SSD disks. I have yet to test it correctly, and in general – use at your own risk.

smartmontools-6-5-1-el6-src