Archive for the ‘Linux’ Category

Windows 7 hammering dnsmasq

Saturday, February 1st, 2014

I migrated to dnsmasq just yesterday, and discovered that a Windows 7 machine was hammering the server with messages like this:

Feb  1 11:06:07 dns dnsmasq-dhcp[1078]: DHCPINFORM(eth0) 192.168.1.77 91:de:87:7b:e5:a8
Feb  1 11:06:07 dns dnsmasq-dhcp[1078]: DHCPACK(eth0) 192.168.1.77 91:de:87:7b:e5:a8 winpc

Googling a bit, I found out this link (with an explanation). The solution is fairly simple. Add the following line to your dnsmasq.conf file to solve the problem:

dhcp-option=252,”\n”

 

Extracting/Recreating RHEL/Centos6 initrd.img and install.img

Tuesday, October 1st, 2013

A quick note about extracting and recreating RHEL6 or Centos6 (and their derivations) installation media components:

Initrd:

Extract:

mv initrd.img /tmp/initrd.img.xz
cd /tmp
xz –format=lzma initrd.img.xz –decompress
mkdir initrd
cd initrd
cpio -ivdum < ../initrd.img

Archive (after you applied your changes):

cd /tmp/initrd
find . | cpio -o -H newc | xz -9 –format=lzma > ../new-initrd.img

/images/install.img:

Extract:

mount -o loop install.img /mnt
mkdir /tmp/install.img.dir
cd /mnt ; tar cf – –one-file-system . | ( cd /tmp/install.img.dir ; tar xf – )
umount /mnt

Archive (after you applied your changes):

cd /tmp
mksquashfs install.img.dir/ install-new.img

Additional note for Anaconda installation parameters:

I did not test it, however there is a boot flag called stage2= which should lead to a new install.img file, other than the hardcoded one. I don’t if it will accept /images/install-new.img as its flag, but it can be a good start there.

One more thing:

Make sure that the vmlinuz and initrd used for any custom properties, in $CDROOT/isolinux do not exceed 8.3 format. Longer names didn’t work for me. I assume (without any further checks) that this is isolinux limitation.

XenServer – increase LVM over iSCSI LUN size – online

Wednesday, September 4th, 2013

The following procedure was tested by me, and was found to be working. The version of the XenServer I am using in this particular case is 6.1, however, I belive that this method is generic enough so that it could work for every version of XS, assuming you're using iSCSI and LVM (aka - not NetApp, CSLG, NFS and the likes). It might act as a general guideline for fiber channel communication, but this was not tested by me, and thus - I have no idea how it will work. It should work with some modifications when using Multipath, however, regarding multipath, you can find in this particular blog some notes on increasing multipath disks. Check the comments too - they might offer some better and simplified way of doing it.

So - let's begin.

First - increase the size of the LUN through the storage. For NetApp, it involves something like:

lun resize /vol/XenServer/luns/SR1.lun +1t

You should always make sure your storage volume, aggregate, raid group, pool or whatever is capable of holding the data, or - if using thin provisioning - that a well tested monitoring system is available to alert you when running low on storage disk space.

Now, we should identify the LUN. From now on - every action should be performed on all XS pool nodes, one after the other.

cat /proc/partitions

We should keep the output of this command somewhere. We will use it later on to identify the expanded LUN.

Now - let's scan for storage changes:

iscsiadm -m node -R

Now, running the previous command again will have a slightly different output. We can not identify the modified LUN

cat /proc/partitions

We should increase it in size. XenServer uses LVM, so we should harness it to our needs. Let's assume that the modified disk is /dev/sdd.

pvresize /dev/sdd

After completing this task on all pool hosts, we should run sr-scan command. Either by CLI, or through the GUI. When the scan operation completes, the new size would show.

Hope it helps!

BackupExec 2012 (14) on newer Linux

Tuesday, August 6th, 2013

In particular – Oracle UEK, which “claims” to be 2.6.39-xxx, but is actually 3.0.x with a lower version number. Several misbehaviors (or differences) of version 3 can be found. One of them is related to BackupExec. The service would not start on OEL6 with UEK kernels. The cause of it is an incorrect use of a function – getIfAddrs. Everything can be seen in this amazing post. The described patch works, at least to allow the service to start. Check out the comments for some insights about how to identify the correct call.

I am re-posting it here, so it can be found for Oracle Universal Enterprise Kernel (UEK) as well.

SABnzbd and high CPU usage on weak CPUs

Sunday, July 21st, 2013

SABnzbd is a nice tool. I just replaced my previous nzbget with it, due to its better handling of the obfuscated names in usenet groups. However, on an Atom CPU, the max download speeds did not go over ~5MB/s on a 100Mb/s link. This is rather sad, because nzbget did get the whole ~11MB/s speeds.
The source of this slowness is the handling of the SSL within python (or SABnzbd, to be exact) which cannot exceed a single core. A workaround for this problem was using stunnel with the SABnzbd, as described here. Because stunnel uses multiple cores, handling of the connections can exceed the rather weak CPU limit into other cores. Following this procedure, I was able to reach 11-12MB/s speeds.

Notice a little correction: The bottom string: connect, has to have the equal sign, like this:

connect = news.giganews.com:563

Juniper NetworkConnect (NC) and 64bit Linux

Tuesday, June 25th, 2013

Due to a major disk crash, I have had to use my ‘other’ computer for VPN connections. It meant that I have had to prepare it for the operation. I attempted to login to aJuniper-based SSL-VPN connection, however, I did get a message saying that my 64bit Java was inadequate. I had a link, as part of the error message to Juniper KB, to which I must link (remembering how I have had to search for possible solutions in the past).

The nice thing about this solution is that it does not replace your default Java version on the system, which was always a problem, as I was using Java for various purposes, but it recognizes that it’s part of the (update-)alternatives list, and makes use of the correct Java version.

Juniper did it right this time!

Oh – and the link to their KB

And to Oracle Java versions, to make life slightly easier for you. You will need Oracle login, however (you can register for free).

TSClient on Ubuntu 12.04

Tuesday, June 25th, 2013

Today there will be a few different posts. This is a day full of events, so…

My first – to allow tsclient to work under Ubuntu 12.04, you should follow this guide: http://superuser.com/a/547102

To sum it up:

  1. Get tsclient to your architecture from http://pkgs.org
  2. Install it using ‘sudo dpkg –force-depends -i tsclient_0.150-3ubuntu1_amd64.deb’
  3. Edit /var/lib/dpkg/status , search for tsclient, and remove the entry containing libpanel-applet2-0

Done.

Target-based persistent device naming

Saturday, June 22nd, 2013

When Connecting Linux to a large array of SAS disks (JBOD), udev creates default persistent names in /dev/disk/by-* . These names are based on LUN ID (all disks take lun0 by default), and by path, which includes, for a pure SAS bus – the PWWN of the disks. It means that an example to such naming would be like this (slightly trimmed for ease of view):

/dev/disk/by-id:
scsi-35000c50055924207 -> ../../sde
scsi-35000c50055c5138b -> ../../sdd
scsi-35000c50055c562eb -> ../../sda
scsi-35000c500562ffd73 -> ../../sdc
scsi-35001173100134654 -> ../../sdn
scsi-3500117310013465c -> ../../sdk
scsi-35001173100134688 -> ../../sdj
scsi-35001173100134718 -> ../../sdo
scsi-3500117310013490c -> ../../sdg
scsi-35001173100134914 -> ../../sdh
scsi-35001173100134a58 -> ../../sdp
scsi-3500117310013671c -> ../../sdm
scsi-35001173100136740 -> ../../sdl
scsi-350011731001367ac -> ../../sdi
scsi-350011731001cdd58 -> ../../sdf
wwn-0x5000c50055924207 -> ../../sde
wwn-0x5000c50055c5138b -> ../../sdd
wwn-0x5000c50055c562eb -> ../../sda
wwn-0x5000c500562ffd73 -> ../../sdc
wwn-0×5001173100134654 -> ../../sdn
wwn-0x500117310013465c -> ../../sdk
wwn-0×5001173100134688 -> ../../sdj
wwn-0×5001173100134718 -> ../../sdo
wwn-0x500117310013490c -> ../../sdg
wwn-0×5001173100134914 -> ../../sdh
wwn-0x5001173100134a58 -> ../../sdp
wwn-0x500117310013671c -> ../../sdm
wwn-0×5001173100136740 -> ../../sdl
wwn-0x50011731001367ac -> ../../sdi
wwn-0x50011731001cdd58 -> ../../sdf

/dev/disk/by-path:
pci-0000:03:00.0-sas-0x5000c50055924206-lun-0 -> ../../sde
pci-0000:03:00.0-sas-0x5000c50055c5138a-lun-0 -> ../../sdd
pci-0000:03:00.0-sas-0x5000c50055c562ea-lun-0 -> ../../sda
pci-0000:03:00.0-sas-0x5000c500562ffd72-lun-0 -> ../../sdc
pci-0000:03:00.0-sas-0×5001173100134656-lun-0 -> ../../sdn
pci-0000:03:00.0-sas-0x500117310013465e-lun-0 -> ../../sdk
pci-0000:03:00.0-sas-0x500117310013468a-lun-0 -> ../../sdj
pci-0000:03:00.0-sas-0x500117310013471a-lun-0 -> ../../sdo
pci-0000:03:00.0-sas-0x500117310013490e-lun-0 -> ../../sdg
pci-0000:03:00.0-sas-0×5001173100134916-lun-0 -> ../../sdh
pci-0000:03:00.0-sas-0x5001173100134a5a-lun-0 -> ../../sdp
pci-0000:03:00.0-sas-0x500117310013671e-lun-0 -> ../../sdm
pci-0000:03:00.0-sas-0×5001173100136742-lun-0 -> ../../sdl
pci-0000:03:00.0-sas-0x50011731001367ae-lun-0 -> ../../sdi
pci-0000:03:00.0-sas-0x50011731001cdd5a-lun-0 -> ../../sdf

Real port (connection) persistence is not possible in that manner. A map of PWWN-to-Slot is required, and handling the system in case of a disk failure by non-expert is nearly impossible. A solution for that is to create matching udev rules which will allow handling disks per-port.

While there are (absolutely) better ways of doing it, time constrains require that I get it to work quick&dirty. The solution is based on lsscsi command, as the backend engine of the system, so make sure it exists on the system. I tend to believe that the system will not be able to scale out to hundreds of disks in its current design, but for my 16 disks (and probably for several tenths as well) – it works fine.

Add 60-persistent-disk-ports.rules to /etc/udev/rules.d/ (and omit the .txt suffix)

 

# By Ez-Aton, based partially on the built-in udev block device rule
# forward scsi device event to corresponding block device
ACTION=="change", SUBSYSTEM=="scsi", ENV{DEVTYPE}=="scsi_device", TEST=="block", ATTR{block/*/uevent}="change"

ACTION!="add|change", GOTO="persistent_storage_end"
SUBSYSTEM!="block", GOTO="persistent_storage_end"

# skip rules for inappropriate block devices
KERNEL=="fd*|mtd*|nbd*|gnbd*|btibm*|dm-*|md*", GOTO="persistent_storage_end"

# never access non-cdrom removable ide devices, the drivers are causing event loops on open()
KERNEL=="hd*[!0-9]", ATTR{removable}=="1", SUBSYSTEMS=="ide", ATTRS{media}=="disk|floppy", GOTO="persistent_storage_end"
KERNEL=="hd*[0-9]", ATTRS{removable}=="1", GOTO="persistent_storage_end"

# ignore partitions that span the entire disk
TEST=="whole_disk", GOTO="persistent_storage_end"

# for partitions import parent information
ENV{DEVTYPE}=="partition", IMPORT{parent}="ID_*"

# Deal only with SAS disks
KERNEL=="sd*[!0-9]|sr*", ENV{ID_SERIAL}!="?*", IMPORT{program}="/usr/local/sbin/detect_disk.sh $tempnode", ENV{ID_BUS}="scsi"
KERNEL=="sd*|sr*|cciss*", ENV{DEVTYPE}=="disk", ENV{TGT_PATH}=="?*", SYMLINK+="disk/by-target/disk-$env{TGT_PATH}"
#KERNEL=="sd*|cciss*", ENV{DEVTYPE}=="partition", ENV{ID_SERIAL}!="?*", IMPORT{program}="/usr/local/sbin/detect_disk.sh $tempnode"
KERNEL=="sd*|cciss*", ENV{DEVTYPE}=="partition", ENV{ID_SERIAL}=="?*", IMPORT{program}="/usr/local/sbin/detect_disk.sh $tempnode", SYMLINK+="disk/by-target/disk-$env{TGT_PATH}p%n"

ENV{DEVTYPE}=="disk", KERNEL!="xvd*|sd*|sr*", ATTR{removable}=="1", GOTO="persistent_storage_end"
LABEL="persistent_storage_end"

 
You will need to add (and make executable) the script detect_disk.sh in /usr/local/sbin. Again – remove the .txt suffix
 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
#!/bin/bash
# Written by Ez-Aton to assist with disk-to-port mapping
# $1 - disk device name
name=$1
name=${name##*/}
# Full disk
TGT_PATH=`/usr/bin/lsscsi | grep -w /dev/$name | awk '{print $1}' | tr -d ] | tr -d [`
if [ -z "$TGT_PATH" ]
then
	# This is a partition, so our grep fails
	name=`echo $name | tr -d [0-9]`
	TGT_PATH=`/usr/bin/lsscsi | grep -w /dev/$name | awk '{print $1}' | tr -d ] | tr -d [`
fi
echo "TGT_PATH=$TGT_PATH"

 
The result of this addition to udev would be a directory called /dev/disk/by-target containing links as follow:

/dev/disk/by-target:
disk-0:0:0:0 -> ../../sda
disk-0:0:1:0 -> ../../sdb
disk-0:0:10:0 -> ../../sdk
disk-0:0:11:0 -> ../../sdl
disk-0:0:12:0 -> ../../sdm
disk-0:0:13:0 -> ../../sdn
disk-0:0:14:0 -> ../../sdo
disk-0:0:15:0 -> ../../sdp
disk-0:0:2:0 -> ../../sdc
disk-0:0:3:0 -> ../../sdd
disk-0:0:4:0 -> ../../sde
disk-0:0:5:0 -> ../../sdf
disk-0:0:6:0 -> ../../sdg
disk-0:0:7:0 -> ../../sdh
disk-0:0:8:0 -> ../../sdi
disk-0:0:9:0 -> ../../sdj

The result is a persistent naming, based on real device ports.
 
I hope it helps. If you get to read it and have some suggestions (or a better use of udev, which I know is far from perfect in this case), I would love to hear about it.

IPSec VPN for mobile devices on Linux

Saturday, December 8th, 2012

I have had recently the pleasure and challenge of setting up VPN server for mobile devices on top of Linux. the common method to do so would be by using IPSec + L2TP, as these are to more common methods mobile devices allow, and it should work quite fine with other types of clients (although I did not test it) like Linux, Windows and Mac.

I have decided to use PSK (Pre Shared Key) due to its relative simplicity when handling multiple clients (compared to managing certificate per-device), and its relative simplicity of setup.

My VPN server platform is Linux, x86_64 (64 bit), Centos 6. Latest release, which is for the time being 6.3 and some updates.

I have used the following link as a baseline, and added some extra about IPTables, which was a little challenge, where I wanted good-enough security around this setup.

Initially, I wanted to use OpenSWAN, however, it does not allow easy integration with dynamic IP address, and its policy, while capable of being very precise, was not flexible enough to handle varying local IP address.

First – Add the following two repositories: Nikoforge, for Racoon (ipsec-tools), and EPEL for xl2tpd.

You can add them the following way:

rpm -ivH http://repo.nikoforge.org/redhat/el6/nikoforge-release-latest
yum -y install http://vesta.informatik.rwth-aachen.de/ftp/pub/Linux/fedora-epel/6/i386/epel-release-6-7.noarch.rpm
yum -y install ipsec-tools xl2tpd

Following that, create a script called /etc/racoon/init.sh:

#!/bin/sh
# set security policies
echo -e "flush;\n\
        spdflush;\n\
        spdadd 0.0.0.0/0[0] 0.0.0.0/0[1701] udp -P in  ipsec esp/transport//require;\n\
        spdadd 0.0.0.0/0[1701] 0.0.0.0/0[0] udp -P out ipsec esp/transport//require;\n"\
        | setkey -c
# enable IP forwarding
echo 1 > /proc/sys/net/ipv4/ip_forward

Make sure this script allows execution, and add it to /etc/rc.local

Racoon config /etc/racoon/racoon.conf looks like this for my setup:

path include "/etc/racoon";
path pre_shared_key "/etc/racoon/psk.txt";
path certificate "/etc/racoon/certs";
path script "/etc/racoon/scripts";
#log debug;
remote anonymous
{
      exchange_mode    aggressive,main;
      #exchange_mode    main;
      passive          on;
      proposal_check   obey;
      support_proxy    on;
      nat_traversal    on;
      ike_frag         on;
      dpd_delay        20;
      #generate_policy unique;
      generate_policy on;
      verify_identifier on;
      proposal
      {
            encryption_algorithm  aes;
            hash_algorithm        sha1;
            authentication_method pre_shared_key;
            dh_group              modp1024;
      }
      proposal
      {
            encryption_algorithm  3des;
            hash_algorithm        sha1;
            authentication_method pre_shared_key;
            dh_group              modp1024;
      }
}
sainfo anonymous
{
      encryption_algorithm     aes,3des;
      authentication_algorithm hmac_sha1;
      compression_algorithm    deflate;
      pfs_group                modp1024;
}

The PSK is kept inside /etc/racoon/psk.txt. It looks like this for me (changed password, duh!):

myHome   ApAssPhR@se

Both said files (/etc/racoon/racoon.conf and /etc/racoon/psk.txt) should have only-root permissions, aka 600.

Notice the bold myHome identifier. As the local address might change (either ppp dialup, or DHCP client), this one will be used instead of the local address identifier, as the common identifier of the connection. For Android devices, it will be defined as the ‘IPSec Identifier’ value.

We need to setup xl2tpd: Edit /etc/xl2tpd/xl2tpd.conf and have it look like this:

[global]
debug tunnel = no
debug state = no
debug network = no
ipsec saref = yes
force userspace = yes
[lns default]
ip range = 192.169.0.10-192.169.0.20
local ip = 192.169.0.1
refuse pap = yes
require authentication = yes
name = l2tpd
ppp debug = no
pppoptfile = /etc/ppp/options.xl2tpd
length bit = yes

The IP range will define the client VPN interface address. The amount should match the expected number of clients, or be somewhat larger, to be on the safe side. Don’t try to be a smart ass with it. Use explicit IP addresses. Easier that way. The “local” IP address should be external to the pool defined, or else a client might collide with it. I haven’t tried checking if xl2tpd allowed such configuration. You are invited to test, although it’s rather pointless.

Create the file /etc/ppp/options.xl2tpd with the following contents:

ms-dns 192.168.0.2
require-mschap-v2
asyncmap 0
auth
crtscts
lock
hide-password
modem
debug
name l2tpd
proxyarp
lcp-echo-interval 10
lcp-echo-failure 100

The ms-dns option should specify the desired DNS the client will use. In my case – I have an internal DNS server, so I wanted it to use it. You can either use your internal, if you have any, or Google’s 8.8.8.8, for example.

Almost done – you should add the relevant login info to /etc/ppp/chap-secrets. It should look like: “username” * “password” * . In my case, it would look like this:

# Secrets for authentication using CHAP
# client    server    secret            IP addresses
ez-aton        *    ”SomePassw0rd”        *

Select a good password. Security should not be taken lightly.

We’re almost done – we need to define IP forwarding, which can be done by adding the following line to /etc/sysctl.conf:

# Controls IP packet forwarding
net.ipv4.ip_forward = 1

and then running ‘sysctl -p’ to load these values.

Run the following commands, and your system is ready to accept connections:

chkconfig racoon on
chkconfig xl2tpd on
service racoon start
service xl2tpd start
/etc/racoon/init.sh

That said – we have not configured IPTables, in case this server acts as the firewall as well. It does, in my case, so I have had to take special care for the IPTables rules.

As my rules are rather complex, I will only show the rules relevant to the system, assuming (and this is important!) it is both the firewall/router and the VPN endpoint. If this is not the case, you should search for more details about forwarding IPSec traffic to backend VPN server.

So, my IPTables rules would be these three:

iptables -A INPUT -p udp --dport 500 -j ACCEPT
iptables -A INPUT -p udp --dport 4500 -j ACCEPT
iptables -A INPUT -p esp -j ACCEPT
iptables -A INPUT -p 51 -j ACCEPT # Not sure it's required, but too lazy to test without

This covers the IPSec part, however, we would not want the L2TP server to accept connections from the net, just like that, so it has its own rule, for port 1701:

iptables -A INPUT -p udp -m policy –dir in –pol ipsec -m udp –dport 1701 -j ACCEPT

You can save your current iptables rules (after checking that they work correctly) using ‘service iptables save’, or manually (backup your original rules to be on the safe side), and you’re all ready to go.

Good luck!

Bonding + VLAN tagging + Bridge – updated

Wednesday, April 25th, 2012

In the past I hacked around a problem with the order of starting (and with several bugs) a network stack combined of network bonding (teaming) + VLAN tagging, and then with network bridging (aka – Xen bridges). This kind of setup is very useful for introducing VLAN networks to guest VMs. This works well on Xen (community, Server), however, on RHEL/Centos 5 versions, the startup scripts (ifup and ifup-eth) are buggy, and do not handle this operation correctly. It means that, depending on the update release you use, results might vary from “everything works” to “I get bridges without VLANs” to “I get VLANs without bridges”.

I have hacked a solution in the past, modifying /etc/sysconfig/network-scripts/ifup-eth and fixing some bugs in it, however, both maintaining the fix on every release of ‘initscripts’ package has proven, well, not to happen…

So, instead, I present you with a smarter solution, better adept to updates supplied from time to time by RedHat or Centos, using predefined ‘hooks’ in the ifup scripts.

Create the file /sbin/ifup-pre-local with the following contents:

 

#!/bin/bash
# $1 is the config file
# $2 is not interesting
# We will start the vlan bonding before any bridge
 
DIR=/etc/sysconfig/network-scripts
 
[ -z "$1" ] &amp;&amp; exit 0
. $1
 
if [ "${DEVICE%%[0-9]*}" == "xenbr" ]
then
    for device in $(LANG=C egrep -l "^[[:space:]]*BRIDGE=\"?${DEVICE}\"?" /etc/sysconfig/network-scripts/ifcfg-*) ; do
        /sbin/ifup $device
    done
fi

You can download this scrpit. Don’t forget to change it to be executable. It will call ifup for any parent device of xenbr* device called at. If the parent device is already up, no harm is done. If the parent device is not up, it will be brought up, and then the xenbr device can start normally.