Xen Networking – Bonding with VLAN Tagging

The simple scripts in /etc/xen/scripts which manage networking are fine for most usages, however, when your server is using bonding together with VLAN tagging (802.11q) you should consider an alternative.

A PDF document written by Mark Nielsen, GPS Senior Consultant, Red Hat, Inc (I lost the original link, sorry) named “BOND/VLAN/Xen Network Configuration” as a service to the community, game me few insights on the subject. Following one of its references, I saw a bit more elegant method of doing a bridging setup under RedHat, which takes managing the bridges away from xend, and leaves it at the system level. Lets see how it’s done on RedHat style Linux distribution.

Manage your normal networking configurations

If you’re using VLAN tagging over bonding, than you should have to setup a bonding device (be it bond0) which has definitions such as this:

/etc/sysconfig/network-scripts/ifcfg-eth0 and /etc/sysconfig/network-scripts/ifcfg-eth1

DEVICE=eth0
BOOTPROTO=none
ONBOOT=yes
MASTER=bond0
SLAVE=yes
ISALIAS=no

/etc/sysconfig/network-scripts/ifcfg-bond0

DEVICE=bond0
BOOTPROTO=none
BONDING_OPTS=”mode=1 miimon=100″
ONBOOT=yes

This is rather stright-forward, and should be rather a default for such a setup. Now comes the more interesting part. Originally, the next configuration part would be bond0.2 and bond0.3 (in my example). The original configuration would have looked like this (this is in bold because I tend to fast-read myself, and tend to miss things too often). This is not how it should look when we’re done!

/etc/sysconfig/network-scripts/ifcfg-bond0.2 (same applies to ifcfg-bond0.3)

DEVICE=bond0.2
BOOTPROTO=static
IPADDR=192.168.0.2
NETMASK=255.255.255.0
ONBOOT=yes
VLAN=yes

Configure bridging

To setup a bridge device for bond0.2, replace the mentioned above ifcfg-bond0.2 with this new /etc/sysconfig/network-scripts/ifcfg-bond0.2

DEVICE=bond0.2
BOOTPROTO=static
ONBOOT=yes
VLAN=yes
BRIDGE=xenbr0

Now, create a new file /etc/sysconfig/network-scripts/ifcfg-xenbr0

DEVICE=xenbr0
BOOTPROTO=static
IPADDR=192.168.0.2
NETMASK=255.255.255.0
ONBOOT=yes
TYPE=bridge

Now, on network restart, the bridge will be brought up, holding the right IP address – all done by initscripts, with no Xen intervention. You will want to repeat the last the “Configure bridge” part for any additional bridge you want to be enabled for Xen machines.

Don’t let Xen bring any bridges up

This is the last part of our drill, and it is very important. If you don’t do it, you’ll get a nice networking mess. As said before, Xen (community), by default, can’t handle bondings or vlan tags, so it will attempt to create or modify bridges to eth0 or the likes. Edit /etc/xen/xend-config.sxp and remark any line containing a directive containing starting with “network-script“. Such a directive would be, for example

(network-script network-bridge)

Restart xend and restart networking. You should now be able to configure VMs to use xenbr0 and xenbr1, etc (according to your own personal settings).

Tags: , , , , , , ,

4 Responses to “Xen Networking – Bonding with VLAN Tagging”

  1. Running Systems » Blog Archive » Xen guests cannot serve NFS requirests Says:

    […] This should do the trick. It is required only on Dom0, and was tested to work well with my own method of configuring bonds and VLAN tags, as described in this post. […]

  2. Grant Says:

    Thanks for the guide, I followed it and it definately helped on my RHEL 5.2 system. My only question is are you able to get your bridge device (xenbr0) to come up when you reboot the server? I’ve noticed that when I reboot, despite the parameter in my bridge config being set to onboot=yes the bridge willl not come up. After the server boots, I have to issue “service network restart” and then the bridge will come up. Then I have to issue “service xend restrat” and start each virtual host before everything starts working. Just wondering if I’m missing something! Thanks again!

  3. ez-aton Says:

    I was required to modify RedHat’s ifup-eth script in /etc/sysconfig/network-scripts/
    This script is buggy, and should be fixed. I was inclined to submit a patch as it was changed through the upgrade to RHEL5.3. Will probably be changed several more times…

    Ez

  4. Tom Says:

    Here is how it is done in Centos 6 and KVM:
    http://henroo.wordpress.com/2011/08/09/forward-bond-interface-vlan-to-guest-kvm/

Leave a Reply