XenServer 6.5 PCI-Passthrough

While searching the web for how to perform PCI-Passthrough on XenServers, we mostly get info about previous versions. Since I have just completed setting up PCI-Passthrough on XenServer version 6. 5 (with recent update 8, just to give you some notion of the exact time frame), I am sharing it here.

Hardware: Cisco UCS blades, with fNIC. I wish to pass through two FC HBAs into a VM (it is going to act as a backup server, and I need it accessing the FC tape). While all my XenServers in this pool have four (4) FC HBAs, this particular XenServer node has six (6). I am intending the first four for SR communication and the remaining two for the PCI Passthrough process.

This is the output of ‘lspci | grep Fibre’:

0b:00.0 Fibre Channel: Cisco Systems Inc VIC FCoE HBA (rev a2)
0c:00.0 Fibre Channel: Cisco Systems Inc VIC FCoE HBA (rev a2)
0d:00.0 Fibre Channel: Cisco Systems Inc VIC FCoE HBA (rev a2)
0e:00.0 Fibre Channel: Cisco Systems Inc VIC FCoE HBA (rev a2)
0f:00.0 Fibre Channel: Cisco Systems Inc VIC FCoE HBA (rev a2)
10:00.0 Fibre Channel: Cisco Systems Inc VIC FCoE HBA (rev a2)

So, I want to pass through 0f:00.0 and 10:00.0. I had to add to /boot/extlinux.conf the following two entries after the word ‘splash’ and before the three dashes:

pciback.hide=(0f:00.0)(10:00.0) xen-pciback.hide=(0f:00.0)(10:00.0)

Initially, and contrary to the documentation, the parameter pciback.hide had no effect. As soon as the VM started, the command ‘multipath -l‘ would hang forever (or until hard reset to the host).

To apply the settings above, run (for a good measure. Don’t think we need it, but did not read anything about it): ‘extlinux -i /boot‘ and then reboot.

Now, when the host is back, we need to add the devices to the VM. Make sure that the VM is in ‘off’ state before doing that. Your command would look like this:

xe vm-param-set uuid=<VM UUID> other-config:pci=0/0000:0f:00.0,0/0000:10:00.0

The expression ‘0/0000’ is required. You can search for its purpose, however, in most cases, your value would look exactly like mine – ‘0/0000’

Since my VM is Windows, here it almost ends: Start the VM, and if it boots correctly, Install Cisco VIC into it, as if it were a physical host. You’re done.

Tags: , , , , , , ,

10 Responses to “XenServer 6.5 PCI-Passthrough”

  1. mx Says:

    Finally a guide that’s current!!
    I am so confused with the numbering with this for instance on the kernel command line you have (0f:00.1) but in the xe vm=param line you have 0/0000:0f:00.0 and no amount of research tells me how that can be ie the difference between the .1 and the .0 I mean is there an auto increment from the root device to the actual device? like does it map to .0 in xen but the kernel sees it as .1?

    I have almost the exact same need but I am trying to pass the whole usb controller (with a usb hard disk attached) to a windows server 2012r2 vm or at the very least a procedure to pass the actual disk so I can unplug it on the host and move it to another windows machine with the file system intact and not a lvm vhd .

  2. ez-aton Says:

    This was a mistake of mine. I just corrected it. So it is 0f:00.0 and 10:00.0. You are most correct.

    About transferring the disk – instead of that, you can connect the USB as RAW (see my previous posts) and then you will not go through LVM, but connect block device directly to Windows. Passing-through the USB seems like the more complex method of achieving it.

    Ez

  3. Jesse Says:

    Thank you, Thank you, THANK YOU!

    I’ve been searching for this exact post all week. Works exactly as it ought, to forward my front panel USB ports (and the USB UltraPro security dongle) from my physical server – a Dell R430 – to the client VM, which is MS Server 2012.

    FYI, and to help any other googlers… 🙂

    [root@XEN2 ~]# lspci | grep USB
    00:1a.0 USB controller: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #2 (rev 05)
    00:1d.0 USB controller: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #1 (rev 05)

    edited the .conf file with pciback.hide=(00:1a.0) xen-pciback.hide=(00:1a.0)

    rebooted, then:
    [root@XEN2 ~]# xe vm-param-set uuid= other-config:pci=0/0000:00:1a.0

    Then booted into the VM, and the front panel USB ports passed through exactly right.

  4. AJ Says:

    You’ll saints. Jesse…this is literally exactly what I needed to do…same server and everything. Thanks all!

  5. AJ Says:

    Spoke too soon. Adding the code to extlinux.conf causes a boot loop. I see the Cirtrix | XenServer logo screen but before the progress bar appears the server reboots. Any suggestions?

  6. AJ Says:

    So I tried the other USB controller 00:1d.0 (Presumably the one on the rear). I was able to add it to the config without getting a boot loop, and I was able to add it to the guest. I now have a device on my guest called “Standard Enhanced PCI to USB Host Controller”. So it appears it’s working. But, nothing I plug into the front or rear USB ports shows up on my client. Any thoughts?

  7. Andreas Balogh Says:

    Hi,

    your recipe does not work for me. After configuring /boot/extlinux.conf
    “… splash pciback.hide=(00:14.0)(00:1a.0)(00:1d.0) xen-pciback.hide=(00:14.0)(00:1a.0)(00:1d.0) — …”
    and
    xe vm-param-set uuid= other-config:pci=0/0000:00:14.0
    I get
    [root@xen1 ~]# xe vm-start uuid=
    The server failed to handle your request, due to an internal error. The given message may give details useful for debugging the problem.
    message: xenopsd internal error: Xenctrl.Error(“38: Function not implemented”)

    According to the xenserver forum there is no PCI passthrough on xenserver 6.5 (http://discussions.citrix.com/topic/291472-pci-passthrough-with-free-xenserver/).

  8. Vit Zikmund Says:

    FYI, there’s also a possibility to do all this without a reboot:
    http://wiki.xen.org/wiki/Xen_PCI_Passthrough#Dynamic_assignment_with_sysfs_.28when_xen-pciback_is_loaded_as_a_module_and_NOT_compiled_into_the_kernel.29
    BTW, don’t listen to the headline, I was doing this on a vanilla (no kernel mods) xensever 6.5. apparently, the xen-pciback module is compiled inside the kernel, but that doesn’t restrict you from using it like it wouldn’t. Mind though, that this is not persistent, so in order to survive reboot, you naturally have to edit the /boot/extlinux.conf

    Regards,
    Vit

  9. Dave B Says:

    Hi.

    Tried this on a Xenserver 6.5 machine and it failed to boot correctly – citrix xen splash came up for a few seconds then rebooted, and went on doing so.

    [root@xenserver-5025 ~]# lsb_release -a
    LSB Version: :core-4.0-amd64:core-4.0-noarch
    Distributor ID: XenServer
    Description: XenServer release 6.5.0-90233c (xenenterprise)
    Release: 6.5.0-90233c
    Codename: xenenterprise
    [root@xenserver-5025 ~]# uname -a
    Linux xenserver-5025 3.10.0+2 #1 SMP Wed Mar 11 11:20:46 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux

    Fix was to interrupt the boot cycle and remove the xen-pciback.hide directive, leaving only the pciback.hide one.

    Finding my target hardware. Use dmesg to find the USB device (I’m specifically after the AUDIO interface):

    $dom0# dmesg | grep USB

    [ 23.995300] hid-generic 0003:05E3:FFE0.0001: input,hidraw0: USB HID v1.00 Keyboard [HID 05e3:ffe0] on usb-0000:00:1a.0-1/input0
    [ 24.002032] hid-generic 0003:05E3:FFE0.0002: input,hidraw1: USB HID v1.00 Device [HID 05e3:ffe0] on usb-0000:00:1a.0-1/input1
    [ 24.002353] hid-generic 0003:05E3:FFE0.0003: input,hidraw2: USB HID v1.00 Mouse [HID 05e3:ffe0] on usb-0000:00:1a.0-1/input2
    [ 24.346124] usb 5-1: new full-speed USB device number 3 using uhci_hcd
    [ 24.679164] input: Burr-Brown from TI USB Audio CODEC as /devices/pci0000:00/0000:00:1d.1/usb5/5-1/5-1:1.3/input/input5
    [ 24.679275] hid-generic 0003:08BB:2902.0004: input,hidraw3: USB HID v1.00 Device [Burr-Brown from TI USB Audio CODEC ] on usb-0000:00:1d.1-1/input3

    Tells me the bus address for the Audio interface is 00:1d.1

    Here’s my full entry in /boot/extlinux.conf (note the missing xen-pciback)

    $dom0# grep pci /boot/extlinux.conf
    append /boot/xen.gz dom0_mem=3072M,max:3072M watchdog dom0_max_vcpus=6 crashkernel=128M@32M cpuid_mask_xsave_eax=0 console=vga vga=mode-0x0311 — /boot/vmlinuz-3.10-xen root=LABEL=root-aytknelj ro hpet=disable xencons=hvc console=hvc0 console=tty0 quiet vga=785 splash pciback.hide=(00:1d.1) — /boot/initrd-3.10-xen.img

    Then I applied the xe vm-param-set as described above, targeting one of my Ubuntu VMs (use your own UUID! – from “xe vm-list”):

    $dom0#: xe vm-param-set uuid=c82a307f-188d-931c-d7f6-f89545f5c03f other-config:pci=0/0000:00:1d.1

    I rebooted my Ubuntu VM, and now I get the following:

    $ubuntu# dmesg | grep USB (filtered)

    [ 1.572035] usb 2-1: new full-speed USB device number 2 using uhci_hcd
    [ 1.895952] usb 2-1: New USB device found, idVendor=08bb, idProduct=2902
    [ 1.897194] usb 2-1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
    [ 1.898370] usb 2-1: Product: USB Audio CODEC
    [ 1.906996] input: Burr-Brown from TI USB Audio CODEC as /devices/pci0000:00/0000:00:05.0/usb2/2-1/2-1:1.3/input/input6
    [ 1.909581] hid-generic 0003:08BB:2902.0002: input,hidraw1: USB HID v1.00 Device [Burr-Brown from TI USB Audio CODEC ] on usb-0000:00:05.0-1/input3

    And finally, the lsusb command that’s available in Ubuntu…

    npf@ubuntu:~$ lsusb
    Bus 002 Device 002: ID 08bb:2902 Texas Instruments PCM2902 Audio Codec
    Bus 002 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
    Bus 001 Device 002: ID 0627:0001 Adomax Technology Co., Ltd
    Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub

    JOB DONE!

    Thanks to everyone on this thread.

  10. etzion Says:

    I am sorry for the late approval. It has been rough around here.
    Thanks for your comments and input. Not a simple task, and PCI passthrough behaves differently for different hardware devices. In my case – I put HBA controllers in passthrough mode, while some others (of the same drivers!) had to remain available to the host. This made it somewhat more tricky.

    I did not get boot loops, although in an HA cluster (three nodes) I have had a case where the host could not join the XenServer HA cluster after update, and I have had to revert the extlinux.conf file and only after a successful reboot, to re-apply it and reboot. This is awkward and annoying, but it is not for me to solve their bugs 🙂

Leave a Reply