Technical Resources for Technical People
Header

SATA Direct Local Disk Access on Xenserver

July 8th, 2011 | Posted by techblog in Linux | Virtualization

There are several reasons why you may wish to attach a physical disk to your existing Xenserver installation, in our case we use it for both a permanent storage solution as well as moving large sets of data around (such as copying out virtual disks or data intensive backups). It’ll treat the SATA (or better yet eSATA) disk similarly to plugging in an external USB drive to the Xenserver host, but at much higher throughput.

One of our servers utilizes a 3 disk RAID5 array which is primarily used for VM and VM data storage. However we do have a NAS virtual machine that stores quite a lot of backup data; and rather then use up a bunch of space on our RAID array and subsequently have to transfer the data off again anyways, we prefer to use a single hard disk formatted in EXT3. This disk can be treated like any typical hard disk data store and is not Xenserver dependent. Once the drive reaches capacity, we can detach from the virtual machine, pull it out and swap in a new one.  (And of course mount it on any Linux box and access the data directly).

In order to get Xenserver to recognize the SATA drive we’ll need to create some symbolic links in the hypervisor’s /dev/xapi/block directory by editing the /etc/udev/rules.d/50-udev.rules file. (The udev daemon in the kernel properly abstracts USB and SATA while placing the appropriate node into the /dev directory).

Using /dev/sdb as the drive we wish to attach, add the following lines to /etc/udev/rules.d/50-udev.rules which are modified from similar USB mass storage detection rules.

ACTION=="add", KERNEL=="sdb", SYMLINK+="xapi/block/%k", RUN+="/bin/sh -c '/opt/xensource/libexec/local-device-change %k 2>&1 >/dev/null&'"
ACTION=="remove", KERNEL=="sdb", RUN+="/bin/sh -c '/opt/xensource/libexec/local-device-change %k 2>&1 >/dev/null&'"

You can add additional lines for additional devices (such as sdc, sdd) if you wish.

Save and reboot Xenserver. Your attached SATA hard disk should now be recognized as a removable storage device, which can now be attached to a virtual machine via the XenCenter console. It’ll be using an emulated SCSI controller instead of USB, which means full read/write throughput! Don’t forget to attach/detach the drive via XenCenter to/from the VM of your choice – while it’s physically “hot swapable”, you can’t just yank the physical drive out of a running VM that’s using it.

This change will be persistent through reboot, but remember that any product updates to the core Xenserver installation will likely overwrite these modifications, please bookmark this page for future reference.

 

You can follow any responses to this entry through the RSS 2.0 You can leave a response, or trackback.

43 Responses

  • netvope says:

    Which version of XenServer are these commands written for?

  • Pingback: Running Systems » Blog Archive » Mapping internal (SATA, SAS, RAID, etc) disks from XenServer host to VM

  • anonymous says:

    any difference for XenServer 6.x? thanks

  • anonymous?2 says:

    works with xen server 6.0.2

  • Metroshica says:

    In CentOS this works fine, but there is something to be aware of. You must attach the storage after you start the VM, though only for the first attachment. If you do not, and attach the storage before you turn on the VM, it seems that the kernel will not release its hold on the storage and give it to the OS. When you try to do anything to the storage (write, mount, fsck, etc) it will respond that the storage is either mounted, busy or in use by the system. As long as you attach it after the boot process, you will avoid this, and on future reboots this problem will not occur. This has been confirmed in CentOS 5.8 and 6.3 and there’s a good chance other Linux distros will have this issue as well.

  • Sven says:

    Works with XCP 1.6. Thanks
    Sven

  • Sacha says:

    yes, works with xen server 6.0.2 me too. thx.

  • xenman says:

    Not working on 6.1.0, I get the error message Unrecognized block type.

    Here is the tail end of my 50-udev.rules

    # mmc block devices
    ACTION==”add”, SUBSYSTEM==”mmc”, RUN+=”modprobe mmc_block”

    # SATA HOT PLUG DRIVES
    ACTION==”add”, KERNEL==”sdb1″, SYMLINK+=”xapi/block/%k”, RUN+=”/bin/sh -c ‘/opt/xensource/libexec/local-device-change %k 2>&1 >/dev/null&'”
    ACTION==”remove”, KERNEL==”sdb1″, RUN+=”/bin/sh -c ‘/opt/xensource/libexec/local-device-change %k 2>&1 >/dev/null&'”

    RUN+=”socket:/org/kernel/udev/monitor”

  • xenman says:

    OK never mind, changed the sdb1 from fdisk to sdb and everything is working fine on 6.1.0

  • Simon says:

    Would it be possible to use /dev/disk/by-id instead of sdX?

  • Chris says:

    Works at citrix xenserver 6.1. Thx!

  • Karl Martinsson says:

    How safe is this in terms of data integrity? Any increased chance of data loss or corruption? I would like to do something like this for ZFS.

  • Jonas says:

    Perfect! XenServer 6.2.

  • Stuart says:

    I’ve been trying to get this working on XenServer 6.2, it was fine in 6.0.2. Can I modify this a little for the changes in 6.2?

  • Peter says:

    Doing this on xenserver 6.2, works great except for one thing. I can not take snapshots of the VM these drives are attached to.

    I’ve tried doing plain snapshots from within xencenter as well as snapshots in alike backup (which allows me to exclude the attached drives from the snapshot) and it won’t take a snapshot. Gives me a “This SR does not support this operation”.

    Any ideas on how to get snapshots to work on these vm’s?

    • Peter says:

      I found out its a trade off… you can’t snapshot a VM with direct access drives attached like in this post. So I snapshot all my other VM’s, and my single VM w/ direct drive access gets backedup in oldschool linux methods (rsync, rsnapshot, crashplan, etc…)

    • JS says:

      Hi Ken,
      I just tried this on Xenserver 6.2.

      My local copy and export also fail, but give messages saying that there is not enough room on the destination. Considering I have mounted a 2TB+ SATA to the VM, that makes sense.

      Perhaps you are seeing a similar issue?

  • Peter says:

    Quick question, anyone know if you can do the same thing w/ mdadm raid drives (example /dev/md0)?

    Thanks

  • Mahesa says:

    What drive does it show up as in /dev on vm?

  • Ken says:

    Hello everyone,

    Thanks for all the post. I’m first time user of Xenserver 6.2, and not a guru of Linux cli. I’m trying to add a external SSD drive that is inside the external enclosure which attached to Esata port. had the same problem where Xenserver will not see the drive as storage. The server system is a dell server which can see the Esata card just fine and as well as the SSD drive. Just can’t see within Xenserver 6.2. I’m trying to build a hypervisor lab Citrix farm for education purpose.

    Reading thru the article but no glue as where to start. I can get into the console of the command line but what all the key steps I need to follow to make this work, it is Greek to me as of now. Thanks again for your helps.

  • makstex says:

    #mkdir /srv/block-devices

    #xe sr-create name-label=”Block devices” name-description=”Block devices” type=udev content-type=disk device-config:location=/srv/block-devices

    #ln -s /dev/sdb /srv/block-devices/sdb

    #xe sr-scan uuid=

    #xe vdi-list sr-uuid=

    Profit !

    • John says:

      this! this is way easy and works in Pre-Release XenServer 6.6 Dundee beta.1

    • Johannes says:

      Hello

      Attention!
      I lost all my data using this way of passthrough-ing a whole disk to a VM. Why? Cause Xenserver (I used version 7) has a limit of disk-size about 2TB. Each bigger drive will by destroyed without prompting. The data is still there, but the partition-table is corrupted, so you have to use something like testdisk to get your data back.
      I had to do long-lasting testdisk-recovery to get my data back. I’m not finished with it, but I have some effort.

      So please be sure to only passthrough-ing drives smaller than 2TB!

      Best regards

  • John says:

    This is working fine for me as well in Xenserver 6.5 with one exception. I am not able to install Xentools, and am stuck at the 3 drive max.

    I tried to attach another to the VM through CLI and and am getting this message “Error: Empty VBDs can only be made for type=CD”. I hunted around a bit and found what I think is the xencode here which shows the message where I am getting with what I think is the parameters. I’m not 100% sure why they are showing up “empty”.

    http://webcache.googleusercontent.com/search?q=cache:kes-ZJLTDgoJ:https://searchcode.com/codesearch/view/3661672/+&cd=1&hl=en&ct=clnk&gl=us

    Any ideas on how to get past this?

    Thanks!

    • Brendan says:

      This is interesting. I’ve been trying and failing to passthrough disks via a PCI storage controller (Mavell based). Can anyone confirm whether this solution is limited to three disks maximum as John referred to above (Thursday, 5th March 2015)? Is this because this is the max number of removable storage devices Xen allows?

  • Tevlon says:

    Confirmed to still work on Xenserver 6.5

    • Kumudini says:

      Hi,

      I am trying to attach a raid array (RAID 5) that is around 5.4 TB, as direct access to a VM. I am doing this in XenServer 6.5. Unless I create an SR for the /dev/sdf, I do not see this as a storage under the host server in XenCenter. Even then, I am not able to attach this as a disk. Any help is much appreciated. Thanks.

  • Josef says:

    Hello, interesting article.
    I want to use the esata drive only for save data of a nas that will run on internal disks, and the data on the external esata raid in order to have separated data.
    I think that using a esata box with embed raid controller is a more stable solution than a raid managed from the nas virtual machine, but can be important to have a sata raid controller that can be analyzed from the nas vps… To see raid status etc..
    Anyone know a nice esata raid box with a controller compatible with a Debian system nas like openmedia vault?

    I have a Dell t310 server with 4 sas disks and Xen server.

    Thanks for the attention, best regards.

  • John says:

    Does not seem to be working anymore as of Pre-Release XenServer 6.6 Dundee beta.1

  • Pingback: [Sammelthread] HP ProLiant G8 G1610T/G2020T/i3-3240/E3-1220Lv2 MicroServer - Seite 214

  • Pingback: XENSERVER 6.5 VS VMWARE ESXI 6.0 ON HIGH END CONSUMER GRADE HARDWARE – Free Software Servers



Leave a Reply to Jeff Cancel reply

Your email address will not be published. Required fields are marked *