VMGuru Lab: Storage Server

Written by Scott Herold

Wednesday, 19 March 2008 05:39

I was able to finally build my iSCSI server a while back and wanted to highlight some of the details of this particular system. I wanted to try to do something unique with this configuration that would help to support a higher-end workload than a typical server.

This article has been updated with some changes to the configuration.

The particular server I chose for my iSCSI system is a Dell 2950. it is a single Quad-Core box with an Intel E5420 processor. I loaded it up with 4GB of memory and upgraded to the Dell PERC 6i SAS RAID Controller. For network throughput purposes I aso added a dual-port Intel 1000PT NIC in addition to the two NIC ports on-board. I took it easy for Round 1 and only picked up 2 @ 400GB 10K SAS drives. They are a lot cheaper if I buy them from somewhere else after the fact.

My operating system of choice was Ubuntu 7.10 Server. Let me start by saying it would have absolutely been 10X easier to build this server had I used Ubuntu 6.06 Server. The iSCSI Enterprise Target software is not available in the universe repositories and had to be compiled, and only then after modification to the make file.

I chose the following partitioning scheme for this particular system:

Partition	Size
/boot	200 MB
/	8 GB
	4 GB
/home	660 MB
UNFORMATTED	197 GB
UNFORMATTED	197 GB
UNFORMATTED	197 GB
UNFORMATTED	197 GB

The unformatted volumes represent iSCSI LUNs that I will present to my ESX Servers.

Once the operating system was up and running I scoured the web for ways to get the iSCSI Enterprise Target software installed in Ubuntu 7.10. My searching led to the only place that had the proper steps, which was the Belgian web site l4l.be. Fortunately, source code and console commands are all in English, which is all I needed to find my particular problem. This article walked me through the exact steps to get the iSCSI Enterprise Target compiled and installed on my sytem. When downloading and configuring iSCSI Enterprise Target for the purposes of VMware ESX Server, make sure you are using version 0.4.15 or higher. There is a critical fix in this release that addresses the Release/Reserve issue highlighted by VMware Support.

I used the following configuration once I finally got iSCSI Enterprise Target up and running on my system:

Target iqn.2008-02.com.vmguru:vmware.p1
Lun 0 Path=/dev/sda5,Type=fileio,ScsiSN=VMGDSK-080209-00
Target iqn.2008-02.com.vmguru:vmware.p2
Lun 1 Path=/dev/sda6,Type=fileio,ScsiSN=VMGDSK-080209-01
Target iqn.2008-02.com.vmware:xenserver.p1
Lun 2 Path=/dev/sda7,Type=fileio,ScsiSN=VMGDSK-080209-02
Target iqn.2008-02.com.vmware:xenserver.p2
Lun 3 Path=/dev/sda8,Type=fileio,ScsiSN=VMGDSK-080209-03

One thing that some people may notice about the configuration is it is unique in the fact that I have specified a ScsiSN value to each LUN. While poking around and trying to get the stupid thing to properly build I saw that there was a README.vmware file in the build directory. I figured it might actually apply to what I was doing so decided to open it up. As I expected, it absolutely applied and made sense of some wierd issues I had seen in the past. I'll paste the contents of this file here:

iet vmware esx (v3) disk identification
=======================================

Disks, also known as LUNs, can be identified by the underlying SCSI
protocol using the vital product pages (VPD). Page 0x83 contains the
unit identification (required by SCSI spec), page 0x80 contains the
disk serial number page (optional as per SCSI spec).

iet provides a feature that allows one to define a ScsiId parameter
within the Lun statement. This parameter controls the value of page
0x83. Before SVN rev 72 (including 0.4.14) did not control page 0x80.

vmware and disk serial numbers
------------------------------
VMware ESX uses page 0x80 to identify disks. If two disks have the same
serial number, then vmware thinks there are multiple paths to that
disk. This becomes obvious if you have multiple targets defined in
iet or you have targets defined on different iet servers. This is an
issue with all versions prior and including release 0.4.14 of iet.

Target iqn.1990-01.edu.amherst.iscsi-target-development:vmware.t1
Lun 0 Path=/mnt/bigslice1,Type=fileio

Target iqn.1990-01.edu.amherst.iscsi-target-development:vmware.t2
Lun 0 Path=/mnt/bigslice2,Type=fileio

So, if the bigslice1 is 100G and bigslice2 is 200G, you will see one
LUN in vmware which might have the size 100G or 200G consisting of 2
paths to that disk. This gets really ugly if one path is down and it
then accesses a different disk.

Prior to SVN rev 75
-------------------
Before SVN rev 75 (this includes 0.4.14), iet emitted a 4 byte long
disk serial number consisting of spaces (0x20). In vmware you can see
this by looking at the Id line in /proc/vmware/scsi/hba40/...

Id: 20 20 20 20 56 49 52 54 55 41

As it turns out, vmware also appends this resulting Id to the first 6
bytes of the product identification field, which is VIRTUA
(VIRTUAL-DISK).

To avoid all this
-----------------
After and including SVN rev 75, you can configure serial numbers in
your ietd.conf:

Lun 0 Path=/dev/sdc,Type=fileio,ScsiSN=AMHDSK-061031-03

Note if you do not specify a ScsiSN, like all the previous versions,
iet emitted a 4 byte long disk serial number consisting of spaces
(0x20).

References:
-----------
http://www.vmware.com/community/message.jspa?messageID=364352

Todo Items
----------
VMware currently does not certify iet as a compatible iSCSI solution.
The open source community is trying to make iet as compatible as
possible.

Written by Steffen Plotner. Last updated 2006/12/05

With that out of the way I was finally able to start the iSCSI Enterprise Target software and go on my merry way with the advanced networking configuration. As I mentioned previously, I have 4 @ GB network connections in this system. I wanted to make this $2000 server outperform some of the far more expensive solutions on the market and figured my largest bottlenecks were my SCSI cache and my network connectivity. Well, network connectivity is easy enough to fix if you know where to look. I knew there was a bonding driver included with nearly all Linux distributions, I just needed to find a nice easy utility to control it. My searching led me to "ifenslave". This control program allows for the very simple configuration and use of the Linux Kernel's bonding driver.

Instead of typing up the step-by-step, I'll just point you to the best DOC out there that discusses the exact use and configuration of the utility. My configuration consisted of using my first on-board NIC for regular network communication on my VLAN 12. The other 3 NICs were all assigned to VLAN 13 and connected to my iSCSI switch. I created a bond (creatively labeled as "bond0") of these 3 NICs and used the "balance-alb" mode, which is "Adaptive Load Balancing". I found it interesting that this type of bond acts nearly identically to a VMware Virtual Switch in just about every fashion.

With the details and configurations highlighted above, I now have a VERY fast and VERY functional iSCSI server that has 4 @ 197 GB LUNs for my virtual machines. Two of those LUNs will be assigned to VMware ESX Hosts, and two will be assigned to Citrix XenServer hosts. Once I migrate my data off my old iSCSI Server, I will be rebuilding it with the same configuration and use it for development purposes for Microsoft Hyper-V and Sun Solaris xVM...that is assuming the rest of my hardware order ever arrives. I'm still 4 servers short and have been waiting very patiently.

For those playing along at home, the following pictures are what the current state of my lab now looks like (I know, my cabling is disgusting).