|
I was able to finally build my iSCSI server a while back and wanted to highlight some of the details of this particular system. I wanted to try to do something unique with this configuration that would help to support a higher-end workload than a typical server.
This article has been updated with some changes to the configuration.
The particular server I chose for my iSCSI system is a Dell 2950. it is a single Quad-Core box with an Intel E5420 processor. I loaded it up with 4GB of memory and upgraded to the Dell PERC 6i SAS RAID Controller. For network throughput purposes I aso added a dual-port Intel 1000PT NIC in addition to the two NIC ports on-board. I took it easy for Round 1 and only picked up 2 @ 400GB 10K SAS drives. They are a lot cheaper if I buy them from somewhere else after the fact.
My operating system of choice was Ubuntu 7.10 Server. Let me start by saying it would have absolutely been 10X easier to build this server had I used Ubuntu 6.06 Server. The iSCSI Enterprise Target software is not available in the universe repositories and had to be compiled, and only then after modification to the make file.
I chose the following partitioning scheme for this particular system:
| Partition |
Size |
| /boot |
200 MB |
| / |
8 GB |
|
4 GB |
| /home |
660 MB |
| UNFORMATTED |
197 GB |
| UNFORMATTED |
197 GB |
| UNFORMATTED |
197 GB |
| UNFORMATTED |
197 GB |
The unformatted volumes represent iSCSI LUNs that I will present to my ESX Servers.
Once the operating system was up and running I scoured the web for ways to get the iSCSI Enterprise Target software installed in Ubuntu 7.10. My searching led to the only place that had the proper steps, which was the Belgian web site l4l.be. Fortunately, source code and console commands are all in English, which is all I needed to find my particular problem. This article walked me through the exact steps to get the iSCSI Enterprise Target compiled and installed on my sytem. When downloading and configuring iSCSI Enterprise Target for the purposes of VMware ESX Server, make sure you are using version 0.4.15 or higher. There is a critical fix in this release that addresses the Release/Reserve issue highlighted by VMware Support.
I used the following configuration once I finally got iSCSI Enterprise Target up and running on my system:
Target iqn.2008-02.com.vmguru:vmware.p1
Lun 0 Path=/dev/sda5,Type=fileio,ScsiSN=VMGDSK-080209-00
Target iqn.2008-02.com.vmguru:vmware.p2
Lun 1 Path=/dev/sda6,Type=fileio,ScsiSN=VMGDSK-080209-01
Target iqn.2008-02.com.vmware:xenserver.p1
Lun 2 Path=/dev/sda7,Type=fileio,ScsiSN=VMGDSK-080209-02
Target iqn.2008-02.com.vmware:xenserver.p2
Lun 3 Path=/dev/sda8,Type=fileio,ScsiSN=VMGDSK-080209-03
One thing that some people may notice about the configuration is it is unique in the fact that I have specified a ScsiSN value to each LUN. While poking around and trying to get the stupid thing to properly build I saw that there was a README.vmware file in the build directory. I figured it might actually apply to what I was doing so decided to open it up. As I expected, it absolutely applied and made sense of some wierd issues I had seen in the past. I'll paste the contents of this file here:
iet vmware esx (v3) disk identification
=======================================
Disks, also known as LUNs, can be identified by the underlying SCSI
protocol using the vital product pages (VPD). Page 0x83 contains the
unit identification (required by SCSI spec), page 0x80 contains the
disk serial number page (optional as per SCSI spec).
iet provides a feature that allows one to define a ScsiId parameter
within the Lun statement. This parameter controls the value of page
0x83. Before SVN rev 72 (including 0.4.14) did not control page 0x80.
vmware and disk serial numbers
------------------------------
VMware ESX uses page 0x80 to identify disks. If two disks have the same
serial number, then vmware thinks there are multiple paths to that
disk. This becomes obvious if you have multiple targets defined in
iet or you have targets defined on different iet servers. This is an
issue with all versions prior and including release 0.4.14 of iet.
Target iqn.1990-01.edu.amherst.iscsi-target-development:vmware.t1
Lun 0 Path=/mnt/bigslice1,Type=fileio
Target iqn.1990-01.edu.amherst.iscsi-target-development:vmware.t2
Lun 0 Path=/mnt/bigslice2,Type=fileio
So, if the bigslice1 is 100G and bigslice2 is 200G, you will see one
LUN in vmware which might have the size 100G or 200G consisting of 2
paths to that disk. This gets really ugly if one path is down and it
then accesses a different disk.
Prior to SVN rev 75
-------------------
Before SVN rev 75 (this includes 0.4.14), iet emitted a 4 byte long
disk serial number consisting of spaces (0x20). In vmware you can see
this by looking at the Id line in /proc/vmware/scsi/hba40/...
Id: 20 20 20 20 56 49 52 54 55 41
As it turns out, vmware also appends this resulting Id to the first 6
bytes of the product identification field, which is VIRTUA
(VIRTUAL-DISK).
To avoid all this
-----------------
After and including SVN rev 75, you can configure serial numbers in
your ietd.conf:
Lun 0 Path=/dev/sdc,Type=fileio,ScsiSN=AMHDSK-061031-03
Note if you do not specify a ScsiSN, like all the previous versions,
iet emitted a 4 byte long disk serial number consisting of spaces
(0x20).
References:
-----------
http://www.vmware.com/community/message.jspa?messageID=364352
Todo Items
----------
VMware currently does not certify iet as a compatible iSCSI solution.
The open source community is trying to make iet as compatible as
possible.
Written by Steffen Plotner. Last updated 2006/12/05
With that out of the way I was finally able to start the iSCSI Enterprise Target software and go on my merry way with the advanced networking configuration. As I mentioned previously, I have 4 @ GB network connections in this system. I wanted to make this $2000 server outperform some of the far more expensive solutions on the market and figured my largest bottlenecks were my SCSI cache and my network connectivity. Well, network connectivity is easy enough to fix if you know where to look. I knew there was a bonding driver included with nearly all Linux distributions, I just needed to find a nice easy utility to control it. My searching led me to "ifenslave". This control program allows for the very simple configuration and use of the Linux Kernel's bonding driver.
Instead of typing up the step-by-step, I'll just point you to the best DOC out there that discusses the exact use and configuration of the utility. My configuration consisted of using my first on-board NIC for regular network communication on my VLAN 12. The other 3 NICs were all assigned to VLAN 13 and connected to my iSCSI switch. I created a bond (creatively labeled as "bond0") of these 3 NICs and used the "balance-alb" mode, which is "Adaptive Load Balancing". I found it interesting that this type of bond acts nearly identically to a VMware Virtual Switch in just about every fashion.
With the details and configurations highlighted above, I now have a VERY fast and VERY functional iSCSI server that has 4 @ 197 GB LUNs for my virtual machines. Two of those LUNs will be assigned to VMware ESX Hosts, and two will be assigned to Citrix XenServer hosts. Once I migrate my data off my old iSCSI Server, I will be rebuilding it with the same configuration and use it for development purposes for Microsoft Hyper-V and Sun Solaris xVM...that is assuming the rest of my hardware order ever arrives. I'm still 4 servers short and have been waiting very patiently.
For those playing along at home, the following pictures are what the current state of my lab now looks like (I know, my cabling is disgusting).
  
Readers have left 5 comments. No.1 Nice article
Nice article. Your lab is looking sweet. Be sure to let us know when you run out of power and start tripping circuit breakers, because that's what's coming next (without the proper preparations).
Been there, done that on the multiple iSCSI targets and VMware seeing them as the same target with multiple paths. I used a similar but slightly different configuration than you to get around the issue. In the end, both solutions work.
I've been using rPath Openfiler Linux for about 18 months or so and it has been pretty good. I haven't had any corruption issues and I'm running multiple ESX hosts with VMotion, DRS, HA, etc. I doubt it's running IET 0.4.15 or higher. I haven't been able to figure out how to check the version and nobody on the rPath or Sourceforge forums replies to my questions when I asked how. Do you know how by chance?
I'm particularly interested in your section on adding bonding because that's one thing I don't have but I'd like to try. I've wondered what my true disk I/O bottleneck is: Processor, Memory (doubtful), Array controller, disk, or network?
My iSCSI target is an HP Proliant ML570G2 which has
4x P4 XEON processors,
9GB RAM
SMART Array 6402 PCI-X array controller with read/write cache
12x 146GB Ultra320 SCSI 10kRPM hot pluggable drives
1x Broadcom GB NIC
My single iSCSI target is currently configured as one large RAID5 array with 1.5TB of available storage for ESX.
As you can see, I've got ample amounts of processor power, memory, plenty of disk spindles (although I wish they were 15kRPM), and the array controller is top shelf for that server model.
Now that I've seen your bonding solution, I'm going to give it a go and see if it improves performance.
Thanks,
Jas No.2 On that top server...
Oh oh Scott. On that top server, the power cable is supposed to make a right angle trun and then 180-wrap around that black clip to the right and then go into the cable management arm. This prevents the power cable from pulling straight out of the power supply when it's stressed with tension. HP servers and Cisco networking equipment have those clips too. No.3 OK On Power
I planned ahead on power. I have 2 @ 20 Amp breakers dedicated to the equipment. According to my PDUs, I'm only pushing 3-4 amps per breaker right now with only 4 more servers coming in (for now). I also have plenty of room to add more juice as necessary. No.4 What kind of LUN/iSCSI performance & iSCSI CHAP
You seem to have configured you're 400Gb SAS drives as a RAID0 Volume. Do you have some idea of what kind of read/write performance you get ? (maybe using something like hdspeed)
You don't seem to be using the CHAP authentication with you're iSCSI target. Maybe adding it to you're article might be a nice addition.
Just for info, I enabled CHAP authentication and the initiators.allow/initiator.deny filtering on my iSCSI Enterprise Target (0.4.15). I was first only allowing the Service Console IP Address, then only the VMkernel IP address. When you use initiators.allow, you need to enable both IP addresses, as the CHAP authentication comes over the Service Kernel network, and the iSCSI traffic over the VMkernel network.
Otherwise it's a Great article !! Thanks for link to the extensive Bonding info.
I'm doing the same iSCSI server with a HP ML150G5 server (QC5410 & 8GB) with 4x 1TB SATA (7200rpm) drives on a P400/512Mb in RAID5. The OS is a RHEL 5.2(beta) with a Xen kernel, so I can use the system both as iSCSI server and compare/test with RHEL Xen 3.1.2 integration.
No.5 Storage Performance
Erik,
After doing some testing, my disk performance simply sucks. I rebuilt the system and generated a new post about the changes I did. I also included CHAP configuration for my LUNs to provide a layer of security. In my home environment I didn't have a need for it so simply skipped the step. It is important for anyone looking to use this solution. |
