After reviewing a comment on my previous post from Erik Bussink I decided to better test my iSCSI solution as I wasn't overly satisfied with what I was seeing. I ran a few rounds of one of my favorite basic I/O write tests #dd if=/dev/zero of=/tmp/test.log bs=1024k count=1000. This command showed my that my throughput absolutely sucked on this relatively beefy system. I was pushing only about 15-17 MB/Sec on a controller and drives that should be able to perform significantly better than that.
I went back to the drawing board and scrapped the iSCSI server. There was nothing overly important on it besides configuration files, which I copied off prior to the wipe. I went into the storage controller and instead of configuring a RAID-0 across the two 400 GB drives, I went with two standalone drives. This also gave me the change to modify my partioning a little bit to give more space to the root filesystem to make sure I could get some ISO images over, as I intend to make this a PXE server at some point as well. The new partition layout is as follows: | Partition |
Size |
DISK SDA
| |
| /boot |
200 MB |
| / |
20 GB |
<SWAP>
|
4 GB |
| UNFORMATTED |
187.6 GB |
| UNFORMATTED |
187.6 GB |
| DISK SDB | |
| UNFORMATTED |
199.7 GB |
| UNFORMATTED |
199.7 GB |
The very first thing I did after completing the server build and required updates was check the disk I/O speed using different variations of block size and block count from my previous dd command. I was now pushing data at over 135 MB/Sec...a slighly more desirable number. In addition, I noticed that there was an update to the iSCSI Enterprise Target software, which brings it up to build 0.4.16. I decided to compile the unmodified source with this new build and it worked flawlessly. There was no need to make any modifications to get it to properly compile. Everything started up without any complaint and I was again ready to map my iSCSI drives. Based on a recommendation, again from Erik Bussink, I also configured CHAP authentication for my LUNs as a security measure. As a result, my new ietd.conf file looks like the following: Target iqn.2008-02.com.vmguru:hyper-v.p1
Lun 0 Path=/dev/sda5,Type=fileio,ScsiSN=VMGDSK-080209-00
IncomingUser hyper-v 12charpasswd
Target iqn.2008-02.com.vmguru:xenserver.p1
Lun 1 Path=/dev/sda6,Type=fileio,ScsiSN=VMGDSK-080209-01
IncomingUser xenserver 12charpasswd
Target iqn.2008-02.com.vmguru:vmware.p1
Lun 2 Path=/dev/sdb1,Type=fileio,ScsiSN=VMGDSK-080209-02
IncomingUser vmware 12charpasswd
Target iqn.2008-02.com.vmguru:vmware.p2
Lun 3 Path=/dev/sdb2,Type=fileio,ScsiSN=VMGDSK-080209-03
IncomingUser vmware 12charpasswd
My wife picked the perfect week to head out to Spain to visit her sister. My final four servers have arrived so I will have a chance to get them installed into the rack and hopefully get some form of hypervisor on them before the weekend. Once she gets back it will be more difficult to escape for the several hours necessary to do a proper job of racking and cabling the new servers.
One person has commented on this article. No.1 Thank you for posting your results
I hadn't actually bench tested my disk I/O prior to reading your results.
On my Openfiler iSCSI target, I performed the same test you did using a local non-VMFS3 partition. I used /tmp/ as you did. Refer to your previous lab articles on my iSCSI setup.
Using iostat:
First try I got 141MB written/sec. @ 84% CPU utilization
Second try I got 124MB written/sec. @ 68% CPU utilization
I then used the same dd command on a VMFS3 volume via vmhba32 (the ESX swISCSI adapter) over a single Gb NIC.
Result was 67MB written/sec. @ 68% CPU utilization.
Will be interesting to see how much bonding NICs changes the above number for swISCSI transfer. There's still some overhead in the ESX swISCSI adapter.
Jas |
