ReadyNAS 3100 iSCSI'd to ESX 4.1 vCenter, very slow

[The Duke of Ice]

I am experiencing very slow VM's performance when using the ReadyNAS 3100 (4.2.17) as a iSCSI Datastore on my VM's.
ESX 4.1 + vCenter 4.1 + ReadyNAS 3100 in private environment.
I have my ESX VMKernal on its own segment connected directly to the ReadyNAS (Single NIC, secondary NIC is used for Web Interface), nothing in the middle.
There are only 20 VM's running on my server, a Dual Xeon 5650 thats only using 5% of CPU and 60% Memory (at load) with my entire environment up and running.

I've tried Teaming both NIC's on ReadyNAS, with no visible difference. Setup TEAM'd VM Kernal no difference, after upgrading to 4.2.17 I did get some faster iSCSI performance, but mainly just eliminated my frequent datastore disconnects from happening.

I am not using Jumbo Frames or any of the advanced features, simply direct connection from Server to Storage.

Am am setting up an nfs store tonight and migrating some VM's over to it to see if I get some performance increases.

What other options are available to me for performance increases? By separating my VM Kernal and storage to their own segments, I believe I've eliminated any interference issues from getting in the way. Upgraded storage and server to latest code with no visible difference. This is a trade show server, so waiting 15+ seconds for the start menu to draw is not acceptable.

Does anyone have similar experiences or any recommendations that I can do in order to speed this up?

Thanks in advance.

-Update-
I have 8 Server VM's and 12 Client VM's.
After moving 4 Server VM's into my NFS Store, my VM speeds are faster than previously experienced.

Have I really reached the upper limit of iSCSI with 20 VM's concurrently running? (This is my first NAS, as before I defaulted to onboard 15K SAS for quick local storage).

[VMColonel]

If you haven't already, take a look at this page:
http://virtualgeek.typepad.com/virtual_ ... phere.html

It was written for/by EMC and NetApp, but most of the theory still applies.

If it were me, I'd be tempted to run iometer on a single VM, and see what kind of rate (IOPs - Cmd's / second) and bandwidth (MB/s) you are pushing, and at what latency (DAVG)

As a SUPER general rule of thumb, you'll want your DAVG counters (from esxtop , then d for disk mode) to be <20 for good performance. I like to see <10. Over 50 and you'll start to notice "slowness"

And lastly, try Windows 2008, I believe it is automagically aligned, for more information on performance best practices, please see here:
http://www.vmware.com/pdf/Perf_Best_Pra ... ere4.0.pdf

Welcome, and good luck

[DAT]

I'm running a pair of environments, one with 2 X ReadyNAS 2100s with 4 TB raw each, 2 X Netgear smart switches, and 3 X Dell 2900 servers running ESX 3.5 u4, and about 25 VMs hosted on NFS shares on the two 2100s. I get fantastic performance out of this set up. Really fast disk access with very low latency. Beats some of the low end NetApps that I used to use. The two ReadyNAS 2100s are still on 4.2.13.

Then I have a near identical setup (I've gone through all of the switch, ESX and ReadyNAS configs over and over again to make sure I haven't missed something) on another network. The main difference is that in the slow environment listed below I have a two port LAG directly connecting the two switches, and on the high performance environment listed above I have a 2 port LAG on each switch that connects back to a "core switch", so really one additional hop, yet much lower latency.

The slow performance network is 2 X ReadyNAS 2100s with 8 TB raw each, 2 X Netgear smart switches, and 3 X Dell r610 servers running ESXi 4.1 u1, and only about 9 VMs hosted on NFS shares on the two 2100s. I get horribly slow performance out of both of them, which I can now see is directly related to incredibly high latency accessing the NFS shares on the ReadyNAS. I also got horribly slow performance when I was running the VMs from the much older HP DL380 G3s (same servers that VMware uses/used for their VCP classes) with ESX 3.5 u4.

Before replacing the 8 year old servers I had thought the slowness was due to the servers being old and cooked, but now, with the new servers I realize that it's a SAN latency issue, since ESXi 4.1 u1 shows you this in the performance monitoring.

When I was using the old HP servers with 3.5, the ReadyNAS had 4.2.13 on both of them. I then upgraded to 4.2.17 on both of them, still had high latency, and they still are running 4.2.17 while being connected to from the new Dell r610 servers.

Your original issue is probably the exact same thing, and shouldn't have anything to do with iSCSI vs NFS, but is a latency issue accessing the ReadyNAS.

I have not been able to figure out what is causing this incredibly high latency, but I agree, it's not normal, nor is it anywhere near acceptable for trying to run servers from it.

Netgear, can you guys take a look at this for us?

[techdruid]

For some reason I've not seen this simple setting posted on any relevant searches for NFS Latency on the ReadyNAS. So here's one thing to check if you have an NFS datastore on your ReadyNAS and are experiencing unusually high latency.

http://techdruid.com/2012/06/03/vmware-readynas-nfs-and-extremely-high-latency/

Log in to your ReadyNAS, then go to "Shares -> Share Listing" and click on the NFS edit icon for the share that your ESXi host is mapped to. Scroll down to the Advanced NFS Options and ensure that the "Enable sync mode" is UNCHECKED.