ReadyNAS 3100 iSCSI'd to ESX 4.1 vCenter, very slow

ReadyNAS 3100 iSCSI'd to ESX 4.1 vCenter, very slow

Postby The Duke of Ice » Tue May 17, 2011 6:22 pm

I am experiencing very slow VM's performance when using the ReadyNAS 3100 (4.2.17) as a iSCSI Datastore on my VM's.
ESX 4.1 + vCenter 4.1 + ReadyNAS 3100 in private environment.
I have my ESX VMKernal on its own segment connected directly to the ReadyNAS (Single NIC, secondary NIC is used for Web Interface), nothing in the middle.
There are only 20 VM's running on my server, a Dual Xeon 5650 thats only using 5% of CPU and 60% Memory (at load) with my entire environment up and running.

I've tried Teaming both NIC's on ReadyNAS, with no visible difference. Setup TEAM'd VM Kernal no difference, after upgrading to 4.2.17 I did get some faster iSCSI performance, but mainly just eliminated my frequent datastore disconnects from happening.

I am not using Jumbo Frames or any of the advanced features, simply direct connection from Server to Storage.

Am am setting up an nfs store tonight and migrating some VM's over to it to see if I get some performance increases.

What other options are available to me for performance increases? By separating my VM Kernal and storage to their own segments, I believe I've eliminated any interference issues from getting in the way. Upgraded storage and server to latest code with no visible difference. This is a trade show server, so waiting 15+ seconds for the start menu to draw is not acceptable.

Does anyone have similar experiences or any recommendations that I can do in order to speed this up?

Thanks in advance.

-Update-
I have 8 Server VM's and 12 Client VM's.
After moving 4 Server VM's into my NFS Store, my VM speeds are faster than previously experienced.

Have I really reached the upper limit of iSCSI with 20 VM's concurrently running? (This is my first NAS, as before I defaulted to onboard 15K SAS for quick local storage).
The Duke of Ice
ReadyNAS Newbie
 
Posts: 1
Joined: Tue May 17, 2011 6:02 pm
ReadyNAS: 3100

Re: ReadyNAS 3100 iSCSI'd to ESX 4.1 vCenter, very slow

Postby VMColonel » Sun Jun 12, 2011 8:58 am

If you haven't already, take a look at this page:
http://virtualgeek.typepad.com/virtual_ ... phere.html

It was written for/by EMC and NetApp, but most of the theory still applies.

If it were me, I'd be tempted to run iometer on a single VM, and see what kind of rate (IOPs - Cmd's / second) and bandwidth (MB/s) you are pushing, and at what latency (DAVG)

As a SUPER general rule of thumb, you'll want your DAVG counters (from esxtop , then d for disk mode) to be <20 for good performance. I like to see <10. Over 50 and you'll start to notice "slowness"

And lastly, try Windows 2008, I believe it is automagically aligned, for more information on performance best practices, please see here:
http://www.vmware.com/pdf/Perf_Best_Pra ... ere4.0.pdf

Welcome, and good luck :)
VMColonel
ReadyNAS VMware Expert
 
Posts: 29
Joined: Tue Aug 17, 2010 8:26 am
ReadyNAS: Duo

Re: ReadyNAS 3100 iSCSI'd to ESX 4.1 vCenter, very slow

Postby DAT » Fri Jul 15, 2011 12:59 pm

I'm running a pair of environments, one with 2 X ReadyNAS 2100s with 4 TB raw each, 2 X Netgear smart switches, and 3 X Dell 2900 servers running ESX 3.5 u4, and about 25 VMs hosted on NFS shares on the two 2100s. I get fantastic performance out of this set up. Really fast disk access with very low latency. Beats some of the low end NetApps that I used to use. The two ReadyNAS 2100s are still on 4.2.13.

Then I have a near identical setup (I've gone through all of the switch, ESX and ReadyNAS configs over and over again to make sure I haven't missed something) on another network. The main difference is that in the slow environment listed below I have a two port LAG directly connecting the two switches, and on the high performance environment listed above I have a 2 port LAG on each switch that connects back to a "core switch", so really one additional hop, yet much lower latency.

The slow performance network is 2 X ReadyNAS 2100s with 8 TB raw each, 2 X Netgear smart switches, and 3 X Dell r610 servers running ESXi 4.1 u1, and only about 9 VMs hosted on NFS shares on the two 2100s. I get horribly slow performance out of both of them, which I can now see is directly related to incredibly high latency accessing the NFS shares on the ReadyNAS. I also got horribly slow performance when I was running the VMs from the much older HP DL380 G3s (same servers that VMware uses/used for their VCP classes) with ESX 3.5 u4.

Before replacing the 8 year old servers I had thought the slowness was due to the servers being old and cooked, but now, with the new servers I realize that it's a SAN latency issue, since ESXi 4.1 u1 shows you this in the performance monitoring.

When I was using the old HP servers with 3.5, the ReadyNAS had 4.2.13 on both of them. I then upgraded to 4.2.17 on both of them, still had high latency, and they still are running 4.2.17 while being connected to from the new Dell r610 servers.

Your original issue is probably the exact same thing, and shouldn't have anything to do with iSCSI vs NFS, but is a latency issue accessing the ReadyNAS.

I have not been able to figure out what is causing this incredibly high latency, but I agree, it's not normal, nor is it anywhere near acceptable for trying to run servers from it.

Netgear, can you guys take a look at this for us?
DAT
ReadyNAS Newbie
 
Posts: 4
Joined: Fri Jul 01, 2011 11:32 am
ReadyNAS: 2100

Re: ReadyNAS 3100 iSCSI'd to ESX 4.1 vCenter, very slow

Postby techdruid » Sat Jun 02, 2012 7:28 pm

For some reason I've not seen this simple setting posted on any relevant searches for NFS Latency on the ReadyNAS. So here's one thing to check if you have an NFS datastore on your ReadyNAS and are experiencing unusually high latency.

http://techdruid.com/2012/06/03/vmware-readynas-nfs-and-extremely-high-latency/

Log in to your ReadyNAS, then go to "Shares -> Share Listing" and click on the NFS edit icon for the share that your ESXi host is mapped to. Scroll down to the Advanced NFS Options and ensure that the "Enable sync mode" is UNCHECKED.
techdruid
ReadyNAS Newbie
 
Posts: 4
Joined: Mon Nov 07, 2011 2:18 pm
ReadyNAS: 2100


Return to VMware



Who is online

Users browsing this forum: No registered users and 1 guest