Number of nfsd threads should be configurable in frontview

[XLION]

Hi,

this request is actually the result of performance measurements I've posted in http://www.infrant.com/forum/viewtopic. ... ght=#67517

After doing my "netcat" tests, the final "suspect" for causing heavy fragmentation on the target file has been NFS or rather nfsd. So I've manually edited "/etc/init.d/nfs-kernel-server" and changed the number of nfsd threads (nproc) from 8 to 1 which -- suprise!!! -- resulted in fragmentation of the target file to be gone, i. e. local read speed on the file copied over the net is now 40 MB/s (compared to 27 MB/s before).

Possibly someone at infrant/netgear should analyse why the use of multiple nfsd instances results in such heavy file fragmentation, but for now I would be happy with a configuration option in frontview ...

Cheers,
Axel

[bhoar]

Is it possible that the server or client needs to be told to lock each client's single file access to a single server thread?

There are probably some negatives to reducing the the total thread count to 1. There's got to be a config item that balances things out.

-brendan

[XLION]

Is it possible that the server or client needs to be told to lock each client's single file access to a single server thread?

There are probably some negatives to reducing the the total thread count to 1. There's got to be a config item that balances things out.

-brendan

I think that not using more than one process/thread for handling a "single file request" -- as you've suggested -- would solve the problem. I always found it quite strange that multiple instances of nfsd were "competing" over a single file ... (I'm using NFS over TCP, bytheway ...)

I've tried to figure out how nsfd is supposed to split load into multiple processes (per TCP-connection, per file, per block, ...), but I haven't found any details in the NFS documentation. There's just this rather generally statement, i. e. "increase number of instances on servers experiencing heavy load"...

ByTheWay: Limiting nfsd to a single instance also increased read performance over the network, i. e. I'm now getting about 40 MB/s when reading a 1GB file from the ReadyNAS. Though write performance is still not better than 12 MB/s, but this is OK my me ...

[bhoar]

Hmm, I'm not sure if this belongs in the Features Request, Beta Release or Performance forum (or all three). It looks like something that could give promising results for NFS users with some simple tweaks on the infrant side.

Perhaps Infrant (Netgear) can comment?

-brendan

[XLION]

Hmm, I'm not sure if this belongs in the Features Request, Beta Release or Performance forum (or all three). It looks like something that could give promising results for NFS users with some simple tweaks on the infrant side.

Whatever ... Considering the entire sequence of copying a large file over the net using NFS (resulting in heavy fragmentation) and then retrieving the very file again over the network, I see a performance boost of 200% (20 MB/s --> 40 MB/s), which is really worth the effort of adding an option to frontview (performance settings), isn't it?

Perhaps Infrant (Netgear) can comment?

Well, I'm having a party anyway ...

[bhoar]

Hmm, I'm not sure if this belongs in the Features Request, Beta Release or Performance forum (or all three). It looks like something that could give promising results for NFS users with some simple tweaks on the infrant side.

Whatever ... Considering the entire sequence of copying a large file over the net using NFS (resulting in heavy fragmentation) and then retrieving the very file again over the network, I see a performance boost of 200% (20 MB/s --> 40 MB/s), which is really worth the effort of adding an option to frontview (performance settings), isn't it?

Er, my point wasn't that the thread should be moved (which I can do), more just an off-hand comment that it didn't fit into a neat category, and also to keep the thread on the Infrant radar by posting to it.

Bump!



-brendan

[yoh-dah]

Nice find XLION. We'll do an analysis of this. Thanks!

[XLION]

I've just noticed that you'll (also) need to modify "/etc/init.d/rc3" to make the change "reboot-resistent". Is "/etc/init.d/nfs-kernel-server" actually used, or rather why does it exist, if it isn't called as a "module" on startup?

ByTheWay: Any news on the analysis of fragmentation caused by multiple nfsd instances?

[XLION]

I did some more research on the nfs performance/fragmentation issues.

1. The fragmentation problem (when running more than one instance of nfsd) isn't specific to the ReadyNAS. The same thing happens on my linux server (linux 2.6.22, nfs-utils-1.1.0).

2. Bad NFS write performance (to ReadyNAS, less than 50% of local write speed) is specific to the ReadyNAS, i. e. when copying files to my Linux Server, the only limiting factor is local disk speed. For "nfsd" uses more than 90% of cpu time during file transfer, I would guess that cpu speed is the limiting factor on the ReadyNAS ...

3. CIFS write speed is slightly better than NFS write speed (16,5 MB/s instead of 12,5 MB/s). CPU usage (smbd) is around 60% during copy.

[XLION]

I've just noticed that you'll (also) need to modify "/etc/init.d/rc3" to make the change "reboot-resistent". Is "/etc/init.d/nfs-kernel-server" actually used, or rather why does it exist, if it isn't called as a "module" on startup?

ByTheWay: Any news on the analysis of fragmentation caused by multiple nfsd instances?

After updating to beta 6 I checked whether the "nfsd instance issue/request" has been worked on. It doesn't seem so.

@yedi council: May I expect that this is addressed before going final?

[yoh-dah]

Not for 4.0, but for the technically savvy, it's a pretty easy change in the shell anyway. Having multiple nfsd running gives you better performance in a multi-user environment and for the most part, the performance is acceptable in a single-user environment. We may consider adding an option for 4.1, but I'd rather keep things less complicated and less cumbersome for the majority of the folks.

[Skywalker]

I've just noticed that you'll (also) need to modify "/etc/init.d/rc3" to make the change "reboot-resistent". Is "/etc/init.d/nfs-kernel-server" actually used, or rather why does it exist, if it isn't called as a "module" on startup?

It is used if the NFS service is stopped or started from Frontview.

[XLION]

Not for 4.0, but for the technically savvy, it's a pretty easy change in the shell anyway. Having multiple nfsd running gives you better performance in a multi-user environment and for the most part, the performance is acceptable in a single-user environment. We may consider adding an option for 4.1, but I'd rather keep things less complicated and less cumbersome for the majority of the folks.

As I said, the performance improvement by itself isn't that much.
But a large file written to the ReadyNAS using the default of 8 instances gets so heavily fragmented that afterwards read speed degrades to 50-60% of what would be possible ... (and this is a permanent condtion for this file)
So please, please change the default to using a single instance and add a listbox to the performance options that allows to increase the number of instances when required ...

[bhoar]

So please, please change the default to using a single instance and add a listbox to the performance options that allows to increase the number of instances when required ...

I concur with XLION here. Otherwise, all that talk about ext3 being such a low fragmentation-risk filesystem vs. NTFS (and therefore no defrag tools are necessary) will be for naught.

-brendan

[MrCyberdude]

Less defragmentation anytime..!!

Defragmentation = Faster DEAD HDD's
Let the users wait, suffer even.
Better them than the admin.

Funny how a system going down changes one's perspective.