4.1.9(T6) Data Corruption

[ahpsi]

Well, I do have great faith in the Jedi's and know they monitor the forums. In the end (and I may be wrong in this) I think Netgear farms the first level tech support out through the same channels their other products are supported through but when the issues get to a certain level they come back into a smaller group that actually knows what is going on and can actually affect product development. I would hope given the severity of the issue (data corruption ranks very high in my list of 'bad' things when we are talking about a data storage device) this thread is already on their radar, assuming you and I haven't done anything monumentally stupid and caused our own issues. I've already reverted the 1100's and Duo's I'd updated and initiated new one way RSYNC jobs to clear any corrupted data so I'm not too worried but will keep watching for an official communication regarding this issue.

Thanks for all the work you've done in identifying and reproducing this, great work!

[slayerking]

T2 - http://www.readynas.com/download/beta/r ... r-4.1.9-T2

I hope they work out the issues with T6, because I really liked the fact there was a newer Samba in it.

The reason I drifted into beta territory in the first place was because 4.1.8 has a speed issue (http://blogx.co.uk/Comments.asp?Entry=855)

Thanks for the link.
Yeah I'm amazed it hasn't been pulled yet

[ianch99]

Reading this thread on data corruption in T6 worries me as I am currently on this version. If I understand things correctly, I download a large(?) file directly onto a network share and then read it back to compare against the same file downloaded locally. The network drive copy will be corrupt and the local one ok.

I used Firefox 11 on XP and downloaded iTunesSetup.exe (75 Mb) twice: once onto a (smb) network share hosted by my Duo v1 and second onto a local D: drive. I then ran cksum against both files:

[:C:/] cksum d:/Downloads/iTunesSetup.exe //ianch-nas/backup/Utility/iTunesSetup.exe
573707582 74982768 d:/Downloads/iTunesSetup.exe
573707582 74982768 //ianch-nas/backup/Utility/iTunesSetup.exe

As you can see, no difference ..

I am unsure if I do or do not have a problem

Have NetGear Support verified this corruption problem on T6 and if so, what is their advice?

[slayerking]

Reading this thread on data corruption in T6 worries me as I am currently on this version. If I understand things correctly, I download a large(?) file directly onto a network share and then read it back to compare against the same file downloaded locally. The network drive copy will be corrupt and the local one ok.

Any file large or small, Let me explain. I have the downloads folder on my NAS mapped to drive W: and the media folder mapped to X:. When I use firefox to download which points to drive W: (The NAS) the file is corrupt. Say pidgin for example it's 9,175KB in size, If I download it on the T6 firmware it will be a different size(say 8,964KB), I downloaded it 4 times and it was never the same size and was never complete.

Trying to run said file results in this error message "Failed to load the specified file! The file signature is invalid. Either the file isn't a KeePass database file at all or it is corrupted." or roundabouts.

[ianch99]

I did exactly this i.e. download a file from the internet directly onto a smb (ReadyNAS Duo v1) share and the file size & cksum was correct .. no corruption

Have NetGear confirmed this as a problem in T6?

Thanks

[slayerking]

I did exactly this i.e. download a file from the internet directly onto a smb (ReadyNAS Duo v1) share and the file size & cksum was correct .. no corruption

Have NetGear confirmed this as a problem in T6?

Thanks

Well continue what you are doing then as you seem to be doing something different. I can replicate it 100% of the time which is confirmed by matthew1471 and ahpsi found out the hard way too.

[ianch99]

I am more concerned that NetGear is silent regards this issue. For a NAS product, what is more important than data corruption? Please can someone from NetGear respond on this thread and update us with their position on this? Do we all keep using T6 or not?

[matthew1471]

Thanks for all the work you've done in identifying and reproducing this, great work!

No problem, my degree is in Computer Science and I probably have a few more tools at my disposal (Visual Studio 2010 etc) than the average user reporting "the NAS is doing something weird", so it only seemed fair to give a full disclosure and save Netgear a couple of weeks of fruitless searching.

I am more concerned that NetGear is silent regards this issue.

To be fair it is something they will need to check themselves before they admit it.. and given they may not have explicitly tested reading file information at the same time as writing it is something they're going to need to take a bit of time to go through (it may even be this problem is due to some weird way Windows 7 uses SMB too, which will make reproducing it in their lab even more fiddly). T2 had no noticeable side effects apart from a few users saying it was slower (which is all T6 fixes over T2) but I'd rather slightly slower than "can't guarantee it writes the file properly".

To be honest I am impressed Netgear provided as much of their time and attention as they have so far, how many other companies will a) not accuse it of being your PC or networking issues (with most call centres and ISPs I have to pull the "trust me, I know what I'm doing and seeing" card) b) refuse to support you because it's a beta and c) refuse to support you because you've replaced the RAM?

I'm definitely impressed with their support, just think they could probably note "Some users are experiencing corruption with T6, as a workaround use T2 if you have any concerns." on the download page.

I did exactly this i.e. download a file from the internet directly onto a smb (ReadyNAS Duo v1) share and the file size & cksum was correct .. no corruption

You only hit this issue if you are attempting to read information about the file (could just be what the file size currently is) while writing to it at the same time. This could either be caused by your anti-virus program (trying to read what file you've just written to see if it's bad), Bittorrent client (to upload a bit of the file you've already downloaded to someone else) or just from refreshing the folder (which Windows 7 seems to even do itself sometimes if you have the folder open and the most active window) a few times while it is downloading.

I download a large(?) file

You (or Windows) are more likely to try to refresh the folder a few times while it's writing if the file is large.. in theory it could really be just about any file size.. but it's more fiddly to try and read the folder at *exactly* the time it's writing, far more easy to reproduce with your 20GB / 12 hour download that you keep going in and out of the same folder for other files etc. My "proof of concept" program generates a file that I think is around 25MB.. and I can corrupt that by refreshing the folder I write to about 5 or 6 times while it's writing so it can be just about any file really.

I managed to "real life scenario" reproduce this while on the phone to Netgear by downloading iTunes using Firefox and setting the file to be saved to my NAS, having that folder open and pressing F5 a few times in that folder while it was downloading (Windows said it wasn't even sure if it was a file it was able to run once I did this).. this is a show-stopper for me.

I agree you could just stop trying to read the file information while it's downloading (and for most files particularly small ones you seem to get away with it) but the risks are too severe for me to see this as a decent workaround to keep running T6. It's definitely reproducible in my set-up and instantly goes away the moment I put T2 back on. I don't mind if you continue to use T6 or not (and I'm happy enough with the program I wrote being able to reproduce it for any Netgear engineer who cares to give it a spin), but for me my decision is clear .

[matt_hargett]

This may explain a weird iTunes database corruption issue I get when I have podcasts/apps downloading in iTunes while my phone is syncing at the same time. I was trying to track it down and make it reproducible before reporting it. Thanks for going to the effort! I'm sure they'll fix it ASAP and release an updated beta now that they have a reproducible case.

[ianch99]

Thanks for all the work you've done in identifying and reproducing this, great work!

No problem, my degree is in Computer Science and I probably have a few more tools at my disposal (Visual Studio 2010 etc) than the average user reporting "the NAS is doing something weird", so it only seemed fair to give a full disclosure and save Netgear a couple of weeks of fruitless searching.

I am more concerned that NetGear is silent regards this issue.

To be fair it is something they will need to check themselves before they admit it.. and given they may not have explicitly tested reading file information at the same time as writing it is something they're going to need to take a bit of time to go through (it may even be this problem is due to some weird way Windows 7 uses SMB too, which will make reproducing it in their lab even more fiddly). T2 had no noticeable side effects apart from a few users saying it was slower (which is all T6 fixes over T2) but I'd rather slightly slower than "can't guarantee it writes the file properly".

To be honest I am impressed Netgear provided as much of their time and attention as they have so far, how many other companies will a) not accuse it of being your PC or networking issues (with most call centres and ISPs I have to pull the "trust me, I know what I'm doing and seeing" card) b) refuse to support you because it's a beta and c) refuse to support you because you've replaced the RAM?

I'm definitely impressed with their support, just think they could probably note "Some users are experiencing corruption with T6, as a workaround use T2 if you have any concerns." on the download page.

I did exactly this i.e. download a file from the internet directly onto a smb (ReadyNAS Duo v1) share and the file size & cksum was correct .. no corruption

You only hit this issue if you are attempting to read information about the file (could just be what the file size currently is) while writing to it at the same time. This could either be caused by your anti-virus program (trying to read what file you've just written to see if it's bad), Bittorrent client (to upload a bit of the file you've already downloaded to someone else) or just from refreshing the folder (which Windows 7 seems to even do itself sometimes if you have the folder open and the most active window) a few times while it is downloading.

I download a large(?) file

You (or Windows) are more likely to try to refresh the folder a few times while it's writing if the file is large.. in theory it could really be just about any file size.. but it's more fiddly to try and read the folder at *exactly* the time it's writing, far more easy to reproduce with your 20GB / 12 hour download that you keep going in and out of the same folder for other files etc. My "proof of concept" program generates a file that I think is around 25MB.. and I can corrupt that by refreshing the folder I write to about 5 or 6 times while it's writing so it can be just about any file really.

I managed to "real life scenario" reproduce this while on the phone to Netgear by downloading iTunes using Firefox and setting the file to be saved to my NAS, having that folder open and pressing F5 a few times in that folder while it was downloading (Windows said it wasn't even sure if it was a file it was able to run once I did this).. this is a show-stopper for me.

I agree you could just stop trying to read the file information while it's downloading (and for most files particularly small ones you seem to get away with it) but the risks are too severe for me to see this as a decent workaround to keep running T6. It's definitely reproducible in my set-up and instantly goes away the moment I put T2 back on. I don't mind if you continue to use T6 or not (and I'm happy enough with the program I wrote being able to reproduce it for any Netgear engineer who cares to give it a spin), but for me my decision is clear .

Thanks for the great explanation. I missed the reading whilst writing part .. I did your "iTunes" test, downloading the 75 Mb file using Firefox (on XP) direct to a mounted NAS network drive. All the while the download was in-progress, I hit F5 (once per second) and could see the file size increasing. I checked the size & cksum at the end with the local copy of the same file and it was the same. I could also run the .exe from the network drive .. strange ..

.. I then switched to a Win 7 PC and repeated the same process ... the cksum is different!! Seems to be a Win 7 only problem? Possibly related to a Samba upgrade in T6?

I think I will go back 4.1.8 and wait for 4.1.9 ..

[ianch99]

Just to confirm, back on 4.1.8 and the Win 7 download corruption is no longer seen ..

[matthew1471]

.. I then switched to a Win 7 PC and repeated the same process ... the cksum is different!! Seems to be a Win 7 only problem? Possibly related to a Samba upgrade in T6?
I think I will go back 4.1.8 and wait for 4.1.9 ..

Wow, thanks for posting your findings. That's very interesting, I wonder if it's some Samba SMB2 bug.

I found some interesting (but not completely appropriate) pages such as http://support.microsoft.com/kb/2618096 and http://support.microsoft.com/kb/2028965 (what's the difference between the 2?) that seems to indicate Windows 7 itself is not perfect with SMB2 and multiple access but more importantly...

...SMB2 support was added in Samba 3.5 and listed as experimental (http://wiki.samba.org/index.php/Samba_3 ... ol_changes:) :

Samba 3.5 also includes experimental support for SMB2.[16] Samba 3.6 fully supports SMB2, except the modification of user quotas using the Windows quota management tools.[17]

Source: http://en.wikipedia.org/wiki/Server_Message_Block#SMB2

Apparently 3.6 is where SMB2 becomes stable. T6 was running 3.5.12 and T2 was running Samba 3.0.

I like that we are running new Samba (for security reasons and also that SMB2 is set to give one hell of a performance increase), so see the solution as Netgear either trying 3.6 Samba or making SMB2 an optional feature in Frontview listing "(reading file information while writing to it may in some instances cause file corruption)".

I might try again, but this time disabling SMB2 on my Windows 7 SP1 x64 computer but am sure it will work.

[slayerking]

.. I then switched to a Win 7 PC and repeated the same process ... the cksum is different!! Seems to be a Win 7 only problem? Possibly related to a Samba upgrade in T6?

I think I will go back 4.1.8 and wait for 4.1.9 ..

Nice find

[Skywalker]

matthew1471, thanks for the great work nailing down how to reproduce this, and the super-handy utility! And sorry everybody for the trouble.

Interestingly, I have never been able to reproduce this issue by downloading straight from Firefox onto a NAS share -- even downloading several large files and making Windows Explorer refresh like a mad-man. But I am able to reproduce it easily with that utility, so there is most definitely a problem here.

So at this point, we don't know exactly what caused the issue, but changing a couple settings for 4.1.9-T9 (which I just posted), I can no longer reproduce the issue. We will continue to dig until we completely track it down, but if you guys could try 4.1.9-T9 and either confirm or deny that it resolves the data corruption issue for you, that would be very helpful.

[slayerking]

matthew1471, thanks for the great work nailing down how to reproduce this, and the super-handy utility! And sorry everybody for the trouble.

Interestingly, I have never been able to reproduce this issue by downloading straight from Firefox onto a NAS share -- even downloading several large files and making Windows Explorer refresh like a mad-man. But I am able to reproduce it easily with that utility, so there is most definitely a problem here.

So at this point, we don't know exactly what caused the issue, but changing a couple settings for 4.1.9-T9 (which I just posted), I can no longer reproduce the issue. We will continue to dig until we completely track it down, but if you guys could try 4.1.9-T9 and either confirm or deny that it resolves the data corruption issue for you, that would be very helpful.

still doing it for me with T9
EDIT: Tried it with several different files and all have landed corrupt. The error this time is the file is not complete as before and the message is that the software is not for the version of windows you have blah blah

Gone back to T2 and it's fine again same files down as before and they are perfect. Yes winblows 7 here, I'll try on my fedora box a bit later