4.1.9(T6) Data Corruption

[slayerking]

After updating to the T6 beta on my ReadyNas Duo, All downloads would be corrupt. For example if I used utorrent as I have done for years to download straight to the NAS all the data would be corrupt, The same goes with other download managers too. I tested with firefox' download manager and same thing corrupt data.

[yoh-dah]

By chance did you change the memory module on the Duo?

[slayerking]

Nope just updated the firmware to the T6. I'm back on 4.1.8 now and it's fine again. It was fine copying from pc to the nas, But if I downloaded a file with say firefox that had the destination of the nas the file would be corrupt 100% of the time.

[mdgm]

What version of Firefox?

[slayerking]

What version of Firefox?

All of them, I tried with 11, 10, 8, 7 and so on down to 3. It's not just firefox, I used firefox as an example. Any program I tried that downloaded from the net produced the corruption. Transferring files from pc to nas worked fine.

[matthew1471]

I had been running this firmware for a while (since Mar 31 10:55:34) and then hit issues.

I tried writing as the disks were powering up, the file that was written was corrupt. The application (KeePass) even acknowledged the file had become corrupt.

---------------------------
KeePass
---------------------------
The new file's content does not match the data that KeePass has written, i.e. writing to the file has failed and it might be corrupted now.
Please try saving again, and if that fails, save the database to a different location.
---------------------------
OK
---------------------------

trying to open the file that had just been written resulted in:

---------------------------
KeePass
---------------------------
Failed to load the specified file!
The file signature is invalid. Either the file isn't a KeePass database file at all or it is corrupted.
---------------------------
OK
---------------------------

Agree with previous poster (and the poster in viewtopic.php?f=17&t=61990#p350165), copying the files using Windows Explorer doesn't seem to trigger the bug, writing directly to the NAS from a program does.

I then opened another file and tried saving, MORE corruption. I rebooted the NAS.

Saving the same file again even after the reboot of the NAS still caused corruption errors. Reverted back to T2 and it instantly fixed it (without even rebooting my PC!). Bug is CLEARLY in T6.

Update: Have reported to ReadyNAS support Case ID 18343146.

[matthew1471]

Right. I think I know what causes it now. Here is what I have sent to NETGEAR support:

I have done some further testing myself and managed to find the cause of this and can now reproduce it 100% of the time without fail.

Basically if you read information about the file while it is still being written to (refresh the folder you are writing in / switch to that folder / have the file highlighted and the folder the current active window and sometimes Windows Explorer itself keeps updating as the file grows / the program tries to read that file while it's writing), the NAS running T6 corrupts it.

So in my case, what must have happened is I had explorer open in that folder (because I must have had it open to have double clicked to open it) while KeePass was trying to write to that file, explorer kept trying to read the file-size while KeePass was writing. That corrupted it.

I have made a program in C# to demonstrate the T6 flaw. This is the best way to cause the error and observe how it has corrupted the file (unless you have a good Internet connection and don't mind re-downloading lots of large files?). Understandably sending you my programs over the Internet is a bad idea (they could be viruses) so I attach my source code instead so that you can see what the code does and decide whether you trust it enough to run it on your computer.

The one thing I can't work out is why some byte patterns seem immune to this corruption but my program writes all of the possible bytes values to the file though and it reproduces the flaw every time (on only T6).

CentralHeating.txt (apologises for the RAIDiator testing pun ) - Simply copy this into the free Microsoft Visual Studio Express (or one of the paid versions if you have it) as a C# Console application. You will need to change the path to a writeable place on your NAS where you want the file to be stored. The program when it runs creates a file with a repeated pattern of bytes. Then it reads back the same file and checks to see if it is still the same pattern. If the pattern is not the same as what it should have written, the program will tell you that the file is bad!

Now normally this will work perfectly ("SUCCESS - Program read back the file exactly as it should have been written."), HOWEVER, if you open up a Windows Explorer window and go to the folder on your NAS where the file is being written to and highlight it (Windows Explorer will keep updating the file size while it is being written to), maybe pressing or even holding F5 to refresh the folder a few times (just to be sure) as my program is running then the NAS will get distracted and corrupt the pattern in the file. Therefore rendering the file useless.

In a real world scenario this would mean if I was downloading a large binary file using Firefox and I had the file I was downloading to highlighted (or I kept pressing F5 or refresh in the folder where the file was being saved) as it was writing in Windows Explorer, my downloaded file would be corrupt (this doesn't seem to happen for all downloads though).

If you would prefer we can arrange a time for you to ring me and I can remotely show you this? and/or If you don't want to compile my program and would rather I sent you one you can just run and you have a computer you don't mind running my programs on (a test virtual machine or something?) then let me know and I will send you the .exe file for you to run.

When I go back to the T2 firmware, my program behaves normally even when I have left Windows Explorer open in the folder I am writing to and refresh it while the file in that folder is being written to. The bug is CLEARLY in T6 and I would guess has something to do with samba file locking. There is a page about Samba locking here (http://www.samba.org/samba/docs/man/Sam ... cking.html) which should help further diagnose why the connection gets broken by clients attempting to read files while another user is performing a write operation.

Hope this helps,
Matthew

P.s. On my NAS the configuration is "Disable full data journaling", "Disable journaling", "Enable fast CIFS writes" and the share ("/temp") allows Guest users to read/write (which is who I am logged in as) but the errors still occur even if I turn all these off and reboot the device.

Code: Select all

using System;
using System.IO;

namespace CentralHeating
{
    class Program
    {
        // Where the file should be written to. The file will be deleted if it already exists.
        const string outputFile = @"\\192.168.1.3\temp\thrash\Radiator.key";

        static void Main(string[] args)
        {
            // Write the program banner.
            Console.WriteLine("----------------------------");
            Console.WriteLine("Matthew1471! CentralHeating");
            Console.WriteLine("----------------------------");
            Console.WriteLine();
           
            // Perform the "write of a binary file and then verify" test (deleting the file if it already exists).
            writeThenVerifyFile();

            Console.WriteLine("Program Completed. Press any key to exit.");
            Console.ReadKey();
        }

        private static void writeThenVerifyFile()
        {
            // Trash the output file if it already exists.
            if (File.Exists(outputFile)) { File.Delete(outputFile); }

            // Whether there was a bad byte in the file.
            bool badByte = false;

            // Update the user.
            Console.WriteLine("Starting Test..");

            // The using statement in C# ensures the resources get cleaned up properly.
            using (FileStream fileStream = new FileStream(outputFile, FileMode.CreateNew, FileAccess.ReadWrite, FileShare.ReadWrite))
            {
                // We want to make a 25MB file.
                for (int loopCount = 0; loopCount < 102400; loopCount++)
                {
                    // Update the user on our progress.
                    if (loopCount % 10240 == 1) { Console.WriteLine("Writing " + (loopCount / 1024) + "%.."); }

                    // Write to the file all the bytes from 0x00 - 0xFF.
                    for (int count = 0; count < 256; count++) { fileStream.WriteByte((byte)count); }
                }
            }

            // Open the file we have just written.
            using (FileStream fileStream = new FileStream(outputFile, FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
            {
                // We want to read the 25MB file.
                for (int loopCount = 0; loopCount < 102400; loopCount++)
                {
                    // Update the user on our progress.
                    if (loopCount % 10240 == 1) { Console.WriteLine("Reading " + (loopCount / 1024) + "%.."); }

                    // Read 256 bytes.
                    for (int count = 0; count < 256; count++)
                    {
                        // Read in a byte from the file.
                        byte readByte = (byte)fileStream.ReadByte();

                        // Check if the read byte matches the expected byte for this position in the file.
                        if (readByte != (byte)count)
                        {
                            // This now could be classed as a "corrupt" file.
                            badByte = true;

                            // Notify the user that a bad byte was encountered.
                            Console.WriteLine("ERROR - While reading the file, what should be 0x" + count.ToString("X2") + ", we instead found 0x" + readByte.ToString("X2") + ".");
                        }
                    }
                }
            }

            Console.WriteLine();

            // If there was not a bad byte in the file then the user will need a success message.
            if (!badByte) { Console.WriteLine("SUCCESS - Program read back the file exactly as it should have been written."); }

            Console.WriteLine();
        }
    }
}

[slayerking]

Unfortunately I don't have explorer open in any of the NAS folders. If I use any program to write to the NAS it gets corrupted, But not if I copy from PC to the NAS myself. But yes it is clearly a bug in T6.

[matthew1471]

Maybe your applications are reading the files back as they are writing them?

[slayerking]

It's a possibility, I know uTorrent would at some point not sure about the others though. All I noticed was the files were never complete on the NAS and gave the errors you posted earlier, I could download the same file time after time and every time it would be a different size and 100% of the time corrupt.

[matthew1471]

Just thinking out loud but I wonder if some anti-virus products as part of their "real time" scanning would re-scan (i.e. read) a file that has just been written to? That way even an application's data where the application does not explicitly read the file would corrupt.

As for bittorent, I guess it depends on whether the particular implementation of it caches the pieces that it is trying to send in memory (if possible, granted some machines might have so little memory that almost everything has to come straight from storage) or if the moment a user requests a piece it immediately looks on the disk.

Either way whatever the cause I'm happy that I have a concrete test case that should be reproducible so the devs aren't having to spend days trying to figure out what is introducing occasional corruption and I'm pretty happy that T2 is pretty safe with my data (and can easily test any future versions with my program rather than waiting until I lose a file again ).

I'm sure by fixing my data issue they will fix yours.

Did you try T2? T2 fixed all the performance issues with 4.1.8 for me. I wouldn't mind betting T2 will not give you corruption either.

[slayerking]

Yeah T2 worked well but I deleted it so just went back to 4.1.8

[matthew1471]

T2 - http://www.readynas.com/download/beta/r ... r-4.1.9-T2

I hope they work out the issues with T6, because I really liked the fact there was a newer Samba in it.

The reason I drifted into beta territory in the first place was because 4.1.8 has a speed issue (http://blogx.co.uk/Comments.asp?Entry=855)

[ahpsi]

Crap, I just corrupted half a directory of inventory Excel files by upgrading to T6 on an 1100. Was running T2 and didn't have any issues but wanted the newer Samba as a new DC goes live next week. Now I must revert to off-site backups for the corrupted files. Should have read this thread before upgrading. Hard to believe T6 is still available for D/L given what it can do to your data.

[matthew1471]

ahpsi, sorry to hear that.

It might be worth you ringing Netgear and asking if you can be added to my case (Case ID 18343146). Might help to get T6 pulled quicker. I think because I've been the only user to raise the issue so far they have hesitated over pulling it.

I hope they do keep the new Samba (and just fix whatever the problem is) rather than just decide T2 *is* the release.