I've spent no less than 72 hours, over a period of months, dealing with all the problems of the ReadyNAS Duo. I've spoken to tech support levels 1, 2, and 3, who knew little about this product's issues or how to solve them. I was able to talk with very nice people, who sincerely tried to help me, but ultimately knew nothing and could do nothing. I don't blame them, because if I were Netgear, I wouldn't want anyone to know about the many flaws in my product's design.
If you search these forums for the "ReadyNAS Duo", you will begin to see the many problems that people are having with blinking blue lights and corrupted drives. I fell victim to this myself, and through much wasted time, managed to gather an idea on how this is happening, and how to fix it.
From my understanding, there is a memory chip inside the ReadyNAS that stores a Linux-based operating system. This operating system is responsible for operating the device and driving the "FrontPage" web interface. It weighs in a little over 200MB in size. The ReadyNAS however, only uses its internal memory to store the original copy. When you insert an empty hard disk into the ReadyNAS, the operating system is copied over from memory and installed on the disks themselves. At this point, the ReadyNAS boots itself from the hard disk like any normal computer.
And herein lies the problem: The integrity of the hard disks, and the stability of the NAS itself, are intricately tied together. If one of these components fail, both of them fail. Which seems to happen fairly often. When the ReadyNAS tries to load the corrupted operating system, it will hang during the boot process, not knowing what to do next. To make matters worse, everything is managed under an LVM partition, meaning that your important data is stored within an additional software layer. This adds an extra level of complication in the event of a disaster. LVM is great on production machines, but I feel backup partitions are better off not using it.
Have you experienced these symptoms?
- The blue light pulses forever, and never changes.
- Frontview is completely inaccessible.
- Your ReadyNAS network shares are not reachable.
- Your ReadyNAS is invisible to RAIDar and HTTP requests from your network.
- The unit refuses to power off and you are forced to pull the plug, or is a power outage, and your Readynas refuses to boot correctly.
- Re-Flashing the firmware works correctly, but still doesn't fix your drive.
- Even after experiencing the above problems, your ReadyNAS will still give a normal error code (3 long flashes, 2 quick flashes) when starting the NAS with no hard disks installed.
Chances are, the ReadyNAS operating system is corrupted beyond repair, and it is unable to boot correctly. If you are really unlucky, your hard disk's partition table may be corrupted as well. The tech support staff are not willing to tell you this, and will probably blame the ReadyNAS unit itself. They will offer to replace it, and even promise that your hard disks will magically come back to life once you place them into a new ReadyNAS unit. The only problem is, it has nothing to do with the ReadyNAS unit. The drives are corrupted. They will fail inside any ReadyNAS unit!
Your only chance is to pray the main LVM partition is still intact, and perform a data recovery.
Data Recovery? Mission Impossible.
When your operating system becomes corrupted, the most common thing to do is mount it inside another operating system. This way, you have full access to all the system files that would normally be restricted, and then you can easily fix or replace them. There are many hard drive deaths every year, but only a fraction of them are mechanical issues. Most drive failures are software based, which can often be fixed or recovered with little effort.
When Netgear first released the ReadyNAS Duo, it used a common ext3 filesystem, and stored data in blocks of 4k size. This was compatible with any Linux operating system, on any type of hardware. Then one day, going against all reason, Netgear decides to release an update that changes the ReadyNAS default filesystem to use an ext4 partition with a whopping 16k block size. While most Linux distros can handle Ext4 partitions, none of them support block sizes above 4k. The only exception is operating systems designed to run on exotic hardware, like SPARC, or in this very unfortunate case, the ReadyNAS Duo.
Not only that, but a 16k block size means that every file must take up 16k of space, even if the file itself is much smaller. This is not such a big problem for data partitions, where almost every file can be expected to closely match or exceed that size. However, operating systems, Linux especially, are full of tiny configuration files. A good operating system maximizes read/write performance by packing these files together and reducing seek time. But when you use a 16k block size, you end up with thousands of small files sitting inside large empty blocks. Not only does this reduce performance, but it increases your chances of file corruption.
A Workaround
As I mentioned above, one of the flaws with the ReadyNAS Duo is that your data is intertwined with the operating system. Despite what Netgear's tech support may tell you, I would not recommend swapping the disks around, or trying to start the ReadyNAS on just a single disk. If you do this, you risk the ReadyNAS formatting the drive and erasing all your data. If the partition tables are corrupted enough, the ReadyNAS unit will format your disks, even if you never engaged the factory reset. I HAVE HAD THIS HAPPEN TWICE!!
What you need to do, is this:
Data Recovery, Step I
- Remove both existing drives from the ReadyNAS.
- Plug one of the drives into your PC using a SATA-to-USB adapter. I have already tested and verified this specific adapter works:
http://www.newegg.com/Product/Product.aspx?Item=N82E16812232002 - If you are running Windows, use 'Computer Management' to check the drive layout. You should see three partitions, with the third partition taking up most of the drive. It should be labeled 0x8LVM or something similar. If you can see this, then the drive should be recoverable.
- If Windows locks up from trying to load the drive, chances are the partition table is corrupted. You will need to use a partition recovery program like TestDisk to fix the partition table before proceeding.
- If the drive loads as 'Not Initialized' and won't let you Initialize it manually, then you will probably need a professional data recovery service. You can try using diagnostics tools provided by the drive manufacturer to fix the drive, but chances are the drive platters will need to be transplanted, or the circuit board replaced.
- If you are running Linux, check to make sure you can access the LVM by using the commands pvscan and vgscan. Try to activate the LVM partition by using 'vgchange -a y' and pay attention to where it's mounted (mine was /dev/c/c). Use the fsck command to scan the LVM. You should get a message saying 'bad block size: 16384' which is normal. If you have a problem running any of these commands, make sure you have the LVM2 software installed by running apt-get install LVM2.
- If you can't see or access the LVM on one disk, try the other one. They are mirrored, so it doesn't matter which one you use, as long as one of them works.
- Once you verify that the operating system can 'see' the LVM, then you need to do a sector-by-sector backup. In fact, this is the only way to backup the partition, because of the 16k block size. I used a windows program called Acronis True Image, but any program capable of disk imaging and sector-by-sector backup should do. Make sure the backup program can backup and restore single partitions within a volume, because we only want to backup the largest LVM partition, which holds all our files.
Now that you managed to back up the LVM partition with all our data, we can proceed to the next step.
Data Recovery, Step II
- We need to install an empty drive into the ReadyNAS, so that it can install a fresh copy of the operating system. The drive needs to be the same size as the one we're recovering, or bigger. You can use your PC to format one of the existing drives to be empty, but make sure you are comfortable with wiping out all the data.
- If the ReadyNAS refuses to setup the new drive, you can use a firmware reset, or factory reset, to nudge it along.
- Once the new drive is setup and running, you should have a solid blue light and solid green light from one of the drive bays. RAIDar, FrontView, and the ReadyNAS itself should all be functioning properly and visible on the network. This is further indication that the ReadyNAS unit is not physically broken, just unable to boot from the corrupted disks.
- Poweroff the ReadyNAS system, and remove the now-functioning drive. Use the USB-to-SATA adapter to connect the disk to your PC.
- Using the same methods in step one, locate the LVM partition taking up most of the drive. This is the new data partition. Delete this partition from the drive, and only this partition. The other two partitions (which I believe are boot and swap partitions) should be left untouched. Windows users can use Computer Management to simply delete the LVM partition. I don't know the proper steps for doing this in a Linux environment.
- There should be a large amount of free space where the new LVM partition used to be. Using the backup software, restore your existing LVM partition into that empty space. It should be a perfect fit, or if you are using a larger drive, you should have extra space left over at the end.
- Once the restoration process is complete, plug this drive back into the ReadyNAS. Using FrontView, check to see that your existing shares have been restored. Hopefully, they have. Make sure you setup user accounts to access your files. If there are any permission issues, you can use the share options to force a new user/group onto all your files.
- If you can see the shares through FrontView, but for some reason can't access them over the network, there might be additional permissions conflicts going on. Try extracting your data by using a Backup Job to dump everything into an external drive.
Conclusion
Once you are successful in recovering your data, I suggest you never use the ReadyNAS Duo ever again. You learned your lesson. If you are still crazy enough to use it, however, I HIGHLY recommend you look into resetting the block size back to 4k or lower. One way to do this, is by downgrading the firmware to a version still using the ext3 file-system. There are other methods on resetting the blocksize posted in these forums as well. But you have been warned!
