ReadyNAS NV locked up, disk failures, and recovering data

[abennett]

I've been a long time ReadyNAS advocate, since the Infrant days, and I have an NV that's been in service since March of 2007, upgraded a few times as my storage needs grew. Currently it has 4x1GB Seagate 7000.12 drives in it (ST31000524AS off the HCL) running X-RAID and is hooked up to a home network serving my family's four Macs. All has been going swimmingly for the last 5 years, until this past week...

At some point earlier this week the NV unit locked up and wouldn't respond to AFP/SMB/HTTP/SSH requests, pings, RAIDar, Frontview, or even the physical power button; I had to resort to pulling the plug (with a cringe.) When I started the NV back up, I was able to see it with RAIDar and it appeared to be running a system check. However, when it finished the checks it once again locked up into a completely frozen state. Pulled the plug again, and connected the ethernet cable directly to my laptop. Again it checked the disks on start up but froze shortly after that finished.

At this point I should mention that over the last months I have been seeing increasing S.M.A.R.T. errors (Reallocated Sector) on drive 1. I had ordered a replacement drive and was ready to replace it. I figured now would be the best time to do so, so I pulled drive 1. Curiously when I booted up with the drive 1 bay empty, the system came up and *stayed* up, however when I hot-plugged the replacement drive the system went back to a frozen state.

Using what I'd learned, I was able to keep the system up by pulling drive 1 again, allowing me to copy off some data that wasn't being backed up (photos get backed up, ripped movies don't since I don't have that kind of space outside of the NAS). I was able to save some of that data before the inevitable happened...

The system froze up again, and upon checking my email I noticed that frontview had tried to notify me of S.M.A.R.T. errors now on drive 4! I fear that whatever happened to drive 1 is how happening to drive 4, and I can't pull a second drive as I was unable to do a resync! Interestingly I'm able to access the data via SSH while it's in in read-only mode doing quota checks. As soon as it flips to read/write everything locks up agian.

What I want to do is save about 1TB of files that would be a huge pain to replace (but not worth data recovery services). Is there a reason that the system would lock up on a drive failure, and is there any way around it? I can't seem to get a resync to happen as it locks up when the new drive was introduced. Most important, is there a way to *keep* the NV in read-only mode and I can pull the files off as quickly as possible?

Any ideas are highly appreciated!
Aaron

[readysecure1985]

Hello Aaron,

The Sparc units are known to lockup when drive 1 dies, due to it being the base drive. The reason the NAS is having issues resyncing is because of that drive 4 having reallocated sectors. The best idea would be to clone drive 4, then try to add drive 1 (with clone 4 already in the system) in order to sync the drive to the array. Below are instructions to clone your drive.


Simple Hard Drive Recovery
Here is a simple guide to quickly recover a failed drive using dd_rescue.
Here is a quick simple guide how to achieve this with a free Linux Live CD and a PC with two SATA connections.
Using a Knoppix 6.2 Live CD for this guide. Can be found at www.knoppix.net
Using dd_rescue command allows you to copy data from one drive to another block for block. This is especially useful for recovering a failed drive. Often when a drive fails, the drive is still accessible, it has just surpassed the S.M.A.R.T. error threshold. dd_rescue allows you to ignore the bad sectors and continue cloning the bad drive to a new healthy drive.
1) Connect your old drive and new drive to your PC
2) Boot up using your Linux live CD
3) Launch a terminal window.
4) Run fdisk -l to make sure the system sees both of the hard drives.
5) Run hdparm -i /dev/sdx on both of the drives to find which drive is your source drive and which drive is your destination drive
6) Once you know which drive is which you can start the clone process.
dd_rescue /dev/sdx(source disk) /dev/sdx(destination drive)
7) You will see the process start, just keep an eye on it, it might take a few hours for the clone job to finish, depending on the size of the drive.

Once the process is complete, there will be no notification, the transfer will just stop and you will see the terminal prompt again.
If you see a lot of errors or see that there is no more data being shown as succxfer: it means the drive got marked faulty by the kernel. At this point reboot the system and make sure you know which drive is which again, as it is possible they lettering might switch. Run the dd-rescue command again but this time with -r option. This will start the cloning again but this time will start from the back of the drive and will make sure to get the data that has not been cloned yet.

There is no guarantee this will recover your data, but there is a very high chance this will work and its free.

[abennett]

Wow, that's a brilliant idea! I'll give this a go as soon as I can!

The other idea I had was to somehow keep the NAS running in read-only mode, then I could go through and SCP the data I need from it... I haven't seen anyone discuss how to keep the NAS in read-only, aside from marking shares read-only.