Successful Readynas X6 X-RAID data recovery

Please post any helpful and useful tips to maximize your fellow ReadyNAS users' experience.

Successful Readynas X6 X-RAID data recovery

Postby mjw » Sat Jun 30, 2012 6:30 pm

Writing this down in case it's useful to anyone.

A few weeks ago my old Readynas X6 would no longer boot but I was pretty sure the disks were all in decent shape. Firmware is 4.1.6 and it has 4x500GB disks with a 4KB block size and 512 byte sectors. It would start to boot and was visible for short periods via RAIDar but seemed to be caught in an endless loop from which it could never fully boot. I thought I had a USB backup of all the critical data, but it turned out that my sister had stored some of her critical files on a non-backed up share! So the recovery hell began.

Recovering large amounts of data on disks is a slow and tedious process. It took me 2 weeks with all the various things I tried that didn't work initially. Ideally I would have found a way to connect all 4 drives simultaneously via SATA cables, but in this case I'd given the NAS to my sister in the UK and all she had was a Windows 7 laptop with one USB 3.0 port.

The first step was to buy StarTech SuperSpeed USB 3.0 eSATA Hard Drive Docking Station so she could attach the disks via USB 3 to a laptop. Then I went to work via Teamviewer.

Getting Ubuntu Running on the Laptop
Initially I tried installing Ubuntu 12.04 Desktop onto a pendrive using Universal USB Installer. The advantage of this method was the disk transfers were quick but I found for some reason (perhaps the pendrive was poor quality) it was not stable and after a couple of days the pendrive would not successfully boot. I then switched to installing Ubuntu on top of VMWare. Disk transfers were much slower, and I couldn't get USB 3 speeds at all but it was stable.

Suggestions:
    Label each of your disks 1,2,3,4 etc so you know exactly which bay they came from.
    Make sure the computer is shutdown before connecting/disconnecting disks to be absolutely safe.

R-Studio for Linux
Next I needed to create images of each disk. I could have used the linux dd command, but I decided to try R-Studio since others in the forums had reported success with R-Studio for Linux. R-Studio for Linux has an option to Create a Disk Image (select the byte-for-byte). To have sufficient space to store 4 500GB disk images we had to buy two external USB drives (1TB plus 2TB). In retrospect it would have been quicker to have created the disk images from within Windows. In the end it took a few days.

Mount LVM-based volumes from loopback full disk images
The above link had some great instructions for mounting the Logical Volume using my disk images.

Using fdisk I was able to find the sector where my LVM physical volume started on each disk:
Code: Select all
Disk /dev/loop0: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders, total 976773168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xc2d98e1c
Device Boot Start End Blocks Id System
/dev/loop0p1 2 4096001 2048000 83 Linux
/dev/loop0p2 4096002 4608001 256000 82 Linux swap / Solaris
/dev/loop0p3 4608002 976736305 486064152 5 Extended
/dev/loop0p5 4608003 976736305 486064151+ 8e Linux LVM

The starting sector is 4608003 and since my disks have 512 bytes per sector, 512 * 4608003 = 2359297536

I ran the following commands in Ubuntu 12.04 64-bit (replace your path with whatever the correct path is to your disk images):
    #Install LVM2
    sudo apt-get install lvm2
    #Map the right offset location in each disk image to a /dev/loop device file using read-only mode
    #I don't know if it's important but I ran these commands in disk sequence i.e. disk1.dsk first, disk2.dsk second, etc
    sudo losetup -r -f "/media/FreeAgent GoFlex Drive/disk1.dsk" -o2359297536
    sudo losetup -r -f "/media/GoFlex Home/disk2.dsk" -o2359297536
    sudo losetup -r -f "/media/GoFlex Home/disk3.dsk" -o2359297536
    sudo losetup -r -f "/media/GoFlex Home/disk4.dsk" -o2359297536
    #See if Linux can now find 3 physical volumes (one drive is parity so we only see 3)
    sudo lvm pvscan

This resulted in the following output showing 3 physical volumes have been found - success!
Code: Select all
PV /dev/loop0 VG c lvm2 [463.50 GiB / 0 free]
PV /dev/loop2 VG c lvm2 [463.50 GiB / 0 free]
PV /dev/loop3 VG c lvm2 [463.53 GiB / 0 free]
Total: 3 [1.36 TiB] / in use: 3 [1.36 TiB] / in no VG: 0 [0 ]


Next I activated the LVM array:
    sudo lvm vgchange -ay c
    #Create a directory where the data will be accessible
    mkdir tera
    #Mount the Logical Volume to this directory
    sudo mount -o ro /dev/mapper/c-c tera

After this, all the data was available intact within the "tera" directory and I was able to copy it off to a spare USB drive.

Here's some useful links
http://home.bott.ca/webserver/?p=306
Also tried the acmtn vmware image mentioned in this thread, but found it was very old and had trouble getting it to work with the disks

Caveats
My device is a Sparc device using X-RAID (not X-RAID2) and has 4KB block size, not 16KB like the newest models. Instructions may differ for other Readynas models. Here's a good place to look if you have a different model: http://home.bott.ca/webserver/?p=306

Dead-ends
I tried Reclaime software which claimed to be able to automatically detect the correct RAID parameters. I initially thought I would need this because R-Studio requires a number of parameters before it can recover a RAID. I ran Reclaime in RAID 0 mode and also RAID 5 mode for several days and it was unable to determine the RAID parameters for my X6. In the end it turned out that all I needed was Ubuntu linux!
Last edited by mjw on Wed Jul 18, 2012 2:42 am, edited 2 times in total.
mjw
ReadyNAS Expert
 
Posts: 220
Joined: Tue Nov 29, 2005 9:05 pm
ReadyNAS: X6

Re: Successful Readynas X6 X-RAID data recovery

Postby mdgm » Sat Jun 30, 2012 6:35 pm

:) Nice writeup.

Are you able to mount the 2GB OS partition?

What are the contents of /var/spool/frontview and its sub-directories?
Useful links: My ReadyNAS Gear|FAQ|Hardware Compatibility List|Docs: Setup Guide, Manual|Downloads|Unofficial Tips|GPL|MDGM on Twitter|MDGM's Unofficial Guides
NB: A ReadyNas is not an excuse not to have a backup. Fire, theft, multiple disk failures, other hardware failure, floods, user negligence etc. can all result in loss of data.
How we users can contact NETGEAR Technical Support | Australia: 1300 361 254 / Other Numbers|Online Submission
Unofficial Guide for Moving from Sparc ReadyNAS to x86 ReadyNAS|Using Gmail with the ReadyNAS|XRAID Volume Size Calculator
User avatar
mdgm
ReadyNAS Enthusiast
 
Posts: 28621
Joined: Tue Feb 17, 2009 9:34 pm
Location: Down Under
ReadyNAS: RN312

Re: Successful Readynas X6 X-RAID data recovery

Postby mdgm » Sat Jun 30, 2012 6:43 pm

From your systems my suspicion is that you've run into an issue with some old firmware where the shutdown spool script is left behind leading to endless reboots. If this was the problem it could've been fixed remotely very quickly.
Useful links: My ReadyNAS Gear|FAQ|Hardware Compatibility List|Docs: Setup Guide, Manual|Downloads|Unofficial Tips|GPL|MDGM on Twitter|MDGM's Unofficial Guides
NB: A ReadyNas is not an excuse not to have a backup. Fire, theft, multiple disk failures, other hardware failure, floods, user negligence etc. can all result in loss of data.
How we users can contact NETGEAR Technical Support | Australia: 1300 361 254 / Other Numbers|Online Submission
Unofficial Guide for Moving from Sparc ReadyNAS to x86 ReadyNAS|Using Gmail with the ReadyNAS|XRAID Volume Size Calculator
User avatar
mdgm
ReadyNAS Enthusiast
 
Posts: 28621
Joined: Tue Feb 17, 2009 9:34 pm
Location: Down Under
ReadyNAS: RN312

Re: Successful Readynas X6 X-RAID data recovery

Postby mjw » Sat Jun 30, 2012 7:25 pm

mdgm wrote::) Nice writeup.

Are you able to mount the 2GB OS partition?

What are the contents of /var/spool/frontview and its sub-directories?


Yes, I just tried and was able to mount the root partition using the disk1.dsk image and an offset of 1024 bytes. Here's the content below /var/spool/frontview:
Code: Select all
ls -lR frontview/
frontview/:
total 12
-rw-r--r-- 1 98 98   89 Nov  8  2010 60_SERVICE_READYNASREMOTE
drwx------ 2 98 98 4096 Sep  2  2009 boot
drwx------ 2 98 98 4096 Aug  2  2009 shutdown

frontview/boot:
total 8
-rwx------ 1 98 98 1197 Mar 19  2007 00_UpdateVendorResource
-rwx------ 1 98 98 1942 Jul 13  2009 01_AddColdSpare

frontview/shutdown:
total 4
-rwx------ 1 98 98 3735 Jul 13  2009 080_Poweron


Does anything look amiss?
mjw
ReadyNAS Expert
 
Posts: 220
Joined: Tue Nov 29, 2005 9:05 pm
ReadyNAS: X6

Re: Successful Readynas X6 X-RAID data recovery

Postby mdgm » Sat Jun 30, 2012 7:28 pm

No. Looks like I was wrong unfortunately.
Useful links: My ReadyNAS Gear|FAQ|Hardware Compatibility List|Docs: Setup Guide, Manual|Downloads|Unofficial Tips|GPL|MDGM on Twitter|MDGM's Unofficial Guides
NB: A ReadyNas is not an excuse not to have a backup. Fire, theft, multiple disk failures, other hardware failure, floods, user negligence etc. can all result in loss of data.
How we users can contact NETGEAR Technical Support | Australia: 1300 361 254 / Other Numbers|Online Submission
Unofficial Guide for Moving from Sparc ReadyNAS to x86 ReadyNAS|Using Gmail with the ReadyNAS|XRAID Volume Size Calculator
User avatar
mdgm
ReadyNAS Enthusiast
 
Posts: 28621
Joined: Tue Feb 17, 2009 9:34 pm
Location: Down Under
ReadyNAS: RN312

Re: Successful Readynas X6 X-RAID data recovery

Postby mjw » Sat Jun 30, 2012 7:29 pm

mdgm wrote: If this was the problem it could've been fixed remotely very quickly.

I was planning tomorrow to put all the disks back into the X6 do a factory reset, but if a jedi is willing to take a look at fixing remotely that would be fantastic. I'd rather not have to wipe and start again if it can be avoided. Please let me know. Thanks.

P.S. Here's some log files that I copied from disk1 of the X6.
mjw
ReadyNAS Expert
 
Posts: 220
Joined: Tue Nov 29, 2005 9:05 pm
ReadyNAS: X6

Successful Readynas X6 X-RAID data recovery

Postby fbmachines » Mon Jul 02, 2012 6:52 pm

Thanks for the write up.

It will come in very useful for many users.
NAS 1 [File Server]: ReadyNAS NV+, Raidiator 4.1.10, 1024 MB, X-RAID, 4x Seagate ST31000340AS 1 TB
NAS 2 [Media Server]: ReadyNAS NV+, Raidiator 4.1.10, 1024 MB, X-RAID, 4x Samsung Spinpoint F4EG HD204UI 2 TB
NAS 3 [HDD Image Server]: ReadyNAS NV+, Raidiator 4.1.10, 1024 MB, X-RAID, 3x Western Digital RE3 WD1002FBYS-02A6B0 1 TB
NAS 4 [Backup for NAS 1,2 & 3]: ReadyNAS NV+, Raidiator 4.1.10, 1024 MB, X-RAID, 4x Hitachi HDS722020ALA330 2 TB
NAS 5 [Off-Site Backup for NAS 1,2 & 3]: ReadyNAS NV+, Raidiator 4.1.10, 1024 MB, X-RAID, 4x Seagate ST31500341AS 1.5 TB
NAS 6 [BitTorrent Client]: ReadyNAS Duo, Raidiator 4.1.10, 1024 MB, X-RAID, 2x Western Digital RE2 WD5001ABYS 500 GB
NAS 7 [CrashPlan Client]: ReadyNAS Pro-6, Raidiator 4.2.23, 4096 MB, X-RAID2, 6x Samsung Spinpoint F4EG HD204UI 2 TB
NAS 8 [On-site Backup]: ReadyNAS Pro-6, Raidiator 4.2.23, 8192 MB, X-RAID2, 6x Samsung Spinpoint F4EG HD204UI 2 TB
NAS 9 [Off-site Backup]: ReadyNAS Pro-2, Raidiator 4.2.23, 1024 MB, X-RAID2, 2x Western Digital RE3 WD5002ABYS-01B1B0 500 GB
NAS 10 [Testing]: ReadyNAS RN102, ReadyNAS OS 6.0.6, 512 MB, X-RAID2, 2x Seagate Pipeline ST3500321CS 500 GB

Network: ProSafe UTM25 :: Netgear GS724Tv3 :: HP Procurve 1810-24G :: Netgear WNDR3700v1 :: D-Link DGS-2205
UPS: CyberPower CP1000AVRLCD :: CyberPower CP850AVRLCD :: 4x CyberPower CP825AVRG :: APC Back-UPS ES 550
User avatar
fbmachines
Advanced ReadyNAS Expert
 
Posts: 594
Joined: Thu Dec 27, 2007 6:23 am
Location: Austin, Texas
ReadyNAS: Pro

Re: Successful Readynas X6 X-RAID data recovery

Postby mdgm » Tue Jul 03, 2012 3:18 am

Your disks look like they're quite old. Have you tried testing them using SeaTools?

Also have you run the memory test on the NAS a few times recently? If not, please do this.
Useful links: My ReadyNAS Gear|FAQ|Hardware Compatibility List|Docs: Setup Guide, Manual|Downloads|Unofficial Tips|GPL|MDGM on Twitter|MDGM's Unofficial Guides
NB: A ReadyNas is not an excuse not to have a backup. Fire, theft, multiple disk failures, other hardware failure, floods, user negligence etc. can all result in loss of data.
How we users can contact NETGEAR Technical Support | Australia: 1300 361 254 / Other Numbers|Online Submission
Unofficial Guide for Moving from Sparc ReadyNAS to x86 ReadyNAS|Using Gmail with the ReadyNAS|XRAID Volume Size Calculator
User avatar
mdgm
ReadyNAS Enthusiast
 
Posts: 28621
Joined: Tue Feb 17, 2009 9:34 pm
Location: Down Under
ReadyNAS: RN312

Re: Successful Readynas X6 X-RAID data recovery

Postby mjw » Tue Jul 03, 2012 4:05 am

I ran the memory test and that was fine. I'll try sea tools.
mjw
ReadyNAS Expert
 
Posts: 220
Joined: Tue Nov 29, 2005 9:05 pm
ReadyNAS: X6

Re: Successful Readynas X6 X-RAID data recovery

Postby mjw » Wed Jul 04, 2012 8:26 pm

I've created a new thread for the X6 troubleshooting to separate from the data recovery discussion.
mjw
ReadyNAS Expert
 
Posts: 220
Joined: Tue Nov 29, 2005 9:05 pm
ReadyNAS: X6


Return to User Submitted Tips



Who is online

Users browsing this forum: devilsrear and 1 guest