ReadyNAS NV+: hot-removed third drive, now its "Dead"

[Kelly64]

I installed my ReadyNAS NV+ three days ago. My configuration consists of three 1.5 TB Western Digital WD15EARS drives. Today I did something that, in retrospect, was probably stupid. I pulled the third drive "hot" from the NV+: hot swap, right, so it should be fine?

I took the drive and ran wdidle3 on it to change the idle settings: that's all that was done to the drive. I put it back in the NV+, again "hot", and now it shows up as "dead".

I've tried restarting the NV+- no dice. The option to resync the array is disabled/greyed out, and the drive is listed as dead with this error message:

Code: Select all

The disk attached to channel 4 could not be used.  The most common reasons are RAID resync in progress, faulty drives, and disks that are too small to be added to the array.


I suppose it *could* be resyncing, but there are no status messages indicating that this is the case. My array is still accessible, but of course it is no longer redundant.

Is there any fix or bypass for this?

[chirpa]

Three disks, but it was placed in Channel 4? Did you put it into a new slot, or same slot? What positions were the disks originally in?

[Kelly64]

The drive was installed in the same slot it was removed from (channel 4).

I have three drives, installed in channels 2, 3 and 4. I hot-removed the drive from channel 4 (it was removed for about an hour), then hot-inserted it back into the same channel.

[mdgm]

Are you able to check the disk using vendor tools. If the disk really is bad, the fix would be to replace it with another disk with at least the same capacity.

[Kelly64]

Mdgm, I could pull it again and test it, but I'm perplexed. The drive was working acceptably before being removed: are you telling me that pulling it from the array while the array was "hot" has permanently and irrecoverably damaged the drive?

If so, maybe these shouldn't be called "hot pluggable"- maybe "hot destroyable" is more appropriate?

[TeknoJnky]

well realize that while hot pull/plugin is generally acceptable, if not careful during the process its completely possible to cause damage to the drive if handled roughly. Remember that the drives are mechanical and sudden jarring of spinning platters can easily cause damage.

Anyway, I would (carefully) hot pull the drive again, make sure it spins up and is recognized in the PC and run a quick smart check, then hot plugin to the nas again.

[mdgm]

well realize that while hot pull/plugin is generally acceptable, if not careful during the process its completely possible to cause damage to the drive if handled roughly. Remember that the drives are mechanical and sudden jarring of spinning platters can easily cause damage.

Yeah, hot-pull is typically recommended when the disk is already dead. See Misconception #3 inthis article. Of course you were pulling the drive for a different reason to the one in the article, but the explanation is still useful.

[Kelly64]

Okay... I'll shut down the ReadyNAS before I pull the drive, hook it up to my PC again, and run diagnostics.

If it comes back clean, what can I do to get the NV+ to re-add it to the array and resync properly? Do I have to perform a low-level format? Or... ?

[mdgm]

If it comes back clean, what can I do to get the NV+ to re-add it to the array and resync properly? Do I have to perform a low-level format? Or... ?

Carefully hot-add the disk.

If you insist on adding it while the NAS is off, then a low-level format is required.

[Kelly64]

If it comes back clean, what can I do to get the NV+ to re-add it to the array and resync properly? Do I have to perform a low-level format? Or... ?

Carefully hot-add the disk.

If you insist on adding it while the NAS is off, then a low-level format is required.

It isn't a matter of insisting: however, when I very carefully hot-added it previously, it showed up as "dead" with the message I've noted. And by the way, the drive is showing up in WD Data Lifeguard as perfectly healthy.

Question: in the event of a "successful" hot-add, is it reasonably to assume that the drive would show up as something other than "dead"? This may sound like a stupid question, but the message from the NV+ suggests that one of the reasons for the drive showing up "dead" is that it is resyncing. Personally, I'd think a drive that is resyncing should show up as... resyncing, not dead.

But as I say, the message makes it kind of unclear what is going on/what to expect.

[mdgm]

Your drive not being on the HCL, does complicate things, but yes if the drive is working, I wouldn't expect it to show up as dead.

Try hot-adding it again, and if it doesn't work, you could then try opening a technical support case. Post your case number here.

[Kelly64]

I tried hot-adding the drive again with the same results (i.e.: it shows up as "Dead").

I've opened a trouble ticket (The reference number is 11252889). I'm tempted to do a low level format on the drive and hope that it will simply rejoin the array and resync afterwards, but I would also like to get to the bottom of the problem.

[Kelly64]

The solution to my problem was to perform a full default reinstall/reformat. I may have been able to get away with a full zero-fill on my "dead-to-the-NAS" (but perfectly functional to manufacturer diagnostics) drive, but that would have taken in excess of 80 hours according to the Western Digital Lifeguard utility.

Things that I learned from Netgear's support:

• A drive that is part of XRAID array is *not* hotswappable. It should never be removed once it is part of a partition per the Netgear tech
  

  Code: Select all

  - Well if the drive is running a Xraid and each of the drives have allocated disk space to the raid then the drives are not hot swappable. Unless you power the unit down and remove the drive then add the drive but only when the unit is off.

- Only when the drive does not have a partition allocated is it a hot- swappable drive.

  
  To be clear, my drive did not become "dead" because it was in any way defective, or because it is not on the hardware compatibility list. It failed because that is what will happen when you remove a drive from an active XRAID partition. Period. Maybe this behaviour is different with different RAID types (e.g.: RAID5 instead of XRAID), or maybe the technician is repeatedly wrong. But I'm inclined to believe what they are saying, since it certainly matches my experience.
• A low level drive format (zerofill) will only reliably restore a drive to life if the entire surface is zero-filled. The WD Data Lifeguard utility's "quick zero fill", which fills the first and last million sectors of the drive, will *not* reliably work.
  

  Code: Select all

  - There isn't a way to format just one drive with out doing a factory reset on the nas.

- You only option is a full zero format on the drive because the quick format sometimes leaves the miscellaneous data that can cause the disk to render as dead in the NAS.

  
  

Note that I did in fact try a quick-zero fill (the drive still showed up dead) but did not complete a full 80 hour zero fill to try that. Performing a full factory reset seemed like a quicker and more reliable way to fix my situation.

The end result: a drive in a ReadyNAS NV+ isn't "hotswappable" in any reasonable sense of the word, since a drive becomes part of the XRAID partition as soon as you insert it, ergo it becomes unswappable at the same time. And there is no reasonably efficient/practical way to force a drive out of an array and cause it to be "re-initialized"- performing an 80 hour + zero fill is not practical.

It would be nice to see at least one of these limitations (e.g.: the ability to force re-initialization of a single drive without bare-metal reformatting) overcome in a future update of RAIDiator.

[mdgm]

A drive that is part of XRAID array is *not* hotswappable. It should never be removed once it is part of a partition per the Netgear tech

A drive is hotswappable. If it dies, you can remove it and replace it with another disk. Or if you no longer intend to use the disk you can remove it and replace it with another. That is what hotswappable is referring to. Removing a disk hot, planning to reinsert it later is not what you should do.

Yeah, hot-pull is typically recommended when the disk is already dead. See Misconception #3 inthis article. Of course you were pulling the drive for a different reason to the one in the article, but the explanation is still useful.

The end result: a drive in a ReadyNAS NV+ isn't "hotswappable" in any reasonable sense of the word

It is hotswappable as I mentioned above. When you remove a disk it hot, you can replace it hot with another. Damage this does to the old disk is pretty irrelevant as the disk should be either dead, at risk of dying or have too small capacity to be of any more use in your array if you hot-remove it. If you're inclined to remove a disk hot and then re-add it, e.g. thinking of this as a method of backup if you have just two disks in your array, then you don't understand what RAID is (See Misconception #3 inthis article). Removing a disk to test using vendor tools should be done cold and the NAS should stay off until the disk is put back in if it passes the tests.

[Kelly64]

The end result: a drive in a ReadyNAS NV+ isn't "hotswappable" in any reasonable sense of the word

It is hotswappable as I mentioned above. When you remove a disk it hot, you can replace it hot with another. Damage this does to the old disk is pretty irrelevant as the disk should be either dead, at risk of dying or have too small capacity to be of any more use in your array if you hot-remove it. If you're inclined to remove a disk hot and then re-add it, e.g. thinking of this as a method of backup if you have just two disks in your array, then you don't understand what RAID is (See Misconception #3 inthis article). Removing a disk to test using vendor tools should be done cold and the NAS should stay off until the disk is put back in if it passes the tests.

mdgm, you might give some second thoughts to your proclamations from on high. I am quite familiar with RAID configurations, having worked with high end EMC and IBM SAN arrays. I first used RAID in the early 90s when it was still mostly experimental. You may have such experience as well, but apparently you are choosing to ignore it here.

A well designed modern hot-swappable device will pre-safe the drive on hot removal- that is, on detection of a removal action, the heads are immediately parked. Some systems are fast enough that they flush buffers correctly in this situation as well- I know this because I have used high end systems.

I accept that a drive removed from a hot array may fail inasmuch as it will not longer be in sync. Further, in the case of an entry level solution like ReadyNAS the drive may be in an "failed" state as the system apparently nothing to "pre-safe" the drive on removal.

However, the drive was in no way physically damaged by being removed from the array. It ran through several manufacturer tests post-removal without a single surface or mechanical error detected. The problem here is that the ReadyNAS s incapable of re-initializing a drive in such a state, meaning that a drive once removed from ReadyNAS is essentially garbage- this is NOT hot removable. It is, as I say, "hot destroyable".

To re-iterate: a drive once removed from an active ReadyNAS XRAID configuration is useless, regardless of its physical state, short of performing a full bare-metal reformat of the entire array in which it participated or performing a complete zero-fill of the drive. I'm not trying to use the removed drive as a backup: I'm expecting to be able to put the drive back in and use it as a drive- that is, to have it resync as it did originally.