A sad tale of ProSupport woe with ReadyNAS Pro Pioneer!

[ensign.fodder]

So I bought and equipped a ReadyNAS Pro Pioneer (6 Bay BYOD) NAS 7/2/2010 and added the ProSupport 24x7 Support and Maintenance upgrade.

Overall I have been pleased with the unit (started with 4 x 1TB 7.2K added 2 more, then converted over to 1.5 TB drives over a few months) as a CIFS/UNC NAS device. Tried iSCSI for VMware but the performance was not there but I teamed the NIC's so it maybe an issue of the overall congestion on the single interface.

What I have not been pleased about is the long wait times to reach a ProSupport tech, the times that the person answering was not in the storage group and had to put me back on hold till someone was available, that I cannot communicate with the phone support via the my.netgear.com section except to upload files, and that I cannot email the tech so I have to use the telephone which circles me back to the top of this set of complaints.

Original Unit - Hardware/backplane failure
It took three separate cases to get me back to an operation state.

1. 16134961 RMA
2. 16143235 Closed without my knowledge once phone call was complete
3. 16144283 Additional help in getting resync to complete

How it went down
7/14/2011 - See dead drive on Web UI (no messages/alert had been sent) and swap in cold spare, I am running X2 with single redundancy.

From Ticket/Tracking log for case # 16134961 Was doing maintenance on a user password and noticed on the volume tab (web interface) that there was a dead volume. Swapped cold spare for failed member but system did not detect and rebuild. ~ 10 minutes after swap the system popped the dead drive warning, sent email, and such.

Your RMA request for ReadyNASRNDP600E[XXXXXXXXXXXXX] has been approved by NETGEAR.
Your RMA number is: XXXXXXX

Completed RMA process 5:28 PM 7/14/2011

When I asked about when I would have the replacement unit the support tech was a little rude and really could not have cared said "Since the RMA was not processed or approved until after 5 PM EST the warehouse would not ship it until Friday for Monday delivery. When I explained that the only reason it was after 5 EST was due to the hold time, he said "Its not like we can make the shipping department work overtime to get your unit out."

Kimroth Y. at Tracelogix who actually handles the RMA for NetGear was absolutely outstanding! Kimroth was the person on the other end of the phone when I called the the RMA status phone number not realizing I was talking to someone at another company. When I explained my issue she explained how the process and procedures ensured I would not get my replacement before Monday. I begged and pleaded a little and she agreed to talk to her boss and see what could be done. From that point on she worked to make it happen, called to let me know it was going to ship, and sent me tracking information so I could follow it to my door.

Once I had the replacement unit I had a great number of issues with the migration to the replacement unit due to incompatible firmware version and eventually had to put the "dead" drive back in the NAS and get it to build consistency across all drives before I could migrate.

From Ticket/Tracking log for case # 16143235 Verify migration - just move drives in same slots, need to pull down configuration from website (currently does not complete), etc...?

*NOTE: Need to bring replacement drive firmware up to 4.2.17 to match existing chassis.

OS/Configs are RAID'ed across drives in hidden partition so migrating the drives migrates settings and such

Currently 60 minutes in to resync after power cycle of unit 0% progress
if no progress after 120 shutdown unit and move "hot" drives only then once running add "cold" spare
If progresses then wait until complete and shutdown unit and move all drives

Waited past 120 minutes and when 0% complete told unit to shutdown.

Waited 30 minutes and now device is not responsive to web admin but front says "shutting down 0.2% Resync" so I assume it will have to sync to shut down.

From Ticket/Tracking log for case # 16144283 Unable to boot replacement unit with a single drive inserted.

Unable to boot replacement unit with five drives.

Replaced drive and allowed to rebuild redundancy

RAID sync started on volume C.

[Fri Jul 15 17:51:52 EDT 2011]

RAID scrubbing finished on volume C.

[Sat Jul 16 17:48:04 EDT 2011]


So at this point I am back in business with a new chassis and life is good.

Till today when same situation of NAS not being available via CIFS/UNC

How it went down
From Ticket/Tracking log for case # 16458558 Unit was not responding via UNC path
Tried web interface and was able to log in but it never fully painted screen/UI
Went and tried various short pushes of power button without any effect
Did a long press of power button until unit shut down
Powered unit back on and watched boot process
When boot/sync completed displayed dead drive
Logged into web interface and confirmed issue
Still no email notice from the unit about issue
23+ minutes of hold time on ProSupport line until being dumped into voice mail
Called normal support line - had to work to get to case number and agent insisted they would get me to a person - several times on hold
Call time +15 - Told by agent that she cannot keep me on hold any longer but she will pass me to the ProSupport line and write up an escalation ticket to make sure someone picks up the call or if no one does someone will call me back.
Unit not responding on web interface but RAIDar showns unit - still able to access file shares
Call time +27 minutes - Dumped into ProSupport voice mail
Rebooting device and swapping disk 5 with prior disk 5 - which passes all Mfg Disk Tool test.
Unit "passed" Disk 5 which had only had SMART error previously but is one that "failed" in prior unit
Received emails from unit about removal/replacement
Resynching on "new" drive but unable to access Web UI

3 1/2 hours since first issue - no response from support.
4 1/2 hours since first issue - no response from support.

I decide to call the ProSupport line again and open web ticket #16460006
Within a minute of submission of web ticket agent on ProSupport line answered.
Tech was friendly and apologetic about prior issues and repeated failures to reach support and assured me he was going to work and resolve the case.

After going over what had happened today (and previously on prior unit) and asking several questions the only guess at cause is a corrupted OS image that made it over from the prior system and I should reload from the flash OS on device which I cannot do until the resynch is complete.

From Ticket/Tracking log for case # 16458558
Wait for full sync and then reload Firmware 4.2.17 then do OS reload on device (4.2.19 Beta T4 is on forums)

So all told today I have spent hours listing to how much NetGear cares and how an agent will be with me in just a moment.

I will update as the recover/restoration completes.

[ensign.fodder]

So device finished resync about 9:30 PM but since web UI is not responsive I will have to wait until I can physically be in the office to complete OS re-install.

[ensign.fodder]

Sun 8/21 @ 8:55 PM

Call Pro Support line and immediately answered call and was with a agent #5116

Was told how to reload OS from flash memory but instructions were wrong - told to use power button to select option not reset button

Called Pro Support back @ 9:01 PM after short time on hold agent #1375 answered

Heavy Indian accent - not technical then passed to #40168

Who:

• Failed to understand network protocol enough to understand that if Telnet to port 80/443 fails then there is no way to connect to ReadyNAS FrontView
• Repeated asked unhelpful questions - I asked to be escalated - more useless questions - I asked what happened to domestic support and told she would collect information for level 3 support
• Asked about number of drives - capacity - model number - my external IP address - asked to open TELNET port through firewall - I told her Telnet (Port 23) is not working inside so why would outside work
• Asked me to open SSH (Port 22) - did so and confirmed working inside - confirmed working outside - told level 3 would investigate and "get back to her and she would get back to me"

11 hours later no response

Mon 8/22 @ 10:37 AM

Called Pro Support and near first ring answer connected to agent #78250

Who explained:

• No need to open port due to prior tech misunderstanding how technology works
• Putting device into TECH MODE actually has it open outbound connection (firewall allows outbound instantiated connections) to NetGear NAS Shell Server
• I had been given wrong instruction and needed to use reset switch not power button to select

Who performed:

• Multiple attempts to fix flashing of OS by remove optional PHP add-on's, attempting to reload OS from new Download
• Having me re-enable Tech Mode and bump up to Engineering with a 24 to 48 hour response time unless identified issue sooner

Meanwhile I received call from MJ (no agent number and another heavy Indian accent) about needing to open port 23/22 to allow static NAT traffic to pass

Agent #78250 called back and believes a fix is in place and I need to reboot unit to confirm but has no idea who MJ is or why I needed to open the port.

Unit reboots and seems to be operating normally. Continue to monitor S.M.A.R.T. errors on slot 5.

PHP Add-ons are still listed and seem to not be able to be removed, but that is for another day.

[ensign.fodder]

So all is good now with the device but I am looking to migrate all my data to another device as soon as the budget allows its replacement.

This support experience will guarantee that I (and anyone I can influence) never buys NetGear equipment if support is going to be required.

[claykin]

May I ask why you waited 5 weeks to begin posting about your issues on the forum?

Separately, I do agree with you that the Pro Support team should act as professionally and expeditiously as is reasonable. You paid extra for this option and Netgear should take that seriously.

I would be interested in learning if you find another vendor with equal or better support options. I've yet to find one unless you want to consider Windows Storage Server.

[ensign.fodder]

May I ask why you waited 5 weeks to begin posting about your issues on the forum?

Separately, I do agree with you that the Pro Support team should act as professionally and expeditiously as is reasonable. You paid extra for this option and Netgear should take that seriously.

I would be interested in learning if you find another vendor with equal or better support options. I've yet to find one unless you want to consider Windows Storage Server.

Claykin - If you read the whole post (long I know) you would see that the only reason I included the 7/14 incident/RMA/etc is up to that point the Pro Support had been fine - or in line with the rest of our support agreements (RIM, IBM, MS, etc) and it was the short window from 7/14 incident to 8/20 incident and the subsequent experience for that problem that led to the post.

P.S. Just updated to 4.2.19 and the ReadyNAS has marked drive five as dead again so it looks like I am back on a support call shortly.

As to alternatives I have not tried another at this stage but I am looking at the QNAP or Synology devices. Several of my VMware friends use them in labs and have good things to say about the devices, will check and see about support.

[ensign.fodder]

After 14 hours without a response (not outside response window for web tickets) I called Pro Support and was almost immediately connected to agent #5040

Discussed several options and after review of logs believes issue is that disk is not spinning up quickly and as such is being marked as "dead"

Other options to be held for later (below) and for now wait until consistency rebuilt and then substitute new larger drive and confirm bay 5 is no longer an issue.

Post/follow up on results on web site

This was an excellent experience - agent answered quickly and had a good grasp on the issues and worked the problem to a reasonable testing/validation of root cause.

****HOLD FOR LATER****

Boot Menu - Test Disk option - several hours to run

Need to backup device and do a factory reset before returning data - backup entire volume to maintain file hierarchy and structure

===========================================================================
Customer Support case #16632923 - No response at last update of this post.

[chirpa]

We've notified the customer service team to review your cases and get an urgent follow up.

[amac27]

Hi ensign.fodder,

We apologize for the inconvience that this has caused you.

As of now we are reviewing all of your cases.

[ensign.fodder]

Received a update to my online case from Sr. Manager, Support Engineering last night and a phone call this morning.

Admitted that there are issues and apologized for them and said that they are working to address them and asked what else can be done at this time to help.

I appreciated the call and the apology it does go a long way in this day and age.

[ensign.fodder]

We've notified the customer service team to review your cases and get an urgent follow up.

Hi ensign.fodder,

We apologize for the inconvience that this has caused you.

As of now we are reviewing all of your cases.

Thanks!

[ensign.fodder]

Update on Sept 7th "drive dead"

Data consistency was built (on swapped drive - same as "dead" drive) Thu Sep 8 16:42:13 EDT 2011 but due to re-racking servers and equipment (to avoid consistency re-build interrupted by power outage) the new drive was not added until 9/12 and consistency was rebuilt and conpleted Tue Sep 13 05:35:11 EDT 2011

Swapped previously used but test OK Seagate ST31500341AS out and New Seagate ST32000641AS in and will continue to monitor the drive stats for any abnormalities.

[ensign.fodder]

9/18- Received notices from Esxpress - Backup still running - Determined ReadyNas is not responding to CIFS/HTTP - multiple short button pushes on front of device did not "wake it up" or produce any results
Pro Support is West Coast and does not start until 9:00 EST (6 AM PST)

Called back @ 9:25 and answered by - Agent #5105 - French accent
Asked to power off unit (hard off) and when I expressed concern about loosing log data helpful to resolving the problem I was told not to worry that we would be able to recover that once restarted.
Did as requested and emailed log bundle for review.
------------FROM REVIEW OF LOGS-------------------------------------------
Claimed we were running Beta 4.2.19 when running gold since Sept 7th
Told everything was fine with device but when pressed for why it was locked and non-responsive no explanation
Asked about network topology and other filler issues
Was told that the problem was a CIFS file transfer was too large and that is what caused the problem
Advised to wait and see what is the problem since it is "working now" but will not escalate - Told him that this behavior is the same as prior ticket #16458558 and after reviewing he said that was not the case.
When I reiterated that it was and asked to be escalated he said said no, that the unit was responding properly now and I would have to wait for the next occurrence of the problem. I asked to know when Agent #5040 was on shift and was told he was already but was unavailable and I should try back in an hour or so. I requested that he update the case with his notes and pass a request to #5040 to call me when he was available. #5105 said he would and the call was ended with niceties @ 10:37 (~70 minutes)

[ensign.fodder]

So now I have someone who is clearly a top support/engineering individual working on the problem. He roots around the NAS and makes a couple of simultaneous changes (listed at bottom) and asks me to make a backup so that in the future we can reset the device. Backup is a four day process for Full and when it completes his and Sr. Manager, Support Engineering email server rejects my mail (multiple times) but only when sent from my domain, Gmail has no problems. Sr. Manager, Support Engineering replies that top support/engineering individual is busy and will respond shortly. Three days later top support/engineering individual contacts me and asks that I send the backup logs. After some back and forth I am told:

Would you like to monitor it for a few weeks and then, if there are no further issues, we can likely say that the original issue was that the backup logs had filled the OS partition? If that does end up being the case, I’ve already spoken with the developers regarding this issue and they are working on the best way to address it.

So now we are six weeks and two days from then and device is back to not responding to CIFS requests and FrontView is not accessible beyond the username/password prompt.



1. Logging has been moved from the hidden partition on the disk to a varlog volume to provide more space to log issues to avoid overwriting but running out of space may have been the cause of the lock.
2. You have removed the two PHP add-on's that I did not need and was unable to remove. The Teaming add-on also does not show on the Add-on's page but the NIC configuration still shows as teamed.

[chirpa]

I poked the Support Manager to have your issue looked at by more eyes.