Robocopy

[brianadkins]

I used Robocopy to initially populate my ReadyNAS NV from a windows XP workstation. The total copy size was around 30 GB and I was very happy at the copy performance over my 100MB lan (much faster than the Drag-n-drop method).

I was happy until I repeated the identical robocopy command only a few minutes later... I noticed that a bunch of files got copied again. Since these had already been copied over once, they should not have been copied the second time.

I finally found that using the FFT option solved my problems. Using the FFT option tells Robocopy to assume FAT File Times (2-second granularity). After adding this option, I had no files recognized as 'newer' or 'older' that required re-synching on the ReadyNAS.


So now I always use this:
Robocopy \\source \\destination /e /z /fft

Links for Robocopy:
http://desktopengineer.com/story_20040426101005321?op=makePrintable&styleId=1010
http://www.ss64.com/nt/robocopy.html

Requires subscription:
http://emea.windowsitpro.com/Windows/Article/ArticleID/44324/44324.html

Snippet from http://www.screamingjungle.com/mt2/files/robocopy.pdf
Some third-party operating systems declare that their volumes are NTFS, but only implement FAT-style file times with a 2-second granularity. When copying to such a destination from a true NTFS volume, file time rounding may occur, along with unnecessary copying of file data in subsequent jobs. If you encounter this situation, you can use the /FFT (FAT File Times) switch to tell Robocopy to consider file times to be identical if they are within 2 seconds of each other, even though both source and destination appear to be NTFS. This eliminates unnecessary copying in this scenario.

Hope this helps ease someone else's frustration in the future...

-Brian

[yoh-dah]

Great tip. Thanks Brian.

[ginooken]

Could this be linked to the copying of unmodified files with the backup scheduler in the ReadyNAS 600 ( to an external USB drive ) ?

Thanks for an answer

Gino

[brianadkins]

In my particular situation, this was a brand new ReadyNAS with no backups configured. This robocopy job was the first time I had put any files on the NAS. Also, I do not yet have an external USB drive hooked up.

FYI, I also have a Linksys NSLU2 "NAS" with an external USB drive. I ran into this exact same problem last year with this setup but was not able to find out the FFT solution.

[ptr727]

I agree, great tip.

I just installed my NV and I was using RoboCopy to sync data from my SATA RAID5 device to my NV. I would have had the FFT problem on my next sync!

Enhancement request: Any chance the NV's filesystem could support high resolution (NTFS) timestamps?


Regards
Pieter

[CharlesLaCour]

I ran into the same issue.

That would be a good addition but I have a feeling that since the underlying file system is ext2/ext3 it will prevent having the same type of file timestamps that are supportd by NTFS. The standard ext2/ext3 inode has 3 time fields, last inode change, last file data change and deletion time. These timestamp fields have a resolution of 1 second. The only time field that maps from NTFS to ext2/ext3 is the file data modification time.

So, I would think that the underlying filesystem would have to be changed to allow for the timestamp compatability with NTFS.

[subs]

For all the years I've lurked on boards, this posting prompted me to actually create a login and post. This was exactly want I needed to know to make my robocopy backup files work (better) - I say better because as someone else pointed out, it's not really NTFS in the ReadyNAS, so there are still some copy timestamp hiccups, and some other strange behaviors, but this might make it actually workable!

Thanks.

[Sageth]

For what it's worth, I found that just using /FFT wasn't enough. For about the past year, I've used the following switches on my backup script:

/Z /MIR /TBD /FFT /NP /XO


/Z is network restartable
/MIR does a mirror image (can delete all of your information. Test first!)
/TBD waits for the share to be available (I had problems with network name cannot be found).
/NP makes it so that it doesn't show the progress (cleaner logs)
/XO excludes older files.

So the full example is:

robocopy t:\scripts \\NAS\scripts /Z /MIR /TBD /FFT /NP /XO /log+:"C:\Documents and Settings\MyUser\Desktop\robocopylog.txt"


Just figured I'd post my solution in the event that /FFT isn't the end-all solution for your needs.

It's pretty quick too. I can recopy my entire NAS (~150GB, Raid 1) in under 30 mins. I'm not sure if I'm the only one, but Radiator 4 p1 was a lifesaver in terms of file copy. It's back to the speeds pre-R3-p6.

[tahir]

Thanks for this, anyone got any idea what the difference between "*EXTRA File" and "New File" is?

I'm assuming "Newer" means that the source file is newer than the destination.

[Sageth]

"Extra File" I've only seen when using /MIR. This means that the file does NOT exist on your source, but DOES exist on your destination. As a result, the "Extra file" will be deleted from the destination.

"New File" means that it DOES exist on the source, but DOES NOT exist on the destination. As a result, the new file is copied to the destination.

Hope this helps.

[tahir]

Many thanks.

[subs]

Extra File appears in the log even if you don't have the MIR setting- which can be a bit unsettling to get used to.

[Lars]

Thanks for a great set of tips!!
Noticed this problem with robocopy right after I got the Readynas, but haven't had time to soak my head into it..

Love to see my robocopy jobs working again!

[Lars]

"Extra File" I've only seen when using /MIR. ...

'Extra file' + 'Extra dir' appears always in ordinary copyjobs for me(not using /MIR) - when target exists but not source as you described.

[JackoMiles]

"Extra File" I've only seen when using /MIR. ...

'Extra file' + 'Extra dir' appears always in ordinary copyjobs for me(not using /MIR) - when target exists but not source as you described.

Is it ok to delete the files or will it mess up?