How does RSYNC "verify" integrity of a backup???

[winger]

I have performed a factory reset my ReadyNAS Pro Pioneer (RNPP) unit. and now want to ensure the restore of the original data from an external USB drive (connected to the NAS' USB port) is copied without any data corruption. I am trying to confirm color=#0000FF]whether RSYNC is a good tool to use to verify the copied data's integrity[/color].

I have ready many posts that suggest using RSYNC to verify a backup. One such example:

viewtopic.php?f=66&t=51339&p=292769

...

After doing the backup to the USB disk you can verify it using a Rsync backup to USB disk

My questions are:'
1) After the initial restore to the RNPP unit (using FrontView's backup job, Source=Share-USBDrive; Target=Share-RNPP Share Drive), I set the backup type to RSYNC.

Do I then perform a FULL backup again?

2) The expectation is nothing changes during this first RSYNC backup. Question - so, how does RSYNC actually verify the backup? Does it to a byte-for-byte comparison or just a date/time comparison?

[mdgm]

You do an incremental backup. It compares the files on the source and destination. If the files are different the changes will be transferred across.

Edit: not comparison of checksums, see below.

[winger]

You do an incremental backup. It compares the files on the source and destination. If the files are different (I.e. checksum doesn't match) the changes will be transferred across.

First, THANKS SO MUCH for the fast response!!!

So, you are saying the (first) RSYNC backup (after the initial full backup/transfer of data) does a checksum verification on each file (source versus target) to determine if two same-named files are different?

I am not clear on this because several posts on this forum state that RSYNC uses timestamp/filesize to determine if a file (between source and target) has changed, then if a difference is detected, RSYNC does the delta calculation to determine what delta gets backed up.

Here is one such post:
viewtopic.php?f=31&t=27318&p=152960

What I want to happen is for rsync to compare dates and checksums of the file and
to not backup files that have not changed so that the network does not get hit very
hard and the disks to not get too much usage.

That's what rsync does, using timestamp and filesize to check if a file needs to be transferred. Using checksums would be very slow....

[mdgm]

Oops. I guess I was wrong then. Using file size and timestamp should be sufficient because one of those is slightly off if the file wasn't copied successfully.

[winger]

Oops. I guess I was wrong then. Using file size and timestamp should be sufficient because one of those is slightly off if the file wasn't copied successfully.

LOL, are you sure you are wrong? Again, I have seen BOTH sides of the story.

OK, let me ask this. IS there a definitive way to do the checksums verification natively on the NAS?

I mean, I can use/run FreeFileSync (http://freefilesync.com/) from my network-connected PC to compare the USB version of the file versus that on the RNPP, but that takes ALONG time because of the time it takes to pull data across my network (100 not 1000Mbps) and my PC is OLD (~4+ yrs laptop), so I prefer not to go that route.

[mdgm]

Just had a look and there is an option for rsync to compare (to decide which files have changed) by checksum. From reading your quote of ewok's it sounds like this option is not used by Frontview backup jobs.

If you have SSH access you could run rsync manually from the command line with the relevant option specified.

[winger]

Can we get additional advice here on whether the FrontView Rsync does do the checksum?

Here is a post that makes it sound like it does. I just got SSH access to the RNPP unit, but am not familiar with the commands and the various options, so my preference is to use FV's Rsync option.

viewtopic.php?f=31&t=53612&p

Yes. Because of the verification process in rsync it is like watching paint dry to do a full backup with rsync. Setting it up as NFS creates the backup, and then rsync only has to verify the check figure for each file to make sure all the data is there. Then every time you run the rsync job, it only copies over the changed portions of the file, so the "incremental" is very fast. If you were going from X-86 to X-86 it would take less than 1/2 the time used by the Duo to X-86, (or X-86 to Duo) but we are still only talking minutes to update gigabytes of data. Yesterday I added about 40GB of video files to my NAS1, and checking the logs from last night, it took about 24 minutes. If this had been from my NVX to my NV+ I would anticipate it would take around 45-50 minutes. But, I don't add 40 GB of data very often.

[mdgm]

ewok is part of the Jedi Council so I would take his explanation over what users (e.g. myself) have thought.

If it notices a different timestamp or filesize Rsync will then do the calculations necessary to transfer just the changes to the file across.

[winger]

ewok is part of the Jedi Council so I would take his explanation over what users (e.g. myself) have thought.

If it notices a different timestamp or filesize Rsync will then do the calculations necessary to transfer just the changes to the file across.

LOL Jedi Council, well OK ; )

Anyways, regarding the size or timestamp. IF there is a slight corruption during the transfer, I assume the target file's timestamp stays the same as the source file's timestamp. Will even the slightest corruption result in a file size difference that can/will be detected by the Rsync process?

[ewok]

Rsync, by default, will use the timestamp and filesize to determine if a file has changed. The Backup Manager has no facility to use rsync checksums, but feel free to create a post in the Feature Requests forum.

From the rsync man page:

Code: Select all

Rsync  finds  files  that  need to be transferred using a "quick check"
algorithm (by default) that looks for files that have changed  in  size
or   in  last-modified  time.


[TeknoJnky]

looks like only an option to add -c to the parameters would be needed.

-c, --checksum
This changes the way rsync checks if the files have been changed and are in need of a transfer. Without this option, rsync uses a "quick check" that (by default) checks if each file's size and time of last modification match between the sender and receiver. This option changes this to compare a 128-bit checksum for each file that has a matching size. Generating the checksums means that both sides will expend a lot of disk I/O reading all the data in the files in the transfer (and this is prior to any reading that will be done to transfer changed files), so this can slow things down significantly.
The sending side generates its checksums while it is doing the file-system scan that builds the list of the available files. The receiver generates its checksums when it is scanning for changed files, and will checksum any file that has the same size as the corresponding sender's file: files with either a changed size or a changed checksum are selected for transfer.

Note that rsync always verifies that each transferred file was correctly reconstructed on the receiving side by checking a whole-file checksum that is generated as the file is transferred, but that automatic after-the-transfer verification has nothing to do with this option's before-the-transfer "Does this file need to be updated?" check.

For protocol 30 and beyond (first supported in 3.0.0), the checksum used is MD5. For older protocols, the checksum used is MD4.

[winger]

Rsync, by default, will use the timestamp and filesize to determine if a file has changed. The Backup Manager has no facility to use rsync checksums, but feel free to create a post in the Feature Requests forum.

From the rsync man page:

Code: Select all

Rsync  finds  files  that  need to be transferred using a "quick check"
algorithm (by default) that looks for files that have changed  in  size
or   in  last-modified  time.


Thanks for responding, Ewok.

Do you know the answer to my question above (when using the FrontView RSync), which is:

.... Will even the slightest corruption result in a file size difference that can/will be detected by the Rsync process?

For your suggestion on placing a post in the requested features section, will do.

[ewok]

Do you know the answer to my question above (when using the FrontView RSync), which is:

.... Will even the slightest corruption result in a file size difference that can/will be detected by the Rsync process?

The answer is no. Consider, for example, the transfer of a text file that contains "hello world". If the file gets corrupted such that the file now contains "hallo wurld", the filesize and timestamp would still be the same. The best you could do without opening up the ReadyNAS would probably be to run checksumming software from your PC, but that would involve a lot of network traffic. An interesting idea for an addon would be one that does an md5sum on every file in the data volume and makes the output available.

[TeknoJnky]

or run rsync with -c and then it will compare checksums of both source and destination to determine whether to copy the file/delta data.

[ewok]

Of course, that would be a good option, but you'd have to open up the NAS.