Backup ReadyNas to ReadyNas with rsync and exclude lists

[datamgmt]

Hi,

I have two ReadyNas that are successfully doing backups (between two sites) via rsync. There is one directory that we do not wish to copy over because it contains large, temporary media files (in excess of 2GB).

Using standard rsync you can do:

rsync --exclude-from=<path>

Does anyone know how I can achieve this on the ReadyNas ?

[tlyczko]

I would also like to know if this is possible too. Thank you, Tom

[ewok]

You would have to ssh in and make the change manually.

[ianmacd]

You would have to ssh in and make the change manually.

I, too, need the ability to pass an arbitrary number of extra flags and parameters to rsync when it performs back-ups.

For example, I need to back up a chroot'ed BIND installation that features a /proc directory within the chroot'ed environment. If I don't use '--exclude chroot/proc' when backing this up, madness ensues and the job fails.

FrontView is inadequate for this. There really ought to be a text field in which one can add the necessary extra options for insertion in the rsync command line that is ultimately invoked.

Lacking this, I finally succumbed to the urge to install the EnableRootSSH patch and go poking around in search of something I could easily hack to get the job done.

Well, I found it. /frontview/bin/backup is the Perl script that handles the back-ups. It's invoked by each of the scheduled back-up jobs, which are listed in /etc/cron.d/frontview-backup.

Happily, the extra functionality can be added to /frontview/bin/backup in about 15 lines of Perl. Here's how I chose to implement it: The patch below works on RAIDiator 4.1.3 (and possibly others).

Code: Select all

--- backup.orig   2008-09-25 15:56:13.000000000 +0200
+++ backup   2008-09-25 22:19:35.000000000 +0200
@@ -244,6 +244,22 @@
   $rsync_option .= " --delete";
}

+my $rsync_option_file = '/frontview/conf/rsync.conf';
+if( $backup_source_share_proto eq 'rsync' && -f $rsync_option_file )
+{
+  open(RSYNC, $rsync_option_file);
+  my @rsync = <RSYNC>;
+  close(RSYNC);
+
+  my $row = (grep(/$backup_source_path/, @rsync))[0];
+  if( $row )
+  {
+    my ($extra_options) = (split $FS, $row)[1];
+    chomp($extra_options);
+    $rsync_option .= " $extra_options";
+  }
+}
+
my $mount_point = "/job_${job}";
my $login_param;
my $source_path = "/$backup_source_share_proto/$backup_source_path";


I wanted to avoid having to repeatedly hack system files when a back-up job with extra rsync options needs to be modified, so I chose to have the new functionality use its own configuration file, /frontview/conf/rsync.conf.

That file should contain one line of additional rsync configuration per back-up job. Each line should contain two fields, separated by the standard FrontView field separator, two exclamation marks (!!).

The first field is the back-up source, which uniquely identifies the back-up job by host::module/path.

The second field consists of the options that should be inserted in the rsync command line before it is invoked.

In my case, /frontview/conf/rsync.conf looks like this:

Code: Select all

foo.caliban.org::slash/var/named!!--exclude chroot/proc --delete-excluded


Now, when an rsync back-up job is run that has foo.caliban.org::slash/var/named as its source, '--exclude chroot/proc --delete-excluded' is added to the rsync command line.

The nice thing about this approach is that /frontview/conf/rsync.conf can be maintained and edited on some other machine and ssh'ed into place on the ReadyNAS. There's no need to work directly on the ReadyNAS, which is important to me. It's an appliance and I want to treat it as such. Logging into it and performing this hack was the last resort for me.

Some would say that, if you need this level of control, you should be configuring the back-up job on the client and pushing your data to the ReadyNAS. However, I need to back-up multiple machines and I want all of my back-up jobs centralised on the ReadyNAS. This is an entirely reasonable approach to managing back-ups.

Obviously, I would like to see this functionality added to FrontView. I don't like the fact that I've potentially endangered my right to tech support by installing EnableRootSSH. I also don't like poking around on an appliance that wasn't really intended to be manipulated at this level. I consider it a dangerous activity.

Anyway, this works for me. If you know what you're doing, it will work for you, too.

[ianmacd]

Here's an improved version of my patch. Previously, the extra rsync options would only be used if the back-up source was an rsync server. The new version will also use the options when the back-up destination is an rsync server.

Let's suppose your /frontview/conf/rsync.conf file looks like this:

Code: Select all

foo.caliban.org::slash/var/named!!--exclude chroot/proc --delete-excluded
127.0.0.1::USB_HDD_1/backup!!--size-only


The first line is the configuration for the job described in my last posting.

The second line shows a new job, that backs up a local ReadyNAS share to a USB-attached hard drive. The job could also be configured as a simple share back-up, but that wouldn't allow files that have since been removed from the source to also be removed from the destination. Only rsync jobs allow this (the --delete flag), so we have to use that, even though both source and destination are actually local.

The --size-only flag is often needed when the source and destination file-systems aren't the same time. If copying from Linux ext3 to Windows FAT, for example, there will be discrepancies in the timestamps that cause all files to be copied from the source to the destination, defeating the main purpose of rsync. --size-only tells rsync to use only the size of the file to determine whether or not the file has changed on the source.

Code: Select all

--- backup.orig 2008-09-25 15:56:13.000000000 +0200
+++ backup      2008-10-04 00:43:28.000000000 +0200
@@ -244,6 +244,29 @@
   $rsync_option .= " --delete";
}

+my $rsync_option_file = '/frontview/conf/rsync.conf';
+if( -f $rsync_option_file && ( $backup_source_share_proto eq 'rsync' ||
+                              $backup_dest_share_proto eq 'rsync' ) )
+{
+  open(RSYNC, $rsync_option_file);
+  my @rsync = <RSYNC>;
+  close(RSYNC);
+
+  my $row = (grep(/$backup_source_path/, @rsync))[0];
+
+  unless( $row )
+  {
+    $row = (grep(/$backup_dest_path/, @rsync))[0];
+  }
+
+  if( $row )
+  {
+    my ($extra_options) = (split $FS, $row)[1];
+    chomp($extra_options);
+    $rsync_option .= " $extra_options";
+  }
+}
+
my $mount_point = "/job_${job}";
my $login_param;
my $source_path = "/$backup_source_share_proto/$backup_source_path";


[StGeorge]

Ian-

What's the prospect for adding a service to the current array of default services on these readynas devices using the basic hacking method that you've outlined above...?

Specifically, I'm thinking of how we could add the tivodesktop server so that our LANs could utilize the readynas to serve up photos and music to tivo boxes in our home, which is currently being done by a pc. The readynas is already storing / backing up all of our music and photo library, so why not take the next step?

Thanks for your thoughts on this....

[ewok]

What's the prospect for adding a service to the current array of default services on these readynas devices using the basic hacking method that you've outlined above...?

You'd probably want to use the add-on mechanism for this rather than hacking files on the NAS.

[jleinbach]

Ian,

Thank you so much for posting this.

I am in a similar boat: I totally agree with you about the purpose of appliances (and the need to not void rights to tech support as well), but i need to get my ReadyNAS to run rsync with parameters that I can't control through FrontView.

Your solution is about as uninvasive as it gets so I am preparing to ride on your coat tails (thanks again for bothering to post the solution). I just wanted to see if you had run into any problems since implementing this before I jump in.

Also, I actually need to get FrontView to STOP adding the "-l" paramter because the rsync server i'm backing up to (from the ReadyNAS) is ibackup and their server rejects connections with this parameter (a documented feature). I'm not familiar with perl at the moment so it's not clear to me if this will be doable/easy.

Lastly, to the tech support folks: is there an effort underway to build an add-on for this? If so I might wait for that as the cleanest way to get the changes.

Thanks in advance for any feedback you can offer.

Jared

[ianmacd]

Ian,

Thank you so much for posting this.

I am in a similar boat: I totally agree with you about the purpose of appliances (and the need to not void rights to tech support as well), but i need to get my ReadyNAS to run rsync with parameters that I can't control through FrontView.

Your solution is about as uninvasive as it gets so I am preparing to ride on your coat tails (thanks again for bothering to post the solution). I just wanted to see if you had run into any problems since implementing this before I jump in.

Also, I actually need to get FrontView to STOP adding the "-l" paramter because the rsync server i'm backing up to (from the ReadyNAS) is ibackup and their server rejects connections with this parameter (a documented feature). I'm not familiar with perl at the moment so it's not clear to me if this will be doable/easy.

Jared

Hello Jared,

Thanks for your kind words. Yes, the lack of rsync control is frustrating. Flags like --exclude are absolutely essential to me.

To remove the -l option from the back-up jobs, just do the following. This solution assumes RAIDiator 4.1.4, but should be similar or identical for other versions.

Go to line 68 of /frontview/bin/backup, where you should find the following:

Code: Select all

my $rsync_option = "v --links ";

Change this to:

Code: Select all

my $rsync_option = "v ";

Of course, this will affect all of your back-up jobs. If this is unacceptable, you'll need to add code to change the options on a per job basis. That's trickier.

And yes, my hack is still working well for me months later. The only issue with it is that installing a new version of the firmware blows it away.

[jleinbach]

Ian, Thanks so much. That's perfect. I never need the --links switch so that fix will be do the trick. I feel safe making an alteration as innocuous as that. I will actually need the exclude too ultimately and will probably use your approach to have a seperate file for the additional paramter(s), but for now just getting things going with removal of the --links switch will do the trick. It does seem (given how it is all implemented) that adding a text box to FrontView to allow for the manual setting of switches in rsync would be a trivial enhancement for them to implement. Maybe we'll see it in the next release. Thanks again.

[ewok]

It does seem (given how it is all implemented) that adding a text box to FrontView to allow for the manual setting of switches in rsync would be a trivial enhancement for them to implement.

It's on our todo list, along with hundreds of other things.

[dmcguire62]

I'd like to implement this on my ReadyNAS, but /frontview/bin/backup on mine is a compiled file:

readynas:/etc# file /frontview/bin/backup
/frontview/bin/backup: perl script text executable

My NAS is a ReadyNAS Pro Business Edition with RAIDiator 4.2.4

Any suggestions where I should look for the raw perl code?

Thanks!

[ianmacd]

I'd like to implement this on my ReadyNAS, but /frontview/bin/backup on mine is a compiled file:

readynas:/etc# file /frontview/bin/backup
/frontview/bin/backup: perl script text executable

My NAS is a ReadyNAS Pro Business Edition with RAIDiator 4.2.4

Any suggestions where I should look for the raw perl code?

It's executable, yes, but that doesn't mean to say that it's compiled code. Just edit the file and you'll see the Perl code. It's interpreted at run-time.

[gdietz]

Thank you so much for this code update. I was looking into it and considering using it as a roadmap - but I kept thinking the lack of the settings being visible to the UI would be problematic to someone other than me looking at the interface. I then looked at the config files and considered that maybe there is even a slightly better way to do this vs having a separate config file.

The /etc/cron.d/frontview-backup file for rsync contains lines that look like...

5 19-19/24 * * sun,mon,tue,wed,thu,fri,sat root /frontview/bin/backup 001 enable 'remote!!rsync!!192.168.1.10::folder!!!!!!' 'local!!backup!!serverx-backup/folder!!!!!!' 'FIRST_TIME!!0!!1!!1!!0' &> /dev/null

Ultimately the "192.168.1.10::folder" variable gets put into $backup_source_path which is then used on 1 of 2 lines similar to...

${password_param} $nice rsync -aA${rsync_option} '${login_param}$backup_source_path/.' '$dest_path' &> \${TEMP_COPYLOG}

It seems it would be possible to use the web-ui "Path" field to quietly embed the command line options. e.g. "folder --exclude blah --delete" - these would make it into the backup file as "!!192.168.1.10::folder%20--exclude%20blah%20--delete!!"

Someone more familiar with perl string parsing could then split the path at the first "--" and auto append the options to the rsync options and remove it from the path...(excuse the pseudo code - I know zero perl)...

if $backup_source_path contains "--" then
$rsync_option += [ everything in $backup_source_path starting after the first -- ]
$backup_source_path = [ everything to the left of the first -- in $backup_source_path ]
end if

The same could be done with the dest path as well. So without requiring separate config files you could override options and see the overrides in the UI. Obviously there are some issues with this - possibly some special characters that can't be saved using the UI (e.g. I would have suggested a "folder[--exclude blah --delete]" syntax except that the [ ( { characters are not allowed by the UI and I am assumming that the path doesn't have a "--" in it.

It could also be extended to have something like "folder --exclude blah --delete_" where the the "_" at the end of the parameters signifies that rather than $rsync_option += it should be a complete override of the options.

I would happily do this myself and post it to the site if I was even remotely competent in Perl - I am hoping that this model makes sense to someone who knows perl and they may want to take a swing at it.

[ianmacd]

Thank you so much for this code update. I was looking into it and considering using it as a roadmap - but I kept thinking the lack of the settings being visible to the UI would be problematic to someone other than me looking at the interface. I then looked at the config files and considered that maybe there is even a slightly better way to do this vs having a separate config file.

I decided to use a separate config file precisely because it was an uninstrusive approach. I needed to patch only one file and no changes were required to the FrontView UI.

As you point out, the integration could be better, using the UI instead of ssh, but the changes to the code would be greater and more difficult to maintain across firmware releases. As it stands, the same patch has worked for me across several firmware upgrades.

Maintaining the extra parameters also has the advantage that the file can be managed on a separate system and rsync'ed into place after changes are made.

I hope to eventually see the feature in a future firmware release.