Encrypted, bandwidth efficient backups with Duplicity

[zero-not-hero]

Adding support for Duplicity: "a compact, elegant tool for automatically encrypting and storing remote backups in a bandwidth efficient manner" would be an incredibly useful (if not glamorous) addition to the ReadyNAS feature set. We use this brilliant backup method ourselves, but frustratingly we have to run this from a client PC (scheduled via cron) not the ReadyNAS unit.

The built in Rsync facility looks great on paper does not give us encryption of data which is absolutely vital. We must be able to store our client data securely encrypted off-site.

See an rsync.net article for a useful discussion and comparison of encrypted backup methods including duplicity: http://rsync.net/products/encrypted.html. A bit of experimentation with this will show you how good duplicity is.

Adding duplicity would be relatively trivial in terms of effort (adding it from standard repositories and then building/adding an interface) but would take the ReadyNAS product line into a different league for real world usefulness (for most enterprises). I really urge you to consider this as a 'killer' feature - it doesn't glitter but it is golden!

z-n-h

[Staffan]

The Pro model has encrypted rsync support. The NV/Duo does not have enough CPU power to handle the encryption and rsync work.

.staffan

[zero-not-hero]

The Pro model has encrypted rsync support. The NV/Duo does not have enough CPU power to handle the encryption and rsync work.

.staffan

Hi Staffan,

1) What we need is encryption of data so that when it arrives at our offsite backup server (in our case provided by rsync.net) it is safely encrypted so no one can access it there. I am assuming that the 'encrypted rsync support' is encryption in transit. If this is the case then this is no good for a large number of businesses that can't have unencrypted client data sitting on a third party server.

2) If the NV/Duo has CPU power to handle rsync it can surely run Duplicity which handles backup data encryption and transmission incredibly efficiently? It provides a smorgasboard of remote backup/restore options that would be incredibly useful. Also backups are likely to involve snapshots and happen at times that the ReadyNAS is barely used (i.e. late night/early morning).

z-n-h

[Jerry Leichter]

The repeated assertion that the older boxes "don't have enough CPU power to do encryption" strikes me as questionable. Encryption isn't that expensive these days. Oh, public key computations can be very CPU-intensive; and if you insist on an obsolete technique like 3DES you'll pay a lot for what isn't even state-of-the-art cryptography. But AES can be computed quite quickly, and if even that's too much, RC4 is very cheap - a couple of instructions per byte - and reasonably secure.

In comparison, rsync does tons of computation: Its whole recursive comparison algorithm trades CPU cost for transmission cost on the (still reasonable, in many situations - but by no means in all) assumption that doing a significant amount of computation is faster than sending data over a network link. (For devices like the NV on a gigabit Ethernet, this assumption is wrong - it's usually faster to just transfer all the data, duplications and all, then to figure out what not to transfer.)

Ultimately, one would have to test this; but I suspect the repeated assertions that "it won't work" have kept people from trying.

[Staffan]

As there is encrypted data on a third party server there is very simple to crack the data, specially if you know what kind of data it is. It might take a week or a year but it is not safe. And trusting another party's statement that the data is encrypted is not what I call good security.

.staffan, got his data in a bank vault

[jquast]

Hi,

After a lot of effort, I was able to compile and install duplicity on my ReadNas NV+.

Two things drove me. Firstly, the built-in backup job to my USB disk works only once, and fails due to a "read-only" filesystem every day thereafter. It simply needs to mount and unmount the drive. It really shouldn't keep the USB disk mounted all the while when its not using it -- I did everything I could to NOT share it and so on, but it requires manual click of the 'disconnect' button? is that right?

Anyway that's when I decided I'd stop messing with this **** web gui and just cronjob the damn backups my damn self. I'm also thinking a lot about remote backups. Something off-site. I recently lost a lot of data to disk failures.

After looking at size limitations and cost of remote site backup solutions, including netgear's own, and a weekend of investigation, I decided to use Amazon's S3 storage.

Which also drove me to the decision to use duplicity, which, for files that grow in size, will only transfer and store the differences within the files -- something vanilla rsync won't do (or 's3sync', the original S3 rsync solution).

From the FUD spread above, I used --no-encryption for a long while, until I finalized my 'frequently', 'daily', and 'weekly' backup actions. I found they all ran in ample time, so I generated the GPG keys and enabled encryption, and it approximately doubled the backup time. That's it.

So no, I don't feel the encryption overhead is too much for the ReadyNas NV+.

This small linux kernel and debian environment is very difficult to monitor system throughput with, but I'm very sure from years of system programming experience, that your internet upload speed, usb disk write speed, and raid disk read speed will be your watermark for all backups, local or remote.

And its a very good idea to encrypt. I didn't really want to either until I thought about it a long while. At least for me, I netboot a lot of unix machines over NFS, and I'm storing files like /etc/passwd, and subsequently backing them up. Anybody who gains access or permission to the backup store has full access to all of these files. And its even more serious for USB: If a thieve or attacker steals my USB disk, I don't want all of my most personal data there free to read by all.

Anyway, I'd like to share this work with everyone, but it's not easy. My coworker who also has a ReadyNAS NV+ immediately asked if it was a .bin package he could install to use it. There isn't.

I'm thinking at minimum a tarball package for ssh-enabled machines to install. Unfortunately I don't think I can really provide a debian package because of the dependency mess.

It also requires a bit of documentation, though a few blogs out there have more or less covered the aspects of gpg keys and duplicity and s3 storage, a 200-line README would be greately appreciated by anybody new to duplicity.

Restore is stupid simple. You just reverse the "source" and "destination" and duplicity knows what to do.

Unfortunately, you need your GPG key to restore, and the documentation needs to make this clear, because the user should know where their GPG key is stored, and to know to keep it secure and recoverable in event of disaster.

Anyway, I'd be happy to make a .BIN package if users weren't such a *****. Though I could build a tarball and a README, anybody interested?

[mdgm]

Two things drove me. Firstly, the built-in backup job to my USB disk works only once, and fails due to a "read-only" filesystem every day thereafter.

What brand and model is it? If it's a SeaGate FreeAgent you need to hook the disk up to a PC and disable the sleep timer using the vendor utility that comes on the disk (the utility is also available online if you deleted it). Otherwise the disk may go to sleep and wake up read-only.

I'm also thinking a lot about remote backups. Something off-site. I recently lost a lot of data to disk failures.

Yes, RAID is not a backup. RAID is of great help, but you should never store important data on just one device.

Anyway, I'd like to share this work with everyone, but it's not easy. My coworker who also has a ReadyNAS NV+ immediately asked if it was a .bin package he could install to use it. There isn't.

I'm thinking at minimum a tarball package for ssh-enabled machines to install. Unfortunately I don't think I can really provide a debian package because of the dependency mess.

It also requires a bit of documentation, though a few blogs out there have more or less covered the aspects of gpg keys and duplicity and s3 storage, a 200-line README would be greately appreciated by anybody new to duplicity.

Restore is stupid simple. You just reverse the "source" and "destination" and duplicity knows what to do.

Unfortunately, you need your GPG key to restore, and the documentation needs to make this clear, because the user should know where their GPG key is stored, and to know to keep it secure and recoverable in event of disaster.

Anyway, I'd be happy to make a .BIN package if users weren't such a *****. Though I could build a tarball and a README, anybody interested?

It does sound interesting. You could write an add-on and make a check early in the script to see if users have SSH enabled. If not cancel the install of the add-on. Add-ons can be better as they should install everything in the right places automatically and if a removal script is properly written remove the add-on cleanly as well.

[advancedstuart]

Hi,

I'm new to ReadyNAS Duo and I'd really like an off site encrypted backup of my data.

After looking around the forum this solution seems to be the best one I've found. Can anyone let me know how to proceed?

Jquast, you said you could build a tarball and readme file for anyone interested, is that an option?

Thanks

[dildano]

I'd also like to see this come to fruition. My issue is that my USB backups are essentially unprotected. I'd really like to encrypt them.

[dildano]

I got the latest version (0.6.09) working over the weekend. I am NOT a Linux expert AT ALL, so please don't ding me if this doesn't work quite properly for you. Here's what I did:

Code: Select all

apt-get install python
apt-get install gnupg
apt-get install gcc
mkdir sources
cd sources
wget http://code.launchpad.net/duplicity/0.6-series/0.6.09/+download/duplicity-0.6.09.tar.gz
tar xvzf duplicity-0.6.09.tar.gz
cd duplicity-0.6.09
python setup.py install
duplicity --version

That last line will tell you if everything worked properly if it returns 0.6.09.

I spent the entire day yesterday figuring out how to get it installed properly and how to use it. In my case, I backed up one of my shares to a connected USB hard drive. It worked like a charm. I disabled compression so that it would go faster. Here's an example:

Code: Select all

duplicity -v4 --encrypt-key=<key> --sign-key=<key> --gpg-options="-z 0" --volsize=256 --exclude="/c/data/Recycle Bin" /c/data file:///USB_Backup_1/data

Unfortunately, this doesn't really resolve my particular backup situation. Duplicity creates a full backup, and then incrementals. What I really want is something to simply create an encrypted mirror of my data on my USB hard drive. Anyone have any suggestions?

[db3l]

Unfortunately, this doesn't really resolve my particular backup situation. Duplicity creates a full backup, and then incrementals. What I really want is something to simply create an encrypted mirror of my data on my USB hard drive. Anyone have any suggestions?

I suppose another approach would be to just encrypt the backup volume in general, so that anything written to it by any tool would be protected.

The modern Linux approach involves the device mapper, but the ReadyNAS kernel doesn't appear to have the crypt support compiled in, so I suspect getting that working would not be for the faint of heart. You could try some of the older approaches involving a loopback device and user space encrypted filesystems, but a better option (though not one I've personally tried) might be TrueCrypt - someone else pointed out recently about running it locally (http://www.readynas.com/forum/viewtopic ... 11&t=41898) just a few firmware revisions ago, so I'm guessing it would still work.

That would let you create a TrueCrypt volume on your backup media and then just use that as the target with any tool.

-- David

[dildano]

You could try some of the older approaches involving a loopback device and user space encrypted filesystems, but a better option (though not one I've personally tried) might be TrueCrypt - someone else pointed out recently about running it locally (http://www.readynas.com/forum/viewtopic ... 11&t=41898) just a few firmware revisions ago, so I'm guessing it would still work.

That would let you create a TrueCrypt volume on your backup media and then just use that as the target with any tool.

My thoughts exactly! In fact, I just got TrueCrypt running based on the post you referenced, and when I get some time, I'll look into creating an encrypted volume on the USB drive. Then it's just a matter of using Rsync to "mirror" my data into the encrypted volume.

[dgJacks0n]

dlidano-

many thanks for the duplicity install instructions. Been wanting this for so long! I just tried them but am getting unment dependency errors from sox, libvorbisenc2 and libvorbisfile3. Tried running 'apt-get -f install' and 'apt-get update' but no help.

What radiator version are you running? I'm at 4.16 on a ReadyNas NV+ I'd appreciate any tips you or others can offer!

Code: Select all

# apt-get -f install python
Reading Package Lists... Done
Building Dependency Tree... Done
You might want to run `apt-get -f install' to correct these:
The following packages have unmet dependencies:
  python: Depends: python2.3 (>= 2.3.4-18) but it is not going to be installed
  squeezeboxserver: Depends: sox but it is not going to be installed
                    Depends: libvorbisenc2 but it is not going to be installed
                    Depends: libvorbisfile3 but it is not going to be installed
E: Unmet dependencies. Try 'apt-get -f install' with no packages (or specify a solution).

[dildano]

I got it up and running on my Pro, so I can't speak for the NV+. Also, I'm a complete Linux newb, so there's no telling what else I may have done on my Pro before trying to get Duplicity to work that may have impacted the install. Having said that, my guess is that the Pro has some stuff installed by default that the NV+ does not have. I'm sure that someone with some more Linux experience can help to resolve your issue.

I Googled "python: Depends: python2.3 (>= 2.3.4-18)", and found this:

"Enable all repositories in apt by editing /etc/apt/sources.list and uncommenting all lines starting with # deb. Then try run apt-get update again."

And this is what is in the sources.list on my Pro:

deb http://www.readynas.com/packages 4.2.13/
deb http://www.readynas.com/packages readynas/
deb http://archive.debian.org/debian sarge main contrib non-free
deb http://ftp3.nrc.ca/debian-archive sarge main
deb http://archive.debian.org/debian etch main

Dunno if that will help you.

[jasonswett]

I'm posting here in hopes that some of you who know something about Duplicity are subscribed to this topic. If you're successfully running Duplicity with your ReadyNAS, any chance you could help me out with my Duplicity command? viewtopic.php?f=31&t=46785