OLD: Making Time Machine work with the ReadyNAS

[toad]

Anyone know what the "hard links" vs. "soft links" deal is re ReadyNAS NV+? See this short note:
http://www.macintouch.com/readerreports ... d26jan2008

(scroll to entry from 26 Jan 2008 if it doesn't jump right there)

I didn't find the ars technica article mentioned, but still...

[btaroli]

OK. A few things to keep in mind... links (hard or soft) are a filesystem feature. TM volumes are sparseimages. So let's say that, like me, your TM configuration has itself pointed at an AFP share (though this works with SMB too) like "tardis" on "cookie-monster". What TM actually does on "tardis" is create a disk image (aka sparseimage) named for your computer and it's ethernet MAC address. Also note that if you point multiple Macs at this same share, TM creates a separate sparsebundle for each.

Code: Select all

cookie-monster:/tardis# ls -l
total 64
-rw-r--r--  1 me users    16 2007-12-21 22:17 :2e001b63a31a0b
drwx--S---  3 me users 16384 2007-12-21 22:20 oscar_001b63a31a0b.sparsebundle

This sparsebundle disk image is it's very own, separate filesystem. So Apple doesn't need to have anyone modify anything in whatever filesystem you might possibly host a TM backup upon. They drop a disk image into whatever share TM is pointing at and can put whatever filesystem they like into that image. The only magic Apple is doing to make people fear putting TM backups on existing NAS solutions is marketing and FUD.

The two biggest items regarding TM on NAS are how to keep TM from using up all your space and a technical issue that seems to prevent sparseimages from properly recognizing space freed by removing files as available for re-use -- which is what TM does expire old files when the backup volume fills to capacity.

There is a nice hint on the sparseimage size limit on Mac OS X Hints. You can read more about the utility for managing sparseimages in the manpage on Apple's Developer Connection.

AFAIK, the re-use of free space issue isn't exactly resolved. I haven't yet played with my sparsebundle to test to see what happens. I've just been keeping my ears and eyes open for updates on the issue once people get Time Capsule into their hands, since Apple will certainly have to have resolved this matter prior to pushing it's own NAS solution out. My NAS has plenty of extra space for me to ride this out until Apple makes it's fix.

[toad]

While a sparseimage is a disk image (I am fairly familiar with them), I'm not sure whether a sparsebundle image, which is what TM is making (and which is new to me), is or not. "Bundles" are pretty common in the Mac OS X environment, and they seem to be just special directories that "act like" a single file. Sorry if I am muddying the waters (hope I'm not!), as I do want to understand these distinctions, but have nothing more to go on than what I've seen on my own computer.

Found a hint on one difference between sparse vs. sparsebundle in the manpage you mentioned:

"Specifying SPARSE creates a UDSP: a read/write single-file image which expands as is [sic] is filled with data. SPARSEBUNDLE creates a UDSB: a read/write backed by a directory bundle."

I'm past my depth here ... But thanks for all the info!

edit: fixed a mish-mash of a partial re-write I did before posting...

[toad]

OK, their terminology here does make it sound like a sparsebundle is indeed a disk image, though I don't get the "bundle-backed" reference in the prior post. Anyway, here's the bit supporting it being a disk image:
compact image
scans the bands of a sparse (SPARSE or SPARSEBUNDLE) disk
image containing an HFS filesystem, removing those parts of
the image which are no longer being used by the filesystem.
Depending on the location of files in the hosted filesystem,
compact may or may not shrink the image. For SPARSEBUNDLE
images, completely unused band files are simply removed.

Anyway, thanks!

EDIT: um ... guess I didn't read the top of the manpage you cited, wherein the overall general command is defined as manipulating disk images... doh! OK, I guess that answers that. So I'll sit back and hope for further word re the outstanding issues you've mentioned...
THANKS!

[toad]

The plot thickens...

http://www.infrant.com/forum/viewtopic.php?t=14456

A disk image that's not really a disk image?

I wonder if this is the root of the apparent problems that have been reported with Time Machine as the drive fills up and TM starts to delete old backups... If Time Capsule is using HFS+ then that might make all the difference in the world... Unless of course people have been reporting the same behavior with networked AFS/HFS+ scenarios (have they?). Or maybe I'm still missing something.

[btaroli]

Red herring, IMO. I appreciate that they'd like afpd to fake ext3 supporting that attribute... but the attribute has no bearing on how the sparse disk image functions or affects TM's capabilities. The attribute just makes the folder look like it's not a folder -- think back to custom icons, but with a twist in the way the Finder behaves when you click it. "Applications" in OS X are bundles by the way... if you visit one in Terminal you'll see they're just folders.

The issue with TM consuming all space as to do with how it creates the sparseimage. There are plenty of postings showing how to change the max size of the image with hdiutil. The behavior with how it seems to delete *all* backups when it fills the image is a known bug.

[toad]

But the point here is (I think) that the sparsebundle images seem to not be treated like a separate filesystem (per your note above), so the constraints of the filesystem on the NAS would bear on the behavior issues, wouldn't they? The "images" are not mounted and treated as we see them on an HFS+ volume, and presumably the metadata issues have a bearing on each file in the contents as well.

If one were to create a sparsebundle image AND a sparse image on an HFS+ volume, then create another sparsebundle image and copy it inside the initial sparesebundle and sparse images... On the HFS+ volume these would behave exactly as one would expect (I think) in that the sparsebundle image within the other images could be "mounted" as a volume once the containing images were mounted. Unmount them all then copy them to the NAS (non-HFS+) volume. If the sparsebundle image is treated as a disk image then you should be able to mount (as volumes!) the sparsebundle images that are stored within the initial two images. But I'm betting that you can only do it with the sparse image.

Wouldn't that indicate that the filesystem of the NAS (and any limitations that may or may not come with that) is what an AFS user is dealing with when using sparsebundle images, but that sparse images are treated as their own filesystems?

I do realize there are hdiutil commands that can be brought to bear, and perhaps that's all TM needs to do to make this all work "right" in the end... but it seems like (at this stage) the attribute in question does in fact have a bearing on how the sparsebundle image functions, and therefore can affect TM's capabilities.

Unfortunately I do not have Dev tools installed, so I cannot use GetFileInfo to check the attribute in the sparsebundle image w/in both sparse image & sparsebundle image cases. Anyone... ?

Anyone know why Apple didn't use sparse images instead of sparsebundle images? Seems like that might have helped here. But then maybe the backing 'bundle' is important to the TM functionality, I don't know...

[btaroli]

If it were relevant, then TM wouldn't work on the ReadyNAS at all since it doesn't support the bundle. I must admin some curiousity as to whether the Time Capsule will use HFS+, but I suspect it won't be much different than AirDisk with the Airport Extreme (which supported sharing of USB attached disk). I guess we'll see soon.

[toad]

I don't know, it seems like it could be relevant without being a critical problem, like it could have a negative side-effect without preventing TM from working at all. Like what we're seeing, by all accounts.

Airport Extreme USB drive sharing doesn't support ext3 either, fwiw:
http://docs.info.apple.com/article.html?artnum=305038

The AirPort Extreme (802.11n) supports USB storage devices that have a block size of 512 bytes, and are formatted as Mac OS Extended (HFS-plus), FAT16, or FAT32. Not all USB storage devices use a block size of 512 bytes.

The AirPort Extreme (802.11n) shares storage devices based on the format used to initialize the storage device. For example, if HFS-plus formatting was used, AFP and SMB/CIFS protocols are used to share the device on the network. If FAT16 or FAT32 was used, SMB/CIFS protocols are used.

My guess is with TC, since it's in their device, it will use something other than ext3 (HFS+ or FAT32?), such that they get the behavior they want/need and leave it at that. Either that or they have to integrate in one of the work-around approaches, like using the hdiutil to shrink image if it's not doing it by default. Or who knows...

Oh well, as you say, we'll see soon.
Cheers!

[btaroli]

Well, they didn't go into specifics -- big surprise -- but one of the notes on the just-released 10.5.2 update for Leopard says that they enhanced device support for TM. Interestingly, all the mounted shares from my NAS suddenly have a new icon in Finder... a trio of stick figures standing together. Interestingly, this is /also/ the icon that appears in... Time Machine. Go figure. Not sure how significant this might be yet, but I think we're seeing adjustments for TC.

I must say I'm really glad they decided to add the menubar item for TM. I won't have to go digging around for it now.

[btaroli]

OK. Although I hadn't been following the story all along, the speculation seems to have been quite correct. The focus on TM fixes in 10.5.2 seems to have been largely focused on the upcoming Time Capsule -- oh, if they had only been able to call it TARDIS... A few links for your pleasure:

• http://babygotmac.com/a/leopard-1052-airport-express-and-time-machine/
• http://www.appleinsider.com/articles/08/02/05/time_machine_remains_focal_point_of_mac_os_x_10_5_2_testing.html
• http://www.computerworld.com/action/article.do?command=viewArticleBasic&articleId=9058862&intsrc=hm_list
• http://www.macsimumnews.com/index.php/archive/leopard_1052_update_to_add_surprise_improvements

I especially found the first one amusing, given our recent chat about disk images. Why? Because the 10.5.2 tester found that, indeed, when they used a USB attached device on Airport Extreme it was created as.... you guessed it.... drumroll please.... a sparsedisk image!

So for all the tribulations about Apple doing something special for their own NAS for TM it seems to be not so. I'm sure we'll learn more as time goes on, but I'm feeling much more comfortable that my TM backups since December are safe(r). YMMV

[francois_75015]

Does anyone really use TM on a ReadyNAS partition ? I cannot go further than the "archive cannot be created"... what's up ?

[btaroli]

Well, it seems 10.5.2 has thrown us another curve ball. When I tested, I found that my existing sparsebundle was working fine. But since I wanted to resize it, I decided to see what would happen if I created a new one (so I didn't jeopardize my old backup data) and resize it to be sure it doesn't eat my entire NAS. But what I found is that TM now fails to create the sparsebundle with an "Operation not supported (45)" error. With a little searching, I found that all sorts of people are being hit by this, even those using AFP shares on Macs that worked previously on 10.5.1.

Digging a little deeper (running hdiutil manually), I found that the operation it's talking about is called FlushBands... so it seems that whatever it's doing during filesystem creation to sync the buffers to media is failing.

The workaround is fairly straightforward though. You will want to create a sparsebundle with the appropriate name on a *local* device (internal HD or USB-mounted) and move it to the NAS under whatever share you're using for Time Machine.

One interesting note in this is that the "defaults" setting I'd made under 10.5.1 to use TM over AFP was gone! In fact, it doesn't appear we need this any more... (fix one problem, Apple, and then create another?)

So, what I did to make this work is the following:

1. Create a sparsebundle image using the following command:

   Code: Select all

   sudo hdiutil create -size 320g -type SPARSEBUNDLE -nospotlight -volname "Backup of <computer_name>" -fs "Case-sensitive Journaled HFS+" -verbose ~/Desktop/<computer_name>_<en0_mac_addr>.sparsebundle

   You could also do this using TM or Disk Utility against a USB drive if you prefer GUIs, but I didn't have one handy and wanted to proceed anyway.
2. Mount TM backup share (I call mine "tardis") over AFP (though others should work too).
3. Drag the sparsebundle this created to my AFP share.
4. Select same share from Time Machine preference panel.
5. Eject AFP share -- not necessary, but cleaner for me since I don't usually keep this share mounted manually.
6. Trigger backup to "run now" in TM menubar widget.

The sparsebundle that results from step 1 is only about 160MB, so I had no trouble doing this on my MBP's internal HD. It was fairly quick too, taking under a minute to complete. Also interesting -- since discussion of this was of interest in this thread -- was that the bundle appeared correctly in the finder (attributes that made it show up as a bundle rather than a folder) when on my HD, but it definitely lost this attribute when copied to the NAS. Not that I care, really, because I know what it is and don't use the TM share for anything other than TM backups but I thought it worth noting anyway.

[davelee]

I tried creating a sparsebundle on the local machine to move to the NAS and I ended up getting a "operation not supported" error when the OS tried to set some permissions or resource fork settings on the NAS.

[btaroli]

I tried creating a sparsebundle on the local machine to move to the NAS and I ended up getting a "operation not supported" error when the OS tried to set some permissions or resource fork settings on the NAS.

Are you certain that the sparsebundle was named correctly when it was placed on the NAS? If not, then TM will simply try to create a new one and you could run into problems (esp on 10.5.2). I find that it must be constructed as [computer name]_[en0 mac addr].sparsebundle where [computer name] is that which is set in the Sharing prreferences panel and [en0 mac addr] is the non-delimited version of what shows up for en0 in ifconfig... or use

Code: Select all

ifconfig en0 | grep ether | awk '{print $2}' | sed 's/://g'

Or, if you really want to be sure, let TM start to create a new sparsebundle on it's own and have a peek at the share you pointed TM at... you will see the newly created sparsebundle there until TM gets an error and drops it.