Single Instance Storage (Block level deduplication)

[omlette brothers]

First post, and a new user Chuffed to bits with my X6 so far...

I've migrated all my data from my W2K3 Server. I have had the benefit of Single Instance Storage as the server also hosted RIS to build my PC's. I have a lot of development I386's which contain a lot of repeated files. Would be great to have the same, transparent support to remove data redundancy.

[bhoar]

Hi - um, care to give more details for those of us who aren't really sure what you're talking about so we can follow along when Infrant responds?

-brendan

[Renegade59]

Single Instance Storage (SIS) allows the server to maintain one copy of a file for multiple "people."

The easiest example is in MS Exchange. If you send an attachment to 30 people, the server used to need to keep 30 different copies of the same file. With SIS, it can keep just the one and use if for everyone.

[yoh-dah]

Single Instance Storage (SIS) allows the server to maintain one copy of a file for multiple "people."

The easiest example is in MS Exchange. If you send an attachment to 30 people, the server used to need to keep 30 different copies of the same file. With SIS, it can keep just the one and use if for everyone.

How would this be applicable in a NAS environment? I would think this is more application dependent as the app would need to create appropriate "symlinks" to the data.

[omlette brothers]

In my environment I have a lot of repeated files from developing unattended Windows 2K3 and XP installs. If you've ever looked in the I386 directory of a Windows install CD you'll see thousands of files.

If you take it a stage further and compare the I386 directory of Windows 2003 Server, Windows 2003 Advanced Server, Windows 2003 Web Server and then multiply that by the different licence types like Volume, Retail and OEM you'll find very few files differ and that thousands are identical.

Single Instance Storage (is a server service), when used with Microsoft's RIS is a way of replacing the physical discrete (ie files) data with (I guess) a Unix like symbolic entry. This cuts down on the amount of *real* disk being used.

My other major uses are DJ'ing and in the future VJ'ing. I see both uses requiring repeated data when editing music / video.

[Xipper]

I'm not sure how you would even go about creating such a service on a storage device. The problem is how do you determine when files are non-unique? md5 isn't accurate enough, file name isn't accurate enough, size is not applicable. It starts to become something that is far more resource intensive, just try running noclone, dedupe or any other software on a PC that is supposed to find duplicate files for you. Its a lot to expect of a system that is optimized to read and write files, not file contents.

I don't believe I have ever come across an enterprise storage system that supports single instance, I'm not aware of many applications that even support this outside of Exchange.

[omlette brothers]

A revival of an old topic, but it now seems to be a hot topic in the world of Enterprise NAS - It goes by the name of block level deduplication. Is the Readynas powerful enough to run a service like this? It would be absolute boon to the device, where only the deltas in file changes are required.

[BobRoss]

I agree, especially for VM shops!!! We have lots of dup'd data in our Win2k3 server installs.

[TeknoJnky]

sis support would be awesome.

for some more info, wiki has a decent explaination.

http://en.wikipedia.org/wiki/Single_instance_store

[omlette brothers]

One day, sometime in the future the Readynas will have dedupe....

Now ZFS has block level dedupe.....http://blogs.sun.com/bonwick/en_US/entry/zfs_dedup

[TeknoJnky]

has anyone thought of or tried running open solaris and zfs with dedupe, on a pro/3200 via virtual box?

I realize it would be slower than using native, but for data that is dedupe friendly it might be worth it.

[omlette brothers]

Block level Dedupe looks to be on the BTFRS roadmap?

[lurium]

Or netgear can make support for LESSFS which is inline dedup in linux.

http://www.lessfs.com/wordpress

[jeremyotten]

+1

[andrewz]

has anyone thought of or tried running open solaris and zfs with dedupe, on a pro/3200 via virtual box?
I realize it would be slower than using native, but for data that is dedupe friendly it might be worth it.

Would be slower.