OpenSolaris Dedupe for home use
Posted by Sam
I know that dedupe will save a ton of space at work. We have lots of Subversion repos checked out on servers and lots of similar files like swfaddress and jquery that are reused in a lot of our projects. But I've been curious how useful it would be with my personal files. Here's what I've found when I deduped files from my home directory.
- Photos: 5.5GB savings (34GB pre-dedupe/28.5 post-dedupe)
- Music: No savings
- Projects: 1GB savings (2.2GB pre-dedupe/1.22 post-dedupe)
As you can see there are some decent space savings to be had even for home/power users. The savings would be even greater if I had enabled compression as well but I was more curious about dedupe since compression is fairly common these days.
Tags: zfs opensolaris deduplication
Dedupe in the real world
Posted by Sam
For those that haven't heard dedupe has made it's way into OpenSolaris 128a. You can download it at Genunix. To test what kind of savings we might expect to see in the real world I fired a VirtualBox with OpenSolaris 128a and turned on dedupe and compression (gzip level 6). Then I rsync'd 10 gigs worth of files from our staging server. The results? Dedupe dropped that 10 gigs to 6 gigs and compression dropped another 2 gigs which left us at 4 gigs. So dedupe gave us a 40% reduction and compression gave us another 20%. Not too shabby, especially when it takes about 10 seconds to enable it. A little bird told me that they are expecting dedupe in the Sun 7xxx series of storage by the end of the year. The Sun boxes are already the winner for best value and with that they jump even further out front!
Tags: opensolaris solaris deduplication zfs
NexentaStor ZFS Based Storage Appliance
Posted by Sam
Anybody that knows me knows that I think ZFS is the best thing since sliced bread. As far as file systems go it's pretty much the best thing to come along in years, maybe decades. I've tried to evangelize it to my Windows brothers but asking them to switch to Solaris for a file system seems to be too much to ask for and it seems unlikely that Microsoft would ever be as progressive as Apple and include it in the operating system. Instead they will no doubt try to clone it, spending millions of dollars in the progress and still ending up trailing ZFS. Ok, enough ranting about the boring giant that is Microsoft. They haven't done anything interesting in years.
NexentaStor takes OpenSolaris and ZFS and packages it up into a neat little NAS appliance complete with a web based GUI for controlling NFS, CIFS (SAMBA), iSCSI, FTP and ZFS snapshots. You provide the hardware and the drives and Nexenta provides the software for running the NAS.
Nexenta provides a free VMware image that you can use to install and test NexentaStor. It appears to be a full featured copy. If you want to run NexentaStor on a physical server you have to purchase it from Nexenta. They don't provide any pricing on their site currently, but I'm sure it will be very cost effective. I've seen pretty poor network performance with Solaris on VMware ESX servers so I don't recommend running NexentaStor on VMware if you require the best performance, but it's great for trying out the product before you buy.
Installing and running NexentaStor was extremely easy under VMware Fusion on my MacBook. I had it running within 30 minutes with no real problems. I've got CIFS and FTP up and running but I haven't been able to get NFS running so far. Also, I'd like to play with joining it to an AD domain. This feature would be great for Windows admins looking for the rock solid reliability of Solaris and ZFS without requiring Windows admins to learn Solaris and everything that goes along with that. For a product that isn't even a 1.0 product it looks pretty amazing so far. I look forward to playing with NexentaStor some more in the future.
Tags: zfs