Recently I had the realization that all of my pictures from the past 10 years were sitting on a single hard drive just waiting to be swept off into deleted land with the next hard drive failure.

My first thoughts were, put them in Dropbox. Ohh wait, I have 80GB of photos and don’t want to pay for an upgraded Dropbox plan(~$10/mo for 100GB). How about Google Drive? Much cheaper for the same amount of storage(~$5/mo for 100GB). Well, maybe I’ll think about how I can do a fun project at home since I actually need more like 2-3TB of storage to hold all of my “backed up” movies.

UPDATE: Here is the post with all of the more detailed setup/specs

I already have a Drobo 2nd Generation, but it is getting older and I’ve noticed that the Drobo products have a bit of a performance problem. I can’t ever seem to get more than 2-3MB/s transfer speed from it through the Drobo Share and the Drobo unit itself seems to get really bogged down when any amount of I/O happens on it.

Requirements

• Redundant storage in case of failures
• Exports NFS & CIFS/Samba
• 2-3 TB of usable space
• Fast(I didn’t really know what Fast was at this point other than >3MB/s)

Options

• Pay for Dropbox
• Pay for Google Drive
• Build a file server

So really, the only option left is building my own file server. This really isn’t a hard task, but I wanted to take it a bit further and try out GlusterFS. Lucky for me I just happened to have some spare computers and five 1.5TB drives as well as a few smaller drives laying around.

Available Hardware

• 1 Dell Vostro 200(4GB RAM)
• 1 home built PC(Intel Core2Duo w/ 2GB RAM)
• 1 Dell “Pizza Box” Server(AMD 4 core w/8GB RAM)
• *4 Seagate 1.5TB Barracuda

• 1 WD Green 1.5TB

• One of the Seagates already had all of my data on it so I couldn’t use it in the setup.

The Plan

Gluster Setup

Run Gluster in a Replica setup over 2 nodes

Gluster Nodes

The Vostro and home built computer would both have 3 drives in them. One drive for the operating system and 2 drives stripped for performance. This would give each node about 2.7TB of usable space. The OS that I chose to use for this was CentOS 6.4 as it seemed to be a fitting choice for this project since Red Hat is the main supporter of Gluster and CentOS is basically a clone of RHEL.

I wanted to try out ZFS so I could play with deduplication and compression. ZFS is ported to Linux and there was a Gluster HowTo so why not try it out.

NFS/CIFS Server

Simply run CentOS on the Dell server and mount the GlusterFS and export it through NFS and Samba4.

Results

ZFS Testing

I got the Vostro setup following the Gluster on ZFS guide that I lined above. I setup the ZFS pool as they stated.

Then I used iozone on the volume to test its performance. It became apparent quite quickly that there was something wrong. The speeds with smaller files was what you would expect(120MB/s+) but as soon as it got to the 256M, 512M & 1GB files the sustained writes went right into the crapper. I was getting sub 1MB/s speeds. Now I don’t know enough about ZFS, nor really disk I/O, but I suspected it had to do with either the ZFS install or the disk controller on the MB. Regardless, I didn’t really want to spend the time to debug the issue. The easy solution was to simply switch to LVM+XFS. A few minutes later I had a LVM volume with XFS on it. At this point I was doing simple dd commands to do the tests with a 1M block size and 512MB test file. XFS was doing anywhere from 180MB/s – 250MB/s. It was inconsistent, but good enough for me.

GlusterFS Testing

Now I was ready to install the next node and setup the GlusterFS. Fast Forward a bit and the 2nd node was up and running. I had both nodes connected to a cheap DLink 5 port Gig switch at this point. I was pretty impressed at how easy it was to setup the replica set given it was more or less just a single command. I did the same quick dd test that I had done on both nodes on the XFS partitions and got something like 70-80MB/s. Seems good enough for a home storage solution. Just to check and see I removed the replica set and setup a strip set over the two nodes and did the same test. Pretty amazing, 105-110MB/s write speeds. I was pretty excited at this point, but had to rein it back in a bit and switched back to the Replica setup.

NFS/CIFS Testing

This really wasn’t anything to special. Just your standard CentOS install with the two internal disks software mirrored. I then installed the nfs server packages and Samba4. The biggest pain in the butt was getting Samba4 setup correctly so that I didn’t have to authenticate. This is a home storage solution so I don’t really care about security that much. I want usability! I don’t remember actually doing the dd tests at this time because I was tired of not being able to watch movies at night(This project was spread over about 4 nights) so just wanted to get my data onto the Cluster and get on with life.

Final Setup Results

All I can say was my wife couldn’t understand how excited I was transferring files from my computer to the storage cluster. She just doesn’t understand that 50-80MB/s transfer speeds is AMAZING! Maybe not anything special if you are transferring over USB3 or something, but over a network! Yes it is pretty great considering the whole setup was basically crappy commodity hardware piled together. I still need to get a small report script going so I get emailed disk status and such every week so I know if a disk dies or something along those lines but for now I’m pretty dang happy with the result.

When I get a chance I’ll post up all of the commands and more specific hardware specs for everything in another post and link it in here. Probably put in some pictures of my Garage Gluster Cluster.