Building a ZFS machine
Back in the day, when I worked at Sun Microsystems, they had the most incredible lab – full of every type of machine, storage array and funky device that Sun manufactured.
In the UK these were maintained by an amazing set of guys headed up by Paul Humphreys and no job was too much for them to tackle, from setting up entire solutions of servers connected via switches to storage arrays (log ticket, let me know when it’s all hooked up) to just simple requests looking for a specific type of server.
When Sun UK were still based at Guillemont Park, the lab took up a huge section of the ground floor for one of the main buildings and it was quite a sight to behold and to hear (thank goodness for the mandatory ear protectors).
I remember my primary ZFS lab box used to be an X4500 because it had lots and lots of disks, thus I could be testing multiple different customer problems on this single piece of hardware, thus reducing the number of lab boxes I had booked.
It was a fantastic box (once those Marvell SATA controller bugs were fixed) and I was exceptionally sad to give up my booking when I left Snoracle.
The Mini (me) X4500
Now that I work from home it would be difficult to own an X4500 / X4540 or variant thereof, frankly the noise in my office would be a complete nightmare and the power consumption would be a little worrying. Sure it’d keep me warm during the winter but my Mac Pro does a pretty good job there.
Therefore I went about looking for a machine that could be used as a SOHO ZFS lab box. The requirements were:
- It had to be quiet
- It had to have a small footprint
- The power consumption had to be reasonable (no getting an electrician in for a new 3 phase power supply)
- It had to have 4 drive bays (one boot drive, three free for playing with various zpool configs)
- It had to be reasonably inexpensive
- The driver support had to work for Solaris / Nexenta / Illumos variants
After some research I settled upon an HP Proliant Microserver N40L as it ticked all these boxes and then some!
The other thing going for the HP N40L was a £100 cashback offer, which certainly helped fund the memory upgrade to 8GB because we all know ZFS likes memory, lots of memory.
HP seem to do this on a regular basis, so right now I see Ebuyer are offering the replacement machine, the ProLiant N54L for £209.99 with £100 cash back, which seems like quite a bargain.
I discovered that not only could I populate this machine with 4 x SATA drives in the front bay but various clever people had purchased a 5.25″ drive bay to sit in the empty slot that would have been taken up by the DVD drive (if you’d configured that as an extra option) and then hooked that up to a SATA or SAS PCI Express HBA.
At the time I remember there were two options – a 6 slot (MB996SP-6SB) or a 4 slot Icy Dock drive cage (MB994SP-4S). I was initially tempted to buy the 6 slot device but the more I thought about this, the more I got worried that there would be insufficient power to drive 6 disks, either HDD or SSD or a combination of the both.
The dock takes two molex connectors to provide power to the disks and there’s just a single molex power connector inside the HP designed to power the DVD drive, which would mean buying a splitter cable.
Then there’s the matter of squeezing 6 disks into such a small space – any disks purchased would have to be very low profile, otherwise they just wouldn’t fit!
Once I’d given it some thought I figured the 4 slot Icy Dock sounded like the best option, mainly due to the worry over power draw. After all I wouldn’t be the first person to experience pool corruption due to power problems, as my friend Andy Harrison found out:
http://www.stormsail.com/zfs-fun-and-games/
Having selected my 4 drive bay dock I then had the job of figuring out which drives to buy and how to connect this up so they’d be seen to the system.
The Icy Dock is both SATA and SAS compatible which gave me an interesting choice – buy a SAS HBA at a higher cost or a SATA HBA which would be much cheaper.
Of course hardware is nothing without the drivers and this was an interesting sticking point. SATA controllers come and go, chipsets come and go which means trying to buy an established card with good driver support (hoping the card is still readily available) or buy a newer card and hope the driver works.
On the other hand because Solaris and its derivatives have an Enterprise background, the SAS support is much better and when you think SAS, invariably you think of LSI controllers.
There is a problem with this approach though – LSI cards tend to be expensive and only available from more specialist dealers.
After a lot of research I found the Intel sasuc8i card, which is a rebadged LSI HBA with the 1068 chipset and been supported in Solaris for a number of years. Also going in its favour was the fact that I could pick one up online for about £130, considerably cheaper than the currently available LSI badged cards.
You can also install the current LSI firmware on the card (with a little bit of a kick) thus ensuring you don’t have to wait for Intel to push out updates.
Having sorted out the drive dock and a SAS HBA this just left:
- A Startech 50cm SAS cable to 4x latching SATA ()
- A Molex 4 pin, Y shaped power splitter cable
- 2 x 500GB Samsung Spinpoint M8 disks
- A USB Samsung SE-208 DVD drive (for O/S installation)
After waiting for all the parts to arrive it was time to fit the Icy Dock, disks, HBA and cables. The thing about the Microserver is that it’s a really compact machine, so fitting the HBA is fiddly – it helps to have small hands. Thankfully my wife is always up for a challenge and got stuck in, disconnecting the various plugs on the motherboard, sliding out the motherboard and fitting the card.
The 4 way SAS/SATA cable proved to be tricky. After attaching this to the Intel sasuc8i card and hooking it up to the Icy Dock, I found that the Samsung disk wasn’t recognised.
The machine was powered off and my wife double checked the cables, only to find that you have to give the SAS/SATA cable a really, really good push to insert into the HBA and get a solid connection.
One boot up later and the disk was still not recognised by the HBA during the probe/discovery phase. At this point I figured I would try the other disk I’d purchased and after screwing this into the caddy and powering up again, success we had a disk that could be seen.
This meant either a DOA disk or a bad connection. Moving the disk to the other slots proved they were fine and putting the suspect disk into a USB caddy proved it couldn’t be seen on my Mac, so it was back to the vendor a replacement.
Once the hardware was seen it was time to burn a NexentaStor DVD, hook up the Samsung DVD drive and then boot and install the software on the internal 250GB SATA drive and create a mirrored data pool on my two 500GB Samsung Spinpoint drives.
I’ll leave the details on this for another post.
jasonbanham 