NexentaStor ZFS Based Storage Appliance
Posted by Sam
Anybody that knows me knows that I think ZFS is the best thing since sliced bread. As far as file systems go it's pretty much the best thing to come along in years, maybe decades. I've tried to evangelize it to my Windows brothers but asking them to switch to Solaris for a file system seems to be too much to ask for and it seems unlikely that Microsoft would ever be as progressive as Apple and include it in the operating system. Instead they will no doubt try to clone it, spending millions of dollars in the progress and still ending up trailing ZFS. Ok, enough ranting about the boring giant that is Microsoft. They haven't done anything interesting in years.
NexentaStor takes OpenSolaris and ZFS and packages it up into a neat little NAS appliance complete with a web based GUI for controlling NFS, CIFS (SAMBA), iSCSI, FTP and ZFS snapshots. You provide the hardware and the drives and Nexenta provides the software for running the NAS.
Nexenta provides a free VMware image that you can use to install and test NexentaStor. It appears to be a full featured copy. If you want to run NexentaStor on a physical server you have to purchase it from Nexenta. They don't provide any pricing on their site currently, but I'm sure it will be very cost effective. I've seen pretty poor network performance with Solaris on VMware ESX servers so I don't recommend running NexentaStor on VMware if you require the best performance, but it's great for trying out the product before you buy.
Installing and running NexentaStor was extremely easy under VMware Fusion on my MacBook. I had it running within 30 minutes with no real problems. I've got CIFS and FTP up and running but I haven't been able to get NFS running so far. Also, I'd like to play with joining it to an AD domain. This feature would be great for Windows admins looking for the rock solid reliability of Solaris and ZFS without requiring Windows admins to learn Solaris and everything that goes along with that. For a product that isn't even a 1.0 product it looks pretty amazing so far. I look forward to playing with NexentaStor some more in the future.
Tags: zfs
Ruby DTrace Probes in Leopard
Posted by Sam
There are a ton of new features in the upcoming release of Mac OS X code named Leopard, but there are a couple of features that I find pretty exciting and interesting but haven't seen mentioned elsewhere. The first is that Apple added DTrace probes to their version of Ruby. See the official page here. These probes were available on Solaris for a while thanks to Joyent, but I didn't realize Apple was including them in Leopard.
The second interesting feature (interesting to developers at least) is an app called Instruments. You can use DTrace probes to see exactly what is happening inside your Rails app while also seeing exactly what affect it's having on your system. It's got a great gui that I think will extremely useful for tracking down performance problems.
Mac OS X was already the platform of choice for Ruby on Rails developers but now it's even better. Apple is doing a great job of picking the best technology available and tying it into their OS. Now the only thing I'm waiting for is full ZFS support! For me it's Solaris on the servers and OS X on the clients. This combination will take over the world. Mark my words.
Tags: macosx dtrace rubyonrails
Easy Custom Environment in LiteSpeed for Rails
Posted by Sam
Update (8/12/2008)
Thanks to LiteSpeed adding a staging environment this work around is no longer necessary for me, but I'm leaving it here in case others need custom environments besides staging.
Ruby on Rails has three standard environments. Development, test and production. Test mode is also called staging by a lot of people, including myself. Unfortunately, my favorite web server and all around best Ruby on Rails application server, LiteSpeed, doesn't have a test mode. For whatever reason they implemented the development and production modes for Rails but didn't include a test mode. This is annoying because any real application development should be going through some formal or informal QA and this should be happening on a dedicated staging environment.
There are a couple of ways around LiteSpeed's lack of a test mode for Ruby on Rails. One of them is to simply set the RAILS_ENV variable in the environment.rb file like so.
# Uncomment below to force Rails into production mode when
I don't like this because if you push this exact same code to production it will override the default value, which should be production. That could cause all kinds of problems. Since I do all my pushes (both personal and professional) straight from subversion I would have to remember to make this change every time I pushed a site update to staging. It just feels wrong.
# you don't control web/app server and can't set it the proper way
ENV['RAILS_ENV'] = 'test'
Another options is to forgo the easy Ruby on Rails setup in LiteSpeed and do a custom setup as described here. But then you will have the problem this guy had. Obviously it's doable but still seems dirty.
I prefer to stick with the standard and obvious way of doing things whenever possible. When you have to quickly migrate a server at 3 in the morning I have learned that keeping things straight forward is the best approach. Sometimes I will go to great lengths to keep something simple when a quick and dirty approach would have sufficed for a while. Keeping things standard and clean is the best approach. So, that being said, this is the approach that I think works best for setting up a test (or staging as we call it) environment in LiteSpeed. Change the environment.rb file in your Rails so that it reads:
# Uncomment below to force Rails into production mode when
Basically you are saying if you don't know what your environment should be, set it to test. Now you need to make sure that the RAILS_ENV is not set by LiteSpeed so go to Configuration > Virtual Host Templates > EasyRailsWithSuEXEC > Context > Edit and select the blank option in the Rails Environment section. Restart LiteSpeed and your custom environment should be loaded.
# you don't control web/app server and can't set it the proper way
ENV['RAILS_ENV'] = 'test' unless ENV['RAILS_ENV']
Tags: rubyonrails litespeed
Installing Solaris 10 update 4 on Dell PowerEdge 2950
Posted by Sam
I flew into Oklahoma City last night to install Solaris 10 on a new Dell PowerEdge 2950. I've been putting Solaris 10 every where I can because it's features like DTrace, ZFS and Zones blow away anything found on Linux. But installing Solaris on non Sun hardware can be a challenge. I'd checked the HCL and it checked out ok as long as you added the appropriate drivers but I knew there were still probably going to be problems. The Perc controller that Dell uses isn't recognized by the Solaris installer so I first tried to install the LSI drivers, but Solaris still couldn't see the drives. So after a little more searching I found drivers from The University of Queensland and they worked without a hitch. After the install I added the Broadcom NetXtreme II drivers to get the networking support going. The Broadcom drivers installed as expected. So far everything is running great. I guess we'll know for sure once this thing goes into production, but as of right now this looks like a successful Solaris installation.
Barracuda Load Balancer 340 Review
Posted by Sam
Last week I tried out the Barracuda 340 Load Balancer. In more informal conversations (is that possible?) I've been referring to the Barracuda as the Barracrappy because that's the impression I was left with. A few weeks ago our Cisco LocalDirector died a sad and lonely death in our NetStandard data center. The LocalDirector had been working flawlessly for better than five years without a single problem and then one day it dies with very little warning. Nothing lasts forever so this wasn't shocking, but it was unexpected. Once the painful recovery process was over I immediately began looking at alternatives. Of course I'd love to go with a BIG-IP, but the cost really isn't reasonable and frankly most of the features are unnecessary for our needs. And so began the search for an inexpensive solution that would be a drop in replacement for the LocalDirector.
After a lot of searching I had basically narrowed down the search to the Barracuda 340 Load Balancer and the Kemp LoadMaster 1500. Both said they would do Layer 4 switching, which I needed so that we could simply drop in the load balancer and have it behave the same as the LocalDirector did. Having to re-ip all the boxes would be a pain and highly disruptive and I really didn't see the benefit. After looking at screenshots and demos I settled on the Barracuda. The price was excellent and as an added bonus it had Intrusion Prevention System which certainly isn't a necessity, but more of a bonus. The other deciding factor was the Barracuda name. I'd never used their products before but I'd heard of them and I like their philosophy on always reaching a human on the phone. So I set out to order the Barracuda 340.
I had a couple problems getting the demo unit from Barracuda. First, I talked to a guy at Barracuda who took my info and was supposed to pass it along to a reseller. I let him know that we were replacing a busted load balancer and I was in a hurry. A full day passed and I didn't hear anything, so I called Barracuda again and talked to somebody else who gave me the information for a reseller nearby and this got the ball rolling. I cut a PO to the reseller who promised to have the unit overnighted. The next day I found out that Barracuda had shipped the load balancer through normal shipping instead of the overnight shipping I'd requested and paid for. So we waited another day for Barracuda to ship another one, overnight this time. Unfortunately, this was only the beginning of the problems. And if this had been the most problematic part of the ordeal it wouldn't have been a big deal.
The demo unit finally arrived, but it was shipped to the wrong person in the company. After I tracked down the unit we immediately went to work to get it configured. My expectation was that we'd simply plug it in, turn it on and after 15-20 minutes of configuration we'd have a load balancer passing traffic. Boy was I wrong! Having configured the LocalDirector through the command line for years I expected the web interface that came with the Barracuda to be a cake walk and it mostly was, except for the fact that it didn't work. We tried everything and I mean everything to get it to work. Finally we called tech support. This really shouldn't have been necessary because like I said it should have been dead simply. After waiting for at least 4 hours for tech support to call me back they finally did, but because I had left for the day I had to have them call me back the next day. Once they called me back I told them what I was trying to do and the guy told me that was impossible in the current setup and when I protested he realized that I had a load balancer and not whatever the heck he thought I had. Somebody else had entered the trouble ticket and they entered it wrong so he was working with faulty information. Grrrr. He transfered me to somebody that can trouble shoot a load balancer. Good call!
Once, I got on the phone with the correct person I was hoping that he'd have some simple solution to our problem. It seems like such a simple thing. The load balancer acts as a bridge, watches the traffic and intercepts and rewrites the traffic it's supposed to handle. I explain our set up to the guy and immediately he grills me on why I'm using Layer 4 load balancing. I explain our setup and that I don't want to re-ip our setup and their instructions on setting up direct server return were less than helpful. And to top it all off they only covered making the changes to Linux. I have happily left the days of Linux behind and we are nearly completely Solaris and Windows. I'd love to leave Windows behind as well but that's currently not an option since our biggest hosting client has to run on Windows. So the direct server return route didn't seem like a viable option and source natting (snat) was definitely out. So I explained most of this and basically told the guy this is how I want it to work, your web site said it will, let's get it working.
The support guy asks all the usual questions and then realizes that our firmware and energize versions are out of date. Energize is the updates for intrusion prevention. Since we have that disabled I'm pretty sure that's not the problem, but the firmware update definitely couldn't hurt. Because we have this on an isolated network (two laptops, two switches and the Barracuda) we have to jump through a bunch of hoops to get this connected to the network. We finally get it setup and update the firmware and the energize versions and try our load balancing setup again. This time it works. Hurray! Unfortunately, that's the last of the good news.
After a couple of minutes of passing traffic through the virtual IP address the load balancer just completely stopped passing traffic for that virtual IP. There was nothing in the logs and no clues to be found. We added a second virtual IP address going to the same two real machines and the same thing happened. It passed traffic for a few minutes and then stopped. We tried a third virtual IP address going to same physical machines and this one worked and seemed to keep on working. I tried deleting the other virtual IP addresses and re-adding the virtual IP addresses and they wouldn't pass traffic to save the world. At this point I was so frustrated with this box and I had zero confidence that I immediately boxed it up and sent it back. When something just doesn't work it's one thing, but when it selectively works that's a very disconcerting thing. Like I said I had zero confidence that this would perform correctly in production and a poorly behaving load balancer is worse than no load balancer.
After packing up the Barracrappy, um I mean Barracuda load balancer I got in touch with my second choice Kemp Technologies' LoadMaster. I'm currently in the process of ordering one and once I've had a chance to play with it I will share my thoughts on it. Hopefully it will be a better experience than the Barracuda load balancer. It certainly couldn't be much worse.
Tags: loadbalancer