SunFire X4150 Strange Drive Behavior
Posted by Sam
We got a couple of SunFire X4150s at work to replace several aging P4 servers. This was the first time I've used Sun equipment in a pretty long time and I'd never setup a Sun x86 server before. For the most part it went pretty smooth but we spent a long time chasing a strange problem. Since I didn't see anything online about it I thought I would explain the problem and what we did to resolve it.
The SunFires we ordered had a total of 5 drives. Four drives were installed at the factory and the fifth was installed by us. When we booted up the machines the first four drives all had green lights indicating they were good to go. The fifth drive had no lights what so ever. If you jumped into the StorageTEK **cough** Adaptec **cough** raid controller you could see and initialize the fifth drive. It's green light would light up and you'd think all was well. Until you rebooted it that is. Once you rebooted it the green light would disappear and wouldn't come back on until you reinitialized it. Sun was little help. Their support folks were friendly and responsive but ultimately we stumbled across the answer on our own. Below is the basics for upgrading the RAID card with the latest version. This fixed the problem for us so hopefully it will help you as well.
- Boot from SunFire X4150 Server Tools and Drivers DVD Version 2.1
- cd firmware\sst
- flash.bat
- Update flash image
- Follow the prompts and reboot when instructed
The Last Word on Rails Scaling
Posted by Sam
Ok, I have to admit that I'm getting a bit tired of all the Rails scaling questions. People seem to think that's Twitter's scaling problems are Rail's scaling problems. They aren't. Twitter's scaling problems do not equal Rails scaling problems or to put it more succinctly Twitter's scaling problems != Rails scaling problems. Rather than try and convince people that Rails can scale I'm just going to include a couple of stories about large companies that have scaled Rails to Billions of page views a month. Will this finally convince people that Rails scales?
Our first Rails app that scales is brought to us by AT&T. Yellowpages.com was rewritten from a Java app to a Rails app. It serves over 1.4 billion requests a month using 25 servers per data center. That's only 4 more servers than it was using while running under Java. The Java code base weighed in at 125k lines of code while the Rails code base came in at under 20k lines of code, including tests. The much larger Java code base didn't include any tests. The entire site was coded with at most five developers over a three month period. One thing that really stands out with Rails site is the maintainability. As a developer would you rather jump into and maintain a site with 125k lines of code or less than 20k lines of code? Over the life of the site AT&T might spend a couple extra grand on servers, but they will more than make it up by having less developers and getting new developers up to speed much quicker! And do you think there are more bugs and security issues hiding in 125k lines of code or in way less than 20k? If you want to look at more details you can check out this presentation from Rails Conf 2008.
Our next app is brought to us by LinkedIn. LinkedIn built a Facebook app called Bumpersticker that handles 1 billion page views monthly and around 100TB of data each month. This app is obviously also built on Ruby on Rails otherwise I wouldn't be including it here. ZDNet has a write up and there's also a video that looks to be from Joyent. This story isn't as interesting as the Yellowpages.com story because it wasn't a rewrite and it's harder to see the benefits of using Rails as compared to a traditional environment like ASP.Net or Java but I'm including it to point out that you can get the advantages of a great framework, cut development time and costs and still scale out to a billion page views a month.
So that's it. Can we please stop asking if Rails can scale and stop using these horribly outdated technologies like Java and .Net? Please? Pretty please?
Tags: rubyonrails
CoyotePoint review coming soon
Posted by Sam
A few weeks back I was contacted by CoyotePoint and asked if I would be interested in reviewing one of their load balancers. Naturally I was ecstatic. This was my first request as a blogger and it made this blog feel a little more real and a little more important. I remember looking at CoyotePoint back when I decided on the LoadMaster from Kemp. I don't recall why I decided against it at the time but now I'm hoping that I don't fall in love with CoyotePoint because I have to send it back. The load balancer reviews get consistenly good reviews and there's definitely a lot of room for improvement over the LoadMaster so I'm happy to be able to provide a third possibility for the lower end load balancer space. Look for a review in the next couple of weeks.
Tags: loadbalancer
Compiling Ruby Enterprise Edition on Solaris 10
Posted by Sam
I wanted to try out Phusion's Ruby Enterprise Edition and hopefully Phusion's Passenger. From what I've read I know they are a pretty Linux centric pair of developers so I wasn't surprised at all that their Ruby Enterprise Edition installer didn't actually install on Solaris 10 despite using gnu tools. Fortunately, it was pretty easy to bypass their installer and compile it like I would with MRI. Here's a quick rundown of what I did to get it compiled.
Here's what I did to get Ruby Enterprise Edition compiled on Solaris 10. This assumes you are using gcc from Blastwave.
bash-3.00# wget http://rubyforge.org/frs/download.php/38084/ruby-enterprise-1.8.6-20080507.tar.gz
bash-3.00# gtar xvf ruby-enterprise-1.8.6-20080507.tar.gz
bash-3.00# cd ruby-enterprise-1.8.6-20080507/source
bash-3.00# ./configure --with-openssl-dir=/opt/csw --with-readline-dir=/opt/csw \
--with-iconv-dir=/opt/csw --prefix=/opt/rubyenterprise
bash-3.00# make
bash-3.00# make install
That's all there is to it. I wish Phusion wasn't so Linux centric. I'm going to try and get mod_rails or passenger or whatever it's called this week compiled next, but I don't think it's going to go quite so easy.
Tags: ruby
Images Belong in the Database
Posted by Sam
Logical Reasons
The more web programming and system administration I do the more I'm convinced that images and other forms of uploaded content belong in the database. I've pretty strongly suspected this for many a years but it's becoming more and more clear that this is just how things need to be. There are many reasons. Some are technical and some are logical, but all signs point to loading images in the database. Below are just a couple.
I'll start with the logical reason first and then get to the technical reasons. Logically all other forms of dynamic content will reside in the database yet programmer after programmer insists on putting images and other uploaded content on the filesystem. Now you've got some content in the database and it's tied to content on the filesystem. Suddenly the nice clean line is broken. You've blurred the lines on who does what and where it goes. What usually happens is programmers forget to clean up after themselves so files references are deleted in the database but still exist on the file system. You know why this happens? Because you shouldn't have put it there in the first place. Repeat after me - dynamic content in the database and application files in the filesystem. Got it?
Technical Reasons
So know that we've got the logical reasons behind us what are some of the technical reasons? For starters your database(s) will always be in sync. That's what they are meant to do. That's what they have to do. Your database(s) have to be synchronized. Your site depends on it. It doesn't matter if it's a simple site with a single database server or a huge site with a dozen database servers. You have to keep your database servers in sync. And databases are good at it. It's much much easier to scale out database servers than file servers. Having all your dynamic content in the database keeps everything in sync.
Some people will wrongly argue that having files in the database will cause slowdowns. To that I say....um maybe, but only if you're doing it wrong. Databases are very fast and have excellent caching but it's highly likely that multiple web servers can server images straight from disk faster than a database. Well that's great, but images still belong in the database. And in this case you can have your cake and eat it too. Check out this article about caching images on the file system with Rails. Hrm, one extra line of code lets you cache your image on the file system. Doesn't seem to bad to me. Elegant, simple and lightning fast.
The Reason for the Rant
Time after time I'm responsible for deploying other people's crappy software. They invariably come up with some stupid solution that lets them click the checkbox that says their product will work on a load balancer, but they neglect to tell you upfront what ridiculous hoops you have to jump through to get it work. The most recent product is called Ektron. I can't say what it's like to work with from a programming stand point but from an admin stand point it's a nightmare. Instead of loading the images in the database like they should they instead force you to share out files and do this dumb little virtual directory linking to the other servers. It's just annoying and not even close to elegant, but then I have yet to see a CMS system that is so I wasn't surprised. Thankfully the Ruby on Rails team understands what it takes to scale apps and they provide you with a nice foundation. If only the rest of the world would catch up.
Tags: rubyonrails web rant