Tag Archives: rails

Hardcore Rails/Server Troubleshooting Session!

So early last night, I noticed that one of my sites, Anime Nano, was flipping out and throwing 502 proxy errors. I tried to do some simple troubleshooting but it seems as though the problem came very suddenly, and I figured it might go away very suddenly as well. The thing about this particular problem was that I had made no changes to the server. So the problem should fix itself! Unfortunately, life is not so simple.

When I woke up in the morning, not only was the site still throwing errors, it was also affecting my other websites as well. MapsKrieg, Notecentric, Basugasubakuhatsu, etc. So I figured I should find out what is going on. I sent a support request to MediaTemple, my host, in case it might’ve been something on their end. It took a while for them to reply, but they eventually just told me it was something taking up all the memory and that I should try to optimize the site. Not too helpful. But I don’t really expect MediaTemple to provide this kind of support anyway. And it turned out it wasn’t their fault.

Upon closer inspection, the site was really behaving weird. I could tell by the logs that the site was actually rendering things, but it always took about 189 seconds. This was odd. I’ve had experiences before where the rendering took less than a second but the page still took more than 3 seconds to load. But 189 seconds! That was a bit too much.

I had suspected it was something to do with the mongrel cluster that I had set the site up to run on. Basically, I followed the script that MediaTemple provided and I still don’t have a great understanding of how mongrel cluster works. That’s definitely bad.

I tried stripping the view of everything but the content for layout. I got rid of before_filters and tried running the site on the mongrel clusters as well as the webrick server on port 3000. The same thing happened. It took way too long for the site to load. Thinking it might be easier to test on my local machine, I got the site and database from svn and mysql and, strangely, it worked fine on my PC.

Now things were slowly fitting together. I was reminded of a post that someone made on Anime Nano about how their feed wasn’t being aggregated into the site. Apparently their feed was unreachable from my server. I had previously chalked this up to weirdness on their server, but I wondered if it wasn’t a problem on my server. The thing about each pageload taking almost exactly 189 seconds pointed to a timeout issue. Then I realized there was a piece of code in Anime Nano that I hadn’t written. It was a plugin for Text Link Ads that I used for some ads on the bottom of the site. That plugin was grabbing a feed of ads and parsing it to show links automatically on the site.

If their site was unreachable, it’s possible that Anime Nano was just waiting on grabbing that snippet of code and timing out. And that’s exactly what happened. I commented out the text for the ads and the site immediately started working again. I’ll have to look into how I can prevent this from happening again before reinstating the ads.

So what did I learn from this whole experience? You should probably understand how code works before randomly integrating it into your site (I added that code a long, long time ago and forgot about it). Also, relying on a third party to provide some information before loading your site is just plain stupid. I can’t believe how badly thought out I had made the organization of the site.

Setting The Crontab Environment For Your Ruby on Rails Jobs

So today is my birthday. In honor of my birthday, I’ve been migrating some websites from my old server to a new one. Okay, it’s not really in honor of my birthday; I’m just doing it because I have some free time today.

Anyway, one of the things that Anime Nano needs to do is run a crontab of a function from a Rails model using a crontab. If that made no sense to you, don’t worry. Just enjoy the pretty picture of the rails logo above. If that made sense to you, wow, you’re pretty hardcore.

This is something that I remember spending time getting to work the last time I set up my server. But unfortunately I didn’t write about how I did it, so I had to spend an hour or so figuring it out again. Apparently, crontab runs in a different environment than the root user. If you try to run a script for Ruby using “ruby script/runner” you’ll just get nothing. No error message, just nothing. I did something on the previous server that set the crontab environment, but I have no idea what I did. I searched. I didn’t find the answer.

So what I ended up doing (this is for future Hung when he has to eventually move his sites to an even newer server) is sticking some lines for my environment into the crontab itself. I just ran ‘env’ and got a bunch of stuff. Comparing the ‘env’ command run from the crontab itself (I just piped the output to a random text file) and running in my shell, I saw a few differences. Most notably these ones:

GEM_HOME=/usr/local/rubygems/gems
GEM_PATH=/usr/local/rubygems/gems
RUBYLIB=/usr/local/rubygems/lib/ruby/site_ruby/1.8

And voila, my crontabbed rails script/runner model script thing ran. Hopefully writing this post will save me some time in the future.

Oh, and happy birthday to me.

Fun With Rails and High-Performance Caching Mechanisms (Memcached)

Since it’s Spring break for University of Michiganers, I’ve had some free time to mess around with learning stuff I don’t have time for when school is in session. Something I’ve been meaning to try out is memcached, which is an uber-efficient way to cache stuff in memory so you don’t end up hitting the database a lot for redundant queries.

I installed memcached on my mediatemple machine using some directions I found online specifically for centos. I don’t have yum or rpm or whatever people use for putting new stuff onto their linux boxes. I just wget, untar, and make/make install. I had to make a symbolic link for the libevent thing that I installed as well since the path wasn’t set up correctly. I’m still trying to get memcached to run automatically as a daemon when I boot up. I tried sticking a line in the /etc/init.d/ or something rc.local but it hasn’t really worked. Oh well.

I’m using memcached right now along with a rails gem called “Cached Model” that does a pretty good job of automating the caching process. What’s important to know is that the gem only caches Active Records that are found via a simple find on id. It won’t cache more complex queries. Right now I have it running on my Animenano site caching a bunch of objects. I’ve seen a small though noticeable difference.

Memcached isn’t without its caveats, however. For one thing, your initial db hit is going to take longer than normal since you’re going to be checking memcached, then grabbing the data from the db, then saving the data into memory. From reading the logs, this doesn’t really take that much extra time. The time saved is likely more worthwhile.

I’m always trying to squeeze more performance out of my sites. Periodically I’ll go check out my log files and see if there’s anything taking an insane amount of time to render (like, .1 seconds or so). I’ve caught a few bad algorithm decisions that I had made in the past, and I think for the most part my sites run pretty efficiently. Memcached is a pretty fun way to slice some time off of the page rendering times for Anime Nano.

If you have a few spare hours to kill and a rails app that seems to be a little slow, it might be worthwhile to install memcached and see if caching your Active Record objects will speed things up. I’m seeing a speedup of maybe 1/2 (it takes half as long to load a page) vs running without memcached. That’s totally non-empirical observation, though.

Ruby on Rails on Media Temple’s (dv) on Plesk on…

rails.png

So I’ve been fiddling around with trying to get my rails applications installed on Media Temple’s (dv) 3 server. It’s a pretty nice setup; basically a dedicated server (though virtual) that you can play around with to your heart’s content.

While they “support” Plesk 8.1 which in turn supports ruby on rails, they do not offer support for ruby on rails installation. Since I’m a pretty hardcore CS guy, I don’t mind learning stuff. It is easier when there’s support people to help you though.

At first I had tried using rails with fastcgi, but that’s apparently a bad idea. The dv supports fcgid anyway. Media Temple’s support told me it was already installed, but guess what? It wasn’t. I had to go into plesk and get the updater to install it.

I’ve been trying to get my apps to run for more than a week now. I’ve been working on it in my free time, and I finally got one of them to actually start doing something. There were a bunch of hurdles I had to jump over just to get it to work.

Here’s the last thing I needed to do:

I was getting some “Application error: Rails application failed to start properly” errors. The rails log wasn’t too helpful. The httpd one wasn’t either. I looked in the /var/log/httpd/suexec_log file and it said stuff like:

[2007-01-30 15:36:58]: file is writable by others: (/var/www/…/public/dispatch.fcgi)

I thought, “well okay, it’s just a warning.” Who cares if it’s group writable? After doing some other random stuff, and changing the permissions accidentally, I got this:

[2007-01-30 15:32:30]: file has no execute permission: (/var/www/…/public/dispatch.fcgi)

Okay, so now it’s complaining that it has no execute permission. Whoops. I chmodded it to something like 775 and I get the “file is writable by others” thing again, along with the application error. So I thought, “what the heck” and chmodded the file to a safer 755. It seems that did the job, so now I can execute the app!

I never knew that apache (or suexec or fcgid or whatever the heck is running dispatch.fcgi) really cared so much about permissions. I’ll have to be more thoughtful of logfiles from now on…