Wednesday, September 30, 2009

why your named_scope isn't behaving the way you want

Let me go into this as briefly as possible.  I had the following named_scope in one of my rails models:

class Alert < ActiveRecord::Base
  named_scope :in_the_past,
              :conditions=>
                ["start_datetime <= ?",DateTime.now]
end
Seems straightforward, right? I thought so too. I used this "in_the_past" method to replace a poorly written load-and-filter algorithm. This would use much less memory and be much quicker. Only one problem it didn't work. It passed all the unit tests. It seemed to work when it was deployed. But over the next few days more and more people were calling me saying they weren't seeing the alerts they expected to. I tried to figure out the problem (maybe related to timezones? or bad data?), and every time I deployed a fix all my users would tell me it looked good again; but then a day or so later everyone was complaining once more. I finally got fed up with it and went back to the old way (loading a collection and filtering out the items in the past using a "select" iterator), cursing the day I'd ever started down this road. And then today, I saw this in a blog post from pivotal labs: Public Service Announcement: your named_scope's get evaluated when the class is loaded, not when an instance is created. I felt like I'd been hit by a truck. The "DateTime.now" call was being evaluated at server startup, so for the first few minutes it seemed fine. However, as the amount of time since class-load increased, the "in the past" scope was missing more and more of the items that were actually now in the past because it's measuring stick was stuck at the time of creation. As a result I dug through my codebase and found no less than 7 scopes to refactor into using a lambda instead so that the system time would be loaded at time of execution instead:
class Alert < ActiveRecord::Base
  named_scope :in_the_past,lambda{
                {:conditions=>
                   ["start_datetime <= ?",DateTime.now]}
              }
end
Thanks to those pros at Pivotal Labs!

Wednesday, September 23, 2009

GMail hijacking

So my email account has been compromised. I don't know how, but what I do know is I'm locked out.  I'm working on it with google, but the short version of the story is this:  someone sent out an email to everyone in my contacts list saying I need money to get home from london.  I'm not in london.  i don't need money.  Ignore this and any other emails that don't sound like me.

Tuesday, September 15, 2009

SpriteMe up, Scotty!

This morning as I perused my rss aggregator, I came upon an article over at Ajaxian about a cool new tool for auto-generating CSS Sprites. This seemed like a cool idea, but I didn't think much of it. After all, I've heard of hundreds of projects that try to do something I think would be really useful, and when I go to try it myself I find it's such a pain to get working it's not worth the effort and I may as well accomplish the original task manually.

However, I decided to give it a try in this case because one of my current startup projects (VacationsTxt) was in dire need of some spriting attention. You see, there were 22 background images on one page, which means load time was pretty much sucking as 22 http requests had to be made just to get all the images alone, nevermind the javascript files and stylesheets. So I told myself I'd give it 5 minutes to spike this new tool and if it looked like it was going somewhere I'd maybe invest a little more effort.

Imangine my surprise, 5 minutes later, when I was done. Yes, done. SpriteMe is just a javascript bookmarklet that you run while you're on the page in question. It auto-magically scans all background images and tries to combine them into a css sprite. Then you just download the new image, integrate the necessary css changes that have been applied to your page (live, I might add, right while you're watching it), and you're done.

So how much did SpriteMe help my homepage? See the screenshot below to find out:



I don't know if that's readable or not, but what it says is this "21 requests eliminated".  That's 21 fewer http calls that my server has to handle for every visit to the homepage, and 21 fewer requests the user has to wait through on load. 

More importantly, how much work did it save me? Well, being no graphics expert, I predict I would have spend 4-5 hours trying to get the sprites right for my homepage. Doing all that in 5 minutes is impressive. Most impressive.

Steve Souders, you've made my day.

Friday, September 11, 2009

Browser Caching and Rails

Alternative title: "Your debugger can't debug what it doesn't execute"

At my startup, we had been experiencing an intermittent user problem for the last few days, with no clue as to the cause.

Users have been reporting that their "alerts" aren't showing up when they login...sometimes....and it goes away a few minutes later....sometimes.

Anytime I hear a report like that, I start dreading the work ahead, because if it's only "sometimes", there's probably not an easy way to reproduce the bug reliably. It could be data-dependant, or a browser quirk, or dependant on the phase of the moon for all I know.

So I started digging in to our code base. Alerts were displayed based on when certain events had gone into the past. Was there a time-zone problem? Was the current time being extracted incorrectly? Was there a problem rendering the partial that handled that block of the homepage? Maybe the CSS was displaying it improperly if there were other dynamic elements on the page?

I started throwing logging in at various points through this controller action. Hours later, reports were still coming in (slowly, so not everyone was experiencing this problem), and the logs looked fine. Then I started to notice something. The people who were complaining the most, had the fewest log reports. In fact, the production log showed that one of the users who was complaining had not accessed the page in question for 2 days.

Enter the browser cache.

For some unknown reason (damned if I could figure this out), this user's browser decided to cache her homepage. She wasn't seeing the alerts because her browser had chosen to cache this page at a moment when she didn't have any. Until it decided to reload the html from that page, she would continue to see no alerts.

With some quick googling, I found a cache buster snippet, and I'll share it here in case anyone else has had the same problem. To summarize:

CLUES THAT A BROWSER CACHE IS CAUSING YOUR BUG:

1) only some users report an issue
2) issue is intermittent
3) no errors being reported from your notification software
4) logging shows no data problems or logic errors
5) symptoms indicate "time displacement" (that's what the page would have looked like "at one point")
6) serious frustration and anger on the part of the software developer

If you are a rails developer, here's the medicine (this goes in application controller):


def set_cache_buster
response.headers["Cache-Control"] =
"no-cache, no-store, max-age=0, must-revalidate"
response.headers["Pragma"] = "no-cache"
response.headers["Expires"] = "Fri, 01 Jan 1990 00:00:00 GMT"
end


set this as a :before_filter on any action you want to protect.

Wednesday, September 2, 2009

Forwarding Emails with Ruby, IMAP, and SMTP

I would hate to tell you just how much it sucked trying to figure this out.

Actually, I would LOVE to tell you. It would be totally cathartic to just unload a days worth of frustration. However, that wouldn't help you figure out how to do the one thing mentioned in the title of this post (which is probably the only reason you're here), and it sure as hell wouldn't make me seem like much of a professional.

So I'll skip the tirade except to say this: Maybe I should have given up 4 hours ago.

That out of the way, here was my problem. I have the email box where all these clients send data files to. I wanted to sort the ones that had attachments (and only ones with attachments) based on who they came from and forward them to different email addresses. There are all kinds of nice ways this could have been done if I wasn't constrained by some external requirements (like not being allowed to use forwarding rules on the server the emails were sitting on), so for the sake of this post assume this had to be done in ruby using net/imap to fetch the emails, and net/smtp to send them to wherever they needed to go.

This was not a fun place to start. These libraries work well, but I'll be honest when I say that MIME and IMAP are a little low level for my accumulated programming experience. So I started by using as a jumping off point a script I wrote for another blog post a while back when I was trying to access my gmail inbox with ruby. I got far enough to where I was polling the email inbox successfully, so now I just needed to figure out the "sending" part. No big deal, right?

I can't tell you how wrong you are.

You see, you can't just pass off a Net::IMAP::FetchData object into the net/smtp library and watch it fly off into the designated mailbox. NO SIR! You see, I had that idea, and this is what I tried:



require 'net/imap'

imap = Net::IMAP.new('mail.yourserver.org')
imap.login('username','password')
imap.select("INBOX")
imap.search(["SINCE", "8-Aug-2007"]).each do |id|
email = imap.fetch(id, "BODY[TEXT]")[0].attr["BODY[TEXT]"]
Net::SMTP.start('smtp.server.org') do |smtp|
smtp.sendmail(email, 'from',['to'])
end
end



It was not very successful. You see, my email client kept showing me the full MIME body, not just the visible components, and attachments were showing up as base64 encoded strings. This was not useful.

I decided the problem was most likely that I was just pulling incomplete MIME information from the imap library and that I needed to find the right attribute to ask for that would have the full MIME package in it's raw form. Enter "BODY[]". It might look very much like the "BODY[TEXT]" attribute I used above, but you would be deceived. "BODY[]" is the whole deal, everything that's there. If you want the raw message, that's what you need.

So that's what I tried:



require 'net/imap'

imap = Net::IMAP.new('mail.yourserver.org')
imap.login('username','password')
imap.select("INBOX")
imap.search(["SINCE", "8-Aug-2007"]).each do |id|
email = imap.fetch(id, "BODY[]")[0].attr["BODY[]"]
Net::SMTP.start('smtp.server.org') do |smtp|
smtp.sendmail(email, 'from',['to'])
end
end



Presto, it worked! No, I'm lying, same problem. So what to do now? Honestly, after the amount of reading I'd done just to arrive at the above conclusion, I was considering giving up. But I thought to myself: "This should be a solved problem. Someone out there should have already written something to take care of dealing with MIME formatting so that I DON'T HAVE TO THINK ABOUT IT!"

Enter rubymail. It looked very promising since the docs said it would parse and generate MIME format behind the scenes of a simple interface. "Great", I thought, "Let's try it":




require 'net/imap'
require 'rmail'

imap = Net::IMAP.new('mail.yourserver.org')
imap.login('username','password')
imap.select("INBOX")
imap.search(["SINCE", "8-Aug-2007"]).each do |id|
email = imap.fetch(id, "BODY[]")[0].attr["BODY[]"]
email = RMail::Parser.read(email).to_s
Net::SMTP.start('smtp.server.org') do |smtp|
smtp.sendmail(email, 'from',['to'])
end
end



This looked promising, but for an unknown reason RubyMail was parsing my emails wrong and making things that were part of the body out to be headers. It wasn't going well. Convinced I was on the right track with this whole "Let somebody else deal with the MIME" idea, though, I substituted "tmail" (another good ruby email library).



require 'net/imap'
require 'tmail'

imap = Net::IMAP.new('mail.yourserver.org')
imap.login('username','password')
imap.select("INBOX")
imap.search(["SINCE", "8-Aug-2007"]).each do |id|
email = imap.fetch(id, "BODY[]")[0].attr["BODY[]"]
email = TMail::Mail.parse(email).to_s
Net::SMTP.start('smtp.server.org') do |smtp|
smtp.sendmail(email, 'from',['to'])
end
end



there you are, scripted forwarding. May that help someone out there to not have a day like mine. :)