Friday, June 27, 2008

using Ruby for IMAP with Gmail

Yesterday, I spent the day working on a problem involving emails. Basically, it can be summed up as follows: the Rails application I'm working on has certain model objects that need to be able to have some data populated via emails (an email gets sent to a certain address, and the app processes it and populates data based on the content).

Now I'd read about ActionMailer before, I knew that it had a nifty way to receive email messages and process them as they arrive, so I went and looked at their documentation on how to tackle that problem.

The most advocated method of handling the incoming email is to configure the Postfix server on you box to forward each email into your mailer script as it comes in, but I'm not a fan of that path for two reasons. First, I don't have any desire to worry about configuring a mail server. That probably makes me less "hardcore" in the eyes of any *nix masters out there, but given our time constraints I am forced to accept that I'm just not strong enough in that area yet to trust a business's success to it. Additionally, and probably more importantly, using a real-time forwarding approach means kicking up a new rails process EVERY TIME an email comes in. What if I get hundreds of emails in a single hour? Thousands? All of a sudden, crashing my server becomes as easy as spamming it's mailbox. No, there must be a better way.

And there is. Rather than try and accept each email as it comes in, you could poll the email box with POP or IMAP every so often and thus use one session to process however many emails have arrived in the time since the last session. To me this was obviously superior, and after checking several blogs of other ruby community members I was ready to give it a shot.

So I started a gmail account for ease of testing, and wrote a script that looked something like this:

pop ="", port)
pop.start('YourAccount', 'YourPassword')
if pop.mails.empty?
puts 'No mail.'
i = 0
pop.each_mail do |m|"inbox/#{i}", 'w') do |f|
f.write m.pop
i += 1
puts "#{pop.mails.size} mails popped."

And this probably would have worked fine if not for one thing. For my purposes, I need SSL to be enabled, and the method on line 2 of the code example above ("enable_ssl"), is an enhancement made in ruby 1.9. Unfortunately, that's a development release, not yet production ready, and 1.8.6 (the current stable version, which I'm using) doesn't have ssl support in it's POP library.

I briefly thought adding the functionality to the 1.8.6 source code myself, but decided that would have to be a last resort tactic. Instead I switched to the IMAP library to see if I'd have any better luck. Here would be a typical code sample you'd see for fetching mail with the ruby IMAP library:

imap ='')
imap.authenticate('LOGIN', 'username', 'password')'INBOX')['ALL']).each do |message_id|
msg = imap.fetch(message_id,'RFC822')[0].attr['RFC822']
MailReader.receive(msg), "+FLAGS", [:Deleted])

This looked promising, but every time I tried to run it I got a very strange error back from the server. Basically, the script would die and print out the words "Not Supported" and a filename coming back from the authentication request (line 2 above, imap.authenticate). I went and looked at the docs, and saw that the authenticate method supports 2 types of authentication: 'LOGIN' (like above), and 'CRAM-MD5', either one can be passed in as the first argument, and further more some servers won't support one or the other.

Sure I had found my fix, I replaced 'LOGIN' above with 'CRAM-MD5', and in an anticlimactic script run I found I got the exact same error.

If you're having this problem when trying to connect to gmail, I'm going to save you some time. The authenticate method in the IMAP library sends an "AUTHENTICATE" IMAP command to the server, which gmail does not support (you can prove this by sending a "CAPABILITY" command to the server, and seeing what options it has available. AUTHENTICATE ain't there).

Instead, using the "login" method (which sends no specific command), you can connect successfully. Here's my final script:

imap =['server'],@config['port'],true)
imap.login(@config['username'], @config['password'])'INBOX')["NOT", "DELETED"]).each do |message_id|
MailFetcher.receive(imap.fetch(message_id, "RFC822")[0].attr["RFC822"]), "+FLAGS", [:Deleted])



Erb said...

Nice job. A lot of my clients have a lot of employees so I just use the IMAP login action to authenticate them so they don't have to maintain a list of users in two systems.

I came across the same problem. Probably should've written something up on it but I'm a dick :o)

This will probably help a lot of people. I think it's becoming a more popular method of authentication especially as more corporate apps are written in Rails...

Luke Francl said...

You can also check out the fetcher plugin which works with Gmail with both POP3 and IMAP (along with other POP3/IMAP servers, of course).

Documentation available here.

This code is also featured in this PeepCode if you want more info.

Ethan Vizitei said...

@luke francl

Cool plugin, thanks for the link!

Element said...

thanks @ethan for the post and thanks @luke for the plugin link. looks very cool and it's great that there is a peepcode pdf for it.

For me as a small independent freelancer gmail/google apps is HUDGE. I've set up a mail server before and it took weeks of reading and frustration. Same with DNS server etc..

I'm glad I have that knowledge, but in the end all I had was a single point of failure and a lot of complicated things to keep track of that I had at best a rough knowledge of.

I offloaded DNS a long time ago, but now that I can offload email my server is just running the rails stack.

I feel a hundred pounds lighter knowing I don't have to deal with spam and storing peoples valuable emails.

ashchan said...

Although by putting the POP3 lib from 1.9 in we can use the POP3 way to fetch mails, IMAP is still preferred IMHO. POP3 seems to fetch all mails every time you run the script, and runs a little slower.

Besides IMAP and POP3, the gmailer gem is another good option, although under some circumstances it fails to work.

siannopollo said...

I had to do the same thing in the past few months and here is what I came up with (basically using the ruby 1.9 libraries to get things to work):

It works with both IMAP and POP and was tested against a gmail account.

Hideaki said...

Great post! Exactly what I was looking for. Thank you!

Vijendra ( ವಿಜೇಂದ್ರ ರಾವ್ ) said...

Nice post.

This may help some one.

Configuration setting required to recive mails from Gmail (using fetcher)

kedar said...

Nice Job And Thanks For Saving My Time