Ethan's Software Blog

Inspired

2011-03-22T21:46:00.000-05:00

I've been reading way more than writing lately, and my current favorite blogger has got to be Alan Skorkin. In particular, I'm a fan of two of his articles, one on the value of fundamentals, and one on the "I'll learn it when I need it" attitude. I love his writing, but more importantly, I'm convicted by his arguments.

With respect to these points, I'm making it a point to encourage my growth at least a little each day, starting right now by taking Alan's advice from the former article and exploring the API of the Array class in Ruby (a class probably used more often than any other in my day to day work). See my next post for anything I might learn...

Tag-it 0.3.1 is released!

2010-11-22T00:15:00.000-06:00

If you've been a consistent reader, you know that last month I released a new gem called "tag-it" to interface with RFID receivers and the tags that come into range with them. Today, there's a new version with a new feature.

In version 0.2.x, there was 1 primary class for continually monitoring a port for tag events. This doesn't do everything I wanted though, because in some cases I really just needed to know what tags were in range at one precise instant. The long-running class really wasn't a good fit, so there's a new class available called TagIt::TagSnapshot which has a "shoot!" method that returns an array of the tag-names currently found:


  require "tag_it"
  port = SerialPort.new("/dev/tty.yourport",{:baud=>9600})
  snapshot = TagIt::TagSnapshot.new(port)
  tags = snapshot.shoot!
  # => ["1nri","1okD","1nrP"]

Get the latest:


  gem install tag-it

json (_pure) ruins my morning

2010-11-16T08:57:00.001-06:00

wrong argument type JSON::Pure::Generator::State (expected Data)

This is what I saw staring back at me from my console this morning when I checked out a project and ran my specs. Specs that had been passing just fine yesterday. The only thing I could think of was that I had added a couple new gems to my gem file, but since I hadn't used them yet surely they couldn't be breaking my test suite?

Well, if you've been a coder for any length of time, by now you should know that the likelihood of a given event contributing to your code failure is inversely proportional to your initial index of suspicion of it's relation.

Fortunately, I didn't have to search long. Googling for "wrong argument type JSON::Pure::Generator::State" quickly turned up this blog post on prettystatemachine that explains the problem. Those gems I'd added? One had "json_pure" as a dependency, so I'll include my own brief version for reference:

ActiveSupport and json_pure both hook into the "to_json" method. ActiveSupport defines to_json for many simple types, but expects fixnum's "to_json" to be handled by Object. json_pure is intercepting that call and not getting the data it wants. Defining "to_json" on fixnum seems to solve the problem.

Hence, I've created an initializer in my rails app called "./initializers/json_patch.rb" and put the following code in it:


class Fixnum
  def to_json(options = nil)
    to_s
  end
end

Specs pass. I would thank "prettystatemachine" by name, but I can't find any reference to the author of the blog. Whoever you are, thanks man.

Hello node

2010-11-13T23:58:00.001-06:00

I've been playing a bit with node.js recently, but have had trouble getting my friends to try it out. They hate javascript, can't see how this framework is any different, won't even give it a look, and further more evented programming sucks.

Well, everyone is entitled to their opinion, but you can't really have a valid opinion without the information. Node.js and other event driven models (EventMachine in ruby, etc) are going to continue to be a huge deal as more applications require more realtime information to be pushed between client and server, let's at least know a little about the tools that are probably carrying the future on their actively-under-development shoulders.

In this spirit, I've put together a very simple "hello world" application in Express (web framework for node.js), available on my github profile here: Hello Node

You see how nice I am? You don't even have to write your own hello world, I just did it for you, all you have to do is check out the code and read through all 4 files therein to get a feeling for why this new tool really isn't all that unfamiliar compared to whatever web framework you're working with now. No fan-boy over-hyped blog post about it's infinite scalability, no drooling and sobbing over how wonderful it is, just a few blocks of harmless code that wants to be your friend. Still hate everything about it? Fine, but at least you'll have some ground to stand on from legitimately having taken a look.

Testing Heroku's SMS addon with Cucumber

2010-11-13T15:27:00.002-06:00

(UPDATE: All code on this page is also available as a gist here: GIST)

I love Heroku's add-ons.

I love Cucumber testing.

But I sometimes hate putting the two together. How do you test the infrastructure you don't know much about? I've been confronted with this recently with the "moonshado-sms" addon, which is a super-simple way to hook your application up to a cheap sms gateway. My integration tests were difficult, though. How do I check that everything got sent appropriately? That the SMS went to the right person, with the right number, and the right message?

I stumbled a bit, but came up with something that I think is pretty clever, and I'm going to share it here in case you have the same insatiable desire to see a series of green dots before deploying that I do. In my env.rb file (for cucumber), at the bottom, I include some monkey patching of Moonshado's API:


module Moonshado
  class Sms
    cattr_accessor :sent_messages
   
    def deliver_sms
      raise MoonshadoSMSException.new("Invalid message") unless is_message_valid?(@message)

      data = {:sms => {:device_address => format_number(@number), :message => @message.to_s}}

      self.class.sent_messages ||= []
      self.class.sent_messages << data
      response = RestClient::Response.create('{"stat":"ok","id":"sms_id_mock"}', "", {})

      parse(response.to_s)
    rescue MoonshadoSMSException => exception
      raise exception
    end
  end
end

Now mix in some clever step defenitions with "sms_steps.rb" in your cucumber "step_definitions" folder:


When /^the sms message "([^"]*)" is sent to "([^"]*)"$/ do |number, message|
  Moonshado::Sms.new(message,number).deliver_sms
end

Then /^there should be an SMS sent to "([^"]*)" saying "([^"]*)"$/ do |number, sms_text|
  messages = Moonshado::Sms.sent_messages
  messages = Moonshado::Sms.sent_messages.select{|data| 
    data[:sms][:device_address] == number and data[:sms][:message] == sms_text
  }
  messages.size.should == 1
  messages.each{|msg| Moonshado::Sms.sent_messages.delete(msg)}
end

Then /^there should be no SMS messages sent$/ do 
  if Moonshado::Sms.sent_messages
    Moonshado::Sms.sent_messages.size.should == 0
  end
end

Take and edit for your own purposes, and may your code always be well tested.

(PS: All code on this page is also available as a gist here: GIST)

Changing Gears

2010-11-13T11:19:00.001-06:00

Sometimes opportunities come into your life that seem too good to pass up. That happened to me this week when a conversation with the "higher-ups" at 12spokes Web Development turned into an offer.

Circumstances are such that this is a good time for me to make a move like that, so I was happy to accept, and for the foreseeable future that's where I'll be contributing the bulk of my Ruby on Rails skills. I've got a good vibe from this company, and am really looking forward to branching out onto bigger projects, so keep watching for any knowledge gems I pick up as I come up to speed on the current projects being worked on at this shop.

And by the way, if you're looking for a web development shop to build your next application....well, I know a guy. :)

Vim, Day 2

2010-11-08T10:36:00.004-06:00

As I mentioned in a previous post, I recently started trying to use Vim instead of textmate as my default editor, with mixed results.

Are some of the command shortcuts awesome? Yes. I love "dd" for deleting a full line, and "o" or "O" for inserting a new line above or below where I am. But that doesn't totally make up for a loss of direction when it comes to moving between files; that's what I DIDN'T love.

Fortunately, one of my friends forwarded me this blog post by Yehuda Katz, where he gives a great summary of his experience in switching into Vim incrementally, but which also (most importantly to me) mentioned the "NERDTree" plugin.

If you don't know about it (I didn't) it gives you a "tree" interface to the directory you specify (I'm using my project home directory) and lets you use a mouse interface or a series of handy shortcuts to navigate. There's nothing else holding me back.

Now I'm sold. Powerful text editing, workable project drawer, I'm a Vim advocate from today forward.

R you programming?

2010-11-07T17:24:00.000-06:00

My wife is in school again getting her masters, and one of the classes she's having to take is statistics. An interesting side effect of this is that she's having to do something that she never thought she would: Programming! Yes, her class is using SPSS, but at home she's starting to work with the "R" programming language, and I'm frankly fascinated.

I'm so excited about sharing something technical with my spouse for the first time, that I'm downloading R onto my laptop myself to learn along with her, and I've bought "R in a Nutshell" from the O'Reilly book series to help me along the way. In this post, I'll be cataloguing some of my first impressions as I play with this new (to me) language.

1) R is interpreted, has a REPL, and so reminds me of when I first started learning Ruby or LISP.

2) R uses vectors a lot, which is expected for a language designed for statistical analysis. Although initially I was thinking of them like an Array, in truth they're different and in some cases a little more powerful than arrays in other programming languages. Vectors provide advanced indexing (integers, ranges, mapped selecting functions) and vector arithmetic both in a concise format.

3) functions in R are just objects. This provides for an interesting source reading technique:

> double_func = function(x,y,z){ c(x*2,y*2,z*2)}
> double_func(2,4,6)
[1] 4 8 12
> double_func
function(x,y,z){ c(x*2,y*2,z*2)}

4) R also has a class for arrays (which is defined as multidimensional vectors), and although I've dealt with multidimensional arrays before, they feel totally different in this syntax:

> a = array(data=c(1:100),dim=c(2,10,3,2))
> a
, , 1, 1

     [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10]
[1,]    1    3    5    7    9   11   13   15   17    19
[2,]    2    4    6    8   10   12   14   16   18    20

, , 2, 1

     [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10]
[1,]   21   23   25   27   29   31   33   35   37    39
[2,]   22   24   26   28   30   32   34   36   38    40

, , 3, 1

     [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10]
[1,]   41   43   45   47   49   51   53   55   57    59
[2,]   42   44   46   48   50   52   54   56   58    60

, , 1, 2

     [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10]
[1,]   61   63   65   67   69   71   73   75   77    79
[2,]   62   64   66   68   70   72   74   76   78    80

, , 2, 2

     [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10]
[1,]   81   83   85   87   89   91   93   95   97    99
[2,]   82   84   86   88   90   92   94   96   98   100

, , 3, 2

     [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10]
[1,]    1    3    5    7    9   11   13   15   17    19
[2,]    2    4    6    8   10   12   14   16   18    20

5) "lists" in R are a little different in that they can have named components and can hold different object types. Building a list in R almost feels like building a struct in C++ or Ruby:

> car = list(make="Honda",model="CRV",mileage=1258)
> car
$make
[1] "Honda"

$model
[1] "CRV"

$mileage
[1] 1258

> car$make
[1] "Honda"
> car$model
[1] "CRV"

6) Everything in R is an object, and a "class" is basically a certain type of function definition. In this way, the language feels reminiscent of javascript, where there isn't really an alternate sytax for defining global functions and object classes.

That's all for one day, don't want to cram in too much at one time. But as an overall impression, I'm getting a feeling for why the language is so valued for statistics. Data is very easy to build and manipulate en masse, and I'm sure that although I haven't gotten there yet, there are probably many built in functions for analysis to help along the way. Subscribe to my RSS feed if you want to get further updates as I dig deeper into the R language in weeks to come.

Embedding Cramp in a Rails App, Part II

2010-11-05T15:31:00.000-05:00

After my last post on getting "Hello World" running on Cramp alongside a full rails application, I thought I'd continue to catalogue my progress on actually getting a websockets action running and doing a real data push back to the client.

First things first, the blog post by Pratik that introduces the concept is invaluable: http://m.onkey.org/2010/1/15/websockets-made-easy-with-cramp.

Following the directions there, I created an initializer (config/initializers/cramp_server.rb for cramp and included the following:

Cramp::Websocket.backend = :thin

Notice that this isn't quite the same as the blog post, because I'm using edge cramp, and it's been refactored to remove the "Controller" module.

Anyway, rebooting the server everything still worked, so the next step was to rebuild my controller as a websockets aware rack application.


class ChatAction < Cramp::Websocket
  on_data :received_data

  def received_data(data)
    render "Got your #{data}"
  end
end

Which did indeed startup and respond to data requests from the client with an echo. Nice!

However, in order to support IE users, it would be necessary to use flash sockets, and although it's not that hard to do, I'm already getting uncomfortable with the number of things that are new to me in this app. So, as a responsible design decision, I'm planning on just using Cramp for long-polling for the time being until better websockets support is available across all the major browsers.

Thanks Pratik for such a cool framework!

Embedding Cramp in a Rails App

2010-11-05T12:33:00.001-05:00

So I have an application that would work great with websockets because I don't want to be doing client polling. Rails doesn't deal with this right now, but Cramp does (cramp is a young web framework built on top of EventMachine). I know, I know, Node.js right? But I've already deployed on Heroku and they aren't taking new participants for their Node.js beta, so I'm going to have to make it work.

Now, Cramp is indeed a Rack-based framework (mostly, sometimes not compliant), so theoretically I should be able to do this, and if it works out you'll have this post to help you along the way (using Rails 3.0, BTW).

First, I added Cramp to my gemfile (gem "cramp") and tried to bundle install it. Failure. The current cramp gem depends on arel (= 0.3.3), and Rails 3 of course need arel 1.1.0.

To fix that, I tried loading edge cramp into my app by using the git repo in my gemfile:

gem "cramp",:git=>'git://github.com/lifo/cramp.git'

So far so good, at least the bundle installed correctly.

Before going any farther, I also added "thin" to my gemfile to use as my webserver in development, because in the Cramp documentation they point out that Cramp only works with Thin or Rainbows!.

Now, to get a cramp application running. I saw a great Screencast by Ryan Bates about Routing with Rack, so that's the approach I'm going to take using the "Hello World" example from this blogpost (IMPORTANT: Edge cramp does not use "Cramp::Controller::Action" anymore like in that blog post, Action is directly under Cramp now). To clarify, I'm dropping the following class into the lib directory:

class ChatAction < Cramp::Action
  on_start :send_hello_world

  def send_hello_world
    render "Hello World"
    finish
  end
end

And then tried to route a path to it (routes.rb):

MyApp::Application.routes.draw do
  match "/chat"=>ChatAction
end

Starting up the server, I got a failure due to an uninitialized constant (It doesn't know about my ChatAction yet). I had to think about the right place to load it, because I haven't really traced the rails startup sequence before. Looking in "application.rb", Bundler loads all it's gems after loading up rails, so I decided to drop in my requirement right after that (load all bundler gems, than load my Cramp application, then start with the rails app config):

require File.expand_path('../boot', __FILE__)

require 'rails/all'

Bundler.require(:default, Rails.env) if defined?(Bundler)
require "lib/chat_action"

module MyApp
  class Application < Rails::Application
    config.autoload_paths += %W( #{config.root}/extras )
    
    config.encoding = "utf-8"

    config.filter_parameters += [:password]
  end
end

That seemed to work; at least everything booted without any more errors when I did this from the application home directory:

rails server thin

and when I traveled to "localhost:3000/chat", I saw my "Hello World" render as expected.

So there you have it, how to get a cramp application running in your Rails 3 application (in the patented "Stumble your way through" methodology). Now to actually make it do something useful!

Vim at last

2010-11-04T12:20:00.001-05:00

I've been using TextMate for a long time, and I love it. Starting my career in Visual Studio, the monster of all IDEs, I've gradually simplified over the years one small step at a time; First by moving to Eclipse, Than XCode, to netbeans, finally to TextMate, which I've been using for 3 years now. However, over that time period I've watched friends of mine blaze away in their code with their mystical mouse-less ways on editors like EMACS and VIM. Today I decided to dive in and try it out myself.

Setup

Downloading the Mac OS X binary for MacVim from the google group is not really in the spirit of the game (if I was feeling hardcore today I'd compile it from the source on github repo), but it sure is fast for setup. Uncompress, move the .app file into the Applications directory, and away you go (you can also move the "mvim" script that comes along with the download into a folder that's on your Path so you can open it from the command line any time; I did). Of course, that's also when I realized that the console version of Vim comes standard on my Mac. Oh well, use which ever you prefer.

Learning

The first thing I notice is that nothing makes sense. I can't seem to click anything. Than I remember that you aren't really supposed to be trying to click on things if VIM is your editor of choice, because it's keyboard focused. Realizing I'm not going to figure this out myself, I opt for running "vimtutor" at the console which gives you a concise walkthrough on how to operate inside this strange program.

Results

...Time passes...

Ok, so now I can move text around in VIM pretty comfortably, and when editing a single file, I'm pretty solid. I'm happy with how short the commands are to do things that require a lot of highlighting and click-dragging in modern editors (delete a full line, jump a few words down the line, etc). What I'm NOT comfortable with yet is jumping around to different files in a project, and the syntax highlighting is still unfamiliar to me. However, the only way to GET used to it is to use it, so although it's going to hurt for the first couple days I'm sure, I think I have enough solid ground under my feet to make a go of it.(EDIT: rails.vim makes a big difference for me)

Stay tuned for updates on whether I find out that this really improves my coding or not.

The options Hash, a classic ruby pattern

2010-11-03T20:46:00.000-05:00

If you use rails, or any of a thousand other ruby gems, you're familiar with using methods like this:


phone = SweetPhone.new
if caller.is_my_friend?
  phone.ring!(:volume=>6,:ringtone=>:smooth_jazz)
elsif caller.is_my_boss?
  phone.ring!(:volume=>11,:ringtone=>:totally_obnoxious)
else
  phone.ring!
end

yes, the conditional could be constructed to just build the hash and only call the method in one place, I know; that's not today's point. What I'm emphasizing is the fact that this "ring" method can just be called by itself, or with any of a series of options. It's nice, because you have intelligent defaults, and easy configuration if necessary.

The problem I see, is that even though this is the simplest of the basic ruby patterns that you see every day, and even though we're all more than happy to use these APIs in third-party libraries, I still run across code every now and then in home-brewed applications that looks like this:


class LamePhone
  def ring!(volume,ringtone,vibrate,max_rings,caller_id)
    #...
    #implementation
    #...
  end
end

and 80% of the code that calls these long-paramed functions uses the SAME SET OF PARAMETERS.

If you've been programming for a while, this isn't news to you, so disregard this post, but if you're new to ruby or software in general, take note: repeating yourself is a bad sign. If you're sending the same message from 50 places in your code, and 40 of them use the same function list, it may be time to set some intelligent defaults. To fix this anti-pattern, the options hash works beautifully:


class SweetPhone
  def ring!(options = {})
    adjust_volume(options[:volume] || 6)
    ringtone = pick_ringtone_from_category(options[:ringtone] || :all)
    sound_the_ringer!(ringtone)
  end
end

the whenever gem and EngineYard AppCloud

2010-10-29T14:26:00.000-05:00

If you have any sort of jobs you need run on a regular basis for your rails app, I can highly recommend the whenever gem. Make a schedule.rb file with all your cron jobs in a pleasent ruby DSL, it's lovely! Every time you deploy, you can setup a hook in whatever deploy process you're using (chef, capistrano) to run the "Whenever" executable script and update your crontab. Sweet!

But we've had a problem recently when we deployed our app to Engineyard AppCloud. We put in a deploy hook to run the whenever script, so far so good. The only problem was deployment wasn't the only time that we needed our crontab updated. You see, as with most cloud setups, we could rebuild our cluster at anytime to make configuration changes or what have you, and it should pick up where it left off. That would bring our code back, but our crontab would be squeaky clean (we found that out the hard way when we rebuilt our cluster and suddenly our billing jobs weren't running).

Fortunately, engineyard uses Chef for their cluster configuration, and they provide you a way to apply Custom Chef Recipes to your environments that will do things during your cluster rebuild.

Taking advantage of this, I built a small custom recipe for "whenever" that simply executes the whenever script in your current app directory (only if it's a utility instance or a solo instance, so your app servers aren't running your big jobs). Check it out at this gist!

Also, if you'd like to see it in it's native environment as part of your ey-cloud-recipes project, you can check out our public git repo Here (recipe is in /cookbooks/whenever/recipes/default.rb).

Cheers!

tag-it 0.2.2 is released!

2010-10-23T20:50:00.001-05:00

If you read this blog regularly, you know that last week I released a new gem called "tag-it" to interface with RFID receivers and the tags that come into range with them. Well, there's a hot new version.

Before, there were only 2 events that were thrown out by the tag tracker; one when a tag arrived (came into range) and another whenever a tag departed. The problem was, unless people were actually moving into and out of range of the receiver, there was nothing going on. That's not really an inherently bad thing, but from my web-app's view it WAS very similar to what would happen if the client were totally non-functional (powered down, error'd out, etc). In order to allow applications to be sure that nothing's happened, tag-it now also sends out a "pulse" event every 180 seconds (including an array of currently in range tags). Get the latest with


  gem install tag-it

Arrival!

2010-10-19T10:09:00.000-05:00

One of my most attainable dreams has finally come true, as my most recent gem was mentioned today on Ruby 5. I've been listening to the show since it's creation, and to hear my own name called out (albeit pronounced incorrectly) was a true joy. :)

How One Person Unknowingly Destroyed our delayed_job queue

2010-10-18T14:55:00.001-05:00

this post is one for the "aren't you glad you found this here on my blog due to some furious googling rather than trying to dig into it yourself" category.

We use delayed_job for our background queue, and it's great. You can do this really cool thing where you call "send_later" on any object, with the method you want eventually called, and it will queue it up for later execution. Very lightweight.

This is what we do for our reporting features. A user selects a date range and which report they want to run, and it gets offloaded to the background queue, sent to them later as an email link to an S3 file.

Today, suddenly things aren't working. Reports aren't coming through, in fact the whole queue has frozen, it's backed up to 85 jobs, and when we look at "top" on one of our EC2 instances, we can see the ruby processed dying as quickly as they spool up. Shit.

Frantically we search the logs. Nothing. Configuration? Still the same, and looks good. We try to manually run a few jobs, no issues. Finally I try spooling up a worker in process and telling it to work off the queue, and I see this:

ArgumentError: argument out of range
 from /usr/lib/ruby/1.8/yaml.rb:133:in `utc'
 from /usr/lib/ruby/1.8/yaml.rb:133:in `node_import'
 from /usr/lib/ruby/1.8/yaml.rb:133:in `load'
 from /usr/lib/ruby/1.8/yaml.rb:133:in `load'

From YAML? Why?!

Fortunately, we got some help from our friends at EngineYard (we've always had top notch support from them) who pointed out this little beauty on the ruby bug list. A bad date could cause YAML to fail hard, and this would be during the "deserialization" of our jobs (when the process loads them out of the database to decide which one to lock and run), so it would effectively hit any worker that came across that job, killing the process, effectively destroying our job queue by halting it in it's path. Upon investigation, we found that someone was indeed trying to run a report from 08/01/2010 to 10/01/12010. Dammit!

Deleting that job allowed our queue to get back up and start processing again (and since we use AppCloud, we were able to spool up another utility instance to work off the now VERY long backlog of jobs). Nevertheless, this now means we need to sanitize all of our report inputs, because it's not enough to be a VALID date (which technically that is, although far in the future), it also has to be within a reasonable distance from now (at least until we're using Ruby 1.9.2, in which this is fixed).

RFID in Ruby

2010-10-18T08:24:00.000-05:00

Anyone who's been reading my blog recently know that I've been playing with some RFID pieces for a local project. The project itself isn't important, though I may go into that in another post. For now, what I want to talk about is the gem I've written to help you get a project up and running with active RFID technology.

It was actually kind of fun to slog through the process of pulling off the input stream from this devices serial port, character by character, and try to put together something useful; However, I wouldn't want to do it again. Enter tag-it, my newest micro-gem hosted on rubygems.org.

The problem was this: I bought this RFID receiver, which works great. It's got a built in serial-to-usb adapter, so as long as you have the right driver installed, you can just plug it into a USB port and it will report tag names and their Relative Signal Strength as they come into range (see my last post for details). Then they just continue repeating all the in range tags every 2.5 seconds until they go out of range again.

That's cool, but it needs to be filtered. If you have a webservice tracking which tags are near which receivers, it's not performance friendly to just keep pinging the server over and over and over again with the same information (this tag is still here, this tag is still here, etc). In my case, I really only care about a tag "arriving" (coming into range) and "departing" (moving out of range). Thus, I built tag-it as a gem that sets up a class monitoring your serial port, and dispatching events as it decides they occur.

Example:


require "tag_it"
port = SerialPort.new("/dev/tty.yourport",{:baud=>9600})
tracker = TagIt::TagTracker.new(port)
watcher = YourCustomWatcher.new
tracher.add_observer(watcher)
tracker.start!

tag-it uses the standard observer class to dispatch events through an "update" method defined on the watcher, and sends three parameters (tag_name, RSSI, and event).

So, your watcher should be built like this:


class YourCustomWatcher
  def update(tag,strength,event)
    if event == :tag_arrived
      # do something because a tag has come into range
    elsif event == :tag_departed
      # do something because a tag has gone out of range
    end
  end
end

Thus far, that's it. Couldn't be simpler (well, maybe it could, but the gem is opensourced on github, so if you COULD make it simpler, send me a patch!).

So far this thing only interacts with this receiver (the one linked to above), but as I procure other sources of hardware, I hope to make this a pretty general library able to interact with whatever other input formats are available out there from other manufacturers devices.

Happy tagging!

Real World Integration

2010-10-17T16:18:00.001-05:00

As part of a new project, I'm playing with some RFID technology, and I decided to document my setup process here so I could duplicate it later if I needed to do it again on another computer. The idea? use active RFID tags to track people as they come and go from a building. Here are the steps for getting the tag processing up and running on my macbook pro:

0) buy RFID receiver and tags from http://cliste.sailwhatcom.com/

1) Download Mac OS X driver for the serial to usb adapter built into the receiver
http://www.prolific.com.tw/eng/downloads.asp?ID=31. run the downloaded pkg file from the unzipped DMG. (will restart your computer)

2) make sure the receiver is sending data. Plug it into a USB port, and you can use (on Mac OS X) "screen /dev/tty.usbserial 9600" to watch the port's data come across your screen. (BTW, I was only able to receive telemetry from my RFID receiver from the USB port on the left side of my Macbook Pro's body. When I hooked into the USB port to the right of the keyboard, I got nothing. No idea why, just something to play with if you have trouble here).

3) Make sure you are using ruby 1.8.7, 1.9.x does not appear to work for this next gem

4) install the ruby-serialport gem with "gem install ruby-serialport"

5) write some code to read from the port, something like this:


require 'rubygems'
require 'bundler/setup'
require 'serialport'

port = SerialPort.new("/dev/tty.usbserial",:baud=>9600,:data_bits=>8,:stop_bits=>1)

count = 0
while count < 1000
  printf("%c",port.getc)
  count += 1
end

port.close

There's a simple program that just reads the tag names as they come in range, along with their relative signal strengths, and prints them to the console one character at a time. Output when I have 2 Active RFID 20 meter transmitters sitting on the table next to the receiver looks something like this:

1nri85 1nwP79 1nri85 1nwP79 1nri85 1nwP79 1nri85 1nwP79 1nri86 1nwP79 1nri85 1nwP79 1nri85 1nwP79 1nri85 1nwP79 1nri85 1nwP79 1nri85 1nwP78 1nri85 1nwP79 1nri85 1nwP79 1nri85 1nwP79 1nri85 1nwP78 1nri85 1nwP78 1nri85 1nwP79 1nri84 1nwP78 1nri84 1nwP79 1nri84 1nwP79 1nri85 1nwP79 1nri86 1nwP79 1nri85 1nwP79 1nri85 1nwP79 1nri86 1nwP79 1nri86 1nwP79 1nri85 1nwP80 1nri86 1nwP80 1nri87 1nwP80 1nri86 1nwP80 1nri85 1nwP79 1nri85 1nwP79 1nri85 1nwP81 1nri85 1nwP79 1nri85 1nwP79 1nri85 1nwP79 1nri86 1nwP79 1nri86 1nwP79 1nri85 1nwP79 1nri85 1nwP79 1nri85 1nwP80 1nri87 1nwP79 1nri88 1nwP79 1nri85 1nwP80 1nri85 1nwP80 1nri85 1nwP79 1nri86 1nwP80 1nri86 1nwP80 1nri86 1nwP78 1nri86 1nwP79 1nri85 1nwP79 1nri85 1nwP81 1nri85 1nwP79 1nri85 1nwP79 1nri86 1nwP78 1nri85 1nwP78 1nri85 1nwP78 1nri86 1nwP79 1nri85 1nwP78 1nri85 1nwP78 1nri85 1nwP78 1nri85 1nwP78 1nri85 1nwP78 1nri85 1nwP78 1nri86 1nwP79 1nri85

Note: BE SURE TO CLOSE YOUR PORT! On a couple occasions I've terminated the program without closing the port object, it seems to leave it open and any future attempts to connect with that port get bounced with a "resource busy" message. Only a restart has solved that problem for me.

you will_paginate in FBML!

2010-10-15T15:51:00.000-05:00

I hate writing facebook applications. I don't know anyone who enjoys it.

Nevertheless, it's a necessity in todays contracting marketplace, and sometimes we just have to live with the fact that we're going to write facebook apps.

One of the big problems is that all these super useful gems and plugins we use to speed up our development when using ruby on rails (including elements of the rails stack itself) don't really work inside the facebook application paradigm (especially if you're using FBML, which I don't want to, but one of the requirements for this app is a profile tab, and right now there are no other options).

Today, I was saddened when my favorite pagination tool, will_paginate, was rendered useless by the fact that when dealing with FBML, any links have to work according to the facebook routing scheme rather than rails's defaults. It makes sense why, but by default will_paginate was generating URLs for my pagination links that pointed to my routes on the server, which are not the same as my routes within the facebook application (maybe I should have done some better configuration up front. Doh!).

Normally I would start googling like crazy for a solution, but I read a very persuasive blog post yesterday by John Nunemaker on just reading the code on gems you want to learn about rather than trying to google up their interface. So (determined to at least try following the advice of an industry icon such as John), I installed gemedit, and opened up the will_paginate source for the 3.0 prerelease, and found that I could actually very easily customize how will_paginate's links get generated.

In will_paginate, you can specify when you call your link-generating-view-helper just what class you want to render the links. The default is WillPaginate::ViewHelpers::LinkRenderer, which does a whole lot of nifty work with boxing up your pagination situation into a nice set of links (and is very well written for readability, in my opinion). In my case, I just wanted to change how the urls were built, so I created a new class, subclassed the LinkRenderer, overrode just one method (the one that builds a link, cleverly named "link"), and reimplemented it to do the routing the way I needed it. Then, in my view, I specified my custom renderer as the class to use for generating links, and presto, my favorite pagination gem is running again even under the arduous unpleasentries of working from within facebook. See the code below:


class FacebookLinkRenderer < 
  WillPaginate::ViewHelpers::LinkRenderer

  def link(text, target, attributes = {})
    if target.is_a? Fixnum
      attributes[:rel] = rel_value(target)
      target = url(target)
    end
    target = target.gsub "/what_is_wrong","/what_is_right"
    attributes[:href] = target
    tag(:a, text, attributes)
  end

end

And in the view:


#...haml haml haml....

=will_paginate(@items,:renderer=>"FacebookLinkRenderer")

#...haml haml haml....

Nice!

Classic ASP on a Mac

2010-10-13T19:07:00.001-05:00

Although I am exclusively working on my own business most of the time, I do occasionally do some side contract work if it's to help out a friend or local organization. Usually this involves doing a small web-application or some sort of data processing, but this week I got a unique request.

My local fire department is interested in redoing their web presence entirely. I'm all game to help with that, but they also need some emergency help because their current application is flat out broken. This means actually diving into their current code, and stabilizing it before we even talk about moving to a more current platform (cough...Ruby on Rails...).

Having been part of these rescue missions before, I know that the problems are usually deeper than the description: "there's just a few things that need tweaking and fixing, no big deal really". In anticipation of this, I've already taken the code I copied off of their production server (yes, some organizations still do this), put it into a git repo, and checked it out to 2 separate computers.

The further complication? The entire app is in classic ASP. And all three of my computers are Macs.

But I'm not the last person who will ever have this problem, so we're going to follow along as I get this dinosaur up and running here on my OS X 10.6 Macbook Pro.

1) Download and Install the Mono Framework

Mono is basically an open source implementation of the .NET Framework. If you want to know more than that, checkout mono-project.com. For now, just know that you need this if you want to run an ASP application on your Mac, and you can download the latest version here. The dmg file has a package installer inside of it, and it should run just fine with the default configuration.

2) Download and Install the Mono CSDK

This is available from the same page, and is actually the link just to the RIGHT of the link for downloading the framework itself. I'd be lying if I said I knew what this was, but many people have said in the forums that it's necessary. :)

3) Download MonoDevelop and copy .app file to the Applications folder

MonoDevelop is a development environment for the .NET framework that you can run on a mac.

4) Forget everything about steps 1-3...

...when you realize that Mono isn't going to be much help running Classic ASP, which is not on the .NET framework.

5) find Apache::ASP

Sweet! Execute your classic ASP in Perl with http://www.apache-asp.org/
6) Make sure Apache is installed with mod_perl

If you're on a Mac, it's already there. Just add this to your httpd.conf:

LoadModule perl_module libexec/apache2/mod_perl.so

and add a perl.conf file to apache's "other" directory to allow the execution of perl scripts:


<ifmodule perl_module="">
AddHandler cgi-script .pl
<ifmodule dir_module="">
DirectoryIndex index.pl index.html
</ifmodule>
</ifmodule>

7) download and install the latest Apache::ASP

tar file can be found here. I used 2.6.1.

First, edit your httpd.conf file again:


<directory $document_root="" asp="" eg="">
  Options FollowSymLinks 
  AllowOverride All
</directory>

Then, in the root directory of the Apache::ASP download, run the following:

perl Makefile.PL

Which will tell you if you're missing any modules. I had to install MLDBM::Sync.
From there you can do your make process like any other:

make
make test
make install

8) test to make sure you can run an ASP script

With these steps in place, you should now at least be able to debug your inherited Classic ASP application on your Mac. Now, hurry up and rewrite it on a better platform!

Refactoring in Ruby with Blocks

2010-10-13T12:54:00.000-05:00

The block can be a mysterious construct for the aspiring rubyist. Sure, they're probably comfortable using it to iterate over a collection, that's common. But when would I ever write my own? Why would this confusing jump in control flow be useful?

Fortunately for you, if you're one who would ask that question, I solved a refactoring problem just today by using a block to de-duplicate some code, and I think it makes a good example for showing what kind of situations blocks are good for helping solve. Enjoy!

The problem

I have a class that merges some active record models together when it turns out we have duplicate information in the database. At first glance, this isn't particularly complex. Merge the authoritative attributes over the secondary attributes, combine collections of child records, and they're merged. The problem is, some of the child record collections need to be reduced. For example, if I have two user models that represent the same person, and they both have collections of child records for the last 12 months indicating that they're eligible for something every month (doesn't matter what), then when merged I'll have one user, with 2 eligibility records for every month back to a year. That's not what I really want, I want to reduce those child collections based on criteria unique to each child collection so that we only have one record representing each discrete period of time. So, to do this for two seperate chile collections, I started out by writing (after the tests) some code like this:


 removed_scripts = []
 user.prescriptions.each do |script|
  if removed_scripts.index(script.id).nil?
   other_scripts = user.prescriptions.find(:all,
              :conditions=>["id != #{script.id} and 
                     start_date = ? and 
                     expiration_date = ?",
                 script.start_date,script.expiration_date])
   other_scripts.each do |os|
    removed_scripts << os.id
    os.destroy
   end
  end
 end
    
 removed_eligibility = []
 user.eligibility_records.each do |rec|
  if removed_eligibility.index(rec.id).nil?
   other_elig = user.eligibility_records.find(:all,
             :conditions=>["id != #{rec.id} and 
               year = ? and month = ?",
              ser.year,ser.month])
   other_elig.each do |oe|
    removed_eligibility << oe.id
    oe.destroy
   end
  end
 end

This was functional, but obviously I have two blocks of code that are more similar than different. How can I combine them into one function that can handle both cases?

The Solution

What makes this situation tough is that one of the things that is different between the two chunks of code above is within an iteration over a set of child records, and is different as it's applied to each record (that is, the conditions array for finding other records to destroy). You can't pass in static conditions, because the conditions are dependent on the particular child record being used at that time for data binding. What you need to do is pass a function that can take that child record, and give you your conditions. Hence, my refactor:


def reduce_recs(child_class,conditions)
 removed_recs = []
 child_class.find(:all,:conditions=>conditions).each do |cld|
  if removed_recs.index(cld.id).nil?
   other_recs = child_class.find(:all,:conditions=>yield(cld))
   other_recs.each do |oth_rec|
    removed_recs << oth_rec.id
    oth_rec.destroy
   end
  end
 end
end

By putting in the "yield" right where the main difference in implementation was, I can submit a different function for each child collection to do the necessary matching. Thus, my new reduce_recs function can be used like this:


reduce_recs(Prescription,"user_id = #{user.id}") do |child|
 ["id != #{child.id} and start_date = ? and 
   expiration_date = ?",
   child.start_date,child.expiration_date]
end

reduce_recs(EligibilityRecord,"user_id = #{user.id}") do |child|
 ["id != #{child.id} and year = ? and month = ?",
             child.year,child.month]
end

Now, are there some other things that could be done to make this code more expressive? Absolutely, and maybe I'll look at those in another post, but the lesson you can take away today is: when you have a duplicated code chunk that only really differs at some point within nested iteration, you might consider using a block to abstract the functionality into a single function that yields out right at that critical point, to allow the calling code to specify what needs to be done for that small difference.

Or, to refactor my last sentence: Blocks are good for making a lightweight template pattern.

Cheers,

~Ethan

Websolr: make your application do a REAL man's search

2010-10-11T16:11:00.003-05:00

Rails 3 and all the various updates that come with it has been a really great experience for me so far. I'm fond of ActiveRelation, I like the routing updates, and I love the fact that Heroku already supports it for deployment (and has for some time).

One thing that's not great when using just the baked in tools, though, is search. Most applications use some kind of searching, but unfortunately many (including a few of my own) do something like this (yes, this is an excerpt from one of my own codebases, I'm sorry, it won't happen again):


class User < ActiveRecord::Base
  named_scope :matching_search_term,lambda{|search_term| 
    {:conditions=>["users.id = ? OR 
       users.last_name LIKE '#{search_term}%' OR 
       users.alt_id LIKE '#{search_term}%' OR 
       users.email LIKE '#{search_term}%'",search_term]}}
end

I shouldn't have to tell you why this sucks. But I will. Firstly, it's really only grabbing the front of the string on each field. If I'm searching for "Vizitei" then I'll find anybody with the last name "Vizitei", but not anyone who has somehow ended up with " Vizitei" in the database (which would never happen because you sanitize your inputs, right?). Additionally, those fields all have to be indexed in the database, or there's going to be a problem. Using OR means that the results of the first part of the query don't actually narrow down what has to be compared in the rest of the query. That is, even though I might have found a few users with the last name "stevens", when the query gets to the "email" part it still has to check the whole table for emails that match "stevens". Third, any long text fields (like "description" or "bio") are going to present a performance problem if you're looking for matching a word anywhere in the body of that text.

What's great, though, is that the problem of integrating the kind of search that would easily cover a situation like this has already been solved, and pretty simply if you're using Heroku for your deployment platform. Today I implemented this kind of fulltext search in about 10 minutes by following the instructions provided here for the "websolr" addon, adding the sunspot_rails gem to my app, and changing my user class to look like this:


class User < ActiveRecord::Base
  searchable do
    string  :first_name
    string  :last_name
    string  :alt_id
    string  :email
    text    :bio
  end
end

Then I just ran

rake sunspot:reindex

and we're done.

Why is this better? Well, for one I can now just call #search on my model class to do a loose search on all those fields that are indexed by my new solr server. For another, there are a ton of cool things you can do with solr that I don't want to go into now, but that you can find here on the solr welcome page. For the impatient: it's faster. Capiche?

I'm a little stunned that this was so easy, but in todays world of cloud deployments and pluggable architecture, tasks like this are stupid-simple. Hooray for finishing early and going home for a beer!

Test helpers for collections

2010-07-13T09:46:00.001-05:00

Something that comes up often in a rails application is testing a new scope into existance on a model object. Often, I pull a collection from the scope, and want to check to make sure that the collection contains an object that I created ahead of time. The gist linked to below are a couple of naive helper methods I use for this, and you're welcome to them if you like them. Paste the "assert_contains" and "assert_does_not_contain" methods into your test_helper.rb and you're good to go.

assert_contains helpers

Panic and creativity

2010-07-02T13:47:00.000-05:00

Nothing sparks ones creativity like an emergency. I know this intimately from my experience as a firefighter in my parallel life, but I encountered a similar situation today when I accidentally axed some data from my production database that I REALLY shouldn't have. I was playing around in the command line in production (always a brilliant idea) trying to find some elusive data to get rid of, and my practiced fingers typed ".each{|x| x.destroy}" on the end of a query that was bringing back data I really didn't want to get rid off. Suddenly my console is filling up with text like this:

[#<ProviderSchedule id: 11231, start_date: "2009-07-01", end_date: "2009-10-01", start_time: "2000-01-01 07:30:00", end_time: "2000-01-01 15:30:00", valid_flag: true, monday_included_flag: true, tuesday_included_flag: true, wednesday_included_flag: true, thursday_included_flag: true, friday_included_flag: true, saturday_included_flag: false, sunday_included_flag: false, created_at: "2009-09-09 04:34:37", updated_at: "2009-09-09 04:34:37", schedulable_id: 12092, schedulable_type: "Provider">, #<ProviderSchedule id: 39252, start_date: "2009-10-01", end_date: "2009-12-31", start_time: "2000-01-01 07:30:00", end_time: "2000-01-01 15:30:00", valid_flag: true, monday_included_flag: true, tuesday_included_flag: true, wednesday_included_flag: true, thursday_included_flag: true, friday_included_flag: true, saturday_included_flag: false, sunday_included_flag: false, created_at: "2009-09-18 17:04:44", updated_at: "2009-09-18 17:04:44", schedulable_id: 12092, schedulable_type: "Provider">...

and I'm starting to feel a little sick. First lesson learned; for God's sake, don't screw around in the production database.

However, what was done was done, and I needed a way to restore the data IMMEDIATELY. Coding through a border-line panic, I quickly produced the code linked to at the following gist:

GIST!

It's quick and dirty, but it saved my ass today. If you're running Rails, and you did something as bone-headed as accidentally destroying a ton of data from the command line, this may save your life. Quickly, before you close your terminal, COPY THE OUTPUT STRING FROM YOUR DELETE OPERATION AND SAVE IT TO YOUR COMPUTER RIGHT NOW!!!!

Now, open up your command line terminal again in production (for the last time ever), copy the gist in to define the method, put the data dump inside a %Q{} stored in a local variable, and run it through the method like so along with the class object (the object which was destroyed):

=>  string = %Q{...huge long output string...}
    restore_from_destroyed_string(string,ClassName)

you'll see some output like this:

creating object 11247...true
creating object 39261...true
creating object 46838...true
creating object 72731...true
creating object 88175...true
creating object 11248...true
creating object 39262...true
creating object 46839...true

Now, wipe your forehead and pound a shot of scotch, you just averted disaster.

PS: NEVER do that again.

patch release of quarter_time (0.3.2)

2010-07-02T11:03:00.000-05:00

This is just a minor release of my mini-gem, quarter_time. Up until now, quarter objects have always been created using integer arguments:

quarter = Quarter.new(2010,3)

However, in cucumber tests and in rails controllers, I often have string versions of these arguments available. After casting in a couple places, I decided to make it part of the gem, so as of quarter_time 0.3.2, you can do this:

quarter = Quarter.new("2010","3")

Simple change, but useful.

using regex groups for ruby's gsub

2010-06-30T12:22:00.000-05:00

I won't spend long on this. If you've done your due diligence and carefully read up on the way regex matching works in ruby, you wouldn't be on this page. If you were like me and figured that you could probably just nail it the first time without looking anything up, you might have written something like this:

cleaned_string = "7.30 AM".gsub(/(\d)\.(\d)/,"#{$1}:#{$2}")

This is about as completely wrong as you can get while still looking somewhat right (as I found out today). You see, the global variables like "$n" do indeed store the matched groups from regex operations....AFTER they've executed. At the time when these parameters to the gsub method are evaluated, these variables are still nil. Thus the string you end up with is something like ":0 AM".

Correct syntax for including matched groups from regex expressions?

cleaned_string = "7.30 AM".gsub(/(\d)\.(\d)/,'\1:\2')

this way it's evaluated later, after the method has been called. Note the single tick marks, as doubles will cause the slash-number combinations to be interpreted.

quarter_time 0.3.1 released

2010-06-22T10:30:00.002-05:00

A small but useful addition to the quarter_time gem I released a couple weeks ago. Normally the scope added to models that include the QuarterTime::QuarterDriven module would be used with other Quarter objects:

class SomeModel < ActiveRecord::Base
  include QuarterTime::QuarterDriven
end

q = Quarter.new(2010,2)
model = SomeModel.for_quarter(q).first

However, in cases where I didn't already have a quarter object built or coming from another model, it seemed like overkill to build the object just to pass a couple parameters. So, here's an additional (equivalent) syntax you can use to select models by quarter:

class SomeModel < ActiveRecord::Base
  include QuarterTime::QuarterDriven
end

model = SomeModel.for_quarter(2010,2)

Off topic but on fire

2010-06-19T16:09:00.000-05:00

Today I wanted to spend some time working on my Lingq.com client gem that I've been working on this weekend. It's a fun litte project, and if you're into learning another language I recommend you go check out lingq.com, it's pretty cool.

But that didn't happen. No, instead today I spent a few hours trying to figure out how I'd been fraudulently charged for $325.50 over the last 6 months. I wish this stuff didn't happen very often, but the guys who got me are pros, and if you're someone who has a credit card (and you are), you should immediately go check your statements for anything that looks like one of these:

MVQ*PRIVACYM
MVQ*CLUBSAVE
MVQ*SHOPESSPLUS
MVQ*SHOPPINGESSENTIALS+
MVQ*BIZMAX
MVQ*BUSINESSMAX
MVQ*DLMAX
MVQ*DEALMAX
MVQ*TWENTY4PROPLUS
MVQ*24PROTECTPLUS
MVQ*TRAVELMEMBER
MVQ*TODAYSESCAPES
MVQ*PRIVACYMID
MVQ*PMIDENTITY
MVQ*HOMERWRDS
MVQ*ATHOMEREWARD
MVQ*VALMAX
MVQ*ValueMax
MVQ*PASSPRT2FUN
MVQ*PASSPORTTOFUN

I'm sure you see the pattern. There are tons more, but who has time to copy and paste all that mess 300 times?

These guys run a ton of billbord sites like this:

http://www.mvq-dlmax.com/

Clue one that this is a problem is how every site of theirs is basically something that says "want to know why this mysterious charge showed up on your credit card statement?". Check out this excerpt from http://www.mvq-todaysescapes.com/

You're probably here because you were searching the billing descriptor MVQ TODAYSESCAPES after finding it on your credit or debit card statement.
The billing descriptor MVQ*TODAYSESCAPES that you saw on your credit or debit card statement is used to indicate a fee for enrolling in Today's EscapesSM, an online membership discount program that offers consumers much-desired discounts on travel plans, dining out, home entertainment and more.

Oh my freaking God. If I had signed up for a membership with you, why would I be searching by the BILLING descriptor for your service? Why in the name of reason would the front page of your website be dedicated to explaining why I shouldn't ask my credit card's fraud department to bring the attorney general down on your slimy ass?

Yep, I called my credit card company, and what did they say? "Yeah, we see this a lot". Wow. Now, I'm glad they were able to help me, they issued me a new card cancelled my old one and credited the charges, but I have to wonder about how this situation reflects on the world of software developers world wide. Obviously someone has shown they can make a quick buck by obtaining credit card numbers and charging them monthly because most people won't investigate charges smaller than $50. And even then, once someone cancels their "account" with these criminals, they usually won't go so far as to file fraud, because it's time consuming. So they have a system that does this automatically, and a phone answering service that lets you cancel your account (it's pretty much the only option on the phone menu, minus some bullshit "explanation of your benefits" speech). And the money rolls in every month.

This probably isn't very coherent, because I'll admit, I'm furious right now, but these people will be allowed to operate only as long as we don't report them because it's too much administrative overhead. Now, I did my part today. I filed a fraudulent charge claim for every transaction they hit me with since they got my credit card number back in september. Today, right now, go to your credit card account search for MVQ. If you're being scammed too, please report it to your credit card company's fraud department so that we can get these lowlifes shut down.

quarter_time: progeny of RailsConf

2010-06-10T08:46:00.001-05:00

Well, it's been a great week so far at RailsConf!

There have been at least 47 times that I've looked at someone and thought "hey, I know that guy!" recognizing them from their avatars on github.com or their names on twitter.

Dr Nic Williams, Obie Fernandez, Greg Pollack, I could go on and on.

Most importantly, I've been overwhelmingly inspired to dive back into self-improvement in my coding ability. Dr. Nic's talk especially galvanized me towards engaging more with the gems and libraries I'm taking advantage of, and contributing back my own. In the spirit of my newly found ambition, I've released a mini-gem today that I use in a couple different applications my startup runs.

quarter_time, here linked to my github profile, is a gem for dealing with the problem of quarter-based business cycles. For instance, we have a medicaid-billing application that processes claims for every quarter of the year (Quarter 1 2010, Quarter 2 2010, etc). The claim models, and many other pieces of supporting data, are stored according to what year and quarter they belong to. Of course, the normal ruby data doesn't have any assistance for moving between quarters, and there are certain things that are common between models that are governed by a quarter (for instance, a scope to find objects by the year/quarter they are part of, a way to find the date range the model covers (start_date and end_date), and so on. This early-stage gem (well-tested, but not fully featured yet) is here to help solve that problem.

Sometimes the best part of going to these conferences is the energy you get from being around so many people who are interested in the same stuff. Thanks RailsConf, and I'll see you next year.

~Ethan

A designers goldmine for software developers

2010-05-19T09:47:00.000-05:00

If you're like me, you're mainly into code. You like writing systems to solve problems that you have and you can do just enough UI design to be functional. You'd like for your projects to be prettier, but spending a ton of time on tweaking the positioning and color of elements is not your idea of a good day.

Helpful Hint: Theme Forest

Hundreds of nicely layed out UI skins that you can buy pretty cheap and use for your pet project. I don't need to say much more about it, just go check it out (This is not an affiliate link, I have no finiancial reason to want you to go buy a template from these guys, I just have bought a couple in the last few weeks and am plenty happy with them).

Cucumber and BackgroundJob

2010-05-11T12:49:00.001-05:00

The problem: you want to have some nice integration tests that cover situations where jobs get submitted to the background queue, but haven't gotten around to writing it yet or haven't wanted to deal with it.

The solution: we at my office have suffered for you. If you are a BackgroundJob user (and someone who uses Cucumber for their acceptance testing), this steps file (and the associated BJ Model) should help you with your workflow:

GIST!

In short, it uses active record to make sure all your jobs are far enough in the past to get run (a problem with BJ and timezones), executes the queue when you want it to, and has a step for making sure that the job you wanted to submit actually exists.

As an aside, BJ takes a while to startup before working off the queue, so I wouldn't use this all the time. There is probably a good way to speed it up that I haven't thought of, but the idiot-proof way would be to switch to DelayedJob (something we're planning on doing in the next couple months).

Enjoy!

Heroku, you are amazing

2010-05-10T12:44:00.002-05:00

I've been spending my weekends trying to keep up with the ruby community. Building toy projects, testing new gems out, and I've found a lot of cool stuff out there.

But what has really struck me as more of a game changer than anything else is Heroku.

You remember when Rails first came out? I remember experiencing it as an epiphany, of sorts. I had spent years writing java apps, configuring every application down to the most minute details. Along come this Rails framework and says "I know what's usually best, and that's what I do by default. My conventions are king.", and that changed web development for me. I could get features done twice as fast because all the config I usually did up front for each piece was already done for me. Yeah, that meant a loss in flexibility, but I didn't mind much. If I have to do something special, I'll cross that bridge when I come to it.

The same sort of feeling came over me this weekend when I tried deploying my first heroku application. I just ran a simple "heroku create" command from my home directory of my app, and it was deployed in seconds, ready to be updated by a simple git push, and already live and kicking. Once again, there's this new guy on the block who says "This is what you usually need, so this is what we always do", and it works so beautifully. Need email? Push an add-on, it's already preconfigured. Performance monitoring? memcached? error notification? it's push-button simple.

And the best part? If you want to spike out an application that you aren't sure you want to build all the way out yet, you can have it deployed and running on their simplest and smallest configuration for FREE! That's right, just try it out, see if it's viable, and when you're ready to bring the traffic you just reconfigure it to spool up more resources. Everything is billed by a running meter (I think around 5 cents an hour per process), and it just works.

To the creators of Heroku, I salute you, you've made Rails deployment fun.

Bug with BigDecimal in Ruby 1.8.7 (patchlevel 173)

2010-04-29T16:35:00.000-05:00

got out my brand new iMac today, and when I checked out my project, I had a bunch of tests failing. I thought I might have checked out an old version, but no everything was current and passing on my old dev machine. It wasn't in in my code. It wasn't even in rails. It was a problem with Ruby 1.8.7 (patchlevel 173). After hours of digging, behold the following console output in irb:


>> require 'bigdecimal'
=> true
>> BigDecimal.new(103123.98.to_s).to_f
=> 103123.98
>> BigDecimal.new(100123.98.to_s).to_f
=> 101239.8

Strangely enough, if you have a "0" as the fourth digit to the left from the decimal point when you build a big decimal object, it cuts it out. I have no idea why this is. I don't have time to investigate it. But I do know it doesn't happen with ruby 1.8.6, which is what I'm running in production, so for the moment it's not a real problem for me (except I had to change all my tests that had a 0 in that decimal place temporarily while I sort out bringing an older version of ruby onto my machine).

*sigh*

RMagick on OS X Snow Leopard

2010-04-29T14:52:00.000-05:00

Just got a new iMac today, so I'm trying to get my project up and running on it. Things were going well until I got to RMagick. I always forget what a pain in the ass that gem is. I was lucky this time, after a little blog surfing, to find a great post by "hocuspokus" that provides some excellent help in the form of a shell script you can run to make it all happen (NOTE: you must have XCode developer tools already installed, it's on your Mac OS X Installation DVD). In the interest of giving credit to the right person, I'm not going to link directly to that script, but am instead just going to give a little link-love to the blog of the guy who made my day so much better when I found his website:

CLICK HERE TO GO TO THE HOCUSPOKUS BLOG SO THAT YOU CAN STOP TROLLING AROUND THE INTERTUBES TRYING TO GET THAT DAMN GEM INSTALLED ON YOUR NEW MAC!

Getting my feet wet with Clojure

2010-04-25T22:53:00.000-05:00

For the next few weeks, I'll be participating in the "Clojure 101" class at Ruby Learning. I don't have any plans to switch over from Rails as my primary platform in the near future, but I have a soft spot for the ol' JVM and an incomplete knowledge of functional languages, so I'm excited to have something of a diversion.

If you're interested in benefiting from my experience, I'm putting my notes/examples/exercises/experiments on my Github Account, and I'll continue to update it as a convenient place to store all my new knowledge (god knows my head's out of room).

Rails has_many scoping and database indexes

2010-04-23T15:06:00.000-05:00

If you are a good rails coder, than you probably form your associations like this:


class Parent < ActiveRecord::Base
  has_many :children
end

class Child < ActiveRecord::Base
  belongs_to :parent
end

Which is correct. Then, to keep things looking pretty, when you want a scoped subset of a given parent's children, you probably write something like this



class Parent < ActiveRecord::Base
  has_many :children
end

class Child < ActiveRecord::Base
  belongs_to :parent
  named_scope :who_can_drive,:conditions=>"age >= 16"
end

parent = Parent.first
children = parent.children.who_can_drive

Which is well and good. Except when you care about performance. You see, I did something like this in my production application. Both fields on the child model are indexed (the foreign key, and the field used for searching by), but guess which one gets used first? If you know your named scopes, you know that in practice the named scope that is APPLIED LAST in the chain is used first in the query, and that's the one MySQL decided to use as the primary index.

Now if a parent has 2 or 3 children, but it's in a table of 15,000,000 kids between the ages of 14 and 21, which index do you think will produce the smaller set to scan? The one that narrows it down to only the 2 or 3 children of that parent and then checks their age, or the one that narrows it down to the 7 million children over the age of 16 and then checks their parent?

So to keep my server from crashing and burning, I've started writing my code like this:



class Parent < ActiveRecord::Base
  has_many :children
end

class Child < ActiveRecord::Base
  belongs_to :parent
  named_scope :who_can_drive,:conditions=>"age >= 16"
  named_scope :belonging_to,lambda{|parent| {:conditions=>{:parent_id=>parent.id}}}
end

parent = Parent.first
children = Child.who_can_drive.belonging_to(parent)

I don't like it, but it sure improved my scores on my newrelic monitoring graphs. Anybody have a more elegant solution to this problem?

Get the gist of data loading (csv,xls,xlsx,ods)

2010-03-12T22:31:00.001-06:00

Common problem: you need to load some spreadsheet data into your Rails application.

Common solution: send it to me as a csv file, I'll take care of it.

Why? Because a csv file is super easy to parse. Using the built-in ruby csv reader, every row comes back as an array as you traverse the file, so it's easy to process one row at a time:

CSV::Reader.parse(file).each do |row|
  #...save some model that this represents...
end

The complication that often comes up is that your customers might not know what a csv file is. So you'll get xls, xlsx, ods, pretty much anything.

You could manually save each file as a csv (kinda frustrating).

You could also write a cludgy class that handles the interfaces for different libraries differently.

Or you could use this gist I'm about to provide you for all your parsing needs, taking the power of the "roo" gem, and the built in "csv" parser, and translating all your file types into the same iterate-able interface that you so love:

GIST!

Now you can just write:

SpreadsheetParser.parse(file) do |row|
  #...pretend its a csv file, every row is just an array
end

Enjoy!

New Mini-Gem in the Wild

2010-03-09T15:19:00.000-06:00

Ok, here's a problem I've had:

I want to deploy my application to a "cloud" setup rather than my current "slice" infrastructure. This is great, but means I have to refactor out many features that used to make use of the local filesystem because I cannot depend on it. I could spawn new servers at any time in a cloud infrastructure, and there's no GFS joining them. For most features, this is no big deal: store your assets on Amazon S3 and get over it. That's what I've done for user uploads, report files, and all manner of other assets.

The problem came with some of my data updates. We have a process at my company where some of our internal users have a page on our application they go to in order to update our data from non-web-enabled sources (mostly through the use of CSV files, since all our data sources seem to be able to generate those). Currently we store those files on the GFS, than kick off a background job passing the file name and let the background job do whatever processing it needs to do on that file.

It's harder to do on the cloud, though, because you can't just store it locally, our utility server instance that's running our background jobs won't be able to get to it since it's a different filesystem entirely. You could put it in the database, and if I were using MongoDB on this project I probably would, but that's not a habit I want to get into with MySQL.

The patchwork solution we're going forward with for now is ditching the file into s3, passing the key to the background task, and re-downloading the file on that end for processing.

In order to make this process a little more palatable, I've quickly built and released a mini-gem called Cumulus CSV. It just wraps a simple interface around storing an uploaded csv file to S3, and iterating over it later. It's available from my github account, or on gemcutter as "cumulus_csv", so if you're struggling with the same problem see if it will help you out!

We're Hiring!

2010-03-08T14:28:00.001-06:00

This is a call to all Rubyists in Columbia, MO; if you're looking for a new (or your first) coding gig, we're interested in talking to you.

Here's the story: we (mostly I) went and built this Rails app, therapylog.com. It lets therapists at public school districts track all of the therapy they do from day to day, in what we think is a pleasant way. No paper, no hassle! We even started a BLOG to let our therapists see how our webapp was growing (you can check it out too to see some sample functionality).

Now the problem: We've grown. We've got a slew of new feature ideas, plus a bit of refactoring work ahead of us, and I can't do it alone.

What I'm looking for is somebody at what an enterprise operation might call the "junior" level. We've got a pretty good code base established, but need someone to step in and keep the gears of progress moving forward. You would be working with me directly, as locally as possible, and we could have you start as soon as you're ready.

Things I'd like:

-at least some experience with Ruby or Python, doesn't have to be professional in nature
-interest in making things better, not just getting the checklist done
-comfortable on a non-windows development environment (*nix, Mac OS X)
-history of making self better
-Not necessarily a design pro, but unwilling to create something ugly.
-good sense of humor (won't help your coding, but should at least help our interaction) :)

Although we'd love to have a 5-7 year rails vet, fact of the matter is, we're talking a $30,000-$40,000+ position ($20/hr if part time), and being a developer myself I know what you're worth when you've been in the business that long, and I can't see you being happy at our current cashflow level.

However, if you're a recent/soon-to-be college grad or a programming hobbyist looking for a career change, this might be for you!

You can reach me directly at ethan.vizitei@gmail.com, can't wait to hear from you!

I've been had!

2010-03-05T10:35:00.001-06:00

Here is a lesson for you that you should take to heart: Trust your mentors, but find out for yourself.

When I first heard about batched finding in ActiveRecord (#find_each adn #find_in_batches), it was a great revelation. My buddy told me how it worked, and I loved it!

According to him, find_in_batches used a "batch_size" hash option, defaulting to 1000, to decide how many records to load into memory at a time, and "find_each" would perform a query for each record one at a time.

Give those options, and wanting to balance my memory conservation against responsible use of database time, I ended up with a lot of blocks like this:


Model.find_in_batches do |models|
  models.each do |model|
    ....
  end
end

It seemed tedious to do a nested block every time, but I didn't want to use the "find_each" option which was going to slam my database with a million single row queries.

Eventually I'd repeated myself enough that I decided I was going to do something about it. I figured I'd create a plugin that would monkeypatch in a method called "find_each_in_batches" that would do the extra block for you, so you could automatically iterate over one model at a time, but still have the finding occur in batches. In order to get the namespacing right, I opened up the source file for ActiveRecord's built in batching methods (batch.rb), and imagine my surprise when I see this:


def find_each(options = {})
  find_in_batches(options) do |records|
    records.each { |record| yield record }
  end
  self
end

What do you know! #find_each actually does exactly what I was planning on making my new plugin do! My buddy had been misinformed, and had in turn misinformed me, and I just took his word for it!

If I had checked out this source code myself in the first place, I would have known this in the first place, and wouldn't have those double iteration blocks littered throughout my code base.

Reading the source code is a good habit anyway, because it introduces you to idioms the pros are using that you may not be familiar with and is a great way to improve your own code. But even just a glance to confirm what i'd been told with the inline documentation would have prevented a world of hurt.

So, while I'm here slapping my forehead, I encourage to you take heed to what I said at the top: Trust your mentors, but find out for yourself.

A New Playground

2010-02-21T22:21:00.000-06:00

Tonight I decided it was time to get my skills up to speed again. With all the new ideas and gems being thrown around in this time of swift evolution in the Rails community, I've felt left in the dust as I continue to toll away on an old project using an old version of rails and a set of older support gems.

So, in an effort to become current once again, I've started a new project (which I'll talk about in another post) utilizing the following:

Rails 2.3.5

Yeah, I know Rails 3 is coming out soon, so why bother coming up to this minor version at all? Well, my previous project was on Rails 2.2.3, and I haven't been getting to play with Rack as much as some of the more current developers have. I think that's one of the major modularity improvements that has pushed rails in the direction it's going with 3.0, and I want on-board.

MongoDB

There's a lot of hype around non-SQL databases right now. As an early user of DB4O, my whole developer career has been littered with occasional forays into the non-relational, and although I have no objection to relation databases per-se, I love the idea of schema-less data. MongoDB does this beautifully, and it's very popular in the ruby community right now in no small part thanks to the MongoMapper gem. Having my data as one hash of properties appeals to my sense of simplicity, and the embedded documents and collections are nifty looking. Time will tell if I will end up feeling like it suits my needs on a larger project.

Heroku

Currently, my large startup project runs on EngineYard. We trust them because their customer service is great and they've done an amazing job of helping us along every step of the way. Their new cloud accounts are awesome (we're going to be switching over to that from our old "slice" architecture soon), and they have a track record a mile long of proven excellence with difficult scaling problems. And they cost what they're worth.

For a smaller project, either of a personal nature or one that's going to be bootstrapped with little revenue to kick it off, it's a little pricy. Heroku, however, is very reasonably priced, and has a slew of posts on the ruby blog network right now that are titled along the lines of "How to deploy a crazy-cool-new-application on Heroku in under 30 minutes". Their development account is free, and I want to see what the buzz is about, so I'll be posting how I find the hosting provider after I've played around a bit.

RSpec

OK, this isn't exactly new, but I just haven't had a project yet where I've been willing to start with a totally new testing library that I'm unfamiliar with, so Test::Unit has been my default. RSpec comes highly recommended, though, and claims to make a nice paradigm shift into the BDD world. Will it make a difference? I don't know, but it's popular enough among rubyists that I'd be silly not to give it a try and see how I feel about the different nature of test presentation.

Delayed Job

I've been a BackgroundJob user ever since my first queue necessitating feature came into existance, and it's never let me down with it's straightforward simplicity. However, I've felt on occasion that a fuller feature set would be desirable for some of my more granular queueing needs. Delayed Job is at the top of the list as far as widespread adoption goes, so if I'm going to jump to a fuller library this is the first place to look.

I love playing with new toys, so the weeks ahead promise to be filled with new and exciting blog material chronicling my exploration of that which is new and popular. Enjoy!

Time for a change

2010-02-17T11:16:00.002-06:00

I've been going as "codeclimber" for a long time with the normal ".blogspot.com" domain, but honestly it's just because it was easier to setup that way and I've been too lazy to go register my freaking name as my domain name. Well, I've finally done it, so from now on "codeclimber.blogspot.com" redirects to "blog.ethanvizitei.com". Watch out, now it's personal!

Making it Happen

2010-02-11T13:41:00.001-06:00

I spend a lot of time trying to make my code good. Maybe too much. I love my codebase like a child. When presented with a problem, my first instinct is to make it somehow fit into what my system already does. Sometimes that's not going to be an easy proposition, though, and in SOME of those cases we need to let the business needs get top priority and just make it happen.

An Example

We have a vendor we ship claims to from time to time, and they use a manual process. Because they are in the public sector, they require a certain format for their claims, which includes a lot of unnecessary Excel formulas because that's what they audit based on (the formulas themselves). Does it matter that we could generate an excel file that has every number they need along the way? No, they must have the FORMULAS in the spreadsheet, or it isn't a valid claim.

Unfortunately for me, our spreadsheet library we use (the "spreadsheet" ruby gem) doesn't have support for formulas yet. So our options were to A) spend some time to work on this gem and add formula support (time we didn't really have). Or B) make the spreadsheet generation a partly manual process on our end as well (Blech, we actually did this for a quarter, generate our numbers and copy them into the claim template one at a time).

This quarter, I went a different route:


Sub Main
  Dim Doc As Object
  Dim InputSheet As Object
  Dim TemplateSheet As Object
 
  SetupVariables(Doc,InputSheet,TemplateSheet,ClaimSheet)
  RowNum = 1
  NameCell = InputSheet.getCellByPosition(2,RowNum)
  Do While NameCell.Type <> com.sun.star.table.CellContentType.EMPTY
    ClaimRow = InputSheet.Rows(RowNum)
    ClaimSheet = Doc.createInstance("com.sun.star.sheet.Spreadsheet")
    Doc.Sheets.insertByName(NameCell.getString(), ClaimSheet)
    CopyTemplateToNewClaim(TemplateSheet,ClaimSheet)
    ProcessClaim(InputSheet,ClaimSheet,1)
    RowNum = RowNum + 1
    NameCell = InputSheet.getCellByPosition(2,RowNum)
  Loop
End Sub

Yep, I used OpenOffice and BASIC and just scripted the damn thing. Ugly? Yes. But it worked, and we don't have to manually calculate our claims. I just generate our spreadsheet that has the claim numbers, and use the macro to copy each row into it's own template sheet in the workbook. It doesn't fit into our rails application at all, but we made it happen, and we're collecting on those claims.

If you're beating your head against the wall in frustration, sometimes it pays to step back and resort to plain old hackery, cause at the end of the day your customers only really care that it works.

Tricky Little find_in_batches (watch your :select clause)

2010-02-04T12:54:00.001-06:00

If you are like me, you have a few background processes that deal with tons of data (reporting, etc). To run these processes, you may use ActiveRecord's "find_in_batches" method which allows you to only pull so many records into memory at a time (a good idea when processing large numbers of records). You may also explicitly use a "select" clause in your AR queries from time to time to only pull in the fields you need for a given purpose. Be wary, as I tripped up over something silly today and you should know about it.

You see, I had something like this:


class SomeModel < ActiveRecord::Base
  named_scope :only_essentials,
              :select=>"some_models.info,some_models.name"
end

class BackgroundProcess
  def run
    SomeModel.only_essentials.
              find_in_batches(:batch_size=>200) do |batch| 
      #....some processing of each record in the batch
    end
  end
end

The problem? It was running way too fast. Over in mere seconds. You might think, "Hey, that's not such a bad problem to have", until you realize that the reason it was running so fast is it was only processing the first 200 records (that is, the first batch).

Why? It's simple if you know how "find_in_batches" works. It uses "order" and "limit" to order your models by primary key and limit the result set. What field did I not include in my named scope? "some_models.id" is correct. Add that field so your reference point is maintained, and everything runs as expected.

Cheers,

~Ethan

Smart quotes and dumb errors

2010-02-01T14:42:00.000-06:00

Wow, this one just sucked. :)

I'll make it quick. We have a model in our rails app that captures large strings and then displays an abbreviated version to the user later. For example:

class Narrative < ActiveRecord::Base
  validates_presence_of :text

  def text_in_brief
    return nil if text.nil?
    (text.size > 20) ? (text[0..19] + "...") : text
  end
end

Hopefully that made sense. We want an ellipses to indicate that this value continues on for some time, so if it's longer than 20 characters, just cut it off and add the three periods.

Enter JSON. We sometimes want to send this value to JSON. And every now and then the to_json method blares this exception:

JSON::GeneratorError: source sequence is illegal/malformed

But only very rarely. What the hell is going on here?

Analyzing the data shows that all the strings that cause this explosion have one thing in common - a smart quote as the 19th or 20th character. I mean the one that is actually represented as "\342\200\235". See where this is going?

By splitting off the string at the character level, it's possible to cut off the smart quote somewhere in the middle, because a smart-quote is actually 3 characters long. This causes an invalid string, which bombs the to_json call. F@(#!

Quick and Dirty solution?

class Narrative < ActiveRecord::Base
  validates_presence_of :text

  before_save :escape_smart_quotes
  
  def escape_smart_quotes
    self.name.gsub! "\342\200\235", '"' 
  end  

  def text_in_brief
    return nil if text.nil?
    (text.size > 20) ? (text[0..19] + "...") : text
  end
end

Maybe when I've cooled off I'll do this better and extract it to be usable in other models. For now, just be glad I figured it out at all. :)

gem Read-through: slim_scrooge

2010-01-27T13:08:00.001-06:00

Ok, new project. I believe it's dangerous to rely on code that you do not understand. As a rails-developer, I have tons of plugins and gems that I do not understand. See the problem?

To rectify this, I'm making it my goal to read through one of my main project's many dependancies each week. Two side benefits:

1) I will probably be better at writing my own open source libraries if I've seen a larger sample of how they're usually constructed.

2) code reading is good for you, but it's tough to find time to just sit down and crack open a library. This will give me a good reason.

So without further ado, today I'm doing a read-through of http://github.com/sdsykes/slim_scrooge, a great ActiveRecord optimizing library that has made a difference in the performance of my current main project. Don't expect anything linear here, I'm just going to record my notes and if you want to use them too you're welcome to them.

Slim Scrooge

The point of the slim scrooge library is to moniter your active record queries, and optimize them so that they only pull back the columns that you end up using in that section of code. Let's find out how it works:

NOTES

1) First thing I noticed. There is a test directory, but no tests. Problem? maybe....

2) Scratch my first note. It appears that SlimScrooge::ActiveRecordTest actually runs the ActiveRecord tests that are included with Rails. I guess this makes sense, as a regression test. Anything that filters activerecord should still pass the activerecord test suite. Still, this definitely means that the code itself is not under test. The gem could do nothing, and the tests would still go green. I'm not here to judge, though. I've written my own share of untested code.

3) first included file in the main library is a C extension called 'callsite_hash'. Looking in the /ext directory of the plugin. My "C" is a little rusty since I've been out of it for 3 years, but I think I get that it's defining the global ruby function "callsite_hash", and mapping it to the c function "rb_f_callsite" in this callsite_hash.c file. I don't know what it does yet, as it's the rb_f_callsite function is a little dense for my limited C skills, but maybe it will make more sense in context. So, moving on.

4) Next inclusion is SlimScrooge::SimpleSet (a subclass of Hash, /lib/slim_scrooge/simple_set.rb). This class stores a set of keys based on a submitted array, all mapped to the value "true". Because of the syntax, each time an element is added, it will only create a new entry if it's not already in the set. So basically it's a set of unique elements with some helper methods to keep operations restricted to only the keys (like a collect method that only runs over the keys array). Knowing what the gem does, at this point I'm guessing this is the structure that column names are stored in so you know which ones were used and which ones weren't after a query. We'll see.

5) Moving on to /lib/slim_scrooge/callsites.rb, which defines the class SlimScrooge::Callsites (no parent class). This class only has static methods, so I guess it's never instantiated. It has a class-level variable called @@callsites, which is a hash. Write access to the hash is synchronized through the uses of a Mutex which is instatiated at the time of class definition as a class-level constant (SlimScrooge::Callsites::CallsitesMutex). Given that I don't know what's being stored here, I don't feel like I can accurately analyze it. Therefore, I'm jumping over to the top-level algorithm in /lib/slim_scrooge/slim_scrooge.rg

6) lib/slim_scrooge/slim_scrooge.rb definately is the meat of the gem. SlimScrooge uses good old alias_method_chain to bring about "find_by_sql_with_slim_scrooge" (defined in the gem) and "find_by_sql_without_slim_scrooge" (the original "find_by_sql" method in ActiveRecord). This is how the gem inserts itself into every activerecord call. In the "find_by_sql_with_slim_scrooge", we see what's being done step by step:

A) if the sql passed in is an array (that is, a custom query directly from a programmer writing Model.find_by_sql("blah")), don't bother. Let it run like normal.
B) if this "callsite" has been seen before, try to optimize it.
C) if it hasn't been seen before, try to monitor it
D) otherwise, let it go (find_by_sql_without_slim_scrooge)

7) So what is a "callsite"? How do you know if you've been here before? Well, apparently that's what the C extension is for "callsite_hash.c". The query is passed into this black-magic-extension which by some occult method creates a unique key for it (called a callsite_key). This is then stored in that class-level hash in the "Callsites" class.

8)There is logic written in here to pass it through unoptimized if the query is not "scroogable", and there are several conditions that meet that. For one, if there's any joining, it won't bother. Also, if it's not a "select" query (that is, it doesn't start with SELECT, include the expected table name, and have a "FROM" in it). [These were limitations I was unaware of before].

9) The monitoring of a query is done by attaching a MonitoredHash to each row in the first query. This hash maintains a reference to the callsite, and can be configured to not monitor certain columns. Anytime a column is accessed that was previously unseen, the callsite is notified.

10) next time the query is run, the callsite has a record of which columns were used and uses "scrooged_sql()" to only produce a select query for those columns.

Well, this was fun. I feel like I've learned a bit about how my site works under the hood, and a little more qualified to comment on the use of this gem in the future. Here are a few things I learned that are not directly about the slim_scrooge gem:

1) The Mutex class can be used to synchronize access to an object.

2) ActiveRecord appears to direct all queries through the "find_by_sql" method. That's the place to hit it if you want to get in some sort of filtering.

3) C extensions for ruby use an "Init_*" method to integrate themselves into the runtime.

Until next time,

~Ethan

Un-Joining your Scopes

2010-01-22T14:04:00.000-06:00

I had a suprising problem today when one of my tests started failing after I had done a little refactoring. You see, I'd had this ActiveRecord class that was doing some reporting (massive data extraction) and it was originally using a pretty ugly SQL statement:


Model.find_in_batches("Massive SQL statement") do |models|
  models.each do |model|
    models_to_compare = Model.scope_with_other_joins
  end
end

Naturally, I wanted to make that SQL statement go away, and use a bundle of named scopes instead. I had good tests wrapping this area already, so I set autotest running and started hacking away:


Model.scope_with_some_joins.find_in_batches do |models|
  models.each do |mdl|
    other_comparisons = Model.scope_with_other_joins
  end
end

Note that both queries (line 1 and line 3) have similar joins involved. Now, my tests started failing on line 3 -- I get a runtime error showing me that for some reason when running the second query it's maintaining the join scope from the outer query, giving me an "ambigious column" error because there is one table that is joined in from both queries. Now, this "some reason" is really just that this is the way it's designed....the whole point of a "scope" is to be able to nest other things inside of it. In my case, though, I needed the line three query to be totally seperate and distinct. It took some googling and stack-overflowing (love that community), but here's what I discovered:


Model.scope_with_some_joins.find_in_batches do |models|
  models.each do |mdl|
    Model.send(:with_exclusive_scope) do
      other_comparisons = Model.scope_with_other_joins
    end
  end
end

this protected "with_exclusive_scope" method resets the scope entirely for that model within that block. Thus, you're able to have a clean query regardless of the surrounding context. Now, I'm not saying that this hack of sending a protected method is a good idea anytime, but in my case I didn't have an easy way to get around it (other than leaving the SQL statement in place). It's still cleaner to me than having that giant SQL string I had in the code before, and maybe once I do a little more reading on the subject I'll get an even better idea. Other suggestions welcome!

Autotest saves the day

2010-01-21T10:46:00.000-06:00

This is not a tutorial on how to setup autotest on your machine. People have already done that plenty of times, a good one is here.

I'm just writing to say what a big difference it's made for me. My first job was at a "test-everything" development shop. I really agreed with the notion of having solid tests surrounding all possible code, and running them before every commit/deployment. The problem for me arose when I moved to start my own business. Without all that peer pressure (I'm working as the only development talent currently), it's easy to fall off the wagon. Especially when using tools that don't exactly integrate testing into your workflow. Typically I'd write tests around all new code, and run them before committing, but if I was making a quick change just to format something better or to fix a bug, I was often hurried enough to not only write no new tests, but to not run any of my current tests before committing just to get the damn thing out the door.

Autotest silently runs in the background, running your tests anytime you change a file. Not only that, it can be configured to use Growl to notify you every time a test breaks. Now I don't even have to think about it. Every time I press command-S, my tests get run and I know that at the least I haven't broken anything that's currently covered.

Of course the limitation is that you must write tests in the first place. Having your tests run all the time without any real coverage doesn't save you much. For me, though, just knowing that my tests are being run consistently gives me more motivation to write more of them, more often. If you're a rails developer, just try it. It's really not too much of a time commitment to set up, and if you're like me you'll be suddenly one giant step closer to being the unit-testing-guru that you always wished you were.

My favorite coding music

2010-01-11T16:40:00.000-06:00

I thought I'd kick off the new year on a slightly less technical note than usual: the tunes I play to boost my productivity. Silence kills me and causes my mind to drift; I personally find that having the right kind of music playing low in the background keeps me going in the right direction when I'm trying work.

This is a very personal subject, so I can't say that what works for me will work for you, but it can't hurt to try out what I'm playing and see what it does for your work life. So, without further ado, here are my 5 top albums for cranking out code:

5) 'Marimbach' - Beverley Johnston

Soft, compelling, and beautiful. This is an album made entirely from classical marimba pieces (mostly solos). It's not distracting at all, but isn't so slow or quiet as to make you lethargic.

4) 'Flight of the Cosmic Hippo' - Bela Fleck and the Flecktones

A nostalgic favorite of mine from high school, this album is a collection of eclectic instruments playing a genre that is entirely it's own. Don't be thrown off by the funky title, these are some serious musicians (including the great Victor Wooten) and it is pretty high-energy the whole way through. My favorite for pressure situations. If I have a deadline today, this is what I play.

3) 'Hush' - Yo-Yo Ma & Bobby McFerrin

This is a collection of duets between a pair of virtuosos: Yo-Yo Ma, the master cellist, and Bobby McFerrin, the most amazing vocalist I've ever listened to. This is a man who has truly made his voice an instrument, and it's almost trance-inducing. I like to put this on first thing in the morning when I sit down to start, although I can't articulate why. Just give it a chance. It's strange at first, but once it grows on you it's an indispensable part of your collection.

2) 'Evolution' - Stefon Harris & Blackout

As a mallet player myself, there are few contemporary musicians I love as much as Stefon Harris. This is much richer and fuller music than anything else on my list, but I think it's perfect for the "deep coding" work. Unhurried, but high energy, and smooth without being "easy listening" (which I despise).

1) 'Libertango' - Gary Burton

My favorite album in my collection, it's steady rhythm always makes me feel motivated. If you try nothing else on my list, give this a listen, especially track one (the title track). It's very dense music, but so accessible even to one who is not a musician. I can't recommend it enough.

So if you're looking for a few new tunes to keep you going during those times of loose focus, see if you can find something you like among my favorites, and drop me a line if you do! I've got more in my back pocket, I just can't share it all in one blog post. :)

Great memory profiler

2009-11-30T14:05:00.000-06:00

Just wanted to link up this post that's helping me solve a tough memory problem with a really lightweight bit of code. If you've got some memory issues with any bit of code in your rails app, go check this out and make use of it:

http://scottstuff.net/blog/2006/08/17/memory-leak-profiling-with-rails

It's over three years old, but still works with the current version of rails quite well.

How to be Humble

2009-11-19T11:01:00.001-06:00

I installed the new Rails Best Practices gem today, and I have to admit it kind of kicked me in the fork. In a good way.

Sometimes it's good to be reminded of just how many things you've done wrong, as it can really motivate you to dig in and get some of that stuff fixed. As for me and my statup's codebase? I have 291 violations of the best practices that need to be addressed. Oh well, at least I've moved from blissful ignorance to gut-wrenching enlightenment.

Want to get schooled on your own code base? Need a good swift kick yourself? Follow these steps:

1) Install ruby_best_practices

sudo gem install rails_best_practices --source http://gemcutter.org

2) cd to your root directory of your rails project

3) execute this line:

rails_best_practices . --debug

(I used the debug flag because it tells you what file was just examined, that way if there's an explosive error and the process bombs, you'll know which file to investigate).

Happy self-humbling!

What to do with expensive and schema-less data in Rails

2009-11-18T11:58:00.000-06:00

OK, the title for this post definitely needs work, but this is an important topic.

Scenario: I have some invoice that needs to be generated once a month for each customer in my database. Each invoice has a table on it that specifies the amount the customer is being charged for each category of services. incomplete ruby-code follows:

class Invoice
  def data_row(category)
    [category.name,
     category.description,
     self.expenses_for(category),
     self.expense_ratio(category)]
  end

  def invoice_data_table
    Category.all.map{|c| data_row(c)}
  end
end

for the sake of argument, assume that "expenses_for" and "expense_ratio" are both defined and are expensive calculations. Also, the categories list changes from month to month. Now, how do I store this data so that I don't need to re-calculate it if I need to re-print this invoice later?

1) static columns
This is probably the worst one, and the one I see most commonly in large corporate applications. Basically you write 50 new columns in your invoices table, each one looking like this: category_1_expense_ratio, category_1_expense, category_2_expense_ratio, etc, etc.

Retrieval is efficient, but every time the category list changes you have to add new columns to your database, and the code for populating these fields is probably ugly and repetitive.

2) child records in join table
Better, but still somewhat expensive in relation to it's purpose. We add an invoice_category_expenses table, and it's modeled as "category_id, invoice_id, expense_ratio, expenses".

This works, and is more robust in the face of change, but in order to reprint the invoice that means you have to fetch 20 other records besides just the invoice. They're not huge, so it's not a real performance problem, but it does feel wrong to me to grace this flat data table with all the trimmings of a full model class.

3) serialized data-structure

This is what I ended up going with, and I'm maddeningly happy with my decision. For readonly structured primitive data that just needs to be stored for possible re-examination (ie, no other behavior necessary) I don't think you can beat it.

Rather than stuffing it in another table, or making your invoices table a schema nightmare, you just add one more column: category_expenses_data. It's a text column with a large size limitation. Now in your invoice model you write the following:

class Invoice
  serialize :category_expenses_data,Array
end

This specifies to rails that this column is going to be a serialized Array (basically a string representation of an array that can easily be re-instantiated). Now when creating the invoice, you generate the expensive data table, and store it by saying "invoice.category_expenses_data = invoice.invoice_data_table". The expensive calculations are done once and stored in the database. When you need to reprint the invoice, you can just ask for "invoice.category_expenses_data" and it will give you back your array (deserialized from the database) ready to be used to generate your PDF again or whatever.

RubyMine 2.0

2009-11-17T14:28:00.000-06:00

I've been using TextMate for Rails development ever since I started programming on a Mac. I've got to say, though, that after trying out the new RubyMine IDE, I'm considering upgrading. Don't get me wrong, I LOVE TextMate. It's been nothing but good to me, and I still plan on using it for Scala and other editing tasks. That being said, my first IDE when I was just starting to program was Eclipse, and in that regard RubyMine feels like I'm coming home.

Cool things I've liked so far:

1) GUI test running reminiscent of JUnit. Yeah, I know, it's just eye candy, but what can I say? I like it, and my preference should count for something if I'm the one doing the development.

2) GUI Git integration. Point and click, baby!

3) intellisense/code completion/inspection/warnings. These can be annoying if things are too bad. Your code looks like it's been torn apart by a highschool honors english teacher, past a certain point. But overall I really prefer having the analysis and information built visually into my editor (another eclipse throwback).

Somethings that make me want to stick with TextMate:

1) Speed Compared to RubyMine, TextMate is blazing fast. It loads quickly, searches quickly, and I definately feel more nimble.

2) Simplicity. TextMate is an editor, and a good one. Your screen isn't cluttered with a hundred helpful windows.

I guess I've got some thinking to do before I lay down any money, but I'll post what I decide when I get around to it.

why your named_scope isn't behaving the way you want

2009-09-30T12:18:00.001-05:00

Let me go into this as briefly as possible. I had the following named_scope in one of my rails models:

class Alert < ActiveRecord::Base
  named_scope :in_the_past,
              :conditions=>
                ["start_datetime <= ?",DateTime.now]
end

Seems straightforward, right? I thought so too. I used this "in_the_past" method to replace a poorly written load-and-filter algorithm. This would use much less memory and be much quicker. Only one problem it didn't work.It passed all the unit tests. It seemed to work when it was deployed. But over the next few days more and more people were calling me saying they weren't seeing the alerts they expected to. I tried to figure out the problem (maybe related to timezones? or bad data?), and every time I deployed a fix all my users would tell me it looked good again; but then a day or so later everyone was complaining once more. I finally got fed up with it and went back to the old way (loading a collection and filtering out the items in the past using a "select" iterator), cursing the day I'd ever started down this road. And then today, I saw this in a blog post from pivotal labs:Public Service Announcement: your named_scope's get evaluated when the class is loaded, not when an instance is created.I felt like I'd been hit by a truck. The "DateTime.now" call was being evaluated at server startup, so for the first few minutes it seemed fine. However, as the amount of time since class-load increased, the "in the past" scope was missing more and more of the items that were actually now in the past because it's measuring stick was stuck at the time of creation. As a result I dug through my codebase and found no less than 7 scopes to refactor into using a lambda instead so that the system time would be loaded at time of execution instead:

class Alert < ActiveRecord::Base
  named_scope :in_the_past,lambda{
                {:conditions=>
                   ["start_datetime <= ?",DateTime.now]}
              }
end

Thanks to those pros at Pivotal Labs!

GMail hijacking

2009-09-23T18:25:00.001-05:00

So my email account has been compromised. I don't know how, but what I do know is I'm locked out. I'm working on it with google, but the short version of the story is this: someone sent out an email to everyone in my contacts list saying I need money to get home from london. I'm not in london. i don't need money. Ignore this and any other emails that don't sound like me.

SpriteMe up, Scotty!

2009-09-15T11:02:00.000-05:00

This morning as I perused my rss aggregator, I came upon an article over at Ajaxian about a cool new tool for auto-generating CSS Sprites. This seemed like a cool idea, but I didn't think much of it. After all, I've heard of hundreds of projects that try to do something I think would be really useful, and when I go to try it myself I find it's such a pain to get working it's not worth the effort and I may as well accomplish the original task manually.

However, I decided to give it a try in this case because one of my current startup projects (VacationsTxt) was in dire need of some spriting attention. You see, there were 22 background images on one page, which means load time was pretty much sucking as 22 http requests had to be made just to get all the images alone, nevermind the javascript files and stylesheets. So I told myself I'd give it 5 minutes to spike this new tool and if it looked like it was going somewhere I'd maybe invest a little more effort.

Imangine my surprise, 5 minutes later, when I was done. Yes, done. SpriteMe is just a javascript bookmarklet that you run while you're on the page in question. It auto-magically scans all background images and tries to combine them into a css sprite. Then you just download the new image, integrate the necessary css changes that have been applied to your page (live, I might add, right while you're watching it), and you're done.

So how much did SpriteMe help my homepage? See the screenshot below to find out:

I don't know if that's readable or not, but what it says is this "21 requests eliminated". That's 21 fewer http calls that my server has to handle for every visit to the homepage, and 21 fewer requests the user has to wait through on load.

More importantly, how much work did it save me? Well, being no graphics expert, I predict I would have spend 4-5 hours trying to get the sprites right for my homepage. Doing all that in 5 minutes is impressive. Most impressive.

Steve Souders, you've made my day.

Browser Caching and Rails

2009-09-11T09:16:00.004-05:00

Alternative title: "Your debugger can't debug what it doesn't execute"

At my startup, we had been experiencing an intermittent user problem for the last few days, with no clue as to the cause.

Users have been reporting that their "alerts" aren't showing up when they login...sometimes....and it goes away a few minutes later....sometimes.

Anytime I hear a report like that, I start dreading the work ahead, because if it's only "sometimes", there's probably not an easy way to reproduce the bug reliably. It could be data-dependant, or a browser quirk, or dependant on the phase of the moon for all I know.

So I started digging in to our code base. Alerts were displayed based on when certain events had gone into the past. Was there a time-zone problem? Was the current time being extracted incorrectly? Was there a problem rendering the partial that handled that block of the homepage? Maybe the CSS was displaying it improperly if there were other dynamic elements on the page?

I started throwing logging in at various points through this controller action. Hours later, reports were still coming in (slowly, so not everyone was experiencing this problem), and the logs looked fine. Then I started to notice something. The people who were complaining the most, had the fewest log reports. In fact, the production log showed that one of the users who was complaining had not accessed the page in question for 2 days.

Enter the browser cache.

For some unknown reason (damned if I could figure this out), this user's browser decided to cache her homepage. She wasn't seeing the alerts because her browser had chosen to cache this page at a moment when she didn't have any. Until it decided to reload the html from that page, she would continue to see no alerts.

With some quick googling, I found a cache buster snippet, and I'll share it here in case anyone else has had the same problem. To summarize:

CLUES THAT A BROWSER CACHE IS CAUSING YOUR BUG:

1) only some users report an issue
2) issue is intermittent
3) no errors being reported from your notification software
4) logging shows no data problems or logic errors
5) symptoms indicate "time displacement" (that's what the page would have looked like "at one point")
6) serious frustration and anger on the part of the software developer

If you are a rails developer, here's the medicine (this goes in application controller):


def set_cache_buster
 response.headers["Cache-Control"] = 
   "no-cache, no-store, max-age=0, must-revalidate"
 response.headers["Pragma"] = "no-cache"
 response.headers["Expires"] = "Fri, 01 Jan 1990 00:00:00 GMT"
end

set this as a :before_filter on any action you want to protect.

Forwarding Emails with Ruby, IMAP, and SMTP

2009-09-02T18:04:00.003-05:00

I would hate to tell you just how much it sucked trying to figure this out.

Actually, I would LOVE to tell you. It would be totally cathartic to just unload a days worth of frustration. However, that wouldn't help you figure out how to do the one thing mentioned in the title of this post (which is probably the only reason you're here), and it sure as hell wouldn't make me seem like much of a professional.

So I'll skip the tirade except to say this: Maybe I should have given up 4 hours ago.

That out of the way, here was my problem. I have the email box where all these clients send data files to. I wanted to sort the ones that had attachments (and only ones with attachments) based on who they came from and forward them to different email addresses. There are all kinds of nice ways this could have been done if I wasn't constrained by some external requirements (like not being allowed to use forwarding rules on the server the emails were sitting on), so for the sake of this post assume this had to be done in ruby using net/imap to fetch the emails, and net/smtp to send them to wherever they needed to go.

This was not a fun place to start. These libraries work well, but I'll be honest when I say that MIME and IMAP are a little low level for my accumulated programming experience. So I started by using as a jumping off point a script I wrote for another blog post a while back when I was trying to access my gmail inbox with ruby. I got far enough to where I was polling the email inbox successfully, so now I just needed to figure out the "sending" part. No big deal, right?

I can't tell you how wrong you are.

You see, you can't just pass off a Net::IMAP::FetchData object into the net/smtp library and watch it fly off into the designated mailbox. NO SIR! You see, I had that idea, and this is what I tried:



require 'net/imap'

imap = Net::IMAP.new('mail.yourserver.org')
imap.login('username','password')
imap.select("INBOX")
imap.search(["SINCE", "8-Aug-2007"]).each do |id|
  email = imap.fetch(id, "BODY[TEXT]")[0].attr["BODY[TEXT]"]
  Net::SMTP.start('smtp.server.org') do |smtp|
    smtp.sendmail(email, 'from',['to'])
  end
end

It was not very successful. You see, my email client kept showing me the full MIME body, not just the visible components, and attachments were showing up as base64 encoded strings. This was not useful.

I decided the problem was most likely that I was just pulling incomplete MIME information from the imap library and that I needed to find the right attribute to ask for that would have the full MIME package in it's raw form. Enter "BODY[]". It might look very much like the "BODY[TEXT]" attribute I used above, but you would be deceived. "BODY[]" is the whole deal, everything that's there. If you want the raw message, that's what you need.

So that's what I tried:



require 'net/imap'

imap = Net::IMAP.new('mail.yourserver.org')
imap.login('username','password')
imap.select("INBOX")
imap.search(["SINCE", "8-Aug-2007"]).each do |id|
  email = imap.fetch(id, "BODY[]")[0].attr["BODY[]"]
  Net::SMTP.start('smtp.server.org') do |smtp|
    smtp.sendmail(email, 'from',['to'])
  end
end

Presto, it worked! No, I'm lying, same problem. So what to do now? Honestly, after the amount of reading I'd done just to arrive at the above conclusion, I was considering giving up. But I thought to myself: "This should be a solved problem. Someone out there should have already written something to take care of dealing with MIME formatting so that I DON'T HAVE TO THINK ABOUT IT!"

Enter rubymail. It looked very promising since the docs said it would parse and generate MIME format behind the scenes of a simple interface. "Great", I thought, "Let's try it":



require 'net/imap'
require 'rmail'

imap = Net::IMAP.new('mail.yourserver.org')
imap.login('username','password')
imap.select("INBOX")
imap.search(["SINCE", "8-Aug-2007"]).each do |id|
  email = imap.fetch(id, "BODY[]")[0].attr["BODY[]"]
  email = RMail::Parser.read(email).to_s
  Net::SMTP.start('smtp.server.org') do |smtp|
    smtp.sendmail(email, 'from',['to'])
  end
end

This looked promising, but for an unknown reason RubyMail was parsing my emails wrong and making things that were part of the body out to be headers. It wasn't going well. Convinced I was on the right track with this whole "Let somebody else deal with the MIME" idea, though, I substituted "tmail" (another good ruby email library).



require 'net/imap'
require 'tmail'

imap = Net::IMAP.new('mail.yourserver.org')
imap.login('username','password')
imap.select("INBOX")
imap.search(["SINCE", "8-Aug-2007"]).each do |id|
  email = imap.fetch(id, "BODY[]")[0].attr["BODY[]"]
  email = TMail::Mail.parse(email).to_s
  Net::SMTP.start('smtp.server.org') do |smtp|
    smtp.sendmail(email, 'from',['to'])
  end
end

there you are, scripted forwarding. May that help someone out there to not have a day like mine. :)

Communication is sometimes the problem

2009-08-14T10:14:00.004-05:00

While waiting for a deployment to run this morning, I decided to quickly peruse the tech industry current events list over at Hacker News. The first article that caught my eye was one by Jeff Atwood where he discussed the current state of web programming within the community and the diverse opinions on the subject that are out there.

There are several directions I could go in to comment on this piece of work. I could talk about the Michael Braude quote Jeff uses at the beginning of his article and attempt to discredit it; I could talk about Mr. Atwood and his tendancy to state his positions a bit strongly (although we all do that from time to time); but what really caught my attention was the comments, both on his website and on HackerNews. Why do we in the tech industry so badly want to paint everyone else in the industry with us as moronic and useless? I hate to oversimplify, but examine these to see if you can find the common thread:

Michael Braude: "The reason most people want to program for the web is that they're not smart enough to do anything else. They don't understand compilers, concurrency, 3D or class inheritance. They haven't got a clue why I'd use an interface or an abstract class. They don't understand: virtual methods, pointers, references, garbage collection, finalizers, pass-by-reference vs. pass-by-value, virtual C++ destructors, or the differences between C# structs and classes."

Summary: Web programmers are stupid. I know better things. I'm smarter than them.

Jeff Atwood: "Let's put aside, for the moment, the absurd argument that web development is not challenging, and that it attracts sub-par software developers. Even if that was true, it's irrelevant. "

Summary: People who think web programmers are stupid, are stupid. I know web programming. I'm smarter than them.

pj: "The web is popular for wannabe's because they get exposure. They think they are hackers, but really they are just factory line workers assembling someone else's code."

Summary: People who use web frameworks are stupid. I would never use a web framework. I'm smarter than them

jeremymcanally: "I wholeheartedly disagree, mostly because there are a lot of people that think he's the next Fred Brooks. He's not. Far from it. He's teaching a new generation of developers these stupid axioms that make no sense. I didn't realize this until I had a few of the more nonsense ones parroted back at me during an interview once."

Summary: People who like Jeff Atwood are stupid. I have no use for Jeff Atwood. I'm smarter than them

Maybe I'm just feeling morose today, but can't we all just get along? Exchanging opinions doesn't have to occur primarily by assaulting your opponent's intelligence. Maybe the byproducts of near-ubiquitous communication mediums like blogging and comments on news articles show that it's not necessarily great to give everyone a voice. Next thing you know, intelligent discussion is tarnished by some otherwise-somewhat-intelligent-software-developer-stricken-by-the-temporary-lunacy-that-comes-with-internet-anonymity screaming "YOUR MOM'S A COMPILER!", and now how will we ever regain the coherent footing of a reasonable discussion?

Anyone want to take odds on someone stumbling upon this post and telling me I'm an idiot for writing it? :)

Internet Explorer, may you die a thousand deaths

2009-06-12T13:02:00.002-05:00

Title says it all.

Processing large files with Ruby and Rails

2009-06-04T18:00:00.006-05:00

UPDATE: As "stefano" pointed out in the comments, the standard "gets" method does indeed accept a parameter. My fault for not checking the documentation!

although we as web developers prefer working on neat features for our websites, sometimes we need to get down and dirty with data processing. I know, I don't like it any more than you do, but if you want to run a business sometimes you have to do stuff that isn't that fun. That's one of the reasons I liked Ruby; the file library for reading and writing files makes data processing a lot simpler than similar tasks I written in C++, Java, and other mainstream languages. Usually I just do something like this:



file = File.open("some_file")
#read all the contents of the file into "str" variable
str = file.read
file.close
...#do some processing...

Wow, that was easy! However, sometimes code like this just won't cut it. For example: if your file is too big to read into memory all wants and could cause performance issues for the server, you may want to process the file iteratively. Again, and Ruby provides a pleasant way for handling the scenario. The "gets" method accepts a block and yields back to you each line of the file one at a time, thus conserving your precious memory. See the example below:



file = File.open("some_file")
while(cur_line = file.gets)
   ...#do some processing...
end
file.close

Also pretty easy. Today, however, I ran into a new problem. What happens when your file is too big to read into memory at one time, but all the data is all in one single line? Don't believe that would ever happen? Check out EDI sometime and see what you think about it (On second thought, never checked that out. Never ever look at EDI. I wouldn't want to make you cry). Sometimes even XML or HTML files are written all on one line in human readability isn't of any particular concern.

Well, I have never dealt with that situation before. I had this file I needed to process, roughly 30 MB, all on one line. now, iterative processing would be okay, because each segment in the file was in the proper order for processing and it was delineated by a pipe character, but there just aren't any built-in methods all in the file object that do what the "gets" method does on a delimiter other than newline. So I wrote one:



class File
  def uber_gets(delimiter)
    segment = ""
    self.each_byte do |byte|
      char = byte.chr
      if char == delimiter
        yield segment
        segment = ""
      else
        segment = "#{segment}#{char}"
      end
    end 
  end
end

with this modification, you can now do small iterative processing based on any delimiter. In my case, using EDI files, each record is separated by a "~". so, I used the above method as follows:


file = File.open("some_file")
file.uber_gets("~") do |segment|
   ...#do some processing...
end
file.close

There you go. The whole file is on one line, but the code is still respecting memory consumption. If it helps you, enjoy. I'll post a link to the gist:

http://gist.github.com/123924

A new way to Blog

2009-06-04T08:39:00.004-05:00

Most of my day I spend in front of a computer. I write code, I answer e-mails, and then when I get home, I blog. All that typing can be hard on the hands. I try to do most of the right things, ergonomically speaking. But I still end up with tendinitis. Being only 22 years old, this is obviously something I really want to avoid if I'd like to continue a career in the software business.

Enter dictation. This isn't the first time that I play with the idea of speaking to my computer. Both Windows Vista, which I had installed in my old computer, and Mac OS X, which I have all my new computers, have built-in speech recognition software. However, this is all mostly for command and control. The software helps you do things; you can open new windows, push menu buttons, click links, and do all sorts of other command based tasks. But when it comes to actually writing, these solutions fall short. Today, though, I'm happy to say that I'm now past that point. every word that you are reading on this page was put there by dictation software. MacSpeech dictation is the program I'm using, and I have to admit I'm impressed. Everything I say seems to just end up on the screen without me having to use my hands or forearms.

there are some drawbacks to the software. For one, it's not cheap. $200 for the current release. Now admittedly, that's not the most expensive piece of software ever seen, but for regular consumer consumption of price point seems a bit steep. On top of that, there is no trial version you can download. In fact, even when you buy it you can't download it. You have to have it shipped to you as if we were back in the 90s. so if you decide you want to buy MacSpeech Dictation, be aware that it's all or nothing.

For someone like me though, the advantage of being able to do my blog posts without my hands far outweighs the drawbacks inherent in MacSpeech's distribution system. I hope that as my experience with the package progresses, I'll be able to say that all my e-mails and all my blog posts are done without putting any unnecessary strain on my forearms. That way, I can save my limited typing capacity for what I enjoy most: code!

If this is something you'd like to try, check out the link below.

Mac Speech

Metric-Fu

2009-05-26T22:48:00.002-05:00

The latest addition to my utility-belt of Ruby tools, Metric-Fu is giving me plenty of ideas on how to refactor my codebase.

Essentially, this library gives you the ability to take 8 of the most common code analyzing tools and run them all on your codebase at once, producing one consolidated report. I love it!

The full list can be found at the Metric_Fu page on rubyforge.org, but 2 of my favorites are listed here:

Roodi gives you some design help as it checks all kinds of common programming problems. Method have too many parameters? Cyclomatic complexity too high? Forget the else clause on a case statement? Roodi will give you the heads up you need.

Flay checks out your code for duplicate constructs and segments. It found several pieces of duplicated logic that I had never noticed before, I was really impressed. It also does soft matches on "similar" code, things that could probably be combined and simplified if you're clever. Very nice.

The great thing about metric_fu is that you just run one command, and all the packages get run for you, giving you one page afterwards containing links to the results of whatever package you want. Check it out, you might be surprised at how much your fingers start itching to go back and fix all the problems you didn't even know you had.

Curse you, rake db:migrate!

2009-05-25T12:32:00.003-05:00

Have you ever been in this situation?

"Hmm, this feature will require a big change to my database! I mean, we're going to have to touch every single record in that table."

If you have, you've probably followed up with this thought:

"Good thing I'm using Rails! They make it so easy!"

And then you went and wrote this migration:


def self.up
  Model.all.each do |m|
    #..some important function 
    #performed on every object..
  end
end

And then you were really proud of how quickly that went, and you run it on your development machine, and it works really well. But THEN you push it to your staging or production server that has way more data than your dev machine, and you get this staring back at you from your command line:


 ** [out :: 123.123.123.100:8063] ==  
 ** YourCrazyMigration: migrating 
  =========================================

And you stare at that for about 30 seconds before shouting:

"Mother F%^&er! I did it again! I can't believe I did it again! I built a stupid migration that uses the stupid 'all' method which is now dominating the memory on that box and I can either kill it and pick up the pieces or let it run for the next 3 hours as it pages the hell out of the hard disk!"

Well, since I did EXACTLY THAT just now, I decided that from now on we'll be using a new migration task at our development shop called "safety_migrate", which you're welcome to take advantage of if you'd like. It runs through every file in your migration directory, checking for the dreaded "all" method, and WILL NOT run the migrations unless every file is clean!

check the gist:

http://gist.github.com/117625

Happy Migrating!

Maximum Impact for Minimal Effort

2009-05-17T20:10:00.002-05:00

Today was a day for working outside, and I got a lot of mulch spread over the landscaping at my house.

I love mulch because it's really easy to use, really cheap to buy, and really makes a difference in the way your yard looks. You can't get much more bang for your buck as far as landscaping goes.

Believe it or not, I was thinking a lot about work as I was hauling chips of cedar over to my flower beds, and I started considering what sorts of things are "mulchy" in the web development world.

Color

Unfortunately, people aren't often impressed by performance or functionality nearly as much as they are by aesthetics. A white page full of blue links that all do really cool stuff is just a huge turn off. The difference that can be made with a simple header, a 3-column layout, and a pleasant color scheme is phenomenal.

Central Navigation

Sometimes you have websites that have spaghetti links all over the place. Trying to get back to the homepage usually means either clicking the back arrow 14 times, or re-typing in the domain address. A very common and successful approach is to have a set of navigational links as part of the header, and it's successful for a reason: people know where to go. No matter where they are, the critical places on the website can be accessed easily.

Intelligent defaults

If you have a dropdown that has a list of states, and 80% of your users are local to your state, go ahead and default the selection to your state. If you have 25 reports users can run on your website, and 3 of them are used more often than any others, put those three at the top. It doesn't take a lot of work to reposition things, but it makes a big difference.

There you go. Three easy things to do that will make a large impact on your users.

Joins and named_scopes in ActiveRecord

2009-05-14T08:18:00.007-05:00

So many of us know how cool named_scopes are in ActiveRecord; they really make building complex queries quite pleasant compared to writing out big hairy SQL strings all over the place. However, in the past week as I have refactored my whole web application to use this excellent feature, I've run into some little-discussed items that I feel should be shared somewhere. Hope you enjoy, and anybody who is more advanced in their Rails-fu is welcome to give me some schooling as to where I've gone wrong on any of these, as most of my discoveries here are through trial and error.

joining "through" associations

Say you have a three-layer association. The following would be an example of this:


class Company < ActiveRecord::Base
  has_many :employees
  has_many :children,:through=>:employees

  validates_inclusion_of :industry,
                 :in=>[“TECHNOLOGY”,”FINANCE”,”AGRICULTURE”]
end

class Employee < ActiveRecord::Base
  belongs_to :company
  has_many :children
end

class Child < ActiveRecord::Base
  belongs_to :employee
  has_one :company,:through=>:employee
end

So a company has many employees, and an employee has many children. This is pretty straight-forward. Now, let's say that we want to find all the children who's parents work for any company in the “TECHNOLOGY” industry. How could we accomplish this? Well, certainly one way would be to iterate over the associations, starting by finding all the technology companies, then iterating over each employee, and adding their children to a growing array. It would look something like this:


children = []
Company.find_all_by_industry(“TECHNOLOGY”).each do |company|
  company.employees.each do |emp|
    emp.children.each do |child|
      children << child
    end
  end
end

But that's not great. That's a lot of lines of code to read for one qeury. Let's put the meat of this finding algorithm into a named scope instead:


class Employee < ActiveRecord::Base
  named_scope :works_in_technology, 
           :joins=>:company,
           :conditions=>”companies.industry = 'TECHNOLOGY'”
end

children = []
Employee.works_in_technology.each do |emp|
  emp.children.each {|child| 
    children << child 
  }
end

That's a little better. You can add in joins to your query just by giving the symbol name of the association, so that's really nice looking. However, as long as we're joining on associations, why not just have another named scope in the Child model and do this whole thing without any iteration?


class Child < ActiveRecord::Base
  named_scope :parent_works_in_technology,
              :joins=>:company,
              :conditions=>
                ”companies.industry = 'TECHNOLOGY'”
end

children = Child.parent_works_in_technology

Hey, that's a lot better! Only one problem: it doesn't work. Yeah, we do have an association between the child and company models, but it's “through” the employee model, and the named_scope doesn't know how to assemble that query properly (try it, you'll see the sql it generates isn't quite right). We can still do this as one named_scope join query, but we're going to have to get a little more detailed level of control:


class Child < ActiveRecord::Base
  named_scope :parent_works_in_technology,
              :joins=>”, employees, companies”
              :conditions=>
                   ”employees.company_id = companies.id and 
                    children.employee_id = employees.id and
                    companies.industry = 'TECHNOLOGY'”
end

children = Child.parent_works_in_technology

Now THAT works just fine. It's a little more verbose in the named_scope declaration itself, but it's really nice for wherever else you need to use that query in your codebase. Notice that you have to do the foreign key joining yourself in the conditions string, don't forget to do that or you will definitely not get the results you are expecting.

Watch out for duplicates in join queries

So now we're joining across multi-layer associations successfully, and now we have a new requirement: We're rolling out a new children's special to all the companies in our system and we need to get a list of all the companies who have employees with children younger than 12. Based on the last named_scope we built in the child model, it seems like we could do another one just like it in the company model.


class Company < ActiveRecord::Base
  named_scope :has_employees_with_young_children,
              :joins=>”, employees, children”
              :conditions=>
                     ”employees.company_id = companies.id and 
                      children.employee_id = employees.id and
                      children.age < 12”
end

companies = Company.has_employees_with_young_children

If you don't know how table joins work, it might seem like this would be fine. The problem comes up when you have a company that has more than one child under age 12: because you're asking it to return you one record for every match of the conditions, you'll get a company record back for every child at that company that's under 12! If “Innitrode” has 50 employees, and those 50 employees have collectively a total of 75 children under age 12, then this named_scope will return 75 innitrode records rather than the 1 you were expecting. This certainly would not be the behaviour you were looking for. Enter the “group” option. Grouping allows you to take the set of records you get back and compress them depending on a matching value in each one. For example:


class Company < ActiveRecord::Base
  named_scope :has_employees_with_young_children,
              :joins=>”, employees, children”
              :conditions=>
                  ”employees.company_id = companies.id and 
                   children.employee_id = employees.id and
                   children.age < 12”,
              :group=>”companies.id”
end

companies = Company.has_employees_with_young_children

Presto, now the results you get back will be flattened so that each record with a company id of n is grouped into a single record, so you will get no duplicate company records in the array you have returned.

ReadOnly records

So now you're using these great joining named_scopes to find records based on all kinds of cross-table conditions, and things are working great. However, they will continue to do so until you need to update one of those records you retrieved, and then you will get a bit of a surprise when your save method throws an error saying that the record you are trying to update is read only.

Actually, there's a very good reason for this, you just have to know a little bit about how active record works. When you run a query with joins in it, you're getting back single records that have WAY more attributes than just the ones found in the model table you're dealing with. If you join employees to companies, than your employee object will have a hash of attributes that has all of the data in it's table, PLUS all the data in the table for the company he works for. So what if you update one of THOSE attributes instead of the ones that are part of the employee table? The update statement to the employee table wouldn't be able to find the column named “corporate address” or whatever, and would crash. So, the records that are returned from a joined query are just marked readonly to prevent that from ever happening. In a rails action, this would rarely be a big deal because you're usually just passing the ID of the object you want to update, plus the parameters that you need to change, so you'd be loading a new record out of the database anyway based on the ID, and that object will be writable. However, it's important to know how to combat this in case the situation every arises where you DO need to update one of these records. There are 2 ways that I can think of around it.

First, you can just reload the record you want to update by running another find just using the ID of the object.


  company = Company.find(company.id)

Another approach that works is to update your named_scope declaration to use the “select” option:


class Child < ActiveRecord::Base
  named_scope :parent_works_in_technology,
              :joins=>”, employees, companies”
              :conditions=>
                ”employees.company_id = companies.id and 
                 children.employee_id = employees.id and
                 companies.industry = 'TECHNOLOGY'”,
              :select=>”children.*”
end

Now that you're only pulling back the attributes specifically from your model table, the objects won't be read only. Be careful with this one, though, because you'll run into a runtime error if you try to use anything that modifies the query like the “size” method.


children_count = Child.parent_works_in_technology.size

This is going to alter your query to call “Select count(children.*)”, and your database will reject this as invalid.

Chaining Join Queries

Finally, what if you have a few named scopes on the same model that join the same tables? Is this a problem?


class Company < ActiveRecord::Base
  named_scope :has_employees_with_young_children,
              :joins=>”, employees, children”
              :conditions=>
                ”employees.company_id = companies.id and 
                 children.employee_id = employees.id and
                 children.age < 12”,
              :group=>”companies.id”
  named_scope :has_employees_with_male_children,
              :joins=>”, employees, children”
              :conditions=>
                ”employees.company_id = companies.id and 
                 children.employee_id = employees.id and
                 children.gender = 'M'”,
              :group=>”companies.id”
end

It's not a problem as long as you don't chain them together. But as soon as you do this, you're in trouble:


  Company.has_employees_with_young_children.has_employees_with_male_children

You're SQL query will now try to join in both of those other tables twice, which will result in ambiguous column references. Usually the answer to this problem is just to not chain joined queries together, but if you need to (like I did in one particular case), you can have the join itself be a different named_scope, so that you only run the joins once:


class Company < ActiveRecord::Base
  named_scope :has_young_children,
              :conditions=>”children.age < 12”
  named_scope :has_male_children,
              :conditions=>”children.gender = 'M'”
  named_scope :join_in_children, 
              :joins=>”, employees, children”
              :conditions=>
                ”employees.company_id = companies.id and 
                 children.employee_id = employees.id”,
              :group=>”companies.id”
end

This will work, you just have to remember to chain it in every time:


   Company.join_in_children
          .has_male_children
          .has_young_children

Hope this helps someone. Any reader with other tricks they've learned with joins, or better ways to do something listed here, put it in the comments and if it's good I'll move it to an update in the post (with credit given to the author, of course).

Git solves all your branching and merging problems! Almost!

2009-05-11T12:25:00.009-05:00

The entire development team here at Research to Practice (that is, Ray and I) have been working entirely out of git for a little while now. The problem was, we were using it just like Subversion. You see, having experienced a bad branch merge before, it's tough to get up the gumption to try again.

Why? Imagine getting food poisoning from a questionable sushi place in the midwest. Yeah, you know that it was totally the health code violations and 12-day-old-fish at that place that made you vomit 9 shades of purple, but you'd still have a hard time bringing yourself to eat anything at an upscale sushi restaurant on the coast of japan, wouldn't you?

Yeah, it's kind of like that.

We know that git is good at this stuff, we know that we want to take advantage of the great flexibility that using branches offers, but when it comes down to actually DOING it, there's always a reason why maybe we shouldn't try this just yet.

Last week, we decided to lay down the law. We wanted to have some features be worked on for a 2.0 release of our website, but we also wanted to continue deploying fixes and small enhancements to the currently running application. No way around it, it was time to branch out.

So it was with great fear and trepidation that we first unleashed this phrase upon our command line:


$> git checkout -b version_2_features

Oh Gods, what have we wrought?!

...Ok, so it wasn't that bad yet. The checkout command with the "b" option created a new branch for us, and after developing a small and inconsequential feature (a necessary precaution because we were fully fearing that the demons-of-parallel-development could swoop in at any moment and destroy everything we had done so far) we pushed this branch to the github repository so that we could both develop on it:


$> git push origin version_2_features

Alright, so far so good. Now we have our "master" branch, which was esentially "What is currently running in production", and our "version_2_features" branch where we could start developing everything we wanted to put into the version we were planning to release at the end of the summer.

Great Success!

So we started developing that way, and everything was moving along splendidly, until we hit something of a snag. Well, a boat anchor, really. See, the whole point of having this 2.0 branch be separate was so that it's content would not be deployed until much later, but SOMEhow, SOMEone, did SOMEthing (I'll let you ask Ray for the details behind that), and suddenly the version_2_features branch ended up being accidentally merged into master.

This resulted in an hour of us assuring ourselves that we could probably find a way out of this without having to admit we (he) ever screwed up, and we came up with a list of options.

1) Use git reset on the repo to bring it back to before the bad merge
2) Check out a previous commit version (before the bad merge), and then commit that as the head
3) Use git reset on a local repo, and push the result to github
4) Revert the merge commit, than when you want to merge for real later revert the revert.

Now out of the first 3 options, it turned out that exactly 0 of them were possible. There's no way (that we could find) to call "RESET" on our remote github repository, checking out a previous commit disconnected you from the head so you couldn't commit and call it the new head, and trying to push a branch that had been reset to the remote repo was met with a regular level of failure. Option 4 looked plausible, but beyond our ken, and neither of us is a git wizard just yet.

So we went with option 5, check out an old version of the repository, copy the entire directory structure into our working directory, commit the deltas, and bam, we're back to premerge state.

NEVER DO THIS

Little did we know at the time that this would later take us beyond regular and straight into the epic level of failure.

"What's the matter?" you might ask. "Isn't your directory back to the way you wanted it?"

The answer to that question is yes, our working directory was back to the production version, which was great for the master branch....but not for the history. You see, we still needed to be able to merge features that were being added to the version_2_features branch back into the master at some point in the future, and that's really hard to do if your repository version believes that those 2 branches have already been merged!!!

By the time we realized this, we'd already pushed these changes to the repo, and were beginning to consider just how screwed we were. Thus began the wailing and gnashing of teeth.

"We were right! Branching was a horrible idea! We've done exactly what should never have been begun!"

So we rebased and merged and copied and merged and went through every combination possible of recursive/3-way/octopus merging until we finally were PRETTY sure that everything was ok (about half a days worth of work), and do you know what I realized after that?

6) delete the remote github repository and create a new one from the one on your local machine that's still in a good state.

Yes. It was possible. It was even a good idea. if only I had thought of it sooner.

Moral of the story? git is great for branching and merging, far better than any solution I've used in the past. Unfortunately, even git can't protect you from your own stupidity. No tool is that powerful (except Java, that's what it was designed for).

On Learning

2009-04-30T07:58:00.002-05:00

In a previous post, I mentioned that in some of my spare time I like to give private music lessons to kids here in Columbia. Well, that part of my life is now temporarily closed as the demands of my business continue to grow leaving me very little "spare time" left over. It's been tough having to tell each one of my students that I'm not going to be working with them any more, because I've really enjoyed working with all of them and have kind of become attached to it, and as I said goodbye to each one I would find myself thinking about how much they had improved since we started working together. Now I'm not going to take all the credit for that, because they are the ones who practiced, so they are responsible for their own gains in skill. However, I feel that something I was doing worked right, better in fact that my previous experiments in the field of teaching.

What was the biggest difference this time around? Well, my approach to teaching in previous years could be summed up by saying "These kids don't know very much yet, so we'd better scale down everything we're working on to make sure it's not overwhelming and just learn a little bit at a time". That could be compared to my approach this year, which was more along the lines of "This kid already knows how to do X, so let's find something they aren't good at yet and specifically pick a piece to work on that requires that skill". Although the evidence is anecdotal, my experience seems to show that people learn better when they are challenged than when they are coddled.

There are three reasons I can think of that support this idea:

1) slow going is boring

Picking up a little at a time is a quick way to lose interest in anything. This is why I still don't speak Russian very well. I really want to learn Russian, but I've been trying to just spend a couple hours a week on it, reviewing many of the words I already know and just adding a few more. You know what? People who go participate in an immersive language program learn fast. People who pick up "Russian for Dummies" and peruse it a few times a week don't, they just get a nice paperweight that looks good on their coffee table.

2) harder to achieve = more satisfaction

Since most of the readers here are programmers, I can feel confident that this point will garner support. If you are trying to tackle something tough that you've never had to deal with before, then your feelings upon success are made up mostly of excitement and joy. If you just finished coding up another CRUD application which you've done 100 of before, your feelings are probably more of relief than anything else. Difficult and new challenges produce satisfaction that makes you want to come back and do more.

3) expectations cause results

This is probably the most important thing a teacher can do. If you expect your student to make minimal progress, and you convey that expectation in the assignments you give and in your assessment of their work, than they will definitely make minimum progress. If, on the other hand, you expect more out of him than even the student himself thinks he is capable of, he will tend to make strides that would surprise anyone because when he is working he is bolstered and driven by the fact that somebody else who has authority in this subject believes that he is capable of doing what he's been assigned. It's amazing, but true (at least in my limited experience).

So as I said, I'm sad to leave this part of my life behind for now, but the lessons that I have learned from it are worth taking with me into other parts of my life. If I want to pick up something new, I think the best way to go about it would be to pick up a challenging project (for my level), dedicate a good amount of time to it, and expect more out of myself than I think I'm capable of. In fact, I'll go so far as to say this would be a good way for you to learn anything too.

Net::HTTP and SSL

2009-04-28T14:32:00.002-05:00

Just ran into an interesting problem today that I thought I'd post the solution for. Everybody knows about the net/http library, it's a core bundle that lets you do web requests. Well I was using it for posting to an experimental web service a friend had set up, and I ran into a problem. Here was my code:


url = URI.parse("https://someaddress:443/web/service/path")
xml = generate_some_xml
request = Net::HTTP::Post.new(url.path,{"Content-Type"=>"text/xml"})
http = Net::HTTP.new(url.host, url.port)
response = http.start {|http| http.request(request) }

Seems like it should have worked, but I kept running into this message:

wrong status line: "\025\003\001\000\002\002"

after googling around a bit, I figured out that this error message comes up whenever you are trying to connect to an SSL enabled site with plain text. I figured that, like curl, if you used "https" in your url, then you were telling the library that you wanted to use SSL. Well, that's not exactly the way it works. Although I put my "https" into the original url, when it gets passed into the HTTP library on line 4 of my code up above, I'm just handing it the host and the port, not the protocol. This requires the use of an aditional line of code:


url = URI.parse("https://someaddress:443/web/service/path")
xml = generate_some_xml
request = Net::HTTP::Post.new(url.path,{"Content-Type"=>"text/xml"})
http = Net::HTTP.new(url.host, url.port)
http.use_ssl = true
response = http.start {|http| http.request(request) }

there you have it. Hopefully by finding this post, you figured this out a lot quicker than I did.

Improving your Rails-Fu

2009-04-24T18:33:00.003-05:00

This month I've been making a conscious effort to improve my competence with the Rails stack. It's not that I feel I'm a bad developer (nobody feels that way, not even bad developers), it's just that the last time I'd been really honing my craft was around rails 1.2, and it's been a while since then (2.3 just released recently). I knew there were some major changes out there that I wasn't taking advantage of, and that there were several things I could improve upon. This was evidenced both by my lack of experience with things being discussed on the rails core list, and the reluctance of other developers to contract in for work on my codebase. Something had to give.

So I went to the same place I always go when I want to work on my skills: The Pragmatic Programmers. I'm not affiliated with these guys in any way, I just really like their stuff. In this instance particularly, I selected a series of screencasts called "Everyday Active Record" done by Ryan Bates of Railscasts fame. Sort version of the story: they were excellent.

I have been a subscriber of the Railscasts feed for sometime, so I knew that Ryan was good at this stuff, but I learned quite a bit from this series in particular that made me slap my forehead as I thought about all the problems they were discussing that I had gone and wasted time trying to solve myself when it was BUILT INTO ACTIVE RECORD THE WHOLE TIME.

A good example of this was named scopes. If you are a Rails developer, you've probably heard of them already, and you're right now navigating away from my article. That's fair, I should have been keeping up better with the industry. Believe me, it didn't take me long to dig into my code and start name_scoping everything more complicated than matching on one column.

If you are like me and HAVEN'T exactly done a stellar job of following the developments in the rails stack (this was introduced in Rails 2.1), you might not know what I'm talking about. Let me point you to a good article on the subject. Read it. Cool, huh?

There's lots more, with easy to follow examples, and at $5 a pop they're a pretty great deal. Do yourself a favor and check them out.

DRY up your views with HAML

2009-04-17T19:59:00.003-05:00

About 6 months ago, I read an article about this great templating engine called HAML. I thought I might try it out. I didn't.

Giles mentioned it back in January. I read that article. "Oh yeah!", I thought. "I read about this! it's cool, I should give it a try!". I didn't.

Daniel Morrison gave it a review in February. I read that article too. He said it makes html unmaintainable for non-rubyists. I thought I should see for myself. I didn't.

Yesterday, Ray Myers (employee #1 of my startup) mentioned that he had found this thing called HAML and wanted to try it. I tried it.

Good things happened. My templates got short. I liked reading my markup. Everything was concise, kind of like this blog post.

Wish I hadn't waited so long.

TRY IT.

Profitable on Paper

2009-04-09T23:02:00.002-05:00

This year has been my first year as a startup founder, so I'd never attempted anything like this before. In my previous jobs, I got a paycheck every 2 weeks, just like clockwork.

Now, being in the midst of a cash crunch, I sometimes miss that security a little. Not enough to want to go back to being chained to an office working on something I don't care about, mind you.

You see, my company has performed services, and invoiced for them, in amounts that would keep our heads above water. However, some of our clients tend to take upwards of 90 days to pay up. If you consider receivables our situation looks pretty great, but without cash in the bank it means almost nothing.

I wrote a post a few months ago that said "Customer Service is King". I think that in light of new information, I'd rank Customer Service as more of a prince or duke. The real king is cash, and managing it's flow is a major art form.

Some things I've learned about managing cash:

1) slash expenses everywhere, all the time. Any money you aren't spending is money you will have in reserve for things you might suddenly be forced to pay for. Don't cut corners, though. Angry customers soon become non-customers.

2) You are an expense. The less you can live on, the better for the long term.

3) Early on in a startup, cash is scarce, so your time is less precious than your money. Spend time negotiating deals or discounts. It's well worth it.

I'll post others as I think of them, but the main thrust here is that as long as you have less than 6 months of operating expenses in reserve, one of your main jobs is to watch what money you have like a hawk.

Big Headaches for MYSQL users

2009-03-27T08:09:00.006-05:00

Here's something I didn't know, but which hopefully will help someone else out in the future:

DON'T USE NESTED SUBQUERIES!

Ok, if you know what that sentence means, and why I said it, you're done with this article, continue to the the next item in your RSS reader.

Now, for all the rest of you, let's start with what a nested subquery is. Let's say, for example, I have 2 tables in my database: students, and schools. Students each have a "school_id" as a foreign key. I want to retrieve from my DB all the students who go to a school that is in the zipcode 65203. I might write a query like this:

"select * from students where school_id in (select id from schools where zip = '65203')"

Or if I was working in rails (which I usually do):

Student.find(:all,:conditions=>"school_id in (select id from schools where zip = '65203')")

It looks really nice, it's easy to understand, and under certain circumstances it can DESTROY your database performance. There's a lot of reasons why this is the case, not the least of which is that query optimizers just don't have the time to figure out each one of your subqueries and fix them up. If you want a nice discussion about the why, I recommend This Blog Post wherein Scott Selikoff does a great job of explaining the problems.

For those of you who just want to fix it, do a join. Yeah, it's a little less pleasant to look at, but your hosting provider will thank you (or, in our case, will stop yelling at you for hurting their other client's database performance times). Here's an example of the previous query joined up:

"Select students.* from students, schools where students.school_id = schools.id and schools.zip = '65203'"

And the rails version:

Student.find(:all,:conditions=>"students.school_id = schools.id and schools.zip = '65203'",:joins=> ", schools")

May your DB feel better, and may you stop getting angry emails! Happy coding!

Opportunities where you least expect them

2009-03-25T15:36:00.004-05:00

So for the last year I've been writing about software, trying to become better at what I do. In the process, I like to think I've become a better writer as well. Apparently, there are other people who think so too, because this last week I was asked to become a featured writer on firelink.com (related to my new experiences as a recruit firefighter). Check the link!

http://www.firelink.com/benefits/articles/6802-firefighter-origins?page=1

What you say and how you say it

2009-03-20T10:29:00.003-05:00

I was sitting in on a meeting with my wife and one of our business service providers a couple days ago, and an interesting thing happened.

We were meeting with two individuals from this company, both from the same department, both with approximately equal authority to get things done. We talked about some problems we were having with their software system, and they told us about the schedule they'd be able to address it on.

After we had left and were driving home, my wife turned to me and said "Which one of those two ladies did you like more?"

I hadn't thought about it, but I immediately said that it was the woman on the left.

"Me too!" my wife said.

We talked about it for a while and realized that both of them were telling us the same thing: "We understand your concerns, we're a little backed up, but we'll get them addressed within X timeframe". However, the woman on the right had mannerisms that were off-putting. She had her arms crossed the whole time, every time we tried to bring up an issue she was quick to come into direct conflict with it, she never smiled. The woman on the left, however, leaned in towards us when she talked, let us finish our sentences, smiled generously, and gave us phrased things with a "we" instead of a "you" in front of them. The first woman seemed to be accusing, the second was accommodating.

The first woman made me want to find another company to work with. The second made me want to return the favor sometime in the future.

It's not the message you're giving that typically affects peoples opinions of you, it's the way you deliver it.

3 Stereotypical Types of Software Labor

2009-03-13T14:32:00.004-05:00

The last month has been really hectic here for both startups that I'm working on. With more and more that needs to be done, there's less and less time for me to get to it all, so the time finally had to come when I said "There's too much for just me to be working on". So we decided to hook up with some contract labor to bridge the gap until we could get a full time employee brought on, but the results have been a mixed bag. That's why I thought I'd put up a post that shows the experiences we've had with the 3 most common types of contract labor. (Please note ahead of time: This is just my honest experience. It does not automatically extrapolate to every person who falls into the categories I will define.)

1. The Cheap Off-Shore Worker

Cool! Software development for less than $10/hr! We couldn't find ANYBODY here to work for that price. When you're in a startup, cashflow is always at the front of your mind, so there's a big appeal to a little less value for a lot less money. When we found a gentleman in India who was willing to work for around half of what we would have to pay local labor, we couldn't pass up the chance to at least try it out.

PROS: Cheap is good. We got a lot of features started that had been waiting on the back-burner while I worked on more important stuff, and for the most part what came back to us was in working order.

CONS: It can be tough to explain what you're asking for when there's a language barrier. Also, frequently the finished work that we received had small problems with it that were not difficult enough for us to want to keep paying someone else to go back in and fix them, but were bothersome enough that they took some of my time to fix before we could deploy them to production.

VERDICT: Good for getting little things done, or maybe for giving you a place to start on larger features. Not to be relied upon for anything mission critical to your application. ALWAYS look over the code yourself before blindly including it in a deployment.

2. The Overpriced Consultant

I had one guy contact me who was very experienced and had a lot of great items on his resume. The problem? He wanted $100/hr to work (but was willing to start at $75/hr). Are you kidding me? Don't get me wrong, I'm sure this guy's very good at what he does, but I wasn't kidding when I mentioned cashflow earlier on. I don't care if this guy has six arms and 3 workstations that he can hack on simultaneously, there's no way he could produce anything that would be worth $100 every hour to a cash-poor startup operating out of the mid-west.

PROS: The gentleman was very nice to talk to, very experienced, and if we had worked with him I have no doubt that he would have produced excellent code and done fine work.

CONS: We have only so much cash, are barely profitable, and any expenditure at that level would quickly decrease our run-way of operation. We just can't afford that kind of recurring expense.

VERDICT: Better leave these blokes for the deep-pocketed mature companies.

3. The Local Student

The dude's never had a shot in the real world yet. Great! He's got lots of academic training, he's probably quite bright, but he won't charge as much as someone who's been in the industry for some time. What a deal, right? Well, maybe. You never know someone's ability level until you've tried them out. In my experience, the huge amount of intelligence you may have in a bright kid doesn't really translate to a great deal of productivity. Why? Well mainly because part of the experience you get from the industry is knowing what things have already been implemented and NOT REWRITING THEM YOURSELF. I don't care if you can implement a super fast bloom filter, what we need right now is the ability to upload pictures for the user's profile.

PROS: Pretty cheap, that's good. Also they're usually on-site, which is to me much easier than working with a remote developer.

CONS: Be prepared to watch every check-in that is made to the repo, because they probably haven't used attachment_fu yet and they may just decide to write their own image attachment library (which is probably very good, but we just don't have the time for that!).

VERDICT: Can consume as much time in supervision as they save you in implementation.

My conclusion: The best contractor we've had is the guy we're going to hire full-time anyway. There's no such thing as a bridge. My impression is that I want the same thing in a contractor that I want in an FTE, but I'm less likely to find it in someone who's not going to have a stake in the outcome. Thank God we're hiring.

a little git

2009-02-13T10:12:00.002-06:00

In a previous post I mentioned that I had switched my SCM solution on the startups I'm working on to "git" away from "svn". Up until now, though, I hadn't really been making the most use of the new powers git gave me. My workflow had been very similar to the way I used to do svn:

checkout the repo (git checkout)
make changes
run pull in any changes that have been sent to repo during my work time (git pull)
commit my changes to the repository (git push)
repeat

Recently I had a need to do more, though. There's one section of one of web applications I'm developing that needs a serious redesign (I mean several days worth of work). So I'm working through it, but then I have a new "emergency" that comes in that requires I deploy some code NOW. Usually, I could solve this problem by making the change in my working directory, and then just selectively adding and commiting that change. However, some of the work I needed to do was in one of the files I had already edited, so there was no way without losing all my work for the day that I could swing getting this fix out (short of checking out from head to a different folder, changing, pushing, and then pulling in to my modded directory again).

Fortunately, git provides a better way:


$> git stash save

I've just saved all my local modifications to the "stash", which you can think of as a temporary shelf you can set stuff on while you're doing branching stuff.


$> git checkout -b working_branch

Now I've created a new branch named "working_branch" on my machine from this repository.


$> git stash apply

My changes that I saved to the stash have now been resurrected and applied to the new branch that I'm working in. I can now make changes and commit to the "working_branch" branch as often as I need.


$> git add .
$> git commit -m "commiting to branch"

Now I need to go back to my master branch to make the emergency change:


$> git checkout master

OK, make my emergency fix, then commit:


$> git add .
$> git commit -m "commiting to master"
$> git push origin master

Now maybe I want to merge these emergency changes into my current working branch so that I can keep the two branches from diverging to far:


$> git checkout working_branch
$> git merge master

Resolve any conflicts, and commit. Done!

There you have it, simple branching in git for great gains in organization and productivity (IMHO).

PDF Attachments with attachment_fu

2009-02-09T15:22:00.002-06:00

The great thing about frameworks like Rails (in my opinion) is that the large number of people doing many of the same tasks results in some really great reusable pieces of code that make common complex features almost trivial to implement. This was brought right to the forefront of my attention again today when I started working on a new feature for our medicaid billing system.

The story is this: We have a physician who writes prescriptions for students who need them. It's better if we don't have to mail him all the documentation for every student he's writing a script for. The best conceived solution was to have school districts upload their docs to our website (associated with a student), and let the physician download them on his side.

Now, surely I'm not the first person to ever want to have uploaded files associated with an entity (that's basically facebook's whole business model, after all).

Sure enough,
attachment_fu to the rescue!

This was dead simple. After installing the plugin (ruby script/plugin install git://github.com/technoweenie/attachment_fu.git), I dug into the docs and started coding.

First I created a new model to house the necessary file for the student (it's called an "IEP", for Individual Education Plan, so don't get thrown by the acronym):


class Iep < ActiveRecord::Base
  belongs_to :student
  
  has_attachment :content_type => "application/pdf", 
                   :storage => :file_system, 
                   :path_prefix => "public/iep",
                   :max_size => 1000.kilobytes

  validates_as_attachment
  
end

Small and simple, right? Here's what it means:
the "belongs_to :student" is the standard ActiveRecord association, indicating that a student entity is the parent of this object (and also implying that the "IEP" object has a foreign key called student_id).

"has_attachment" is the method that configures the file you're going to upload. :content_type, obviously, is the type of file you expect. In my case (and yours if google brought you to this entry) that type is "application/pdf", and if anything else comes in it won't be accepted.

":storage" is how you tell attachment_fu where you want to store these files. Other options besides the filesystem are the database and amazon_s3, but those are both beyond the scope of this article. I just wanted to get these items onto the disk, and then back off on another computer, and the filesystem was the way to go that required the least configuration.

":path_prefix" is the file path you want prepended to the storage location for all files stored by attachment_fu. In my case, "public/iep" is just a directory on my local filesystem, but in production it's a symlink to a location outside of the application folder so that the uploaded files will persist across new deployments (handy strategy for anyone who needs to upload files, thanks to EngineYard for handling that for me).

":max_size" should be self-explanatory. If it isn't, you may be in the wrong job.

Next I set up a couple controller actions to handle the actual uploading of the file:


def upload_iep
  @student = Student.find(params[:id])
  @iep = Iep.new
end
  
def exec_upload_iep
  @student = Student.find(params[:id])
  if @student.iep.nil?
    @iep = Iep.new(params[:iep])
    @iep.student_id = @student.id
  else
    @iep = @student.iep
    @iep.update_attributes(params[:iep])
  end
    
  if @iep.save
    flash[:notice] = 'IEP Uploaded!'
    redirect_to :action => :all_students
  else
    redirect_to :action => :upload_iep, 
                :id => @iep.student_id
  end
end

It ain't restful, but it works. One action to load the form, the other to accept the actual upload. Nothing in there should look different than dealing with regular rails forms. Next we have the view:


<% form_for(:iep, :url => {:action=>:exec_upload_iep}, 
    :html => { :multipart => true }) do |f| -%>
  <p>
    <label for="iep">Upload An IEP:</label>
    <%= f.file_field :uploaded_data %>
    <%= hidden_field_tag :id,@student.id%>
  </p>
  <p><%= submit_tag 'Upload' %></p>
<% end -%>

The only important bits to remember here are the ":multipart => true" which makes sure that the form can accept file uploads, and the ":uploaded_data" attribute which is what attachment_fu searches for when manipulating attached files. Now it's easy to upload an IEP attached with a student anytime. And the best part is, retrieval is a snap:


<td>
  <a href='#{student.iep.public_filename}'>Download IEP</a>
</td>

that "public_filename" method returns the necessary path to retrieve the file, making it easy for the physician to download the IEP on his computer. Done and done. Thanks to technoweenie for writing and maintaining this plugin for the community!

Yahoo killed my buzz

2009-02-04T11:47:00.003-06:00

So I was having a really good week until today. Billing is coming along nicely for our medicaid project, and the other startup I'm working on will be demoing at the Lake on saturday (I'll post a link on this blog after that date, so that I can get feedback from all of you readers as well).

Unfortunately, today my experience with Yahoo really destroyed that "high-on-life" feeling I'd been having. You see, we had this idea for handling incoming email with the application. I wanted to have users be able to send email to the app that would cause things to happen. So I went to our friendly and efficient hosting provider, EngineYard, and asked them to setup a mailbox for us to accept incoming emails. Like usual, engineyard's customer support got done what we needed and did it with a smile. They then told me through email that in order to start receiving email there I would need to add an MX record to my DNS listing that would make "msg.domain.com" point to their mail server. Basically, I just wanted that subdomain to be redirected, while leaving all regular mail sent to "domain.com" to continue as normal.

Figuring that this would not be an issue, I asked our business expert to take care of it. As the title of this post probably indicates to you, it WAS AN ISSUE. This poor woman spent the first half-hour of this project playing with Yahoo's web interface for their small business registrar system. No dice. She called them and spent half an hour waiting, and then longer actually on the phone with a customer service representative who didn't understand what we were asking for. I got on the next call with her and together we talked with a Yahoo support guy named "Phillip" who told us in an apathetic voice that what we were wanting to do simply wasn't possible.

Bullshit! I'm willing to accept something along the lines of "I don't know how to do that", because there are plenty of things that I don't know how to do. But you're going to say to my face that something is downright impossible you'd better be sure that I'm not going to be able to embarrass you with something as simple as a google search: I have Many Online Resources that say this is a perfectly reasonable request that is implemented in DNS records all over the world all the time. Knowing that I have to demo by Saturday, I'm probably just going to have to work around this by directing all email traffic to the engineyard server and forwarding the regular stuff back to our personal email addresses, so I can get past this, but what a freaking drag!

Long story short, don't be suckered in by Yahoo's low prices for domain names. They will screw you if they get the chance. As soon as this demo is over, we're switching. UPDATE: I just checked, and GoDaddy supports this kind of MX record, so that's who we'll be switching to. I recommend you all do the same.

Variable Naming Matters!

2009-01-30T10:48:00.002-06:00

I woke up this morning with a production bug on my plate. Those of you who read this blog regularly probably know that one of the startups I'm working on is a medicaid billing software suite for school districts. The problem I was having was that there were some transactions being added to our billing batches that shouldn't have been. See, in order to bill a transaction to medicaid, several things must be in place: the student must have a prescription, the therapy provided must be one of the federally approved therapies, the therapist must have a federally issued NPI number on file, etc.

These transactions were failing validation, obviously, because they were invalid. They should never have been billed in the first place. When I dug into my code, I couldn't see the problem, although based on the title of this post you will probably see it immediately:


def get_student_detail(s)
  group = s.appts.select{|sa| s.is_medicaid_eligible} 
  if group.size > 0
    claim = Claim.new(group,{:batch_id=>self.id})
  end
end

I spent a lot of time on this. I wrote more unit tests, couldn't figure out why they were failing. I built mocks and tested many underlying methods to prove that they were working properly. Eventually the only way I figured it out was when I kicked up an "irb" session on my production box and started stepping through this code one piece at a time. what an embarrassment when I realized what I had done.

Yes, I went and named my "student" object instance "s". And then I named my "StudentAppointment" object instance "sa". And they both have a method called "is_medicaid_eligible" that returns a boolean value. So, if the student is eligible, all his appointments get billed regardless of whether those appointments are eligible or not, because I mistakenly wrote "s" instead of "sa" in the block on line 2 of the above code. There are plenty of arguments out there about naming conventions and whether they really matter, and I may not be qualified to participate in many of them, but here's one lesson I won't soon forget:

if I had used the name "student" for my student variable, and "appointment" for my appoitment variable, I probably wouldn't have spent an hour trying to find this bug.

Breaking my own Rules

2009-01-27T15:15:00.004-06:00

OK, I'm going to post a link to somebody elses blog post. I almost never do this, because I usually feel like there's a comment section on the other person's blog for a reason, so if you have something to say you should "Say it to their face" so to speak. But in this one case, what I read made so much sense that I had to share it with my audience:

http://www.defmacro.org/ramblings/taming-perfectionism.html

Check it out, and note especially the very last sentence.

Permanent bug fixes

2009-01-24T15:52:00.002-06:00

One of the startups I'm working on right now involves a lot of SMS communication. Threading and queueing messages, although not difficult, can present some edge cases that need attention. One such case was brought to me this week when the application seemed to freeze and error out after a certain message type was sent, followed immediately by a cancellation of the same message.

When presented with a bug, my first instinct is always to "think through" the code that I believe is the most probable culprit. This is usually not the best approach. Yeah, that's an opinion, and I only have anecdotal evidence to support my viewpoint, but this is MY blog so if you don't want to hear my opinion you don't have to read it.

To me there are two important components of a GOOD bug fix. The first is natural: make sure the error/undesireable/unpredictable event stops happening. The second is too often overlooked: make sure it doesn't happen again. Because of the second point, and also because you don't always have a stack trace to show you where the problem is (if, for instance, the problem is not an "error" but an undesireable outcome), I think it's important to suppress the "dig in and find it" urge, and to instead start writing tests. Why? 3 reasons:

1) unit/functional tests allow you to precisely recreate the circumstances under which a bug occured (or under which you believe it to have occured). This means that if you don't yet know what's causing the problem, you can use assertions at every step along the way to make sure every bit of state maintained in the code is as you would expect. In the case of my SMS processing bug, there were many many objects involved in the transaction (because various notifications had to be delivered) and no error was being given so I had no way to know instantly where my data was being monkeyed with. If I wanted to do an "intuitive" diagnosis (as my insticts would have me do) I'd have been dealing with maybe 60 lines of code to look through. By setting up my test case and writing assertions at every point, I was able to isolate the place where the problem was occuring (where my first assertion failed) and thus the number of lines of code I had to apply my diagnosis-thought-process to was exactly one. It's always good to reduce the problem space anyway possible, it can only make your job easier. (As a bonus, if you're using test-coverage as a metric, then you're improving your numbers WHILE bug-fixing)

2) By using tests to diagnose the problem, you also have a feedback mechanism that will tell you as soon as it's fixed. Isn't that handy? When the test is passing, the bug is dead. You might write a few more tests afterwards if this bug has opened up a new path in the code that needs coverage, but basically you won't have to go through the process of "Let's try this, deploy it, did it help? no? Ok, what about this, let me deploy the website again. How bout now? No? Damn! OK, one second". You still might be following the same path of "try this, try that" but your feedback iteration is reduced significantly. Running your test suite takes a lot less time than redeploying the code over and over again.

3) Never experience the same bug twice. With that test in place (as long as your team is disciplined about running the test suite before deployments) that bug is dead for good. Were anyone to adjust the code to the point where it would reoccur (something I've seen happen on more than one project), the test will fail, stopping the release cycle until the problem is solved. You shouldn't ever have to solve the same problem more than once.

When it comes to pest control, your number one weapon should be an up-to-date suite of unit tests.

Advanced Reporting with Excel in Rails

2009-01-19T16:14:00.002-06:00

This is a bit of an expansion and synthesis of two previous posts, one referring to Rails Reporting in Excel, and one dealing with Processing large amounts of ActiveRecord Data. I had to combine those two skills today as our database size is increasing to the point that some of our larger user reports are having to aggregate more data than is pleasant to store in memory at once. In the interest of providing a recipe that others could use, I've pasted an example of this approach on Gist. Essentially, it uses a pagination library (paginating_find, in this case) to page through result sets for reports, limiting the amount of memory space being taken at any one time. Check it out, leave a comment if it helps you.

Simplify, Simplify

2009-01-16T16:15:00.007-06:00

I was presented with a challenge in my work today, one which I was not immediately ready to solve. One of the startups I am working on is a google maps mashup, and a certain requirement we'd decided on earlier involved having certain "markers" on the map blink so that the user's attention would be drawn to them. The following is a sort of "stream-of-conciousness" recording as I thought about making an image on my google map blink:

"OK here's my image:"

"I need to make it blink. It's part of a GMarker object (from the google maps API), so maybe there's some sort of animation API call I don't know about yet....Let's go check the Docs [navigate to google maps API docs].....OK, there's a hide method and a show method, those might be useful.....Not much else though.....I guess I could maintain an array of all the markers that need to blink, and use the "SetInterval" javascript function to toggle their visibility once a second.......yeah, that would work, but if I have a hundred or a thousand markers it might slow down a bit.....still, workable....maybe I'll google and see if anyone's done this kind of thing before....[google for "GMarker blink"].....now we're talking...cool, there's like three open source libraries that actually add a "blink" method to the GMarker class, that's handy....OK, so I just need to plugin one of these libraries and then call the "blink" method only on the ones I want to blink, seems pretty simple....It's going to add a whole new javascript file, but hey, that's not really much of a load time drag....."

By this point I had already constructed the code in my head to an extent, and I was ready to implement it. Then, as I was downloading the library, I noticed something. As I watched the progress bar on this third party website, I was mesmirized by the little circle-spinny-thingy....you know, it was using that usual symbol for "please wait"......the segmented circle that gets highlighted constantly around and around. I always thought that was a cool effect, and I remembered that I'd created that effect a couple weeks ago for one of my other projects. It was pretty neat putting this image together, layer by layer, timing out how long each layer should show for and then saving it as an animated gif.

Then I slapped my forehead, opened up GIMP, and produced this little baby:

Moral of the story, stop and think before you dive into something complicated, there's probably a better way.

Crunch Time

2009-01-13T21:17:00.002-06:00

No matter the industry, or the development methodology, or the size of your company, or the scale of your project, everyone has experienced a time of "push". Call it a hard deadline, or non-negotiable due date, or whatever you want, it all means the same thing: you'll be working extra hard between now and then on little else.

Well, that's where I am right now. We're already sending out marketing information regarding a new product that is theoretically ready but perpetually not quite there, and now the race is on to put the finishing touches together. Stress is high, task list is endless, and I ask myself "Is being part of a startup worth it?"

Certainly there are people who would answer "no" to this question, and you couldn't fault them for it. It is perfectly valid to say "I value my home life enough that I never want my work to encroach upon it", and if that were your point of view than working in a large industry where many people share the responsibility for the success or failure of a project would make perfect sense.

I can't quite go there, myself. It's not that I don't value my home life, it's just that I feel I'm an all around happier person when the things I do at work feel like they have impact: when I have a stake in the outcome, so to speak. I would not suggest that this feeling can't be found in a large company, but I guarantee that when you are the sole technical muscle behind a small and rapidly growing project, EVERYTHING you do carries weight. The cost is just that when it comes to crunch time, you endure it alone.

A price, yes, but one I pay gladly.

Modifying ActiveRecord fields in place

2009-01-05T20:16:00.002-06:00

So here's a "gotcha" for Rails that had me chasing my tail for a few minutes today.

Imagine that you have a model object that is storing a string that is encoded to represent a large amount of data in a small amount of space. For instance, let's say you have a calendar item that is using a string that is 31 characters long to represent the state of each day in the month (this is purely hypothetical, of course). Let's say you want to be able to block of segments of the month at a time, labeling them as "AVAILABLE" or "UNAVAILABLE". You might have a method on your model object that looks something like this:


def mark_days(day_1,day_2,status)
    marker = (status == "AVAILABLE") ? "A" : "U"
    while(day_1 <= day_2)
      self.availability[day_1 - 1] = marker[0]
      day_1 += 1
    end
    self.save!
  end

Cool. Now you run your unit tests, and you notice a strange thing: they're failing. Adding some debug output, you can see that the model object is indeed having it's field updated correctly, and that all the validation is passing, and that the save method is indicating that it saved properly, but when you reload the object from the database nothing has changed!

"Bummer" you might say to yourself. I know I did. After some digging into the ActiveRecord documentation, though, I have found the solution, and I hope that in the future someone who needs it comes across this blog post.

When dealing with a string field on an AR model object, the model will not save that field unless it thinks it has been changed. Usually this is done by direct assignment. For example, if you have a person object named steve, and you want to change his name to fred, you would simply say:


  person.name = "Fred"

When the name attribute is assigned to, AR knows that the field has been changed, and when the next save call comes the name will be persisted. However, if you will note my example above, I'm not reassigning the field, I'm just modifying parts of it in place. This causes active record to basically say "Since the reference hasn't changed, the data hasn't changed, and I don't need to save anything".

Never fear, though, as there's an easy way around it. If you want to modify a field "in-place", you must notify ActiveRecord of the incoming change. Observe this corrected code snippet:


def mark_days(day_1,day_2,status)
    marker = (status == "AVAILABLE") ? "A" : "U"
    self.availability_will_change!
    while(day_1 <= day_2)
      self.availability[day_1 - 1] = marker[0]
      day_1 += 1
    end
    self.save!
  end

By using the "[attribute name]_will_change!" method, you tell ActiveRecord that despite all of the usual indicators, this field has changed and must be saved next time the call comes in. Problem solved, and the tests are passing. May someone benefit from this tidbit of knowledge.

Contract Work: Blackberry Mobile Development

2009-01-03T15:41:00.002-06:00

So my startup is coming along nicely, but now I find myself in a unique situation: we have some work that needs to be done to build a thin blackberry app that does a few basic transactions with our website. Since my plate is getting a little full, I'm interested in contracting the work out, but I don't know anyone personally who's a great mobile developer. If that's you, and if you're looking for an opportunity to make some extra cash, email me at my personal address (ethan.vizitei@gmail.com) and let's talk shop. For the right person, this relationship could turn into a full time position eventually with an equity share.

Big pluses:

-previous experience contracting for small businesses
-worked on mobile apps before, specifically on the blackberry

I know you're out there, future friend. Looking forward to hearing from you.

Happy New Year

2008-12-31T18:41:00.002-06:00

My first post on this blog happened in January 2008. At that time, I figured I'd be doing this for a couple months, and then move on to something else. Instead it's turned into something I truly have enjoyed, as I've gotten all kinds of valuable feedback and advice from readers. As a final post of the year (before I go find some eggnog and an X-Box party), I just wanted to thank everybody who took the time to read what I had to say and start a conversation with me about it. I hope those opportunities come up even more frequently ini 2009.

And now, by the authority of Google Analytics, I give you the 10 most popular posts of 2008:

10)Rails Reporting with Excel Why write functionality into your webapp that already exists in a spreadsheet program that's been around for years? Just give your users the data in an excel sheet, and they can manipulate it any way they want (this post will show you how).

9)Centurytel: What an Embarrassment. Hey kids! Want to see some comedy? Well, everyone enjoys a good rant, so step on up and watch as i totally lose my cool and blast my old ISP for doing exactly what all the other ISPs do all the time!

8)Background Jobs in Rails Have some work that needs to be done in a background thread? Here's a Q&D way to get it done.

7)Maintaining Discipline: A remorseful post I wrote the morning after I decided to go ahead and "get shit done" without writing tests or refactoring. A case study in technical debt.

6)Refactoring in Ruby: You might call this a "lab" in refactoring. Basically I just walked through a real-life example of fixing up some bad code that I had run into the day of that post.

5)Remote NHibernate Apparently I'm not the only person who's ever had trouble getting NHibernate to work over a network. This post doesn't have all the answers, but it was everything I learned when I first attempted it and I've gotten many emails thanking me for the information it does have, so it must be useful to some people.

4)Becoming Less Obtrusive I wrote this post when I finally caught up with the times and started using an unobtrusive approach to javascript. Not much in the way of implementation here, but if you want to know why I write all my javascript this way now, than this post will tell you.

3)Sending email with Rails and Gmail. Ever wanted to have your rails application send out emails, but don't want to configure postfix? Here's your ticket. I don't use this trick anymore because I eventually was sending out so many emails to users that google thought I was a spammer, but if your email volume is relatively low this might be the perfect solution for you.

2)Burning the midnight oil. Honestly I have no idea why this one became as popular as it did. All I was saying was that I seemed to work better at night, but it became a huge deal. If you like midnight-coding as well, check it out.

1) All I need is a programmer: a short diatribe I wrote after a friend of mine expected me to do a programming project for him for free. For some reason it must have struck a nerve with a lot of people out there, because I got a lot of very positive feedback from programmers in the same position.

Thanks again for keeping this blog alive (not through money, because I don't accept donations or run ads, but through attention, because that's really what I crave anyway).

Happy 2009!

ActiveRecord and Large Datasets

2008-12-31T18:01:00.004-06:00

So I ran into a neat problem today. I should clarify right off the bat that when I say "neat" I really mean "ate up all the memory on my server and caused it to slow to a crawl where it was serving requests up at an average of 20 seconds each". Now on a regular day that would be cause for panic, but our product is used by school districts, so we had next to no traffic today, so I had the luxury of considering my problem a "challenge" instead of an "emergency". Here's what I found:

Those of you who are repeat readers of my blog will know that I've done several posts in the past on the topic of generating reports via a background thread so that they don't take up time on one of your mongrels. That's where my problem was. One of the "Reports" I can run as the administrator of my startups product is an audit that checks every transaction ever put into our system and reports on the ones that need attention. The specific business rules aren't important for this post, suffice it to say that we have to filter through every record in our transactions table.

Now, the way I had written this before looked something like this:


District.all.each do |dist|
    transactions = Transaction.find(:all, :conditions=>"..belongs to district..")
    transactions.each do |trans|
         report_on(trans) if (trans.should_report)
    end
end

That's obviously way oversimplified, but the gist of it is there. In previous months this has worked fine, but every month the transactions table picks up another 200,000 or so records, and I guess this month was the breaking point for the audit report. You see, the reason I had split up the transactions by school district instead of iterating through all of them at once was to diffuse the problem of bringing too many records into memory at once. Clearly that's no longer enough of a precaution, because the table has grown to the point where even the number of records for just one school district is just too much to process in memory at once. We need a new solution.

ActiveRecord (IMO) is great as a database interface. What it's not so super at (out of the box) is very large sets of data. Usually that's not a problem for a website, because for almost everything a user needs to do for a website you'll either be dealing with just one single record per action, or a very small set of records that are pertinent to that specific user. Now that I need to deal with millions of records in one go, I've kind of exceeded what AR alone is going to be able to do for me.

However, let's not forget that there is another case common to web-apps where we need to take a dataset in chunks rather than as a whole. What happens when you need to show a user 100 records in a table? Do you just display it all on one page? You can, but most people wouldn't. That's why we have so many great plugins that deal with pagination, and that's what I used to dig me out of this hole.

At first it seemed unorthodox to use a piece of functionality designed to make views more pleasant as an asset for solving my data processing problem, but after thinking about it for a few minutes I realized that the problem I was trying to solve was exactly the same: I want to deal with only small chunks of data at a time.

If you don't have a pagination plugin, I recommend highly the one that I use, Paginating Find. Here's what my code looks like now:


    arr = Transaction.find(:all, :page=>{:size=>100})
    proceed = true 
    while arr.next_page? and arr.next_page! do
        arr.each do |trans|
            report_on(trans) if (trans.should_report)
        end
    end

Done. Now we're only processing the transactions in batches of 100. The running time is a little longer, but it happily uses just a very little amount of memory the whole time, so the web-server can keep on cranking.

Hope someone else out there benefits from the fix!

Binary Search

2008-12-30T22:51:00.012-06:00

This is another journal entry as I follow along with an MIT algorithms class from iTunesU. If you haven't read the first two posts in this series (Insertion Sort and Merge Sort), what I'm trying to do here is record what I'm learning so that I can solidify it in my mind and catalog any information that others might find useful later. As I've said in my previous posts, I'm presenting this information as a student, not as a master, so any corrections would be most welcome.

Today I want to look at the following problem:

I have a sorted array of 10,000 elements. I am asked to find a certain element in the array, and return the index of it's location (or some other value like -1 or nil if it is not in the array). What's the best way to do this?

Well, before I give you the good answer (which has really already been given away in my title), let's look at the naive solution: iteration. This is the easy one to implement, and frankly is the first one that would have occurred to me (although I know better now). Pseudocode for an iterative-searching algorithm follows:


1   define linear_search(array,token)
2      for each i in 0...(array.size - 1)
3          value = array[i]
4          if(value == token)
5              return i
6          else if(value > token)
7              return -1
8      end
9      return -1
10 end

Not bad really, short and sweet. Why don't we do a quick analysis to determine what the runtime is?

Let's see, I want to do a worst-case analysis, so I'm going to assume that the desired token is the last element in the array. Line 2 (the loop initializer) is run exactly 1 time, regardless of the size of the array, so there's one operation, and Line 5 is run exactly one time as well (on the loop iteration when we find the token), so now we're up to 2 operations:

2

lines 3 and 4 will be run once for every item in the array (in the worst case), so thats 2 operations n times added on at the front:

2n + 2

line 6 is run for every element in the array except the last one when it finally finds the token, so that's 1 operation occurring (n-1) times:

2n + (n - 1) + 2

which simplifies to:

3n + 1

So the naive iteration search runs in O(n) [remember, big-O notation is the upper bound/worst case. If the token were found near the beginning of the array it would finish much quicker]. You may want to observe as a side note here that our FASTEST sorting algorithm we've looked at so far (merge sort) runs in O(n log2 n), which is actually slower than our slowest searching algorithm. Be aware that sorting is comparatively expensive, it's just something good to know when working with large collections.

Now, you probably have guessed from the title of this post that there is a faster way to accomplish all this searching, and even if you've never heard of "Binary Search" before, I guarantee you already know how it works. To prove it, let me ask you how you might look up the last name "Johnson" in the phone book. Would you start at the beginning and read through every name until you found the one you were looking for? Of course not! You'd just open the book to about the middle and see where you were. Based on the page you opened to, you'd say either "I'm too far, I need to go back to the left" or "I'm not far enough in, I need to continue somewhere to the right". In that one action, you've eliminated half of the set of names you have to dig through. That action, performed recursively, is binary searching. See the pseudo-code below:


1    define binary_search(array, token, low, high)
2       if (high < low)
3           return -1 
4       mid = (low + high) / 2
5       if (array[mid] > token)
6           return binary_search(array, token, low, mid-1)
7       else if (array[mid] < token)
8           return binary_search(array, token, mid+1, high)
9       else
10           return mid 
11     end
12 end

If you run the benchmarking I've already done for you in my searching code on github, you will see empirically that this is MUCH faster than our original iterative search. But, if we want to do this right, we should go ahead and analyze the binary search algorithm so we can give a mathematical comparison between the two.

First, in order to analyze binary search, we need to produce a recurrence that describes it. Although we did this in a pretty lengthy way in my post on merge sort, the quick way to do this is to ask "what are the recurrent subproblems" and "how much work is done non-recursively in each call". For binary search, this is very easy. As you can see from the pseudo-code above, binary search will decide whether to go to the left or to the right half of the array based on the element it pulls from the middle, so on each successive call it will be considering half of the previous array (n/2). Unlike merge sort, which performed a recursive call on BOTH halves of the array [2T(n/2)], binary search only considers one or the other [T(n/2)]. So the beginning of our recurrence looks like this:

T(n) = T(n/2)

Then the only question we have to ask now is "how much work is being done non-recursively in each recursive call"? Remember, in merge sort we were merging both halves of the array together, so we had to do one operation for each member of the original array, giving us "n" work per level. In the case of binary search, we just do the same 3 comparisons regardless of how many elements there are in the array, so there is always some constant "K" of work being done.

T(n) = T(n/2) + K

Pretty simple. Now, we could use substitution and try to guess what the right answer is (this one is simple enough that it would not be too hard to guess "log n", since the problem set halves every time). If we weren't sure of that, we could use a recursion tree, like we did in merge sort, to show us that we were doing "K" work over "log n" levels, giving us "K * log n" which in Big-O notation would be "O(log n)", and then we could use substitution to prove that log n was a reasonable upper bound. However, the simplicity and form of the algorithm means this is a great time to introduce the "master theorem", a quick way to solve recurrences of this type.

In the lecture series I'm listening too, they made the master theorem sound very complex (it could also be because it's all audio, so I couldn't see the blackboard). After listening to the same section 3 or 4 times though, it began to click. Here's the process:

The master theorem can only be used to solve recurrences of a certain type:

T(n) = aT(n/b) + f(n)

Now, don't balk just because it's a multi-variable equation, it's really quite simple. "a" is just the number of subproblems (recursive calls). in merge sort, we had 2 of these because we called merge_sort once on each half of the original array. For binary search, "a" is 1, as we just call search again on one half of the array or the other. It must, logically, be at least one, otherwise you wouldn't need to be using a recurrence because it wouldn't be a recursive algorithm. "b" is what's used to reduce the size of the problem set on each recursive call. In both merge sort and binary search, the problem set is being divided in half for each recursive call, so "b" is 2. Logically, "b" must be greater than 1, because if it is 1 than the problem set is not getting smaller on each go around, and it will recurse infinitely (or until you run out of memory, pragmatically). "f(n)" is just some function describing the amount of work done non-recursively in each call. As we said a moment ago, for binary search there is a constant amount of work being done for each recursive call, so "f(n)" would be "K" (some constant). Remember that for merge sort, we must merge the two arrays together, which is done in O(n) time, so the work is "n". Looking again at our recurrence for binary search:

T(n) = T(n/2) + K

We can see that binary search does indeed meet the qualifications for the master theorem. It has an "a" of 1, a "b" of 2, and an "f(n)" of K. Now, this is where it get's scary if you are intimidated easily. There are three cases of the master theorem, and you must decide which one your recurrence falls into. Fortunately, it's easy to remember them because they're all concerned with the relation of "f(n)" to the quantity "n ^(logb a)" (it's hard to show complex equations in a blog. That last quantity, in english, is "n to the log base b of a". Although I may be over simplifying, if f(n) is less than "n ^(logb a)", you're in case one. If f(n) is roughly equal to "n ^(logb a)" (multiplied by log n to some power k, and "k" can be 0), you're in case 2. And if f(n) is greater than "n ^(logb a)", you're in case three. Each one of the cases basically tells you what the answer is.

Case 1 => T(n) = O(n^[logb a])
Case 2 => T(n) = O(n^[logb a] * (log n)^(k+1))
Case 3 => T(n) = O(f(n))

Now a full proof and analysis of this theorem is beyond the scope of this blog post. A good resource for more information on the master theorem is HERE. All I want to do is apply the master theorem to my binary search recurrence. So, first I need to compare f(n) to (n^[logb a]) so I can decide which case to use. Remember, we said earlier that our "f(n)", or "work done non-recursively on each call", was some constant "K". Now I need to calculate (n^[logb a]) so I have something to compare it to. For binary search, b = 2, and a = 1, so we're calculating "n to the log base 2 of 1". What power do we need to take 2 to in order to produce 1? 0, of course, so we're looking at n^0, which is 1.

So which case do we fall into? Is K less than, roughly equal, or greater than n^0? Well, K is a constant, n^0 is a constant (1), so they're roughly equal. Looking back, we see that "roughly equal" is defined as equal to n ^(logb a)"(multiplied by log n to some power k, and "k" can be 0)". we don't really need that log n to some power in order to be roughly equal, so we'll use 0 as our "k" (little k, not to be confused with the big K we're using to represent our "f(n)"). Looking at the above statement of the answer for Case 2, I know that our runtime is:

O(n^[logb a] * (log n)^(k+1))

We already showed that n ^(logb a) in our case is n^0, which is 1, so we can make this:

O(1 * (log n)^(k+1))

and of course 1 is the multiplicative identity so:

O((log n)^(k+1))

and we're using 0 for our little k, so:

O((log n)^1)

and 1 is the power identity:

O(log n)

so binary search runs in O(log n). MUCH faster than the O(n) we have for our linear search above.

So there you have it. We've now covered all three methods for solving recurrences, and we're successfully analyzing and comparing algorithms. Pretty neat, huh? Nothing like a little academic exercise to get your mind going in the morning.

As I've mentioned before, I'm implementing each of these algorithms as we discuss them and the code (and the harnesses that benchmark them) are available on github. Also, don't forget that I'm learning all of this material off of iTunesU, and you can too. There are hundreds of interesting courses from top schools available online, for free. It's an incredibly rich opportunity for self-improvement, and there's no reason not to take advantage of it.

Merge Sort

2008-12-30T09:00:00.005-06:00

In my last post, I mentioned that I've started listening through an online lecture series on algorithms, and that I'm recording what I'm learning here both to solidify it in my memory, and to try to present the information in a non-intimidating way so that others who share my lack of formal computer science background (and passion for software) can benefit.

Last time, I demonstrated the "insertion sort" algorithm, and then gave a worst-case analysis that showed that insertion sort performs "n^2 + 2n -2" operations to sort an array, meaning that it scales at On^2. This is an interesting piece of information in and of itself, but it becomes USEFUL when you can compare insertion sort to other sorting algorithms to see which one is a better choice for your present needs. To conduct such a comparison, I'll be showing the "merge sort" algorithm in this post, conducting an analysis, and then comparing it to the insertion sort analysis done in my last post.

Merge Sort

Merge Sort, unlike insertion sort, is a recursive algorithm (if you don't know what that means, the short version is that it's a method that calls itself. Google for more detail if necessary). It has two paths on each call:

1) if the input array has a size of 1 or 0 elements, it's already sorted, and can be returned.
2) if the input array has 2 or more elements, divide the array in half and call merge_sort on each one of them, than merge the two results together.

Pseudo code for this algorithm is below:


0  define merge_sort(array)
1     if array.size <= 1
2         return array
3     else
4         split_index = array.size / 2
5         first_half = merge_sort(array[1...split_index])
6         second_half = merge_sort(array[split_index...array.size])
7         return merge(first_half,second_half)
8      end
10 end
11 
12 define merge(left,right)
13    result = []
14    while length(left) > 0 and length(right) > 0
15       if first(left) ≤ first(right)
16           append first(left) to result
17           left = rest(left)
18       else
19           append first(right) to result
20           right = rest(right)
21    end while
22    while length(left) > 0 
23        append left to result
24    while length(right) > 0 
25        append right to result
26    return result
27 end

Ok, so far so good. Now, how do we analyze this? Well, it's tougher than insertion sort, that's for sure. Because of the recursive nature of the function, it's difficult to just start adding up the number of times a line gets executed as a function of the size of the input array. A better place to start to analyze a recursive function is to write a recurrence for it (A recursive definition of the sequence. ). How does that work? Well, let's take a look at the english version of the merge sort algorithm again:

1) if the input array has a size of 1 or 0 elements, it's already sorted, and can be returned.
2) if the input array has 2 or more elements, divide the array in half and call merge_sort on each one of them, than merge the two results together.

So the time it takes to sort an array is equal to the time it takes to sort the left half of the array, plus the time it takes to sort the right half of the array, plus the time it takes to merge the two together. This can be written as follows:

T(n) = T(n/2) + T(n/2) + (time to merge)

Which simplifies to:

T(n) = 2T(n/2) + (time to merge)

Now what are we going to do with the "time to merge" statement (the one that combines the two sorted arrays)? Do you see why that's easy to take care of? Maybe it would be easier if you saw it written out, rather than in psuedo code; the merge function works like this:

1) take two sorted lists
2) compare the first element in each list.
3) place the smaller element of the two into the output array (and delete it from the list it came from).
4) repeat until each element has been merged into the output array.

You should be able to intuitively spot that there will be exactly as many iterations as there are elements to merge, so the runtime is (x*k*n) where x = the number of operations in an iteration (which I think is 4, in our case), k = the constant time it takes to execute one operation, and n is the number of elements in each of the two lists added together. Thus, in big-O notation, merge has a runtime of On (linear time). Now, to simplify the final equation, we will take the liberty of saying that since each iteration is a constant number of operations, each iteration runs in constant time, and thus it's easiest to say that it takes n "steps" to merge the two sorted lists. This means that our above time calculation can be rewritten like this:

T(n) = 2T(n/2) + n

And this is our "recurrence". Remember, all that means is that the amount of time it takes to merge_sort a list is equal to the amount of time it takes to merge_sort the first half of the list, plus the amount of time it takes to merge_sort the second half of the list, plus the amount of time it takes to merge them together.

Now, how do we take that recurrence and turn it into a formula that represents the growth of the runtime in proportion to n (which is what we need to do if we want to compare it to insertion sort)?

Well, it's not going to be pretty. The only method I know of solving this is to use substitution, and that method basically goes like this:

1) guess the right answer
2) prove it

You can most likely guess that a lot hangs on step one. Now, you may know off the top of your head that merge sort is "O(n log2 n)", but let's pretend that we didn't know it. I might just toss out a guess of O(n^2). What's cool about this is that technically it's correct. Big O notation just tells us the upper bound, like a <= sign, which means that if an algorithm has a runtime of On^2, it also fits into the set of On^1000 because it's runtime will grow proportionately with n at a rate that is LESS THAN OR EQUAL to n^1000. So since we know that merge sort's tight upper bound is n log n, we should definitely be able to prove that O(n^2) is a viable loose upper bound. So, to demonstrate, let's say that I take a guess and say "I think that merge sort will run in less than O(n^2)". Remember, big-O notation removes all coefficients, so what I'm really saying is that the highest order term will be n^2 and it will have some constant C as it's coefficient. This means that I'm saying T(n) is actually less than or equal to Cn:

T(n) <= Cn^2

This could also be said as follows: "If I pick a C that is big enough, merge sort will ALWAYS execute with less than C*n^2 operations". Now, how do we prove it? Well let's look back at our recurrence:

T(n) = 2T(n/2) + n

Since we are trying to prove that T(n) is <= Cn^2, let's substitute in Cn^2 for T(n) in the right side of the equation and solve from there. Now, we don't actually have any T(n) on the right side, but we do have a T(n/2), and since we know that n/2 is less than n, we can safely say T(n/2) < T(n), so at most T(n/2) is still less than Cn^2, so the following inequality will hold:

T(n) <= 2C(n/2)^2 + n

Which can be rewritten to:

T(n) <= 2C(n^2/4) + n

and the leading 2 can be eliminated by dividing numerator and denominator by 2:

T(n) <= C(n^2/2) + n

and to make it easier to read, I can take out the fraction to the front:

T(n) <= (1/2)Cn^2 + n

We're getting close to the inequality we need to prove, which is T(n) <= Cn^2, but we need to get rid of the (1/2) somehow. The simplest solution in this case is to just add (1/2)Cn^2 to the first term, and then subtract the same amount from the back end of the equation, which gives us this:

T(n) <= Cn^2 + n - (1/2)Cn^2

So now we've got Cn^2, but we also have all this trailing junk that's throwing off the inequality. What you need to remember in order to go through this next step is what an inequality really is. Consider the following:

If I asked you to prove that x < 5, but I already told you before hand that x < 3. You have no work to do at all. See, 3 is already less than 5, so anything less than three is less than 5 automatically. We have the same property in play here, when I rewrite the equation thus:

T(n) <= Cn^2 - [(1/2)Cn^2 - n]

It's going to be tough to show that T(n) is less than Cn^2 by itself, but if we can be sure that the quantity we're comparing it to is ALREADY less than Cn^2, then we've no more work to do. So how do we make sure that Cn^2 - [(1/2)Cn^2 - n] is less than Cn^2? well, X minus any positive number is less than X, so as long as everything inside of those brackets comes out to a number that is >= 0, then we know our whole right side is less than Cn^2, and we've proved that On^2 is a reasonable loose upper bound for T(n). So to prove this, we just need to set up the following inequality:

(1/2)Cn^2 - n >= 0

And now we solve:

(1/2)Cn^2 - n >= 0
(1/2)Cn^2 >= n
Cn^2 >= 2n

So now we see that as long as C is greater than or equal to 2, the quantity [(1/2)Cn^2 - n] will be greater than 0, which means the entire quantity Cn^2 - [(1/2)Cn^2 - n] will be less than Cn^2, which means that we've shown On^2 to be an upper bound for the merge sort. Note carefully here that we have not shown that it is the tightest upper bound, only that it is AN upper bound (as is On^3, On^4, On^5, etc). All we've said is that T(n) is guaranteed to be less than Cn^2 if C is at least 2:

T(n) <= 2n^2

So that's an interesting piece of information. However, it's not super useful. Remember, we want to compare merge sort to insertion sort. Insertion sort has a worst case operation count of "n^2 + 2n -2", and the upper bound we've just proved for merge sort (2n^2) is much bigger than n^2 + 2n - 2 for large n. This let's us know that we haven't proven a very tight upper bound, because empirically if you benchmark insertion sort against merge sort for n = 10000, merge sort actually performs at least two orders of magnitude BETTER than insertion sort (download my example implementations from a github and run them yourself for proof).

We need to be able to prove a tighter upper bounds, and it would be helpful if we could do it without guessing.

The way that the MIT course recommends finding out what sort of bound we're looking for involves using what's called a recursion tree to visually picture the algorithm and determine the time spent based on it. For merge sort, this is fortunately very simple, as it's hard to convey something as visual as a recurrence tree on a blog.

The idea is this: We want to compute the amount of time it will take to perform T(n) (the full sort). T(n) is defined as the sum of T(n/2) [sorting the lefts side] plus T(n/2) [sorting the right side] plus n work on the side that is done non-recursively (that's the merging part).

If I were to draw this in a tree, for an 8 element array, it would look something like this:

T(n)
/ \
T(n/2) T(n/2)
/ \ / \
T(n/4) T(n/4) T(n/4) T(n/4)
/ \ / \ / \ / \
T(n/8) T(n/8) T(n/8) T(n/8) T(n/8) T(n/8) T(n/8) T(n/8)

In order to determine the amount of work done, you find the amount of work done on each level, and sum it up. This is easy in the case of merge sort, because if you look at each level you will notice a pattern. The root will have to merge 8 elements, which is our n, so level one does n work. level 2 has two nodes, each one works on 4 elements, which is n/2, so the amount of work done on that level is n/2 + n/2 which is n. level 3 has 4 nodes, each one working on 2 elements, which is n/4, so the total work is n/4 + n/4 + n/4 + n/4. See the pattern? The sum of work on each level is n, so in order to find the total we just need to know how many levels there are, and multiply that number by the work done on each level (n). Since we see that an array of 8 elements has 4 levels, and if we had 16 there would be 5 levels, and if we had 32 there would be 6 levels, we can tell immediately that the recursion tree for merge sort will have (log2 n + 1) levels. What does that give us?

n(log2 n + 1)
n log2 n + n

which is in big-O notation:

O(n log2 n)

Alright! Now we're getting somewhere! Now, we could use substitution to prove beyond a shadow of a doubt that this is a valid upper bound, but for the purposes of this blog post it's unnecessary. What's important to us now is the ability to compare one algorithm's runtime to another. We saw before that insertion sort performs "n^2 + 2n -2" operations, which in big-O notation would be O(n^2). We just showed that merge sort grows at O(n log2 n). Which one is better for a large array? Merge sort, clearly, and the bigger the array the more true it is. On a small set it hardly matters (in fact, on very small sets insertion sort is actually faster than merge sort), but the more elements you add the greater the disparity between the two.

That's it for today, so I'll take this last post to remind everyone reading that I'm approaching this subject not as an expert, but as an amateur. I want to get better, so I welcome your corrections if you find fault in the above, but please don't think that I'm maliciously trying to mislead anyone or that I believe I'm some sort of authority, any mistakes in the above are the honest errors of the apprentice.

Again, code for everything I've covered in this series can be obtained from github:

http://github.com/evizitei/ruby_algorithms/tree/master

Insertion Sort

2008-12-29T14:50:00.009-06:00

I've discovered a really fun new past-time this week as I've been looking for ways to increase my facility and skill in programming without necessarily spending any money (cash is short in the startup world). For those of you who don't know about this great opportunity, go check out iTunesU immediately. Essentially, it's got real lectures from top universities on pretty much any topic you could be interested in (many in video format as well as audio), and all the content I've run into so far has been free. Now, is it as good as taking the class yourself? Probably not, as you obviously don't get college credit, and you can't ask questions to your professor, but for my purposes (pure self-improvement) and with my research ability (I'm pretty sure I can find the answer to any questions I might have online somewhere), I feel like it's something worth my time.

Currently, I'm listening to the first lecture in an MIT course on algorithms, and as an exercise to see if I'm retaining the information at all, I decided to do a little report here on my blog regarding what I learned in that short lecture recording. If it all comes out well, I'll continue throughout the series and see if I just might make a computer scientist out of myself yet. ; )

If you want to brush up on your familiarity with algorithms, or if you're an expert and would like to critique my presentation of the information for accuracy and clarity [I'm looking at you, (cadr life)], I welcome you as a reader.

Is performance important?

The professor for this course began by asking this question, but in the form of "what is MORE important than performance?". Many answers came to the front: usability, features, maintainability, security, and a host of others. The question is than raised, since all these things are often desired above performance, should we care about learning algorithms at all? Of course, since the professor probably wants to give the other 24 lectures in the series, the answer was "yes". He says it's because good performance gives you the currency you need to spend on other necessary aspects of the software, and I'm sure there's a long discussion that could be had about what aspects of improving performance for your software are and aren't important, but for the sake of moving on to the real meat of the course I will avoid spending any unnecessary energy on fleshing out that discussion. Instead, I want to proceed directly to the first example of an algorithm given and the analysis conducted upon it.

Insertion-sort

So far I'm not out of my depth; insertion sort isn't an unfamiliar algorithm to me. You pull elements off the unsorted portion of the array (which initially is the entire array), and insert them at the correct point in the sorted portion of the array (which starts out as none of the array, but grows by one element with each iteration). Some non-standard over-simplified pseudo-code follows:


0   define insertion_sort(A)
1        for i = 1 to length[A]-1 do
2           element = A[i]
3           j = i-1
4           while j ≥ 0 and A[j] > element do
5                A[j + 1] = A[j]
6                j = j-1
7           end
8           A[j+1] = element
9        end
10 end

So, how do analyze this algorithm's runtime in a meaningful way so that we can compare it to other algorithms that accomplish the same thing? (this is where my lack of a formal background is shown, so correct me where necessary, but be nice).

As I understand it, one good way to analyze and compare algorithms is to first assume that each "operation", or "step", takes some constant time, and then count how many times each operation in the algorithm is executed in the worst case scenario (ie, operating on the input that would cause it to perform the most work). What follows is my attempt at performing this analysis on the insertion sort pseudo-code above:

OK, line 0 is a method definition, so we can discard that as part of the discussion. Next is line 1, which is really only executed once (since the loop is not redefined on each iteration). So far, our analysis looks like this:

L1

Now, I think I can look at lines 2,3, and 8 all on the same level, since they are each executed once for each iteration of the outer for loop. Since that loop starts with the second element in the array (index 1) and moves through each element, each of these lines will be executed "n-1" times ("n" being the number of elements in the array to be sorted). So including each of these in the equation, our analysis now looks like this:

L1 + (n-1)*(L2+L3+L8)

The only lines we have left to include in the analysis are lines 5 and 6, but this part is the trickiest of the whole process. You see, those lines are only run as many times as it takes to find the right spot to insert the current element being considered. If that element is smaller than all the elements in the already sorted array, it will have to iterate through every sorted element to arrive at the correct insertion point, but if the current element is BIGGER than all the sorted elements, then those lines won't even get executed once on that particular run through the outer for loop. Thus, in an average scenario, you'll have a variable number of executions for that particular segment. However, remember now that we're trying to consider the worst case scenario. That means that we should assume that the list begins sorted in reverse order, and that each element will always have to be inserted at the beginning of the already sorted portion of the array, thus requiring a full traversal of the sorted portion on each iteration of the for loop. This means that on the first for loop iteration, those lines will be run 0 times, than once, than twice, than three times and so on for each iteration. So on the LAST iteration of the for loop, they will be run "n-1" times (one less than the full size of the array). So now we're looking at the following for our analysis:

L1 + (n-1)*(L2+L3+L8) + (L5+L6)*(0+1+2..(n-3)+(n-2)+(n-1))

Now this probably looks rather intimidating, but it can be simplified rather easily. First, since we said earlier that we were assuming all operations took some constant time, I can replace each of the line numbers with a "k". This gives us the following:

k + (n-1)*(3k) + (2k)*(0+1+2..(n-3)+(n-2)+(n-1))

That's a little better. Next, if you've had any higher math, you'll know that the arithmetic series at the right of the equation (the summation of 1 to n-1) can be simplified as well. I seem to remember that the sum of 1 to n can be written as follows:

(1+n)*(n/2)

if this is the case (and I just looked it up again, so I know that it is) than if we only wanted the sum up until "n-1", not all the way to n, all we need to do is subtract the final term in the series (n):

[(1+n)*(n/2)] - n

I actually prefer it to be multiplied out, so I would write it this way:

[(n+n^2)/2] - n

which gives us the following formula for the length of time in our worst case insertion sort as a function of the size of the input array:

f(n) = k + (n-1)*(3k) + (2k)*([(n+n^2)/2] - n)

Still sort of big and ugly, but stick with me here. The next step towards simplifying it is realizing that we don't want to know the actual runtime, we just want to be able to compare it's runtime to the runtime of similar algorithms. We want to be able to relate it to other algorithms using a common measuring stick. As I understand it, this means we aren't looking for "amount of time" so much as "number of operations". Since k is one operation in our equation, what we are really asking is "how many k's will occur in the worst case scenario?". This means we can move the k constant out of the equation ("divide both sides by k", if you want to think about it that way) and just look at the whole thing in terms of n:

f(n)/k = 1 + 3(n-1) + 2([(n+n^2)/2] - n)

which can be multiplied out to be:

f(n)/k = 1 + 3n - 3 + n + n^2 - 2n

And now we can combine like terms:

f(n)/k = n^2 + 2n -2

Hey! That's a quadratic equation! That's not so bad at all! (Check my math on this, BTW; I'm no pro and have no easy way to validate my answer). Now I can say that for any array of size "n", insertion sort (in the worst case) will execute "n^2 + 2n -2" operations to sort it, and I can compare that function to similar functions for other sorting algorithms to get an idea of what's faster. See? This stuff isn't nearly as hard as it seems.

This would probably be as good a time as any to mention that you needn't go through this whole process when analyzing an algorithm, as the most significant factor in the final performance appears to be the highest order term with disregard for constants or lower order terms. Thus, in the common Big O notation , insertion sort's worst-case scenario analysis is really just "On^2".

Well, this post has probably gone on for long enough. Check back soon, as I'm having a blast with this and hope to continue down this path for a few posts, and I really encourage everybody to go check out iTunesU, there's a ton of interesting material there.

By the way, I had to play around with some code while I was doing this to make sure I understood the sorting algorithm correctly. If you'd like to see my implementation, you can check it out on github here:

http://github.com/evizitei/ruby_algorithms/tree/master

Merry Christmas to Me!

2008-12-23T17:53:00.003-06:00

It's funny how my RSS reader structured things for me today. The first article I picked up was one from a developer's blog where he talked about how Rails was slowing down in it's evolution and was beginning to frustrate him as it failed to reinvent itself in the same exciting ways it did when it was cutting edge. I could sympathize with that to a point. After all, that's a natural consequence of having more and more people involved with and dependent on the framework in question. It was humorous to me, though, that the very next article I looked at was the announcement that Merb 2.0 and Rails 3.0 would be the same entity. Woot!

Merb was built after rails, and learned a lot of lessons from it's development, and now those lessons are going to be brought back to their source as all the performance optimizations and component-agnosticism that make Merb such a great framework are brought into Rails to allow the old workhorse to make a major leap forward. Looks like the beta will be attempting a release around May, but I'm sure that one way or another we'll have a functional version to play with by the end of next year. Hooray for cooperation!

Ramen Profitable

2008-12-23T14:10:00.003-06:00

Big news from the startup I'm working on: we've hit a major milestone. After about 8 months of work, we've finally reached the level of income that Paul Graham describes as Ramen Profitable. Yes, this month for the first time we'll actually take in slightly more money than we spent. No telling if this will continue steadily, but it's a very encouraging sign. Theoretically, as long as we contain our expenses, this gives us the runway we need to stay alive while we continue to bring on new customers and realize the full potential of the ones who are already on board with us. So in my house, we'll be celebrating more than just the holidays this year. ; )

Branching Out

2008-12-23T09:30:00.002-06:00

Just a one time announcement for readers who have an interest in other areas of my life besides technology: I started a second blog yesterday on blogger that will be following my attempt to become a member of the local volunteer fire department. If that kind of thing interests you, head on over and take a look.

How to advance medical science while you sleep

2008-12-16T17:49:00.002-06:00

A friend of mine recently sent me a link to BOINC. If you're familiar with the program SETI@home, than you already know about this middleware. Basically, BOINC is the framework that allows you to donate your unused computer time (via a screensaver that kicks in when you aren't working) to the CPU-intensive project(s) of your choice. That's right, you can contribute to furthering science just by not using your computer from time to time (if you're one of those people who requires sleep, this is particularly easy since you already have 1/3 of the world's rotation cycle where your computer processor is unused that you can give away to needy science projects).

Personally, I have all three of my computers running Rosetta@home. The full description of what they do is here, along with the medical relevance of the project, but the short version is as follows:

Proteins are made up of amino acids. There are 20-some amino acids used in the construction of proteins, and each one of them has a different shape, and exerts different forces on it's surroundings. The forces exerted by the amino acids cause the protein (which could be thought of as a chain of amino acids) to take on a particular shape (which they call a "fold"). The basic goal of Rosetta@home is to run through many likely folds for each protein, trying to produce the one with the lowest energy, which is the greatest predictor of the proteins true shape. The results of this research can potentially be applied to problems ranging from Malaria, to Cancer, to HIV. As you can imagine, this is a CPU-intensive project, so distributed computing works quite well.

Hence, it's a good cause, and you've got a computer anyway that you don't use 24-hours daily. Why not give?

Internet Explorer and the Z-Index (or, how I wasted a whole 3 hours of my life)

2008-12-11T12:47:00.004-06:00

As my most recent post mentioned, I frequently get frustrated over small inconsistencies between the way firefox/safari renders my projects, and the way they look in internet explorer. However, as any pragmatic web-developer will tell you, IE ain't leaving anytime soon, so I didn't have the option of just telling all my users to switch browsers (though it would have pleased me greatly). Fortunately, IE-8 looks like it will be fixing a lot of the quirks that IE has had in the past. UN-fortunately, IE6 is still hanging around on many corporate/government/home systems as a plague to all developers, even a couple years after the release of IE7, which means that if we follow the pattern than we won't be able to rely on windows users having IE8 until a full decade after it's release. So, until that happy day arrives, here's a little tip that may save someone out there time when they are having a bad day with IE.

z-index

I've got this layout I've been working on for my startup. It has a header, and three vertical columns on the page (should seem familiar, this is a very common layout scheme). At one point about halfway down the page, I have this really large element in the center column that expands a bit over the borders of the column it's in. That's totally intentional, because my design called for this large element to overlap and stand in front of the edges of elements in both of the side columns, making it look kind of like the top-most magazine clipping in a child's collage.

Now, the way I was achieving this effect was through the use of the "z-index" css property. If any of you readers are software developers who are not familiar with HTML/CSS, the "z-index" basically is an integer that you use to guide the browser into placing elements "on top" of each other (with respect to the perspective of the viewer. x-axis runs left to right, y-axis runs top to bottom, z-axis runs front to back). The higher an element's z-index, the closer it is to the viewer, or the closer to the "top of the stack" it is. For the sake of example, let's say I gave the elements in the left and right columns of my layout all a z-index of "1", and then gave my large center element a z-index of "2". In Firefox and Safari, this works no problem. Open it in IE, and God only knows what will be on top of what.

Now, I had no idea what was going on when I saw this, and it took quite a bit of research for me to figure out what the problem was. Aren't you lucky that you came across this blog post before YOU had to do that kind of research? I'll answer that for you: "Yes, you are very lucky". I can say this confidently because the answer is so stupidly simple that you would have cried if you had spent hours trying to find it (I know I did).

Internet explorer (even version 7) does not respect the z-index property when the elements in question have different parent elements. In my case, the "div" tags that represented my three columns didn't have z-indexes assigned to them. The elements in the center column were children of one div tag, the elements in the left column were children of another, and the elements in the right column were children of a third div tag, none of which had z-indexes assigned to them. Basically, it seems IE only uses z-indexes of children with the same direct parent to calculate positioning. The fix? I gave my columns Z-index properties ("1" for each of the flanking columns, "2" for the center) instead of the elements inside of them. Since the columns were siblings, IE was finally able to figure out what was on top, and now it looks fine in all three browsers. So there you have it, a special fix for a special browser.

UI Pain

2008-12-09T16:52:00.002-06:00

Time for a mini-rant.

Everytime I get something working just the way I want it: looking a certain way, positioned a certain way, colored a certain way. Everytime, I end up slapped in the face as soon as I look at it in Internet Explorer. It's like it's actively rejecting my work, telling me that it just isn't content to render content the way the other browsers do (Firefox and Safari, of course). No; instead it requires custom attention and special handling, kind of like a worthless train-wreck of a person strung out on crack who happens to be the nephew of a powerful politician.

Screw it, I guess I've got more work ahead of me.

Scaling Images with GIMP

2008-12-09T08:38:00.006-06:00

Here's a UI tip for all of you who are software developers but happen to need to get something done with an image.

As readers of this blog probably know, my primary skill-set is software. That's what I've spent the most time learning, and it's what I'm most comfortable with. In previous jobs, when it came to the User Interface, I'd have an HTML template handed to me, built by a talented web designer, and then I would make it actually do stuff. But now, working on a startup, we don't have a dedicated UI guy. I AM the UI guy. I'm also the software guy, the server guy, the internal IT guy, and the customer service guy. Because of the position that puts me in, I've learned a lot about web user interfaces in a sort of trial-by-fire pedagogy, and this morning I figured something out that just hadn't occurred to me before, and thought I might share it with other "non-Ui-guys" who are in the position of doing the best they can.

For a web page I was working on, I had three images that had to be resized way way down to fit up in a banner. I have a passing knowledge working with GIMP, so I fired it up and cropped and scaled until it was about the right size. But you know what? It looked terrible. It was so pixelated that you couldn't even tell what it was by just looking at it (a speedboat, for the curious).

Not knowing what else to do, I started redoing the process from the beginning, using slightly different configuration settings, trying to scale down less, saving to different formats, and it just looked bad and worse. I had to actually THINK for a minute before I figured out what the deal was. Basically, when you scale an image using GIMP, you have the following dialog box to work with:

Notice the resolution settings, as I just didn't mess with them as I was scaling. If you think about it, of course you will lose quality as you shrink a picture. Why? Well if you leave the resolution at a constant 72 pixels per inch, then scale the picuture down by half, it can only fit every other pixel from the original image into the new one, so you've lost half the data of the original file. What I did instead was to increase the resolution by the same percentage that I was decreasing the size of the picture. Thus, if I was making it a 4th of the original width, I quadrupled the number of pixels per inch along the X-axis (and likewise matched the ratio of heights to the Y-axis resolution), thus preserving all the data from the original picture as the size shrank. What resulted was a much sharper and clearer tiny picture, that now looks great on our webpage.

Overall word to the wise: I was about to just hire a "professional" to do this stuff for me, and if it was a really complicated layout that I needed it might have made more sense. But for something simple, it's not to challenging to just sit down and grind out something that will work. If you're smart enough to write good code, I guarantee you that you are capable of understanding HTML/CSS and image editors. At least give this stuff a shot before you outsource it, it might not be as challenging as you think, and when you're bootstrapping almost any cash savings is a win.

Cross Training

2008-12-07T07:40:00.005-06:00

I spend most of the day, every day, writing RoR code for my startup. It's fun, to a point, but sometimes you need to take a break. I've got a list of hobbies, things I usually do in my free-time: Rock-Climbing, Piano, etc. But recently I've been into video-game development as my down-time filler, and so I've found some new fun things to do in that sector. One of those things is 3D-modeling.

Now I'm not going to claim to be any kind of authority on this subject. I've been playing around in Blender for maybe 8 hours total, and my lack of artistic skills is apparent to no-one so much as me. Nevertheless, I've quickly come to appreciate just how much work artists have to do in order to make the spectacular renderings I've seen in most of the video-games I own.

I think that one of the most valuable skills a person can have is the ability to learn new things. As I've said in previous posts, being an expert in your craft is great, but I'm more interested in a person who's an expert at becoming an expert. And what's the best way to get good at learning new things? By learning new things, of course! So that's why I work on side projects like this. Not only are they enjoyable, but I think they make me better at my job because they give me practice in continuing the evolution of my skillset.

Anyway, I've been having a great time going through tutorials and resources on 3D modeling, and I finally have my own first project that I did without any tutorial or guidance. Yes, I'm proud of it; it's the kind of proud that a 7-year-old feels when he makes a clay pot for his parents not realizing how much more advanced the art of ceramics is beyond his skill level, but proud nonetheless. So here it is, my primitive model of a Colonial Viper Mark II from the re-imagined Battlestar Galactica series:

Now, before all you real artists pounce on me for my amateurism, let's remember that no field would ever be advanced if not for the encouragement of amateurs.

Also, you've been doing it a lot longer than me, so back off, k? ; )