Setting up Background Job (BJ) Properly

June 23rd, 2008

I dealt with a customer today who was a bit miffed that his Background Jobtasks were taking a while to kick off after the Rails app submitted them to the queue.

I dug a little deeper into the problem - he was using our standard configuration, moving control of the runner out to cron instead of being controlled from within the app itself.

1

Bj.config["production.no_tickle"] = true

and then a cronjob such as

*/2 * * * * bj run --forever --rails_env=production --rails_root=/data/user/current

Read the rest of this entry 0 comments »

Hpricot Case Sensitivity

April 24th, 2008

Recently we were trying to use the awesome Hpricot to do some HTML parsing. Problem is that Hpricot doesn't easily allow you to do case insensitive searches for elements.

This means that we're missing any element where someone has placed an uppercase character in the tag we're explcitly looking for.

1
2
3
4

>> h.at("meta[@name=description]")
=> {emptyelem meta name"description" content"David Smalley, Ruby hacker based in Leeds, UK"}
>> h.at("meta[@name=Description]")
=> nil

This problem has been talked about on the Mofo Mailing List and even on the Hpricot challenge page - but I didn't really like the look of those solutions as downcase-ing everything would distort the original HTML.

So I devised this (horrible and hacky) bit of code that loops through all element names and attributes, downcase-ing them all without touching any innerHTML.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

  def normalize(element)
    element.children.each do |child|
      if child.respond_to?(:name=)
        child.name = child.name.downcase if child.name
      end
      if child.respond_to?(:raw_attributes=)
        attribs = {}
        child.raw_attributes.each_pair do |key,value|
          attribs[key.downcase] = value.downcase if value
        end
        child.raw_attributes = attribs
      end
      normalize(child) if child.respond_to?(:children)
    end
    return element
  end
>> h = Hpricot(open("http://davidsmalley.com"))
=> Hpricot.....<snip>
>> normalize(h)

Now all your element names and tags will be downcase-ed and all innerHTML will be left alone. Comments, feedback and suggestions very much welcome!

Shout - at a glance status

April 21st, 2008

Inspired by a recent 37 Signals blog post A peek at In/Out, an internal app at 37signals, we decided this would be great for the Litmus team as well. A quick glance overview of what everyone is working on at this moment in time.

Not wanting to go too overboard and create something we never went and updated, we decided to tie it into Twitter which we all use anyway. Direct Messages fit the bill perfectly as we weren’t too keen on broadcasting the inner workings of Litmus across the internet.

Merb was the framework of choice for this little web app. Using the Twitter Gem to check the direct mail of an account we set up, it then republishes the messages to a page we can all view inside our management app. Merb worked a treat, and we got the app up and running in a couple of hours, a quick style overhaul from Paul and we had the app live and in use from today.

We hooked the app into a Fluid browser and Paul knocked up a little applescript app that posts a direct message to the correct Twitter account.

We’re all really impressed with the result…

The plan is to release this app via Github once it has been cleaned up a bit and properly tested, check back here for details of its release.

I should point out that while we were playing around with this we called it “in/out” (that’s what appears in the screen shot) but obviously the fine people at 37 Signals have already used this name. Therefore our app is known as “Shout”.

Exciting times

April 9th, 2008

This last weekend I was up at Scotland on Rails – met loads of great people and saw lots of interesting talks. Well done to the organisers for putting together a Ruby/Rails conference in the UK. It had been a long time coming since RailsConf Europe came (and went) to London a few years ago (as an aside, is this always going to happen in Berlin now? I had hoped it was going to do a grand tour of Europe).

At the conference I was offered a job with Engine Yard and I’m delighted to say that I accepted the offer and will be starting in May!

Big thanks to Jamie van Dyke for giving me the opportunity.

Extension-less Formats in ActiveResource

March 26th, 2008 When trying to integrate Litmus with Fetch recently we came across an API that whilst appearing RESTful, didn't quite map onto ActiveResource conventions because their URLs 404'd if we included .xml as the extension.

So I quickly whipped up a little plugin to let us specify new formats that don't include extensions. 

1
2
3
4
5
6
7

class Something < ActiveResource::Base
  
  self.site = "http://someone:password@an-app.com"
  self.prefix = "/api/"
  self.format = :xml_no_extension
  
end


Now, calls to 


Something.find(:all)
will generate http://an-app.com/somethings rather than http://an-app.com/somethings.xml

Perfect!

You can grab the code from the ExtensionlessFormat GitHub repository

New Async Observer Start/Stop scripts

March 6th, 2008

I've updated the Async observer start/stop scripts that I've been using extensively for pushing off long running tasks in Litmus. Combined with a few extra cap tasks this has been working perfectly for us for several weeks now.

Lots of the code is pulled out of the ferret_server and spec_server code from the Ferret and Rspec projects respectively. Download it here async_worker.zip

and get some Cap tasks from below... 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

namespace :beanstalk do
  
  task :start, :roles => :app do
    run "/usr/bin/beanstalkd -d -l 127.0.0.1"
  end
  
  task :stop, :roles => :app do
    run 'killall beanstalkd'
  end
  
  task :restart, :roles => :app do
    stop
    start
  end
  
  namespace :worker do
    
    task :start, :roles => :app do
      run "cd #{deploy_to}/current && RAILS_ENV=#{rails_env} script/async_worker start"
    end
    
    task :stop, :roles => :app do
      run "cd #{deploy_to}/current && RAILS_ENV=#{rails_env} script/async_worker stop"
    end
    
    task :restart, :roles => :app do
      stop
      start
    end
    
  end
  
end

Leeds Ruby Thing

February 19th, 2008

After a fantastic meet-up earlier this month, the Leeds Ruby Thing has a website and, hopefully, a regular monthly slot down at the wonderful Mr. Foley’s on The Headrow in Leeds city centre.

For some reason I am grimacing in this photo from the last meet-up.

Ruby on Rails: Quick tip for Find :conditions

January 11th, 2008 I stumbled across a sneaky tip for conditions when using find. When you might normally be doing this: 

1
2

Account.find(:all, :conditions => ['name LIKE ? AND updated_at < ?', 
"aname", 3.days.ago])
You can just do this instead: 
1
2

Account.find(:all, :conditions => ['name LIKE :name AND updated_at < 
:date', {:name => "aname", :date => 3.days.ago}])

This isn't the best example of using this symbol placeholder method, but imagine using it in a situation where you're repeating the same search param a few times. I know I've had a few times that my conditions has been something like 
1
2

['created_at > ? AND updated_at < ? 
AND invoiced_at > ?'], Time.now, Time.now, Time.now].
Using the hash at the end you could just pass this in as :time => Time.now and reuse :time in multiple places in your conditions hash. Neat.

Update: See ActiveRecord::Base on Noobkit for more info

Async Observer start/stop scripts

January 11th, 2008

Recently I’ve been adding some filters to Litmus which are firing off calls to remote services based on actions performed by our customers. Unfortunately this started to bring some “Timeout::Error” exceptions bubbling up when the remote services didn’t respond in time.

The functions these remote services provide are easily hidden if we get a failure when trying to contact them, the problem for us is that this meant a long wait time for customers whilst we tried to contact the service, followed by a 500 error once the connection had timed out.

I read about beanstalkd – its ability to asynchronously process jobs from a memcached like queue would be the perfect solution to the problem, allowing us to take a shot at reaching the remote services but not tying up mongrels and making our users wait.

Using it with Rails was easy using the async-observer plugin – I managed to get the whole thing up and running in around 30 minutes.

When it came to deploy I noticed that the async observers were in need of scripts to start and stop them so I hacked up a couple of scripts you can stick in your scripts directory and chmod +x ready to use in Capistrano.

I’ve literally just finished these scripts and they’ve had about 5 minutes of testing, so all comments/suggestions/corrections and even your own modifications posted elsewhere are welcome. Just drop any links/suggestions in the comments.

async-start-stop.zip

Update: I’ve just noticed that the “puts” returned by the async observer is causing Capistrano to hang during deployment. I’ll have to have a look at it when I get a chance. Any suggestions welcome. Ideally the scripts will still output some text to let you see something is happening during deployment…

Update: I’ve updated the script and it now works really well for us… get it here

Convert OPML into rspec spec files

August 20th, 2007

I’ve been doing lots of work recently by writing up my specs in OmniOutliner and then converting them into spec files before I start writing my code. I knocked up a quick ruby script to handle the conversion for me.

First off, as I write my specs in a certain way in OmniOutliner the script follows my own conventions so will probably need modifying if you use it in a different way. In OPML form they should look like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

<?xml version="1.0" encoding="UTF-8"?>
<opml version="1.0">
  <head>
    <title>billing_specs</title>
    <expansionState>0,1,5,9,10,14,18,19,23,26,27,34,42,48,57,60,69,78,85,89,94</expansionState>
  </head>
  <body>
    <outline text="Account">
      <outline text="when placing a first order">
        <outline text="should do something"/>
      </outline>
      <outline text="when upgrading">
        <outline text="should do something"/>
      </outline>
   </outline>
  </body>
</opml>


This will produce a spec file like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

describe Account, "when placing a first order" do

  it "should do something" do

  end

end

describe Account, "when upgrading" do

  it "should do something" do

  end
end

And there you have it, this has speeded up my development and saves me the tedium of rewriting everything into spec files.

You can download the script here – you’ll need to have the ‘xmlsimple’ gem installed for it to work

Standardise flash messages across your app

July 19th, 2007 In the admin areas of apps I have been writing recently I've found myself writing the same block of code over and over. In these pretty mundane admin areas, after any action has completed or a rescue block has been triggered for ActiveRecord::RecordNotFound I like to show a quick "XXXXX Record Created" /"Could not find XXX with that ID" and then redirect people back to a safe place.

To solve my problem I created a double team-up of TextMate snippets and an addition to ActionController to make my coding quicker, and to standardise these messages across the whole app.

Read the rest of this entry

Ruby: Require everything in a directory

July 3rd, 2007 Recently I've been working a lot with the Liquid templating system and in our application initialisation we needed to quickly require all the files in a particular directory that aren't in the normal rails load path. Normally a good way to do this is to just add the new paths onto your Rails load path 



$LOAD_PATH << File.join(File.dirname(__FILE__), '../app/drops')

or, in environment.rb 


config.load_paths += %W( #{RAILS_ROOT}/app/drops #{RAILS_ROOT}/app/filters )

This particular app uses engines so most of the configuration resides in our core plugin in order not to dirty up environment.rb and make is easier to keep all the configuration centrally managed. This means we can't really touch environment.rb or anything in the Rails initializer block for that matter.
Then I discovered the power of .glob 



Dir.glob(File.join(File.dirname(__FILE__), '../app/drops/*.rb')).each {|f| require f }

Perfect.

Edited: Yes - that second snippet of code is from Mephisto - at the moment it's the best reference for how to implement a liquid based templating system. Thanks Rick, you've really helped us out!

UK Nova Signup Tool in Ruby

April 16th, 2007

After giving the Paul Bain UK Nova Signup tool a bash and not ever receiving a confirmation email to start using it, I thought I’d just write my own in Ruby and so far it seems to work!

I’ve included the script here to do with as you please, but please make sure you do not use this script for evil! In my mind I don’t see much wrong with refreshing a page and filling in a form programatically – the script should only check the page between every 30 – 60 seconds.

You’ll need to run it on a Windows box with the Watir and Hpricot gems installed, just call: ruby novasignup.rb username email password and sit and watch the log.

Download novasignup.rb

ActiveRecord Delegation and demeter

March 18th, 2007

A while ago I looked into the various aspects of the laws of demeter sparked by a post I read whilst trying to find out why Mephisto had the word “delegate” sprinkled throughout it.

If you haven’t come across demeter before then its worth reading the post I linked above and the Wikipedia Article

To sum up the concept in one short sentence, the magic that AR gives us that allow us to, for example, say: person.address.city is a dangerous thing to do. It’s all lovely and magic when its working but as soon as our person doesn’t have an address every reference to that throws a lovely nil.city error and promptly blows up our app.

Rails provides a delegate method which seems to have wrongly been assumed to handle this and protect delegated methods from returning nil errors but this isn’t the case. Using delegate you would expect to say:

delegate :city, :to => :address

and then expect to be able to call

person.city

without it all blowing up horribly if there is no address object attached to the person.

Well, after moaning about this to Dave we devised a new AR mixin called ar_delegate that lets you safely delegate methods to other classes and protect against the horrible nil errors that occur when you start nesting relationships using AR magic.

Dave has really come up with some lovely syntax that feels Railsey and hopefully we’ll be able to get this into core if it looks like its a good fit (and we write some tests). All credit to Dave for the code, I was merely an IM-advisor as he was writing it.

See ActiveRecord Delegation over at Dave’s blog for a much better write up and links to the plugin.

Rails Plugin - Shiny-render

December 21st, 2006

Dave of Shiny Development and Afeeda fame has released a new plugin for Rails called Shiny Render

It’s a simple patch but one I think would be quite suited for merging with simply_helpful

Basically it allows you to pass an HTML tag to wrap around any collections that are rendered in a partial and suppresses the tag if nothing is rendered.

The example Dave gave me was:

<%= render partial => "tag", :collection => current_user.tag_names, parent_tag => 'ul' %>

Very nice, visit the RubyForge project for Shiny-render to submit your patches and ideas to Dave