Distributed Processing with EC2 using PoolParty

09Jul08

So PoolParty seems nice.  Tonight I’ll be attempting to use it (primarily because it gives me shared disk across instances via S3 for free) to replace a script’s use of Sun Grid Engine (SGE) with EC2.  This happens to be all I need to pull this off fairly easily.

So I’m using this to parallelize FSL, a bit of software that analyzes fMRI data.  Specifically, I’m modifying the parallelization function to handle EC2 on top of the Sun Grid Engine support it already has.  The current process ends up with a file called commands.txt.  Each line needs to be run on its own EC2 instance with access to the shared data directory, in which the output will be placed.  The current shell script will then take over, as that’s the behaviour that SGE ends up with for this task.

To help with this, Ari (the poolparty guy) wrote up a nice PoolParty Plugin for this very task.

We’ll ignore the actual use case (just makes a good story) and simplify the problem to: launch a pool party that outputs a command to /data/log/output1, output2, etc.

My commands.txt file looks like:

ls /bin > /data/output1
ls /root > /data/output2

Then it’s just a matter of pool -v -i -I ami-1234 -b shared_bucket.  This launches the appropriate number of instances and at the end of the process all of the data has been written to the appropriate place.

I actually wrote no ruby code for this.  Ari wrote the plugin that handled my whole use case, but it was really nice to see how tiny the plugin can be.  Ultimately, this gem looks intensely interesting for our generic use case, which was simply distributed processing of a massively parallelizable computation.  It completely replaces the need for Sun Grid Engine in this case.

Everyone join the Pool Party.

NOTE: Documentation is sparse at present, but Ari is available on Freenode#poolpartyrb and I would be glad to help anyone that wants to talk about how to use PoolParty in whatever small way I can.  Ari has said that this week is to be documentation week for the project.  And go donate to the project if you can.

Filed under: Uncategorized   |  1 Comment
Tags: Amazon, AWS, EC2, pool party, ruby

Gitting git-svn working on mac

17Feb08

I woke up fired up and ready to try git; unfortunately, it is three hours later and I’m just gitting started. The problem was getting git-svn to work. Despite the numerous posts including: post1 post2 post3 and post4 I absolutely could not get it to work. Finally, here are two techniques that worked for me:
Port Method
sudo port install subversion-perlbindings
sudo port install git-core

or Method2

follow Fraser Spiers post

except

export PERL5LIB="/opt/subversion/lib/svn-perl:$PERL5LIb"

In other words, the package installs the subversion perl bindings in /opt/subversion/lib/svn-perl not /usr/local/lib/svn-perl.

Time to start gitting it done!

Filed under: Uncategorized   |  1 Comment

First contribution from Brian: a class for recursive time-based operations, and our use in a Rails project

20Nov07

by Josh Adams

Ever wanted to be able to do something like an Outlook-style recursive event generator? Say, schedule an event each MWF for the next six weeks from 8:00 – 8:30 AM? There’s a nice library called RUNT that I ran across today that makes this kind of temporal work easier.

Then Brian, the newest hire at isotope11, created a library to ease the particular use case we had, which was using a form similar to the following:

The code for the GroupEvent class is as follows:

require 'runt'

class GroupEvent
  include Runt
  def self.schedule(start, finished, weekdays, type, &date)
    case type
    when /w/i
      self.weekly(start, finished, weekdays) { |day| date.call(day) }
    when /b/i
      self.biweekly(start, finished, weekdays) { |day| date.call(day) }
    when /m/i
      self.monthly(start, finished) { |day| date.call(day) }
    when /y/i
      self.yearly(start, finished) { |day| date.call(day) }
    end
  end 

  def self.weekly(start, finished, weekdays, &day)
    day = nil
    exp = nil
    for i in weekdays
      if (exp.nil?)
        exp = DIWeek.new(i)
      else
        exp ||= DIWeek.new(i)
      end
    end
    for day in (start..finished)
      if(exp.include?(day))
        yield(day)
      end
    end
  end

  def self.biweekly(start, finished, weekdays, &biday)
    self.weekly(start, finished, weekdays) do |biday|
      if((biday.day%2) == (start.day%2))
        yield(biday)
      end
    end
  end

  def self.monthly(start, finished, &month)
    current = start
    begin
      yield(current)
      current = current.to_time.months_since 1
    end while(current.to_time < finished.to_time)
  end

  def self.yearly(start, finished, &year)
    current = start
    begin
      yield(current)
      current = current.to_time.years_since 1
    end while(current.to_time < finished.to_time)
  end
end

Inside the controller that form points to, I call the following (*horrendously ugly*) method:

  def create_events
    if params[:recursive]
      start_date = Date.new(params[:start_date][:year].to_i, params[:start_date][:month].to_i, params[:start_date][:day].to_i)
      end_date = Date.new(params[:end_date][:year].to_i, params[:end_date][:month].to_i, params[:end_date][:day].to_i)
      GroupEvent.schedule(start_date, end_date, params[:days].map(&:to_i), params[:period]) do |date|
        starts_at = DateTime.new(date.year, date.month, date.day, params[:start_time][:hour].to_i, params[:start_time][:minute].to_i)
        ends_at = DateTime.new(date.year, date.month, date.day, params[:end_time][:hour].to_i, params[:end_time][:minute].to_i)
        DateEvent.create( :starts_at => starts_at, :ends_at => ends_at, :description => params[:event][:description])
      end
    else
      @event = DateEvent.new(params[:event])
    end
  end

Please, please forgive me for that code. It was well past five and we just wanted to get it working, okay? That code was mine, the good looking stuff above was Brian’s. Don’t pin my trespasses on him.

Anyway, the code appears to do exactly what we want so far, haven’t found any quirks yet. We’ll get better test coverage of this stuff tomorrow.

Filed under: Uncategorized   |  1 Comment
Tags: rails, recursive events, ruby, time-based

RSS to synthesized speech in 47 inefficient lines

07Aug07

A few posts back I mentioned a ruby aggregator in 26 lines of code. I’ve spent the last fifteen minutes or so tying it to festival so it can read out RSS feeds.

If you just want to read the script fast, click here.

The code’s all there, but the coolest thing is the library I found, FestivalTTS4Ruby.

To actually run the code, you’ll need festival installed (I just shell out to it). Fire up irb, and:

require 'rss_to_speech'
RSSToSpeech.new

That will read in each line of feeds.txt, suck down the posts for the feed, pipe it through a python script called html2text.py, then pipe that through festival –tts. More lines than I’d like it to be, but I don’t feel like working on it any more

Filed under: Uncategorized   |  1 Comment

Ruby Require Idiom

21May07

I just posted an article about ruby’s require on my blog: http://ghouston.blogspot.com/2007/05/ruby-require-idiom.html

It is fairly basic ruby stuff, but if you don’t know how Kernel#require works you can get bitten by files getting loaded multiple times.

= Greg

Filed under: Uncategorized   |  1 Comment

Thanks O’Reilly

13Apr07

Rubyham wants to say thanks to O’Reilly!  Through their User Group program, they have sponsored Rubyham with books.  At next month’s TechMixer 3.0 you can find Rubyham along with some of O’Reilly’s Ruby books to browse. O’Reilly is offering 35% off to Rubyham members, and “Orders over $29.95 qualify for free shipping in the US.”

Filed under: O'Reilly   |  Leave a Comment

RSS Aggregator in 26 tasty lines of Ruby code

22Mar07

By Josh Adams

This story on an RSS Aggregator written in Ruby, brought to you by me via an RSS Aggregator written in something else.

Get your library knowledge on. From the article:

A liberal RSS aggregator in 26 lines of Ruby – may be hard to believe, but it’s true. On top of that, it is also capable of serving static files, has a templating engine, and accepts an arbitrary number and types of RSS feeds – talk about squeezing functionality out of every line!

Filed under: Uncategorized   |  1 Comment

Deprec currently broken, but there’s an easy fix.

16Mar07

Mike Bailey, creator of Deprec, is apparently in Africa or something right now. Apache’s been updated, and the version of httpd that deprec points to (2.2.3) has been removed and replaced (2.2.4). To get deprec working for you, just edit your deprec recipe to correct this. Mine was at /usr/lib/ruby/gems/1.8/gems/deprec-1.3.1/recipes.rb I think. Search for 2.2.3 and replace it with 2.2.4. Fini.

Oh yeah, for the googlers, here’s the error I was seeing.

  * executing "sudo      sh -c \"cd /usr/local/src && test -f httpd-2.2.3.tar.gz  && echo 'f72ffb176e2dc7b322be16508c09f63c  httpd-2.2.3.tar.gz' | md5sum -c -  || wget --timestamping http://www.apache.org/dist/httpd/httpd-2.2.3.tar.gz\""
    servers: ["208.75.85.114"]
    [208.75.85.114] executing command
 ** [out :: 208.75.85.114] --14:56:39--  http://www.apache.org/dist/httpd/httpd-2.2.3.tar.gz
 ** [out :: 208.75.85.114] => `httpd-2.2.3.tar.gz'
 ** [out :: 208.75.85.114] Resolving www.apache.org...
 ** [out :: 208.75.85.114] 140.211.11.130
 ** [out :: 208.75.85.114] Connecting to www.apache.org|140.211.11.130|:80...
 ** [out :: 208.75.85.114] connected.
 ** [out :: 208.75.85.114] HTTP request sent, awaiting response...
 ** [out :: 208.75.85.114] 404 Not Found
 ** [out :: 208.75.85.114] 14:56:40 ERROR 404: Not Found.
    command finished
command "sudo      sh -c \"cd /usr/local/src && test -f httpd-2.2.3.tar.gz  && echo 'f72ffb176e2dc7b322be16508c09f63c  httpd-2.2.3.tar.gz' | md5sum -c -  || wget --timestamping http://www.apache.org/dist/httpd/httpd-2.2.3.tar.gz\"" failed on 208.75.85.114

Filed under: Uncategorized   |  1 Comment

Google Spreadsheets Go Camping

07Mar07

Being such a pretty day in Birmingham, I decided to take google spreadsheets camping. I’m starting to use Google Spreadsheets more and more and have very little to complain about. However, the interface leaves much to be desired. My chief problem is the reliance on opening up multiple tabs( windows for non firefoxers ) for each spreadsheet. For instance:

What I’d really like is a simple:

And something lightweight would be great. First off, getting a list of the available sheets borrowing some code from the gdata rubyforge project

require 'net/http'
require 'net/https'
require 'uri'
require 'hpricot'

module Net
  class HTTPS < HTTP
    def initialize(address, port = nil)
      super(address, port)
      self.use_ssl = true
    end
  end
end

class GoogleSpreadsheet
  attr_accessor :id, :links, :title, :link, :key
  SPREADSHEETS_ROOT = "http://spreadsheets.google.com/ccc"
  URL_ROOT = "#{SPREADSHEETS_ROOT}?key="
  NEW_LINK = "#{SPREADSHEETS_ROOT}?new"
  def initialize(entry)
    @id = entry['id'][0]
    @key = @id.split('/').last
    @links = entry['link']
    @link = URL_ROOT + @key.to_s
    @title = entry['content']['content']
  end

  def document
    XmlSimple.xml_in(@link)
  end
end

class GoogleUser
  GOOGLE_SPREADSHEET_LIST_URL = '/feeds/spreadsheets/private/full'
  GOOGLE_LOGIN_URL = URI.parse('https://www.google.com/accounts/ClientLogin')
  attr_accessor :email, :password, :token, :spreadsheets
  def initialize(email, password)
    self.email=email
    self.password=password
    self.authenticate
  end

  def authenticate
    $VERBOSE = nil
    response = Net::HTTPS.post_form(GOOGLE_LOGIN_URL,
      {'Email'   => email,
       'Passwd'  => password,
       'source'  => "formula",
       'service' => 'wise' })
    @headers = {
     'Authorization' => "GoogleLogin auth=#{response.body.split(/=/).last}",
     'Content-Type'  => 'application/atom+xml'
    }
  end

  def get_spreadsheets
    the_list = XmlSimple.xml_in(spreadsheets_list)
    @spreadsheets = []
    the_list['entry'].each{|entry|
      @spreadsheets << GoogleSpreadsheet.new(entry)
    }
    @spreadsheets
  end

  def spreadsheets_list
    request(GOOGLE_SPREADSHEET_LIST_URL)
  end

  private
  def request(path)
    response, data = get_http.get(path, @headers)
    data
  end

  def post(path, entry)
    get_http.post(path, entry, @headers)
  end

  def get_http
    http = Net::HTTP.new('spreadsheets.google.com', 80)
    http
  end
end

Then for the single file camping interface:

#!/usr/bin/ruby
%w(rubygems camping gspreadsheet).each { |lib| require lib }

Camping.goes :CampingInSheets

module CampingInSheets::Controllers
  class Index < R '/'
    def get
      render :index
    end
    def post
      @user = GoogleUser.new(input.username, input.password)
      @spreadsheets = @user.get_spreadsheets
      render :index
    end
  end
  class Style < R '/styles.css'
    def get
      @headers["Content-Type"] = "text/css; charset=utf-8"
      @body = %{
          iframe{width: 100%; height: 80%;}
      }
    end
  end
end

module CampingInSheets::Views
  @@mab = Markaby::Builder.new
  SPREADSHEETS_ROOT = "http://spreadsheets.google.com/ccc"
  NEW_LINK = "#{SPREADSHEETS_ROOT}?new"
  def spreadsheet_link spreadsheet
    link_to spreadsheet.title, spreadsheet.link, :target => 'sheets'
  end
  def spreadsheet_dropdown spreadsheets
    options = spreadsheets.map{|x| "<option value='#{ x.link }'>#{x.title}</option>"}
    options = "<option value='#{NEW_LINK}'>New Spreadsheet</option>" + options.join
    options = "<option></option>" + options
    @@mab.html{
     form{
      select(  nchange => "var l = this.options[this.selectedIndex].value; if( l != '' ){ sheets.location.href=l; }"){
       self << options
      }
     }
    }
  end
  def layout
    html do
      head do
        title 'Camping in Sheets'
        link :href=>R(Style), :rel=>'stylesheet', :type=>'text/css'
      end
      body do
        div.content do
          self << yield
        end
      end
    end
  end
  def index
    if @user
      h1 "#{@user.email.gsub(/@.*/,'')} google spreadsheets"
      self << spreadsheet_dropdown(@spreadsheets)
      iframe(:name => 'sheets')
    else
      _login
    end
  end
  def _login
    h1 'Google Email and Password needed to Camp'
    form :action => R(Index), :method => 'post' do
      label 'Username', :for => 'username'; br
      input :name => 'username', :type => 'text'; br

      label 'Password', :for => 'password'; br
      input :name => 'password', :type => 'password'; br

      input :type => 'submit', :name => 'login', :value => 'Login'
    end
  end
end

And hardcore campers could rightly say that the code is horribly inefficient but it works for me. Combining this with a mousehole script to proxy all request to google spreadsheets to the camping interface would make this even better.

In order to make the two files work together, install the gems necessary for camping, save the second code snippet as camping_in_sheets.rb, the first as gspreadsheet.rb, and then from the command line type:

camping camping_in_sheets.rb

and enjoy. If anyone gets it going with mousehole, let me know.

Filed under: Uncategorized   |  1 Comment

JSON like ruby object

07Mar07

Recently when creating a rails backed dsl for interacting with Quickbooks SDK, I found myself desiring the ability to have json like behavior on a ruby object. By this I mean being able to create an object with attributes and subobjects on the fly and then being able to access them easily. My solution and with quite a bit of help from jadams was first to reopen the Hash class and insert a method missing to allow for accessing hash['attribute'] with the more rubyesque hash.attribute. The technique and my first ever metaprogramming was inspired by Ola Bini’s excellent blog post, specifically method 4 which uses an example from why’s fantastic camping project.

class Hash
  def method_missing(m,*a)
    if m.to_s =~ /=$/
      self[$`] = a[0]
    elsif a.empty?
      self[m]={} unless self[m]
      self[m]
    else
      raise NoMethodError, "#{m}"
    end
  end
end

The next step was simply to create a generic object with an attr_accessor to create a hash:

class JSONLikeObject
  attr_accessor :type, :attributes
  def initialize(type, attributes = {})
    @type = type
    @attributes = attributes
  end
end

Great! Now I’m able to say foo = JsonLikeObject.new and then say foo.attributes.attribute to access foo.attributes['attribute']. Closer, but I really wanted to be able to say foo.attribute, so I defined method missing for the JSONLikeObject class:

def method_missing(m,*a)
    if m.to_s =~ /=$/
      @attributes.send(m,a[0])
    else
      @attributes[m]={} unless @attributes[m]
      @attributes[m]
    end
  end

Mission accomplished, but I really would like a more complicated/richer tree. For example, customer = JSONLikeObject.new and be able to write customer.contact.address.line1 = ‘Ruby Drive’ and it create customer.attributes['contact']['address']['line1'] = ‘Ruby Drive’ without blowing up. Off to read Ola’s post again and there in example 1 lies the solution:

module Kernel
  def singleton_class
    class << self; self; end
  end
end

Now it is simply a matter of taking the method missing from the class and wrapping it in a module that can be included upon initialization:

class Hash
  def method_missing(m,*a)
    if m.to_s =~ /=$/
      self[$`] = a[0]
    elsif a.empty?
      self[m]={} unless self[m]
      self[m]
    else
      raise NoMethodError, "#{m}"
    end
  end
end
module Kernel
  def singleton_class
    class << self; self; end
  end
end
module Mod
  def method_missing(m,*a)
    if m.to_s =~ /=$/
      @attributes.send(m,a[0])
    else
      @attributes[m]={} unless @attributes[m]
      @attributes[m]
    end
  end
end
class JSONLikeObject
  attr_accessor :type, :attributes
  def initialize(type, attributes = {})
    @type = type
    @attributes = attributes
    singleton_class.class_eval do
      include Mod
    end
  end
end

And bingo, a json like object in ruby.

Filed under: Uncategorized   |  Leave a Comment