So I’m using this to parallelize FSL, a bit of software that analyzes fMRI data.  Specifically, I’m modifying the parallelization function to handle EC2 on top of the Sun Grid Engine support it already has.  The current process ends up with a file called commands.txt.  Each line needs to be run on its own EC2 instance with access to the shared data directory, in which the output will be placed.  The current shell script will then take over, as that’s the behaviour that SGE ends up with for this task.

To help with this, Ari (the poolparty guy) wrote up a nice PoolParty Plugin for this very task.

We’ll ignore the actual use case (just makes a good story) and simplify the problem to: launch a pool party that outputs a command to /data/log/output1, output2, etc.

My commands.txt file looks like:

ls /bin > /data/output1
ls /root > /data/output2

Then it’s just a matter of pool -v -i -I ami-1234 -b shared_bucket.  This launches the appropriate number of instances and at the end of the process all of the data has been written to the appropriate place.

I actually wrote no ruby code for this.  Ari wrote the plugin that handled my whole use case, but it was really nice to see how tiny the plugin can be.  Ultimately, this gem looks intensely interesting for our generic use case, which was simply distributed processing of a massively parallelizable computation.  It completely replaces the need for Sun Grid Engine in this case.

Everyone join the Pool Party.

NOTE: Documentation is sparse at present, but Ari is available on Freenode#poolpartyrb and I would be glad to help anyone that wants to talk about how to use PoolParty in whatever small way I can.  Ari has said that this week is to be documentation week for the project.  And go donate to the project if you can.

or Method2

follow Fraser Spiers post

except

export PERL5LIB="/opt/subversion/lib/svn-perl:$PERL5LIb"

In other words, the package installs the subversion perl bindings in /opt/subversion/lib/svn-perl not /usr/local/lib/svn-perl.

Time to start gitting it done!

Then Brian, the newest hire at isotope11, created a library to ease the particular use case we had, which was using a form similar to the following:

The code for the GroupEvent class is as follows:

require 'runt'

class GroupEvent
  include Runt
  def self.schedule(start, finished, weekdays, type, &date)
    case type
    when /w/i
      self.weekly(start, finished, weekdays) { |day| date.call(day) }
    when /b/i
      self.biweekly(start, finished, weekdays) { |day| date.call(day) }
    when /m/i
      self.monthly(start, finished) { |day| date.call(day) }
    when /y/i
      self.yearly(start, finished) { |day| date.call(day) }
    end
  end 

  def self.weekly(start, finished, weekdays, &day)
    day = nil
    exp = nil
    for i in weekdays
      if (exp.nil?)
        exp = DIWeek.new(i)
      else
        exp ||= DIWeek.new(i)
      end
    end
    for day in (start..finished)
      if(exp.include?(day))
        yield(day)
      end
    end
  end

  def self.biweekly(start, finished, weekdays, &biday)
    self.weekly(start, finished, weekdays) do |biday|
      if((biday.day%2) == (start.day%2))
        yield(biday)
      end
    end
  end

  def self.monthly(start, finished, &month)
    current = start
    begin
      yield(current)
      current = current.to_time.months_since 1
    end while(current.to_time < finished.to_time)
  end

  def self.yearly(start, finished, &year)
    current = start
    begin
      yield(current)
      current = current.to_time.years_since 1
    end while(current.to_time < finished.to_time)
  end
end

Inside the controller that form points to, I call the following (*horrendously ugly*) method:

  def create_events
    if params[:recursive]
      start_date = Date.new(params[:start_date][:year].to_i, params[:start_date][:month].to_i, params[:start_date][:day].to_i)
      end_date = Date.new(params[:end_date][:year].to_i, params[:end_date][:month].to_i, params[:end_date][:day].to_i)
      GroupEvent.schedule(start_date, end_date, params[:days].map(&:to_i), params[:period]) do |date|
        starts_at = DateTime.new(date.year, date.month, date.day, params[:start_time][:hour].to_i, params[:start_time][:minute].to_i)
        ends_at = DateTime.new(date.year, date.month, date.day, params[:end_time][:hour].to_i, params[:end_time][:minute].to_i)
        DateEvent.create( :starts_at => starts_at, :ends_at => ends_at, :description => params[:event][:description])
      end
    else
      @event = DateEvent.new(params[:event])
    end
  end

Please, please forgive me for that code. It was well past five and we just wanted to get it working, okay? That code was mine, the good looking stuff above was Brian’s. Don’t pin my trespasses on him.

Anyway, the code appears to do exactly what we want so far, haven’t found any quirks yet. We’ll get better test coverage of this stuff tomorrow.

If you just want to read the script fast, click here.

The code’s all there, but the coolest thing is the library I found, FestivalTTS4Ruby.

To actually run the code, you’ll need festival installed (I just shell out to it). Fire up irb, and:

require 'rss_to_speech'
RSSToSpeech.new

That will read in each line of feeds.txt, suck down the posts for the feed, pipe it through a python script called html2text.py, then pipe that through festival –tts. More lines than I’d like it to be, but I don’t feel like working on it any more

It is fairly basic ruby stuff, but if you don’t know how Kernel#require works you can get bitten by files getting loaded multiple times.

= Greg

Rubyham wants to say thanks to O’Reilly!  Through their User Group program, they have sponsored Rubyham with books.  At next month’s TechMixer 3.0 you can find Rubyham along with some of O’Reilly’s Ruby books to browse. O’Reilly is offering 35% off to Rubyham members, and “Orders over $29.95 qualify for free shipping in the US.”

This story on an RSS Aggregator written in Ruby, brought to you by me via an RSS Aggregator written in something else.

Get your library knowledge on. From the article:

A liberal RSS aggregator in 26 lines of Ruby – may be hard to believe, but it’s true. On top of that, it is also capable of serving static files, has a templating engine, and accepts an arbitrary number and types of RSS feeds – talk about squeezing functionality out of every line!

Oh yeah, for the googlers, here’s the error I was seeing.

  * executing "sudo      sh -c \"cd /usr/local/src && test -f httpd-2.2.3.tar.gz  && echo 'f72ffb176e2dc7b322be16508c09f63c  httpd-2.2.3.tar.gz' | md5sum -c -  || wget --timestamping http://www.apache.org/dist/httpd/httpd-2.2.3.tar.gz\""
    servers: ["208.75.85.114"]
    [208.75.85.114] executing command
 ** [out :: 208.75.85.114] --14:56:39--  http://www.apache.org/dist/httpd/httpd-2.2.3.tar.gz
 ** [out :: 208.75.85.114] => `httpd-2.2.3.tar.gz'
 ** [out :: 208.75.85.114] Resolving www.apache.org...
 ** [out :: 208.75.85.114] 140.211.11.130
 ** [out :: 208.75.85.114] Connecting to www.apache.org|140.211.11.130|:80...
 ** [out :: 208.75.85.114] connected.
 ** [out :: 208.75.85.114] HTTP request sent, awaiting response...
 ** [out :: 208.75.85.114] 404 Not Found
 ** [out :: 208.75.85.114] 14:56:40 ERROR 404: Not Found.
    command finished
command "sudo      sh -c \"cd /usr/local/src && test -f httpd-2.2.3.tar.gz  && echo 'f72ffb176e2dc7b322be16508c09f63c  httpd-2.2.3.tar.gz' | md5sum -c -  || wget --timestamping http://www.apache.org/dist/httpd/httpd-2.2.3.tar.gz\"" failed on 208.75.85.114

What I’d really like is a simple:

And something lightweight would be great. First off, getting a list of the available sheets borrowing some code from the gdata rubyforge project

require 'net/http'
require 'net/https'
require 'uri'
require 'hpricot'

module Net
  class HTTPS < HTTP
    def initialize(address, port = nil)
      super(address, port)
      self.use_ssl = true
    end
  end
end

class GoogleSpreadsheet
  attr_accessor :id, :links, :title, :link, :key
  SPREADSHEETS_ROOT = "http://spreadsheets.google.com/ccc"
  URL_ROOT = "#{SPREADSHEETS_ROOT}?key="
  NEW_LINK = "#{SPREADSHEETS_ROOT}?new"
  def initialize(entry)
    @id = entry['id'][0]
    @key = @id.split('/').last
    @links = entry['link']
    @link = URL_ROOT + @key.to_s
    @title = entry['content']['content']
  end

  def document
    XmlSimple.xml_in(@link)
  end
end

class GoogleUser
  GOOGLE_SPREADSHEET_LIST_URL = '/feeds/spreadsheets/private/full'
  GOOGLE_LOGIN_URL = URI.parse('https://www.google.com/accounts/ClientLogin')
  attr_accessor :email, :password, :token, :spreadsheets
  def initialize(email, password)
    self.email=email
    self.password=password
    self.authenticate
  end

  def authenticate
    $VERBOSE = nil
    response = Net::HTTPS.post_form(GOOGLE_LOGIN_URL,
      {'Email'   => email,
       'Passwd'  => password,
       'source'  => "formula",
       'service' => 'wise' })
    @headers = {
     'Authorization' => "GoogleLogin auth=#{response.body.split(/=/).last}",
     'Content-Type'  => 'application/atom+xml'
    }
  end

  def get_spreadsheets
    the_list = XmlSimple.xml_in(spreadsheets_list)
    @spreadsheets = []
    the_list['entry'].each{|entry|
      @spreadsheets << GoogleSpreadsheet.new(entry)
    }
    @spreadsheets
  end

  def spreadsheets_list
    request(GOOGLE_SPREADSHEET_LIST_URL)
  end

  private
  def request(path)
    response, data = get_http.get(path, @headers)
    data
  end

  def post(path, entry)
    get_http.post(path, entry, @headers)
  end

  def get_http
    http = Net::HTTP.new('spreadsheets.google.com', 80)
    http
  end
end

Then for the single file camping interface:

#!/usr/bin/ruby
%w(rubygems camping gspreadsheet).each { |lib| require lib }

Camping.goes :CampingInSheets

module CampingInSheets::Controllers
  class Index < R '/'
    def get
      render :index
    end
    def post
      @user = GoogleUser.new(input.username, input.password)
      @spreadsheets = @user.get_spreadsheets
      render :index
    end
  end
  class Style < R '/styles.css'
    def get
      @headers["Content-Type"] = "text/css; charset=utf-8"
      @body = %{
          iframe{width: 100%; height: 80%;}
      }
    end
  end
end

module CampingInSheets::Views
  @@mab = Markaby::Builder.new
  SPREADSHEETS_ROOT = "http://spreadsheets.google.com/ccc"
  NEW_LINK = "#{SPREADSHEETS_ROOT}?new"
  def spreadsheet_link spreadsheet
    link_to spreadsheet.title, spreadsheet.link, :target => 'sheets'
  end
  def spreadsheet_dropdown spreadsheets
    options = spreadsheets.map{|x| "<option value='#{ x.link }'>#{x.title}</option>"}
    options = "<option value='#{NEW_LINK}'>New Spreadsheet</option>" + options.join
    options = "<option></option>" + options
    @@mab.html{
     form{
      select(  nchange => "var l = this.options[this.selectedIndex].value; if( l != '' ){ sheets.location.href=l; }"){
       self << options
      }
     }
    }
  end
  def layout
    html do
      head do
        title 'Camping in Sheets'
        link :href=>R(Style), :rel=>'stylesheet', :type=>'text/css'
      end
      body do
        div.content do
          self << yield
        end
      end
    end
  end
  def index
    if @user
      h1 "#{@user.email.gsub(/@.*/,'')} google spreadsheets"
      self << spreadsheet_dropdown(@spreadsheets)
      iframe(:name => 'sheets')
    else
      _login
    end
  end
  def _login
    h1 'Google Email and Password needed to Camp'
    form :action => R(Index), :method => 'post' do
      label 'Username', :for => 'username'; br
      input :name => 'username', :type => 'text'; br

      label 'Password', :for => 'password'; br
      input :name => 'password', :type => 'password'; br

      input :type => 'submit', :name => 'login', :value => 'Login'
    end
  end
end

And hardcore campers could rightly say that the code is horribly inefficient but it works for me. Combining this with a mousehole script to proxy all request to google spreadsheets to the camping interface would make this even better.

In order to make the two files work together, install the gems necessary for camping, save the second code snippet as camping_in_sheets.rb, the first as gspreadsheet.rb, and then from the command line type:

camping camping_in_sheets.rb

and enjoy. If anyone gets it going with mousehole, let me know.

class Hash
  def method_missing(m,*a)
    if m.to_s =~ /=$/
      self[$`] = a[0]
    elsif a.empty?
      self[m]={} unless self[m]
      self[m]
    else
      raise NoMethodError, "#{m}"
    end
  end
end

The next step was simply to create a generic object with an attr_accessor to create a hash:

class JSONLikeObject
  attr_accessor :type, :attributes
  def initialize(type, attributes = {})
    @type = type
    @attributes = attributes
  end
end

Great! Now I’m able to say foo = JsonLikeObject.new and then say foo.attributes.attribute to access foo.attributes['attribute']. Closer, but I really wanted to be able to say foo.attribute, so I defined method missing for the JSONLikeObject class:

def method_missing(m,*a)
    if m.to_s =~ /=$/
      @attributes.send(m,a[0])
    else
      @attributes[m]={} unless @attributes[m]
      @attributes[m]
    end
  end

Mission accomplished, but I really would like a more complicated/richer tree. For example, customer = JSONLikeObject.new and be able to write customer.contact.address.line1 = ‘Ruby Drive’ and it create customer.attributes['contact']['address']['line1'] = ‘Ruby Drive’ without blowing up. Off to read Ola’s post again and there in example 1 lies the solution:

module Kernel
  def singleton_class
    class << self; self; end
  end
end

Now it is simply a matter of taking the method missing from the class and wrapping it in a module that can be included upon initialization:

class Hash
  def method_missing(m,*a)
    if m.to_s =~ /=$/
      self[$`] = a[0]
    elsif a.empty?
      self[m]={} unless self[m]
      self[m]
    else
      raise NoMethodError, "#{m}"
    end
  end
end
module Kernel
  def singleton_class
    class << self; self; end
  end
end
module Mod
  def method_missing(m,*a)
    if m.to_s =~ /=$/
      @attributes.send(m,a[0])
    else
      @attributes[m]={} unless @attributes[m]
      @attributes[m]
    end
  end
end
class JSONLikeObject
  attr_accessor :type, :attributes
  def initialize(type, attributes = {})
    @type = type
    @attributes = attributes
    singleton_class.class_eval do
      include Mod
    end
  end
end

And bingo, a json like object in ruby.