So PoolParty seems nice.  Tonight I’ll be attempting to use it (primarily because it gives me shared disk across instances via S3 for free) to replace a script’s use of Sun Grid Engine (SGE) with EC2.  This happens to be all I need to pull this off fairly easily.

So I’m using this to parallelize FSL, a bit of software that analyzes fMRI data.  Specifically, I’m modifying the parallelization function to handle EC2 on top of the Sun Grid Engine support it already has.  The current process ends up with a file called commands.txt.  Each line needs to be run on its own EC2 instance with access to the shared data directory, in which the output will be placed.  The current shell script will then take over, as that’s the behaviour that SGE ends up with for this task.

To help with this, Ari (the poolparty guy) wrote up a nice PoolParty Plugin for this very task.

We’ll ignore the actual use case (just makes a good story) and simplify the problem to: launch a pool party that outputs a command to /data/log/output1, output2, etc.

My commands.txt file looks like:

ls /bin > /data/output1
ls /root > /data/output2

Then it’s just a matter of pool -v -i -I ami-1234 -b shared_bucket.  This launches the appropriate number of instances and at the end of the process all of the data has been written to the appropriate place.

I actually wrote no ruby code for this.  Ari wrote the plugin that handled my whole use case, but it was really nice to see how tiny the plugin can be.  Ultimately, this gem looks intensely interesting for our generic use case, which was simply distributed processing of a massively parallelizable computation.  It completely replaces the need for Sun Grid Engine in this case.

Everyone join the Pool Party.

NOTE: Documentation is sparse at present, but Ari is available on Freenode#poolpartyrb and I would be glad to help anyone that wants to talk about how to use PoolParty in whatever small way I can.  Ari has said that this week is to be documentation week for the project.  And go donate to the project if you can.


I woke up fired up and ready to try git; unfortunately, it is three hours later and I’m just gitting started. The problem was getting git-svn to work. Despite the numerous posts including: post1 post2 post3 and post4 I absolutely could not get it to work. Finally, here are two techniques that worked for me:
Port Method
sudo port install subversion-perlbindings
sudo port install git-core

or Method2

follow Fraser Spiers post

except

export PERL5LIB=”/opt/subversion/lib/svn-perl:$PERL5LIb”

In other words, the package installs the subversion perl bindings in /opt/subversion/lib/svn-perl not /usr/local/lib/svn-perl.

Time to start gitting it done!


by Josh Adams

Ever wanted to be able to do something like an Outlook-style recursive event generator? Say, schedule an event each MWF for the next six weeks from 8:00 – 8:30 AM? There’s a nice library called RUNT that I ran across today that makes this kind of temporal work easier.

Then Brian, the newest hire at isotope11, created a library to ease the particular use case we had, which was using a form similar to the following:

Screenshot

The code for the GroupEvent class is as follows:


require 'runt'

class GroupEvent
  include Runt
  def self.schedule(start, finished, weekdays, type, &date)
    case type
    when /w/i
      self.weekly(start, finished, weekdays) { |day| date.call(day) }
    when /b/i
      self.biweekly(start, finished, weekdays) { |day| date.call(day) }
    when /m/i
      self.monthly(start, finished) { |day| date.call(day) }
    when /y/i
      self.yearly(start, finished) { |day| date.call(day) }
    end
  end 

  def self.weekly(start, finished, weekdays, &day)
    day = nil
    exp = nil
    for i in weekdays
      if (exp.nil?)
        exp = DIWeek.new(i)
      else
        exp ||= DIWeek.new(i)
      end
    end
    for day in (start..finished)
      if(exp.include?(day))
        yield(day)
      end
    end
  end

  def self.biweekly(start, finished, weekdays, &biday)
    self.weekly(start, finished, weekdays) do |biday|
      if((biday.day%2) == (start.day%2))
        yield(biday)
      end
    end
  end

  def self.monthly(start, finished, &month)
    current = start
    begin
      yield(current)
      current = current.to_time.months_since 1
    end while(current.to_time < finished.to_time)
  end

  def self.yearly(start, finished, &year)
    current = start
    begin
      yield(current)
      current = current.to_time.years_since 1
    end while(current.to_time < finished.to_time)
  end
end

Inside the controller that form points to, I call the following (*horrendously ugly*) method:


  def create_events
    if params[:recursive]
      start_date = Date.new(params[:start_date][:year].to_i, params[:start_date][:month].to_i, params[:start_date][:day].to_i)
      end_date = Date.new(params[:end_date][:year].to_i, params[:end_date][:month].to_i, params[:end_date][:day].to_i)
      GroupEvent.schedule(start_date, end_date, params[:days].map(&:to_i), params[:period]) do |date|
        starts_at = DateTime.new(date.year, date.month, date.day, params[:start_time][:hour].to_i, params[:start_time][:minute].to_i)
        ends_at = DateTime.new(date.year, date.month, date.day, params[:end_time][:hour].to_i, params[:end_time][:minute].to_i)
        DateEvent.create( :starts_at => starts_at, :ends_at => ends_at, :description => params[:event][:description])
      end
    else
      @event = DateEvent.new(params[:event])
    end
  end

Please, please forgive me for that code. It was well past five and we just wanted to get it working, okay? That code was mine, the good looking stuff above was Brian’s. Don’t pin my trespasses on him.

Anyway, the code appears to do exactly what we want so far, haven’t found any quirks yet. We’ll get better test coverage of this stuff tomorrow.


A few posts back I mentioned a ruby aggregator in 26 lines of code. I’ve spent the last fifteen minutes or so tying it to festival so it can read out RSS feeds.

If you just want to read the script fast, click here.

The code’s all there, but the coolest thing is the library I found, FestivalTTS4Ruby.

To actually run the code, you’ll need festival installed (I just shell out to it). Fire up irb, and:


require 'rss_to_speech'
RSSToSpeech.new

That will read in each line of feeds.txt, suck down the posts for the feed, pipe it through a python script called html2text.py, then pipe that through festival –tts. More lines than I’d like it to be, but I don’t feel like working on it any more :)


I just posted an article about ruby’s require on my blog: http://ghouston.blogspot.com/2007/05/ruby-require-idiom.html

It is fairly basic ruby stuff, but if you don’t know how Kernel#require works you can get bitten by files getting loaded multiple times.

= Greg


O'Reilly Discount Code DSUG for 35% Off

Rubyham wants to say thanks to O’Reilly!  Through their User Group program, they have sponsored Rubyham with books.  At next month’s TechMixer 3.0 you can find Rubyham along with some of O’Reilly’s Ruby books to browse. O’Reilly is offering 35% off to Rubyham members, and “Orders over $29.95 qualify for free shipping in the US.”


By Josh Adams

This story on an RSS Aggregator written in Ruby, brought to you by me via an RSS Aggregator written in something else.

Get your library knowledge on. From the article:

A liberal RSS aggregator in 26 lines of Ruby – may be hard to believe, but it’s true. On top of that, it is also capable of serving static files, has a templating engine, and accepts an arbitrary number and types of RSS feeds – talk about squeezing functionality out of every line!




Follow

Get every new post delivered to your Inbox.