So PoolParty seems nice. Tonight I’ll be attempting to use it (primarily because it gives me shared disk across instances via S3 for free) to replace a script’s use of Sun Grid Engine (SGE) with EC2. This happens to be all I need to pull this off fairly easily.
So I’m using this to parallelize FSL, a bit of software that analyzes fMRI data. Specifically, I’m modifying the parallelization function to handle EC2 on top of the Sun Grid Engine support it already has. The current process ends up with a file called commands.txt. Each line needs to be run on its own EC2 instance with access to the shared data directory, in which the output will be placed. The current shell script will then take over, as that’s the behaviour that SGE ends up with for this task.
To help with this, Ari (the poolparty guy) wrote up a nice PoolParty Plugin for this very task.
We’ll ignore the actual use case (just makes a good story) and simplify the problem to: launch a pool party that outputs a command to /data/log/output1, output2, etc.
My commands.txt file looks like:
ls /bin > /data/output1
ls /root > /data/output2
Then it’s just a matter of pool -v -i -I ami-1234 -b shared_bucket. This launches the appropriate number of instances and at the end of the process all of the data has been written to the appropriate place.
I actually wrote no ruby code for this. Ari wrote the plugin that handled my whole use case, but it was really nice to see how tiny the plugin can be. Ultimately, this gem looks intensely interesting for our generic use case, which was simply distributed processing of a massively parallelizable computation. It completely replaces the need for Sun Grid Engine in this case.
Everyone join the Pool Party.
NOTE: Documentation is sparse at present, but Ari is available on Freenode#poolpartyrb and I would be glad to help anyone that wants to talk about how to use PoolParty in whatever small way I can. Ari has said that this week is to be documentation week for the project. And go donate to the project if you can.
Filed under: Uncategorized | 1 Comment
Tags: Amazon, AWS, EC2, pool party, ruby
Gitting git-svn working on mac
I woke up fired up and ready to try git; unfortunately, it is three hours later and I’m just gitting started. The problem was getting git-svn to work. Despite the numerous posts including: post1 post2 post3 and post4 I absolutely could not get it to work. Finally, here are two techniques that worked for me:
Port Method
sudo port install subversion-perlbindings
sudo port install git-core
or Method2
follow Fraser Spiers post
except
export PERL5LIB="/opt/subversion/lib/svn-perl:$PERL5LIb"
In other words, the package installs the subversion perl bindings in /opt/subversion/lib/svn-perl not /usr/local/lib/svn-perl.
Time to start gitting it done!
Filed under: Uncategorized | 1 Comment
by Josh Adams
Ever wanted to be able to do something like an Outlook-style recursive event generator? Say, schedule an event each MWF for the next six weeks from 8:00 – 8:30 AM? There’s a nice library called RUNT that I ran across today that makes this kind of temporal work easier.
Then Brian, the newest hire at isotope11, created a library to ease the particular use case we had, which was using a form similar to the following:
The code for the GroupEvent class is as follows:
require 'runt'
class GroupEvent
include Runt
def self.schedule(start, finished, weekdays, type, &date)
case type
when /w/i
self.weekly(start, finished, weekdays) { |day| date.call(day) }
when /b/i
self.biweekly(start, finished, weekdays) { |day| date.call(day) }
when /m/i
self.monthly(start, finished) { |day| date.call(day) }
when /y/i
self.yearly(start, finished) { |day| date.call(day) }
end
end
def self.weekly(start, finished, weekdays, &day)
day = nil
exp = nil
for i in weekdays
if (exp.nil?)
exp = DIWeek.new(i)
else
exp ||= DIWeek.new(i)
end
end
for day in (start..finished)
if(exp.include?(day))
yield(day)
end
end
end
def self.biweekly(start, finished, weekdays, &biday)
self.weekly(start, finished, weekdays) do |biday|
if((biday.day%2) == (start.day%2))
yield(biday)
end
end
end
def self.monthly(start, finished, &month)
current = start
begin
yield(current)
current = current.to_time.months_since 1
end while(current.to_time < finished.to_time)
end
def self.yearly(start, finished, &year)
current = start
begin
yield(current)
current = current.to_time.years_since 1
end while(current.to_time < finished.to_time)
end
end
Inside the controller that form points to, I call the following (*horrendously ugly*) method:
def create_events
if params[:recursive]
start_date = Date.new(params[:start_date][:year].to_i, params[:start_date][:month].to_i, params[:start_date][:day].to_i)
end_date = Date.new(params[:end_date][:year].to_i, params[:end_date][:month].to_i, params[:end_date][:day].to_i)
GroupEvent.schedule(start_date, end_date, params[:days].map(&:to_i), params[:period]) do |date|
starts_at = DateTime.new(date.year, date.month, date.day, params[:start_time][:hour].to_i, params[:start_time][:minute].to_i)
ends_at = DateTime.new(date.year, date.month, date.day, params[:end_time][:hour].to_i, params[:end_time][:minute].to_i)
DateEvent.create( :starts_at => starts_at, :ends_at => ends_at, :description => params[:event][:description])
end
else
@event = DateEvent.new(params[:event])
end
end
Please, please forgive me for that code. It was well past five and we just wanted to get it working, okay? That code was mine, the good looking stuff above was Brian’s. Don’t pin my trespasses on him.
Anyway, the code appears to do exactly what we want so far, haven’t found any quirks yet. We’ll get better test coverage of this stuff tomorrow.
Filed under: Uncategorized | 1 Comment
Tags: rails, recursive events, ruby, time-based
A few posts back I mentioned a ruby aggregator in 26 lines of code. I’ve spent the last fifteen minutes or so tying it to festival so it can read out RSS feeds.
If you just want to read the script fast, click here.
The code’s all there, but the coolest thing is the library I found, FestivalTTS4Ruby.
To actually run the code, you’ll need festival installed (I just shell out to it). Fire up irb, and:
require 'rss_to_speech'
RSSToSpeech.new
That will read in each line of feeds.txt, suck down the posts for the feed, pipe it through a python script called html2text.py, then pipe that through festival –tts. More lines than I’d like it to be, but I don’t feel like working on it any more 
Filed under: Uncategorized | 1 Comment
Ruby Require Idiom
I just posted an article about ruby’s require on my blog: http://ghouston.blogspot.com/2007/05/ruby-require-idiom.html
It is fairly basic ruby stuff, but if you don’t know how Kernel#require works you can get bitten by files getting loaded multiple times.
= Greg
Filed under: Uncategorized | Leave a Comment
Thanks O’Reilly
Rubyham wants to say thanks to O’Reilly! Through their User Group program, they have sponsored Rubyham with books. At next month’s TechMixer 3.0 you can find Rubyham along with some of O’Reilly’s Ruby books to browse. O’Reilly is offering 35% off to Rubyham members, and “Orders over $29.95 qualify for free shipping in the US.”
Filed under: O'Reilly | Leave a Comment
By Josh Adams
This story on an RSS Aggregator written in Ruby, brought to you by me via an RSS Aggregator written in something else.
Get your library knowledge on. From the article:
A liberal RSS aggregator in 26 lines of Ruby – may be hard to believe, but it’s true. On top of that, it is also capable of serving static files, has a templating engine, and accepts an arbitrary number and types of RSS feeds – talk about squeezing functionality out of every line!
Filed under: Uncategorized | 1 Comment
Mike Bailey, creator of Deprec, is apparently in Africa or something right now. Apache’s been updated, and the version of httpd that deprec points to (2.2.3) has been removed and replaced (2.2.4). To get deprec working for you, just edit your deprec recipe to correct this. Mine was at /usr/lib/ruby/gems/1.8/gems/deprec-1.3.1/recipes.rb I think. Search for 2.2.3 and replace it with 2.2.4. Fini.
Oh yeah, for the googlers, here’s the error I was seeing.
* executing "sudo sh -c \"cd /usr/local/src && test -f httpd-2.2.3.tar.gz && echo 'f72ffb176e2dc7b322be16508c09f63c httpd-2.2.3.tar.gz' | md5sum -c - || wget --timestamping http://www.apache.org/dist/httpd/httpd-2.2.3.tar.gz\""
servers: ["208.75.85.114"]
[208.75.85.114] executing command
** [out :: 208.75.85.114] --14:56:39-- http://www.apache.org/dist/httpd/httpd-2.2.3.tar.gz
** [out :: 208.75.85.114] => `httpd-2.2.3.tar.gz'
** [out :: 208.75.85.114] Resolving www.apache.org...
** [out :: 208.75.85.114] 140.211.11.130
** [out :: 208.75.85.114] Connecting to www.apache.org|140.211.11.130|:80...
** [out :: 208.75.85.114] connected.
** [out :: 208.75.85.114] HTTP request sent, awaiting response...
** [out :: 208.75.85.114] 404 Not Found
** [out :: 208.75.85.114] 14:56:40 ERROR 404: Not Found.
command finished
command "sudo sh -c \"cd /usr/local/src && test -f httpd-2.2.3.tar.gz && echo 'f72ffb176e2dc7b322be16508c09f63c httpd-2.2.3.tar.gz' | md5sum -c - || wget --timestamping http://www.apache.org/dist/httpd/httpd-2.2.3.tar.gz\"" failed on 208.75.85.114
Filed under: Uncategorized | 1 Comment
Google Spreadsheets Go Camping
Being such a pretty day in Birmingham, I decided to take google spreadsheets camping. I’m starting to use Google Spreadsheets more and more and have very little to complain about. However, the interface leaves much to be desired. My chief problem is the reliance on opening up multiple tabs( windows for non firefoxers ) for each spreadsheet. For instance:
What I’d really like is a simple:
And something lightweight would be great. First off, getting a list of the available sheets borrowing some code from the gdata rubyforge project
require 'net/http' require 'net/https' require 'uri' require 'hpricot' module Net class HTTPS < HTTP def initialize(address, port = nil) super(address, port) self.use_ssl = true end end end class GoogleSpreadsheet attr_accessor :id, :links, :title, :link, :key SPREADSHEETS_ROOT = "http://spreadsheets.google.com/ccc" URL_ROOT = "#{SPREADSHEETS_ROOT}?key=" NEW_LINK = "#{SPREADSHEETS_ROOT}?new" def initialize(entry) @id = entry['id'][0] @key = @id.split('/').last @links = entry['link'] @link = URL_ROOT + @key.to_s @title = entry['content']['content'] end def document XmlSimple.xml_in(@link) end end class GoogleUser GOOGLE_SPREADSHEET_LIST_URL = '/feeds/spreadsheets/private/full' GOOGLE_LOGIN_URL = URI.parse('https://www.google.com/accounts/ClientLogin') attr_accessor :email, :password, :token, :spreadsheets def initialize(email, password) self.email=email self.password=password self.authenticate end def authenticate $VERBOSE = nil response = Net::HTTPS.post_form(GOOGLE_LOGIN_URL, {'Email' => email, 'Passwd' => password, 'source' => "formula", 'service' => 'wise' }) @headers = { 'Authorization' => "GoogleLogin auth=#{response.body.split(/=/).last}", 'Content-Type' => 'application/atom+xml' } end def get_spreadsheets the_list = XmlSimple.xml_in(spreadsheets_list) @spreadsheets = [] the_list['entry'].each{|entry| @spreadsheets << GoogleSpreadsheet.new(entry) } @spreadsheets end def spreadsheets_list request(GOOGLE_SPREADSHEET_LIST_URL) end private def request(path) response, data = get_http.get(path, @headers) data end def post(path, entry) get_http.post(path, entry, @headers) end def get_http http = Net::HTTP.new('spreadsheets.google.com', 80) http end end
Then for the single file camping interface:
#!/usr/bin/ruby %w(rubygems camping gspreadsheet).each { |lib| require lib } Camping.goes :CampingInSheets module CampingInSheets::Controllers class Index < R '/' def get render :index end def post @user = GoogleUser.new(input.username, input.password) @spreadsheets = @user.get_spreadsheets render :index end end class Style < R '/styles.css' def get @headers["Content-Type"] = "text/css; charset=utf-8" @body = %{ iframe{width: 100%; height: 80%;} } end end end module CampingInSheets::Views @@mab = Markaby::Builder.new SPREADSHEETS_ROOT = "http://spreadsheets.google.com/ccc" NEW_LINK = "#{SPREADSHEETS_ROOT}?new" def spreadsheet_link spreadsheet link_to spreadsheet.title, spreadsheet.link, :target => 'sheets' end def spreadsheet_dropdown spreadsheets options = spreadsheets.map{|x| "<option value='#{ x.link }'>#{x.title}</option>"} options = "<option value='#{NEW_LINK}'>New Spreadsheet</option>" + options.join options = "<option></option>" + options @@mab.html{ form{ select(nchange => "var l = this.options[this.selectedIndex].value; if( l != '' ){ sheets.location.href=l; }"){ self << options } } } end def layout html do head do title 'Camping in Sheets' link :href=>R(Style), :rel=>'stylesheet', :type=>'text/css' end body do div.content do self << yield end end end end def index if @user h1 "#{@user.email.gsub(/@.*/,'')} google spreadsheets" self << spreadsheet_dropdown(@spreadsheets) iframe(:name => 'sheets') else _login end end def _login h1 'Google Email and Password needed to Camp' form :action => R(Index), :method => 'post' do label 'Username', :for => 'username'; br input :name => 'username', :type => 'text'; br label 'Password', :for => 'password'; br input :name => 'password', :type => 'password'; br input :type => 'submit', :name => 'login', :value => 'Login' end end end
And hardcore campers could rightly say that the code is horribly inefficient but it works for me. Combining this with a mousehole script to proxy all request to google spreadsheets to the camping interface would make this even better.
In order to make the two files work together, install the gems necessary for camping, save the second code snippet as camping_in_sheets.rb, the first as gspreadsheet.rb, and then from the command line type:
camping camping_in_sheets.rb
and enjoy. If anyone gets it going with mousehole, let me know.
Filed under: Uncategorized | 1 Comment
JSON like ruby object
Recently when creating a rails backed dsl for interacting with Quickbooks SDK, I found myself desiring the ability to have json like behavior on a ruby object. By this I mean being able to create an object with attributes and subobjects on the fly and then being able to access them easily. My solution and with quite a bit of help from jadams was first to reopen the Hash class and insert a method missing to allow for accessing hash['attribute'] with the more rubyesque hash.attribute. The technique and my first ever metaprogramming was inspired by Ola Bini’s excellent blog post, specifically method 4 which uses an example from why’s fantastic camping project.
class Hash def method_missing(m,*a) if m.to_s =~ /=$/ self[$`] = a[0] elsif a.empty? self[m]={} unless self[m] self[m] else raise NoMethodError, "#{m}" end end end
The next step was simply to create a generic object with an attr_accessor to create a hash:
class JSONLikeObject attr_accessor :type, :attributes def initialize(type, attributes = {}) @type = type @attributes = attributes end end
Great! Now I’m able to say foo = JsonLikeObject.new and then say foo.attributes.attribute to access foo.attributes['attribute']. Closer, but I really wanted to be able to say foo.attribute, so I defined method missing for the JSONLikeObject class:
def method_missing(m,*a) if m.to_s =~ /=$/ @attributes.send(m,a[0]) else @attributes[m]={} unless @attributes[m] @attributes[m] end end
Mission accomplished, but I really would like a more complicated/richer tree. For example, customer = JSONLikeObject.new and be able to write customer.contact.address.line1 = ‘Ruby Drive’ and it create customer.attributes['contact']['address']['line1'] = ‘Ruby Drive’ without blowing up. Off to read Ola’s post again and there in example 1 lies the solution:
module Kernel def singleton_class class << self; self; end end end
Now it is simply a matter of taking the method missing from the class and wrapping it in a module that can be included upon initialization:
class Hash def method_missing(m,*a) if m.to_s =~ /=$/ self[$`] = a[0] elsif a.empty? self[m]={} unless self[m] self[m] else raise NoMethodError, "#{m}" end end end module Kernel def singleton_class class << self; self; end end end module Mod def method_missing(m,*a) if m.to_s =~ /=$/ @attributes.send(m,a[0]) else @attributes[m]={} unless @attributes[m] @attributes[m] end end end class JSONLikeObject attr_accessor :type, :attributes def initialize(type, attributes = {}) @type = type @attributes = attributes singleton_class.class_eval do include Mod end end end
And bingo, a json like object in ruby.
Filed under: Uncategorized | Leave a Comment
Search
Recent Entries
- Distributed Processing with EC2 using PoolParty
- Gitting git-svn working on mac
- First contribution from Brian: a class for recursive time-based operations, and our use in a Rails project
- RSS to synthesized speech in 47 inefficient lines
- Ruby Require Idiom
- Thanks O’Reilly
- RSS Aggregator in 26 tasty lines of Ruby code
- Deprec currently broken, but there’s an easy fix.
- Google Spreadsheets Go Camping
- JSON like ruby object
- EC2, Ubuntu, and Rails
Categories
- O'Reilly (1)
- Uncategorized (27)
