Brainspl.at

BackgrounDRb initial release

Posted by ezmobius Mon, 15 May 2006 17:57:00 GMT

I’m happy to announce the first alpha release of BackgrounDRb for rails. I plan to extend this framework to meet the needs that arise for long running background tasks. It is currently in a very good working state but I want to release it and get feedback from people as to how they will use this setup.

So please use and abuse this and let me know how it works or doesn’t work for you.

You can get BackgrounDRb from subversion here:

svn://rubyforge.org//var/svn/backgroundrb

Click Thru for the README on what this is all about

[EDIT] Added ActiveRecord caching with a simple interface. Like so:

def fill_cache
  @posts = Post.find(:all, :include => :comments)
  MiddleMan.cache_as(:post_cache, @posts)
end  

def retrieve_cache
  @posts = MiddleMan.cache_get(:post_cache)
end

Copyright (c) 2006 Ezra Zygmuntowicz

BackgrounDRb is a small framework for divorcing long running tasks from
Rails request/response cycle. With HTTP it is usually not a very good
idea to keep a request waiting for a response for long running actions.
BackgrounDRb also allows for status updates that in combination with
ajax can render live progress bars in the browser while the background
worker task gets completed. The MiddleMan can also be used as a cache.
You can store rendered templates or compute intensive results in the
MiddleMan for use later. 

The MiddleMan drb server is a front controller or factory/delegate type 
of object. It takes instructions from you and instantiates your worker 
objects with the args you send from rails. It uses a hash to keep a  key 
pointing to a running worker that is also put into the session in rails 
so railscan find the same Worker object that is running its job on 
subsequent requests. The MiddleMan front object has a method that takes 
a class type as a symbol  and instantiates a new instance of said 
class type. Then it returns a job_key to the client(rails). Rails can
then call a method on that object through the MiddleMan class.

There are many possible use cases for this system. More will be implemented
soon. This is an open request for comments and feature requests. Let's 
look at how this system works in detail.

To install BackgrounDRb you need to follow these steps:

1. Copy the backgroundrb folder to RAILS_ROOT/script/
2. Make sure RAILS_ROOT/script/backgroundrb/start and RAILS_ROOT/script/backgroundrb/stop
   are executable and have the correct #! lines that will work with your system.
3. Make a workers directory in RAILS_ROOT/lib/workers. This is where you will put your
   custom worker classes. 
4. Once BackgrounDRb is installed in your rails app you will have a start and stop
   script to use. You must start the drb server before you start rails
   To start the drb server you use this command from your RAILS_ROOT:

   # to start
   $ script/backgroundrb/start -p 22222 -d

   That will start the drb server on port 22222 in the background and give you
   back control of the shell. Without the -d flag the drb server will run in
   the foreground which is helpful for debugging and experimentation. If you 
   don't specify the -p port number it will default to 22222.

   # to stop
   $ script/backgroundrb/stop   

   You will need to add a few lines to your RAILS_ROOT/config/environment.rb file.

require "drb" 
DRb.start_service
MiddleMan = DRbObject.new(nil, "druby://localhost:22222")
class << MiddleMan
  def cache_as(named_key, data)  cache(named_key, Marshal.dump(data)) if data end
  def cache_get(named_key) Marshal.load(self[named_key]) end
end

   Make sure to set the port number to the one you started the drb server with.
   The port num defaults to 22222.

Lets look at a simple worker class.

class FooWorker
  include DRbUndumped

  def initialize(options={})
    @progress = 0
    @results = []
    @options = options
    start_working
  end

  def start_working
    # Work loop goes inside a new thread so it doesn't block
    # rails while it works. A neat way to do progress bars in
    # the browser is to have a @progress instance var that is
    # initialized to 0 and then gets bumped up by your long 
    # running task. This way you can poll for the progress
    # of your job via ajax and update a client side progress bar.
    Thread.new do
      # main work loop goes here. do work and update the
      # progress bar instance var.
      while something
        @results << foo(@options)
        @progress += 1
        break if @progress > 99
      end
    end  
  end

  def results
    @results
  end

  def progress
    puts "Rails is fetching progress: #{@progress}" 
    @progress
  end      
end

Your worker classes go into the RAILS_ROOT/lib/workers/ directory.
You can then use your worker class in rails like this:

# in a controller

# start new worker and put the job_key into the session so you can 
# get the status of your job later.
def background_task
  session[:job_key] = MiddleMan.new_worker(:class => :foo_worker,
                                           :args => {:baz => 'hello!', :qux => 'another arg!})
end

def task_progress
  if request.xhr?
    progress = MiddleMan.get_worker(session[:job_key]).progress
    render :update do |page|
      page.replace_html('progress', 
                        "<h3>#{progress}% done</h3>" +
                        "<img src='/images/progress.gif' width='#{progress * 2}' height='15' />")
      if progress == 100
         page.redirect_to :action => 'results'
      end   
    end 
  else
    redirect_to :action => 'index'   
  end
end

def results
  @results = MiddleMan.get_worker(session[:job_key]).results
  MiddleMan.delete_worker(session[:job_key])
end

Please note that when you use new_worker it takes a hash as the argument.
the :class  part of the hash is required so MiddleMan knows which
worker class to instantiate. You can give it either an underscore
version like :foo_worker or normal like :FooWorker. Also the :args key
points to a value that will be given to your worker class when initialized.
The following will start a FooWorker class with a text argument of "Bar" 

session[:job_key] = MiddleMan.new_worker(:class => :foo_worker,
                                         :args => "Bar")

In the background_task view you can use periodically_call_remote
to ping the task_progress method to get the progress of your job and update
the progress bar. Once progress is equal to 100(or whatever you want) you 
redirect to the results page to display the results of the worker task.

There are a few simple examples in the workers dir. These are the worker classes
I show being used here for proof of concept:

http://brainspl.at/drb_progress.mov
http://brainspl.at/drb_ajax_tail.mov

If you want to play with the demo app that implements those two movies then 
you can check out the rails app here to play with:

http://opensvn.csie.org/ezra/rails/plugins/backgroundrb/

If you want to have a named key instead of generated key you can specify the
key yourself. This is handy for creating shared resources that more then one 
user will access so that multiple users and backends can get the same object
by name.

MiddleMan.new_worker(:class => :foo_worker,
                     :args => "Bar" 
                     :job_key => 'shared_resource')

For caching ActiveRecord Objects you can now use the following syntax:

def fill_cache
  @posts = Post.find(:all, :include => :comments)
  MiddleMan.cache_as(:post_cache, @posts)
end  

def retrieve_cache
  @posts = MiddleMan.cache_get(:post_cache)
end

For caching text or simple hashes or arrays or even rendered views you 
can use a hash like syntax on MiddleMan:

MiddleMan[:cached_view] = render_to_string(:action => 'complex_view')

Then you can retrieve the cached rendered view just like a hash with:

MiddleMan[:cached_view]

You could create this cache and then have an ActiveRecord observer
expire the cache and create a new one when the data changes. Delete
the cached view with:

MiddleMan.delete_cache(:cached_view)

Best practice is to delete your job from the MiddleMan when you are done with
it so it can be garbage collected. But if you don't want to do this then you 
can use another option to clean up old jobs. Using cron or maybe RailsCron
for a time, you can call the gc! method to delete jobs older then a certain time.
Here is a ruby script you could run from cron that will delete all workers
older then 24 hours.

#!/usr/bin/env ruby
require "drb" 
DRb.start_service
MiddleMan = DRbObject.new(nil, "druby://localhost:22222")
MiddleMan.gc!(Time.now - 60*60*24)

** ROADMAP **
1. **DONE** Add better ActiveRecord caching facilities. Right now you can cache text, hashes, arrays and many 
   other object types. But I am still working on the best way to cache ActiveRecord 
   objects. I will probably use Marshal or YAML to do the right thing here
2. More examples. A chat room is forthcoming as well as an email queue.
3. More documentation.
4. Profit.. ?

Thoughts and feature requests most welcome.