BackgrounDRb updated.

Posted by ezmobius Sun, 02 Jul 2006 01:44:00 GMT

Howdy Folks-

I'm happy to announce a new release of BackgrounDRb! I have added quite a few new features and included some nice patches from folks on the list.
$ script/plugin install svn://rubyforge.org//var/svn/backgroundrb

Click through for the details.

All of the code now stays within the plugin and the start and stop scripts are now just stubs. This makes it easier to tweak or figure out how it works. The backgroundrb.rb file in the root of the plugin dir is the drb server part of MiddleMan and the pluginroot/lib/backgroundrb.rb file is the rails client side MiddleMan.

One of the new things is much better threading job control. This is mostly invisible to you as a user of the plugin but it makes the whole system a lot more sturdy. It gives your worker objects the chance to clean up after themselves when they get deleted instead of just getting the kill brick dropped on them ;) Thanks to David Lemstra for the job management code!

The other big new feature is a built in timer and a time to live argument for workers and caches. This allows you to set how long a worker or a cache survives before it gets reaped. The :ttl param defaults to :immortal for worker classes and 10 minutes for cached objects. If you don;t specify the :ttl param then your workers will only get reaped if you directly call delete_worker on them or use gc! to override. If you want longer or shorter ttl's then specify :ttl in *seconds* like this:
#worker
session[:job_key] = MiddleMan.new_worker(:class => :foo_worker,
                                         :args => "Bar",                                                                                          
                                         :ttl => 300) # 300 seconds == 5 minutes

#caching

#fill_cache, expire in 5 minutes
@posts = Post.find(:all, :include => :comments)
MiddleMan.cache_as(:post_cache, 300, @posts)
#OR
@posts = MiddleMan.cache_as :post_cache, 300 do
  Post.find(:all, :include => :comments)
end

#retrieve_cache
@posts = MiddleMan.cache_get :post_cache do
  Post.find(:all, :include => :comments)
end

By default the timer_sleep is set to 60 seconds. This means that the timer will wakeup every 60 seconds and delete caches and worker that are past their time to live. You can change this in the config file if you want it to run faster or slower.

All your worker classes should now inherit from BackgrounDRb::Rails so they get all the good new thread management features. If you are already using the plugin you should delete the plugin from your rails app and reinstall it. You should also delete the script/backgroundrb directory and the config/backgroundrb.yml file. Then run:

# to install the new code.
$ rake backgroundrb:setup

Your workers should not need to change as long as they inherit from BackgrounDRb::Rails. Please test this and give me feedback if you are using it. I would also like to start collecting worker classes or at least hear what folks are using this plugin for.

*Home page: http://backgroundrb.rubyforge.org *svn repo: svn://rubyforge.org//var/svn/backgroundrb *Mailing list: http://rubyforge.org/mailman/listinfo/backgroundrb-devel

Tags , ,  | 18 comments

Comments

  1. dvdplm said 2 days later:
    Upgraded today and it works seamlessly! Nice nice nice... :-) I use BackgrounDRb to do some heavy CSV parsing. Some files contain some about 10 000 rows and for each row there are quite a bunch of SELECTs (through AR) and, if needed, a few INSERTs. Processing times are around 5 to 15 minutes and I do some AJAX stuff to poll progress and show some feedback to users. I'm very new to rails and to ruby, so it might just be me, but I'd very much appreciate some more details/tutorials regarding theBackgrounDRb caching meccanism: why should I use the cache? Would the scenario outlined above benefit from caching in any way? Isn't A-R already caching some of my stuff? Great job. I do plan to use this for other projects as well. My compliments.
  2. Ezra said 3 days later:
    dvdplm-

    Thanks for the kind words ;) The caching feature probably won't help you with the cvs parsing your talking about. The main use of the cache is to store complex queries in memory so that you don't have to hit the database everytime someone requests that data. You can also cache views that have been render_to_string. Its kind of like a very simple version of memcached. You only use the caching from the rails side of thing. So as an example, say you have a huge tag cloud that you need to render on every page. And you want it to expire and get rebuilt every 5 minutes. So you cache it using cache_as and then this big query only runs once every 5 minutes and all requests for it get served from an in memory cache. And when you use the block form of cache_get you can specify a query that happens when the cache is deleted. So your tag cloud gets automatically updated every 5 minutes but gets served from the cache on all other requests.

    For example:
    # fill cache, expire in 5 minutes
    @tags = MiddleMan.cache_as :tags_cache, 300 do
      Tag.find(:all, :include => :meta_tags)
    end
    
    #retrieve_cache, if the cache for the :tags_cache key
    # is empty, rebuild it with the data inside the block
    # and return the query to fill @tags. 
    @tags = MiddleMan.cache_get : tags_cache, 300 do
      Tag.find(:all, :include => :meta_tags)
    end
    
  3. Max said 6 days later:
    Good work! That's exactly what I've searched for. Unfortunately I keep getting "undefined method `[]' for #" ... My worker does heavily us my ActiveRecord Models. I've already tried to name them all in application.rb through "model :user, :account ..." but with no luck. Any ideas?
  4. Max said 6 days later:
    Sorry, the comment ate the brackets. Its "undefined method [] for #<DRb::DRbUnknown:0x12345%gt;"
  5. Damien Tanner said 8 days later:
    Yo Ezra, I'm getting the same error as Max. When I set :args to a hash or array (string is fine) it throws back with "undefined method `[]' for #" http://pastie.caboo.se/3963 has the full trace. Using ruby 1.8. Damien.
  6. Ezra said 8 days later:
    If you get these errors it means that you are trying to send something to the drb server that cannot be marshalled over drb. This is the case when you want to send an active record to a worker as an arg. To get around this you need to add include DRbUndumped into the class def of whatever model you want to send as and arg. like so:

    class Post < ActiveRecord::Base
      include DRbUndumped
      # rest of code...
    end
    

    If you still have errors then please send them to me on the backgroundrb maling list with code samnples and error traces.

    Damien- Its unclear if you are trying to send AR objects in the has as args. If you look in the test cases for the plugin you can see that sending hashes and arrays work fine. To get this error it means you are trying to send something across the wire that cannot be marshalled. So try to track down what that is and include DRbIUndumped in it and you will be fine.
  7. akshat said 10 days later:
    Hi, I want to start Backgroundrb when my system(linux) boots up.....but am unable to do so from my rc.local script....any idea how i could do so thanks, akshat
  8. John said 46 days later:
    I know it's been kind of mentioned before, but I'm trying to send an AR object to a worker. I get none of the above errors, but in the worker when I do:
    @logger.warn args[:test].inspect
    I get a DRb::DRbObject. I'm not sure what to do with this.
    I've already put DRbUndumped in the model in question (and restarted everything), but this didn't change anything.
    Am I missing something obvious, or should I just be sending an ID and loading the AR object from the database inside the worker?
  9. Ezra said 50 days later:
    DRb::DRbObject is what you get when you access an object over drb that includes DRbUndumped. So that is actually your AR object there, inspect just says that because of the way drb works. you can still call your methods on it like it was an ar instance. You can also just pass id's over the wire as well. Either way works fine.
  10. Kaszuby said 72 days later:
    It,s something wrong with this side. Best regards.
  11. Andrew said 79 days later:
    What would be the implications of say, having a timer_sleep of 1. The reason I ask is, if I leave it at 60, some screwy stuff happens. For instance, in the following code, I get the correct result if I have a timer_sleep of 1, however I get a nonsense result if I leave it at 60 (more updates than are rows in the database, rows updated more than once). class ReminderWorker < BackgrounDRb::Rails repeat_every 10.minutes first_run Time.now @secondreminders = ExampleItem.find(:all, :conditions=>["emailconfirmed=0 and confirmation_reminders=1"]) @secondreminders.each do |secondreminder| @logger.info "Second reminder" secondreminder.update_attribute(:confirmation_reminders,2) end @firstreminders = ExampleItem.find(:all, :conditions=>["emailconfirmed=0 and confirmation_reminders=0"]) @firstreminders.each do |firstreminder| @logger.info "First reminder" firstreminder.update_attribute(:confirmation_reminders,1) end end thanks for your help
  12. kodecharlie said 102 days later:
    I am using a version of Ruby 1.8.4 installed as a cygwin package on Windows XP.

    I tried installing backgroundrb like this:
      script/plugin install svn://rubyforge.org//var/svn/backgroundrb
    
    This failed with this error report:
    svn: '\home\Charlie\cdd\ruby\vendor' is not a working copy
    svn: '\home\Charlie\cdd\ruby\vendor' is not a working copy
    ./script/../config/../vendor/rails/railties/lib/commands/plugin.rb:257: warning: Insecure world writable dir /usr/local/bin, mode 040777
    svn: Can't create directory '\home\Charlie\cdd\ruby\vendor\plugins\backgroundrb'
    : The system cannot find the path specified.
    
    I assume the path problem is caused by some cygwin-specific issue. So, I then tried this:
      cd vendor/plugins
      svn co svn://rubyforge.org/var/svn/backgroundrb
    
    This retrieved the backgroundrb package. I then tried running unit tests:
      cd backgroundrb
      rake
    
    It appears one unit-test is failing:
    $ rake
    (in /home/Charlie/cdd/ruby/vendor/plugins/backgroundrb)
    /usr/bin/ruby -Ilib:lib "/usr/lib/ruby/gems/1.8/gems/rake-0.7.1/lib/rake/rake_te
    st_loader.rb" "test/test_backgroundrb.rb" "test/test_helper.rb" "test/test_worke
    rs/cron_worker.rb" "test/test_workers/test_worker.rb"
    /usr/lib/ruby/gems/1.8/gems/rake-0.7.1/lib/rake.rb:728: warning: Insecure world
    writable dir /usr/local/bin, mode 040777
    Loaded suite /usr/lib/ruby/gems/1.8/gems/rake-0.7.1/lib/rake/rake_test_loader
    Started
    .F.......
    Finished in 1.397 seconds.
      1) Failure:
    test_caching(BackgrounDRbTest) [./test/test_backgroundrb.rb:105]:
     is not true.
    9 tests, 65 assertions, 1 failures, 0 errors
    rake aborted!
    Command failed with status (1): [/usr/bin/ruby -Ilib:lib "/usr/lib/ruby/gem...]
    
    I then tried this:
      rake --trace backgroundrb:setup
    
    This also had problems:
    (in /home/Charlie/cdd/ruby/vendor/plugins/backgroundrb)
    rake aborted!
    Don't know how to build task 'backgroundrb:setup'
    /usr/lib/ruby/gems/1.8/gems/rake-0.7.1/lib/rake.rb:1449:in `[]'
    /usr/lib/ruby/gems/1.8/gems/rake-0.7.1/lib/rake.rb:455:in `[]'
    /usr/lib/ruby/gems/1.8/gems/rake-0.7.1/lib/rake.rb:1906:in `run'
    /usr/lib/ruby/gems/1.8/gems/rake-0.7.1/lib/rake.rb:1906:in `each'
    /usr/lib/ruby/gems/1.8/gems/rake-0.7.1/lib/rake.rb:1906:in `run'
    /usr/lib/ruby/gems/1.8/gems/rake-0.7.1/bin/rake:7
    /usr/bin/rake:18:in `load'
    /usr/bin/rake:18
    
    Is there anything special I need to do for the cygwin/Ruby/backgroundrb setup?

    Thanks,
    kodecharlie
  13. Ezra said 103 days later:
    I have never tried this on cygwin and I think it should work fine but I don't have cygtwin to test this out with. Make sure the rake tasks exist in the plugin your downloaded, they are in there and should work fine. Do you have an old version of rake perhaps?
  14. Matt White said 140 days later:
    What's the best way to debug Workers? For some reason I'm not getting any errors thrown back when code fails in my workers, so debugging is quite a pain. The script just terminates on the failing line, but with no explanation. I've tried wrapping the code in rescue, but that doesn't always work. It seems that everyone else that has posted hasn't had any issues with this, so perhaps something isn't configured correctly on my end. I'm running Tiger, and I've been scanning the backgroundrb.log and the backgroundrb_server.log files, but they never seem to give me any reason why my code is failing. Aside from that, it's working quite well! Thanks for the excellent work, Ezra!
  15. Jarszon said 233 days later:
    DRbObject is what you get when you access an object over drb that includes DRbUndumped.
  16. big tits said 306 days later:
    Ruby block with link adding in a line seems to be useless, imho.
  17. nice ass said 307 days later:
    How to include PHP using smarty tags?
  18. bang bros said 308 days later:
    I use psvn as my Subversion client: it’s not perfect but it’s more than good enough!

(leave url/email »)

   Preview comment