Dead Simple Deployment

Posted by ezmobius Wed, 26 Apr 2006 05:20:00 GMT

I have been experimenting with mongrel clusters for the past week or so. I tested pound, balance and finaly settled on pen http://siag.nu/pen/ for a load balancer in front of the mongrel cluster. Pen was the fastest out of those three load balancers in my tests, and is dead simple to set up.

I created a pool of 6 mongrels on my macbook pro and ran pen in front of them. I was able to get 800-900req/sec for static files and 70-150 req/sec for dynamic full blown rails pages. It was also rock solid. I abused the hell out of it with httperf and ab and was very impressed with the performance and stability.

I want to say a huge thank you to Zed Shaw (author of mongrel) and the folks supporting his effort finacially. Mongrel is an awesome addition to the ruby world.

Not once did I get any 500 errors or zombied mongrel processes.

Pen can do either round-robin balancing or ‘sticky session’ adaptive routing. ‘sticky sessions’ means that you can get session affinity between requests. But in my tests round-robin was faster by a little bit. And if sessions are in the db or memcached or drb then sticky sessions don’t matter.

Here is a simple way to test this setup out for yourself. First compile pen: 
$ curl -O http://siag.nu/pub/pen/pen-0.17.0.tar.gz
$ tar xzvf pen-0.17.0.tar.gz
$ cd pen-0.17.0
$ ./configure
$ make
$ sudo make install
Lets install the mongrel & mongrel_cluster gems 
$ sudo gem install mongrel mongrel_cluster --include-dependencies
mongrel_cluster is a gem plugin for mongrel that makes it easy to manage a cluster of mongrels. Thanks to Bradley Taylor for making this available. You can read his blog about it here: http://fluxura.com/articles/2006/04/24/easy-mongrel-clustering-with-mongrel_cluster. Lets configure our rails app to use it. cd into your rails application’s root directory and run: 
$ mongrel_rails cluster::configure -p 5000 -e production -a localhost
This will create the file: 
$ cat config/mongrel_cluster.yml
--- 
num-procs: 1024
docroot: public
timeout: 120
cwd: /Users/ez/ez
log_file: log/mongrel.log
port: 5000
config_script: 
debug: false
environment: production
servers: 6
pid_file: log/mongrel.pid
address: localhost
mime_map:
This is the config file that the cluster::configure just made. The only thing I changed was the number of servers: which i set to 6. Now lets start the mongrel cluster: 
$ mongrel_rails cluster::start
You can now stop or restart the cluster with these commands: 
$ mongrel_rails cluster::stop
or
$ mongrel_rails cluster::restart
Then once the cluster is running we can fire up pen. This command runs pen in debug mode in the foreground so you can see the requests being handled by different mongrels. This is also using round robin. If you turn off -d(debug) and -f(foreground) it will be a little bit faster still. 
$ sudo pen -r -a -f -d  localhost:80 localhost:5000 localhost:5001 \
localhost:5002 localhost:5003 localhost:5004 localhost:5005 localhost:5005

With pen running in the foreground and debug mode like this you can see pen balancing requests between our six mongrels. You will want to turn off the -d and -f switches when you want to run in production.

Then you can go use ab or preferably httperf to test the setup. I think you will agree this little pack of wolves…err.. mongrels is hungry! ;-)

I am very impressed with this setup and it consumes very little resources. Pen is the fastest, lightest HTTP load balancer that can be a front end for https as well, that I could find for free. Its about 5-10% faster then balance and pound in my tests. And I am getting better performance with this setup then I do with lighttpd->standalone fcgi listeners on my macbook.

Very cool stuff.

EDIT: With this type of setup you will need one IP address for each app you want to run. Or you can put lighty or apache in front of pen to use vhosting capabilities. I will post more instructions on how to do this if people need to see them.

Tags , , ,  | 69 comments

Comments

Leave a response

  1. Roberto said about 1 hour later:
    Great article. Have been playing with pen myself and also liked it more than the others. How do you serve statice files ? With mongrel as well ? If yes, do you nfs-mount the public dir (in distributed cluster) or has each mongrel its own public-copy and does page-caching independant ? I think this redundancy would make sense in case one mongrel dies ..
  2. Seth said about 8 hours later:
    What sort of concurrency are you seeing with this configuration? Relative to lighty / fastcgi?
  3. Chris said about 9 hours later:
    Just curious ... now that we're fully out of March and nearly through April, when should we *really* be expecting the (beta) book? Thanks, Chris
  4. Josh H said about 10 hours later:
    Looks like a great setup. I setup my Debian system back in the day with your Debian Rails stack, and I'm wondering how you install Ruby 1.8.4 (need for mongrel) on Debian Sarge. Build from source?
  5. Ezra Zygmuntowicz said about 12 hours later:
    Roberto- Right now I have this setup for mongrel to serve static files. with just under 1000 req/sec for static files with just mongrel it works very well. When mongrel can't handle it anymore I will probably put lighty or apache infront to serve the static files and have lighty or apache proxy to pen and then fan out to the mongrel cluster from pen still. Right now I have only tested this setup on one machine. When it comes time to spead it out on multiple machines I will mount /public with nfs so all mongrels can write to the same page cache.

    Seth- I was running ab with concurrency set to 20-25 and getting great results.

    Chris- I know I'm lagging a bit on the book but it is a ton of work. Rest assured that I am doing everything I can to get it out as a beta before railsconf.

    Josh- You can either use apt-pinning or just add anunstable apt source to get ruby from, Or you can compile your own ruby and use chkinstall to make a .deb package out of it.
  6. Bradley Taylor said about 14 hours later:
    Great write up! Thanks for the bring Pen to light. I'll try it out. If you want more than the default 2 servers, you can call cluster::configure with "-N 6" for 6 servers. In addition, you can specify any mongrel_rails parameter in the call to cluster::configure.
  7. Joe said about 21 hours later:
    Ezra - doesn't this require Lighty as a frontend? If so, what does your relevant Lighty config look like? If not, any reason to use Lighty?
  8. ix said about 21 hours later:
    lighty is useful if youre serving up lots of static files, or parts of the same site are written in PHP or Python.. is your 'testing' carried out on debian? everyone knows OSX's poor threading and syscall overhead makes it unsuitable for deployment..so mongrel/linux besting lighty/osux even in the static dept wouldnt surprise..
  9. Ezra Zygmuntowicz said about 22 hours later:
    Actually the results I am getting is on my intel macbook pro. And is even faster on linux.

    Lighty will beat this setup if you need absolute performance on static files hands down. But close to 1000req/sec for static assets from mongrel is plenty for most sites.

    Joe- This setup is without lighty. Only pen and mongrel. Since mongrel is a full http server pen is just load balancing requests between 6 webservers(mongrels).

    When you need to spread this out to multiple physical hosts, you will need to put /public on nfs or something similar.
  10. Joe said about 24 hours later:
    If you used just Mongrel (and not Pen, Lighty, etc.), how could it serve multiple Rails apps on a single server? My current setup is Lighty->Mongrel->Rails apps (each of which gets its own range of ports).
  11. Stoyan Zhekov said 1 day later:
    > $ sudo install mongrel mongrel_cluster --incldue-dependencies
    think you mean
    $ sudo gem install mongrel mongrel_cluster --incldue-dependencies
  12. Stoyan Zhekov said 1 day later:
    aaa also:
    --include-dependencies
  13. Roberto said 1 day later:
    For the scenarion of scaling up, instead of nfs-mounting, maybe Mogile-FS is an interesting alternative, but I hav not experience with it. Anybody has ? (I made baad experience with NFS-mount and SUN machines and I have heared with LINUX, nfs gives even more problems)
  14. Chris said 1 day later:
    Ezra: Awesome, thanks for the update. =)
  15. Greg said 1 day later:
    If you were to add SSL support to this setup, how would you?
  16. Ezra Zygmuntowicz said 1 day later:
    Read the pen docs in the website I link to. Pen can handle the ssl side of things as well. So pen runs ssl for the external conection and runs standard http to the backend mongrels.
  17. Josh H said 2 days later:
    Looks like the support for SSL in pen is pretty experimental, but you can always use something like stunnel (http://stunnel.org). That brings up another question. If you have pen or stunnel handling the ssl decryption and forwarding http to mongrel, is there any way for the Rails app to know that the incoming request was secure. I use the request.ssl? method in my app, but I have a feeling that would break. Perhaps something could be done with custom HTTP headers being added by pen/stunnel and recognized by Rails.
  18. Austin Godber said 32 days later:
    Nice article Ezra. (You may have already encountered this, but ...) With this setup you might check to see that local_request? tells the truth. I found that telling pen to use localhost:port all the requests appeared to be local requests. This is easily fixed by telling pen to use the host's real IP rather than the loopback IP.
  19. some1else said 34 days later:
    Once again, your post is a revelation to me. Thanks for writing up such a beauty. BTW: Youre the first person I noticed being struck with Trackback spam. Haha
  20. Michael Kovacs said 46 days later:
    Hey Ezra, The SSL question is an important one to me as well. Did you delve into that at all with this setup? I would imagine that even if it's an outage that it can't remain very long because the use of SSL is so prevalent.
  21. Ezra said 47 days later:
    Yeah I was able to get it working with a self signed cert. I haven't tested it extensively yet though. Pound might be a better choice if you need ssl at this point. I will have a new article up soon about apache2.2 mod_proxy_balancer so watch for it.
  22. nate said 122 days later:
    This is a fantastic article! I was hoping you could go into detail for using apache as a frontend through ProxyPass. This is my current setup and it would be very nice to know the last little bit for my configuration. Great job!
  23. Ryan Norbauer said 123 days later:

    "Or you can put lighty or apache in front of pen to use vhosting capabilities. I will post more instructions on how to do this if people need to see them."

    Hi Ezra,

    Yes, please do post on this. I'm running multiple virtualhosts/rails apps on Apache + mod_proxy_balancer + mongrels.

    I hate dealing with ugly un-DRY Apache configs and would love to switch back to Lighttpd's pretty config files. Knowing how to make Lighty + pen + mongrels work with vhosts would be the last step in my move away from Apache to something with less obscurantist configs. :)

    PS: looking forward to the pragprog book! any idea when the beta is due out?

  24. anon-e-mouse said 138 days later:
    Ezra, How do you test for req/sec? Thanks, Mouse
  25. backlink said 222 days later:
    xchange links exchange system exchange service web link exchange travel links exchange recipricol link exchange link exchange english link exchange recipricol link exchange
  26. search engine traffic said 239 days later:
    I’ve use up all my search engine sick days so I’m calling in dead.
  27. Andy said 247 days later:
    I am a linux newbie so please forgive me if this is a stupid question, how can you make instances on mongrel run on other servers and keep all the data in sync (db, etc)?
  28. Pozycjonowanie said 254 days later:
    hate dealing with ugly un-DRY Apache configs and would love to switch back to Lighttpd's pretty config files. Knowing how to make Lighty + pen + mongrels work with vhosts would be the last step in my move away from Apache to something with less obscurantist configs. :)
  29. mac said 287 days later:
    http://www.universityofnorthtexas.info/ http://www.universityofpheonixonline.info/ http://www.universityofsoutherncalifornia.info/ http://www.eye-cream.net/ http://www.free-instant-credit-report.info/ http://www.free-t-shirt.info/ http://www.free-yearly-credit-report.info/ http://www.horses-directory.com/ http://www.music-future.com/ http://www.neck-tie-knots.info/ http://www.painting-books.info/ http://www.bestfreebiesamples.com/ http://www.best-anti-wrinkle-cream.info http://www.craft-book.net/ http://www.credit-card-deal-maker.com/ http://www.credit-free-online-score.info/ http://www.cross-stitch-book.info http://www.entertainment-coupon-book-2006.info/ http://www.4affordablehealthcare.info/ http://www.4laptop-memory.info/ http://www.4costume-jewelry.info/ http://www.4laptop-repairs.info http://www.4jewelry-supply.info http://www.best-antique-store.info http://www.4jewelry-supply.info http://www.any-laptop-accessory.info http://www.4notebook-battery.info http://www.4laptop-reviews.info http://www.4notebook-battery.info http://www.buy-laptop-now.info http://www.4notebook-computers.info
  30. lesbian videos said 323 days later:
    The optimizer could be applied to other things such as well! Anyway good position bro, peace:-)
  31. http://forum.opel24.com said 327 days later:
    I hate Apache configs, but yours website is very usefull
  32. <a href="http://forum.opel24.com" rel="nofollow" rel="nofollow">Opel</a> said 327 days later:
    I hate Apache configs, but yours website is very usefull
  33. Opel said 327 days later:
    I hate Apache configs, but yours website is very usefull Opel
  34. Comment said 327 days later:
    I think these blog is really useful for new comers and Excellent resource list Opel 24 Firanki Kredyty Auto-Moto Forum Opel SEO Karty plastikowe
  35. Black Shemale said 327 days later:
    Am I correct in assuming that this only works when the helper is called directly from the template, as opposed to being abstracted away by some custom defined helper?
  36. Anal Sex said 330 days later:
    It’s actually functionally very similar to Ruby blocks, but with an applied assignment (to whatever the function name is). I wonder if Cheetah can do this, or should do this?
  37. Alice Buses said 335 days later:
    thanks for the update,i hate Apache configs, but yours website is very usefull..
  38. Free Shemale Sex said 342 days later:
    The optimizer could be applied to other things such as well! Anyway good position bro, peace:-)
  39. myspace news blog said 349 days later:
    I love the firefox browser, i think its one of the really good browser. I hope the svg format will be supported from more browser in the future
  40. fee said 355 days later:
    I hate Apache configs, but yours website is very usefull
  41. Sexy Ass said 355 days later:
    Free access to BSD - how to grant it to user from a company, by ssh or ftp. That is my question.
  42. news said 356 days later:
    Hey man, thanks for article, but I still have recurrent problems with explore your site in Internet Explorer 4.0. please fix this problem. it's not very convinient to browse your site if there are no design :(
  43. Anime Girls said 357 days later:
    Posting or comments in this thread about article created above prohibited here or not?
  44. Free Cumshot Videos said 358 days later:
    I am a newbie here, just kidding around:-) anyone from Manhattan?
  45. Nipple Slip said 359 days later:
    Creation from Ruby or from Wordpress is useless from some points of view. IMHO.
  46. Nipple Slip said 359 days later:
    Creation from Ruby or from Wordpress is useless from some points of view. IMHO.
  47. nipple slip said 363 days later:
    Buenos Noches indigo valenico grates tuta!
  48. bang bros said 363 days later:
    I am here to notify you that in the end of April 2007 will be Ruby and WP community party at NY, US.
  49. captain stabbin said 364 days later:
    Captain on the bridge:-) kiddin' around, nice article! :-)
  50. We Live Together said 368 days later:
    Hello everyone, wanna be part of some kind of community, possible here? anyone here?
  51. ass parade said 369 days later:
    Pretty being in Berlin watched their show, welcome everyone - i can pay for invitation!
  52. baghdad dog from lava love marine named war said 369 days later:
    This site is interesting and very informative, nicely interface. Enjoyed browsing through the site.
  53. bang bus said 369 days later:
    Hello everyone. Nice to meet you here ;-) interesting article - i'll put it in bookmarks, i guess will use it any day.
  54. bang bus said 369 days later:
    Hello everyone. Nice to meet you here ;-) interesting article - i'll put it in bookmarks, i guess will use it any day.
  55. bang bros said 373 days later:
    Hello everyone, wanna be part of some kind of community, possible here? anyone here?
  56. big tits said 373 days later:
    Ruby block with link adding in a line seems to be useless, imho.
  57. cartoon wallpaper said 374 days later:
    very good site
  58. coloring pages said 374 days later:
    nice
  59. coloring pages said 374 days later:
    very good
  60. http://insuranceready.googlepages.com said 374 days later:
    a
  61. http://www.m150.org said 374 days later:
    http://www.m150.org
  62. a said 374 days later:
    a
  63. a said 374 days later:
    aa
  64. a said 374 days later:
    aaaa
  65. a said 374 days later:
    a
  66. a said 374 days later:
    a
  67. a said 374 days later:
    a
  68. nice ass said 374 days later:
    How to include PHP using smarty tags?
  69. Bang Bros said 376 days later:
    Hello everyone. Nice to meet you here ;-) interesting article - i'll put it in bookmarks, i guess will use it any day.

RSS feed for this post trackback uri

(leave url/email »)

   Preview comment