Merb *is* Rails

Posted by ezmobius Tue, 23 Dec 2008 18:21:00 GMT

Wow this has been a hectic, emotional week. The Rails and Merb core teams have been silently working together towards this monumental announcement. The announcement is that Merb is Rails and Rails is Merb.

Merb and Rails are joining forces and merging the two code bases for Rails-3.0. We’re targeting RailsConf for the first cut of this magic.

When I first started merb way back in the day it was just a fun little project to hack on. But it became more serious as I explored the options. I always considered merb-core a clean room rewrite of ActionPack and I think this tended to upset some people who were very attached to rails.

Fast forward to current day and Merb-1.0 has been out for a while and there is a whole brouhaha of Rails VS Merb memes on the twitters and blogs. This has to stop as it is tearing apart the community and is very non productive.

So our two teams started talking to see if we could put our differences aside and come together for the common good. We’ve laid out a roadmap of what and how to integrate merb’s best features into rails-3.0. I think we have a great plan that will make Rails the best framework in existence. It will be a blend of the best things we have discovered while working on merb, while keeping the Rails aesthetic people have grown to love.

You can expect to get a kick ass, fast, memory efficient version of rails(merb) this spring!

Merb folks will not be left out in the cold. We will continue to support bug and security fixes for the merb 1.0.x line. And we will provide a clear upgrade path to Rails 3.0 for merb apps. We still have quite a few merb apps running internally at ey and will want an upgrade path for our own apps as well.

This is going to be a lot of work for everyone involved but in the end I think this may just be the most inspiring open source story in history.

I’m so impressed with everyone involved. All the core team members from both sides have been able to put away the weapons and egos and really come together, committed to making this transition work seamlessly.

2009 is going to be a seriously strong year for the ruby community. With everyone working together, we have such a kick ass team of smart people that this whole framework thing is going to be a solved issue.

Merb is dead, long live Merb(Rails-3.0)!



http://rubyonrails.org/merb
http://weblog.rubyonrails.org/2008/12/23/merb-gets-merged-into-rails-3
http://yehudakatz.com/2008/12/23/rails-and-merb-merge/

Tags , ,  | 65 comments

Merb master/worker monit control setup

Posted by ezmobius Sun, 07 Dec 2008 19:55:00 GMT

Going into 1.0, Merb has changed the way it handles options and processes. This tends to break people’s existing monit configs. So today I’m releasing a custom Engine Yard monit script for controlling merb master/worker processes.

You can get the script from here: monit_merb_mpc. Put this somewhere in your $PATH, or make it live here: /engineyard/bin/monit_merb_mpc and make sure it is executable.

In order to make proper use of this script it assumes certain paths are setup. In this example we will assume you have an app called ‘iggy’ that lives at /data/iggy/current, where /data/iggy is the main app dir that contains the capistrano releases and current directory. The script assumes these paths so if you want top use different paths you will need to hand edit the script as desired.

This script also assumes a logging directory of /var/log/engineyard/$appname, you can edit the script if you want to change this as well.

You need to have one master configured per app, as well as monit configuration for each worker.

Master Config

check process merb_iggy_master
  with pidfile /var/log/engineyard/iggy/iggy-production-merb.main.pid
  start program = "/engineyard/bin/monit_merb_mpc iggy start_master -c3 -n5000" 
  stop program = "/engineyard/bin/monit_merb_mpc iggy stop_master" 
  #if totalmem is greater than 80.0 MB for 2 cycles then restart       # eating up memory?
  group merb_iggy

Note that the start program needs to be adjusted if you need more/less instances or want to run the workers on different ports /engineyard/bin/monit_merb_mpc iggy start_master -c -n . Also, memory monitoring is turned off for this process because the memory includes all processes under its control, including the workers and they are already monitored.

Worker Config (one for each port)

check process merb_iggy_5000
  with pidfile /var/log/engineyard/iggy/iggy-production-merb.5000.pid
  start program = "/engineyard/bin/monit_merb_mpc iggy register_worker 5000" 
  stop program = "/engineyard/bin/monit_merb_mpc iggy restart_worker 5000" 
  if totalmem is greater than 80.0 MB for 2 cycles then restart       # eating up memory?
  group merb_iggy

How it works

Since Merb 1.0 (actually version 0.9.8 and above) uses a master process and a spawner process in addition to the worker processes, the monit_merb_mpc (multi-process-control) script needs to control the master process separately from the worker processes. To accomplish this, there is a fair amount of trickery to make monit treat the merb processes in a way monit was never intended to be able to handle. When the master process is started, it will create all the worker processes, so when monit starts the worker processes, they are infact already started – so the start command for the worker simply “registers” the worker – in other words, it just ensures that the pid files contains the worker pid. Also since the master process will automatically restart the worker processes, the stop action for the workers is actually a restart.

Notes:

Although this looks like it should use the monit dependency directive, the scripts are actually written to not need it, and making the worker processes dependent on the master process, it will cause problems during shutdown. since the master specified the number of worker processes using the -c option, you can’t simply shutdown a single worker process by stopping it with monit – it will simply restart by itself and operate unmonitored, thus making it a possible runaway process.

Thanks to all the EY support guys who worked on this one to make it smooth. Hopefully this clears up how to deploy merb with the master/worker server cluster setup under monit.

7 comments

Ruby Deployment and Engine Yard as a Service

Posted by ezmobius Tue, 18 Nov 2008 20:54:00 GMT

I gave a talk today at the #prorubyconf on Ruby Deployment, past present and future. You can view my slides below.

We will start doing some private beta testing of our new EY platform in December so let me know if you want to beta test and havce some interesting apps to run in the cloud.

Ruby Deployment
View SlideShare presentation or Upload your own. (tags: cloud engineyard)

Tags ,  | 16 comments

Merb 1.0

Posted by ezmobius Sat, 08 Nov 2008 06:08:00 GMT

Umm, w00t! It’s been a little over two years since merb was a twinkle in my eye, and a pastie. Since then it has undergone many drastic transformations, working its way towards a very solid, fast foundation for people to build their homesteads on.

It’s very exciting for me to see it grow into what it wants to be, based on the awesome merb community that has popped up around the project. It just shows that if you put in the work on something you love, polishing every little corner it will attract other like minded individuals and they will help you take the project to levels higher then your wildest dreams.

So, gem install merb and get the 1.0 final now, it is propogating its way through the rubyforge tubes. If you can’t get it from the ‘forge yet then you can get it from our gem server:


gem install merb—source http://edge.merbivore.com


Thank you to everyone who helped make this happen with code, feedback and blood, sweat and tears. It’s been totally worth the effort IMHO.

The merb is dead, long live the merb.

Tags  | 53 comments

MerbCamp Keynote and Introducing Nanite

Posted by ezmobius Sat, 11 Oct 2008 18:41:00 GMT

Just a quick post to share some news. I just finished giving my keynote at MerbCamp about past and present merb and the core tenets of merb development.

I also introduced Nanite: self assembling cluster of ruby processes

You can watch the full video of my talk here: Keynote Video

WMV format ^^ Thanks to Matt Jaynes for mirroring the video on a higher bandwidth connection

And you can see my slides on slideshare below.

Merb Nanite
View SlideShare presentation or Upload your own. (tags: merb nanite)

Thanks to all the merbcamp organizers, the venue is awesome and I have never spoken at a ruby conference that streamed all of the talks live. Very cool setup here.

I will follow up with some more posts on nanite and the what why and how of it shortly. But you can get a pretty good overview of it by watching my talk.

Tags ,  | 15 comments

EY Looking for Mad Scientists

Posted by ezmobius Tue, 19 Aug 2008 22:51:00 GMT

We’ve got a few job openings I wanted to float out there for your consideration. Here are the listings:

Ruby Developer

Systems Engineer in Sacremento

Systems Engineer

We need another team member for the Vertebra project. You know ruby like the back of your hand and are interested in distributed systems as well as learning erlang.

Systems engineers work on our cluster systems and will help with our custom Gentoo linux build as well as be involved with the Load Balancers, Switching Fabric, Linux, Xen, Coraid and helping design the latest and greatest platform for running Ruby applications in the cloud.

If you think you have the chops for any of these jobs then we want to hear from you. Please send a resume and a short email describing who you are and why you kick ass to jarnold@engineyard.com and cc ezra@engineyard.com

Tags ,  | 3 comments

More info on Vertebra

Posted by ezmobius Wed, 30 Jul 2008 20:52:00 GMT

I gave a talk on Vertebra today at Pivotal Labs. This time it was a bit more technical then my last talk at railsconf as the system has evolved since then. We’re working furiously on this thing and hope to have the first release very soon.

We will be opening up an early beta for a few folks so if you have some wicked cool idea you want to use Vertebra for get in touch and a few people can get added to the early access.

Video was taken today of the talk and should get posted online within a few weeks.

Slides are on slideshare here if you cannot view the embedded version below.

Tags , , ,  | 5 comments

Nginx Upload Module

Posted by ezmobius Sun, 20 Jul 2008 20:32:00 GMT

Valery Kholodkov has written a very cool nginx module for handling uploads.

The way this works is that you specify a location block to handle the uploads. So if you are using the standard nginx.conf for rails apps then you would add this in your server block right above your “location /” block:

    # Upload form should be submitted to this location
    location /upload {
      # Pass altered request body to this location
      upload_pass   /internalupload;

      # Store files to this location
      upload_store /tmp;

      # Set specified fields in request body
      upload_set_form_field $upload_field_name.name "$upload_file_name";
      upload_set_form_field $upload_field_name.content_type "$upload_content_type";
      upload_set_form_field $upload_field_name.path "$upload_tmp_path";
    }

    # Pass altered request body to a proxy
    location /internalupload {
        proxy_pass   http://mongrel;
    }

Then make a simple upload form that does a multipart POST to /upload. Now you can have your Rails or Merb app on the backend with a route called /upload. In the action of your app that responds to the /upload route you will get a set of params that look like this(assume the name of your upload fields is called ‘file1’ and ‘file2’):

{"file2.path"=>"/tmp/0000123459", "file1.path"=>"/tmp/0000123458",
"file2.content_type"=>"image/png",  "submit"=>"Upload",
 "file2.name"=>"Picture 2.png", "action"=>"index", 
"file1.name"=>"Picture 1.png",  "controller"=>"test", 
"file1.content_type"=>"image/png", "test"=>"value"}

What this is doing if parsing the multi-part mime boundaries in C in the nginx plugin, putting the parsed files into files in /tmp and then stipping the multipart stuff out of the POST body and replacing it with the info you need to get the name and location of the file on disk.

This means that by the time the request hits your application, the expensive mime parsing is already done and you simply move the file to it’s final resting place. This is a huge win since now the hard work is done in C in nginx before your app ever gets involved.

Of course this is a fresh new module so do your own testing and deciding whether or not this is a fit for your needs. But I think this is a great plugin and have verified it works as advertised.

Tags ,  | 10 comments

Engine Yard takes $15mill Series B Round from NEA, Amazon and Benchmark

Posted by ezmobius Mon, 14 Jul 2008 00:17:00 GMT

Man it seems like yesterday that Engine Yard was a small 3 person startup with big ideas and little cash. People seemed to like what we were offering and the business grew into a thriving startup.

In the beginning we only had plans to be the best fully managed rails/ruby hosting company. And I think we met this goal early on. But it became apparent that we had a strong brand and we started kicking around ideas of becoming an open source software company as well.

We believe in Ruby as a platform, and we’ve put our money where our mouth is when we sponsored Rubinius and Merb, hiring many developers to work on both projects.

Engine Yard has become much more then just a managed rails hosting company, we want to strengthen the Ruby ecosystem for everyone by providing the infrastructure and open source software for the next wave of Ruby deployments in the cloud.

We’ve also been delving into the cloud computing arena as I think the next 5 years are going to see huge transition from standard hosting models into the cloud. Our upcoming Vertebra project is a new application programming platform for building distributed cloud applications with XMPP. You can expect to see the first open source release of Vertebra this summer, I think this is a truly unique and very fun project to work on. I think a lot of folks out there will have tons of different use cases for it.

So fast forward to right now, Engine Yard now has more then 80 employees worldwide. Half of these people are Application Support and SysAdmins working to support all the awesome applications we host. Providing 24/7 support. But we also have a growing Engineering team working on all kinds of exciting stuff. I consider myself lucky to work with such a talented team.

So with all that being said, I’m excited to annouce that we have just closed our Series B round of VC funding totalling $15million dollars(insert austin powers joke here)! This round includes investment from NEA, Amazon and Benchmark.

We’re going to use this money towards making Ruby the platform of choice for cloud computing and web development in startups and the enterprise alike.

Watch this space, we have lots of exciting announcements in the coming months. I’d like to thank all of our customers and users of our open source software and the Ruby community in general. It has been a wild ride these last 2 years and I expect many more exciting things to come!

Tags  | 31 comments

Rubyology Podcast Interview

Posted by ezmobius Mon, 23 Jun 2008 02:06:00 GMT

Chris Matthieu and Steven Bristol interviewed me for the Rubyology Podcast a few days ago and they have posted the interview here. I talk about my early days of computing, merb, vertebra, rubinius and the story behind how Engine Yard got started and has grown. I’ll warn you though it’s a long interview, 120 minutes! Have a listen if you want to hear the story.

Tags , , ,  | 2 comments

Older posts: 1 2 3 4 ... 15