Brainspl.at

You got ur Erlang in my Ruby

ezmobius — Thu, 30 Apr 2009 18:56:00 +0000

Here are the slides from my keynote this morning at the SF Erlang Factory conference:

Engine Yard hiring developers to help build our cloud

ezmobius — Tue, 07 Apr 2009 20:19:00 +0000

I’m looking for a few good folks to join my team. You need to be able to work on site in San Francisco, we can assist with moving expenses for the right candidate.

You would work with my team(currently 5 devs). We do no bs agile style development with pairing where appropriate and test infected mentality . generally we have a fun, tight team and get to work on some of the hardest and most rewarding areas of software development relevant today.

You would be on my team working on http://engineyard.com/solo and it’s successors as well as Engine Yard’s on premises infrastructure.

You would be working on/with the following ruby technologies:

Ruby
Merb
Rails
Sinatra
DataMapper
Nanite
Chef
Event Machine
Many others..

You would be working on the following problem domains:

Scalable Ruby Deployment Architectures
Scalable Cloud Computing Architectures
Scalable Database Architecture
Distributed Computing
Key Value Data Stores
Infrastructure automation
Cloud provider API's
Multi-Cloud portability
Monitoring and alerting systems
Security in the Cloud
Horizontally Scalable Architectures
Many other interesting areas of computing

Familiarity with the following tech is nice but not necessarily required if you can learn fast:

Gentoo Linux
Nginx
Apache
Monit
Daemontools
Mysql
Postgresql
Memcached
Passenger
Mongrel
Thin
SSL Certs
Iptables/Networking
Erlang

If you think you have the “right stuff” and you have the kick ass take names attitude then send a resume with a short intro about who you are and why you are the person we should hire.

ez@engineyard.com

Cooking with Chef 101

ezmobius — Sat, 31 Jan 2009 19:00:00 +0000

One of the things I really like about chef is that it scales down as well as up. You can get started using it right away with chef-solo and then graduate to chef-client and chef-server when you need more cowbell. And then as you grow you can horizontally partition each component of the chef infrastructure, web servers, couchdb servers with messaging queues to decouple each part so you can go from tiny to large and in charge with the same recipes.

So let’s get started with a simple set of recipes to install some rubygems, and setup some directories for deploying a few apps with capistrano after chef configures the base system.

First you will need to clone my chef-101 repo from github:

http://github.com/ezmobius/chef-101

$ git clone git://github.com/ezmobius/chef-101.git

First we need to get chef installed.

$ sudo gem install chef ohai --source http://gems.opscode.com --source http://gems.rubyforge.org

(the two sources are needed so that dependencies can be installed form rubyforge even though chef is hosted on gems.opscode.com. Rubygems quirk but at least it works this way ;)

Now lets just run the recipe set so you can see chef in action! (this will install a few rubygems on your system so you can skip this step if you want. It will also create a few directories underneath /data.)

You will need to change the user attribute in the config/dna.json file to a user that exists on your system before this will run(unless you have an ‘ez’ user like I do ;)

$ cd chef-101
$ sudo chef-solo -l debug -c config/solo.rb -j config/dna.json

You will see a bunch of output scroll by, first you will see the ohai output, ohai is the systems information collector. The you will see a bunch of info about compiling recipes and applying recipes. It will end with this line if it was successful:

INFO: Chef Run complete in 1.093957 seconds

Congratulations! You are now cooking with fire. Now let’s take a look at the recipes and how they are put together so you know how this thing works.

First off what is this dna.json file we passed in to chef-solo?

{
  "apps": [
    "beast",
    "mephisto" 
  ],
  "user": "ez",
  "gems": [
    {
      "name": "rake",
      "version": "0.8.3" 
    },
    {
      "name": "tmm1-amqp",
      "source": "http://gems.github.com",
      "version": "0.6.0" 
    }
  ],
  "recipes": ["main"]
}

This is what I am calling the JSON DNA of your recipe set. These are the parameters that will be available in your recipes as the node object. So you can see we have some applications, a username and some rubygems to install.

You can see that we have set the “recipes” array to contain ‘main’. Let’s take a look at the main recipe, as this is our entry point into the system.

include_recipe 'gems'
include_recipe 'applications'

template "/data/someservice.conf" do
  owner node[:user]
  mode 0644
  source "someservice.conf.erb" 
  variables({
    :applications => node[:apps]
  })
end

You can see that we use include_recipe, this command will pull in the named recipe and run it in place top down right there in the recipe. This allows for chef recipes to run in deterministic order every time, if you want one resource to run before another, just make sure it comes first in the recipe run.

So we are including the gems recipe and then the applications recipe. After that we render a template for some config file. You can see that the template resource is easy to define, you name it with the target location, and you tell it what permissions, what the source template is named and any variables to pass in to the template.

Now on to the gems recipe:

node[:gems].each do |gem|
  gem_package gem[:name] do
    if gem[:version] && !gem[:version].empty?
      version gem[:version]
    end
    if gem[:source]
      source gem[:source]
    end
    action :install
  end
end

What we’re doing here is looping over all of the gems in our dna.json hash and creating a gem_package resource for each gem. Gems can have versions and sources, but we take care to not set these if they are empty. If you don’t specify a version it will install the latest available and if you don;t specify a source it will default to rubyforge.org. finally we say action :install, which will run the install action for each rubygem. This will not install the gems again if they are already installed on your system.

Now let’s look at the applications recipe:

directory "/data" do
  owner node[:user]
  mode 0755
end

node[:apps].each do |app|

  cap_directories = [
    "/data/#{app}/shared",
    "/data/#{app}/shared/config",
    "/data/#{app}/shared/system",
    "/data/#{app}/releases" 
  ]

  cap_directories.each do |dir|
    directory dir do
      owner node[:user]
      mode 0755
      recursive true
    end
  end

end

First we create the /data directory and give it the permissions we want and owner we specified in our json. Then we loop over all the applications from our json and create the proper set of capistrano deploy directory resources for each app.

So ends our first installment of chef-101, this should be a good starting point for you to take on more advanced recipes.

Pro tip: once you have a nice set of recipes, you can create a tarball of them, put them on an http server somewhere and then use a quick capistrano script to fire the off on remote machines.

Say we create a chef-101.tgz recipe set and we put it up on the web at http://foo.com/chef-101.tgz. We can then log into a server with chef-solo installed vias ssh or via a cap script and run this command:

$ sudo chef-solo -l debug -j /etc/chef/dna.json -r http://foo.com/chef-101.tgz

That will download the recipe set, untar it and run chef on it with the json you passed in. You will first need to create /etc/chef directory and put a solo.rb as well as your dna.json in there.

/etc/chef/solo.rb should look like this:

cookbook_path     "/etc/chef/recipes/cookbooks" 
log_level         :info
file_store_path  "/etc/chef/recipes/" 
file_cache_path  "/etc/chef/recipes/"

Hope you liked cooking with chef-101. See you all next time.

Chef: Suck on my chocolate salty balls

ezmobius — Thu, 15 Jan 2009 21:35:00 +0000

Yesterday we announced http://engineyard.com/solo which is our new engine yard on AWS platform. If you haven’t yet watched the screencast then you should check it out:

Engine Yard Solo Screencast

I’m using a really sweet configuration management engine called Chef that is being open sourced today. My good friends at OpsCode are responsible for the project.

If you have ever used or thought about using a system like Puppet then you owe it to your self to give Chef a try. Chef is a state based, declarative configuration management engine. You define recipes of how you want your system to look and then chef makes it so.

The big advantage chef has over puppet is that the codebase is 1/10th the size and it is pure ruby, including the recipe DSL. Puppet’s biggest flaw is its configuration language that is not quite ruby and not quite turing complete. So you end up wrestling with it to get it to do anything.

Chef is just pure ruby so anything you can do in ruby you can do in your recipes. It also uses a simple merb-core/couchdb server for centralized storage of node configs and each server you manage gets an openid for identity.

If you have a hand in building or configuring any servers, you need to learn chef now.

You need two repos to get chef running, chef and ohai:

chef ohai

Ohai is a gem that interrogates your system and gathers facts about it. Chef is the gem that runs your recipes and actually does the work of configuring the system.

Here are the docs to get started with:

Chef Wiki

I believe that chef will play a very important role in the future of building and managing servers in the ruby community as well as anyone else who needs repeatable server build outs.

We will expose a way to have your own custom chef recipes applied via our management app so that anything we haven’t already written recipes for you can write yourself and get applied as part of a server build.

Put em in your mouth and suck em.

Merb is Rails

ezmobius — Tue, 23 Dec 2008 18:21:00 +0000

Wow this has been a hectic, emotional week. The Rails and Merb core teams have been silently working together towards this monumental announcement. The announcement is that Merb is Rails and Rails is Merb.

Merb and Rails are joining forces and merging the two code bases for Rails-3.0. We’re targeting RailsConf for the first cut of this magic.

When I first started merb way back in the day it was just a fun little project to hack on. But it became more serious as I explored the options. I always considered merb-core a clean room rewrite of ActionPack and I think this tended to upset some people who were very attached to rails.

Fast forward to current day and Merb-1.0 has been out for a while and there is a whole brouhaha of Rails VS Merb memes on the twitters and blogs. This has to stop as it is tearing apart the community and is very non productive.

So our two teams started talking to see if we could put our differences aside and come together for the common good. We’ve laid out a roadmap of what and how to integrate merb’s best features into rails-3.0. I think we have a great plan that will make Rails the best framework in existence. It will be a blend of the best things we have discovered while working on merb, while keeping the Rails aesthetic people have grown to love.

You can expect to get a kick ass, fast, memory efficient version of rails(merb) this spring!

Merb folks will not be left out in the cold. We will continue to support bug and security fixes for the merb 1.0.x line. And we will provide a clear upgrade path to Rails 3.0 for merb apps. We still have quite a few merb apps running internally at ey and will want an upgrade path for our own apps as well.

This is going to be a lot of work for everyone involved but in the end I think this may just be the most inspiring open source story in history.

I’m so impressed with everyone involved. All the core team members from both sides have been able to put away the weapons and egos and really come together, committed to making this transition work seamlessly.

2009 is going to be a seriously strong year for the ruby community. With everyone working together, we have such a kick ass team of smart people that this whole framework thing is going to be a solved issue.

Merb is dead, long live Merb(Rails-3.0)!

http://rubyonrails.org/merb
http://weblog.rubyonrails.org/2008/12/23/merb-gets-merged-into-rails-3
http://yehudakatz.com/2008/12/23/rails-and-merb-merge/

Merb master/worker monit control setup

ezmobius — Sun, 07 Dec 2008 19:55:00 +0000

Going into 1.0, Merb has changed the way it handles options and processes. This tends to break people’s existing monit configs. So today I’m releasing a custom Engine Yard monit script for controlling merb master/worker processes.

You can get the script from here: monit_merb_mpc. Put this somewhere in your $PATH, or make it live here: /engineyard/bin/monit_merb_mpc and make sure it is executable.

In order to make proper use of this script it assumes certain paths are setup. In this example we will assume you have an app called ‘iggy’ that lives at /data/iggy/current, where /data/iggy is the main app dir that contains the capistrano releases and current directory. The script assumes these paths so if you want top use different paths you will need to hand edit the script as desired.

This script also assumes a logging directory of /var/log/engineyard/$appname, you can edit the script if you want to change this as well.

You need to have one master configured per app, as well as monit configuration for each worker.

Master Config

check process merb_iggy_master
  with pidfile /var/log/engineyard/iggy/iggy-production-merb.main.pid
  start program = "/engineyard/bin/monit_merb_mpc iggy start_master -c3 -n5000" 
  stop program = "/engineyard/bin/monit_merb_mpc iggy stop_master" 
  #if totalmem is greater than 80.0 MB for 2 cycles then restart       # eating up memory?
  group merb_iggy

Note that the start program needs to be adjusted if you need more/less instances or want to run the workers on different ports /engineyard/bin/monit_merb_mpc iggy start_master -c -n . Also, memory monitoring is turned off for this process because the memory includes all processes under its control, including the workers and they are already monitored.

Worker Config (one for each port)

check process merb_iggy_5000
  with pidfile /var/log/engineyard/iggy/iggy-production-merb.5000.pid
  start program = "/engineyard/bin/monit_merb_mpc iggy register_worker 5000" 
  stop program = "/engineyard/bin/monit_merb_mpc iggy restart_worker 5000" 
  if totalmem is greater than 80.0 MB for 2 cycles then restart       # eating up memory?
  group merb_iggy

How it works

Since Merb 1.0 (actually version 0.9.8 and above) uses a master process and a spawner process in addition to the worker processes, the monit_merb_mpc (multi-process-control) script needs to control the master process separately from the worker processes. To accomplish this, there is a fair amount of trickery to make monit treat the merb processes in a way monit was never intended to be able to handle. When the master process is started, it will create all the worker processes, so when monit starts the worker processes, they are infact already started – so the start command for the worker simply “registers” the worker – in other words, it just ensures that the pid files contains the worker pid. Also since the master process will automatically restart the worker processes, the stop action for the workers is actually a restart.

Notes:

Although this looks like it should use the monit dependency directive, the scripts are actually written to not need it, and making the worker processes dependent on the master process, it will cause problems during shutdown. since the master specified the number of worker processes using the -c option, you can’t simply shutdown a single worker process by stopping it with monit – it will simply restart by itself and operate unmonitored, thus making it a possible runaway process.

Thanks to all the EY support guys who worked on this one to make it smooth. Hopefully this clears up how to deploy merb with the master/worker server cluster setup under monit.

Ruby Deployment and Engine Yard as a Service

ezmobius — Tue, 18 Nov 2008 20:54:00 +0000

I gave a talk today at the #prorubyconf on Ruby Deployment, past present and future. You can view my slides below.

We will start doing some private beta testing of our new EY platform in December so let me know if you want to beta test and havce some interesting apps to run in the cloud.

Ruby Deployment

View SlideShare presentation or Upload your own. (tags: cloud engineyard)

Merb 1.0

ezmobius — Sat, 08 Nov 2008 06:08:00 +0000

Umm, w00t! It’s been a little over two years since merb was a twinkle in my eye, and a pastie. Since then it has undergone many drastic transformations, working its way towards a very solid, fast foundation for people to build their homesteads on.

It’s very exciting for me to see it grow into what it wants to be, based on the awesome merb community that has popped up around the project. It just shows that if you put in the work on something you love, polishing every little corner it will attract other like minded individuals and they will help you take the project to levels higher then your wildest dreams.

So, gem install merb and get the 1.0 final now, it is propogating its way through the rubyforge tubes. If you can’t get it from the ‘forge yet then you can get it from our gem server:

gem install merb—source http://edge.merbivore.com

Thank you to everyone who helped make this happen with code, feedback and blood, sweat and tears. It’s been totally worth the effort IMHO.

The merb is dead, long live the merb.

MerbCamp Keynote and Introducing Nanite

ezmobius — Sat, 11 Oct 2008 18:41:00 +0000

Just a quick post to share some news. I just finished giving my keynote at MerbCamp about past and present merb and the core tenets of merb development.

I also introduced Nanite: self assembling cluster of ruby processes

You can watch the full video of my talk here: Keynote Video

WMV format ^^ Thanks to Matt Jaynes for mirroring the video on a higher bandwidth connection

And you can see my slides on slideshare below.

Merb Nanite

View SlideShare presentation or Upload your own. (tags: merb nanite)

Thanks to all the merbcamp organizers, the venue is awesome and I have never spoken at a ruby conference that streamed all of the talks live. Very cool setup here.

I will follow up with some more posts on nanite and the what why and how of it shortly. But you can get a pretty good overview of it by watching my talk.

EY Looking for Mad Scientists

ezmobius — Tue, 19 Aug 2008 22:51:00 +0000

We’ve got a few job openings I wanted to float out there for your consideration. Here are the listings:

Ruby Developer

Systems Engineer in Sacremento

Systems Engineer

We need another team member for the Vertebra project. You know ruby like the back of your hand and are interested in distributed systems as well as learning erlang.

Systems engineers work on our cluster systems and will help with our custom Gentoo linux build as well as be involved with the Load Balancers, Switching Fabric, Linux, Xen, Coraid and helping design the latest and greatest platform for running Ruby applications in the cloud.

If you think you have the chops for any of these jobs then we want to hear from you. Please send a resume and a short email describing who you are and why you kick ass to jarnold@engineyard.com and cc ezra@engineyard.com