Cooking with Chef 101

Posted by ezmobius Sat, 31 Jan 2009 19:00:00 GMT

One of the things I really like about chef is that it scales down as well as up. You can get started using it right away with chef-solo and then graduate to chef-client and chef-server when you need more cowbell. And then as you grow you can horizontally partition each component of the chef infrastructure, web servers, couchdb servers with messaging queues to decouple each part so you can go from tiny to large and in charge with the same recipes.

So let’s get started with a simple set of recipes to install some rubygems, and setup some directories for deploying a few apps with capistrano after chef configures the base system.

First you will need to clone my chef-101 repo from github:

http://github.com/ezmobius/chef-101

$ git clone git://github.com/ezmobius/chef-101.git

First we need to get chef installed.

$ sudo gem install chef ohai --source http://gems.opscode.com --source http://gems.rubyforge.org

(the two sources are needed so that dependencies can be installed form rubyforge even though chef is hosted on gems.opscode.com. Rubygems quirk but at least it works this way ;)

Now lets just run the recipe set so you can see chef in action! (this will install a few rubygems on your system so you can skip this step if you want. It will also create a few directories underneath /data.)

You will need to change the user attribute in the config/dna.json file to a user that exists on your system before this will run(unless you have an ‘ez’ user like I do ;)

$ cd chef-101
$ sudo chef-solo -l debug -c config/solo.rb -j config/dna.json

You will see a bunch of output scroll by, first you will see the ohai output, ohai is the systems information collector. The you will see a bunch of info about compiling recipes and applying recipes. It will end with this line if it was successful:

INFO: Chef Run complete in 1.093957 seconds

Congratulations! You are now cooking with fire. Now let’s take a look at the recipes and how they are put together so you know how this thing works.

First off what is this dna.json file we passed in to chef-solo?

{
  "apps": [
    "beast",
    "mephisto" 
  ],
  "user": "ez",
  "gems": [
    {
      "name": "rake",
      "version": "0.8.3" 
    },
    {
      "name": "tmm1-amqp",
      "source": "http://gems.github.com",
      "version": "0.6.0" 
    }
  ],
  "recipes": ["main"]
}

This is what I am calling the JSON DNA of your recipe set. These are the parameters that will be available in your recipes as the node object. So you can see we have some applications, a username and some rubygems to install.

You can see that we have set the “recipes” array to contain ‘main’. Let’s take a look at the main recipe, as this is our entry point into the system.

include_recipe 'gems'
include_recipe 'applications'

template "/data/someservice.conf" do
  owner node[:user]
  mode 0644
  source "someservice.conf.erb" 
  variables({
    :applications => node[:apps]
  })
end

You can see that we use include_recipe, this command will pull in the named recipe and run it in place top down right there in the recipe. This allows for chef recipes to run in deterministic order every time, if you want one resource to run before another, just make sure it comes first in the recipe run.

So we are including the gems recipe and then the applications recipe. After that we render a template for some config file. You can see that the template resource is easy to define, you name it with the target location, and you tell it what permissions, what the source template is named and any variables to pass in to the template.

Now on to the gems recipe:

node[:gems].each do |gem|
  gem_package gem[:name] do
    if gem[:version] && !gem[:version].empty?
      version gem[:version]
    end
    if gem[:source]
      source gem[:source]
    end
    action :install
  end
end

What we’re doing here is looping over all of the gems in our dna.json hash and creating a gem_package resource for each gem. Gems can have versions and sources, but we take care to not set these if they are empty. If you don’t specify a version it will install the latest available and if you don;t specify a source it will default to rubyforge.org. finally we say action :install, which will run the install action for each rubygem. This will not install the gems again if they are already installed on your system.

Now let’s look at the applications recipe:

directory "/data" do
  owner node[:user]
  mode 0755
end

node[:apps].each do |app|

  cap_directories = [
    "/data/#{app}/shared",
    "/data/#{app}/shared/config",
    "/data/#{app}/shared/system",
    "/data/#{app}/releases" 
  ]

  cap_directories.each do |dir|
    directory dir do
      owner node[:user]
      mode 0755
      recursive true
    end
  end

end

First we create the /data directory and give it the permissions we want and owner we specified in our json. Then we loop over all the applications from our json and create the proper set of capistrano deploy directory resources for each app.

So ends our first installment of chef-101, this should be a good starting point for you to take on more advanced recipes.

Pro tip: once you have a nice set of recipes, you can create a tarball of them, put them on an http server somewhere and then use a quick capistrano script to fire the off on remote machines.

Say we create a chef-101.tgz recipe set and we put it up on the web at http://foo.com/chef-101.tgz. We can then log into a server with chef-solo installed vias ssh or via a cap script and run this command:

$ sudo chef-solo -l debug -j /etc/chef/dna.json -r http://foo.com/chef-101.tgz

That will download the recipe set, untar it and run chef on it with the json you passed in. You will first need to create /etc/chef directory and put a solo.rb as well as your dna.json in there.

/etc/chef/solo.rb should look like this:

cookbook_path     "/etc/chef/recipes/cookbooks" 
log_level         :info
file_store_path  "/etc/chef/recipes/" 
file_cache_path  "/etc/chef/recipes/" 

Hope you liked cooking with chef-101. See you all next time.

28 comments

Comments

  1. Mihai said about 1 hour later:
    Can you give more details about "We can then log into a server with chef-solo installed via ssh or via a cap script and run this command" ? I still need to install ruby, gems and chef on the remote machine (I'm thinking EC2 ubuntu AMIs). Do you propose making an AMI with chef installed then customize it using chef ? From what I seen sprinkle runs the commands remotely via capistrano/ssh. Also, how much resources/memory does chef-server uses ? Is it worth using chef-server for a couple of EC2 instances dynamically scaled or just use chef-solo? Thank you, EY and Opscode :)
  2. Ezra said about 1 hour later:
    @Mihai- Yeah the best way to go on ec2 is to have a base AMI with chef installed and then send the json as user-data when you boot the instance and have chef-client or chef-solo run on first boot and do it's thing. I'd say get yourself running and writing recipes with chef-solo and then explore the client server architecture and see if you need it for anything. It has some great features like searching across other node's attributes which makes it nice for setting up clustered services. The nice thing is that you can start with chef-solo and graduate to chef-server when you need it.
  3. anon said about 1 hour later:
    i must be stupid but i don't really see the point. Why not just whip together a small bash script with the commands that you would run for deployment? insteed of obscurifying it with some wierd ruby dsl? Sure i love ruby but a plain bash-script feels alot less "darkness" to me...
  4. Ezra said about 1 hour later:
    Plain old bash script is not idempotent. The whole idea of something like chef is that you can run the recipes over and over again on the same server and it will always converge on the proper state that you have declared. We did not go into any advanced resources in this example just because it was more about getting started. But once you get complex recipes it will really save your ass. with plain old shell scripts, what happens when it crashes half way through the run? what state is your system in now? how will you finish the script? Comment out what already succeeded and try to run the rest? If so you're doing it wrong. With chef you just run the recipes again and it will not do anything it has already done and it will not harm a live system to run the recipes over and over again. This is the power of idempotent recipes and it is much harder to get idempotency in shell scripts.
  5. jacques said about 3 hours later:
    #4 comment is very interesting! That actually helps me understand the need for Chef. Yet another thing to learn... but it looks like it could be a gamechanger for system management.
  6. Ezra said about 3 hours later:
    @Mihai chef-server is just a merb app that usually runs with two processes, so we're talking about 40-70Mb total for chef-server depending if your on 32 or 64bit machines. chef-client or chef-solo use about 14Mb ram when running but they do not run as daemons.
  7. Mihai said about 4 hours later:
    Thank you very much for your answers. I'll experiment with chef-solo and EC2.
  8. Don said about 5 hours later:
    Thanks for the simple tutorial. Their documentation looks decent, but having a short step-by-step targeting Ruby developers is perfect.
  9. Adam said 6 days later:
    I had to add the "eventmachine" gem to the dna.json file's "gems" structure before I could get chef-solo to run successfully since "tmm1-amqp" depends on it. Perhaps there's a way to get chef's gem_package to install dependencies?
  10. Dick Davies said 6 days later:
    Thanks for the quickstart, really useful; I tried to grok Puppet for years but never found an easy path from 'here' to 'there' - chef-solo looks like a good first step. Is there any particular reason chef doesn't create the 'ez' user automatically?
  11. Ezra said 6 days later:
    @Dick- you can have chef create the ez user with a user resource, but the user resource is currently busted on OSX and I figured a lot of folks would try it out first on osx. On a real deploy you woudl definitely have chef manage the users too, this was just an abbreviated short start.
  12. Dick Davies said 8 days later:
    Thanks Ezra. It was fairly straightforward to figure out where I'd plug my own Ruby to do that sort of thing; Can't beat frameworks that keep out of your way when you need them to ;)
  13. LR Kosmetik said 11 days later:
    I was always looking for this solution. Thanks for that it's very useful for my problem.
  14. dream promo said 18 days later:
    I am a newbie to Ruby and this is just the kind of tutorial I was looking for, thanks
  15. Mallorca Immobilien said 66 days later:
    Ruby is not that easy. I tried to implementate Ruby in my cms, it works wonderfull!
  16. Mallorca Immobilien said 66 days later:
    Ruby is not that easy. I tried to implementate Ruby in my cms, it works wonderfull!
  17. De Puta Madre said 67 days later:
    To my opinion Ruby is not a good extension for projects based on CMS Systems.
  18. De Puta Madre said 67 days later:
    To my opinion Ruby is not a good extension for projects based on CMS Systems.
  19. Aloe Vera said 81 days later:
    Thanks a lot of this help. It was very easy.
  20. Free SMS said 81 days later:
    It was wonderful. Thank you. With best regards...
  21. Notebook Handy said 84 days later:
    @ puta madre: it was exactly what I wanted to know. So I can implemente ruby on rails in drupal for ex.?
  22. Kosmetik Parfüm said 86 days later:
    Hi, a very interesting site. I love it :-)
  23. Kosmetik Parfüm said 86 days later:
    Hi it is a very interesting site.I love it.
  24. SMS gratis senden said 86 days later:
    Aha, ich verstehe zwar nicht viel, aber dennoch genug um zu sagen "Kochen mit dem Chef"
  25. http://www.mallorca-properties.co.uk said 95 days later:
    a great site i thing one of the best blogs!! Regards from sunny mallorca
  26. Mallorca Makler said 95 days later:
    a great site i thing one of the best blogs!! Regards from sunny mallorca
  27. Muskelaufbau mit Eiweiss said 105 days later:
    Interesting! We have some proteine and eiweißand many more - maybe for cocking?
  28. free sms senden said 105 days later:
    Fine words. Greetings...

(leave url/email »)

   Preview comment