Posted by ezmobius
Mon, 07 Nov 2011 08:59:00 GMT
Just a quick follow up post to my last one about my makerbot and how I have fallen in love with 3d printing.
If you just want to skip to the photo gallery of the build log then here you go all you TLDR’rs, for the rest of you read on…
Posted by ezmobius
Sun, 23 Oct 2011 08:10:00 GMT
Self Recursive 3d Printing and Scratching Your Own Itches.
This past summer I built a Makerbot Thingomatic 3d printer kit. It took about 5 weeks to get the kit after I ordered it and it took me a 22 hour straight session to assemble and get it printing reasonably well. Since then I have been fine tuning it to get better prints and re-learning 3d modeling and CAD software I used to use when I wanted to go into computer animation in the mid 90's ;)
If you just want to skip to the photo gallery here it is
or you can read on to get the full story.
Posted by ezmobius
Thu, 02 Jun 2011 02:32:00 GMT
Funny little thing hit me a few minutes ago. I logged into the vps that runs this blog(this is a very rare occurrence indeed;) and I noticed how ancient this setup is, I mean this is a rails setup from before I wrote my book on deployment and it still hums along.
Here’s the fun stuff that made me write this little post:
Posted by ezmobius
Sat, 16 Apr 2011 01:09:00 GMT
Wow, so it looks like I haven’t posted on my blog since last August when I decided to move on from the company I helped found (Engine Yard), it’s awesome to see how great they have done since I left. Makes me sure that I made the right decision and left the company in very capable hands. But now it is time to talk about what I’ve been working on since then ;) I’ve been lucky enough to join the team at VMware that built/is building the Cloud Foundry project. Cloud Foundry is our implementation of the open platform as a service (PaaS) idea that we’ve been talking about for a while now. It has been interesting for me because I am used to tweeting and blogging all the time about what I’m working on day to day, but I’ve had to keep quiet about this project until now. I’m stoked that I can finally talk about what we’ve been working on in the Cloud Foundry engineering team.
Now keep in mind that this is my personal blog so anything I say here should be considered my personal opinion and not that of my employers, so blame me not VMware if I say something out of line here … ;)
This team I had the honor of joining at VMware is full of rock stars. I was ready for a change where I had lots of folks way smarter then I am to look up to and work with. This team is by far the highest caliber team I’ve worked on since the Engine Yard engineering team, which is, also chock full of awesome. But I feel that this whole Cloud Foundry project is truly a game changer and since it is open source it naturally falls on my shoulders to try and help build the OSS community around the project. So my new role as Ruby/Rails/Cloud Foundry Evangelist here at VMware will allow me to help steward the OSS project and try to help build a great community around Cloud Foundry.
I’ve been lucky enough to be on the team building this platform within VMware and I’m very happy to finally be able to talk about it in the open. I’m even more excited that VMware has decided to release Cloud Foundry under the Apache2 license as a fully open OSS project! That means I can talk tech with you about the implementation and you can actually go to github and download the source yourself and follow along.
There is a ton of stuff to talk about, here are a few tidbits that can get you started.
Cloud Foundry is on github here:
The VCAP repo is the meaty part or what we call the “kernel” of Cloud Foundry
NATS (Internal Messaging) here:
Cloud Controller here:
Droplet Execution Agent here:
Health Manager here:
For the deep dive and details that I’ve written go here to the Cloud Foundry Blog.
Look forward to much more content around the technical aspects of the Cloud Foundry Kernel in the coming weeks and months.
Posted by ezmobius
Fri, 13 Aug 2010 08:27:00 GMT
So by now most folks who know me know that I have resigned from the startup I co-founded, namely Engine Yard Inc. It’s a long strange story that has lead up to this point and a lot of it I cannot speak of but I am going to give a short history of a 4 year startup that took $38million in VC money over 3 rounds and my view from the cockpit and trenches both.
Back in February of 2006 Ruby on Rails was still in it’s infancy and things were still changing very fast with regards to deployment especially. It was obvious that Rails was going to be a smashing success and I had actually launched one of the first commercial Rails applications in late 2004 when I worked at the Yakima Herald-Republic newspaper in Yakima WA. They had a website already written in PHP that got about 250k uniques/day, so not a small amount of traffic but it was no twitter or anything like that. Still back then almost no one knew how to efficiently deploy Rails apps and scale them without them falling over. And the landscape was going through major changes, CGI, webrick, Fast-CGI, mod_ruby, mongrel etc. With a plethora of front end static file web servers to put these things behind.
I was the only technical person at the whole newspaper and I was tasked with rewriting their entire website, intranet and classified and obituary entry system. THey had a small data center in the newspaper building and they bought me two Apple X-Serves and said “here you go, your Rails application must take the 250k uniques on day one without faltering or we will roll back to the php app and you will be in hot water buddy”
So I had to become the deployment expert in the Rails community, if you search the ruby on rails mailing lists from 2004-2007 you will find literally thousands of my posts helping folks figure out how to deploy and keep their apps running. I was approached by Dave Thomas of the Pragmatic Programmers and asked if I’d like to write a book on Rails Deployment. “Hell yeah I’d like to write that book I said”. Little did I know how much work that was going to be with all the changes Rails deployment was about to go through. But it gave me a name in the community as the “Rails Deployment Expert”. And I did finally finish the book “Deploying Rails Applications” 2 years later with the help of two other authors. But I rewrote most of the book a few times as things changed so often.
Anyway back to February of 2006, Tom Mornini and Lance Walley called me up out of the blue(I did not know either of these guys before this call) and told me about an idea they had to build a “Rackspace for Rails”. They asked if I was interested and said they found me through my thousands of posts on the rails mailing list and my Pragmatic beta book and figured I was the obvious expert to help them build their vision. So I agreed to become the 3rd founder with them and we started the planning stages of what was to become Engine Yard Inc.
The first Railsconf was in June of 2006 and we had a teaser website up and I was giving a talk on Rails Deployment at the conference. I announced the then vaporware Engine Yard as my “One more thing” at the end of my speech and linked to our site where we took emails.
We rounded up about $120k by begging, borrowing and stealing(not really) from friends and family. I must admit that Tom and Lance raised most of this first money here while I was busy writing the first code for Engine Yard. We raised this seed money in August of 2006 and that was when I quit my full time job and took the plunge. We all flew down to Sacramento where we had rented our first rack at Herakles data and bought our first 6 Super-micro servers and first Co-Raid JBOD disk array.
I still remember building the servers and racking them while at the same time fast talking folks on the phone trying to sell early accounts on our not yet running first cluster.
Now you need to know that this was way back before everything was “Cloud” Amazon had not yet come out with EC2 or any hints of it. But we knew what we needed to make our vision happen so we ended up building a cloud before the word cloud meant what it means today. We went with Xen and commodity pizza boxes and hand rolled automation written in ruby and python.
At this point Tom and I had realized we were out of our league with the low level linux, Xen, networking and basic ponytail type of stuff. This is when we took on our 4th founder Jayson Vantyl. He was the guy answering all our Xen and coraid questions on the mailing lists just like I had been the guy answering the rails deployment stuff. So we first hired him as a contractor and flew him out to help build our first cluster. We quickly realized we needed this guy as the 4th founder and made him an offer and he joined the company as the 4th and final founder in September I believe of 2006.
So we were able to cobble together our first cluster and get it working well enough to take some trusting early customers. I personally hand deployed the first 80 or so customers. I mean I literally took their app code and worked with them to learn about their app, and personally wrote the caspistrano deployment scripts for each of these customers myself. I was the only support staff on call 24/7 for the first year as far as application support went and Jayson was the only cluster support guy for the first 6 months or so as well.
I quickly built up a collection of best practices and a set of automation tools that let me do this easier and easier for each new customer. Slowly building out our Engine Yard automation toolkit.
I was also responsible for putting together the “Rails Stack” we used. Jayson chose Gentoo linux because that was what he was most familiar with and it was the distro we could hand optimize to get the best speed and flexibility out of. But I built the entire stack that ran after a blank VM was online with networking, storage and all that jazz.
I chose Nginx, Mongrel, Mysql, Memcached, Monit, Sphinx and a handful of other open source items that we made custom builds of highly tuned for ruby and running Rails apps as fast as possible. Many of you will remember me sharing my nginx and other configs on this blog and I am sure that my nginx.con for running rails apps with mongrel clusters is probably one of the most widely used nginx.conf files in existence ;)
Fast forward a year or so and we hired a few folks to help me. We were having massive success. Hardly any other hosting companies knew how to host, scale and keep rails apps online. We specialized in rails apps only and we also did the full white glove service. Anything you needed you could file a ticket for and we would bend over backwards to help you get it done.
We helped many folks get through slashdottings and digg and techcrunch ‘attacks’ by temporarily giving them more ram or cpu or bigger databases and then scaling them back down after the onslaught was over. We were one of the first hosting companies to be truly ‘cloud like’ IMHO. EC@ came out in 2007 and looking back if it had come out before we built our own cloud I would have chosen to use them as a substrate instead of building our own cloud. But also looking back I think rolling our own gave us a unique perspective on the whole thing as we have seen the cloud stack all the way from the blank cages to the PaaS we currently run on top of Amazon.
But in late 2008 I was between projects and we had taken our series B VC round which included Benchmark, NEA and Amazon as investors. So I decided to go off into a cave for a month or two and see if I could decouple our ‘stack’ from our hardware and make it run on top of EC2. It took me about 6 weeks to have a working prototype of what is now called AppCloud. I’d like to shout out here to Adam Holoway from Opscode here for training me on how to use Chef. I hired him for 3 days to come down and help me convert my Engine Yard automation system that was mostly based on capistrano to chef and a central gui to manage it. He helped me jump start the whole thing and Engine Yard owes him and Opscode a debt of gratitude. We were the first company to use chef commercially, heck we even used chef in production for pay before opscode did! So big ups Adam.
This was all done in october and november of 2008. Then I showed it to the rest of the company and we started to build a team around me to help productize the prototype spike and thus was born AppCloud.
We launched it in late January 2009 as Engine Yard Solo as it started out only managing single machines at a time with the app servers, database servers and cache servers all running on one box. The only way to scale was to move to a bigger EC2 instance. But we launched and pretty fast too!
We started taking customers rapidly and taking customer feedback to rapidly iterate the system. By the end of the summer of 2009 we launched Engine Yard Flex(horrible names here, bear with me) which was the same system but supported full clusters on EC2 with load balanced self healing application tiers, Database tiers with read only slaves and utility servers so anything that didn’t fit into the other two boxes like nosql databases or search engines etc.
Then we took on even more customers and rapidly iterated on this until it was pretty nice and solid. The team has been doing this ever since and now there are way over 1000 customers on AppCloud(we removed the solo and flex names and just called it AppCloud). Many features have been added and the current incarnation of AppCloud is the vision I had in my head when we first started Engine Yard way back in 2006. We just were victims of our own success and could not deploy customers fast enough to get free time to build the automated system that AppCloud is today.
And we started moving people out of our own data center and off our own cloud onto AppCloud as it got more and more featureful. As far as I see it there is zero value in racking and stacking servers anymore. The money is going to be in management platforms for other peoples IaaS clouds like EC2 and others that are popping up all over the place.
But I believe the true future of cloud computing for developers is to not think about servers at all. It is now time to focus on the Application and new levels of abstraction that allow folks to use the computing resources in easier and easier ways.
So here it is the summer of 2010, I’ve been working my ass off at Engine Yard for 4 years now pouring every ounce of blood sweat and tears that I could muster to make it the best it could be.
February of this year my wife Regan and I had a beautiful baby boy named Ryland and that changed the game entirely. Now is the time for me to focus on spending as much time with my son and family as I can and I can no longer commit to 100 hour weeks at Engine Yard. I also wanted to move to Portland where my folks live so my son can grow up near his grandparents and my wife and I can have trustworthy babysitters so we can have a social live of our own(even if it’s just a little bit;)
After 4 years at Engine Yard I was so close to being fully vested on my stock that I decided to resign and leave the company in the competent hands and team that has been built up there over the years. I wish all of you the very best and will be watching closely to make sure you make me a ton of money someday when you sell the company like you promised me you would ;)
So here we are at the beginning of a new era. What comes next for me? Something truly awesome and even more challenging then Engine Yard, but that will allow me to work from home in Portland. I cannot yet reveal where I am going to work next but I can promise it will be another game changing project and I am chomping at the bit to get started.
I will make another post when I am allowed to reveal my new job and what I will be working on, but for now I just wanted to write down a short history of “Engine Yard, the Good Times”. Don’t get me wrong it has definitely not all been roses and taking lots of VC money is a hard thing to do and keep the balance of your vision of a company. But I believe I did the best I could for Engine Yard, I gave them many great inventions and lots of good code and I practically hired most of the devs, ops and support folks there just by knowing them from online or at conferences or by working with them on open source projects or talking with them on irc. That is the new way to build companies, College degrees don’t matter much IMHO anymore, for developers anyway. It’s more how you interact with the open source community and what you release yourself. Your github account has become your new resume and what you say on Twitter and in various IRC channels are more likely to get you the best jobs then any recruiters ever will.
I just want to say a fond farewell to Engine Yard and all my people there. I wish you all the best of luck and I know you will be a success with all the smart folks that work there.
It was just time for a new challenge for me, time to move on and time to focus a bit more on family then on startup life 24/7.
Farewell fond memories and friends at the Yard, I will miss each and everyone of you. Oh the stories I could tell if only I could. But I feel that telling this positive story of the history of EY is the classy way to go out and I wish Engine Yard all the best in the world.
Posted by ezmobius
Wed, 11 Aug 2010 02:09:00 GMT
Holy shit bad luck. I just somehow accidentally deleted my @ezmobius twitter account!
I’ve created a new temp account at @ezmob but I hope someone at twitter can help me get my @ezmobius account back with all my followers.
I’ve reached out to a contact at twitter to see if I can get some help but please retweet this far and wide to help me get my @ezmobius account back if at all possible.
Posted by ezmobius
Thu, 30 Apr 2009 18:56:00 GMT
Here are the slides from my keynote this morning at the SF Erlang Factory conference:
Posted by ezmobius
Tue, 07 Apr 2009 20:19:00 GMT
I’m looking for a few good folks to join my team. You need to be able to work on site in San Francisco, we can assist with moving expenses for the right candidate.
You would work with my team(currently 5 devs). We do no bs agile style development with pairing where appropriate and test infected mentality . generally we have a fun, tight team and get to work on some of the hardest and most rewarding areas of software development relevant today.
You would be on my team working on http://engineyard.com/solo and it’s successors as well as Engine Yard’s on premises infrastructure.
You would be working on/with the following ruby technologies:
You would be working on the following problem domains:
Scalable Ruby Deployment Architectures
Scalable Cloud Computing Architectures
Scalable Database Architecture
Key Value Data Stores
Cloud provider API's
Monitoring and alerting systems
Security in the Cloud
Horizontally Scalable Architectures
Many other interesting areas of computing
Familiarity with the following tech is nice but not necessarily required if you can learn fast:
If you think you have the “right stuff” and you have the kick ass take names attitude then send a resume with a short intro about who you are and why you are the person we should hire.
Posted by ezmobius
Sat, 31 Jan 2009 19:00:00 GMT
One of the things I really like about chef is that it scales down as well as up. You can get started using it right away with chef-solo and then graduate to chef-client and chef-server when you need more cowbell. And then as you grow you can horizontally partition each component of the chef infrastructure, web servers, couchdb servers with messaging queues to decouple each part so you can go from tiny to large and in charge with the same recipes.
So let’s get started with a simple set of recipes to install some rubygems, and setup some directories for deploying a few apps with capistrano after chef configures the base system.
First you will need to clone my chef-101 repo from github:
$ git clone git://github.com/ezmobius/chef-101.git
First we need to get chef installed.
$ sudo gem install chef ohai --source http://gems.opscode.com --source http://gems.rubyforge.org
(the two sources are needed so that dependencies can be installed form rubyforge even though chef is hosted on gems.opscode.com. Rubygems quirk but at least it works this way ;)
Now lets just run the recipe set so you can see chef in action! (this will install a few rubygems on your system so you can skip this step if you want. It will also create a few directories underneath /data.)
You will need to change the user attribute in the config/dna.json file to a user that exists on your system before this will run(unless you have an ‘ez’ user like I do ;)
$ cd chef-101
$ sudo chef-solo -l debug -c config/solo.rb -j config/dna.json
You will see a bunch of output scroll by, first you will see the ohai output, ohai is the systems information collector. The you will see a bunch of info about compiling recipes and applying recipes. It will end with this line if it was successful:
INFO: Chef Run complete in 1.093957 seconds
Congratulations! You are now cooking with fire. Now let’s take a look at the recipes and how they are put together so you know how this thing works.
First off what is this dna.json file we passed in to chef-solo?
This is what I am calling the JSON DNA of your recipe set. These are the parameters that will be available in your recipes as the node object. So you can see we have some applications, a username and some rubygems to install.
You can see that we have set the “recipes” array to contain ‘main’. Let’s take a look at the main recipe, as this is our entry point into the system.
template "/data/someservice.conf" do
:applications => node[:apps]
You can see that we use include_recipe, this command will pull in the named recipe and run it in place top down right there in the recipe. This allows for chef recipes to run in deterministic order every time, if you want one resource to run before another, just make sure it comes first in the recipe run.
So we are including the gems recipe and then the applications recipe. After that we render a template for some config file. You can see that the template resource is easy to define, you name it with the target location, and you tell it what permissions, what the source template is named and any variables to pass in to the template.
Now on to the gems recipe:
node[:gems].each do |gem|
gem_package gem[:name] do
if gem[:version] && !gem[:version].empty?
What we’re doing here is looping over all of the gems in our dna.json hash and creating a gem_package resource for each gem. Gems can have versions and sources, but we take care to not set these if they are empty. If you don’t specify a version it will install the latest available and if you don;t specify a source it will default to rubyforge.org. finally we say action :install, which will run the install action for each rubygem. This will not install the gems again if they are already installed on your system.
Now let’s look at the applications recipe:
directory "/data" do
node[:apps].each do |app|
cap_directories = [
cap_directories.each do |dir|
directory dir do
First we create the /data directory and give it the permissions we want and owner we specified in our json. Then we loop over all the applications from our json and create the proper set of capistrano deploy directory resources for each app.
So ends our first installment of chef-101, this should be a good starting point for you to take on more advanced recipes.
Pro tip: once you have a nice set of recipes, you can create a tarball of them, put them on an http server somewhere and then use a quick capistrano script to fire the off on remote machines.
Say we create a chef-101.tgz recipe set and we put it up on the web at http://foo.com/chef-101.tgz. We can then log into a server with chef-solo installed vias ssh or via a cap script and run this command:
$ sudo chef-solo -l debug -j /etc/chef/dna.json -r http://foo.com/chef-101.tgz
That will download the recipe set, untar it and run chef on it with the json you passed in. You will first need to create /etc/chef directory and put a solo.rb as well as your dna.json in there.
/etc/chef/solo.rb should look like this:
Hope you liked cooking with chef-101. See you all next time.
Posted by ezmobius
Thu, 15 Jan 2009 21:35:00 GMT
Yesterday we announced http://engineyard.com/solo which is our new engine yard on AWS platform. If you haven’t yet watched the screencast then you should check it out:
Engine Yard Solo Screencast
I’m using a really sweet configuration management engine called Chef that is being open sourced today. My good friends at OpsCode are responsible for the project.
If you have ever used or thought about using a system like Puppet then you owe it to your self to give Chef a try. Chef is a state based, declarative configuration management engine. You define recipes of how you want your system to look and then chef makes it so.
The big advantage chef has over puppet is that the codebase is 1/10th the size and it is pure ruby, including the recipe DSL. Puppet’s biggest flaw is its configuration language that is not quite ruby and not quite turing complete. So you end up wrestling with it to get it to do anything.
Chef is just pure ruby so anything you can do in ruby you can do in your recipes. It also uses a simple merb-core/couchdb server for centralized storage of node configs and each server you manage gets an openid for identity.
If you have a hand in building or configuring any servers, you need to learn chef now.
You need two repos to get chef running, chef and ohai:
Ohai is a gem that interrogates your system and gathers facts about it. Chef is the gem that runs your recipes and actually does the work of configuring the system.
Here are the docs to get started with:
I believe that chef will play a very important role in the future of building and managing servers in the ruby community as well as anyone else who needs repeatable server build outs.
We will expose a way to have your own custom chef recipes applied via our management app so that anything we haven’t already written recipes for you can write yourself and get applied as part of a server build.
Put em in your mouth and suck em.