4 years at Engine Yard, what a long strange trip it's been.

Posted by ezmobius Fri, 13 Aug 2010 08:27:00 GMT

So by now most folks who know me know that I have resigned from the startup I co-founded, namely Engine Yard Inc. It’s a long strange story that has lead up to this point and a lot of it I cannot speak of but I am going to give a short history of a 4 year startup that took $38million in VC money over 3 rounds and my view from the cockpit and trenches both.

Back in February of 2006 Ruby on Rails was still in it’s infancy and things were still changing very fast with regards to deployment especially. It was obvious that Rails was going to be a smashing success and I had actually launched one of the first commercial Rails applications in late 2004 when I worked at the Yakima Herald-Republic newspaper in Yakima WA. They had a website already written in PHP that got about 250k uniques/day, so not a small amount of traffic but it was no twitter or anything like that. Still back then almost no one knew how to efficiently deploy Rails apps and scale them without them falling over. And the landscape was going through major changes, CGI, webrick, Fast-CGI, mod_ruby, mongrel etc. With a plethora of front end static file web servers to put these things behind.

I was the only technical person at the whole newspaper and I was tasked with rewriting their entire website, intranet and classified and obituary entry system. THey had a small data center in the newspaper building and they bought me two Apple X-Serves and said “here you go, your Rails application must take the 250k uniques on day one without faltering or we will roll back to the php app and you will be in hot water buddy”

So I had to become the deployment expert in the Rails community, if you search the ruby on rails mailing lists from 2004-2007 you will find literally thousands of my posts helping folks figure out how to deploy and keep their apps running. I was approached by Dave Thomas of the Pragmatic Programmers and asked if I’d like to write a book on Rails Deployment. “Hell yeah I’d like to write that book I said”. Little did I know how much work that was going to be with all the changes Rails deployment was about to go through. But it gave me a name in the community as the “Rails Deployment Expert”. And I did finally finish the book “Deploying Rails Applications” 2 years later with the help of two other authors. But I rewrote most of the book a few times as things changed so often.

Anyway back to February of 2006, Tom Mornini and Lance Walley called me up out of the blue(I did not know either of these guys before this call) and told me about an idea they had to build a “Rackspace for Rails”. They asked if I was interested and said they found me through my thousands of posts on the rails mailing list and my Pragmatic beta book and figured I was the obvious expert to help them build their vision. So I agreed to become the 3rd founder with them and we started the planning stages of what was to become Engine Yard Inc.

The first Railsconf was in June of 2006 and we had a teaser website up and I was giving a talk on Rails Deployment at the conference. I announced the then vaporware Engine Yard as my “One more thing” at the end of my speech and linked to our site where we took emails.

We rounded up about $120k by begging, borrowing and stealing(not really) from friends and family. I must admit that Tom and Lance raised most of this first money here while I was busy writing the first code for Engine Yard. We raised this seed money in August of 2006 and that was when I quit my full time job and took the plunge. We all flew down to Sacramento where we had rented our first rack at Herakles data and bought our first 6 Super-micro servers and first Co-Raid JBOD disk array.

I still remember building the servers and racking them while at the same time fast talking folks on the phone trying to sell early accounts on our not yet running first cluster.

Now you need to know that this was way back before everything was “Cloud” Amazon had not yet come out with EC2 or any hints of it. But we knew what we needed to make our vision happen so we ended up building a cloud before the word cloud meant what it means today. We went with Xen and commodity pizza boxes and hand rolled automation written in ruby and python.

At this point Tom and I had realized we were out of our league with the low level linux, Xen, networking and basic ponytail type of stuff. This is when we took on our 4th founder Jayson Vantyl. He was the guy answering all our Xen and coraid questions on the mailing lists just like I had been the guy answering the rails deployment stuff. So we first hired him as a contractor and flew him out to help build our first cluster. We quickly realized we needed this guy as the 4th founder and made him an offer and he joined the company as the 4th and final founder in September I believe of 2006.

So we were able to cobble together our first cluster and get it working well enough to take some trusting early customers. I personally hand deployed the first 80 or so customers. I mean I literally took their app code and worked with them to learn about their app, and personally wrote the caspistrano deployment scripts for each of these customers myself. I was the only support staff on call 24/7 for the first year as far as application support went and Jayson was the only cluster support guy for the first 6 months or so as well.

I quickly built up a collection of best practices and a set of automation tools that let me do this easier and easier for each new customer. Slowly building out our Engine Yard automation toolkit.

I was also responsible for putting together the “Rails Stack” we used. Jayson chose Gentoo linux because that was what he was most familiar with and it was the distro we could hand optimize to get the best speed and flexibility out of. But I built the entire stack that ran after a blank VM was online with networking, storage and all that jazz.

I chose Nginx, Mongrel, Mysql, Memcached, Monit, Sphinx and a handful of other open source items that we made custom builds of highly tuned for ruby and running Rails apps as fast as possible. Many of you will remember me sharing my nginx and other configs on this blog and I am sure that my nginx.con for running rails apps with mongrel clusters is probably one of the most widely used nginx.conf files in existence ;)

Fast forward a year or so and we hired a few folks to help me. We were having massive success. Hardly any other hosting companies knew how to host, scale and keep rails apps online. We specialized in rails apps only and we also did the full white glove service. Anything you needed you could file a ticket for and we would bend over backwards to help you get it done.

We helped many folks get through slashdottings and digg and techcrunch ‘attacks’ by temporarily giving them more ram or cpu or bigger databases and then scaling them back down after the onslaught was over. We were one of the first hosting companies to be truly ‘cloud like’ IMHO. EC@ came out in 2007 and looking back if it had come out before we built our own cloud I would have chosen to use them as a substrate instead of building our own cloud. But also looking back I think rolling our own gave us a unique perspective on the whole thing as we have seen the cloud stack all the way from the blank cages to the PaaS we currently run on top of Amazon.

But in late 2008 I was between projects and we had taken our series B VC round which included Benchmark, NEA and Amazon as investors. So I decided to go off into a cave for a month or two and see if I could decouple our ‘stack’ from our hardware and make it run on top of EC2. It took me about 6 weeks to have a working prototype of what is now called AppCloud. I’d like to shout out here to Adam Holoway from Opscode here for training me on how to use Chef. I hired him for 3 days to come down and help me convert my Engine Yard automation system that was mostly based on capistrano to chef and a central gui to manage it. He helped me jump start the whole thing and Engine Yard owes him and Opscode a debt of gratitude. We were the first company to use chef commercially, heck we even used chef in production for pay before opscode did! So big ups Adam.

This was all done in october and november of 2008. Then I showed it to the rest of the company and we started to build a team around me to help productize the prototype spike and thus was born AppCloud.

We launched it in late January 2009 as Engine Yard Solo as it started out only managing single machines at a time with the app servers, database servers and cache servers all running on one box. The only way to scale was to move to a bigger EC2 instance. But we launched and pretty fast too!

We started taking customers rapidly and taking customer feedback to rapidly iterate the system. By the end of the summer of 2009 we launched Engine Yard Flex(horrible names here, bear with me) which was the same system but supported full clusters on EC2 with load balanced self healing application tiers, Database tiers with read only slaves and utility servers so anything that didn’t fit into the other two boxes like nosql databases or search engines etc.

Then we took on even more customers and rapidly iterated on this until it was pretty nice and solid. The team has been doing this ever since and now there are way over 1000 customers on AppCloud(we removed the solo and flex names and just called it AppCloud). Many features have been added and the current incarnation of AppCloud is the vision I had in my head when we first started Engine Yard way back in 2006. We just were victims of our own success and could not deploy customers fast enough to get free time to build the automated system that AppCloud is today.

And we started moving people out of our own data center and off our own cloud onto AppCloud as it got more and more featureful. As far as I see it there is zero value in racking and stacking servers anymore. The money is going to be in management platforms for other peoples IaaS clouds like EC2 and others that are popping up all over the place.

But I believe the true future of cloud computing for developers is to not think about servers at all. It is now time to focus on the Application and new levels of abstraction that allow folks to use the computing resources in easier and easier ways.

So here it is the summer of 2010, I’ve been working my ass off at Engine Yard for 4 years now pouring every ounce of blood sweat and tears that I could muster to make it the best it could be.

February of this year my wife Regan and I had a beautiful baby boy named Ryland and that changed the game entirely. Now is the time for me to focus on spending as much time with my son and family as I can and I can no longer commit to 100 hour weeks at Engine Yard. I also wanted to move to Portland where my folks live so my son can grow up near his grandparents and my wife and I can have trustworthy babysitters so we can have a social live of our own(even if it’s just a little bit;)

After 4 years at Engine Yard I was so close to being fully vested on my stock that I decided to resign and leave the company in the competent hands and team that has been built up there over the years. I wish all of you the very best and will be watching closely to make sure you make me a ton of money someday when you sell the company like you promised me you would ;)

So here we are at the beginning of a new era. What comes next for me? Something truly awesome and even more challenging then Engine Yard, but that will allow me to work from home in Portland. I cannot yet reveal where I am going to work next but I can promise it will be another game changing project and I am chomping at the bit to get started.

I will make another post when I am allowed to reveal my new job and what I will be working on, but for now I just wanted to write down a short history of “Engine Yard, the Good Times”. Don’t get me wrong it has definitely not all been roses and taking lots of VC money is a hard thing to do and keep the balance of your vision of a company. But I believe I did the best I could for Engine Yard, I gave them many great inventions and lots of good code and I practically hired most of the devs, ops and support folks there just by knowing them from online or at conferences or by working with them on open source projects or talking with them on irc. That is the new way to build companies, College degrees don’t matter much IMHO anymore, for developers anyway. It’s more how you interact with the open source community and what you release yourself. Your github account has become your new resume and what you say on Twitter and in various IRC channels are more likely to get you the best jobs then any recruiters ever will.

I just want to say a fond farewell to Engine Yard and all my people there. I wish you all the best of luck and I know you will be a success with all the smart folks that work there.

It was just time for a new challenge for me, time to move on and time to focus a bit more on family then on startup life 24/7.

Farewell fond memories and friends at the Yard, I will miss each and everyone of you. Oh the stories I could tell if only I could. But I feel that telling this positive story of the history of EY is the classy way to go out and I wish Engine Yard all the best in the world.

Tags , , , ,  | 64 comments