Searching...
EngineYard Technology Stack
Posted by ezmobius Sun, 24 Sep 2006 19:06:00 GMT
People have been asking me for a little technology overview of how Engine Yard is structured. I’ll give a small run down here.
Our servers each have two dual-core AMD Opterons processors and 8Gigs of ram. The servers don’t have any hard drives in them, just 128Mb ATA flash chips. The servers boot up the Xen dom0(the hypervisor) off of the flash memory. And then the domU’s(user VPS’s) boot off of the Coraid SAN.
The Coraid SAN in the first cluster is 6 terrabytes and uses AoE(ATA over Ethernet) as its data transfer mechanism. So each coraid unit has dual Gbit connections to the cluster. Once the base user OS images are loaded into the domU’s, they then mount your slice’s storage space as a GFS mount off the SAN. This allows you to have multiple slices that all use the same filesystem storage so rails caching just works and the static webservers can also mount the public dir of your rails apps to serve cached and static pages.
Everything in this cluster is redundant so there are two or more of each unit. Coyote Point hardware load balancers with hardware ssl are up front and balance the traffic between the internet and your slices of the cluster. There are also two or more of every type of switch and power source in the cluster. This makes for a very reliable setup and when we need to reboot or cycle any one part of the system we have it set up to leap frog by taking one thing down and back up at a time so your sites don’t have to go down. Or if we privision more resources for you and need to reboot your slices, they do it one at a time.
Now let e explain what a ‘slice’ actually gets you. A slice consists of one Xen VPS running gentoo linux. This VPS only runs your rails application on a cluster of mongrels. So this VPS doesn’t have to run anything else except ruby, rails, your gems, a mongrel_cluster and something small like nginx or pound to balance between mongrels. This allows them to focus on your app and makes them perform very well. So also as part of each slice you get access to clustered mysql db’s, clustered front end static webservers, dns, outgoing mail, subversion and other infrastructure services. The sweet thing is that none of these other services run on your VPS. So a slice is more then just a VPS, it really is a ‘slice’ of the entire cluster architecture. So while one slice is a very nice deployment environment, the real benefits of our service happen at two or more slices. Two is the minumum number of slices to run if you want your rails app to be totally redundant.
The control panel is still under development but will manage your slices and applications easily. You will have environments like production or staging assigned to different slices. The you will take an svn revision or tag of your app and create a packge from it. A package is simply a wrapper around a certain revision of your app with all its versioned gem and unix dependencies. Then you push this package to one of your environments as a deployment. Everything uses acts_as_versioned so you can have the exact info about a certain deployment you made for QA.
EngineYard is kinda like a mini EC2 built only for rails ;)
So can people get SSH access to their VPS? If so, in theory people could quite easily have a development and production environments in once slice, right? Can you choose the number of mongrels you have running in your slice or is it a one mongel per slice setup? My site (which goes live in a few months) uses a custom cache server for general persistent object storage, how would something like this become redundant on your cluster? Looks like an interesting service, and I like the idea of being able to scale easily without having to move host, but SSH access is a deal breaker for me. "What you get" isn't very clear at this point.
Hey Ian-
You are correct the managed slices you don't get ssh access. But you do get a rake terminal where you can run any rake tasks you have written.
Of course people will have specific requirements that don't match the managed slices sometimes. So we will also be offering root slices. These are in the same cluster and setup the same way but you do get shell access to install any custom daemons or what have you. So these are unmanged slices that you can do whatever you need with and still utilize the cluster resources and tie them in with your managed app slices. We already have some clients that need custom daemons and so a root slice or two as part of your managed environment will give you the flexibility thast you want. We can also setup entirely dedicated custom clusters to your specs with whatever you need installed. Including a branded version of our control panel ;)
We aren't out of beta yet so we are being a bit vague on some details. This is mainly because we are still tuning and benchmarking and working out these issues. Rest assured we are going to make this one of the most kick ass rails hosts around. So if you have any concerns we are happy to address them with you. ez@engineyard.com
Mmmmm. When do we get to buy?
A root slice sounds perfect. I'm an ex-Gentoo developer by the way, I'm glad to see it being used here! The PR team would no doubt love to hear about your deployment (you'd get a bit of free advertising via the GWN too ;)). I'll give you guys another look in a couple of months.
"EngineYard is kinda like a mini EC2 built only for rails ;)" I'd say so. Man this is totally awesome, great work!
Quick question: Since the "slices" are basically VPS images with the filesystem mounted from the SAN, how are you provisioning the resources for each "slice"? Particularly CPU and RAM resources. (Also HD space?) Are static values set for each slice and clients buy more slices to get more resources or is it on some kind of dynamic scale depending on how much is going on within each slice?
Each slice will scale dynamically up to a certain point. After that point you add more slices to scale farther. We are still working out the exact details of slice resource allocation and will announce when we have a better idea. But we plan on making each slice capable of quite a bit of traffic. Its a different way of looking at rails hosting so we are treading new ground here. As we work out more and more details we will let everyone know more about the service and what to expect from each individual slice.
@Ian-
Yeah Gentoo ended up being the perfect distro to build our infrastructure on top of. It is by far the most cust6omizable and tuneable linux I have come across. We have built our own Engintoo distro ;) Its is super small and totally optimized for serving rails applications. We love Gentoo so once we are a bit further along we will contact the pr team at gentoo and give them a run down. Thaks for everyones interest. I'm super excited about engine yard.
Hi Ezra, Quite impressive the yard setup :-) Start my weekend lecture with Xen architecture and is evry interesting. We use a flash too to pre-boot our windows systems (and upgrade using a image from a network server). How the domU gentoo know which slice should it load? (hehe, thats the trick, right?) Anyway, looked at Coraid, but will stick to Areca (local RAID, we need high IO and too-much-to-count MB/s for video). Good weekend!
I'm very interested in signing up! Can you let me know when this service is ready to go? I'd love to port all my rails apps to a more reliable server. Dreamhost just isn't cutting it for me. Lately I've been working with our sysadmin to get Apache w/ mod_proxy_load_balancer load balancing mongrel clusters while also allowing Tomkat 5.0.28 to work with our Java stack. Very interesting project. Anyways, great site I'll be viewing it regularly.
The PR team would no doubt love to hear about your deployment
@Luis Lavena : I guess you never used a SAN before...When you use a clustered SAN your speeds are way above local raid sets. And it's so easy to migrate VM's. We use management software that watches all VM's with heartbeat. When single VM's die the system will reboot them and bring back online. When a hardware node fails all VM's on it will be migrated to a spare node immediately and they continue to run. Storage on a SAN is also much easier to maintain...all storage is managed in one tool. You have the benefits of snapshots and you can pu t a clustered filesystem upon it and give more than one VM access for a loadbalanced environment for example. And for the speed thing...I've seen single units providing over 700Mb/s iSCSI througput....imagine what it can do when clustered and using loadbalanced multi path io.
Newbie here but not newbie to the article, nice work!
Predicted values in template editing can be done through any PHP script?