Posted by ezmobius
Tue, 07 Apr 2009 20:19:00 GMT
I’m looking for a few good folks to join my team. You need to be able to work on site in San Francisco, we can assist with moving expenses for the right candidate.
You would work with my team(currently 5 devs). We do no bs agile style development with pairing where appropriate and test infected mentality . generally we have a fun, tight team and get to work on some of the hardest and most rewarding areas of software development relevant today.
You would be on my team working on http://engineyard.com/solo and it’s successors as well as Engine Yard’s on premises infrastructure.
You would be working on/with the following ruby technologies:
Ruby
Merb
Rails
Sinatra
DataMapper
Nanite
Chef
Event Machine
Many others..
You would be working on the following problem domains:
Scalable Ruby Deployment Architectures
Scalable Cloud Computing Architectures
Scalable Database Architecture
Distributed Computing
Key Value Data Stores
Infrastructure automation
Cloud provider API's
Multi-Cloud portability
Monitoring and alerting systems
Security in the Cloud
Horizontally Scalable Architectures
Many other interesting areas of computing
Familiarity with the following tech is nice but not necessarily required if you can learn fast:
Gentoo Linux
Nginx
Apache
Monit
Daemontools
Mysql
Postgresql
Memcached
Passenger
Mongrel
Thin
SSL Certs
Iptables/Networking
Erlang
If you think you have the “right stuff” and you have the kick ass take names attitude then send a resume with a short intro about who you are and why you are the person we should hire.
ez@engineyard.com
Tags automation, aws, chef, cloud, datamapper, deployment, merb, nanite | 8 comments
Posted by ezmobius
Sun, 20 Jul 2008 20:32:00 GMT
Valery Kholodkov has written a very cool nginx module for handling uploads.
The way this works is that you specify a location block to handle the uploads. So if you are using the standard nginx.conf for rails apps then you would add this in your server block right above your “location /” block:
# Upload form should be submitted to this location
location /upload {
# Pass altered request body to this location
upload_pass /internalupload;
# Store files to this location
upload_store /tmp;
# Set specified fields in request body
upload_set_form_field $upload_field_name.name "$upload_file_name";
upload_set_form_field $upload_field_name.content_type "$upload_content_type";
upload_set_form_field $upload_field_name.path "$upload_tmp_path";
}
# Pass altered request body to a proxy
location /internalupload {
proxy_pass http://mongrel;
}
Then make a simple upload form that does a multipart POST to /upload. Now you can have your Rails or Merb app on the backend with a route called /upload. In the action of your app that responds to the /upload route you will get a set of params that look like this(assume the name of your upload fields is called ‘file1’ and ‘file2’):
{"file2.path"=>"/tmp/0000123459", "file1.path"=>"/tmp/0000123458",
"file2.content_type"=>"image/png", "submit"=>"Upload",
"file2.name"=>"Picture 2.png", "action"=>"index",
"file1.name"=>"Picture 1.png", "controller"=>"test",
"file1.content_type"=>"image/png", "test"=>"value"}
What this is doing if parsing the multi-part mime boundaries in C in the nginx plugin, putting the parsed files into files in /tmp and then stipping the multipart stuff out of the POST body and replacing it with the info you need to get the name and location of the file on disk.
This means that by the time the request hits your application, the expensive mime parsing is already done and you simply move the file to it’s final resting place. This is a huge win since now the hard work is done in C in nginx before your app ever gets involved.
Of course this is a fresh new module so do your own testing and deciding whether or not this is a fit for your needs. But I think this is a great plugin and have verified it works as advertised.
Tags nginx, upload | 10 comments
Posted by ezmobius
Fri, 06 Jun 2008 22:47:00 GMT
I’d like to point folks to our new Engine Yard Express VMWare image. This is a replica of an EY slice that you can download and use locally whether you are an EY customer or not. When you boot it up it will autotune mysql to use the proper amount of ram, boot up a merb and a rails app along with nginx and monit et all. It will give you a user/pass and show you the IP address as part of the boot process. So just boot it up and peek at the IP address on port 80 and port 81, the index pages of the rails and the merb app have detailed instruction on how to get a deploy.rb and deploy your own code.
Enjoy!
8 comments
Posted by ezmobius
Thu, 21 Feb 2008 23:32:00 GMT
So after being in the works for almost 2 years now and after going through many rewrites as Rails deployment tech has changed, my book is finally done! I just have to thank all of my coauthors who put in tons of work to get this thing as polished as it is now. Especially Bruce Tate and Clinton Begin, thanks guys!
New in this release is the chapter on Apache and Nginx scaling as well as a performance chapter. ALso included is setting up your own Xen installation and setting up Mysql Master → Slave and Master → Master replicated databases.
If you were holding off on getting this book until the advanced content was in there then now is the time to snap it up. This is the final beta release and the book will be going to print fairly soon. I really like the way the book has come together and I think it is a wealth of knowledge about deploying Rails applications.
\m/ Yay! Writing a book was a lot more work then I thought it would be and I could not have done it without my wonderful co-authors and all of the great feedback from the early beta readers.
Thanks to everyone who provided feedback. You can grab your copy of the book here:
Deploying Rails Applications
Tags deploy, server | 18 comments
Posted by ezmobius
Wed, 06 Feb 2008 20:25:00 GMT
Want to work for the best and with the best? Are you interested in merb, rails, rubinius, and other ruby projects? Are you passionate about learning what you don’t know?
This position requires direct interaction with customers through the phone, ticketing system, irc, and email. You will be supporting some of the largest rails deployments on the internet. Can you handle it?
Desired: Experience with memcached, merb, monit, postgres, swiftiply.
Required: Solid ability to speak and write English. Good Attitude. Great work ethic. Experience with capistrano, mongrel, mysql, nginx, rails, subversion. Basic networking knowledge.
This position is right someone who isn’t afraid of hard work, loves facing new challenges each day, and wants to contribute to Engine Yard’s projects.
We offer comprehensive dental and medical insurance as well as the ability to work from home (or anywhere). Stock options are around the corner too.
Drop us a line at info@engineyard.com and tell us why you would be a great addition to our team. If you are the right person—we will send you an offer immediately. It’s a bonus if you are on the West Coast, maintain open source projects, or have a great community presence.
Posted by ezmobius
Thu, 06 Dec 2007 23:26:00 GMT
We are looking for a few killer Sysadmins/Cluster Engineers.
The engineer will build, maintain and support clusters of machines. This role also provides application and network support and communicates with users on systems status, including system changes, problems, enhancements and potential impacts.
This is a telecommute position. You will get to play with badass hardware.
Requirements:
- 3 years experience managing server class hardware
- 3 years experience as a Linux Administrator. Including configuration and maintenance.
- Experience with Gentoo Linux. Other distros a plus.
- Working knowledge with most of the following software/technologies: LVS (Linux Virtual Server), Bash shell scripting, Linux iptables, CoRAID AoE devices, RedHat Cluster Suite, TinyDNS, Exim, nginx, apache, mysql, postgresql, ruby, python, nagios.
- Experience performing 1st and 2nd level 24×7 production support
- Capable of implementing small to medium sized technology projects from implementation documents.
- Strong verbal and written communication skills.
- Capable of developing documentation to support completed tasks.
- Ability to create and maintain positive and professional business relationships with both internal associates and external customers.
- Must be able to work effectively in a team environment.
If you think you have the chops for this gig please send your resume and some text about why you would kick ass to:
Edward Muller emuller@engineyard.com
Tags sysadmins | no comments
Posted by ezmobius
Mon, 26 Nov 2007 21:15:00 GMT
Here is a nice little snippet for nginx that took me forever to figure out. The scenario is that you have a facebook app and you want to use rails built in page caching that writes out an html file that the webserver serves directly instead of hitting rails for the page.
The problem is that facebook only sends POST requests and Nginx and most other webservers do not allow serving a static file in response to a POST request. Nginx will return a 405 error.
The way to solve this is to catch the 405 error with an error_page directive and change it to a 200 ok request and serve the page cached file. Of course this still has to work with checking for html extensions and falling back to mongrel if there is no page cache. So here is the solution .. add this to your nginx vhost server{} block at the bottom and you will be good to go:
error_page 405 =200 @405;
location @405 {
index index.html index.htm;
# needed to forward user's IP address to rails
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_redirect false;
proxy_max_temp_file_size 0;
proxy_next_upstream error;
if (-f $request_filename) {
break;
}
if (-f $request_filename.html) {
rewrite (.*) $1.html break;
}
if (!-f $request_filename) {
proxy_pass http://mongrel;
break;
}
}
Enjoy ;)
Tags facebook, nginx | 1 comment
Posted by ezmobius
Fri, 09 Nov 2007 22:50:00 GMT
Nginx is my favorite webserver as you astute reader all probably know. It’s very fast, light and rock solid in production. My only complaint was that the proxy modue was round robin only. This means that if you had 3 mongrels and one of them performed and action that took say 20 seconds, for that 20 seconds every 3rd request would get stuck in line behind the mongrel doing the long blocking request. This is because nginx was merrily doing round robin and didn’t care if a mongrel was busy or not.
Now we all know that you should not have actions that take this long in our apps right? Of course we do, but that doesn’t stop it from happening quite a bit in a lot of apps I’ve seen.
So EngineYard.com put out a bounty on getting a fair proxy balancer that would keep track of when a mongrel is busy and not send requests to it until it has finished and is ready to take requests. This means that now we can effectively queue inside nginx very efficiently and only send requests to non busy mongrels. Thereby avoiding the waiting in the wrong line for service problem described above.
Grzegorz Nosek has stepped up to claim the bounty and has done a wonderful job of making this work very nicely. this is not a production release but I wanted to get this out there to get some testing from interested parties.
You can grab a tarball of the git repo here:
http://git.localdomain.pl/?p=nginx.git;a=tree;hb=upstream_fair-0.6
click the snapshot link to download a tarball. Then you can compile nginx with the new module like this:
$ cd nginx.git
$ ./configure—with-http_ssl_module
$ make
$ sudo make install
The only difference in configuration will be to add the “fair;” directive to your upstream block. For example:
upstream mongrel {
fair;
server 127.0.0.1:5000;
server 127.0.0.1:5001;
server 127.0.0.1:5002;
}
Here is an easy way to try this out and verify it’s working for you. Create a rails or merb app with one action that does a “sleep 15” and another action that returns some text immediately. Fire up a cluster of 3 mongrels behind your newly compiled nginx with the fair directive commented out.
Now open up 2 browser windows and hit the sleepy action in one window and repeatedly hit the other action in the second window. with the fair directive commented out every 3rd request for the fast action will hang until the sleeping action is done. This is because mongrel is queuing requests and nginx is just using round robin.
‘Now uncomment the “fair;” line and restart nginx and do the experiment again. This time you will notice that nginx avoids sending any requests to the sleeping mongrel and only balances between the free ones until the sleep finishes, then it is added back to the available pool.
Please give this a shot and beat it up some folks. Report any problems you have here in the comments. This is going to be huge for any rails apps running under nginx and I;’d like to ferret out any bugs before we have an official release.
Tags nginx, proxy, server | 23 comments
Posted by ezmobius
Fri, 31 Aug 2007 17:46:00 GMT
Hey folks-
Due to overwhelming influx of new customers we need to find some more help ASAP.
Are you a hacker at heart? Do you know rails, capistrano, linux and sysadmin stuff? Can you learn fast things you don’t know? Can you solve problems on your own and work with many diverse technologies?
We are very much still a startup company so we work long hours, doing cool stuff. Everyone wears multiple hats.
We need some people that aren’t afraid of anything on this list:
linux
mongrel
rails
nginx
swiftiply
capistrano
ruby
mysql
postgres
subversion
python
lvs
xen
some advanced network knowledge
coraid experience
vlad
god
You don’t have to be an expert in all of these things, but you need to be willing to learn from us and on your own.
Work would consist of doing new customer deployments, customer support(tickets, phone), writing automation tools for anything you do more then twice and pitching in on some of our open source projects like rubinius or merb.
If this sounds good to you and you aren;’t afraid of hard work then you may be the person we are looking for. Please send a quick note about why you would kick ass at this job and include a resume and any ruby code or tales of sysadmin heroics you want to show us.
We need some folks to start ASAP like today even. It’s a bonus if you are in the east coast usa timezone or even australia would be good.
So drop me a line at info@engineyard.com and tell me why we should hire you and why you want to work for us.
Thanks!
Tags deployment, engineyard, jobs | no comments
Posted by ezmobius
Sun, 05 Aug 2007 23:11:00 GMT
Wow it’s been 55 days since I last posted here! I’ve been very busy for the last few months. So here is a little update on what I’ve been up to.
Just released Merb 0.3.7 as a gem. Check it out, lots of new stuff in there. Lots of new code contriubtors as well. Merb has an open door svn policy> If you send me one or two good patches then I will give you commit rights.
I’ll be giving a talk on Merb at The Ruby Hoedown this coming weekend. WIll post the slides and hopefully have audio/video recording.
At Engine Yard we just finished our new third generation architecture and we are starting to dole out new slices on the new cluster tomorrow. We also launched Zaadz.com on their own dedicated cluster with 28cpu cores, 100Gigs of RAM and 14Tb SAN storage. They have a lot of room to grow into this cluster now and it was great fun designing the ultimate system with all services separated out nicely into their own VM’s
Anyway.. I will be blogging more here shortly. Have some nice tips on nginx + memcached and a new mongrel_leash customized monit dog watcher.
Tags ey, merb, speaking | 4 comments
Posted by ezmobius
Wed, 03 Jan 2007 22:28:00 GMT
I have been working on optimizing my nginx.conf file as I use it on more and more sites. Thanks to folks on the nginx and mongrel mailing lists for some of these fixes. Main improvements is in gzip of dynamic and other content as well as setting the proxy buffers to 0 and a few improvemets in the proxy header settings. This conf also includes a second vhost for ssl that points to the same mongrel cluster so you can hanlde ssl and non ssl with the same cluster and rails request.ssl? helper will work correctly.
Another big improvement is with static file handling. The credit on this one goes to Zed Shaw. He notcies that the rewrites and regex tests for rails cached pages were getting run for every request including static images. So a request for /images/foo.jpg woudl get served in this manor:
/images/foo.jpg/index.html # no file found continue
/images/foo.jpg.html # no file found continue
/images/foo.jpg # good send it!
This is obviously a performance hit for static files as it has two extra regexes and two extra file stats for every static image. Suck, I wish I found this sooner. To fix this all we need to do is add this test right at the top before the other rewrite tests:
# If the file exists as a static file serve it directly without
# running all the other rewite tests on it
if (-f $request_filename) {
break;
}
In my benchmarks this does increase the speed of static files about 8%. Also allowing gzip on
HTTP 1.0 requests made sure that the proper static assets are gzipped when possible. Also we force all proxied requests to be gzipped if they are the right content type. I also added more gziped content types and corrected a wrong one for javascript.
This is an important lesson about your production environment folks. With regards to what you accept as gospel from someone like me. “Trust but verify”. Zed was trying to track down how to do a custom rewrite for alternate cache directories and so he at my behest turned on the debug logging and consequently saw the rewrites happening on every static file. I haven’t been in the rewrite log for a while since I thought my config was good. So when you get config files from people like me off the net, please trust but verify on your own systems.
This new config is running close to 100 instances of nginx at Engine Yard. Nginx has proven to be an excellent choice for a front end proxy balancer for mongrel. It beats apache in all my tests now as far as non gzipped proxy requests, static file requests and it ties with apache for gzipped proxy requests. And its about 30 times more lightweight then apache.
If you were holding off to try nginx until it was proven out by others then let me tell you it has proven itself to me time and time again already as a very easy to manage, high performance options for rails deployment on mongrel. I don’t personally use apache2.2 right now for anything other than mod_svn for subversion over HTTP. But there is stirring of upcoming support for nginx and svn over webdav so stay tuned. When that happens I will happily drop apache from our rotation entirely. Of course take this with a grain of salt as always, I don’t have to support any php apps or legacy stuff right now, just rails. I haven’t personally used php with nginx yet but I see others getting fantastic results so I am not worried for when I may need it.
All in all this new config file performs significantly better then my old one. Somewhere on the order of 20% total improvement overall over the last config file. I have updated the link to poitn from the old article to the right confi file and I am linking it here again for your enjoyment:
New Nginx.conf
Tags deployment, loadbalance, nginx, proxy, server | 17 comments
Posted by ezmobius
Thu, 30 Nov 2006 03:44:00 GMT
I have just released Merb 0.0.7 as a gem. The docs are updated at:
http://merb.rubyforge.org
Here is the changelog:
* added respond_to
* added json POST body to be parsed into params
* added configurable log level
* added before filters. :symbols or Proc's
* added :include and :exclude hashes to before filters.
* merb cluster works better. pid files work with monit
* added merb_drb_server
* added drb session container
* added support for mongrel_upload_progress
* added nginx_send_file and send_file methods
* improved dispatching speed.
* keep mime parsing and route recorgnition out of mutex
* only mutex around places that can call ActiveRecxord objects.
* Improved memory profile with fastthread
* uses fson if installed falls back to json
* added basic auth module and filters.
* changed default template extensions to .herb .jerb and .xerb for html, js and xml
* added many helpers
* add start of framework specs built with rspec
I hope to have some real tutorials RealSoon Now™. Enjoy!
Rdocs and README
Merb Trac and tickets.
Merb svn repo
Tags erb, merb, mongrel, server | 5 comments
Posted by ezmobius
Sun, 24 Sep 2006 19:06:00 GMT
People have been asking me for a little technology overview of how Engine Yard is structured. I’ll give a small run down here.
Read more...
Tags cluster, deploy, engineyard, hardware | 14 comments
Posted by ezmobius
Tue, 12 Sep 2006 16:02:00 GMT
[UPDATE] The config file has been updated and commented so it is easier to figure out. It also sets the right header when it proxies to mongrel and doesn;t choke on the foo.js?394732323 urls that rails generated for static assets.
OK this is very sweet. We have a new Nginx conf file that works perfectly with rails page caching. nginx serves all static files and all rails cached file. Fast
I want to thank Alexy Kovyrin and James Cox for their help in getting this config perfected. This makes nginx truly one of the best options for fronting a cluster of mongrels.
Might as well get the latest version while we’re at it.
curl -O http://sysoev.ru/nginx/nginx-0.4.0.tar.gz
tar -xvzf nginx-0.4.0.tar.gz
cd nginx-0.4.0
./configure --sbin-path=/usr/local/sbin --with-http_ssl_module
make
sudo make install
Now for the new config file. Here you go folks, get it while its hot!
nginx.conf
Tags deployment, loadbalance, nginx, proxy, server | 28 comments
Posted by ezmobius
Wed, 23 Aug 2006 20:27:00 GMT
I have been on the lookout for a better front end for mongrel_clusters. I wanted one that had the flexibility of pound when it comes to proxying requests. But I also want something to serve static and cached files fast. Apache2.2/mod_proxy_balancer does a good job of this but it is way overkill if you don’t need apache for other sites with php or whatever. Lighttpd has always been one of my favorites but the current release has a broken mod_proxy that will not re-enable backends without a restart if they go down and back up again. Lighty will have a much better mod_proxy_core when 1.5 is released but thats a ways off it looks like and is not currently usable.
So I am very happy that I recently found a lightweight webserver that can serve static files very fast and that also has very complete proxy and rewrite modules. Its called nginx(which I have been pronouncing in my head as ‘engine x ;). I am very impressed with its performance and features. It is also super lightweight, taking up only a few Mb of ram when fully loaded while benchmarking. Nginx also has mod_fcgi. So it will work for externally spawned dispatch.fcgi’s or php-fcgi scripts. It does balancing and failover for fcgi as well. I haven’t tested the fcgi support yet but will do so in the near future. The only drawback I have found is that it is developed in Russia and while there are some english translations of the docs, they are not complete and don’t cover the proxy and rewrite modules yet.
But don’t fret, I have done most of the hard work for you and figured out most of the configuration needed to use nginx as a great front end for mongrel clusters. Nginx is my new favorite front end for my rails apps and has been very stable during my testing and abuse of it. It is actually faster then lighty for static files and the proxy and rewrite modules are way more complete. In my benchmarks on my macbookpro, nginx serves static files at 13+k requests/sec. And it performs better for proxied requests then pen, pound or apache2.2. It also handles ssl and vhosts like pound does, including regex conditionals and other niceties.
Click throuhg for the full details.
Read more...
Tags deployment, loadbalance, nginx, proxy, server | 48 comments
Searching...
