I don't fly so I was hoping for some audio or video recordings. I found a live video feed from some bloggers upstairs but nothing downstairs (at least not that I could find). I need to hunt down the slideshows for Crazyhorse, BuddyPress, Nginx etc. Finding a schedule of the tech stuff would probably be a good start, I found one for the general stuff upstairs but not downstairs. ZDnet covered only the non-dev side, but in detail.

Was there *no* bbPress presentation this year at WordCamp? Really?
Twitter went insane with WordCamp stuff!
added: some really good notes from Jeremy Person

ps. I can't believe Matt stopped using LiteSpeed and switched to Nginx. As fast as Nginx is, LiteSpeed is still faster and definitely has a better interface and tech support, but I guess Matt can afford his own techs now to support open source stuff so it makes him feel better to have that flexibility. LiteSpeed isn't cheap, especially when you have as many servers as Automattic must have.

pps. I can't believe Matt didn't think of all the vegans in SF and only had a BBQ lunch

De hecho, he encontrado este manual sobre como instalar Wordpress (usando nginx, una versión ligera de Apache muy eficiente) en Slicehost, incluyendo los pasos para hacer tu servidor más seguro. Una pequeña joya.

Como bonus he descubierto "Just-ping", una web para hacer pings desde varios servidores alrededor del mundo a tu web, viendo tiempos de respuesta. Y con un enlace a "just-traceroute" para ver los caminos que recorre la comunicación.

I have nginx working as a reverse proxy for Apache / mod_python serving Django, somewhat according to the instructions here:

http://lethain.com/entry/2007/jul/17/dreamier-dream-server-nginx/

Basically, Nginx sits in the front, and forwards all requests but /media/ to apache.

My problem arose when I wanted to implement this SSL Redirect middleware: http://www.djangosnippets.org/snippets/880/

Last time I set up SSL on Django, I had nginx forward HTTPS requests to a separate apache virtual host, also listening on port 443. The SSL certification was on apache, so Djangos request.is_secure() returned True with no problems.

This time though, I have nginx dealing with the SSL, and forwarding the request to the same apache as regular http (one apache vhost), so apache is blissfully unaware of ssl. This means request.is_secure() always returns false, which means the SSLRedirect middleware endlessly loops around a redirect. Endlessly redirecting because is_secure() always returns False.

To fix the problem I just set up my nginx to add a header "HTTP_X_FORWARDED_PROTOCOL" = "https"

and replaced the request.is_secure() in our SSLRedirect middleware with a check to first see if the above mentioned header is in request.META, and then if its value is 'https'. Return true if that is the case, and we successfully get a redirect.

I guess the value of it doesn't matter, just that it exists vs not, but that feels very dirty.

Now, the documentation for attachment_fu indicates that the standard way to do downloads using the file system store is Attachment#public_filename. That works great, if you're comfortable with relying on somewhat obfuscated URLs for privacy, and nothing else. You can always do something like this in your view:
[sourcecode language='ruby']
<%= link_to @attachment.filename, @attachment.public_filename -%>
[/sourcecode]

So how do we add some access control to this? Let's start with your AttachmentsController, and add a #show action.

[sourcecode language='ruby']
def show
@attachment = current_user.attachments.find(params[:id])
# Do some other stuff if we need to
head(:x_accel_redirect => @attachment.public_filename,
:content_type => @attachment.content_type,
:content_disposition => "attachment; filename=\"#{@attachment.filename}\"")
end
[/sourcecode]

The X-Accel-Redirect header tells nginx to serve up the assigned path directly, letting Mongrel get on to serving the next request.

Now, in your nginx.conf, you need to configure your attachments path, which should work something like this (inside your main server section).

[sourcecode language='jscript']
location /attachments {
root /var/www/myapp/public;
internal;
}
[/sourcecode]

You'll want to replace the root directive with the actual path to RAILS_ROOT/public. The internal keyword tells nginx that it should only serve up files under that location when directed to via X-Accel-Redirect.

Hope this helps.

Spread it: |||||

Basic HTTP features:

1. Handling of static files, index files, and autoindexing; open file descriptor cache;
2. Accelerated reverse proxying without caching; simple load balancing and fault tolerance;
3. Accelerated support without caching of remote FastCGI servers; simple load balancing and fault tolerance;
4. Modular architecture. Filters include gzipping, byte ranges, chunked responses, XSLT, and SSI. Multiple SSI inclusions within a single page can be processed in parallel if they are handled by FastCGI or proxied servers.
5. SSL and TLS SNI support.

How to install Nginx

sudo aptitude install nginx

How to start Nginx

sudo /etc/init.d/nginx start

Testing Nginx

http://localhost/

Start and Stop nginx webserver

Following are the commands to start, stop and restart nginx

Start
sudo /etc/init.d/nginx start

Stop
sudo /etc/init.d/nginx stop

Restart
sudo /etc/init.d/nginx restart

I have a VPS that hosts my Django application via FastCGI behind nginx on Ubuntu 7.10. Nginx comes with an init.d script which starts it at boot time (for me on Ubuntu it runs as /etc/rc5.d/S20nginx -> /etc/init.d/nginx) but I was having to turn the Django app on (via FastCGI) every time the server was rebooted. Since the only reason nginx runs was for my Django app, I wanted the app to start up just like nginx does so that on a reboot, server crash, etc I don't need to ssh in and restart the app. To do this, I started with the posting on the Django wiki (http://code.djangoproject.com/wiki/InitdScriptForLinux), and then tweaked it for my own purposes, mainly: one single Django app.

The idea here is that I'd simply run `/etc/init.d/my_django_app start` and be done with it while the server itself would do this automatically when it boots up. Here is what I did to let me do this.

1. Tweaked the init.d script, you can find it here:
   http://static.geewax.org/my_django_app
2. Copied that script into `/etc/init.d/my_django_app`
3. Ran `update-rc.d my_django_app defaults`
4. Rebooted the machine to test it out

Note that the update-rc.d (I'm told) is only for Ubuntu (or Debian-like) systems.

Now, if I wanted to run more than one app, I can copy the script to a different name and follow the procedure again which lets me do fine-grained application enabling/disabling of Django services should the need arise.

There was also the possibility that the static website (for whatever reason) could start getting hammered with requests, and this tiny xen instance would definitely not scale very well. (And we all know how scalability is the best thing since bubble gum...)

To deal with this situation, I sat down and started brainstorming on how to let Amazon and S3 handle my static web hosting and here is my result:

There are three components here: DNS entries (I use bind9) , a web server (I use nginx), and of course, S3.

Web Server (nginx):

I decided that I wanted people to be able to go to http://example.org and http://www.example.org (and have http://example.org redirect to the www subdomain). Unfortunately S3 doesn't support a default index page (that is, 'bucket.s3.amazonaws.com/' returns the XML of a list result rather than showing the contents of index.html), so I need to do some rewriting:

1. Rewrite the non-www address to the www address;
2. Rewrite the root get request (/) to index.html; and
3. Rewrite all requests to a www1 subdomain which would handle the static data.

My nginx.conf and sites-enabled/example.org files are here:

• http://static.geewax.org/nginx.conf
• http://static.geewax.org/sites-enabled.example.org

DNS Entries (bind9):

Note: Although I use bind because I have a full xen instance, you could easily just set your zone file to have these similar records.

We want to be able to handle the root, and the www subdomain, and send off the www1 (static) subdomain onto s3 to do the actual hosting. This translates into the following records:

(This assumes your root A record points to 1.2.3.4)
www IN A 1.2.3.4 # (IP address of your server)
www1 IN CNAME s3.amazonaws.com. # (Note the trailing dot)

My zones/example.org.db file is here: http://static.geewax.org/example.org.db

S3:

In your S3 console (I use the S3 Organizer Firefox plugin) you should create a bucket called www1.example.org (this gives you the URL www1.example.org.s3.amazonaws.com). The CNAME you created in the DNS records will point to S3 and this bucket will be accessed for data. Now all you have to do is upload your content to S3 and you're all set.

Issues:

Obviously the main issue here is that the first hit to the website will go through my server, but afterwards all data is strictly sticking to S3; I think I can live with a first hit penalty of a couple rewrites in nginx.

References:

1. http://wiki.codemongers.com/NginxHttpRewriteModule

I'm going to compile in nginx server from the source. I'm assuming that you've packages like ruby, Rails etc. pre-installed on your system/server.

I've compiled in nginx from the source, so you'll need a gcc complier.

Download the source.
wget http://sysoev.ru/nginx/nginx-0.5.35.tar.gz

Untar it
tar zxvf nginx-0.5.35.tar.gz

Run.

cd nginx-0.5.35
./configure --prefix=/usr/local/nginx
make
make install

If everything does fine, nginx will get installed /usr/local/nginx

Here is a sample nginx configuration file. Just copy it as nginx.conf(/usr/local/nginx/conf/nginx.conf) and modify it accordingly four your application setup.

user  www www;
worker_processes  3;

error_log  /var/log/nginx/error.log;

pid        /var/run/nginx.pid;

events {
    worker_connections  1024;
}

http {
    include       /usr/local/nginx/conf/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] $status '
                      '"$request" $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "http_x_forwarded_for"';

    #access_log  logs/access.log  main;

    sendfile        on;
    tcp_nopush     on;
    tcp_nodelay    on;

        gzip  on;
    upstream mongrel {
     server 127.0.0.1:4000;
     server 127.0.0.1:4001;
    }

        #Rails App here
            server {
        listen       80;
        root /var/www/railsapp/public;
        index index.html index.htm;
        server_name yourdomain.com www.yourdomain.com;
        client_max_body_size 50M;

        access_log  /var/log/nginx/localhost.access.log;

        location / {
         proxy_set_header X-Real-IP $remote_addr;
         proxy_set_header X-Forwarded_for $proxy_add_x_forwarded_for;
         proxy_set_header Host $http_host;
         proxy_redirect false;
         proxy_max_temp_file_size 0;

         if (-f $request_filename) {
            break;
          }
         if (-f $request_filename/index.html) {
            rewrite (.*) $1/index.html break;
         }
         if (-f $request_filename.html) {
            rewrite (.*) $1.html break;
         }
         if (!-f $request_filename) {
            proxy_pass http://mongrel;
            break;
         }

        }
        # redirect server error pages to the static page /50x.html
        #
        error_page   500 502 503 504  /500.html;
        location = /500.html {
            root  /var/www/railsapp/public;
        }
    }

}

You may start the nginx server using the command

/usr/local/nginx/bin/nginx -c /usr/local/nginx/conf/nginx.conf

Now you may want an init startup script to start|stop|restart the server. Just copy the below script as /etc/init.d/nginx and set it executable(chmod 755 /etc/init.d/nginx)

#!/bin/sh

# Description: Startup script for nginx webserver on Debian. Place in /etc/init.d and
# run 'sudo update-rc.d nginx defaults', or use the appropriate command on your
# distro.
#
# Author:       Ryan Norbauer
# Modified:     Geoffrey Grosenbach http://topfunky.com

set -e

PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
DESC="nginx daemon"
NAME=nginx
DAEMON=/usr/local/nginx/sbin/$NAME
CONFIGFILE=/usr/local/nginx/conf/nginx.conf
PIDFILE=/var/run/$NAME.pid
SCRIPTNAME=/etc/init.d/$NAME

# Gracefully exit if the package has been removed.
test -x $DAEMON || exit 0

d_start() {
  $DAEMON -c $CONFIGFILE || echo -n " already running"
}

d_stop() {
  kill -QUIT `cat $PIDFILE` || echo -n " not running"
}

d_reload() {
  kill -HUP `cat $PIDFILE` || echo -n " can't reload"
}

case "$1" in
  start)
        echo -n "Starting $DESC: $NAME"
        d_start
        echo "."
        ;;
  stop)
        echo -n "Stopping $DESC: $NAME"
        d_stop
        echo "."
        ;;
  reload)
        echo -n "Reloading $DESC configuration..."
        d_reload
        echo "reloaded."
  ;;
  restart)
        echo -n "Restarting $DESC: $NAME"
        d_stop
        # One second might not be time enough for a daemon to stop,
        # if this happens, d_start will fail (and dpkg will break if
        # the package is being upgraded). Change the timeout if needed
        # be, or change d_stop to have start-stop-daemon use --retry.
        # Notice that using --retry slows down the shutdown process somewhat.
        sleep 1
        d_start
        echo "."
        ;;
  *)
          echo "Usage: $SCRIPTNAME {start|stop|restart|force-reload}" >&2
          exit 3
        ;;
esac

exit 0

Also don't forget to add the user www using "useradd -s /sbin/nologin www" if the user doesn't exists.

Now time to setup mongrel for your app. Install the mongrel gem, if it's not installed already.

gem install mongrel

Then configure the mongrel cluster for your app and start the mongrel server.

cd /var/www/railsapp (your application directory)

mongrel_rails cluster::configure -e production -p 4000 -N 2

mongrel_rails cluster::start

You can of course use the mongrel_rails cluster::start|stop|restart commands to manage your mongrel instance.

Hope that this tutorial is useful...

A pesar de que en ese último link explica por encima como hacerlo, lo voy a contar yo también.

• Bajamos la versión 0.6.31 de nginx en http://sysoev.ru/nginx/nginx-0.6.31.tar.gz. Según el comentario de Marlon (el creador del parche) debería funcionar también con las versiones 0.6.30 y 0.6.29, y mirando los changelogs pienso que tambíen debería funcionar en la última que es la 0.6.32. Si alguien lo prueba que me lo diga.
• Descomprimimos:
  # cd /usr/src/
# tar xvzf nginx-0.6.31.tar.gz
• Bajamos el parche:
  # wget http://bugs.gentoo.org/attachment.cgi?id=153345 -O nginx_syslog.patch
• Parcheamos:
  # patch -p0 < nginx_syslog.patch
• Compilamos e instalamos nginx:
  # cd nginx-0.6.31
# ./configure --with-syslog
# make
# make install

En mi caso antes de compilar he tenido que hacer un cambio en una linea de los fuentes del nginx. He tenido que substituir en el fichero auto/cc/gcc la siguiente linea:
CFLAGS="$CFLAGS -Werror"
por:
CFLAGS="$CFLAGS"

Esto únicamente hace que no se rompa la compilación al encontrar algun warning. En mi caso los warning que lanza la compilación se pueden ignorar tranquilamente, por lo que resulta seguro continuar con ellos.

Si todo ha ido bien deberíamos de tener funcionando nginx. Faltaría unicamente configurarlo y ponerlo en marcha. En el fichero de configuración no hace falta indicar nada para que mande correctamente los logs al syslog. Por defecto los manda a la facility daemon.

Yo lo he probado ya en servidores en producción y por el momento funciona estupendamente.

Desde aquí agradezco el esfuerzo a Marlon de Boer, que no he conseguido encontrar un blog suyo donde hacerlo.

My first analysis was that this problem was caused by "direct" requests (those with a server cookie) always being considered before the load balanced ones. But while fixing this design idiocy, I discovered a real problem : it was perfectly possible for a fresh new request to be served immediately without passing through the queue, causing requests in the queue to be delayed for at least as long as the queue timeout, until they might eventually expire. Now *that* explains the horrible peaks on Alexander's graphs. My problem was that it was a real misdesign, which could not be fixed by a 3-liner patch. So I spent the whole week reworking the queue management logic in a saner manner and running regression tests.

The fix has further repercussions:

[T]he good news is that not only this fixes a number of 503 errors and long response times when running with a low maxconn, but as an added bonus, the "redispatch" option is now naturally considered when a server's maxqueue is reached, so that it will now not be necessary anymore to trade between large queues and the risk of returning 503 errors.

Willy also realized that his redesign work would lead the way to priority-based request scheduling in the future, which is great news.

With the new release in hand, I have finally found the time to sit down and do a rematch. The conclusion? In short, the patch works as intended: It eliminates the odd spikes while still providing smoother performance than Nginx. The spikes that remain are present with Nginx as well, and their regularity implies some kind of periodic activity, possibly on the box itself, although a much more likely culprit is Ruby's garbage collection. Damn you, curiously slow and old-fashioned interpreter implementation!

Finally, some people requested CPU usage data from vmstat. For this new benchmark I updated my scripts to run vmstat concurrently with ab, hoping there would some meaty differences for charting, but it turns out that there is no significant difference between HAProxy and Nginx — at best, CPU usage looks a trifle smoother with HAProxy, but this could be a fluke. I suspect you have to amp up the load considerably to achieve a sensible comparison. Still, I have included the vmstat data in the raw data tarball for anyone who is interested.

Anyway, enjoy the graphs. Many thanks to Willy for working out a solution so promptly and expertly.

Nginx vs HAProxy at 3 concurrent connections

Nginx vs HAProxy at 10 concurrent connections

Nginx vs HAProxy at 30 concurrent connections

With a product like WebMynd, where part of the service we provide is to save and index a person's web history, this sort of explosive expansion brings with it some growing pains. Performance was a constant battle with us, even with the relatively low user numbers of the first few months. This was due mainly to some poor technology choices; thankfully, the underlying architecture we chose from the start has proven to be sound.

I would not say that we have completely solved the difficult problem in front of us - we are still not content with the responsiveness of our service, and we're open about the brown-outs we still sometimes experience - but we have made huge progress and learned some invaluable lessons over the last few months.

What follows is a high level overview of some of the conclusions we've arrived at today, best practices that work for us and some things to avoid. In later weeks, I plan to follow up with deeper dives into certain parts of our infrastructure as and when I get a chance!

Scaling is all about removing bottlenecks

This sounds obvious, but should strongly influence all your technology and architecture decisions.

Being able to remove bottlenecks means you need to be able to swap out discrete parts which aren't performing well enough, and swap in bigger, faster, better parts which will perform as required. This will move the bottleneck somewhere else, at which point you need to swap out discrete parts which aren't performing well enough, and swap in bigger, faster, better parts... well you get the idea. This cycle can be repeated ad infinitum until you've optimised the heck out of everything and you're just throwing machines at the problem.

At WebMynd, for our search backend, we've done this four or five times already in the five months we've been alive, and I think I still have some iterations left in me. Importantly, I wouldn't say that any of these iterations were a mistake. In a parallel to the Y Combinator ethos of launching a product early, scaling should be an iterative process with as close a feedback loop as possible. Premature optimisation of any part of the service is a waste of time and is often harmful.

Scaling relies on having discrete pieces with clean interfaces, which can be iteratively improved.

Horizontal is better than vertical

One of the reasons Google triumphed in the search engine wars was that their core technology was designed from the ground up to scale horizontally across cheap hardware. Compare this with their competitors' approach, which was in general to scale vertically - using larger and larger monolithic machines glued together organically. Other search engines relied on improving hardware to cope with demand, but when the growth of the internet outstripped available hardware, they had nowhere to go. Google was using inferior pieces of hardware, but had an architecture and infrastructure allowing for cheap and virtually limitless scaling.

Google's key breakthroughs were the Google File System and MapReduce, which together allow them to horizontally partition the problem of indexing the web. If you can architect your product in such a way as to allow for similar partitioning, scaling will be all the more easy. It's interesting to note that some of the current trends of Web2.0 products are extremely hard to horizontally partition, due to the hyper-connectedness of the user graph (witness Twitter).

The problem WebMynd is tackling is embarrassingly partitionable. Users have their individual slice of web history, and these slices can be moved around the available hardware at will. New users equals new servers.

Hardware is the lowest common denominator

By running your application on virtual machines using EC2, you are viewing the hardware you're running on as a commodity which can be swapped in and out at the click of a button. This is an useful mental model to have, where the actual machine images you're running on are just another component in your architecture which can be scaled up or down as demand requires. Obviously, if you're planning on scaling horizontally, you need to be building on a substrate which has low marginal cost for creating and destroying hardware - marginal cost in terms of time, effort and capex.

A real example

To put the above assertions into context, I'll use WebMynd's current architecture:

The rectangles represent EC2 instances. Their colour represents their function. The red arrow in the top right represents incoming traffic. Other arrows represent connectedness and flows of information.

This is a simplified example, but here's what the pieces do in general terms:

• All traffic is currently load balanced by a single HAProxy instance
• All static content is served from a single nginx instance (with a hot failover ready)
• Sessions are distributed fairly across lots of TurboGears application servers, on several machines
• The database is a remote MySQL instance
• Search engine updates are handled asynchronously through a queue
• Search engine queries are handled synchronously over a direct TurboGears / Solr connection (not shown)

One shouldn't be timid in trying new things to find the best solution; almost all of these parts have been iterated on like crazy. For example, we've used Apache with mod_python, Apache with mod_proxy,  Apache with mod_wsgi. We've used TurboLucene, looked very hard at Xapian, various configurations of Solr.

For the queue, I've written my own queuing middleware, I've used ActiveMQ running on an EC2 instance and I'm now in the process of moving to Amazon's SQS. We chose to use SQS as although ActiveMQ is free as in beer and speech, it has an ongoing operations cost in terms of time, which is one thing you're always short of during hyper-growth.

The two parts which are growing the fastest are the web tier (the TurboGears servers) and the search tier (the Solr servers). However, as we can iterate on our implementations and rapidly horizontally scale on both of those parts, that growth has been containable, if not completely pain free.

 
Amazon's Web Services give growing companies the ideal building blocks to scale and keep up with demand. By iteratively improving the independent components in our architecture, we have grown to meet the substantial challenge of providing the WebMynd service to our users.
 

Don't think it will work in 5 minutes and you don't need some knowledge about how to setup an webserver and a php application, you definitely need it.

If you think this is a step by step description for people which only want to copy and past commands and it works then you are wrong here, please take a websearch engine for a alternative description.

What you need:

1. nginx http://www.nginx.net/
2. php-fm http://php-fpm.anight.org/
3. php http://www.php.net/
4. a compile environment
5. the development libraries for the wanted php features and libxml2
6. A clear test scenario and target how many requests per second should be reached on static and dynamic part.

How you can build this environment:

1. Extract, build and install nginx http://wiki.codemongers.com/NginxGettingStarted
2. Extract, build and install php-fm http://php-fpm.anight.org/docs.html
3. configure nginx for fast-cgi proxying http://wiki.codemongers.com/NginxHttpFcgiModule

A brief description how the flow is:

• A request comes to nginx
• He takes a look into his config to find the right location or server config
• If the static content rule matches then he will deliver the content directly from the disc
• If the dynamic rule matches the he will contact php-fm, which manage the php-instances
• php-fm execute the php script and send the dynamic content back to nginx
• nginx sends the content back to the client

There are some pitfalls:

1. client_body_buffer_size
2. fastcgi_buffers*

You should take your time to find the right config setup for nginx and php-fm to be able to fulfill the assumed traffic.

I have started to translate the russian documentation, which is still not complete.

You can take a look here:

why_fastcgi_en

current_php_fastcgi_problems_en

extra_features_en

By default nginx do not use this event methode due to the fact that there was some security related problems Security Vulnerabilities in The Solaris Event Port API May Result in a Denial of Service (DoS) Condition and  Vulnerability in the Solaris 10 Event Port Implementation May Lead to a System Panic, Resulting in a Denial of Service (DoS) if you have a newer Version or applied the Patch you should use the event ports in nginx.

You only need to add

use eventport;

in the events { ... } section. If you don't add this then nginx by default us the /dev/poll method.

More about the possible polling methods can be found here.

1.) get it

wget http://sysoev.ru/nginx/nginx-0.7.1.tar.gz

2.) extract it

gtar xfvz nginx-0.7.1.tar.gz

3.) build it

cd nginx-0.7.1 && ./configure --prefix=$HOME/servers/nginx --with-http_ssl_module --with-debug --with-cc=/opt/SunStudioExpress/bin/cc

It's so easy, thanks Igor.

Here's how to get it working on Nginx Ubuntu.

Follow the instructions for the source installation. Do not apt-get apache and apache associated libraries. Source installation.

I presume you already have mysql and php installed.


apt-get install imagemagick wv pdftohtml html2text html2ps htmldoc aspell links mono-gmcs mono-mcs libmono-sqlite2.0-cil libmono-sqlite2.0-cil libmono-system-web2.0-cil


Note the above may not be all you need. The complete package list is at Source installation.

Make sure pear is installed.... aptitude install php-pear

There are problems with /skins/ace/neutral/css.php files etc. Sometimes they these files can't find the includes files so you may have to edit these and put the full path for these includes.

After the full install you may have to also aptitude install libmono-i18n2.0-cil otherwise you might get some 500 errors with mysql.

If you get an error with @api you may have to change the /etc/dekiwiki/mindtouch.host.conf

# hostname to listen on
IP="*:8081"

Note I also had to:

cp source_folder/config/dekiwiki-init.debian /etc/init.d/dekiwiki && chmod +x /etc/init.d/dekiwiki

mkdir /var/www/.wapi

server {
listen 80;
server_name wiki.mookiesplace.com;
access_log /usr/local/nginx/logs/campaign.access.log;

location ~* ^.+\.(html|jpg|jpeg|gif|png|ico|css|zip|tgz|gz|rar|bz2|doc|xls|exe|pdf|ppt|txt|tar|mid|midi|wav|bmp|rtf|js)$
{
root /home/mookiesplace/dekiwiki;
expires 10d;
break;
error_page 404 /index.php?title=$uri;
}

location / {
root /home/jollymarketing/campaignwiki;
index index.php;

rewrite ^/$ /index.php?title= last;
rewrite ^/@gui/(.*)$ /proxy.php?path=$1 last;

if (!-e $request_filename) {
rewrite ^/(.+)$ /index.php?title=$1 last;
}
}

location /@api {
proxy_pass http://127.0.0.1:8081;
}

location ~ \.php$ {
fastcgi_pass 127.0.0.1:8084;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME /home/mookiesplace/dekiwiki$fastcgi_script_name;
include /usr/local/nginx/conf/fastcgi_params;
}
error_page 500 502 503 504 /50x.html;

location = /50x.html {
root /var/www/nginx-default;
}

}

First, a bit of background. For a long time we at Bengler have been using Nginx as the main web server for our projects (1, 2), as well as to proxy Rails running under Mongrel. Nginx is a superb little open-source web server with a small footprint, sensible configuration language, modern feature set and buckets of speed. However, we quickly realized that the load balancing features of the proxy are not up to scratch.

The core problem is the proxy load balancing algorithm. Nginx only comes with a round-robin balancer and a hash-based balancer. Only the former is of interest to us since our object is to distribute the load evenly across a pack of Mongrel back ends. The round-robin algorithm is often an acceptable tool: if every request finishes within a few milliseconds, there's no problem.

But if a page takes a while to load, Nginx will start routing requests to backends that are already processing requests — as a result, some backends will be queueing up requests while some backends will remain idle. You will get an uneven load distribution, and the unevenness will increase with the amount of load subject to the load-balancer.

So when Gzegorz Nosek, backed by EngineYard, announced his fair load balancer module, we naturally pounced on it. Gzegorz's module routes requests to the back end with the fewest outstanding requests, and this improved performance a lot.

Unfortunately, Gzegorz's patch is not completely stable, and turned out to be the main source of our stability problems of late. Sometimes it sits down chewing the carpet while backends go idle and requests pile up, or worse, goes into tailspin and refuses to serve requests, for which the only remedy is a cold restart of Nginx. Even in normal operation, however, it will often send multiple connections to a backend even when some are idle, since there is no limit on the number of connections each backend can receive.

After reading about HAProxy (there's nice blog Rails-oriented blog post here), I felt the itch to try out this product myself. HAProxy has a handsome feature set:

• It's is proxy — and only a proxy. It can't serve files, for example: proxying is all its does.
• It can proxy anything TCP-based — not just HTTP.
• Plenty of load-balancing algorithms, including a "least connections" strategy that picks the backend with the fewest pending connections. Which happens to be just what we want.
• Backends can be sanity- and health-checked by URL to avoid routing requests to brain-damaged backends. (It can even stagger these checks to avoid spikes.)
• A dedicated status page gives you backend status, uptime and lots of yummy metrics. There's also a way to read metrics from a Unix domain socket.
• Requests can be routed based on all sorts of things: cookies, URL substrings, client IP, etc.

In a real-world situation, some of these requests would simply time out, hopefully to be rescued by a friendly "sorry, we're overloaded" error page. Is this an acceptable compromise between performance and responsiveness? I think it is, given that they should only occur during exceptional load; in such situations I prefer serving really fast pages to most users and possibly disappointing an extremely small number of users, rather than letting everyone suffer.

I think these results show that HAProxy is a better choice for us. The additional features and fine-grained proxy control are also extremely welcome. HAProxy's lack of support for sharing static files means that we will also put Nginx behind HAProxy and route requests accordingly.

You can download the raw data here.


server {
listen 80;
server_name yourhotdomain.com;
access_log logs/yourhotdomain.access.log;
location / {
proxy_pass http://localhost:8080;
}
}


If you want to change the datastore location do this:


sudo ./dev_appserver.py helloworld/
ctrl c
cp /tmp/dev_appserver.datastore /some_new_path/dev_appserver.datastore
sudo ./dev_appserver.py --datastore_path=/some_new_path/dev_appserver.datastore helloworld/


Your App Engine app should now be running on 'yourhotdomain.com'.

If you want to run it on your local computer just change your /etc/hosts file to include 'yourhotdomain.com'


127.0.0.1 yourhotdomain.com yourhotdomain.com


After several tests with this deployment configuration , we encountered some problems of performance (i will detail what our application does in a future article) :

- First with Apache server, after an idle time, then accessing a page of our application, we should wait for a long time before our page loads. Seems fcgid processes take a long time to (re)start.

- In our application, we deliver some files with a download controller : for bigger files, the download was very long.

We decided recently to switch the Apache2 / fcgid configuration to Mongrel (Ruby HTTP server) + nginx (as proxy in front of Mongrel).

I will explain you how to proceed.

We suppose here you have already installed Ruby , Rails, mySQL.

Connect as root and follow these steps :

First stop Apache2 :

#/etc/init.d/apache2 stop

Install Mongrel

# gem install mongrel

# gem install mongrel_cluster

Install nginx:

# aptitude install nginx

Configure what we have installed

We have to setup our server apps to boot when the server starts, and shutdown gracefully when the server reboots.

Concerning nginx, it already did automatically this setup at install:

# ls -l /etc/rc?.d/*nginx
lrwxrwxrwx 1 root root 15 2008-05-05 11:00 /etc/rc0.d/K20nginx -> ../init.d/nginx
lrwxrwxrwx 1 root root 15 2008-05-05 11:00 /etc/rc1.d/K20nginx -> ../init.d/nginx
lrwxrwxrwx 1 root root 15 2008-05-05 11:00 /etc/rc2.d/S20nginx -> ../init.d/nginx
lrwxrwxrwx 1 root root 15 2008-05-05 11:00 /etc/rc3.d/S20nginx -> ../init.d/nginx
lrwxrwxrwx 1 root root 15 2008-05-05 11:00 /etc/rc4.d/S20nginx -> ../init.d/nginx
lrwxrwxrwx 1 root root 15 2008-05-05 11:00 /etc/rc5.d/S20nginx -> ../init.d/nginx
lrwxrwxrwx 1 root root 15 2008-05-05 11:00 /etc/rc6.d/K20nginx -> ../init.d/nginx

As you can see, for runlevels 0, 1 and 6 there is a K at the beginning of the link, for runlevels 2, 3, 4 and 5, there is a S. Those two letters stands for Kill and Start.
On Debian (and Ubuntu), runlevels 2, 3, 4 and 5 are multi-users runlevels.
Runlevel 0 is Halt.
Runlevel 1 is single user mode
Runlevel 6 is reboot

Remove Apache 2 service

By hand you should remove every link /etc/rc.X/*apache2.

Using update-rc.d is as simple as :

# update-rc.d -f apache2 remove

Removing any system startup links for /etc/init.d/apache2 ...
/etc/rc0.d/K09apache2
/etc/rc1.d/K09apache2
/etc/rc2.d/S91apache2
/etc/rc3.d/S91apache2
/etc/rc4.d/S91apache2
/etc/rc5.d/S91apache2
/etc/rc6.d/K09apache2

Add Mongrel service for automatic startup/stop

# cp /usr/lib/ruby/gems/1.8/gems/mongrel_cluster-1.0.5/resources/mongrel_cluster /etc/init.d/mongrel_cluster

Edit /etc/init.d/mongrel_cluster and add the following environment setup:

PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local:/usr/local/sbin:/usr/local/bin

# chmod +x /etc/init.d/mongrel_cluster

# update-rc.d mongrel_cluster defaults
Adding system startup for /etc/init.d/mongrel_cluster ...
/etc/rc0.d/K20mongrel_cluster -> ../init.d/mongrel_cluster
/etc/rc1.d/K20mongrel_cluster -> ../init.d/mongrel_cluster
/etc/rc6.d/K20mongrel_cluster -> ../init.d/mongrel_cluster
/etc/rc2.d/S20mongrel_cluster -> ../init.d/mongrel_cluster
/etc/rc3.d/S20mongrel_cluster -> ../init.d/mongrel_cluster
/etc/rc4.d/S20mongrel_cluster -> ../init.d/mongrel_cluster
/etc/rc5.d/S20mongrel_cluster -> ../init.d/mongrel_cluster

But as you can see, the default value is 20 which is pretty different than 91 ... a S20 link is started before a S91 and and K91 is kill before K20.

I decided to startup nginx after mongrel, and stop nginx before mongrel. I'll use 23 for nginx.

First remove nginx symlinks:

# update-rc.d -f nginx remove

Then create symlinks for nginx with custom priorities :

# update-rc.d nginx defaults 23 23
Adding system startup for /etc/init.d/nginx ...
/etc/rc0.d/K23nginx -> ../init.d/nginx
/etc/rc1.d/K23nginx -> ../init.d/nginx
/etc/rc6.d/K23nginx -> ../init.d/nginx
/etc/rc2.d/S23nginx -> ../init.d/nginx
/etc/rc3.d/S23nginx -> ../init.d/nginx
/etc/rc4.d/S23nginx -> ../init.d/nginx
/etc/rc5.d/S23nginx -> ../init.d/nginx
Create the folder /etc/mongrel_cluster that our Rails app Mongrel config will live in.

# mkdir /etc/mongrel_cluster

Generate the Rails app Mongrel config file:

# mongrel_rails cluster::configure -e production -p 8001 -N 3 -c /path/to/rails/app -C /path/to/rails/app/config/mongrel_cluster.yml -P /path/to/rails/app/log/mongrel.pid -l /path/to/rails/app/log/mongrel.log --user <user> --group <group> -a 127.0.0.1

Verify the config file (/path/to/rails/app/config/mongrel_cluster.yml)

user: <user>
cwd: /path/to/rails/app
log_file: /path/to/rails/app/log/mongrel.log
port: "8001"
environment: production
group: <group>
address: 127.0.0.1
pid_file: /path/to/rails/app/mongrel.pid
servers: 3

Create a symlink to this file into /etc/mongrel_cluster :

ln -s /path/to/your/rails/app/mongrel_cluster.yml /etc/mongrel_cluster/YOURAPPNAME.yml

Configure nginx

We now need to edit our nginx.conf file found in /etc/nginx/nginx.conf to set it up. Edit the values found in my nginx.conf file and modify it to suit your needs.

Nginx is a powerful UNIX tool.

You have noticed we used here nginx as Reverse-Proxy in front of a Mongrel cluster; if the web-server (Mongrel in our case) cannot handle more load , you can even put nginx before the web-server to use it as web-server to handle requests to static files.

Thanks to nginx flexibility, you can pass any types of requests to web-server server by using location sections (all files, only dynamic content requests or some specific locations in your web-server tree):

It should be fine now! Reboot your server and test everything is up after startup :

# shutdown -r now
Nota : Another tool to test: Varnish, an high-performance HTTP-accelerator.

• Lack of true configuration reload support made managing our 20+ load balancers cumbersome.  We had a solution (hack) in place, but it was getting to be a pain.
• When something would break on the backend and cause 20-50k connections to pile up, the thread creation would cause huge load spikes and sometimes render the servers useless.
• As we started to push 700-1000 requests per second per load balancer, it seemed things started to slow down.  Hard to get quantitative data on this because page load times are dependent on so many things.

So...  A couple weeks ago we finished converting all our load balancers to Nginx.  We have been using Nginx for Gravatar for a few months and have been impressed by its performance, so moving WordPress.com over was the obvious next step.  Here is a graph that shows CPU usage before and after the switch.  Pretty impressive!

Before choosing nginx, we looked at HAProxy, Perlbal, and LVS. Here are some of the reasons we chose Nginx:

• Easy and flexible configuration (true config "reload" support has made my life easier)
• Can also be used as a web server, which allows us to simplify our software stack (we are not using nginx as a web server currently, but may switch at some point).
• Only software we tested which could handle 8000 (live traffic, not benchmark) requests/second on a single server

I will be working on this WP Super Cache replacement using Nginx but in the mean time here is a set up for Nginx and WP Super Cache rewrites.

First you need install 0.6.29 or up which has a module called Gzip Pre-Compression Module. Download and unzip the latest Nginx. Then set it up like this:


./configure --with-http_gzip_static_module --with-http_ssl_module
make
sudo make install


In the http area of your nginx.conf file include:


gzip_static on;
gzip_http_version 1.1;
gzip_proxied expired no-cache no-store private auth;
gzip_disable "MSIE [1-6]\.";
gzip_vary on;


Now what the above setting does is makes Nginx first see if there is a file in the directory already gzipped. If there is no gzip file Nginx will gzip the file on the fly then send the file out.

Now for the rewrites:

server {
listen 80;
server_name yourmublogs.com *.yourmublogs.com;

location ~* ^.+\.(html|jpg|jpeg|gif|png|ico|css|zip|tgz|gz|rar|bz2|doc|xls|exe|pdf|ppt|txt|tar|mid|midi|wav|bmp|rtf|js)$
{
root /home/yourmublogs/public_html;
rewrite ^/.*(/(wp-admin|wp-includes)/.*\.(html|jpg|jpeg|gif|png|ico|css|zip|tgz|gz|rar|bz2|doc|xls|exe|pdf|ppt|txt|tar|mid|midi|wav|bmp|rtf|js))$ $1 last;
rewrite ^/.*(/wp-content/(themes|plugins|mu-plugins)/.*\.(html|jpg|jpeg|gif|png|ico|css|zip|tgz|gz|rar|bz2|doc|xls|exe|pdf|ppt|txt|tar|mid|midi|wav|bmp|rtf|js))$ $1 last;
rewrite ^.*/files/(.*(html|jpg|jpeg|gif|png|ico|css|zip|tgz|gz|rar|bz2|doc|xls|exe|pdf|ppt|txt|tar|mid|midi|wav|bmp|rtf|js))$ /wp-content/blogs.php?file=$1 last;

rewrite ^.*/files/(.*(html|jpg|jpeg|gif|png|ico|css|zip|tgz|gz|rar|bz2|doc|xls|exe|pdf|ppt|txt|tar|mid|midi|wav|bmp|rtf|js))$ /wp-content/blogs.php?file=$1 last;
expires 10d;
break;
}

location / {
root /home/yourmublogs/public_html;
index index.html index.htm index.php;

if (!-e $request_filename) {
rewrite ^.+?(/wp-.*) $1 last;
rewrite ^.+?(/.*\.php)$ $1 last;
}

if ($http_cookie !~* "comment_author_|wordpress|wp-postpass_" ) {
rewrite ^(.*)$ /wp-content/cache/supercache/$http_host/$1index.html;
break;
}

error_page 404 = @tricky;
}

location @tricky {
rewrite ^ /index.php last;
rewrite ^/.*(/wp-login.php)$ $1;
rewrite ^/.*(/wp-admin/.*\.php)$ $1;
rewrite ^/.*(/wp-includes/.*\.php)$ $1;
fastcgi_pass 127.0.0.1:8084;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME /home/yourmublogs/public_html$fastcgi_script_name;
include /usr/local/nginx/conf/fastcgi_params;
}

error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}
}

server {
server_name www.yourmublogs.com;
rewrite ^/(.*) http://yourmublogs.com/$1 permanent;
}

Ajax could be used for commenting.

To configure Nginx to load balance multiple FastCgi servers use this type of configuration:

upstream servercom {
server 127.0.0.1:44440;
server 127.0.0.1:44441;
server 127.0.0.1:44442;
server 127.0.0.1:44443;
server 127.0.0.1:44444;
}

location ~ \.php$ {
fastcgi_pass servercom;
fastcgi_index stream.app;
fastcgi_param SCRIPT_FILENAME /var/www/htdocs$fastcgi_script_name;
include /etc/nginx/fastcgi.conf;
}

If you're using spawn-fcgi from lighttpd consider replacing it with php-fpm.

• Erlang - Almost everything is built atop the Erlang/OTP application system. I like functional programming. It is easier to manage complexity in a language with reasonably isolated side effects. Erlang's concurrency primitives have been touted many times before, and conceptualising concurrent systems as communicating sequential processes now feels a lot more natural than other paradigms. As Damien Katz recently mentioned, systems built atop Erlang/OTP don't seem to suffer the same failures under high load as certain other garbage collected languages do.
• Distel on Emacs - When writing code (and actually quite a bit more), I like Emacs. When writing Erlang, I like Distel. It provides a natural entry point for debugging pieces of Erlang code. Getting Emacs to communicate with an inferior Erlang VM is especially useful for rapid prototyping. Being able to do everything within Emacs is good for productivity.
• Mnesia - Arbitrary term storage is a simple way to avoid having to interface with another database management system. Because Mnesia is part of Erlang/OTP, you get it for free.
• Mochiweb and nginx - Thanks to Bob Ippolito and Matthew Dempsky (and others I'm sure I've missed) for the mochiweb HTTP toolkit. It's lightweight and embeddable, so it was very easy for us to integrate with our backend. Nginx performs well and is very simple to integrate with a backend powered by mochiweb.
• sgte and StringTemplate - Filippo Pacini's sgte template engine ("inspired by" StringTemplate) was the basis for our templating system.
• YXA - While we don't specifically deploy any SIP software, YXA is a great example of well-written Erlang. I learned a lot from reading through Magnus Ahltorp and Fredrik Thulin's code. It's been around for several years and has matured quite a bit as a result.
• Flat files - Writing a binary file to disk is sometimes overlooked as a solid way to persist sequential data that's only needed locally. I'm not sure who I should thank here, but I think the list would probably be pretty long.