Basic Verbs

Benchmark of django deployment techniques

I made a benchmark of different Django deployment techniques and configurations mostly for my personal purposes. The results are published in the hope that it would save others some time. The benchmark was not designed to test the speed of Django itself, only to give relative comparison between different techniques for running a Django application in production environments. I’m not affiliated with any of the following open source projects: Apache,  Nginx, Cherokee, mod_wsgi, mod_python, Cherrypy or uWSGI. Further, I’m not claiming to be an expert in configuring any of the software mentioned here.

What was measured

The Django project used for the benchmark was a simple application for displaying rows from 3 different tables with pagination. Each page had references to 3 static files (css, javascript and an image). Each page involved rendering a simple template inheriting from a base template and including another one, built-in filters were also used. The database had more than a million records in all three tables combined. I browsed different pages of the application over a proxy which recorded the URLs of the browsing session. So for each request that was handled by the application there were 3 more requests for static files. For each deployment technique I ran 4 tests at different concurrency levels for 1 minute making GET requests to the recorded URLs. For each test run I recorded throughput (number of requests served per second), response time (average time in which a request was served) and longest request (the longest time a request was served in each run). Only for the tests at highest concurrency levels I recorded memory usage. I tried to make sure that only the necessary processes for each test were running at a time. Automatic maintenance tasks on the system and the database were turned off. Every test cycle was repeated at least 3 times to recheck the results.

Hardware and software details

For generating the loads I used Siege and ran all the benchmarks over gigabit ethernet from a 2.16 GHz machine with 2 GB of RAM running OS X Snow Leopard.

The system that served as a web and database server ran in a VMware appliance on a 2.8 GHz Core2Duo PC with 8GBs of RAM. The appliance was given only 1GB of working memory and assigned both cores of the CPU. Software details:

• Ubuntu 9.10 32b
• Python 2.6
• PostgreSQL 8.4
• Apache 2.2.12 worker MPM
• Nginx 0.7.64
• Cherokee 0.99.39
• Django 1.1
• psycopg2 2.0.8

Tested configurations

Apache with mod_wsgi

This was the first configuration I tested. Apache was serving both the static files and dynamic content via mod_wsgi which ran in daemon mode with 5 processes and 1 thread per process.

Nginx + Apache with mod_wsgi

My preferred configuration for running django sites. Apache with mod_wsgi was used only for the dynamic content, requests to these urls were proxied by nginx. Static files were served by nginx directly.

Nginx + fcgi

Here nginx is used for serving the static content while the dynamic content was handled by FastCGI processes. I used all the defaults from the runfcgi management command and used a TCP socket instead of a socket file because I was bumping into issues with access to the socket file at large numbers of concurrent requests.

Cherokee + SCGI

This was set up from the Cherokee web based wizard for deploying django applications. Static files were served by Cherokee directly. I have to say this is by far the  easiest method of deploying Django applications in production environments.

Cherokee + Apache with mod_wsgi

This is essentially the same as Nginx + Apache and mod_wsgi except here Cherokee was used as a proxy and for serving static content. Since it’s the first time I’m using Cherokee everything was configured via the web based admin interface and all parameters were left to default values.

Nginx + Cherrypy WSGI server

Here I used the django-cpserver management command to run the application in the Cherrypy WSGI server. 5 Instances of the WSGI server were running behind Nginx as a load balancer. Nginx was serving the static files.

Configurations that are left out

Apache with mod_python

This was left out because I couldn’t get consistent results at 250 concurrent requests and the application would often error out at this concurrency level. The benchmark already took a significant amount of my time and I’m not experienced with mod_python so I decided not to proceed with locating the problem or publishing any shaky results.

uWSGI

I tried to deploy the application on uWSGI with the Cherokee web based wizard. With the default configuration (1 process) the tests ran 4 times slower at high concurrency compared to the other configurations. Bumping up the number of process to 5 still didn’t yield comparable results. At 25 processes I got comparable results but memory usage skyrocketed. I still suspect I was doing something wrong here, so I didn’t publish the results.

lighttpd

Personally I avoid using lighty for a number of subjective reasons. When I have the time I may update this post with some configurations based on lighty, although I wouldn’t expect the results to be much different than the ones for the Nginx or Cherokee configurations.

The results

Memory usage

I measured memory usage only at 250 concurrent requests. All the tests ran for 60 seconds, so you can work out where the load on the server happened from the graphs.

Apache with mod_wsgi

Nginx + Apache with mod_wsgi

Nginx + FCGI

Nginx + Cherrypy

Cherokee + SCGI

Cherokee + Apache with mod_wsgi

DOM Carousel

In one project of ours, we needed a simple and light DOM carousel that will fallback on list when javascript is turned off. By the term “DOM carousel” i mean a carousel that will show/hide/slide DOM elements, complete container (div, p, h) not just images or special types of content. This domCarousel takes the containers one by one and slides them one after another.

Works in a simple manner. Just attach it on the parent container of the elements that will slide, fill in the options object (or use the default values) and you are good to go.

$.fn.domCarousel = function(options){
        // keeping a reference to the carousel object
	var container = this;

        //setting the class of the items that will rotate or default
	var itemClass = (options.itemClass)?(options.itemClass):("dc-item");

        //interval length of transition
	var ti = (options.transitionInterval)?(options.transitionInterval):(10000);

        //label for the start button or default
	var startLabel = (options.startLabel)?(options.startLabel):("Start");

        //label for the stop button or default
	var stopLabel = (options.stopLabel)?(options.stopLabel):("Stop");

        //label for the back button or default
	var backLabel = (options.backLabel)?(options.backLabel):("Back");

        //label for the forward button or default
	var forwardLabel = (options.forwardLabel)?(options.forwardLabel):("forward");

        //display buttons true or false
	var displayButtons = (options.displayButtons)?(options.displayButtons):(true);

       //attach a class to the buttons
	var buttonClass = (options.buttonClass)?(options.buttonClass):('button');

	//initializing and help vars.
	var intervalID = ""; //id of the interval object so we can control it
	var mutex = false; //mutex so the transitions wont overlap
	var min_height = 0; 

        //hide all the elements
	$(container).find("."+itemClass+":gt(0)").hide();
	//make them position relative and add some style to them
        //(you can change this so it suits you better)
	$(container).css("position","relative");
	$(container).css("margin-top","20px");
	$(container).find("."+itemClass).css("margin","0px");

	var currentIndex = 0; //this is the currently active em

       //This is where the magic happens. Using current item reference
 	//and the mutex, we animate blocks one by one. See why mutex
       // is important
	container.changeItem = function(){
		if(!mutex){
			mutex=true;
			$(container).find("."+itemClass+":eq("+currentIndex+")")
                                .hide("slide",{direction: 'left'},300,function(){

			nextToShow = (currentIndex == $(container)
                                       .find("."+itemClass).length-1)
                               ? (0):(currentIndex+1);				

				$(container).find("."+itemClass+":eq("+nextToShow+")")
                                .show("slide",{direction: 'right'},300,function(){
					currentIndex = nextToShow;
					mutex=false;
				});
			});
		}
	}

	// same function from above but for back
	container.changeBack = function(){
		if (!mutex) {
			mutex = true;
			$(container).find("." + itemClass +
                                  ":eq(" + currentIndex + ")").hide("slide", {
				direction: 'right'
			}, 300, function(){
			nextToShow = (currentIndex == 0) ?
                        ($(container).find("." + itemClass).length - 1) : (currentIndex - 1);

			$(container).find("." + itemClass + ":eq(" + nextToShow + ")")
                        .show("slide", {
					direction: 'left'
				}, 300, function(){
					currentIndex = nextToShow;
					mutex = false;
				});
			});
		}
	}

	//Initialize the interval and animate
	container.play = function(){

		if(!intervalID){
			$(container).find("input.dc_button_ss").val(stopLabel);
			$(container).find("input.dc_button_ss").click(function(){
				container.stop();
			});
			intervalID = setInterval(function(){
				container.changeItem();
			},ti);
		}
	}

	//find the interval and remove it
	container.stop = function(){
		if(intervalID){
			$(container).find("input.dc_button_ss").val(startLabel);
			$(container).find("input.dc_button_ss").click(function(){
				container.play();
			});
			clearInterval(intervalID);
			intervalID = null;
		}
	}

	//Should we display buttons or not.
	if(displayButtons){
		$(container).append('<span style="position: absolute; right: 5px; top: -28px;">
               <input class="dc_button_back '+buttonClass+'" type="button" value="'+backLabel+'" /> 
               <input class="dc_button_ss '+buttonClass+'" type="button" value="'+stopLabel+'" /> 
               <input class="dc_button_forward '+buttonClass+'" type="button" value="'+forwardLabel+'" />');

	        //init button click events
		$(container).find("input.dc_button_back").click(function(){
			container.stop();
			container.changeBack();
		});

		$(container).find("input.dc_button_forward").click(function(){
			container.stop();
		        container.changeItem();
		});
	}
        //start the carousel
	container.play();
}

The plugin is used same as all other jquery plugins : $(selector).domCarousel(options). Options is an object that has these attributes:
itemClass : css class of the transition items,
transitionInterval : time in milliseconds of transition,
startLabel : start button label,
stopLabel : stop button label,
backtLabel : back button label,
forwardLabel : forward button label,
displayButtons : will we display buttons or not,
buttonClass : css class of the buttons

To use this plugin, you will need the jQuery library, the jQuery-UI library and the plugin itself. Live example of the plugin, you can see here the fourth box from top.

Enjoy!

Announcing django-audit-log

For those unfamiliar with the term, a definition from Wikipedia:

Audit trail or audit log is a chronological sequence of audit records, each of which contains evidence directly pertaining to and resulting from the execution of a business process or system function.

How does this come into play in a web application?

Lets examine the case of a simple application for keeping track of a store’s inventory:

In the most simple case there would be a single database table in which we’d keep details on different products in the store. In the case where multiple users would have access to INSERT/UPDATE/DELETE records in the products table, one user could insert a product with name, description and price, later another user could change the description or even delete the whole record. If at some later point we wanted to restore the original record or see who made the latest changes we’d have to ask all the users to remember what they did. An audit log for this table would provide the means of keeping track of all the changes that were made to it and who made the changes in a chronological order.

django-audit-log provides such facilities for your Django models.

It’s designed to be very simple to add chronological tracking to any django model with the least amount of changes to your existing code. Adding an audit log for your models is done in three steps:

1. Add a middleware class in settings.py.
2. Add a manager property to every model you need to keep track of.
3. Execute the syncdb management command.

To keep track of all the changes a separate table will be created for each tracked model. This table would have the same column structure as the original model plus columns for tracking the time, type of action (create, change or delete) and user who did the action. Queries on the audit log for a model are made via the manager added in step 2.

The project is still under heavy development and there’s no official release yet. Keep that in mind if you consider using it in production. The code can be downloaded from the mercurial repository:

hg clone https://django-audit-log.googlecode.com/hg/ django-audit-log

Basic usage instructions can be found on this wiki page.  Feature requests are always welcome.

Details on how it works and extension points will be coming up on the project wiki page soon.

/objects – Ton of Dev FREEBIES

Ton of random snippets of code I frequently whip out during design. Particularly the fixed footer. The single-level dropdown is pretty custom/sleek too.

Anyway, they are all pretty bare-bones, so you should be able to figure out how to implement them from source.

http://basicverbs.com/objects

Feel free to check back as I add something new almost every week.

Livescape – a jQuery live landscape

Few days ago i was bored to death and decided to do something to pass time – write a jquery plugin. Didn’t have any idea what to do, so Cory said “Lets do something that will give life and movement to the designs without using flash”. I started on LiveScape – the live landscape.

The idea behind this plugin is to have somekind of landscape container that will contain objects moving inside of it e.g. sky with the clouds moving. The plugin is built simple, has simple implementation, it is very easy to use, yet very flexible. You only need the landscape image, the object images and one container.

Lets get technical: You have a div #livescape. This is how the plugin in used.

$('#livescape').livescape({

    height : '360px',  //height of the landscape (can be omitted)
    width : '1000px', //width of the landscape (can be omitted)
    background_image : 'img/bg.jpg', //background image (can be omitted)

    // Next define the objects that are moving:
    objects : [
        {
            //the positions are relative to the landscape
            //if omitted, random assumed - same for duration
            start_x : "1700px", //x of start position
            start_y : "190px", //y of start position
            end_x : "-4000px", //x of end position
            end_y : "190px", //y of end position
            image : 'img/car.png', //image of the object (url)
            duration : 18000, // duration of the animation
            loop : true, // the animation loops
            href : "ThisisBatCountry.png", //the object is anchor
            pause: 1000, //pause in between loops in ms
            fade : true //the objects fade in on start and out on end
        },
        //another object
        {
            start_x : "-250px",
            start_y : "5px",
            end_x : "1250px",
            end_y : "5px",
            duration : 49000,
            image : 'img/cloud1.png',
            loop : true,
            fade : true
        }
    ]
})

This is setup is for 2 objects. The implementation is pretty intuitive and i think you will get a hold of it very fast.

The plugin is still beta. Needs refactoring. Time of development was around 30 min. Nevertheless you will have access to an early access version.

Anyway [DOWNLOAD] the plugin and enjoy the “Fear and Loathing in Las Vegas” example:

Portfolio Press by Blog Oh! Blog