Another milestone has been reached, and the functionality gap has been narrowing drastically the past 3 months. However, I will be brave enough to admit that the gap is still there, and at least for me, there’s still a mountain to climb ahead.

Enjoy the new releases, and as usual, your feedback is appreciated. Do give a shout in the forums, or just raise a support ticket. You can contact me via this blog as well.

Procfs in Linux is full of information, and CPU utilisation can be be calculated from the outputs of /proc/stat. Have a look at man proc for more details.

Here’s a small python app that reads /proc/stat and prints out the CPU utilisation time.

#!/usr/bin/python
import time

FNAME='/proc/stat'

def readBody():
    fp = open(FNAME, 'r')
    lines = []
    try:
        lines.extend([l.strip() for l in fp])
    finally:
        fp.close()
    return lines

def splitBody():
    lines = []
    lines.extend(l.split() for l in readBody())
    return lines

class CPUTime:
    user = 0
    nice = 0
    system = 0
    idle = 0
    total = 0

    def parse(self, line):
        self.user = long(line[1])
        self.nice = long(line[2])
        self.system = long(line[3])
        self.idle = long(line[4])
        self.total = float(self.user + self.nice + self.system + self.idle)

    def __repr__(self):
        return 'user=%s, nice=%s, sys=%s, idle=%s, total=%s' % (self.user, self.nice, self.system, self.idle, self.total)

    def usageUser(self):
        return self._doPercentage(self.user)

    def usageNice(self):
        return self._doPercentage(self.nice)

    def usageSystem(self):
        return self._doPercentage(self.system)

    def usageIdle(self):
        return self._doPercentage(self.idle)

    def delta(self, other):
        self.user -= other.user
        self.nice -= other.nice
        self.system -= other.system
        self.idle -= other.idle
        self.total -= other.total

    def copy(self):
        t = CPUTime()
        t.user = self.user
        t.nice = self.nice
        t.system = self.system
        t.idle = self.idle
        t.total = self.total
        return t

    def _doPercentage(self, a):
        return a / self.total * 100.0;

def main():
    print 'Collecting first sample'

    first = CPUTime()
    first.parse(splitBody()[0])

    while True:
        time.sleep(1)
        second = CPUTime()
        second.parse(splitBody()[0])
        secondCopy = second.copy()
        second.delta(first)
        print 'user=%s, nice=%s, sys=%s, idle=%s' % (second.usageUser(), second.usageNice(), second.usageSystem(), second.usageIdle())
        first = secondCopy

if __name__ == '__main__':
    main()

The source code can be downloaded from http://svn.xp-dev.com/svn/rs_scripts/trunk/get_cpu.py
as well.

Here’s an example run output:

rs@laptop:~/projects/scripts/pub$ ./get_cpu.py
Collecting first sample
user=2.5, nice=0.0, sys=0.5, idle=97.0
user=0.995024875622, nice=0.0, sys=1.49253731343, idle=97.5124378109
user=1.96078431373, nice=0.0, sys=0.490196078431, idle=97.5490196078
user=0.980392156863, nice=0.0, sys=0.980392156863, idle=98.0392156863
user=1.9512195122, nice=0.0, sys=0.975609756098, idle=97.0731707317
user=1.9801980198, nice=0.0, sys=0.990099009901, idle=97.0297029703
user=2.89855072464, nice=0.0, sys=1.44927536232, idle=95.652173913
user=2.42718446602, nice=0.0, sys=1.45631067961, idle=96.1165048544
user=6.34146341463, nice=0.0, sys=1.9512195122, idle=91.7073170732
user=1.9512195122, nice=0.0, sys=2.43902439024, idle=95.6097560976
user=3.98009950249, nice=0.497512437811, sys=1.49253731343, idle=94.0298507463
user=1.94174757282, nice=0.0, sys=0.970873786408, idle=97.0873786408
user=2.94117647059, nice=0.0, sys=0.490196078431, idle=96.568627451

Feel free to take it and do something useful from it. It’s in the public domain.

One of the problems is hardware resource.

The truth is: Apache is a memory hog, and to keep things scalable for serving Subversion repositories, I decided to remove all PHP websites out from apache and run them under nginx and PHP-CGI (sudo apt-get install php5-cgi). To be honest, I did not notice any difference in performance of the web sites (apache/mod_php vs nginx/fastcgi/php-cgi), however, the main motivation of this exercise is to limit the maximum amount of memory that my non-critical PHP web sites take, and at the same ti

me, giving apache more room to grow for serving the Subversion repositories. I could have had two apache installations, and give them different limits (by tweaking MaxSpare*, MaxRequests* and friends), but that’s an outright pain to manage. Moreover, I needed a simple webserver that can just serve static content as well.

And lets not forget the users of virtual private servers (VPS) with limited amount of memory. Nginx and PHP-CGI is a much appropriate solution for those memory limited configurations.

I had a look around, and it was basically down to lighttpd or nginx as a replacement to serve the PHP websites, and I picked nginx as there were some odd bugs with lighttpd serving large files. The FastCGI performance is almost the same (I did not really do any scientific benchmarks). However, the part that really got me sold on these two was that it used a master-slave threading model, rat

her than the (out of date) one thread/process per client model, which does not scale at all. Both of them are event driven, rather than “client socket” driven. BTW, this includes the awesome J2EE web container Jetty (if you use the SelectChannelConnector).

Migrating the websites across from apache to nginx/fastcgi/php-cgi was an absolute breeze and here are a few pointers that will help ease the burden.

Strategy

Just to clarify, in the apache/mod_php world, PHP files are served via the apache process itself. The strategy under nginx is to get nginx to pass on the request to another set of long running php-cgi processes that do the actual PHP processing. The response will then be passed back to nginx, which will send it back to the web browser.

Documentation

Use the English Nginx wiki extensively. There’s a lot of documentation there on configuring and tweaking nginx, especially the module reference pages. Here’s a quick and dirty howto on getting nginx+fastcgi and php-cgi working.

PHP FastCGI Start/Stop Scripts

Save yourself the trouble of writing a custom PHP FastCGI start/stop script. Install lighttpd and use their spawn-fcgi script wrapper. Its really going to save you a lot of painful hours. I wrote a simple wrapper around that script as I wanted PHP cgi to startup on every server bootup, or if I wanted a quick restart of the processes. You might rant to adjust the variables pidfile and cgidir for your setup.

#!/bin/bash

me=`whoami`
if [ $me != "root" ]; then
        echo Not root!
        exit 1
fi

pidfile=/root/php.PID
pid=`cat $pidfile`
cgidir=/var/run/php-cgi
sock=$cgidir/unix.sock

[ ! -d $cgidir ] && echo creating $cgidir && mkdir $cgidir && chown www-data.www-data $cgidir

if [ "$pid" != "" ]; then
        echo Killing $pid
        kill $pid
        rm $pidfile
        sleep 1
fi

[ -f $sock ] && chown www-data.www-data $sock

/usr/bin/spawn-fcgi -f /usr/bin/php-cgi -s $sock -C 5 -P $pidfile -u www-data -g www-data

Stop serving .htaccess

Plenty of web apps out there have built in support for apache, and include .htaccess files in their distribution to reduce the configuration overhead for the installer. However, nginx will serve these files by default, which maybe fine for most of the cases, but its always good practice to deny access to it. Simple config for nginx does the trick

location ~ /\.ht {
    deny  all;
}

Serving PHP files

To serve PHP files, nginx will pass the request to the PHP-CGI handlers.

location ~ .*\.php$ {
	fastcgi_pass   unix:/var/run/php-cgi/unix.sock;
	fastcgi_index  index.php;
	include /etc/nginx/fastcgi_params;
	fastcgi_param  SCRIPT_FILENAME  /home/rs/local/wordpress/$fastcgi_script_name;
}

Notice that I’ve included a /etc/nginx/fastcgi_params file above. This file contains all the regular FastCGI directives, and I’ve put it in a seperate file to avoid too much repetition. The content of the file /etc/nginx/fastcgi_params is below:

fastcgi_param  QUERY_STRING       $query_string;
fastcgi_param  REQUEST_METHOD     $request_method;
fastcgi_param  CONTENT_TYPE       $content_type;
fastcgi_param  CONTENT_LENGTH     $content_length;

fastcgi_param  SCRIPT_NAME        $fastcgi_script_name;
fastcgi_param  REQUEST_URI        $request_uri;
fastcgi_param  DOCUMENT_URI       $document_uri;
fastcgi_param  DOCUMENT_ROOT      $document_root;
fastcgi_param  SERVER_PROTOCOL    $server_protocol;

fastcgi_param  GATEWAY_INTERFACE  CGI/1.1;
fastcgi_param  SERVER_SOFTWARE    nginx/$nginx_version;

fastcgi_param  REMOTE_ADDR        $remote_addr;
fastcgi_param  REMOTE_PORT        $remote_port;
fastcgi_param  SERVER_ADDR        $server_addr;
fastcgi_param  SERVER_PORT        $server_port;
fastcgi_param  SERVER_NAME        $server_name;

Wordpress Rewrite

The final tip is for all those Wordpress junkies out there. To get nice urls for Wordpress, you will need the following rewrite directive. If I’m not mistaken, one will be given to you for apache when you’re setting up custom urls via the admin screen, but not for nginx:

if (!-e $request_filename) {
    rewrite ^(.+)$ /index.php?q=$1 last;
}

And that’s about it. I really do hope these tips will help someone out there. I know it would have shaved a couple hours off my setup time had I known them beforehand.

So, being really frustrated with hitting refresh so many times when browsing the web, I decided to just take the bullet and install my own BIND server locally. I’m not a big fan of running my own name server – its a pain as it eats up resource, and I really shouldn’t have to worry about resolving names from my home laptop – that’s the responsibility of the darn ISP! To get it working on Ubuntu is as simple as:

sudo apt-get install bind9

If you’re using DHCP, edit /etc/dhcp3/dhclient.conf. And ensure that there’s an entry for prepend domain-name-servers:

prepend domain-name-servers 127.0.0.1;

And thats it!

If you don’t want to run your own bind server, then you’re not out of luck just yet! You could OpenDNS and plugin their servers locally or on your router. I’ve set them up for my router as other PCs and devices use the router to resolve names.

I decided to take it easy the past couple of days, as I’ve been working on XP-Dev.com like a maniac for the past 4 weeks. Just to give this some context – I began rewriting the whole of XP-Dev.com from 16th October 2008 and released it on 17th November 2008, which is about a months worth of work. I current site has 13k lines of Java (not including unit tests) and 2k lines of HTML/Velocity templates. There’s about 1k lines of XML, but most of that is just Spring configuration (NOT their web framework – I’m only using Spring for IOC), and I had a day job to take care of. There was plenty of refactoring done during this time, and these numbers are only the final figures. All in all – a crazy month, but well worth it!

It is a complete re-write with a whole new custom MVC framework and object relational mapper (ORM) layer. I’m seriously considering releasing the MVC framwork as open source (the ORM needs some work!), which I think is pretty darn good – writing new dynamic pages and forms is a breeze! Now, I’m sure a lot of you might be wondering why I embarked on such an ambitious journey ? Well, stay tuned as I will be posting a series of blog posts on how I re-engineered and re-architected XP-Dev.com, and the reasons behind those decisions.

The new XP-Dev.com has only been released for 3 days and I did get quite a few mails from users who expressed their gratitude. I can’t begin to thank all of you for those kind emails and support tickets – it is really good to know that there are plenty of you out there who are happy users, and that the past month has been well worth it! I’ve got many more ideas to put into XP-Dev.com, so, think of the recent release as the beginning of something new

Why are you offering Subversion Hosting for free ? Is it too good to be true ?

Let me set something straight:

I offer it free because I really do not believe that anyone should pay for something so simple to setup and run as Subversion.

Here is the reality: I setup Apache using mod_svn, mod_dav, mod_ssl and mod_auth_mysql once. Believe me: only once and never ever ever ever (ever!) touched it again. No, I am not kidding – only once! No tinkering needed, it just runs like Forrest Gump (no pun intended to all you Gump fans out there).

It does cost $$$ to host it, including my time to add more features to it. Disk space and bandwidth is getting cheaper. They are not free, but then again, if you average it across the number of users that I have on XP-Dev.com, the figure looks really, really small. It is a cost nonetheless, which I’ll try to cover below.

So, we’ve established it does cost money, how are you covering these costs ? Are you really rich ?

OK. I wish I was rich, but the truth is – I am not. I could claim I was rich and lie to you all, but then I would not get any glory every time I look at my monthly bank statements.

So, where does the money come from to pay for the services ? Well, at the moment, I am paying for it. But I won’t be doing this forever.

I have got a few models to generate revenue and these models will be implemented in the next few months. I can’t reveal them to the public just yet, but rest assured that the usage of Subversion and project tracking on XP-Dev.com will always remain free. This is how I started and envisaged XP-Dev.com, and that is how it will always be.

Free Subversion Hosting and Project Tracking on XP-Dev.com is a life-time guarantee.

You’re offering a free service. There’s a catch to it, right ? Are you selling our code to someone else ?

No. Nada. No catch. I am not a petty code trader. I don’t go around knocking on other peoples doors saying “PHP codez $4 per line! .. $3.50 per line! .. $3.40 per line! ..”. I could not even be the least bothered about what everyone else is coding. I have my own ideas to push forward and materialise (one of them is XP-Dev.com, there are a lot more in the pipeline).

So, your code is safe on our servers. No one else other than the ones you have permissioned are looking at your repositories. We do have backups that run every night and copied over off-site, but they are all encrypted before leaving the server.

I put all my code on XP-Dev.com. I am a consumer of my own service. I believe that anyone who offers a service should always be their own users/clients/customers. You should see your service from the customers point of view.

If someone else looked at my code and data, I’d be really worried. I respect that tremendously and try my very best to lock down the server.

What you see is what you get – WYSIWYG. There are no catches at all. Your code and data are safe. We have a “no prying eyes” and “mind your own business” policy.

OK. So it is a genuine service that is FREE with no strings attached. Then I suppose it will have to be an overloaded, slow service ?

Never! This is one of the things that come out from being a consumer of your own service. If the services do get slow, there’s going to be one really noisy, angry, verbal user – me. And I’m really scared of him.

On a serious note, I’d be disappointed with myself if the service ever comes to a unacceptable quality. At the moment it’s fast and quick and I intend on keeping it that way. If it every becomes slow, I’ll be there in front of the queue shouting.

I’m not too sure if this is a good thing, or a bad thing – I’ve only ever worked in the Front Office for Investment Banks building real-time (well, its near real-time) trading and pricing system. They are all high performance scalable systems. The systems I work on can cost a trader anywhere between $100,000 to $500,000 if latency went up a nudge above 10ms (yes, that’s milliseconds!). XP-Dev.com is a testament of my experience building & architecting these crazy systems (trust me, they are crazy!).  If performance degrades, it will be a major failure on my part and I’m a really proud person .

It is a great service. How can I help ?

This reply is a cliche. There are a few ways you can help.

If you are not a user, register now!

If you are a user, and have any problems, queries or just want to say thank you, then please tell me, or email admin@xp-dev.com. Every single non-spam email that goes there gets a reply. If you don’t get a reply in a few hours, then it’s probably SpamAssassin acting up. You should use this form instead.

If you are a user, or not even one just yet – you can help by telling your friends, mom, dad, brothers, sisters, relatives, neighbours, cats, dogs, fish and everyone else about XP-Dev.com. Digg it, Buzz it, Reddit. Do whatever. Just keep spreading the word. I really appreciate it.

If you have any other questions or concerns, please post them as comments to this blog entry, or do contact me directly.

CodeWeavers have long been the helping hand in getting Windows applications running nicely in Linux and Mac. Now, due to the Great Lame Duck Presidential Challenge, they are giving away their products for free – just for TODAY! Get your serial number and grab a copy for yourself. I’ve grabbed my copy already.

Wikipedia, who use a mix of Red Hat and Fedora distributions for their infrastructure are moving over to Ubuntu, as part of an effort to standardise their infrastructure.

Ubuntu is very extremely popular as a desktop Linux distribution, but seem to have taken a while to penetrate the server market. If you have a look at Distrowatch.com, Ubuntu has had the top position in their Page Hit Rank since 2005, which just generally means that it is pretty darn popular!

My general take on Ubuntu is that it has always been promoted as a desktop distribution. I remember having a discussion with a former colleague about using Ubuntu as a server. He laughed out loud in disbelieve and discounted the fact at face value. It will take time to adjust to the fact that Ubuntu is actually a very viable server distribution as there’s a stigma attached to it – imagine a system administrator coming up to you with a solution to a business proposal and suggesting “Oh yes sir, we’ll be using the Windows XP”. To the average Joe, that’ll be a textbook failure in making first impressions. Ubuntu’s image is that deep in the desktop zone.

However, not many people know this fact – the current server that hosts this blog runs on Ubuntu, and as far as I can remember, Ubuntu has always had a “minimal”/”server” installation option along with their standard “desktop” option. In the not so far past, I ran a bunch of servers from home behind an ADSL line (hardly the enterprise environment!) and as soon as Ubuntu came out, I wiped out the mix of Debian/Gentoo/FreeBSD that I was running and installed Ubuntu everywhere – desktops and servers!

Why would I do such a thing ? Why would I use a desktop distribution on a server ? Why would Wikipedia use Ubuntu on their enterprise environment which process 50,000 HTTP requests during peak times ?

Well – the answer is actually very simple. Canonical took Debian, gave it a major cleanup, ease of use and (importantly) standardised the whole distribution. They basically wanted something that was easy to use and maintain. There would be a few releases a year and they will be properly supported, but only for a certain period of time. The best part is that the distribution feels like it was not standardised from a desktop point of view, i.e. a top-down approach. It really feels like they took a bottom-up approach in modifying and cleaning up Debian – which results in a great distribution that has everything consistent! It takes more work to implement the bottom-up approach, as you’re not only cleaning up Gnome or KDE, but you’re cleaning up the thousands, upon thousands of standard software packages that come with Ubuntu.

So, having Ubuntu basically means that I get ultra awesome support for security patches and bug fixes, and at the same time, I get a Debian distribution that is extremely clean, standardised and consistent. All the default config files are at the right locations. It makes it faster, easier and more intuitive to setup a package under Ubuntu as the default options for most packages are good enough.

I can see why it makes sense for Wikipedia to switch to Ubuntu – it keeps the servers very standardised, which in turn means administration costs are kept low and architecture is less prone to failure. My only hope is to actually see more adoption of Ubuntu in the data centres worldwide. So, if you’re in a team that uses Linux, or even Windows, do convince them to switch over to Ubuntu.

I’ve been playing with the idea of having a “always online” smart phone so that I can check my emails at any time, do the occasional web surfing and help sort out user issues and queries from XP-Dev.com and Duzle.com. So far, my best solution has been to grab hold of a HTC TyTN II and a sim-only deal from either O2 or Vodafone that give tons of bandwidth for internet usage. The HTC phone will set me back some £400, and the contract about £30 a month (12 months contract) (TCO: £760). An iPhone costs £45 a month on an 18 month contract (TCO: £810).

So, the HTC idea is cheaper – my only quirk: it runs Windows. 3 months down the line with the HTC, I can see myself getting frustrated with Windows Mobile. I might even try installing Linux and risk bricking the phone. However, since the Google G1 out, it just adds a different perspective to the choices that I have. Morever, its not Windows. It uses Google Andriod, which is a big bonus point in my books. Suddenly, the ideal smart phone has come about – the only thing that’s going to hold me back could be the price.

Java on FreeBSD has been working for a long time. While most things worked, it was never officially supported, as it was not a complete implementation. Not everything on the JVM worked as expected on FreeBSD. Well, that has changed (I might be late to join the party on this one!) and Java 6 is officially supported now, and the downloads include pre-built binaries.

BTW, they’ve dropped official support for the Java 5 JVM. From their Java project page:

Packages for JDK/JRE 1.5.0 are still available for download although they are not supported any longer.

I stopped using FreeBSD since I started using a multi-core desktop and FreeBSD did not have SMP support . Even that has changed, and they have come a long way from FreeBSD 5. I’ll be keeping a close watch on them. Who knows, maybe the whole of XP-Dev.com might run on FreeBSD in the future ?