Posts Tagged ‘Linux’

Converting videos for the Sony Playstation Portable PSP in Linux

Thursday, May 15th, 2008

I’ve had this script lingering around for a bit to convert almost any video file that MPlayer can read to one that can be played on a Sony Playstation Portable (PSP).

First, you have to ensure that you’ve got the latest PSP system software (I’ve got version 3.72 installed, at time of blogging, version 3.90 is the latest one).

Then, install mplayer and mencoder. If you’re using Ubuntu, this is as simple as:

$ sudo apt-get install mplayer mencoder

Finally, copy and paste the following script to a file. Lets call the file ‘convert_psp’:

#!/bin/bash 

if [ -z "$1" ]; then
    echo "Please specify movie filename"
    echo "$0 [-n] "
    echo "where -n is for 4:3 mode"
    exit 1
fi

scale=368:208
while getopts n: o
do
    case "$o" in
    n)    scale=320:240;;
    [?]) print >&2 "Usage: $0 [-n] file ..."
         exit 1;;
    esac
done

let "x=$OPTIND-2"
if [ $x -gt 0 ]; then
    shift $x
fi

src="$1"
title=`basename "$1"`
title=${title// /_}
title=${title%.*}
out=${src%.*}
out=${out}.MP4

echo Converting $src with title $title using scale $scale to output file $out

mencoder -ofps 30000/1001 -af lavcresample=24000 -vf harddup,scale=$scale -of lavf \
  -oac lavc -ovc lavc -lavcopts aglobal=1:vglobal=1:vcodec=mpeg4:acodec=libfaac  \
  -lavfopts format=psp \
  -info name="$title" "$src" -o "$out" 2> /dev/null

Do remember to chmox +x the file:

$ chmod +x convert_psp

Using it is pretty simple, just run the script and pass in the video filename:

./convert_psp <filename>

The script will generate a video file with an extension .MP4. Transfer this file to your PSP’s ‘VIDEO’ folder and you’re done!

One thing, I can’t and won’t claim all the credit for this script. It was not me who did all the work, and I remember copying the initial script from somewhere and I’ve retro fitted it for my needs.

Ext3 – handling large number of files in a directory

Saturday, May 10th, 2008

If you’ve used Linux in the past, I am pretty sure that you’ve heard of the Ext3 file system. It is one of the most common file system format out there, used mainly on Linux based systems.

I’ve noticed something really annoying about how it handles large number of files in a single directory. Essentially, I have a directory with almost a million files and I found that creating a new file in this directory took ages (in the region of tens of seconds), which is not ideal at all for my purpose.

After some reading, and much research, I learnt that Ext3 stores directory indices in a flat table, and this causes much of the headache when a directory has many files in a directory. There are a couple of options.

One, restructure the directory so that it does not contain that many files. I did some tests, and in a default (untuned) Ext3 partition, each subsequent write degrades horribly past the 2000 file limit. So, keeping the items in a directory to within 2000 files should be fine.

Second, is to enable the dir_index option on the Ext3 file system. Run the following as root and you should find that it improves a lot. Do note that the indexing will take up much more space, but then hard disk space is not too expensive nowadays:

$ sudo tune2fs -O dir_index /dev/hda1

Finally, just use something like ReiserFS which stores directory contents in a balanced tree, which is pretty darn fast and you don’t have to muck around tweaking things.

If you’ve got your main partition as an Ext3, and can’t really afford to reformat it into ReiserFS, there might be an alternative: create a blank file and format that as a ReiserFS file system and mount it using loopback.

So, lets create the file first. This depends on how much data you need to handle, and in this example, I’ll just create a ~100MB file full of zeros:

$ dd if=/dev/zero of=reiser.img bs=1k count=100000

Next, format the file using ReiserFS as below. It will complain about the file ‘reiser.img’ not being a special block device (and we know that!). Just say yes and carry on.

$ mkreiserfs -f reiser.img

Finally, mount it where you would like to read/write files into it (need to do this as root):

$ sudo mount -t reiserfs -o loop reiser.img /tmp/listdir

You might need to do some chown so that your normal user can write into it. Moreover, if you need it to startup during boot, do remember to put it in /etc/fstab !

FYI, I used a Python script below to see how long it took to write new files:

import os
import time

count = 1000000
total = 0.0
for i in range(count):
	if i % 1000 == 0:
		print 'Creating %i' % i
	start = time.time()
	open('/tmp/listdir/%s' % i, 'w').close()
	total += (time.time() - start)
print 'Avg is %0.8f' % (total / count)

MCE – A light weight media center

Tuesday, May 6th, 2008

I have an XBox (not the 360, but the really old one that is going for dirt cheap: £98.99 brand new from Amazon, and £29 from the Marketplace) that runs Linux (specifically Xebian) and it seems to work pretty nicely. Video playback is not the clearest or sharpest in the world, but it is sufficient, and I am content with the cost/quality ratio. I only use it to play videos and DVDs, and that’s about it. I don’t use it for surfing the web, email or anything more demanding. So, it makes sense to install a media center front-end like MythTV. The only thing is that MythTV is a resource hog, and only having 64MB of physical RAM it does not cope very well out of the box. However, there are some nice folks out there who have released a modified version of it that works pretty well. Now, considering that the prime (and only!) reason that I use my XBox is to watch videos, the natural component of MythTV that I would find useful is MythVideo. I have followed some instructions very closely, and always end up with the MythVideo module failing to load up due to some unresolved symbols. This is probably due to a library mismatch within one of the shared libs in MythTV. So, there I was with an XBox and I wrote a simple Java/Swing app that can invoke mplayer remotely (yes, yes, I know this is nasty!) on the XBox. Was doing this for about 1-2 years. The only problem with this approach is that I needed another PC/Laptop working to run the Java/Swing app. This was really inconvenient, and in one afternoon, wrote a simple GTK app. I call it MCE – Media Center E (I have no idea what ‘E’ stands for, just that it sounded natural when I was thinking of a name. Feel free to make something up!). The goal of the app is:

  • Light weight on resources (memory, disk, etc)
  • Simple to use
  • Minimalistic eye-candy
  • Lease amount of dependencies with other applications, libraries, packages, databases
  • Able to run on as many system as possible – Linux, BSDs, AIX, Solaris, Windows, etc

In a nutshell, it should be small and simple! So, I took an afternoon (2-3 hours) and wrote MCE. Now, it’s only in its works, but it is pretty functional as a first cut. To install it, you’ll need to install python, pygtk and any command line media player (like Xine, MPlayer, etc) If you’re using Xebian, or wish to run this on any other non-XBox machine running a Debian (Ubuntu, etc) based (or any other distro that uses dpkg and apt), all you have to run is:

$ apt-get install python python-gtk2

The latest release (version 1) is can be downloaded from: http://roopindersingh.com/releases/mce-1-release.zip There are some instructions in the readme file (under mce/1/docs). I’ll paste here for easy reading:

Prerequisite:-------------You will need the following installed:

1. Python (anything version 2.3 and above is good!)2. PyGtk (GTK2 bindings for Python)3. Mplayer (or any other command line media player)

Installation:-------------1. To install, just unzip it somewhere:$ unzip mce-version-release.zip

2. Then go into the 'bin' directory:$ cd mce//bin

3. And add execute bits to mce.sh$ chmod +x mce.sh

Configuration:--------------

All configuration is done in a file called 'mce.properties'.The file is always under the directory mce//config

The parameters that you're interested in are:media.directories    This is a comma seperated list of directories where    your media files aremedia.extensions    The file extensions that will be picked up by MCE. This    is case insensitive    media.player    The command line player that will be playing the files.    You will need to specify a single %s as that's where    the media filename will be placed ingui.fs    True - run MCE in full screen mode    False - run MCE in windowed mode (if you're debugging it, etc)

Running:--------To run it, go into the bin folder in mce//bin and run the'mce.sh' file:$ ./mce.sh

At the moment, you should be able to navigate the file system hierarchy using a mouse, keyboard or the XBox joystick and find your media files and play them. There is some gap around the list of files to allow some space for widescreen TVs. Screenshot:

Future releases will have:

  • Support for LIRC – to be able to move around with a remote control
  • A little bit of eye candy so make it look appealing – at the moment it looks like an afternoon’s hack job (well, it is!)
  • Playing DVDs
  • Display photos from files/CD/DVD

And that is all I am going to make it do at this stage. I don’t want it to be one of these all singing and dancing media center frontends that can even make a cup of coffee for you, as that usually means bloat and resource hungry. License? Well, its got the source there – so, help yourself, I put it in the public domain. If you have any questions or requests about support, help, even features, do drop me a quick note by clicking here.

Moving emails

Saturday, May 3rd, 2008

I managed to move emails and mail lists for Duzle.com and Midearth.co.uk to a faster and bigger server in about 3-4 hours.

My current setup, which I’ve had tweaked over the past couple of years, is pretty simple and straight forward:

  1. Postfix to handle SMTP including using SASL for SMTP authentication and supporting TLS/SSL
  2. Courier Maildrop as the mail delivery agent. Maildrop filters it past spamassassin and put it in the Spam folders
  3. Squirrelmail as the web frontend that uses Courier IMAP for retrieving emails

All user settings are stored in a MySQL database, and this includes the SpamAssasssin settings and bayes learned tokens.

Postfix, Courier IMAP, Mailman was an easy migration, as most of the setup is in the MySQL database.

The most tedious part of the whole process was getting maildrop to work nicely with my setup above. FYI, a handy command for testing maildrop is:

maildrop -V 4 -d email_test@test.com < /etc/motd

For some reason, the Maildrop versions supplied in Ubuntu Gutsy are not backwards compatible to Ubuntu Edgy (at least I had to debug my maildroprc to get it working correctly).

After about 3-4 hours of hacking, copying, tar-ing, untar-ing and some careful planning to avoid missing out on emails during the migration, we’re back running on faster hardware and even more disk space!

Subversion Hosting

Wednesday, March 12th, 2008

Something I’ve always wanted to do – offer Subversion hosting, and for free. Why free you may ask? Well, in my opinion, comoditised things like Subversion (or any other version control) hosting should never, ever be charged for. Version control is something that everyone should use, and while the barriers to entry are getting lower (in the sense that the client tools are getting easier to use, and the concepts of version control getting easier to understand) every year, we still find that it is only being used by:

  • People who truly believe in it
  • Folks in the software industry – let it be developers, managers, or any member of your QA team (the examples carry on, but I’ll stop here)

I probably would categorise myself at 60% on the first, and 40% on the second. Case in point – I really do put everything under version control – right from the most obvious things, like source code, to the not so obvious things like images, documents / reports and server config files. Ever since Subversion came out and offered binary file handling that actually works, I’ve been finding more and more interesting ways of using it.

Sadly, your average Joe will still be saving backups of their thesis on a thumb drive, and risk either deleting the thing, or potentially just loosing the thumb drive entirely. Or when they do end up making a mistake in the content (for e.g. rewriting a whole chapter), its just going to make them even more upset as they can’t roll it back to how it was before they rewrote it. Here is where I think version control is really useful and powerful – as long as Joe is in the habit of saving his work often. Tracking changes (comparing and reverting them) is such an easy task if the document was put under version control from the beginning.

Coming back to the hosting, I’ve offered this via XP-Dev.com, and it should be really easy to startup – just create a user account, login and start creating your repositories. Moreover it has virtually no space limits.

Give it a go, and as usual, if there’s anything you’d like to see, do just drop a note!

Task and project management

Tuesday, February 26th, 2008

I’ve always found myself in the situation where I need to keep track of the list of things to do (or plan to do), especially when it comes to developing systems and managing projects. The problem is that there aren’t any tools out there that is simple enough to use, where the process of tracking does not become a significant overhead. I really liked XPlanner, but the problem it is that it’s trying to satisfy way too many people and the process of project planning/tracking can sometimes really be an overhead, especially when it comes to managing the stories and tasks.

The way I like to think of a project is that it is defined as a set of stories, and each story is defined as a set of tasks. To track a project, and ensure that you can meet your deadlines and delivery dates, the estimates (metric should always be in no. of hours!) should really be assigned at the task level – NOT at the story level. Stories should be best described as an atomic user case in a system, and the underlying tasks should define the implementation of the story. Think of it this way – a story is a plan/proposal and the tasks actually get you there.

So, ideally for me, a project planning tool should capture these fundamentals:

  • Each project has a set of stories
  • Each story has a set of tasks
  • Each task should have an estimate no. of hours
  • Each developer adds hours spent to a task
  • Each iteration should be defined as a set of stories (from any project)
  • Managing the above should be as simple as possible, and not an overhead to the actual development of the project

The are other nice things to know: for e.g. reports that tell you how far along you are in a project/story (and whether you’ll have to tuck your tail between your legs to senior management in the weekly review). But I’ll get back to this later.

XPlanner is the closes tool out there that manages to get most things right. However, it can be a little clunky when you actually get down to managing tasks and story. I used it about a year back and maybe it has moved on. The clunky interface is actually a burden on the developer. I was working on a pretty high profile project in my bank when I was using XPlanner, and I found that the developers were spending more time surfing through the tool, trying to look for one single feature (for e.g. adding hours to a task), rather than developing. Now, this is an investment bank, where 3 developers wasting 20 minutes each a day is a big major NO NO practice. Every minute counts! So, XPlanner was not ideal for me.

Coming back to project management. Let me reiterate the traditional constraints: scope, quality, time and cost. Very important to always keep this in mind when running any project. I have found that many project managers are actually missing these fundamentals when pushing for projects to meet deadlines and not reporting back to management/clients on these. There will be times where the clients are really impatient, and won’t really understand the constraints. But generally, most of them do understand it and will decide on which of the fundamentals they would like to have delivered (Funnily enough, I’ve found that at times quality is not one of the fundamentals in Investment Banking. However time is almost always one of that clients would like delivered).

From my experience in the past, you can at most keep 2 or even 3 of the fundamentals at any one time, and your tool that you use for managing projects should relay this information to you as clearly as possible. The problem here is that I have yet to find a tool out there that can do this seamlessly.

The other issue that I noticed is that the development team that I used to manage always worked on multiple projects. No matter what period of the year it was, there was always 2 to 3 concurrent projects running for a team of 5-6 developers. This is a constraint that I have seen in many project management tools out there. They tend to defined an iteration based on a project. So, project A will have a current iteration and you can add stories A1, A2 and A3 to it and kick off the iteration. This never worked for me, at work and on my own personal projects. I needed a tool that I could arbitrarily define an iteration as a delivery target for a number of projects and systems (time constraint). In each one of these iterations, we will deliver some functionality/fixes (scope constraint) given the current resource levels (cost constraint). So, what does this really mean ? Say, I had a release cycle of every 2 weeks. Within those 2 weeks I really need to convey to all the business lines that I can deliver parts of project A and B. Within those 2 weeks, I will get out functionally defined by stories A1, A2, B1, B2 and B3. I kick off the iteration and keep track on how they are progressing. There will be ongoing risks that I will convey back to the clients. My tool will tell me how much work is outstanding, and I can then give early warnings to the business that we need to readjust our constraints. Sound simple enough ? Again, no tool.

Long and short of it – I decided to write my own project tracking tool. I’ve put it up on XP-Dev.com. Its currently only single user: i.e. if you’re in a team, then you can’t really collaborate with each other on the projects. Its a little rusty here and there, but it does what it says on the tin. At any one point, I can see where I stand on all my outstanding work and projects.

I use it extensively for my personal projects and seems to work pretty good. The one thing I really liked about XPlanner (and I’ve shamelessly copied) is the task list. At any one point I can see the tasks that are outstanding to me from all projects and keep knocking them one by one off the list. This is follows the Getting Things Done time management (Next Actions list).

Give it a go, and if you have any requests, please do contact me and I’ll ensure that it gets done.