Archive for the ‘Programming’ Category

« Older Entries
Newer Entries »

More on XP-Dev.com

Wednesday, November 5th, 2008

Doug Bromley has made a comparison list of free Subversion Hosting providers on his blog, and XP-Dev.com is listed there (thanks Doug)! If you’re just skimming it, then it might look as if XP-Dev.com is a little “thin” on the features end of it. The odd thing I found was the “Tracking” portion had XP-Dev.com listed as a “NO”, which is not entirely true. Having said that, the project tracking portion is not as polished as I want it to be. There are more features in the works, and one of the main ones that most users have asked for is the Project tracking part to be multi-user.

I will keep everyone posted on the progress as soon as I can confirm dates, but it is really really close.

On a side note, I’m going to start a blog for XP-Dev.com at blog.xp-dev.com and wish to invite anyone out there who is enthusiastic about agile programming and software development (and delivery!) in general and can contribute some time to blog on best practices and other fun stuff, do drop me a note and we can discuss specifics!

Tags: , ,
Posted in Programming | No Comments »

IntelliJ 8

Sunday, November 2nd, 2008

I have blogged in the past on a very sensitive issue – Eclipse vs IntelliJ. I come from the IntelliJ camp where I strongly feel that IntelliJ really improves developer efficiency tremendously. It might be a little opinionated, but here’s the drill: I have been working at my current job (no, its not a career, but that is content for another blog) for roughly 8 months, and I can only use Eclipse, for a multitude of reasons. So, I might be able to claim that I do use Eclipse and IntelliJ a lot.

Here’s what I found when comparing Eclipse to IntelliJ: Eclipse has way too much clutter. It does not refactor right. It is really slow in auto-completion. It just does not offer the right suggestions during auto-completion. The list goes on. It could be that I’m facing these issues because I’m using Eclipse 3.3. I really should upgrade to Eclipse 3.4 and hopefully some of these might go away.

My IDE has to be really fast, and has to be smart enough to give me the right auto-completion. My IDE has to be able to help me develop real quick. Eclipse falls short – way short. I tend to develop much faster (about 2x productivity) thanks to IntelliJ’s pretty smart auto-completion, and somehow the code comes out cleaner, as I don’t have to worry about the IDE screwing up big refactors. I end up spending less time typing, and more time refactoring.

However, the one thing that I always liked about Eclipse is that it supported various other languages and IntelliJ fell short in the category. Well, not for long – IntelliJ is catching up and now with version 8 – there’s a ton of features added with new language support. This list is just the tip of the iceberg on the new features.

Grab your evaluation copy, and try it out. Tips to using IntelliJ properly:

  • Shortcuts, shortcuts, shortcuts! Learn all the keyboard shortcuts. Use the reference card (Windows + Linux, Mac) and memorise it.
  • Stop touching that mouse! Every single time you move one hand from the keyboard to the mouse you have just wasted a few seconds of good coding time. Break this habit.
  • Learn to type fast, without looking at your keyboard.
  • Get in the habit of refactoring. There maybe once, or twice in my 3 years of using IntelliJ that it failed on my when refactoring, and I use that bloody rename shortcut like it is going out of fasion. The more often you refactor, the cleaner your code will become. It will be more maintainable as well.
  • When you’re editing something that is not exactly code, like XML for instance, don’t use notepad or some old crummy text editor! Chuck it into IntelliJ and you might get a suprise that it does auto-completion for you.

Some useful sites to help you get your head around IntelliJ:

Tags: , , ,
Posted in Programming | 5 Comments »

Free Subversion Hosting

Saturday, November 1st, 2008

Many people over the past few months, have been asking the same questions over and over again about the services over at XP-Dev.com. I don’t mind answering them with the same answers, but I think it is time to put all of these questions into one place and discuss them.

Why are you offering Subversion Hosting for free ? Is it too good to be true ?

Let me set something straight:

I offer it free because I really do not believe that anyone should pay for something so simple to setup and run as Subversion.

Here is the reality: I setup Apache using mod_svn, mod_dav, mod_ssl and mod_auth_mysql once. Believe me: only once and never ever ever ever (ever!) touched it again. No, I am not kidding – only once! No tinkering needed, it just runs like Forrest Gump (no pun intended to all you Gump fans out there).

It does cost $$$ to host it, including my time to add more features to it. Disk space and bandwidth is getting cheaper. They are not free, but then again, if you average it across the number of users that I have on XP-Dev.com, the figure looks really, really small. It is a cost nonetheless, which I’ll try to cover below.

So, we’ve established it does cost money, how are you covering these costs ? Are you really rich ?

OK. I wish I was rich, but the truth is – I am not. I could claim I was rich and lie to you all, but then I would not get any glory every time I look at my monthly bank statements.

So, where does the money come from to pay for the services ? Well, at the moment, I am paying for it. But I won’t be doing this forever.

I have got a few models to generate revenue and these models will be implemented in the next few months. I can’t reveal them to the public just yet, but rest assured that the usage of Subversion and project tracking on XP-Dev.com will always remain free. This is how I started and envisaged XP-Dev.com, and that is how it will always be.

Free Subversion Hosting and Project Tracking on XP-Dev.com is a life-time guarantee.

You’re offering a free service. There’s a catch to it, right ? Are you selling our code to someone else ?

No. Nada. No catch. I am not a petty code trader. I don’t go around knocking on other peoples doors saying “PHP codez $4 per line! .. $3.50 per line! .. $3.40 per line! ..”. I could not even be the least bothered about what everyone else is coding. I have my own ideas to push forward and materialise (one of them is XP-Dev.com, there are a lot more in the pipeline).

So, your code is safe on our servers. No one else other than the ones you have permissioned are looking at your repositories. We do have backups that run every night and copied over off-site, but they are all encrypted before leaving the server.

I put all my code on XP-Dev.com. I am a consumer of my own service. I believe that anyone who offers a service should always be their own users/clients/customers. You should see your service from the customers point of view.

If someone else looked at my code and data, I’d be really worried. I respect that tremendously and try my very best to lock down the server.

What you see is what you get – WYSIWYG. There are no catches at all. Your code and data are safe. We have a “no prying eyes” and “mind your own business” policy.

OK. So it is a genuine service that is FREE with no strings attached. Then I suppose it will have to be an overloaded, slow service ?

Never! This is one of the things that come out from being a consumer of your own service. If the services do get slow, there’s going to be one really noisy, angry, verbal user – me. And I’m really scared of him.

On a serious note, I’d be disappointed with myself if the service ever comes to a unacceptable quality. At the moment it’s fast and quick and I intend on keeping it that way. If it every becomes slow, I’ll be there in front of the queue shouting.

I’m not too sure if this is a good thing, or a bad thing – I’ve only ever worked in the Front Office for Investment Banks building real-time (well, its near real-time) trading and pricing system. They are all high performance scalable systems. The systems I work on can cost a trader anywhere between $100,000 to $500,000 if latency went up a nudge above 10ms (yes, that’s milliseconds!). XP-Dev.com is a testament of my experience building & architecting these crazy systems (trust me, they are crazy!).  If performance degrades, it will be a major failure on my part and I’m a really proud person :) .

It is a great service. How can I help ?

This reply is a cliche. There are a few ways you can help.

If you are not a user, register now!

If you are a user, and have any problems, queries or just want to say thank you, then please tell me, or email admin@xp-dev.com. Every single non-spam email that goes there gets a reply. If you don’t get a reply in a few hours, then it’s probably SpamAssassin acting up. You should use this form instead.

If you are a user, or not even one just yet – you can help by telling your friends, mom, dad, brothers, sisters, relatives, neighbours, cats, dogs, fish and everyone else about XP-Dev.com. Digg it, Buzz it, Reddit. Do whatever. Just keep spreading the word. I really appreciate it.

If you have any other questions or concerns, please post them as comments to this blog entry, or do contact me directly.

Tags: , , , ,
Posted in Linux, Programming | 78 Comments »

Hibernate, MySQL and Case-Sensitivity

Wednesday, October 29th, 2008

Since going live with XP-Dev.com a few months ago, there has been a nasty little bug that I’ve been trying to chase down where a weird exception is thrown in some obscure situations. When I say “weird”, I mean “I look at the code and keep saying to myself: this exception can’t possibly be thrown here”. Now, the exception is very domain specific and its almost pointless for me to attempt describing it here, as that will take a few posts, rather than just the one.

In a nutshell, a user could login to XP-Dev.com using valid username and password credentials, but then found that he/she could not perform any tasks. It turns out that this only happens if a user logs in using a slightly different case for his/her username. For e.g. When user ‘developer‘ logs in using ‘Developer‘, he’ll basically hit this bug.

The problem is two fold:

First, MySQL stores varchar columns as case insensitive. So, when you run a SQL query like:

select * from Users where username='Developer'

MySQL will return the rows with username ‘developer’.

This is not a problem per-se, as I’ve found this design feature pretty useful when attempting to do web oriented work in the past. The web in general is case insensitive, and system facing the web should reflect that. There is no difference between XP-Dev.com and xp-dev.com.

So, from this perspective, its not really a security bug as well, as you can’t really register the user ‘Developer’ anyway and attemp to ’steal’ the other user.

Here comes the annoying second part of this fiasco.

I use Hibernate as a Object Relationship Mapper/Persistence Layer on the current version of XP-Dev.com. Carrying on with the MySQL example above, pulling out a User object from the Users table using Hibernate can be done as follows:

User user = (User) org.hibernate.Session.get(User.class,
                                             "Developer")

The problem is that the User object returned will have the member ‘username’ set to ‘Developer’, rather than ‘developer’ as it is on the database. The object that Hibernate returns to you does not reflect the database state at all.

Coming back to the bug: Whenever a user logs in, I keep his username in memory attached to his session on the server-side, and refetch a fresh new User object on every HTTP request. The problem was that performing any tasks failed because it relied on strong Foreign Keys on the database tables. For e.g. if user ‘developer’ had 3 repositories, and logs in using ‘Developer’, I will refetch ‘Developer’ using Hibernate and get the username ‘Developer’ back, rather than ‘developer’. The user ‘Developer’ does not have any repositories and the permissions layer that I wrote (which is case sensitive) will throw a nasty exception when user ‘Developer’ tries to access repositories for the user ‘developer’. The permissions layer should be case sensitive as it’s a very critical part of the architecture and should be as restrictive as possible.

It’s nasty the way Hibernate does not return the exactly values back on the data object as it is on the database. The work-around is to use something like below:

org.hibernate.Session.createQuery("from User where username = ?")
                             .setParameter(0, "Developer")
                             .list()

Magically, Hibernate DOES return a User object with username set to ‘developer’. This approach has an extra overhead of creating a list on every call, but I suppose I’ll have to live with that for now.

On the new version of XP-Dev.com, I got rid of Hibernate and Wicket. I’ve had growing pains with Wicket as well, but that’s a rant for another blog post. Hibernate is just too bloated, and I can’t afford the performance overhead.

Tags: , ,
Posted in Programming | 1 Comment »

Python and Multi-threading

Tuesday, October 7th, 2008

It has been a few days since Python 2.6 has been out, and the word on the street is that it’s meant to ease the transition into Python 3k. Python3k is not backwards compatible to the 2.X releases. I haven’t had much time on my hands to get down and dirty with the new 2.6 release, but have had some time to read up on it.

Most people know that Python does have a threading API that is pretty darn close to Java’s. However, the way it has been implemented is that all threads need to grab hold of the Global Interpreter Lock to ensure that only one thread at any one time can execute within the Python VM. This is to ensure that all threads have the same “view” of all variables. Apparently they tried to avoid this by making the Python VM thread safe, but it did get a terrible performance hit.

Java tends to get around this by having a rather complex memory model within the Java VM where each thread has it’s own virtual memory. That’s why you have to synchronize various sections of your code to ensure that threads see the same variable states. I highly recommend reading up Doug Lea’s article on synchronization and the Java Memory Model for anyone who wants to do very intensive multi-threaded applications in Java.

So, what are the implications of having to grab hold of the Global Interpreter Lock in Python ? The problem is that it is not TRUE multi threading. You, as the programmer and designer (you DO design your solutions first, right?), will have to  plans on when threads should just go to sleep and allow other threads to run. The VM will not do this for you, and one might say that it really is closer to a single threaded VM. From past experience, I’ve found Python’s Threads to be really useful when I’m making blocking calls (for e.g. grabbing a DB connection, blocking APIs (yuck!)), and can do something else in the background while the main thread is sleeping. You could get around this problem by using sub-processes, but there was no easy way to do it, and you had to add a lot of boiler plate code every single time. There was just no support for a clean true multi-threaded interface out of a standard installation.

Now, in Python 2.6, there’s a new package for creating sub-processes called multiprocessing. After a quick glance, it looks very similar to the threading API, BUT instead of running threads, it creates a child process which has it’s own memory and in turn does not need to share it’s Global Interpreter Lock. My own prediction is that it comes at a cost of creating a new process and memory space efficiency. However, you do end up with a TRUE multi-threaded application that really uses all the available processor cores on a multi-core CPU. Considering that RAM is getting cheaper, and processors getting more cores built into them, I think this is a fair trade off.

As always, and this applies to Java as well – writing a true multi-threaded application is not trivial, and always do your homework before you get started! In the past, I always had to fallback to Java for the more intensive applications that I wrote because I always thought creating sub-processes in Python was too tedious. From now on, I have no excuses! The new package in Python 2.6 looks very neat and removes the need to write tons of boiler plate.

Python

Tags:
Posted in Programming | No Comments »

Marsenne Prime and Python

Saturday, September 6th, 2008

A couple of weeks ago, I read the Slashdot entry on the discovery 45th Marsenne Prime. In a nutshell, a Marsenne Prime is a prime number that is calculated via: Mn = 2n – 1

This got me thinking – how does one calculate such a large number (the 44th prime has 9.8 million digits). The obvious problem in calculating this number would be the fact that the largest number most compilers and languages support would be an unsigned long long, which is not even close enough to hold such a large number. At this point I recalled reading somewhere that Python has support for handling numbers of any size. I knocked up the following script and ran it on my dev “box”:

#!/usr/bin/env python
import sys

if len(sys.argv) != 2:
	print 'Need power'
	sys.exit(1)

exp = long(sys.argv[1])
print 'Exponent %s' % exp
ans = pow(2, exp)

fp = open('out.write', 'w')
try:
	fp.write(str(ans))
finally:
	fp.close()

print 'Done'

All it does is grab the exponent from a command line parameter and writes the output to a file called out.write. I only cared in calculating the exponent part (2n) as the subtraction is trivial. The processor is a 1GHz Transmeta Crusoe with 256MB RAM. Not exactly the fastest CPU on the block, but it gets the job done as my development machine. CPU details below:

processor	: 0
vendor_id	: GenuineTMx86
cpu family	: 6
model		: 4
model name	: Transmeta(tm) Crusoe(tm) Processor TM5800
stepping	: 3
cpu MHz		: 993.322
cache size	: 512 KB
fdiv_bug	: no
hlt_bug		: no
f00f_bug	: no
coma_bug	: no
fpu		: yes
fpu_exception	: yes
cpuid level	: 1
wp		: yes
flags		: fpu vme de pse tsc msr cx8 sep cmov mmx longrun lrti constant_tsc up
bogomips	: 2026.55
clflush size	: 32

In total, it took 4020 minutes (2 days 19 hours) to calculate the 44th Marsene prime on this weakling machine:

rs@small:~/apps$ time ./dopow.py 32582657
Exponent 32582657
Done

real    4020m29.975s
user    3836m26.106s
sys     5m49.542s

And the number has 9,808,358 digits!

rs@laptop:~/mounts/petite/apps$ wc out.write
      0       1 9808358 out.write

Have a look at the file (10MB RAW), or grab the compressed version (5MB ZIP).

I am impressed by Python’s ability to handle such large numbers. Kudos to Guido and his team of brainiacs for coming up with this feature.

Tags: ,
Posted in Programming | No Comments »

Java 6 on FreeBSD, anyone?

Thursday, August 28th, 2008

Beastie

Java on FreeBSD has been working for a long time. While most things worked, it was never officially supported, as it was not a complete implementation. Not everything on the JVM worked as expected on FreeBSD. Well, that has changed (I might be late to join the party on this one!) and Java 6 is officially supported now, and the downloads include pre-built binaries.

BTW, they’ve dropped official support for the Java 5 JVM. From their Java project page:

Packages for JDK/JRE 1.5.0 are still available for download although they are not supported any longer.

I stopped using FreeBSD since I started using a multi-core desktop and FreeBSD did not have SMP support . Even that has changed, and they have come a long way from FreeBSD 5. I’ll be keeping a close watch on them. Who knows, maybe the whole of XP-Dev.com might run on FreeBSD in the future ?

Tags: , ,
Posted in Linux, Programming | No Comments »

Software Metrics

Friday, July 4th, 2008

I’ve been reading Bruce F. Webster’s articles and blogs lately. He’s written an article on project management metrics (more directed on IT and software development) on BaselineMag.com. In his article, he quotes The Metric Law of 90s:

The first 90 percent of a development project
takes 90 percent of the schedule.
The remaining 10 percent of the project
takes the other 90 percent of the schedule.

I have to agree with this! When I first started working (not 3 years ago!), a former colleague once told me (tounge-in-cheek): If you ever have to put a date on a delivery for the business, think in your head how long you think it will take, and then multiple that estimate by 3.

Tags: ,
Posted in Management, Programming | No Comments »

Firefox 3 Released

Wednesday, June 18th, 2008

Firefox 3 is out! Grab it from http://www.mozilla.com/firefox/.

Firefox

One thing that I noticed is that at times the site will take a long time to respond. I suppose all the buzz about Firefox is really hitting their servers hard.

Initial impressions: much faster, much more light-weight than Firefox 2. I would definately recommend anyone with a lower end machine to upgrade. Downside: finding all my favourite Firefox extensions and plugins that are compatible with Firefox 3. The search begins!

Tags: , ,
Posted in Linux, Programming | No Comments »

Applications and Software

Monday, May 26th, 2008

I’ve added an application and software section to this blog, and plan to add all the little bits of apps and scripts that I’ve been writing over the past couple of years. All of them will be open source and free.

Tags:
Posted in Programming | No Comments »

« Older Entries
Newer Entries »