Planet SCM

Over the years, I have tried many different approaches to software estimation.  From the Ceiling and Weller method (look up to the ceiling, scratch your chin and say "Well-errr"), FITA analysis (Finger In The Air) through various variations (both formal and informal) of metric based estimation techniques to things like IFPUG etc.

Today, I had something happen that reminded me why software estimation is so hard. I've just wasted about 8 hours trying to figure out an issue with what I thought was some bizarre firewall problem with Windows Vista SP1 x64 when it turns out that some code that I wrote was actually working correctly and was picking up a HTTP proxy preference that was pointing to a server that no longer exists.  I'd just forgotten I'd fixed this bug, and that I had the proxy set in my preferences.

I took two lessons from this frustrating day.

1. We need a better error message when your proxy is no-longer available
2. Software estimation is hard

In discussions with my wife she frequently struggles when some nights I finish work complaining that I am 3 days behind and the very next afternoon I can be caught up or even a little ahead.

The best way I have come up with to explain this, is to get her to imagine that I had given her 60 Suduko puzzles all rated as 10 minute puzzles.  Should take you 10 hours right?  Now see how long it takes you to do each puzzle.

Suduko is the closest analogy to computer programming that I can find for "normals" - i.e. people that don't code.  This only really works if the person does Suduko puzzles, but my wife does so it works in our house.  Suduko is a numeric analytical problem solving activity.  While there are tricks and techniques to solving some puzzles, there is a significant challenge and difference to each one. Looking at the puzzle, it is hard to know if it is going to be hard or easy.  You can get stuck down blind alleys and have to start all over. Also when you "get into the zone" you can often make surprising intuitive leaps that often defy verbal explanation afterwards. Finally, solving a hard Suduko puzzle quickly involves a fair degree of luck and depends on your state of mind at the time of trying the puzzle.  There is a great amount of satisfaction to be gained from solving a Suduko puzzle along with a high degree of frustration when you cannot solve one - you know that it must be possible after all.

So.  Each task (solving a Suduko puzzle) should take about 10 minutes, with-in a certain error range.  In software estimation a really good developer doing a well known and well defined task can only hope to get their estimate to around 25% accuracy (a "hard estimate") - and then there are always the odd random occurrences that throw you way off.

I've been in interviews where the candidate swears blind that they always finish a task on time.  This tells me two things about that candidate:

1. They are a liar
2. They either never estimate how long something is going to take up-front or they over-subscribe to the Scotty principle of estimation. (Or as Microsoft are fond of saying, under promise - over deliver)

Does this mean that we should give up on estimation?  Of course not.  Planning a project is going to be pretty hard if you have no idea of roughly when you are going to be finished, what the end result is going to do or how much it is going to cost.  Not to forget that when estimating a large number of tasks you can rely on the fact that some of the work you finish early will offset the work that takes longer than expected.  However, understanding the nature of software development and how it differs from, say, laying bricks, helps you be more likely to succeed.

The problems inherent in software estimation also help me to understand why Agile software development methodologies work. 

• You are forced to break problems down into small bits that can be managed, tracked and measured.
• Small iterations means that you can only get so far behind.
• You re-estimate and re-prioritise work at each iteration when a hard estimate on that task is possible (and therefore your estimates are more likely to succeed)
• You frequently listen to the person who will be using the software
• You can load your iterations so that there is always work to do of the correct priority to the customer when you have finished a task early and things of the appropriate priority that you can drop off the list when something takes longer

There are also many other techniques that people can adopt from Agile methodologies or just from common sense to reduce the amount of time taken when a task is taking longer than expected, such as

• Limit distractions. You need time to concentrate on a single problem otherwise you will never finish it.
• Pair programming. Identify when a second pair of eyes is needed and quickly as for help in your team (can be hard when your team are distributed across multiple time-zones)
• Daily progress meetings. The guy that has been stuck for 24 hours on what he thinks is a firewall issue is quickly identified)
• Lack of ego.  I'm the "Windows guy" in our company, if I say something is a firewall issue with Windows Vista SP1 x64 and a probably collision with the new Eclipse 3.4 launcher executable - then the new intern who only knows about the Mac he used at college should feel like he can ask a "dumb" question and say "Have you checked your proxy settings after that bug you were fixing yesterday".

Anyway.  All common sense stuff and writing this post has been a nice way to get rid of the frustration of finding why I was stupidly stuck for the past day.  Now to stop distracting myself further and get on with some more work :-)

"Flow" is a the most productive state of programmer's mind. You can induce the flow by starting your day with writing unit tests.

All programmers know this state of "flow", when your productivity is at its highest. The design decisions are coming naturally, the written code is perfect and a week's work can be done in a couple of hours. And it feels really good. I think it's just the sense of creative fulfillment. Some call it creative Zen.

The problem is, the flow does not come on demand. It usually requires some kind of warming-up. What I have noticed is that it is much easier to get into the flow after you are doing something that you like, something rewarding but not laborious. Yes, meetings are ultimate killers of the flow. Want to be productive, don't go to meetings. After trying many activities, I found one that helps to get into the flow. It works great for me and may work for you, especially if you are into eXtreme Programming a.k.a XP. This activity is writing unit tests.

It works the following way: Start your day with finding classes or methods that not covered with tests. There are always that are not covered. Write tests for some of them. As you are adding tests, you may be noticing small deficiencies in the code. Refactor them. You may even find bugs while writing tests.

Even if you didn't not feel like programming at all in the beginning, in 30 minutes you will notice that interest appears. The test coverage improves, refactoring is already helping to improve design. And you have just found a couple of subtle bugs (you will for sure) thanks to the new tests. All this feels really good and you are already coding. You are in the flow.

This "programmer's warm-up" helps to get into flow even when it seems that creativity has gone forever. Try it and quite possible it works for you. And don't forget the nice side effect in form of increased test coverage.

Helpful Resources on Unit Test and Test Coverage

Some of these tools are not free, but, believe me, they worth every cent.

• Clover code coverage tool for Java (We use it at Cacheonix)
• EMMA: a free Java code coverage tool
• NCover - Code Coverage for .NET Developers

The new "Gutsy Gibbon" version of Ubuntu was officially released this week.  While some of the developers at Teamprise use Ubuntu as their primary OS, I personally prefer Windows Vista, however I keep an Ubuntu image around so that I can quickly test any new UI code I write in an alternate OS.  Getting Ubuntu to install in VMWare is pretty trivial - the standard installation seems to work just fine.  However getting it to work in Microsoft Virtual PC is a little more complicated - the various tricks are located in various forum and bug posts around the web that I can never find when I need them so I am collating them here for my own use later - hopefully you may also find them useful.  Please bear in mind that I know enough about Linux to be dangerous rather than useful, so this may not be the "best" way - however it is my way ;-)

There are a few basic issues with getting Ubuntu 7.10 working inside Virtual PC.

1. The default display color settings use a higher color depth than Virtual PC supports
2. Ubuntu has a hard time recognizing the emulated Virtual PC mouse
3. Networking does not work initially
4. Time syncronization with the host can be a little off.
5. Soundcard does not work initially
6. The ubuntu splash screen "usplash" causes problems when shutting down/restarting.

Anyway, here we go.  Obviously first things first, you need Virtual PC 2007 and Ubuntu 7.10 (x86 desktop).

Installation:

• In Virtual PC, create a new virtual machine by pressing "New..." in the console.  I use the following settings:
  • In the OS selection select "Other"
  • Ram 512Mb
  • New Disk of 8192 Mb
• One created, go back into settings and Enable Hardware Virtualization if available (only available if you have a modern processor, and even then you probably had to enable this in your host machines BIOS first).  This can make a serious difference if your hardware supports it, however having the flag enabled makes the image less portable as some machines cannot support this option so they have to switch it off before using the image for the first time. 
• Start the VM.  Then go to Edit, Capture ISO Image... and select the downloaded Ubuntu install image.  You may need to restart your Virtual PC to get it to boot from the mounted DVD
• When the ubuntu install menu comes up, move down to "Start Ubuntu in safe graphics mode" then press "F6" for more options. 
• In the boot options, add the text "i8042.noloop" without the quotes.  This is a parameter we are passing to the kernel to get the emulated Virtual PC mouse to work correctly.  You entire boot options line should look like this:
  file=/cdrom/pressed/ubuntu.seed boot=casper xforcevesa initrd=/casper/initrd.gz quiet splash  -- i8042.noloop 
• Press enter to start the boot into the installation live cd.  Sometime when I do this, when the UI comes into view and the mouse is captured but it is like the virtual PC is only showing the top left hand corner of the screen.  Not sure why, but to fix it hit CTRL-ALT-F1 to go into a console and then CTRL-ALT-F7 to restart the window manager.  You should be able to see the whole screen now.
• Install by double-clicking on the Install icon on the desktop.  I use the guided partition manager to assign the whole virtual disk.
• Once the install has completed, unmount the DVD from Virtual PC by going to "CD" and Release "ubuntu-7.10-desktop-x386.iso" (remembering that in Virtual PC, the right ALT button releases your mouse from the guest machine).  Then go to Action, Reset to restart the VPC.

Initial Configuration:

When you boot for the first time, there are a few things that you need to do to get everything working properly.  We need to get all the things (including the mouse etc) working again.  These changes will then be stored on the virtual machine disk so you won't need to do them again for this image.

• Log in (remember the mouse isn't working yet), so press CTRL-ALT-F1 to drop into a console.
• Let's now get the mouse working again.  type sudo nano /boot/grub/menu.lst.  Page down to the bottom of the file and find the first kernel line after the phrase " ## ## End Default Options ##".
• Add i8042.noloop clock=pit to the kernel parameters.  You'll recognize the i8042.noloop thing from before, the clock=pit is a fix for the time drift issues (KB918461)
• Now we want to ensure the modules are loaded for the Virtual PC soundcard and Network card.  type sudo nano /etc/modules.  At the end of the file add the lines
tulip
snd-sb16
  Ensuring that each is on a line of it's own, followed by a carriage return.
• Then we want to remove the fancy Ubuntu splash screen that has some issues under Virtual PC.  Type sudo apt-get remove usplash -y
• Restart by typing sudo reboot
• When you restart your soundcard will be working - you might want to get rid of the login/logout sounds...  System, Preferences, Sound, Sounds, setting Log out and Log in to "None".
• As this is a virtual pc, depending on the usuage you may also want it to automatically log-in with the user account created on install.  In which case go to System, Administration, Login Window, Security and check the "Automatic login" box - selecting your user.  Also, because this is a virtual PC you may also want to disable the display power-management (System, Preferences, Power Management, Display) and disable the screen saver (System, Preferences, Screen Saver, uncheck Activate Screensaver when computer is idle).
• To get networking fully enabled, I've found that I have problems with the "Roaming Mode" introduced in 7.10.  Therefore I go to System, Adminitstation, Network.  Click on the Wired Network and select properties.  Disable the "enable roaming mode" check box and then select DHCP configuration then press ok.  Finally, check the checkbox next to the "Wired connection" box to make this the default connection used on startup.

Configuring to run Teamprise

Now I have a booting Ubuntu instance, I usually want to configure it with Java, Eclipse and Teamprise.  This part may be of interest to less folks, but I need to do it so I'll write down the steps but feel free to ignore.

• Installing Sun Java.  Teamprise needs some encryption stuff to support NTLM authentication that is not available in the CGJ - therefore another JVM needs to be installed - in this case I'll use Sun's.  Go to Applications, Add/Remove programs, change the drop down to "All available applications" and then in the search box type "Sun Java".  In my case I'm going to pick Sun Java 6 Console then I press "Apply".
• Next is Eclipse.  Easy way is from Add/Remove programs, search for "Eclipse" and Apply - however that was Eclipse 3.2 at the time of writing.  Nothing wrong with Eclipse 3.2 of course, I just like to be on the Early Adopter curve so I go to the Eclipse.org site and download the latest for linux.
• Then finally, I install Teamprise.  That part is easy for me (copy paste the update site URL into Eclipse and unzip Teamprise Explorer)

Hope that helps if you are going on a similar journey.

At the recent MVP summit, I attended a Mash-up lounge event.  It was a bit Web 2.0-ey, but basically they had a bunch of bean bags, fake palm trees, and members of the Windows Live team and some Virtual Earth MVP's around to help folks get up to speed with the Windows Live API's.  (Thanks to Tyler and Robert, Virtual Earth MVP's from InFusion Development for helping me with some sample code to grok).  This is a whole area that I've not really been paying attention to in the past year, so I dropped by to see what the deal was and have a play.

I have to say, I'm impressed.  It is super easy to control the Virtual Earth map control from your JavaScript, and the documentation is pretty decent.  Now, as I freely admit, I am no expert on the differences between the various mapping providers around and their web interfaces - but I will say this; The local.live.com data for Ireland and Northern Ireland is by far the best of the mapping sites that I've seen.  I also love the driving instructions and the custom print capabilities.  I get the feeling that if local.live.com was not made by Microsoft then there would probably be more fuss about it - but hey ho.  I've just come back from a trip to Redmond so I must have had my borg implant secretly added if I'm feeling Microsoft are being unjustly treated :-)

Anyway, on my return, I was having a chat with my good friend Rob Burke about some of the exciting possibilities that the mapping API gives you.  As an example, I did this very quick mash-up of traffic cameras from the Dublin City Council website and a Virtual Earth Map.  It is very crude as it only took me 5 minutes to write the HTML (in Notepad2) and then a further 15 figuring out the lat/long positions of the cameras - but hopefully you'll get the point.  If anyone wants the GeoRSS data I used to create the map then here it is.

Dublin City Center Traffic Cameras - data from Dublin City Council.

Every source control system I have worked with until recently supported keyword expansion.  My code comment blocks have always tended to be something like (in Java):-

/**
 * Description.
 * @author $Author: martin $
 * @version $Revision: 12 $ $Date: 2007-01-02 16:15 $ 
 */

The source control system updates the block in-between the $tag: $ on check-in and that's just how it works.  Over time, I standardized on author, version, date because that is what was common between all the major SCM tool vendors and things like the history tag just got in the way.  Keyword expansion did have it's downsides thought, the main one is that it always gave you problems when diffing and merging files and had the habit of giving you un-necessary conflicts to resolve.

I say that every source control system "until recently" supported keyword expansion.  Over the past couple of years there has been a notable exception - the version control provided by Team Foundation Server does not have this capability.  I found this very surprising at first - just because I expected to have them at my disposal, I never stopped to ask why I wanted them.

The reason why I used to want them was because I used to review code by printing it out on candy stripe paper and taking it along to a code review meeting.  Also, my source control tool was always a separate (frequently hard to use) application that meant me swapping tools and then finding the file I was working on in that separate application when all I wanted to know was to who to talk to about the line of code I was looking at.  I'll admit that was in the days when I did all my coding in PL/I with the odd sprinkle of JCL - but still.  It was awfully handy for the source control system to help you keep your comment blocks up to date, to tell you who was the last person who touched the code, therefore the first person you spoke to if your suspected the code was broke.

Now-a-days, it is rare that a single person "owns" a file.  If they do, you get a much better idea of what has happened by looking at the history.  As all this is nicely integrated inside the IDE, checking the history is a matter of right-clicking on the file in the IDE and selecting "History", I don't have to jump tools, re-navigate to find files etc.  With diff between versions and annotate functionality I can easily see who did what and when, what other changes they made at the same time and what work items they linked all this stuff too.  Maybe keyword expansion was just a solution for a problem I no longer have (except possibly when dealing with Stored Procedures, but IDE integration has come a long way there recently).

Yet I still miss it.  Does anyone out there have a reason why they *need* keyword substitution as part of their software development process, or is it just one of those things that you've always done that has long lost the value?