Planet SCM

The agile community has written much about this in the past year or so:

• Iterative vs Incremental - from the first (and original) Wiki Web
  
• The Neglected Practice of Iteratation - by Jeff Patton
  
• Difference between Iterative and Incremental Development - nice visual depiction
• No Waterfall Trap -- an even better visual depiction of the difference between iterative and incremental
• Alistair Cockburn (pronounced "Coh-burn") has written several papers on the subject:
  • Using Both Incremental and Iterative Development -- Crosstalk, May 2008
  • Iterative and Incremental Development: How & Why you should be doing BOTH -- Better Software, April 2008
  • Incremental means adding, Iterative means reworking -- Jan 2008 (I dont care for this definition, but the article has lots of good references/resources)
• Iterative and Incremental Development: A Brief History -- by Craig Larman and Victor Basili, IEEE Computer, June 2003

Apologies in advance for being a "stick in the mud" on this one - I'm not particularly happy with the definitions so far. I searched around some more on the WWW and came across one I like a lot that I think better meets our needs.

It is from the paper What is Iterative Development? (part 1), by Ian Spence and Kurt Bittner,

Iterative and Incremental Development:
A style of development that involves the iterative application of a set of activities to evaluate a set of assertions, resolve a set of risks, accomplish a set of development objectives, and incrementally produce and refine an effective solution:

• It is iterative in that it involves the successive refinement of the understanding of the problem, the solution's definition, and the solution's implementation by the repetitive application of the core development activities.²


• It is incremental in that each pass through the iterative cycle grows the understanding of the problem and the capability offered by the solution.


• Several or more applications of the iterative cycle are sequentially arranged to compose a project.

Sadly, development can be iterative without being incremental. For example, the activities can be applied over and over again in an iterative fashion without growing the understanding of the problem or the extent of the solution, in effect leaving the project where it was before the iteration started.

It can also be incremental without being truly iterative. For example, the development of a large solution can be broken up into a number of increments without the repetitive application of the core development activities.

To be truly effective the development must be both iterative and incremental. The need for iterative and incremental development arises out of the need to predictably deliver results in an uncertain world. Since we cannot wish the uncertainty away, we need a technique to master it. Iterative and incremental development provides us with a technique that enables us to master this uncertainty, or at least to systematically bring it sufficiently under control to achieve our desired results.

I like that this definition separated iterative from incremental and then defines them together. I would summarize it as follows (but I like the above better, even if it is longer):

Iterative development is the cyclical process of repeating a set of development activities to progressively elaborate and refine a complete solution. The “unit” of iterative development is an “iteration”, which represents one complete cycle through the set of activities.

Incremental development is the process of developing and integrating the parts of a system in multiple stages, where each stage implements a working, executable subset of the final system. The “unit” of incremental development is an “increment”, which represents the executable subset of the system resulting form a particular stage

Iterative and Incremental development is therefore ...

the application of an iterative development lifecycle to successively develop and refine working, executable subsets (increments) of a solution that evolves incrementally (from iteration to iteration) into the final product.

• Each iteration successively elaborates and refines the understanding of the problem, and of the solution's definition & implementation by learning and adapting to feedback from the previous iterations of the core development lifecycle (analysis, design, implementation & test).
• Each increment successively elaborates and refines the capability offered by the solution in the form of tangible working results that can be demonstrated to stakeholders for evaluation.
  

An Agile Iteration is a planned, time-boxed interval (typically measured in weeks) whose output is a working result that can be demonstrated to stakeholders:

• Agile Iterations focus the whole team on collaborating and communicating effectively for the purpose of rapidly delivering incremental value to stakeholders in a predictable fashion.
• After each iteration, the resulting feedback and data can be examined, and project scope & priorities can be re-evaluated to adapt the project's overall performance and optimize its return-on-investment

So in addition to the non-agile-specific definitions above, we see that Agile iterations are adaptive, in that they use the previous results and feedback to learn, adjust and recalibrate for the next iteration. And Agile increments are tangible, in that they can be executed and made accessible to stakeholders for demonstration and evaluation.

That's my story and I'm sticking to it!

InfoQ.com summarized an email-list discussion thread on the subject of using a Traceability Matrix in an Agile Project.

I contributed quite a lot to the thread, and InfoQ apparently included many of the key things I said along with the related URLs to articles I've written. (Thanks guys!)

Some of the discussion with my co-authors on our May 2008 CM Journal article on Agile Release Management spurred some additional thoughts by me that I hope to refine and work into a subsequent article later this year.

Release Management is about so much more than just the code/codeline (and it being "shippable") it's not even funny. Some other articles to reference and mention some key points from are:

• Breaking the Major Release Habit
• Release Management - Making it Lean and Agile

Kevin Lee has written some GREAT stuff on Release Management that relates to Agile. The best is from the first and last chapters of his book on "The Java™ Developer's Guide to Accelerating and Automating the Build Process" but bits of pieces of it can also be found at:

• Agile SCM in the Enterprise
• Software Release Management Best Practices
• Agile Software Delivery
• Architecting the Build Process

ANY discussion about Release Management also needs to acknowledge that there is no single "the codeline", not just because I may have different codelines (Development-Line plus Release-Line) working toward the same product-release, but ESPECIALLY because no matter how Agile you are, the reality is that you will typically need to support MULTIPLE releases at the same time (at the very least the latest released version and the current under development version, but often even Agile projects need to support more than one release in the field)

So, when dealing with multiple release-line, and any "active development lines" for each of those, and the overall mainline, we really should say something overall about how to manage this "big picture" of all these codelines across multiple releases and iterations:

• What is the relationship between development line, release-line and release-prep codeline?
• How do the above three "lines" relate to "mainline"
• What is the relationship between the different release-lines for the different supported releases
• What is the overall relationship between the mainline and the release-lines (and if the mainline is also a release-line, which release is it?)

The above questions and the ability to give some big picture "advice" on relating it all together (the stuff of pattern languages) is precisely where Laura Wingerd's writing on "channeling the flow of change" and her chapter on "How Software Evolves" fits in! It tells us

• The overall Mainline model
• The different types of codelines ("line" patterns), and what kinds of builds take place on each of them
• The relationships of those to the mainline
• When+Why to branch (and from which "types" of codelines)
• When+Why to merge across codelines (as a general rule)

These are where Laura's rules for "the flow of change" apply. And her concept of "change flow" is very much applicable to the Lean/Agile concept of "flow of value". The Tofu scale and "change flow" rules/protocol have to do with order+flow of codeline policies across the entire branching structure when it comes to making decisions about stability -vs- speed. One codeline's policy might make a certain tradeoff, but it is the presence of multiple codelines and how they work together, and how their policies define the overall flow of change across codelines, that forms the "putting it all together" advice that is key to release management across multiple releases+codelines.

In some way's you could make an overall analogy to the Laws or Thermodynamics and the realities of codeline management. Software and codelines tend, over time, to grow more complex and, if unchecked,
"Entropy" (instability) quickly becomes the most dominating force to contend with in their maintenance. See

• "The 8 'Laws' of Software Evolution", and ...
• "The Laws of Software Complexity"

The "entropy" (instability) doesnt just happen within a codeline. It can actually get far more hideous when it happens across codelines via indiscriminate branching from, or merging to, other codelines. This is what happens when you don't respect the principles and rules of "change flow" (from Wingerd) which ultimately stem from the rules of smooth and steady (value-stream) flow from Lean.

The Laws of Thermodynamics are about energy, entropy, and enthalpy. In the case of release management and codelines ...

• energy relates to effort & productivity
• entropy relates to stability/quality versus complexity
• enthalpy relates to "order" (i.e., in the sense of structure and architecture as Christopher Alexander uses the term "order"). It is the "inverse" of entropy.

We could call them them "Laws of Codeline Dynamics" :-)

Energy misspent degrades flow, creates waste, and hurts productivity/velocity. In traditional development, we often see "fixed scope" with resources and schedule having to vary in order to meet the "scope" constraint. IN Agile development we deliberately "flip" that triangle upside down (see the picture in the article at here under the title "The Biggest Change: Scope versus Schedule - Schedule Wins"). So we are fixing "resources" and "schedule" and allowing scope to vary.

This might be one way of viewing the law of conservation of energy. If we fix resources and time (and insist on "sustainable pace" or "40hr work week") then we're basically putting in the same amount of effort over that time-box, but the key difference is how much of that effort results in "giving off energy" in the form of waste ("heat" or "friction") versus how much of that energy directly adds value. Both "Value" and "Enthalpy" degrade or depreciate over time, and adding more energy (effort) doesnt necessarily mean value is increased.

To make sure that energy goes toward adding value (and minimizing waste) we need to focus on the flow of value, and hence the flow of change/efforts to create value (the latter is one reasonable definition of a "codeline" or a "workstream"). to ensure a smooth, steady, and regular/frequent flow, there are certain rules we need to impose and regulate stability within and across codelines to better manage all those releases.

Zeroth Law of Thermodynamics (from Wikipedia)

- If two thermodynamic systems are each in thermal equilibrium with a third, then they are in thermal equilibrium with each other.

Translation to codelines ... this law of "thermal equilibrium" is a law of "codeline equilibrium" of sorts. (Does this mean If two codelines are are "in equlibrium" with a third codeline, then they are "in sync"? and with each other? here "in sync" doesnt mean they have the same frequency, it means their is some synchronization pattern regarding their relative stability and velocity. In Lean Terms, this would refer to "nested synchronization" and "harmonic cadence"). This might imply the "mainline" rule/pattern or one of Wingerd's rules of change-flow.

First Law of Thermodynamics

- In any process, the total energy of the universe remains the same.

This is the statement of conservation of energy for a thermodynamic system. It refers to the two ways that a closed system transfers energy to and from its surroundings - by the process of heating (or cooling) and the process of mechanical work.

This relates to effort & changes expended resulting in the creation of value and/or the creation of waste. We have activities that add value (which we hope is development), activities that preserve value (which is what much of SCM attempts do, given that it doesnt directly create the changes, but tries to ensure that changes happen and are built/integrated with minimal loss of energy/productivity/quality), and then we have activities (or portions of activities) that create waste (and increase entropy rather than preserving or increasing enthalpy/order)

Second Law of Thermodynamics

- In any isolated system that is not in equilibrium, entropy will increase over time

So this is the law of increasing instability/complexity/disorder. The "key" to preventing this from happening is achieving and then maintaining/preserving "equilibrium". How do we achieve such equlibrium? we do it with the "release enabler" patterns for codeline management (which help ensure "nested synchronization" and "harmonic cadence" in addition to achieving a balance or equilibrium between stability and velocity (to smooth out flow).

Third Law of Thermodynamics

- As temperature approaches absolute zero, the entropy of a system
approaches a constant minimum.

In our case, "Temperature" could be regarded as a measure of "energy" or "activity". As the energy/activity of a codeline approaches zero (such as a release in the field that youve been supporting and would LOVE to be able to retire that codeline sometime real soon), it's instability approaches a constant minimum.

This is perhaps another more polite way of saying something we already said in our article on "The Unchangeable Rules of Software Change", namely that "absolute stability" means dead (as in, "no activity"), and should serve as a reminder that our goals is not the prevention of change in order to achieve some ideal "absolute stability", for such an absolute would mean the project not just "done" but "dead".

On the other hand, it also speaks to us as a guideline for when it is safe to retire old codelines, and when to change their policy in accordance with their "energy level"

My Agile SCM co-authors Rob Cowham, Steve Berczuk, and myself have written an article for the May CM Journal on An Agile Approach to Release Management

We're relatively pleased with the article, and all collaborated together quite well.

My review of Jim Brosseau's Software Teamwork: Taking Ownership for Success is available in the May issue of the Agile Journal. It is nothing less than outstanding!

I found Software Teamwork to be an immensely helpful, intensely practical, profusely insightful field guide to improving team outcomes and changing team behaviors by focusing on interpersonal action and personal leadership. This book belongs on any software team-leader's bookshelf, along with Jean Tabaka's Collaboration Explained and Murray Cantor's Software Leadership.

Other articles in this issue on the theme of "Challenges with Distributed Agile" are:

• Softening Iterations - Setting up for success
• Overcoming Resistance to Change with Distributed Agile
• Agile Adoption Patterns
• KPIs for Agile Teams

I recently learned that Michelle Sliger, author of the wonderful 4 part series of articles on Relating PMBoK to Agile Practices, is co-authoring a book with Stacia Broderick entitled the Software Project Manager's Bridge to Agility. You can even download an excerpt from her website.

I'm looking forward to this book a great deal, judging by the excellent articles and presentations of hers that I've read.

Yet Another Fad
We’ve heard it all before. The Segway, UML, Virtual Reality, TQM, object-oriented databases, Artificial Intelligence. Agile’s no different, right? Just the latest fad? That’s what I thought.

Back in 2005 at the SD Best Practices conference, I had my first run-in with Agile development. At a speaker’s reception, my blissful ignorance of Agile came to an end. I was surrounded by Agilists. They looked perfectly normal to me, but then they started to talk about the advantages of keeping track of requirements on 3x5 cards, pair programming, and lots of other crazy sounding stuff.

3x5 cards! Are you kidding me? What is this, the seventies? How can people who make a living writing software advocate the use of 3x5 cards? And what about pair programming? Two people sitting together all day sharing the same keyboard and mouse switching between coding and sitting on their hands, having to hear phone calls about the results of medical tests. Yuck!

I wasn’t just skeptical, I was outraged that all these people were trying to push this nonsense. Keep in mind that I work at a company that provides software tools to companies that are looking to improve their process. On the surface, Agile seems to be diametrically opposed to that. I felt that it was time for somebody to step forward and do something about it. I nominated myself. To gather evidence I read the books, talked to the experts, and attended the seminars. But then one day a funny thing happened. I had an aha moment and I realized there might actually be something of value buried under all of the rhetoric.

To illustrate what I realized, let's compare traditional development and Agile development using a typical development scenario.

Next: Traditional Development Scenario

Agile 2008 - August 4th - 8th
I will be doing three sessions at Agile 2008 in Toronto. If you are going, I hope you will consider attending one of my sessions. In any case, I look forward to meeting you there. Agile 2008 promises to be a truly great conference. The planning process was amazing with lots of great discussion of potential proposals.

Here are the three sessions that I will be doing:

Questioning the Perception of Agile (Full Listing)
There are many misperceptions of Agile. This session is mainly for Agile advocates. It is a facilitated discussion about common misperceptions/misconceptions and ways to alleviate them.

Calling All Agile Skeptics, The Curious, and Die-Hard Non-Agile (Full Listing)
If you are looking for a place to either air your skepticism/questions or respond (helpfully) to skeptics/questions, this will be a facilitated forum for both. Please bring an open mind, your sense of humor, and at least one anecdote to share. Participants will be expected to share the floor and help to keep the atmosphere fun and relaxed.

See Large Scale Multi-Stage Continuous Integration in Real Time (Full Listing)
Multi-Stage Continuous Integration is a way to scale CI to larger teams as well as a way to make CI more effective for teams of any size. For a preview, I have a series of posts on the topic. This session will show a real-time environment using AccuRev, Cruise Control, and Jira with hundreds of scripted developers resolving issues, making changes, causing builds to fail and pass, etc.

SD Best Practices, October 27th - 30th

Introduction to Multi-Stage Continuous Integration (Full Listing)
Multi-Stage Continuous Integration is a way to scale CI to larger teams as well as a way to make CI more effective for teams of any size. For a preview, I have a series of posts on the topic.

Agile Development Practices, November 10th - 14th
A full list of sessions is TBD soon, but all other details are available now and registration is open.

Introduction to Multi-Stage Continuous Integration (Details TBD)
Multi-Stage Continuous Integration is a way to scale CI to larger teams as well as a way to make CI more effective for teams of any size. For a preview, I have a series of posts on the topic.

Calling All Agile Skeptics, The Curious, and Die-Hard Non-Agile (Details TBD)
If you are looking for a place to either air your skepticism/questions or respond (helpfully) to skeptics/questions, this will be a facilitated forum for both. Please bring an open mind, your sense of humor, and at least one anecdote to share. Participants will be expected to share the floor and help to keep the atmosphere fun and relaxed.

One of the principles of Agile, mostly related to design and architecture, is “The Simplest Thing That Could Possibly Work.” This is sometimes taken as a license for cowboy coding. But that is not the intention. A better way to express it would probably be something like “The Simplest Solution That Could Possibly Satisfy Your Requirements.” For instance, if you have a requirement to create the back end for a web site like amazon.com, then while a perl/cgi solution on a single core machine could possibly “work,” it doesn’t work from the point of view of high availability, fast response time, or reliability.

From Oversimplification To Rube Goldberg
On the one hand, there is a wide spectrum of complexity of construction ranging from doing nothing to Rube Goldberg level complexity. On the other hand, there is the set of solutions that work, meaning that they meet all of the requirements. TSTTCPW refers to the solution which works and which is lowest in complexity.

Part of being simple means simple to read, maintain, use, design, understand, and implement balanced against the time it takes to get the job done. Spending too much time to create the ultimate in simplicity starts to get you into a different kind of trouble.

As somebody that struggles to apply this principle on a regular basis, I was happy to stumble upon an example of this principle which can be captured in a picture and kept in mind as I am working on a new design. Perhaps you will find it to be useful food for thought as well.

A Bridge Too Far
There’s a construction project that you’ve probably heard of which is affectionately called the “Big Dig.” Part of this project was the construction of the “Leonard P. Zakim Bunker Hill Bridge” aka the “Zakim bridge.” This part suspension bridge, part cantilever bridge is an enormous one of a kind architectural marvel. It supports five lanes of traffic in either direction for a total of ten lanes. It was built at a cost of approximately $11M per lane.

Running parallel to the Zakim (to the left in the photo) is the Leverett Circle Connector Bridge. It serves a total of four lanes of traffic. It was built at a cost of approximately $5M per lane.

Part of the requirements for the Zakim bridge were clearly “create a stunning new Boston landmark.” On the other hand, the Leverett Bridge is a very simple but also very strong bridge. It could have been made even more simply, but not without a safety risk and/or a shorter lifespan. In other words, it is “The Simplest Thing That Could Possibly Work.”

Next: The Faberge Egg

TOC: Zero to Hyper Agile in 90 Days or Less

[Note: revised 7/3/08 to reflect comments on reddit. Clearly the original post didn't work. :) ]

When choosing how to allocate resources, it can be difficult to do an effective cost benefit analysis in a short period of time. A technique that I stumbled upon is to look at things from the perspective of “what would I do if I was the only person on the team?”

In 1992, I convinced my manager at the Open Software Foundation (OSF) to create the position of tool smith and allow me to work on the OSF Development Environment (ODE) full time. As I now had much more time for doing development and made many more changes to ODE than I had ever made before, there was much more testing to be done.

Initially, I didn’t realize just how much testing was fully required for ODE. Not only were there no automated tests for ODE, there were no documented test cases either. The test plan consisted entirely of “round up as many volunteers as you can and have each volunteer test on one of the nine platforms.” The testing that each volunteer did was entirely up to their discretion. It was different volunteers every time, and there was no record of what they did, only a “seems fine to me” or “I found these problems.” Once the number of problems got down to an acceptable level, we’d ship the new version.

After a couple of releases it started to sink in that I had to do something differently. My first attempt was to document a standard set of test cases. At first this seemed to work really well. Testers commented that it was much easier and took much less time. I felt like I had gotten a consistent set of results from a consistent set of tests. But a test/fix cycle requires running the same tests over and over again until things stabilize. Following the same steps over and over again can become pretty mind-numbing. Pretty soon I couldn’t get the volunteers I needed and I was starting to suspect that people were skipping steps.

I also discovered another problem. As bug reports came in from production use, and as I thought of test cases that were missing, the list of test cases mushroomed. I was very careful not to include overlapping test cases, but still the list grew and grew and grew.

Since the QA of the tool was ultimately my responsibility, I had to pick up the testing slack. The combination of the ever increasing list of test cases and the diminishing pool of volunteers soon made the QA of a new release pretty much impractical. I would end up spending much more of my time testing than coding.

But then I remembered how I had gotten the job of tool smith in the first place, by automating myself out of my previous job of manually kicking off builds on all platforms. Test cases are basically a set of steps to take and the expected results. Automating test cases is basically coding, and that was much more fun than manual testing. A couple of inspired weekends later I had automated all of the test cases and added many more new ones as well.

That was the last time I ever relied on manual tests.

Read more from: "Zero to Hyper Agile in 90 Days or Less"

There are really only five ways to increase the profitability of a business based on software development: reduce costs via outsourcing, reduce headcount, reduce other expenses, increase productivity or increase revenues. Reducing expenses can only go so far. The most expensive part of software development is the people. Thus, one of the most successful ways to increase profits is to increase the productivity of the software development team.

The Agile Workplace
At Litle & Co., developers like the fact that Agile provides the additional challenge of solving business problems instead of just technical problems which requires thinking at a higher level. Developers at Litle report that they have a higher level of job satisfaction than in previous companies that were not using Agile because they see the results of their software development efforts installed into production every month. Also, they like the fact that there is much less work which amounts to “implement this spec.”

Your development infrastructure is really no different than the general company infrastructure which includes your cube or office, the carpet, the artwork on the walls, the company cafeteria, your phone, your computer, and the company location. These are all part of your work environment. If you have a computer that is 5 years old, your work environment is not as good as if you have a computer that is only 2 years old. If you are writing in C rather than C++, C# or Java, your work environment is sub-optimal.

The closer that your development infrastructure is to the ideal environment for your circumstances, the more productive your team will be. This principal extends to all aspects of the development environment, from development language, to build system, to build farm, to issue tracking system, to the process that you follow.

Next: Your Development Process is Part of Your Work Environment

Your development process (regardless of how it is implemented), is also part of your work environment. If as a result of your development process you regularly end up redoing work because problems weren’t discovered until just before the release, or projects get cancelled or shelved, then this is also likely to reduce productivity and job satisfaction. As this process improves, so does your work environment. The smoother it operates, the more pleasant your working environment will be.

There are many problems which you may think of as being unrelated to your development process. For instance, broken builds. Broken builds are simply the result of somebody making an idiotic mistake, right? Perhaps that’s true some of the time, but most of the time it is due to the complexity of integrating many changes made by many people for software that has many interdependencies.

To be sure, a “perfect” process does not guarantee happiness, success, or the absence of problems. You still have to debug complicated problems, port to new platforms, deal with unforeseen circumstances, etc. However, the state of your process impacts the efficiency with which your effort is applied.

If your process is perfect and completely frictionless, then 100% of your effort will be applied to the work that creates value. If it is rife with problems, it may mean that only 50% (or less!) of your effort will be applied to work that creates value. If there are problems with the process, then you are already expending effort which is essentially wasted. You would be better off investing some of that effort in removing the problems permanently instead of losing it to friction on a regular basis.

Next: Quick Summary of The Benefits of Adopting Agile

Here's an Agile development scenario. For simplicity, let’s use two month iterations. Marketing says the three features with the highest ROI are a Facebook plug-in, a Second Life plug-in, and an RSS feed plug-in.

You start with the Facebook plug-in. You plan. You design. You create a test plan while the code is being written. You discover potential problems and deal with them. You automate those tests while the code is being written. As the code is written, it is integrated, built and tested continuously; problems are found and fixed immediately. At the end of development, the only problems that remain are the ones that could only be found at the end of development. If you wanted to, you could cut a new release with very little overhead.

Marketing announces that the Second Life plug-in is not as marketable as they had hoped and iPhone support is showing signs of becoming very lucrative. So, you start on the RSS feed support. Now marketing says the Second Life plug-in is worthless but iPhone support is hot, so you add iPhone support. During the whole process, you kept your options open. At the end you have produced more business value than using the traditional process, and you were in a position to start realizing that value much earlier.

To get the primary benefits of Agile, as described above, you don't have to use 3x5 cards or pair programming, and you don’t have to do frequent releases. These things are optional and you can use them or not according to your preference and circumstances.

Next: Achieving the Primary Benefits of Agile

The benefits of moving to Agile development can be split into two categories: benefits to the organization, and benefits to you personally. As a result of increased productivity, higher quality, and responding more rapidly to market demands, Agile can provide the following benefits to the organization:

• Increased revenues
• Reduced costs
• Increased market share
• Higher customer satisfaction

Each of these benefits lead to a stronger organization which is then in a better position to reward you for your efforts both directly and indirectly. Some of these benefits include:

• Getting a raise and/or bonus
• Having more discretionary income to buy cool stuff
• Improving your working conditions
• Actually using all of your vacation time
• The opportunity to spend more time working on cool stuff

In addition, Agile can provide the following direct benefits:

• A less stressful environment
• Less canceled or shelved work
• Career advancement due to learning new skills
• Having the resume that gets you your dream job

Many of these same claims have been made in the past about tweaks to traditional development, but nothing ever came of it. Why will this be any different with Agile? Let's take a look at why attempts to "fix" traditional development haven't panned out.

Next: Having Dev Team Performance Problems Using Traditional Development? Try Niagra!

There was a developer that worked for me once, I’ll call him George. He wrote a lot of really good code. But one day he decided he wanted to make his mark on things. George believed that the way to do it was to create a beautiful widget. He wanted it to be so beautiful and so useful that it could be used generically and would be something that we could sell separately on its own merits.

The widget was part of a Java/Swing user interface for an issue tracking system. It was responsible for the data model used by a form, a query editor, and a handful of other objects. So, it did need to be fairly general purpose, but it didn't need to be so generic and so functional that it would be something that people would want to buy separate from the application.

George referred to this widget as his “Faberge Egg.” It was an apt name for the widget. It was overly ornate, intricately detailed, and supported a very wide range of functionality that we had no immediate use for and still don’t to this day.

I was very clear with George that I only wanted a “Cadbury Egg” and tried hard to convince him that he could provide much more value to the company in other ways and the company would reward him for that, not the Faberge Egg. I like folks to have the freedom to grow and explore. Even though we were under deadline pressure, I gave George some wiggle room while at the same time trying to guide him towards the simplest design that would work. Unfortunately, after weeks of work and many conversations, George’s desire to produce a masterpiece of a widget prevailed.

Over time, that widget has been the source of more than its share of problems, both in bugs and in making it more complicated for other developers to maintain it and extend it. It has been partially refactored several times, but it seems there has never been the time to really do the full job required.

The same functionality that the widget provides was needed in our new web interface. Instead of reusing the code as we have done for other functionality, we decided to start from scratch. Doing just what was needed for the job at hand took only two days and that code has been very reliable right from the start.

Next: Frequent Releases Improve Code Quality Faster

Related: The Iterative Design of a Transforming Lego Sports Car

In a traditional project, the demand for resources from the four major aspects of software development see-saws dramatically over the course of a project. These aspects are project management and planning, architecture and design, development, and QA. You need resources on hand to serve the peak demand level, but during periods of low demand those resources will either be idle or used for activities which have a lower ROI.

A common circumstance is that there are insufficient resources on hand for the peak demand level and so people end up working in “crunch mode.” During crunch time, people tend to make more mistakes. Agile levels demand out over time and removes this see-saw effect which simplifies resource planning and removes the need for crunch time.


In the figure above, the straight green lines represent the resource load in an Agile project and the zig-zagging purple lines represent the resource load in a traditional project.

With traditional development, delays during development compress most of the testing to the end of the process which then requires taking shortcuts due to schedule pressure. I used to think that one way of compensating for insufficient QA resources was to delay the release until QA finishes. On the surface it seems to make sense. But only if the folks writing code sit on their hands while QA does their work. Ok, so you have multiple projects and the developers work on another project. But then they finish that. Now QA starts on the second project and the developers move to the third. The problem is still there.

On the other hand, as a result of the need for increased QA resources during testing, you may have two other problems. If you have enough QA resources to handle the pressure of the endgame, you may have too many QA resources during the rest of your development cycle. Alternatively, you may bring on additional QA resources on a short-term basis to compensate. Both of these options are obviously undesirable.

There’s a natural balance between the amount of effort required for developing something and the amount of effort required to QA it. No matter what you do, if you have the wrong ratio of development resources to QA resources, it will cause problems. If development creates more than QA can absorb, you will create a backlog of QA work that will always grow.

There are six options for dealing with a QA backlog: do less QA than needed and thus shift the burden of finding problems that you could have found onto your customers, increase your QA capacity, decrease your development capacity, have development idle, have development help with QA or allow the backlog to grow. The larger your testing backlog, the longer it will take to ship it and the greater your opportunity cost.

The imbalance may be in either direction. After you transition to Agile development, you may find that you have more QA resources than are needed. In that case, you have the option of having QA take on some of the work currently done by developers. See The Role of QA in an Agile Project for more on that topic.

This natural balance holds between all four aspects of software development. Depending on your organization, there may be an imbalance between supply and demand at any stage in the pipeline. Wherever there is an imbalance you have the same six options as described above. For example, you may end up with project plans that are never used, developers idle because the design isn’t ready yet, etc. To the extent that some of the resources are actually the same people you can use that fact to manage this problem.

When using short iterations, resource imbalances are easier to detect and correct. Having balanced resources means that all development activities are done consistently and on a regular basis and there is no need to take the shortcuts that are typical of traditional development.

Next: The Usability of Short Iterations

TOC: Zero to Hyper Agile in 90 Days or Less

Stand-up meetings are a great way to reduce delays in communicating important information. Another benefit of stand up meetings is the elimination of time-wasting status and progress meetings.

Stand up meetings are most closely associated with Scrum and are called “Daily Scrum Meetings” within Scrum, but have become populare independent of any particular methodology which is a good indicator of suitability for mainstream use.

A stand-up meeting is simple to implement. There are just a handful of guidelines:

• Limit the time to fifteen minutes.
• Pick a regular time for the team to meet, preferably in the morning.
• Start on-time regardless of who is absent.
• Each person answers these three questions:
• What have you accomplished since the last meeting?
• What are you working on next?
• What impediments do you have?
• All discussion and problem solving is deferred until the end of the stand-up meeting.
• Follow-up as needed in smaller groups

Although it is called a stand-up meeting and standing is encouraged, the time limit is the most important part and standing is optional.

The point of a stand-up meeting is to improve communication and to discover and resolve impediments, not to have a meeting just for the sake of having a meeting. If the team feels that other practices make the stand-up meeting redundant, then by all means reduce their frequency or even discontinue use until such time as it appears to be necessary again.

To help make this decision, let’s take a look at the expense side of stand-up meetings. First, people have to get to it. And then they have to get back to their computers. Scrum discusses how to minimize this time, but practically speaking, there is more overhead than just the ideal 15 minute meeting. If you are at a larger company, somebody has to book the room and let people know where it is. Let’s call the cost of the meeting 20 minutes per person. If you have 12 people in a stand-up meeting, that’s 4 person hours per day. That’s the equivalent of half of a person. Those meetings had really better be worth it!

Now let’s take a hard look at the stand-up meeting itself. One of the basic ideas of Agile (and Lean) is continual self improvement. If the value of the meeting exceeds the cost, then there’s no problem with the meetings, especially if they are eliminating other meetings. If the stand-up meeting is the only remaining meeting, that seems like a good thing. However, continuous improvement means we’re never satisfied. Now that you are down to just the one meeting, you should still ask the question: “is it providing more value than the cost? Is there a better way?”

What is the purpose of a stand-up meeting? To quickly find out if people are getting their assigned work done and if not why not. If it is more efficient to do that via e-mail, IM, an issue tracking system, or other means, then use those means. Someone might say “but seeing folks face to face is worthwhile.” Ok, so why not just do that then? Go out to lunch together or something like that.

Or perhaps the stand-up meeting is needed because otherwise folks wouldn’t complete their work, or people wouldn’t speak up when they run into an impediment. In that case the stand-up meeting isn’t a solution at all, it is a crutch. For instance, perhaps somebody isn’t completing their work because they don’t like it, but the constant peer pressure of the standup meeting is goading them into completing their work anyway. So then the real problem is lack of job satisfaction or low moral or something along those lines. Until you fix that problem, the stand-up meeting is just acting as a band-aid.

The real measure of project status and health is having an increment of shippable value at the end of every iteration. A standup will only expose problems that are on people’s minds, but the forcing function of the increment of shippable value is where you will get the true picture of how things are going. A one month iteration interval is good, but if you can get it down to 2 weeks or even 1 week, that may do far more to expose real problems than a standup will.

Next: How Agile Helped Litle & Co. Get to #1 on the Inc. 500

Subversion 1.5 has just been released and has a whole raft of new features that developers have been asking for. To take full advantage of all the new features you’ll need to upgrade both your server and clients, but both will inter-operate so you can upgrade gradually if you wish. Server upgrade does not require a dump and reload, but as usual with any major upgrade you should back up your repositories first. Some of the new features in 1.5 require a repository upgrade—after installing the 1.5 server software use svnadmin upgrade to bring your repositories up to the newest format, Subversion won’t do this automatically.

The killer new feature in Subversion 1.5 is merge tracking. This is a feature that Perforce has had for years and that I always missed in Subversion. It’s a major change and the Subversion developers have been working on it for several years—there’s literally been a design document in the Subversion repository since the 1.0 days.

Usually when creating a release, you’ll create a production release branch of your code. This branch will be where you get your software ready for final release, fix the last few bugs, that sort of thing. You’re also likely to use this branch for production support, fixing production issues when they arise. Using this strategy, detailed in Pragmatic Version Control using Subversion, you will frequently need to merge release branch changes back down to the trunk. Until now, you had to manually track which changes you had already merged, and ask Subversion to only merge new changes. This meant a fair amount of manual bookkeeping, writing down revision numbers, looking at log entries, and so on.

Subversion’s new merge tracking fixes the need to manually figure out which changes need to be merged between branches. Instead you just tell Subversion you want the branch changes merged to the trunk, and it figures out what to merge. You can run the same command every week to merge changes, no revision numbers required. Subversion 1.5 also makes it easier to merge entire branches back down to the trunk, for example when merging an experimental changes branch to the trunk.

For merge tracking to work, you need to upgrade both your server and client to 1.5, and upgrade your repository with svnadmin upgrade. Merge tracking isn’t quite finished in Subversion—1.5.1 will address performance issues and some edge cases such as cyclic merges.

Another great feature in Subversion 1.5 is change list support. As you are working on a change, you can organize your changes into named change lists, and check them in independently. This is really useful in cases where you’re working on a feature but then someone asks you to fix a bug, and you want to do a quick fix on the bug and check it in. You can now just fix the bug, keep the bug changes in a different change list to the feature changes, and then commit the bug fix without committing the unfinished feature. This is a client-only feature which many people did manually previously—I know I’ve deliberately done half-commits so someone else could see my work sooner. Unlike the Perforce version of change lists, no-one else can see your in-progress change lists, they’re stored on the client only.

As usual TortoiseSVN, everyone’s favourite Windows Subversion client, has full support for all the new Subversion features.

Once More Unto the Breach, Dear Friends
As Helmuth von Moltke (the Elder) said, “No battle plan survives contact with the enemy.” In the case of software development, the “enemy” is reality. What we think we need to do to satisfy customers and how we think we need to implement it generally changes as we implement and as we get customer feedback.

Similarly, Dwight D. Eisenhower said, “In preparing for battle I have always found that plans are useless, but planning is indispensable”. Planning (short and long, tactical and strategic) is still important in the Agile world. The difference is that Agile is specifically designed to take advantage of the fact that planning, design, and development have a learning component and to accommodate and embrace the resulting need to change plans on a regular basis. As you discover new information during an iteration, you can easily adjust your plans to take what you have learned into account in future iterations.

Too Busy to Plan
Short iterations and other Agile practices such as “You Aren’t Going to Need It” (YAGNI), may seem to indicate that Agile is tactical in nature. However, I would say that it is when looked at through a Waterfall lens. For instance, with Waterfall the planning horizon and iteration length (the full start to finish time) are the same. So if you look at Agile with this perspective you might think that the longest that Agile folks look ahead is 30 days. You might then conclude from this that Agile can only be used tactically. But in fact there is no connection between the planning horizon and the iteration length.

Many teams use very short iterations; two weeks or one week. What happens within an iteration is whatever has been planned for that iteration. Different Agile methodologies use different approaches for planning iterations. In any case, you can plan as far out as you would like. As an Agile product owner I can tell you that some of my products have plans going out multiple years.

Sometimes there is work that can’t be fit into a single iteration (whatever iteration length you have chosen). In that case, you can use multiple overlapping iterations. For instance, a one-week iteration for most stuff and an overlapping one month iteration for the exceptions. Over time though, teams generally get better at avoiding the need for exceptions.

Wherever You Go, There You Are
In my experience, whether the software development of a company is done tactically or strategically is connected to the planning culture of an organization, not the methodology. If an organization thinks and plans strategically, that will be reflected in their software development whether they are using Agile or Waterfall.

I’ve seen plenty of companies that use Waterfall but think tactically, packing mostly tactical features into a large release. Feature creep is a good example of how tactical thinking is fairly common in Waterfall projects. While of course it is true that “that shouldn’t happen” the facts on the ground are that it does.

I’ve also seen plenty of Agile projects that were strategic in nature. For instance, an Agile company that thinks strategically will have a strategic plan which they carry out using multiple iterations. As they discover that they need to do something tactical in the middle of their strategy, or they need to change their tactics in support of their strategy, it is simple to do with short iterations. They just change their plans for future iterations. As soon as their current iteration is done, they begin the next iteration which has now changed to take into account the new information.

It may well be that folks that think and plan strategically are less inclined to look deeply at Agile because it appears on the surface to be tactical in nature. I know that was true in my own case. I believe that people and organizations that think strategically will bring that thinking with them into the Agile world. Having crossed over I can report that I have been able to very happily and successfully bring strategic thinking with me to the other side.

Read more about Agile in "Zero to Agile in 90 Days or Less" .

[Note: this post was inspired by Jordan Bortz’ post, Agile and Waterfall are Two Sides of the Same Coin where he states that ‘ “Waterfall” is basically a Strategic method of attaining goals, and “Agile” is basically a Tactical method for achieving goals.’ ]

Once, when I was just starting to snowboard, I was at Sugarloaf for the weekend and they had very little cover and very few trails open. But then Saturday night, they got 33” of powder. A friend and I came to a trail that was closed. It looked like a great trail; endless powder with no tracks. The problem was that it had been rocks and grass the day before and there was no base underneath, so it was just the same as riding on rocks and grass. It was not a pleasant experience. Adopting Agile without understanding it and without creating a proper ecosystem for it is destined for a similar fate.

Adopting Agile development requires breaking down mental barriers and building up new skill sets. There is nothing particularly hard about actually doing any of the Agile practices, but many of them are counterintuitive and do take a bit of practice to get used to. That said, don’t underestimate the amount of effort required. The effort required is at least on the order of taking a team which is very used to writing in C++ and moving to Java. There’s nothing particularly difficult about such a transition, but there are many subtleties which must be learned and it takes a while to build up the same base of experience.

Self Education
Before getting too far along, make sure that you have done your homework. Read other books on Agile, find other folks in your organization that have done Agile development before. Go to conferences, join the local Agile user group, become a certified Scrum Master, do whatever you do to find people that you can lean on when you need it.

Scope
Determine the best scope of the adoption. As with most things, it is best to think big, but start small. Is there a small project with no more than 12 people that is amenable to piloting something new? There are two advantages in starting small: minimizing disruption and leveraging what the pilot group learns about doing Agile in your environment when transitioning other groups.

Scouting
Agile development has certain perceptions related to it. One of the most prevalent perceptions is that it is “for small groups.” That was certainly my perception when I first started hearing about it. Another perception is that small iterations aren’t a good thing because customers don’t want frequent releases, there’s more overhead involved, the quality will be lower that way, and it makes marketing’s life more difficult.

If you just advocate Agile without knowing the landscape, you run the risk of alienating the people whose support you need in order to go Agile. Find out how receptive your organization is to going Agile. Think about who is in a position to help or hinder its adoption. Those are the key stakeholders. You will need to find out where they stand, what they like about the idea of going Agile, and what their objections are. This information will come in handy later in the adoption process.

Prepare Your Organization
Once you have a basic lay of the land, see what you can do to raise people’s awareness and understanding of the advantages and potential pitfalls of Agile. Do a presentation for folks that are interested, invite in somebody from the Agile community to do a presentation or workshop. Recommend books and websites that you found helpful.

Transition Development and QA Together
The most important component of reducing the rework period that comes from long iterations is improving your testing process. For many if not most organizations, this is the hardest goal to achieve in practice.

If you don’t have any automation at all, it is a good bet that there is an ingrained belief that automated testing is either a bad idea, doesn’t work as well as manual testing, is too expensive, or that the tools are too expensive. As a result, it may be that there are no QA automation experts in the building and possibly nobody with scripting skills in the QA group. The best course of action in this case is to concentrate on introducing the idea of QA automation.

If it is clear that there is a bias against automated testing that is too strong to overcome any time soon, another tactic is to have the development organization champion automation with an eye towards handing it over to QA once the idea catches on. A good place to start is with unit tests. It should be clear from the start that your goal is to have QA own test automation. Developers will write good tests, but they are too optimistic by nature. Developers start from the assumption that “it will work.” QA people start from the assumption that “it doesn’t work and I’ll prove it to you.” Pessimism is a good trait for a person creating tests.

Keep Your Healthy Skepticism
Think carefully about the value of each practice that you plan to adopt and make absolutely sure that it is appropriate for your exact circumstances before you adopt it. You shouldn’t be adopting practices simply for the sake of saying that you are adopting Agile practices. Every practice you adopt should be because now that you know about it you simply can’t imagine getting by without it.

Don’t Throw Out the Baby With the Bath Water
I’ve seen many first-time Agile projects fail because they threw out everything they already knew about developing software. Most of the individual steps of developing software with an Agile methodology are the same as traditional software development. You still have to talk to customers, decide what you are going to do, write code, write tests, do testing, etc. Agile development is simply a different framework for those steps. You have a business to run, and you don’t really need to introduce large and sudden risk factors. Before you decide to chuck everything that you know and start from scratch, spend some time developing your knowledge and understanding of Agile. Look closely at your existing practices and see which ones will fit well within Agile, and which ones may cause problems. Create a game plan and start out gradually.

Next: Agile Adoption Stage 2: Establishing a Natural Rhythm