In Part I, I talked about how the design of a Lego car that can transform into a robot illustrates the process of iterative design. At the beginning of the project, I had the exact requirements and they didn't change at all from the start of the project to the end. However, just having the right requirements did not mean that I could immediately design something that would meet the requirements. At the end of Part I, I had a Lego car that met many of the basic requirements, but still didn't transform. I was stuck because I didn't have the technology that would meet the rest of the requirements.
Part of iterative design is keeping your eyes open, always looking for ways to intelligently expand the solution space. If the information you need is not already at hand, trying variations of things you already know may not lead to a good solution.
Thinking Outside The Box
I had always eschewed the Bionicle Legos as “not really Lego,” so I hadn't originally considerd those. But then two things happened that breathed new life into my Lego project. My son’s third birthday was coming up, and my wife suggested that we make it a Lego birthday. I was responsible for filling the gift boxes for the guests, and I decided to start by visiting the Lego store to see what they might have.
The Lego store had a wide variety of gifts for $5 or less including Duplo dinosaurs, a pony and princess set, knights on horses, and most importantly (for the project) small Bionicle sets. I had never bought a Bionicle set before, so I had never noticed that many of them had articulated joints. The joints were ball-and-socket parts which were small and had enough friction to make them poseable. They were perfect for the project.
There is another kind of Lego set called Exo-Force which has a completely different technology for doing articulated joints and also has a part which when combined with a Bionicle part provides a third technology for articulated joints.
Creating limbs which can fully fold up is a difficult proposition at best. But, I managed to create a couple of variations that allowed me to create the next version of the project. It didn’t really meet the goals of looking like a sports car or hiding the fact that there was more to it than met the eye. It was quite boxy, had odd proportions, and some of the joints were exposed. It was also fairly fragile. But, it could transform!
Sometimes Flexibility is Constraining
Sometimes software that is too generic or two flexible creates another set of problems such as taking too many resources or being too difficult to configure for the task at hand. The knee and elbow joints of the robot form had the same problem. Because they were fully articulated they were just too darn big and difficult to hide. I knew that these joints didn’t need to have the same degrees of freedom as the shoulder, hip, and ankle joints, but had not previously been able to create a hinge that could fold flat and have the full range of motion required. I repeated the exercise of determining all of the available hinge options and found a couple of new ones. The most promising one was a dual Technic hinge which was perfect for the knees.
For the arms, this hinge was too large. But then I realized that less resistance was required for the arms and a single hinge would do. Perfect! Now I had the robot skeleton, I just needed the car exterior.
Inspired by Elegant Design
Many months later, when looking for a new Lego set for my son, I saw a new car set (#4939) that was perfect for the project. I’m not an artist by any means, so one of the challenges has been finding Lego car designs to emulate. Previously they had always been either too big or too small. At one point I had been very excited about the Ferrari that comes with both red and yellow exterior pieces, but it was double the size I needed. Set #4939 seemed to be the perfect size. It didn’t hurt that it was very cool looking and reasonably priced. I had the feeling that this was it, I was finally going to achieve the goal.
Using set #4939 as inspiration, I was able to layer a car exterior onto the robot skeleton without too much trouble. As a bonus, the head, which had previously never really worked out well, was just there. I was within reach of the goal. But still one problem remained. I couldn’t cover the shoulder and hip joints without preventing transforming. I needed something that would allow the top and bottom parts to slide out and then lock in place. I had created parts that slid from place to place before for other Lego projects, but never something that could also lock in place.
Sliding Into Home
After trying many variations using small hinges to act as the locks, I had something that worked, but it was just too big. It was also very clumsy. Luckily, I was in denial. I forged ahead anyway. I wasn’t sure what to do about it being clumsy, but I was determined to create more room in the model. The most obvious step was to do what they do to create stretch limousines: take a regular limo and stretch it out. I made the whole model a bit longer, wider, and taller while keeping the same overall proportions. I also removed all unnecessary internal parts or replaced them with parts that would still do the job but took up less space.
It wasn’t enough, and I couldn’t stay in denial about the sliding part being clumsy forever. But while I was stretching things out I observed that some of the technic parts would slide around a little bit when I exerted too much pressure on them. Of course, the part that would give depended on the amount of friction. Whichever had less friction was what moved and the other one did not. From that came the solution: two bricks together to anchor the axle, and one brick that moved freely but with enough friction that it would stay wherever it was slid to.
The slider was the last major challenge. Once I had this part, it was just a matter of trial-and-error shifting things around until it all worked. There are two sliders; one for the lower body, and one for the upper body, the following picture shows the car with the upper and lower body pulled out as the first steps of transforming.
After the car has been extended, the legs are unfolded.
Lastly, the arms are extended. The head is also fully articulated.
There is still room for improvement, but at this point I think it is more a matter of aesthetics than mechanical design. I’m happy with the current design and hope that perhaps a Lego fan with more art skills than I will take the design to the next step. My other hope is that perhaps the amazing folks at Lego will start producing transforming Lego sets. And of course, if they were Transformer branded, that would be a wonderful (but not necessary) bonus.
[Get the LDraw model for the car here . ]
Motivation, skill, and experience all play important roles in the design process, but they are no guarantee that you are going to be able to design, let alone build what you have set out to do. It almost always requires trial-and-error, serendipity, creative inspiration, research, and an open mind.
See Also: Designing Software is the Same as Predicting the Future
Extending the analogy of software architecture views and quality attributes for software process architecture, I'd like to spend some time discussing how software product lines relate to software process architecture and "common processes" across an enterprise.
Many organizations strive for standard common processes, often as part of a CMM/CMMI-based process improvement. All too often I have seen the mantra of "common process" misused and abused to make the practitioners serve the process instead of the other way around.Processes don't create great software. People and Teams do!
And while the process needs to meet the needs of the business and the needs of the customer, it has to first and foremost serve the needs of the practitioners so that they in turn may better serve the needs of the business to deliver operational business value.
Many in management seem to have the mis-impression that "common process" means "no tailoring" and everyone does everything the same way across products and projects throughout the organization. Process variation across products and projects is regarded as something to eschewed and stamped out, beating the offenders into compliance with top-down dictates and mandates and sanctions. If everyone does everything the same way then the people are more or less "plug-and-play replaceable" and can quickly and easily be reallocated to another project or product with zero learning-curve and associated start-up costs.
This is a dangerous myth that causes irreparable harm to process improvement and common/standard process efforts. Anything that focuses on individuals and interactions as subservient to common processes and standard tools is doomed to fail, and those organizations often end-up with the processes they deserve (along with many disgruntled, frustrated workers).
The purpose of such common processes and tools is not to be a rigid restrictive straightjacket for replaceable people. The intended purpose is to recognize that such people are irreplaceable and to provide a flexible knowledge framework to guide and enable them as the help each other collaborate to learn, grow, and lead in the discovery, practical application and effective execution of practices and improvements that are the best fit for a particular product, product, community and business-environment.
The intended purpose common software processes is quite simply that of process and knowledge reuse! And as such it shares many of the same fundamental problems and solutions as that of software reuse. Indeed it could even be argued that software process reuse is but a special case of software reuse. And current prevailing industry wisdom on the subject suggests that software product-lines show the greatest promise of leveraging software reuse for greatest business value.
In software reuse, we seem to recognize that "one size does not fit all." We acknowledge that even though different products, components, and platforms may share common features, that each one may have different project parameters and environments with different quality attributes and engineering-tradeoffs that need to be "preferred" and optimized that particular application: dynamic versus static, performance versus memory, storage versus latency, throughput versus bandwidth, single versus multi processing, optimistic versus pessimistic concurrency, security versus availability, and on and on.
Software process reuse is no different. Different products and projects have their own uniquely differentiating value proposition (if not there would be no business-need to attempt them in the first place). And those differentiating aspects warrant many kinds of process variation across projects, products, technologies and teams.
Those coming from a SixSigma background may point out how SixSigma strives for reducing process variation. But it's all too easy to forget the context for that is when repeatably reproducing the same output from the same inputs for the same desired set of quality attributes and tradeoffs (not to mention the "kinds" of variation SigSigma is appropriate for trying to eliminate).
So I would advocate the translation and application of software product-line practices to software processes (software "Process-Lines" or "Process Families" if you will) and the treatment of such common processes as first class architectures that need to accommodate the views and perspectives of ALL their critical stakeholders, and which should identify their essential quality attributes and tradeoffs, approaches to managing commonality and variability, and apply appropriate patterns and tactics (such as modifiability tactics for software processes and projects) to meet those objectives.
In light of the above, Lean & Agile software development define an architectural style for such process families and their architecture. Lean and Agile principles identify some of the critical process-quality attributes for such efforts. And the corresponding enterprise and its product offerings and their market segments may identify additional quality attributes that needs to met (such as security, regulatory auditability/compliance, large-scale and/or distributed projects and teams, etc.)
In the lead-up to the debut of the movie “Transformers,” there was already plenty of Transformers merchandise available. My son, who was 2 ½ years old at the time was instantly entranced. While my wife was strolling him through the bookstore he grabbed a Transformers calendar without her noticing. When she got to the register he then used his charms to convince her to buy it.
The Challenge
This got me to thinking. Wouldn’t it be fun to build my son a car out of Legos that could transform into a robot? After all, how hard could it be? I decided to give it a shot and I set myself the following design goals and constraints:
- Build a Lego model that has two forms: a sports car and a robot
- Folks looking at it agree that each form is clearly recognizable as that form
- On casual inspection, it doesn’t look like it transforms
- 100% Lego, no other parts, and no glue
- Easily held in one hand
- Strong (parts don’t fall off easily during use)
- Robot form can stand on its own
- The model can easily transform from one form to the other
- Model is completely connected and transformation does not require the addition or removal of any parts
- Fully articulated
As the project progressed, I wasn’t thinking of it as an exercise in iterative design or a formal project, it was always just something I did for fun in my spare time. I didn’t realize the link to iterative design until I started talking to people about the history of the project.
Transforming Requirements Into Working Software
When working with software, even if you have clear requirements that you have validated with users, it is not always clear how you will implement them. The search for a solution has two possible components: thinking about how to do something and implementing something. When thinking about how to do something, that’s really just fantasizing. Of course, it is fantasizing that is focused on producing something real, and it is more likely to produce something real than fantasizing about winning the lottery, but it is still fantasizing.
Requirements, analysis, specifications, and design are all examples of fantasizing. Yes, it is useful to do it, and you will often hit on solutions or eliminate problems much more quickly and much more economically than creating something real without giving it the same amount of thought, but fantasizing is only an approximation of reality, an interlocking web of educated guesses. Not until you actually attempt to create something real will you find out how all of the individual requirements interact and the real costs and difficulty involved. You don’t always find a clear match between requirements and implementation that is cost-effective to implement. In this case you have four paths forward: implement with the closest match and hope for the best, search for new possibilities, invent new possibilities, or shelve the project.
Transforming Lego Car, Iteration 1
The first attempt to build the transforming Lego car didn’t go very well. I was able to create a reasonable looking car which had lots of interior space for the mechanics needed to transform, but creating the sliding parts and the joints proved to be more difficult than I had imagined. I had lots of different hinges, flat parts, special joints, and Technic parts, but all of the possibilities I came up with were either too big, too fragile, too loose, or didn’t support articulation well enough. I was stumped. After a quick trip to the nearby Lego store to see if there was something I had missed, I reluctantly decided to shelve the project.
From a requirements perspective, I had satisfied requirements 2-6, which is half of the requirements. Of course, it was easy to satisfy "doesn't look like it transforms" because it didn't transform at all. :-) From an Agile perspective, the car was fully functional as a car in its own right, and I had discovered lots of valuable information that would help with later iterations of the car.
Little did I know that a hidden bias had kept me from seeing what was right in front of me. It wasn't until a couple of months later that serendipity broke through that bias and brought together the necessary ingredients for the first transforming version of the car.
Next: Part II (of 2)
This is an evolving web version of a book that I'm writing titled "Zero to Hyper Agile in 90 Days or Less." If you have the time to read it straight through, that would be great. If you don't have the time or inclination for that, how about picking a section or two and leaving your frank feedback? Any and all feedback will be appreciated. Many thanks to all who have commented so far!
The end of each section of the "book" links to the next section.
Table of Contents
Introduction
Agile in a Nutshell
What is Agile?
Primary vs Secondary Benefits
Agile Development, What's in it for Me?
The Problems With Traditional Development
Dev Team Having Performance Problems? Try Niagra!
The Chinese Finger Trap of Traditional Development
The Chinese Finger Trap: Architecture, Development, and QA
Short Iterations
Unconciously Agile
Natural Law
Agile Development is People Oriented
The Agile Waterfall
Hyper Agile Development
There is no Bug. It is Not The Bug That Bends, it is Only Yourself
Apply Elegant Architecture to Your Dev Team
How Agile Solves Problems
Hyper Agile Lifecycle
Agile Product Management
The Role of Defect Management in Agile Development
Designing Software is the Same as Predicting the Future
Frequent Releases Improve Code Quality Faster
The Role of QA in an Agile Project
The Role of QA in an Agile Development Project Part II
Integration
Multi-Stage Continuous Integration
Releasing
It is Better to Find Customer Reported Problems as Soon as Possible
Customers Don't Want Frequent Releases
Tuning the Frequency of Your Release
Complimentary Practices
Refactoring and the Law of Unintended Consequences
Bibliography
More to come!
Yet Another Fad
We’ve heard it all before. The Segway, UML, Virtual Reality, TQM, object-oriented databases, Artificial Intelligence. Agile’s no different, right? Just the latest fad? That’s what I thought.
Back in 2005 I had my first run-in with Agile development. At a speaker’s reception, my blissful ignorance of Agile came to an end. I was surrounded by Agilists. They looked perfectly normal to me, but then they started to talk about the advantages of keeping track of requirements on 3x5 cards, pair programming, and lots of other crazy sounding stuff.
3x5 cards! Are you kidding me? What is this, the seventies? How can people who make a living writing software advocate the use of 3x5 cards? And what about pair programming? Two people sitting together all day sharing the same keyboard and mouse switching between coding and sitting on their hands, having to hear phone calls about the results of medical tests. Yuck!
I wasn’t just skeptical, I was outraged that all these people were trying to push this nonsense. Keep in mind that I work at a company that provides software tools to companies that are looking to improve their process. On the surface, Agile seems to be diametrically opposed to that. I felt that it was time for somebody to step forward and do something about it. I nominated myself. To gather evidence I read the books, talked to the experts, and attended the seminars. But then one day a funny thing happened. I had an aha moment and I realized there might actually be something of value buried under all of the rhetoric.
Traditional Development Scenario
Here’s what I realized. Let’s say you want to add new features to stay competitive. In a traditional project, you have a known timeframe for major releases. Too short and you’ll spend too much time on overhead, too long and you’ll miss opportunities. For the sake of argument, let’s pick a timeframe of six months and say that allows you to provide three “big features.” Marketing says the three features with the highest ROI are a Facebook plug-in, a Second Life plug-in, and an RSS feed plug-in.
Midway through the design process, marketing announces that the Second Life plug-in is not as marketable as they had hoped and iPhone support is showing signs of becoming very lucrative. You think, oh well, nothing we can do about that now. Just after you finish coding marketing declares that the Second Life plug-in is going to be a complete flop and when can they get iPhone support?
Now that the functionality has settled down, QA starts writing tests and running them. Planning and development took longer than expected and the release deadline is looming. Testing time is compressed, QA concentrates on the most critical stuff and gives the rest a spot check. Here’s a mystery. If your full test cycle takes two days, why does testing take a month? The answer is that you aren’t really doing a month of testing. Problems have been creeping in all along the way that you are just now finding out about and it takes many test/fix cycles to expose them and fix them. Once the find/fix rate gets down to an acceptable level, you declare victory and deliver the new release.
Now Let’s Try Agile
For simplicity, let’s use two month iterations. You start with the Facebook plug-in. You plan. You design. You create a test plan while the code is being written. You discover potential problems and deal with them. You automate those tests while the code is being written. As the code is written, it is integrated, built and tested continuously; problems are found and fixed immediately. At the end of development, the only problems that remain are the ones that could only be found at the end of development. If you wanted to, you could cut a new release with very little overhead if you wanted to.
Marketing announces that the Second Life plug-in is not as marketable as they had hoped and iPhone support is showing signs of becoming very lucrative. So, you start on the RSS feed support. Now marketing says the Second Life plug-in is worthless but iPhone support is hot, so you add iPhone support. During the whole process, you kept your options open. At the end you have produced more business value than using the traditional process, and you were in a position to start realizing that value much earlier.
I also realized that you don’t need to do any of those things that at first really turned me away from Agile. You can still get the primary benefits without resorting to 3x5 cards or pair programming and you don’t have to do frequent releases. These things are optional and you can use them or not according to your preference and circumstances. My advice to you is don’t pass by the opportunity to find out if Agile can work for you the way it has worked for others. I’m not suggesting you throw out your skepticism; try a pilot project!
Next: No Really, What is Agile?
In parts 1 through 3 of this series, I described how Multi-Stage Continuous Integration can be used at a team and multiple-team level. In this post, I will describe how Multi-Stage CI can also encompass other development stages.
The Development Hierarchy
So far, we’ve split the mainline up into a hierarchy of branches. Each developer has their own workspace and is part of a team, each team has its own branch, and each team branch is based on and merges back to an integration branch, which may be the mainline. Sometimes work is organized at a feature level instead of at a team level. For simplicity, I’ll just refer to teams. Each part of the hierarchy corresponds to a stage of development. This hierarchy enables multiple parallel development pipelines where changes move from a low level of maturity to a higher level of maturity and the work in the parallel pipelines is continuously integrated.
We’ve covered three stages of development: individual developer coding, team integration, and full project integration. But there are many more stages in a typical project lifecycle, even when you are doing Agile development. Traditionally, the stages of the lifecycle occur in several different places including project plans, the SCM system, and the issue tracking system. This adds development stages such as: code reviewed, system tested, UAT passed, demoed, etc. Each of these stages can be added as a branch to your branch hierarchy.
When using Multi-Stage CI, unlike the typical use of branches, changes do not linger on a branch any longer than required to pass through the stage that branch is associated with. The difference between any given branch and its parent in the hierarchy cycles rapidly between almost nothing and nothing. The goal is to push changes through the hierarchy as rapidly as possible.
When working within a development hierarchy, the impact of any given change is limited in scope. If there are 4 members on a team, then only 3 other people are affected when a change is merged into the team branch. Because changes are made on the team branch only when the developer has things working in their own version, changes at the team level are made less frequently and have a lower probability of destabilizing the build. Once the team branch is considered stable, changes are then merged into an integration branch which consolidates the work for multiple teams. Again, changes are made less frequently and have a lower probability of destabilizing the build. Thus, the graph at the integration level shows fewer changes and the changes are smaller. As you proceed towards the root of the hierarchy, changes are less frequent and less likely to destabilize the build until finally you reach the root where you have code that is always shippable.
Getting Started
Getting to Multi-Stage CI takes time, but is well worth the investment. The first step is to implement Continuous Integration somewhere in your project. It really doesn’t matter where. For a recommendation on an excellent book on Continuous Integration, see the bibliography. The next step is to implement team or feature based CI. Once you have that working, consider automating the process. For instance, you can set things up such that once CI passes for a stage, it automatically merges the changes to the next level in the hierarchy. This keeps changes moving quickly and paves the way for easily adding additional development stages.
I’ve seen Multi-Stage Continuous Integration successfully implemented in many shops and every time the developers say something like: “I never realized how many problems were a result of doing mainline development until they disappeared.”
Next: It is Better to Find Customer Reported Problems As Soon As Possible
[Note: this is a re-mix from what I originally posted on my blog experiment on Wordpress]
In the movie “The Matrix,” when Neo goes to visit the Oracle, he has a conversation with one of the potentials who is bending a spoon with his mind. The boy explains to Neo that “there is no spoon” and that “It is not the spoon that bends, it is only yourself.” While this is rather mystical, it occurred to me that this is a good way to think about bugs!
Often, getting a bug to bend to your will (go away), is about as easy as bending a spoon with your mind. For instance, consider the problem of coloring maps such that no two contiguous regions are the same color. Let’s say that in your experience, you’ve only ever needed 3 colors. You need to color maps using software, so you write it with the assumption that you only ever need 3 colors. You may encounter some bugs which end up putting the same color on two contiguous regions in a map that can be colored with just 3 colors. Eventually you will find and fix those bugs. But if you run your program on a map that requires 4 colors, no amount of bug fixing your program that uses 3 colors will ever work. You must realize that your assumption that 3 colors will always work is wrong.
Another problem with bugs is that sometimes fixing the bug doesn’t actually fix the root problem. You think you know what the bug is, so you make a change so that the software now behaves as desired under the given circumstances. However, there are still 5 other ways that the problem can manifest itself and you didn’t fix any of those. Plus, you’ve introduced a new problem. You don’t know about these 6 problems yet, but they are there. You’ve only fixed the symptom, not the cause. You have not bent the spoon.
Even when you find and fix the root cause of that bug, you will write more software using your existing process and it will have new bugs to replace the old bugs. The progress was an illusion. The spoon is still unbent. If anything, it is even more unyielding than before.
It is tempting to think of a bug as something that creeps into the software after it is written, something that is separate that can be isolated and removed. But really, bugs are introduced during the process of developing software. They are not separate and isolated, they are actually flaws in the process itself. The problems lie within. Only when we acknowledge this ("there is no bug") and accept it can we hope to affect real change. Agile development is an excellent platform for seamlessly incorporating the required introspection and continual improvement.
I’m not saying that real software with zero bugs is an attainable goal, only that you consider a change of perspective in order to bend the spoon a bit. The ultimate goal is to make behavioral and process changes such that no bug is found in the software that you deliver.
Next: Apply Elegant Architecture to Your Dev Team
In Part I of this series I introduced the idea that traditional development is like a Chinese Finger Puzzle. Solutions to problems often make the problem worse instead of better. This post will cover two more areas that illustrate this phenomenon.
Future-Proof Architecture, Major Releases and Large Feature Sets
A common sentiment when using traditional development is that because it is so hard to redo architecture, because it takes so long to make changes, because it takes so long to produce a release, we had better get the architecture right the first time. We had better take into account all possibilities and eventualities. Since we are going to spend a long time on the architecture, we better plan to do a lot of features for the release. All of those features mean we need to take more variables into account and have an even more thoroughly thought out architecture. In a traditional project, the interplay between architecture, release timeframe, and feature set create a vicious cycle which tends to increase the scope of all three.
In an Agile project, the practices of short iterations and iterative design reinforce each other to keep the amount of architecture that is required as small and simple as possible yet no less than necessary. The product backlog and short iterations keep the feature set focused on those features which will produce the highest customer and market value instead of trying to build a product which takes into account all possible future requirements.
Read more about iterative design: "Designing Software is the Same as Predicting the Future."
Separation of Development and QA
A common pattern in traditional development is the separation of development and QA. Of course everybody always says that “QA is everybody’s job” and “you can’t test quality into a product” and “QA should be involved right from the start,” but what usually happens in practice is that QA occurs at the end. One contributor to this problem is that “during development” the product isn’t stable enough to test and you want to get the most bang for the buck out of your testing because it is so expensive to do and it takes so long to qualify a release. The natural result from these circumstances is the separation of development and QA which is constantly reinforced. Using traditional development patterns, all efforts to unify development and QA are inevitably defeated. Nobody is consciously trying to make it happen, it just happens.
But sometimes it is also on purpose. The separation of development and QA can feel “more honest.” Sometimes development doesn’t want QA to know what is going on because they are sure that they can catch up and fix things and then deliver something to QA which is good. But in order to get the benefits of QA without the help from the official QA folks, development ends up having to do QA themselves which reduces the time spent on development. In any case, traditional development practices tend to separate development and QA.
Agile practices do just the opposite, they tend to bring development and QA closer together. The writing of tests for a particular change is done either before the coding starts or during the coding, not in a separate phase at the end of all coding. Tests are run constantly using the practice of Continuous Integration. QA doesn’t have to wait until the traditional code-freeze to start the bulk of their work. This creates a positive feedback loop where QA gets to show its value on a regular basis, development spends less time fixing things at the last minute, QA learns more about the product, and development and QA become even closer.
Next: Short Iterations
Aside from personal preference, the only reason to make a change to the way you develop software is to realize a benefit. It could be to increase quality, customer satisfaction, employee satisfaction, productivity, or profits. In the end, these should all result in increased profits. If profits are not an important part of your organization, for instance you work at a not-for-profit, then another way to look at this is reducing expenses. For simplicity, I will focus on profits.
What do Profits have to do with Me?
There are really only five ways to increase the profitability of a business based on software development: reduce costs via outsourcing, reduce headcount, reduce other expenses, increase productivity or increase revenues. Reducing expenses can only go so far. The most expensive part of software development is the people. Thus, one of the most successful ways to increase profits is to increase the productivity of the software development team. The other way to significantly increase profits is to increase revenues. Agile development can do both.
Why should you care about the profitability of your company? There are a number of reasons. If you are a stockholder, you benefit directly. The more profitable a company is, the more likely it is to be secure. The more profitable a company is, the more likely it is to embark on exciting new opportunities which mean more opportunities for you. It also means that the company is more likely to invest in its development infrastructure.
If you were smirking while reading the previous paragraph because you know that extra profits will never be invested into the development organization, perhaps you are working for the wrong company. Or perhaps nobody ever thought of investing more into the development organization and a suggestion or two in the right place is all that is needed. In either case, you can still take to heart the idea of re-investing profits into the development engine and work on strategies to make it happen.
Your Development Infrastructure is Part of Your Working Conditions
The development infrastructure is really no different than the general company infrastructure which includes your cube or office, the carpet, the artwork on the walls, the company cafeteria, your phone, your computer, and the company location. These are all part of your “working conditions.” If you have a computer that is 5 years old, your working conditions are not as good as if you have a computer that is only 2 years old. If you are writing in C rather than C++, C# or Java, your working conditions are sub-optimal. This extends to all aspects of the development environment, from development language, to build system, to build farm, to issue tracking system.
Your development process (regardless of how it is implemented), is also part of your working conditions. If as a result of your development process you regularly end up redoing work because problems weren’t discovered until just before release, or projects get cancelled or shelved, then this is also likely to reduce productivity and job satisfaction. As this process improves, so do your working conditions. The smoother it operates, the more pleasant your working environment will be.
To be sure, a “perfect” process does not guarantee happiness, success, or the absence of problems. You still have to debug complicated problems, port to new platforms, deal with unforeseen circumstances, etc. However, the state of your process impacts the efficiency with which your effort is applied. For instance, if your process is perfect and completely frictionless, then 100% of your effort will be applied to the work you want to do. If it is rife with problems, it may mean that only 50% (or less!) of your effort is applied to the work you need to do. If there are problems with the process, then you are already expending effort which is essentially wasted. You would be better off investing some of that effort in removing the problems permanently instead of losing it to friction on a regular basis.
Potential Benefits of Adopting Agile
Here are some reasons for you personally to consider and/or champion implementing Agile development:
• Getting a raise and/or bonus
• Increasing revenues
• Decreasing costs
• Increasing market share
• Increasing customer satisfaction
• Experiencing less stress
• Improving your working conditions
• Actually using all of your vacation time
• Advancing your career
• Spending more time working on cool stuff
• Having the resume that gets you your dream job
• Learning new skills
• Having more discretionary income to buy cool stuff
Next: Dev Team Having Performance Problems? Try Niagra!
In order for something to become widely adopted, it has to provide an advantage that is significantly greater than whatever it replaces, and the advantage has to be easily accessible to a mainstream audience. Another way to describe “easily accessible” is to use the term “usability.” That which has high usability is more likely to become mainstream.
There’s a lot of literature on the usability of both software and everyday objects. But what does usability mean when applied to the process of software development itself? If we think of the myriad steps we use to produce software as the implementation of a software development process, then we can think of that implementation as software and consider its usability.
To consider the usability of a particular process, we first have to have a measuring stick for usability. Here’s a set of categories loosely based on some of the chapters from Joel Spolsky’s excellent book “User Interface Design for Programmers” and also adding in “visibility.”
Feedback
We get feedback our whole lives. When we first come into the world, we need constant feedback in order to learn how to survive. Touching something hot is painful, going without food is painful, falling down stairs is painful. In school we get feedback as to which subjects we are good at and which subjects we are not so good at. We can use this feedback to decide what to do more of, what to avoid and as an aid to improve, if we are able to.
The software development process, along with its rules, is just one process that you are a part of. Process, rules, and laws are all around you. You don’t have to consciously think about them but you are constantly reacting to them and they are second nature to you. On your way to work you drive on the correct side of the road, you stop at stop signs, and you stay in your own lane. When you go to lunch, you pay for your food. After work you go to your house instead of going to somebody else’s house.
It is easy to remember the laws, rules, and processes because you interact with them every day and you get feedback in the form of reminders and possibly even unpleasant consequences when you go astray. A great example of this is the rumble strip that has become commonplace on the side of highways. If for some reason you start to drift off the road, you’ll hear a loud warning noise as your tires hit the rumble strip. If you are really violating the law, you’ll hear an even louder sound accompanied by flashing lights. This is also known as feedback.
With traditional development, the development timeframe is much longer and thus the feedback loop is much longer. With Agile, the feedback loop is usually no longer than a month and often on a weekly basis.
Consistency
An important part of absorbing all of these laws, rules, and processes is that they are clearly defined, easy to understand, and easy to remember. For instance, once you have mastered basic physical laws such as the fact that running at full speed into a door is painful, it is easy to understand the reasoning behind having an upper limit on the speed of vehicles on public roads. You may disagree with that limit, but you can generally guess what the limit is for a given stretch of road and it is posted at regular intervals to help you score your guesses. Sometimes there is even a referee.
In traditional development, the long timeframe makes it difficult to absorb what the actual process is and usually everybody has a different mental model of that process. When people have different mental models for the same process, bad things happen. In Agile development, the process is far simpler and much easier to absorb. You have a chance to see the full process from start to finish on a very regular basis.
Expected Behavior
When something happens unexpectedly, it interrupts our train of thought and slows us down. It is a little bit like taking a wrong turn and becoming lost. We have to take time to figure out where we are and how to get back to familiar territory. Most of the time, when working on a software project we are concentrating on a single task that we alone are responsible for.
Consider software development from your own perspective. I’ll bet that you identify with most or all of the following statements:
- When working on a software project on your own, you make changes and use the new version right away.
- One of the reasons that you got into software development is the fact that you can make a change and see the result right away.
- When you have something new working for the first time, you want to show it to somebody.
- It isn’t easy to create something that the customer thinks is exactly right the first time.
- Even when creating something for yourself, it isn’t easy to create what you want the first time.
With traditional development, there is generally much more process to cope with the greater complexity of trying to develop a much larger increment of functionality over a much longer time period. Because of the large gap between how one would develop software on one’s own and how one develops software as part of a much larger process, the steps in the process are often counter-intuitive and thus it is much harder to establish a natural rhythm.
Visibility
It is hard to know what to do next if you aren’t sure where you are. Visibility is akin to project status. Do you feel like you know exactly where you are and what to do next at every step of the process? Or do you feel pulled in a million different directions at once and lose track of what you’ve already done and what you need to do next? If you do feel like you’ve finally “got” how software is developed in your organization, how long did it take to get to that point?
In a traditional project, it is almost impossible to see at a glance what the real project status and progress are. You know that you won’t really know until sometime after code freeze. In an Agile project, the next milestone is much closer, so it is much simpler to see where you are. Granted, if your real goal is six iterations out, you don’t have the same sort of visibility into the next five iterations, but the chances of risks hiding out until code freeze is substantially reduced.
The Usability of the Software Development Process
Based on just these few criteria, it is clear that traditional software development suffers from poor usability and hinders the ability of otherwise competent people to get their work done. While it is true that people can succeed on projects that have very long timeframes and involve a high degree of stress at the end, that mode of working does not align well with our strengths as human beings.
People are much more productive and much less prone to error when they work at a constant and sustainable pace. This also means that when there are unforeseen circumstances, people are more likely to be able to respond well.
Agile Development provides frequent feedback via short iterations, a simple process which is more in keeping with natural human rhythms and capabilities, and significantly more visibility into project status and progress. This creates a highly usable and people-oriented software development environment.
Next: It is Not the Bug That Bends, it is Ourselves
Reddit is a great way to surf the net for things that you are interested in, and you can participate in determining what's good and what's not. Now you can surf using reddit for all things Agile! Check out the new Agile reddit!
After my previous postings on Software Architecture Views and Perspectives, Software Architecture Quality Attributes and Software Modifiability Tactics, the question remains as to what all this has to do with Agile processes or with CM.
Well, about a year ago I wrote that Software CM is NOT a Process! ...Software CM creates the medium through which software development changes & activities must flow. Therefore, Software CM is the intentional architecture of software development change-flow.
The elements of this Software CM architecture include practices, tools & technology, teams & organizations, valued deliverables & intermediate work-products, changes to and assemblies of these deliverables & work-products, and the set of needed status/tracking reports & measures.
So now I want to take the perspective of Software CM as an architecture, and I want to consider questions like:
I think I answered the first of these questions in my Dimensions and Views of SCM Architecture. I take the perspective of a 4+2 views model comprising Product, Project, Evolution, Environment, Process (+1), and Enterprise (+2). These views straddle the conceptual and physical aspect of both the content and context of the different kinds of "containers" that are to be managed and interrelated. And the dimensions of SCM complexity that prove most challenging are those of scale and diversity, and of differences between artifact change/creation time and decision binding-time.
The next question involves translating what Availability, Modifiability, Performance, Security, Testability, and Usability mean for a "process" architecture. I'll make an initial stab at that (feedback is encouraged):
What are the "quality" attributes of an "agile" process? Do they include ALL of the above? what about: adaptive? lean? result-driven (i.e. "working software")? self-organization? iterative? collaborative?
How about some of the traditional "quality" attributes of a CM system: traceability (vs. transparency?), reproduceability? repeatability?
Have you ever thought “why do the companies I work at have so many problems with developing software?” Well, I can tell you from interacting with literally thousands of software development organizations that most development organizations have problems with reliably and predictably producing high quality software. What I hear over and over again is “if only we could hire the right people” or “if only people could be more disciplined.” The other thing I hear over and over again are various tweaks to traditional development that people believe will fix things but that “other people just don't get it.”
Well, there are only so many people out there to hire so unless we find some magical place to hire more of “the right people” from, we are going to have to figure out how to use the people we have. And I hate to break it to you but you just aren’t going to get more discipline out of people. Whatever discipline that exists today, that’s all you are going to get. So what’s left? Changes to the way we do things.
Let’s say there was some combination of techniques which would mean that most development groups could use traditional development to reliably and predictably produce high quality results. Let’s call it the “Niagra” method. Furthermore, you (or whoever wants to make this claim) get to define the Niagra method. It works every time. Just use Niagra and performance improves overnight (when used as directed).
If the Niagra method existed, it would have become widespread by now. It would have become the #1 prescribed solution to project performance problems. If it existed and produced the claimed results, people would start referring to it, and it would spread rapidly. That’s how C++, Java, and HTML became mainstream. People rapidly adopted them because they provided benefits that anybody could realize. In fact, there are many proposed Niagra methods, but none of them have become mainstream because they only work under special conditions.
This raises an obvious point. If traditional development is so bad and is such a big failure, then why has the software industry done so incredibly well? The software industry is an incredible success, that’s absolutely true. It is incredible how much of our daily lives runs on software and how much software has positively influenced our quality of life. The benefits of software, when it does finally ship and it does finally have a high enough level of quality are worth waiting for. Otherwise, of course, there would no longer be a software industry.
There have been incremental improvements to the software development process which have produced incremental improvements and have become widespread. Examples are: the use of software development tools such as source control and issue tracking, breaking down large tasks into small tasks, nightly builds, one-step build process, moving from procedural programming to object-oriented programming and many others. The important point here is that each of these improvements have become widely adopted and made things somewhat better, but still didn’t solve the root cause. The process is still unpredictable and unreliable.
This is akin to the O(n) approach to problem solving. If you have an algorithm which takes 100*n^2 operations, then getting that down to 50*n^2 is a good thing, but changing the algorithm to (for instance) be 200*n is much better. To date, changes to the software process have been of the first variety, shrinking the constant.
Perhaps Agile development is the answer. But isn’t Agile “yet another fad” which will blow over any day now? After all, we have had CMM, 9001, TQM, Critical Chain, and many other “next big thing” ideas in the past. What makes Agile any different? Previous ideas have all been centered around the idea that the framework underlying traditional development is fine, it is the implementation, discipline, people, or management of the people that is the problem. No, it is the underlying framework that is the root problem.
Certainly, people issues are important. Getting a team gelled and working well together with the right combination of skills is essential. But the success rates of these teams isn’t exactly stellar either! Yes, it is better, but they have similar problems with predictability and quality. A root problem remains. I do not claim that solving this problem means that suddenly teams that have insufficient skills and hate each other will start pumping out high quality software on a regular basis. But I do claim that Agile development has the ability to dramatically increase the potential of any team. And to truly succeed you have to do both: create a good team, and use Agile development.
Let's take a deeper look at the fundamental problems with traditional development.
Next: Traditional Development is a Chinese Finger Trap
Have you ever wondered how your organization compares to other organizations when it comes to the process of developing software? Do you feel like there is an area that really needs improvement, but everybody else says "that's the way everybody does it?" While looking at what is involved in mainstream Agile adoption, I realized that it would be useful to create a list of the current mainstream software development practices for comparison purposes and I thought other folks might find it useful as well.
I define a practice as being mainstream if it is a practice that most people would agree that most people do. It may not be that they do that exact practice, they may do something equivalent or further along the spectrum of what constitutes a good practice, but they at least go to the level described.
The word "practice" is the key word here. This is about what people can be observed as actually practicing in their day to day real work. Things which are described by policy or documented as the correct way of doing something but avoided and worked around don't qualify. A mainstream practice is something that is in common usage that people do because they believe the value is worth the effort and not doing it is sort of like saying “electricity and hot and cold running water aren’t for me, I much prefer candles and fetching water from the well.”
While each of the following individual practices is considered a mainstream practice, that does not mean that it is mainstream to being doing all of the following practices. That is, in some of the areas that these practices cover, any particular development organization may be operating at a lower level than these practices, but for any particular practice, most development organizations operate at or above the level of these practices.
Overall
Basic flow – the common development activities are: talk to customers and/or do market analysis, document market needs via requirements, create a design, implement the design, write tests, execute the tests, do find/fix until ready to ship, ship.Preparation
Requirements documented using Microsoft Word or Excel – while there are actually quite a few off-the-shelf requirements tools available, most people do not use them. The most common method for documenting requirements is via Word and Excel.Recording of defects in Bugzilla – there are very few organizations which aren’t using at least Bugzilla to track bugs and Bugzilla is definitely the most popular choice. And recording of defects is pretty much as far as it goes. Enhancements, RFEs, requirements are tracked separately.
Basic initial project planning – this is the simple act of picking a bunch of work to be done, estimating it, dividing it up among the folks that are available to do the work and determining a target ship date. Some teams use MS-Excel, some teams use MS-Project. This does not mean sophisticated project planning. The extent of re-planning is a simple “are all tasks done yet” with the occasional feature creep and last minute axing of important functionality at the last minute.
Basic design – prior to coding features or defect fixes that will take more than two days to do or are highly impactful, a design document is created, circulated, discussed, and updated.
Development
All source code is stored in a source control system – this is not to be confused with “all work is done in a source control system.” It is actually surprisingly common to see the source control system used for archiving of work done rather than as a tool for facilitating the software development process.Source control and issue tracking integration via a hand-entered bug number in the check-in comment, without enforcement – while there are certainly more sophisticated integrations, most people at least type in the bug number when checking in changes so that they can later find the changes associated with that bug.
Defects have a defined workflow that is defined in and enforced by the bug tracking system – even if it is as simple as Open, Assigned, Closed.
Mainline development – all developers on a project check-in to the mainline (aka the trunk).
Refactoring – while the term “refactoring” has come to mean just about any change to the software, the idea that code should be periodically cleaned up and simplified is pretty much universal at this point.
Nightly build – the entire product is built every night. This does not necessarily mean that tests are also run automatically or that there is an automatic report of the results, just that there is a nightly build.
Quality
Mostly manual testing – this one is the hardest to understand. The process of testing software is one of the most backwards parts of software development.Unit tests – while the overall testing of software is still mostly manual, unit testing has caught on rapidly in a very short span of time.
Using defect find/fix rate trend to determine release candidacy – this is not actually a particularly good practice, but it is what most people do.
Separation of developer, test, and production environments – you might think this is so obvious it isn’t even worth mentioning, but this principal is violated enough that it is worth mentioning that it is actually a mainstream practice. I mention it to emphasize that if you aren’t doing this, you really need to.
Releasing
Basing official builds on well-defined configurations from source control – the official process for producing production builds includes the step of creating an official configuration in the SCM system and using that configuration to do the build.Releasing only official builds – all builds that go into production are produced using the official process.
Major and minor releases - creating a major release every 6 to 12 months and minor and/or patch releases on a quarterly basis.
What do you think? Are these right on target or way off base? Do you think there are any missing areas? Do you think the mainstream practice level is higher or lower for some of these? Let me know what you think!
Next: "There is no Bug. It is not the Bug That Bends, it is Only Yourself."
The Agile Manifesto, created in 2001, is an attempt to describe a better set of values and principles for developing software. This may sound a bit Zen, but the Agile Manifesto does not actually describe the best way to develop software. Or, if you prefer, it does not describe the best principles to develop software. It is only the attempt to describe the way or the principles.
Another way to describe this is to look at what Agilists say about requirements documents. Written requirements do not actually describe the best product for the customer, they only attempt to do so. In fact, the best product for the customer can only be found via the constant feedback of discussion and delivery. I believe that one of the core values of Agile is the constant search for better software development techniques and thinking. What is the Manifesto if not a user story that attempts to describe the needs of software development itself?
Therefore, doesn’t the Manifesto imply, no demand, that it be constantly reviewed and, if needed, updated? Isn’t that the whole point of Agile? Doesn’t the fact that this is not happening point out a major problem that needs to be immediately addressed?
What happened to changing requirements? The requirements of software development itself haven’t changed since 2001? Nonsense! How can we face the skeptics if we don’t even believe in the Manifesto enough to change it? Shouldn’t we be embracing the idea of updating and revising the Agile Manifesto?
You could take the position that all of the books and blogs and seminars that have been produced since 2001 have turned the Manifesto into merely a historical document. But in practice the Manifesto is much more than a historical document. It is the first footnote or reference in most books, papers, and articles on Agile. Journalists and other media folk refer to it directly and often by URL all the time. As a result, many people read it.
It is perfectly reasonable for somebody new to Agile to assume that Agile Manifesto 1.0 is what we mean when we say “Agile” and that it will be one of their first stops on their trip to understanding Agile. New people read it every day. There are new signatories every day. Consider the impression that people have today when they visit the Manifesto and see a dusty web page that hasn't changed in seven years. Now consider the impression that people would have if they saw a revision history, a forum for proposing and discussing proposed updates, and the date for the next meeting to produce the next iteration of the Manifesto.
I personally have visited the manifesto at least 10 times in the past year including today to see if maybe it was versioned. It isn’t. Each time I visit the Manifesto I think “I’ve changed my thinking on this or that, am I ready to sign it yet?” The answer is still no. I realize now that one reason is that it doesn’t take its own medicine.
The idea for writing this post came as a result of participating in the proposal submission system for the Agile 2008 conference. There is a proposal to discuss revising the Agile Manifesto (account creation required). I have to say that the resistance to revisiting the manifesto is a bit shocking to me. It reminds me of many of the things that I thought that Agilists don’t like such as requirements documents that are set in stone, resistance to change, long iterations, rigidity, etc.
I can’t speak for others, but it doesn’t matter to me if I’m personally involved in the review and revision process itself. For me, it is about practicing what we preach and setting an example for those that are still doing traditional development or doing Agile poorly (in my experience most projects still fall into one of those two camps, unfortunately).
Let’s say the existing Manifesto is a historical document. Let’s enshrine it, put it behind glass, do some sort of ceremony etc. But to me, the fact that there is no version 1.1 or 1.0.1 or 2.0 or anything other than “the document” says to me that something is wrong here. Or conversely, creating a new version, regardless of the size of the change would be a good thing and indicate that we are in fact practicing what we preach.
For those that think it is futile or noise, why comment at all? Why does it matter? Worse case: the folks that want to see this happen go and waste their own time and the signatories say “we aren’t changing it.” So what? What’s the big deal? Is it somehow sending a bad message that some folks think that it is a bit odd that the Manifesto itself isn’t responding to change?
The treatment of this document as some sort of holy artifact feels to me like non-Agile thinking. It feels like a discussion that one would have with somebody about a giant standards document or process document. That reminds me, I did just hear such a discussion at Deep Lean in which somebody in the audience was saying “the coding standards document can’t change! It is a standards document.”
Let’s respond to change instead of following a plan. Let’s talk about if and how the Agile Manifesto should change to take into account all that the Agile community has learned since 2001 and set a good example for those that have not yet discovered the benefits of Agile. What do you think?
You may also be interested in a recent post on Deconstructing the Agile Manifesto.
After my previous postings on Software Architecture Views and Perspectives, Software Architecture Quality Attributes and Software Modifiability Tactics, the question remains as to what all this has to do with Agile processes or with CM.
Well, about a year ago I wrote that Software CM is NOT a Process! ...Software CM creates the medium through which software development changes & activities must flow. Therefore, Software CM is the intentional architecture of software development change-flow.
The elements of this Software CM architecture include practices, tools & technology, teams & organizations, valued deliverables & intermediate work-products, changes to and assemblies of these deliverables & work-products, and the set of needed status/tracking reports & measures.
So now I want to take the perspective of Software CM as an architecture, and I want to consider questions like:
I think I answered the first of these questions in my Dimensions and Views of SCM Architecture. I take the perspective of a 4+2 views model comprising Product, Project, Evolution, Environment, Process (+1), and Enterprise (+2). These views straddle the conceptual and physical aspect of both the content and context of the different kinds of "containers" that are to be managed and interrelated. And the dimensions of SCM complexity that prove most challenging are those of scale and diversity, and of differences between artifact change/creation time and decision binding-time.
The next question involves translating what Availability, Modifiability, Performance, Security, Testability, and Usability mean for a "process" architecture. I'll make an initial stab at that (feedback is encouraged):
What are the "quality" attributes of an "agile" process? Do they include ALL of the above? what about: adaptive? lean? result-driven (i.e. "working software")? self-organization? iterative? collaborative?
How about some of the traditional "quality" attributes of a CM system: traceability (vs. transparency?), reproduceability? repeatability?