In Part I of this series I introduced the idea that traditional development is like a Chinese Finger Puzzle. Solutions to problems often make the problem worse instead of better. This post will cover two more areas that illustrate this phenomenon.
Future-Proof Architecture, Major Releases and Large Feature Sets
A common sentiment when using traditional development is that because it is so hard to redo architecture, because it takes so long to make changes, because it takes so long to produce a release, we had better get the architecture right the first time. We had better take into account all possibilities and eventualities. Since we are going to spend a long time on the architecture, we better plan to do a lot of features for the release. All of those features mean we need to take more variables into account and have an even more thoroughly thought out architecture. In a traditional project, the interplay between architecture, release timeframe, and feature set create a vicious cycle which tends to increase the scope of all three.
In an Agile project, the practices of short iterations and iterative design reinforce each other to keep the amount of architecture that is required as small and simple as possible yet no less than necessary. The product backlog and short iterations keep the feature set focused on those features which will produce the highest customer and market value instead of trying to build a product which takes into account all possible future requirements.
Read more about iterative design: "Designing Software is the Same as Predicting the Future."
Separation of Development and QA
A common pattern in traditional development is the separation of development and QA. Of course everybody always says that “QA is everybody’s job” and “you can’t test quality into a product” and “QA should be involved right from the start,” but what usually happens in practice is that QA occurs at the end. One contributor to this problem is that “during development” the product isn’t stable enough to test and you want to get the most bang for the buck out of your testing because it is so expensive to do and it takes so long to qualify a release. The natural result from these circumstances is the separation of development and QA which is constantly reinforced. Using traditional development patterns, all efforts to unify development and QA are inevitably defeated. Nobody is consciously trying to make it happen, it just happens.
But sometimes it is also on purpose. The separation of development and QA can feel “more honest.” Sometimes development doesn’t want QA to know what is going on because they are sure that they can catch up and fix things and then deliver something to QA which is good. But in order to get the benefits of QA without the help from the official QA folks, development ends up having to do QA themselves which reduces the time spent on development. In any case, traditional development practices tend to separate development and QA.
Agile practices do just the opposite, they tend to bring development and QA closer together. The writing of tests for a particular change is done either before the coding starts or during the coding, not in a separate phase at the end of all coding. Tests are run constantly using the practice of Continuous Integration. QA doesn’t have to wait until the traditional code-freeze to start the bulk of their work. This creates a positive feedback loop where QA gets to show its value on a regular basis, development spends less time fixing things at the last minute, QA learns more about the product, and development and QA become even closer.
Next: Short Iterations
Have you ever wondered how your organization compares to other organizations when it comes to the process of developing software? Do you feel like there is an area that really needs improvement, but everybody else says "that's the way everybody does it?" While looking at what is involved in mainstream Agile adoption, I realized that it would be useful to create a list of the current mainstream software development practices for comparison purposes and I thought other folks might find it useful as well.
I define a practice as being mainstream if it is a practice that most people would agree that most people do. It may not be that they do that exact practice, they may do something equivalent or further along the spectrum of what constitutes a good practice, but they at least go to the level described.
The word "practice" is the key word here. This is about what people can be observed as actually practicing in their day to day real work. Things which are described by policy or documented as the correct way of doing something but avoided and worked around don't qualify. A mainstream practice is something that is in common usage that people do because they believe the value is worth the effort and not doing it is sort of like saying “electricity and hot and cold running water aren’t for me, I much prefer candles and fetching water from the well.”
While each of the following individual practices is considered a mainstream practice, that does not mean that it is mainstream to being doing all of the following practices. That is, in some of the areas that these practices cover, any particular development organization may be operating at a lower level than these practices, but for any particular practice, most development organizations operate at or above the level of these practices.
Overall
Basic flow – the common development activities are: talk to customers and/or do market analysis, document market needs via requirements, create a design, implement the design, write tests, execute the tests, do find/fix until ready to ship, ship.Preparation
Requirements documented using Microsoft Word or Excel – while there are actually quite a few off-the-shelf requirements tools available, most people do not use them. The most common method for documenting requirements is via Word and Excel.Recording of defects in Bugzilla – there are very few organizations which aren’t using at least Bugzilla to track bugs and Bugzilla is definitely the most popular choice. And recording of defects is pretty much as far as it goes. Enhancements, RFEs, requirements are tracked separately.
Basic initial project planning – this is the simple act of picking a bunch of work to be done, estimating it, dividing it up among the folks that are available to do the work and determining a target ship date. Some teams use MS-Excel, some teams use MS-Project. This does not mean sophisticated project planning. The extent of re-planning is a simple “are all tasks done yet” with the occasional feature creep and last minute axing of important functionality at the last minute.
Basic design – prior to coding features or defect fixes that will take more than two days to do or are highly impactful, a design document is created, circulated, discussed, and updated.
Development
All source code is stored in a source control system – this is not to be confused with “all work is done in a source control system.” It is actually surprisingly common to see the source control system used for archiving of work done rather than as a tool for facilitating the software development process.Source control and issue tracking integration via a hand-entered bug number in the check-in comment, without enforcement – while there are certainly more sophisticated integrations, most people at least type in the bug number when checking in changes so that they can later find the changes associated with that bug.
Defects have a defined workflow that is defined in and enforced by the bug tracking system – even if it is as simple as Open, Assigned, Closed.
Mainline development – all developers on a project check-in to the mainline (aka the trunk).
Refactoring – while the term “refactoring” has come to mean just about any change to the software, the idea that code should be periodically cleaned up and simplified is pretty much universal at this point.
Nightly build – the entire product is built every night. This does not necessarily mean that tests are also run automatically or that there is an automatic report of the results, just that there is a nightly build.
Quality
Mostly manual testing – this one is the hardest to understand. The process of testing software is one of the most backwards parts of software development.Unit tests – while the overall testing of software is still mostly manual, unit testing has caught on rapidly in a very short span of time.
Using defect find/fix rate trend to determine release candidacy – this is not actually a particularly good practice, but it is what most people do.
Separation of developer, test, and production environments – you might think this is so obvious it isn’t even worth mentioning, but this principal is violated enough that it is worth mentioning that it is actually a mainstream practice. I mention it to emphasize that if you aren’t doing this, you really need to.
Releasing
Basing official builds on well-defined configurations from source control – the official process for producing production builds includes the step of creating an official configuration in the SCM system and using that configuration to do the build.Releasing only official builds – all builds that go into production are produced using the official process.
Major and minor releases - creating a major release every 6 to 12 months and minor and/or patch releases on a quarterly basis.
What do you think? Are these right on target or way off base? Do you think there are any missing areas? Do you think the mainstream practice level is higher or lower for some of these? Let me know what you think!
Next: "There is no Bug. It is not the Bug That Bends, it is Only Yourself."
There’s a saying that “you can’t test quality into a product.” That may be true, but it is very difficult to assess the level of quality of a product without testing it. I find that one of the problems with quality is the way that it is talked about. There are all different categories of testing such as regression, black-box, white-box, exploratory, automated, manual, etc. Each of these categories serves a useful purpose, but what is the overall purpose of testing?
Some people argue that testing is how you verify that the product works. Other people argue that the purpose of testing is to uncover what does not work. My view is that the purpose of testing is to show that the software works. By “works” I include that it correctly handles bad input, bad data, and diabolical usage. If you try to use the product in a way that was not intended and it crashes or gives no result at all, then you have found a way in which it does not work.
Quality Assurance and testing are not the same thing. You can implement all of the QA you want: good process, code reviews, coding standards, coding for testability, and whatever else you want to assure quality short of actual testing, but without testing, you have no idea how the product will actually perform. Let’s say you produce a product with absolutely no test plan or you have a great test plan but you don't execute it. I suppose that it is possible that the quality that the customer experiences will be exactly the same as if you had a test plan and executed it. The difference is that your knowledge of the quality of the product will come from your customer after you have already shipped your product. The purpose of testing is to derive knowledge about the quality of the product prior to shipping it so that you can make good decisions about what to do now that you have that knowledge.
So, testing itself does not produce higher quality. It only produces knowledge of current quality. There are only two things that produce higher quality: changes to the process that you use to create the product itself (not including testing), and the responses taken based on the knowledge gained from testing. Testing can influence quality, but it does not directly affect it.
Product quality is an ongoing effort. No matter how much testing you do, users will find ways to use your product that find holes in your test plan. For any change, whether it is a bug fix or a new feature, you don't really know the impact until customers start using it. That means that the maturation of a change doesn't really start until the change ships. If you have a long release cycle, you will have changes that are made early that are basically gathering dust until you ship. You can't really say that they are tested until you test the whole product as it will ship, so again, true maturation doesn't start until you ship.
The moral of this story? Release as often as you reasonably can; which may be once a week, once a month, or once a year, but always strive to release often.
Next: The Role of QA in an Agile Project
The best way that I currently know to convey why I think that Agile produces higher quality is via a 1 hour PowerPoint presentation. I thought it would be worthwhile to attempt to put it into a blog post. But, I couldn't figure out how to condense it down to a reasonable length. There's lots of diagrams and animation, etc. But while I was working on it I created a list of points to make. Looking at the list I thought "Hey! Here's yet another top ten list!"
Please note that in this list I am referring to truly Agile projects. See “No Really, What is Agile?" for details.
10. More appropriate distribution of test coverage. Typically, QA gets a big dump of functionality near the end of a release and has to figure out how to best use the time available. That often means that the "most important" new features get very thorough testing and the rest get a "spot check." In an Agile project, test plans and automated tests are created throughout the project and each work item is given the amount of QA resources that is appropriate for it.
9. Because testing is done in the same timeframe as the coding, problems are found earlier.
8. Writing tests early catches requirement and design problems earlier.
7. Because problems are found and fixed faster, there is less chance of the quality of a project being poor for long stretches of time. When there are lots of tests that don’t pass, it is difficult to get accurate feedback on new code. In contrast, code written on a stable base is more likely to be stable itself because there will be accurate and timely feedback on the results of the changes.
6. It is hard to succeed in an Agile environment without automated testing. Automated testing helps to increase the consistency of testing.
5. One effect of short iterations is the evening out of resource demands. That means that testing is done consistently and on a regular basis and there is no need to take shortcuts in contrast to typical development which compresses most of the testing to the end of the process which then requires taking shortcuts due to schedule pressure.
4. More frequent customer input on direction. Part of quality is usability and match of features to needs.
3. More frequent customer input on results. Customers are the ultimate arbiter of quality and their level of expectation is often different than you expect.
2. The Development and QA organizations must be integrated for Agile success. Integrated development and QA is far better than the typical “separation of church and state”.
1. Especially when doing one piece flow, there is significantly more opportunity to detect process problems, diagnose them, try corrective action, and gauge the results of the corrective action.
Many of the above deserve further elaboration, but I wanted to get this out there for discussion.
First, many thanks to Isaac Rodriguez whose comments on my previous post elicited a comment from me which I have re-written as a post here. I've also updated my previous post a bit to better state the original point.
Let's say that you develop your software in 30-day iterations. At the end of every iteration all functionality introduced during that iteration had all of its tests written during that iteration instead of at the end of the one year cycle. All of those tests pass. You take testing very seriously; you've got very high test coverage numbers, you are using decision based coverage instead of line coverage, and you fix every bug you find.
Let's say that you could release every month but instead you choose to release every year. Now at the end of this year you release your product. I guarantee that your customers will find problems. Let's say for the sake of simplicity that they find exactly 12 and each one is linked to functionality introduced in one of the 12 iterations. This means that for the bug in functionality in iteration 1, you waited 11 months to find the issue that your customer would have found right away.
I'm not saying that you should forgo testing or rely on your customers to be part of your QA department. I'm only saying that despite your best efforts, it is inevitable that there will be issues that you only find after you release, so keep on testing, don't stop that. But release as often as you can.
Also in this (contrived) example, your customers would still be exposed to the same number of bugs, just not all at the same time.
Moral to this story: "In addition to keeping your testing standards high, it is better to find problems that you are likely to only find by releasing to customers as soon as you possibly can. Therefore, release as often as you can."
Are you interested in Agile Development and Software Configuration Management? Then check out the new Agile CM forum at CM Crossroads!
My experience is that with automation, branching, and the supporting process, the goals of determining the quality of an iteration and creating stable baselines can be achieved in parallel. The first step is automation. Without a high degree of automated testing, I'm not sure it makes sense to try to do this in parallel. However, with continuous integration which includes full builds and full test suite runs, there should be few if any problems at the end of the iteration.
Some might say that their automated build and test takes a very long time. That’s a whole separate topic. In the meantime, let's assume that the full automated build and test cycle takes on the order of hours.
By branching, you can start the next iteration without disturbing the previous one. If QA finds any problems, development can fix them on the previous iteration branch. At some point that branch will be declared "done." Those changes are then merged into the current iteration. At that point, there is very little if any difference between where things are and where they would have been if development had started from the "done" baseline.
The advantage here is that development did not have to stop, and you still have a stable baseline. My experience is that once a team gets into a rhythm, the amount of time spent on "touch-up" is minimal.
This also applies to iterations that become releases. The "previous iteration" branch becomes work towards the release and then hopefully soon thereafter the release itself. Meanwhile, work continues towards the next release.
An interesting question in Agile Development is “how do you incorporate QA?” I guess that all depends on what you see as the role of QA. For instance, if you have your QA people involved from start to finish, then you really wouldn't have to change a thing. For instance, a QA person can:
- help to determine if your stories are well defined
- suggest adding stories related to testability
- make sure that people are creating good unit tests and/or write unit tests
- increase your code coverage by adding more automated tests
- write tests for stories before (test first) or after they are completed
- organize usability testing
- add stories for improving usability
- do exploratory testing on early builds
- verify the completion of stories as they are completed
If they are involved from start to finish in this manner, then you eliminate the need for a well-defined QA stage. If you find at the end of an iteration that QA is "falling behind" then it could be argued that your ratio of development to QA is off. If so, then you can either add more QA resources or have developers take on some portion of the above tasks to restore the balance.
For AccuWorkflow, I had the partial use of multiple QA resources over time, so I had a "virtual QA person" which was different people at different times including developers doing the various tasks I mentioned.
With short iterations, it is likely that there will still be some touch-up work to be done on the previous iteration when the next one starts. Perhaps QA has not finished evaluating the iteration, or perhaps somebody wants to do a demo but runs into a bug that needs to be fixed first.
We were doing variable length iterations where the iterations were between 2 days and a week, so we ran into this hand-off period on a regular basis. The way we handled it was to create a stream for the “hand-off” period.
As an iteration ended, we created a new stream based on the existing development stream. Depending on the SCM system you are using that probably translates to either creating a branch or a label. In any case, the new stream then represented a stable version of the “previous iteration.” If somebody needed to do a little follow-on work, for instance to get things ready for a demo, they used another configuration called “demo” based on the “previous iteration” stream. To prepare for the demo they made changes in the demo stream without affecting the previous iteration or the current one. Once they finished the demo they then merged any changes into the current stream.
Having streams representing multiple iterations also made it possible to easily find out what had changed between iterations which aided in tracking down problems.
Of course, there are all of the standard things that a good QA organization brings that operate at a higher level than stories, but I'll just take those for granted for the purpose of this post.
At a high level, you should work to keep the ratio of development to QA balanced. The ratio will vary from project to project, so I'm not commenting on that, but once you have a ratio there are several ways to maintain it. In an Agile project, you may quickly find out that the ratio itself is off and need to adjust the ratio. In a project with long iterations, you might not find out that the ratio is out of whack until it is "too late."
In any case, once you find that you are short of QA resources, there are a couple of options. The first option of course is to get more QA resources. Failing that, you can use development resources. It may also be that you have a short-term need for more QA and in that case using development resources might be exactly the right short-term solution.
I think a common rule of thumb, which especially applies in the context of compliance, is that different people do the development and testing of a particular requirement. That is, if Sue is responsible for issue 12345, then Sue doesn't do the testing of it.
As to the exact use of development resources, I advocate that developers take on a sub-set of QA activities, thus allowing for better leveraging of the existing QA resources. For instance, creating automated unit tests or the general generation of automated test cases.
You can of course set whatever policy makes sense for your organization. Having people only write test cases for other people's code is always a good idea. Beyond that, I would assume that QA would take what the developers had done as a starting point, supplementing with additional tests, reviewing what the developers had done, running the official test cycle according to the test plan and reporting on the results.
