J. Hank Rainwater
HankRainwater@CelticTestingExperts.com
Chief Technical Officer
Noel Kierans
NoelKierans@CelticTestingExperts.com
Founder/Chief Executive Officer

Celtic Testing Experts
http://www.CelticTestingExperts.com

Table of Contents
Process or Results? 
Bugs Still Exists 
What to Do? 
Manage the Process or Be Managed by It 
Review the Scope 
Adapt to Change 
Prove the Process 
Requirements Gathering 
Test Driven Development (TDD) 
Team Composition 
Phases, Iterations and Other Magic Bullets 
Inspect What You Expect 
Document with Flexibility 
Size Up the Effort 
Appendix

Process or Results?
Over the past several decades, software development life cycle (SDLC) methodology battles have been waged between those that consider the process just another flavor of engineering and those who view it as more a matter of craftsmanship. While not quite a struggle between art and science, the skirmishes have been fierce at times and, depending on whom you ask, victories have been declared. True: adoption of agile methods seems to be on the rise, but many still find linear models, often referred to as waterfalls¹, producing good results. A history of the discipline however, shows that nothing particularly new has occurred under the sun: both linear and iterative models have been around since the dawn of programming. Yes, improvements to programming languages and architectural techniques as well as SDLC management tools have occurred, but the very human activity of turning code into product still longs for a magic bullet.

Standing to the side of these struggles are those who simply want a methodology that works, is repeatable, and can be made to work within their corporate culture. In short, managers want methods that led to success. This article is aimed at bringing about some reconciliation in the methodology wars and to remind us that methods always have an academic appeal but what really matters is the result: did the software help the company’s bottom line. You may measure this result however you please but busted schedules and budgets and buggy software should be at the top of your metric list. And, finally, the management of the process is at least as important as the details of the process.

Bugs Still Exists
Ever since the first literal bug was found blocking an electromechanical relay from performing its logical operation in the late 1940s, problems with software development and the costs to enterprises because of these issues have abounded and formed part of the legend and lore of the profession. If software is a manufactured product – and it is – it is unique in that as a human creation founded on logic it often behaves in unpredictable ways. One would think that given enough time and thought that all the possible use cases for a program could be mapped out and tested. This is theoretically possible but the constraints of project schedules driven by market forces often render this goal impractical no matter what methodology is in use. A recent survey conducted by Electric Cloud® in partnership with Osterman Research has shown ‘that the majority of software bugs are attributed to poor testing procedures or infrastructure limitations rather than design problems’². Most of the developers surveyed said that their software testing process is often more painful than preparing taxes!

The results of this survey are shown below. Figure 1 illustrates that the typical developer’s experience with testing is often not as helpful as one might hope.

¹ The origin of the term ‘waterfall’ is hotly disputed other than the obvious picture suggested by certain process flow charts that seem to imply you can’t start a process over again in the same way that water doesn’t flow uphill.
² www.electric-cloud.com/news/2010-0602.php

When it comes to developers’ experience with how bugs affect the overall delivery, there was little surprise in the survey’s result as shown in Figure 2.

Bugs cost money, affect corporate reputations and remain one of the most difficult areas of software development to mitigate.

What to Do?
What’s a development manager to do in light of this survey? Well, for one, examine your methodologies and be sure you’ve got the best mix among all those time-tested ways of delivering software. No one method can fit every organization or project, and, in spite of the trend toward agile methods in the past 15 years, waterfall still has much to recommend it.

Whatever school you follow and practice, every development project consists of common activities and associated outputs as well as relationships between these activities. Whether you’re building a product with one function or a hundred, you’ll have to proceed through the activities shown in the flow chart of Figure 3.

The only magic involved in optimizing your SDLC is to mix, overlap and stir all these activities into a consistent recipe that you can duplicate from project-to-project and that leads to successful delivery. Do note, however, that while testing might appear to occur late among the various activities – and it often does when execution is sequential – based on the survey described in the preceding section the role, scope and timing of testing should be reconsidered; more about this later.

Even after you think you’ve obtained the best mix in your methodology, you should always be open to learning from past results: after all, if agility means anything it means learning and adapting. By comparison, waterfall, sometimes maligned by agile proponents, often ensures that significant thought and planning has surrounded the development activities. Of course analysis-paralysis sometimes afflicts waterfall shops too so be cautious
at this end of the spectrum. And, to appease agile critics, this school can put programmers in charge of the process which isn’t always a good idea³.

But the focus here is on success, regardless of your method. We all know of great products created by cowboy programmers as well as those produced by carefully managed teams using strict processes of various flavors.
What is critical to any method is the role of those who are managing the process and responsible for the results. This would be you.

³ J. Hank Rainwater. Herding Cats: A Primer for Programmers Who Lead Programmers. New York: Springer-Verlag (Apress), 2002.

Manage the Process or Be Managed by It
While metaphors often become clichés and acronyms frequently defy reason and tax our memory, a marriage of various SDLC techniques and the management of them can be consummated by use of the picture brought
to mind by the word ‘RAPIDS’. Certainly if your development project feels like it is moving downstream and encountering lots of bumps along the way you can identify with this term. It also combines the physical nature of waterfalls – albeit small ones – with the motivation behind agile, namely time-to-delivery. Let’s take ‘RAPIDS’ apart and see what can be learned.

Review the Scope
Rapid implies a development velocity that often can only come from careful management of the scope of what is to be delivered prior to any other activity. This doesn’t mean that a full requirements document can’t be
complied, but it does mean that development/testing groups can only concentrate on small portions of a product at one time – this is just human nature. In spite of all the hype about multi-tasking aided by the proliferation of communication channels today, the reality is now beginning to dawn on managers that a loss of focus results from resources spread too thinly over too many demands. Being overwhelmed by the vastness of scope is the best way to derail progress and success. Have you considered using a versioned requirements document that iteratively builds up the functionality you need in a product? This is, of course, a hallmark of agile methods and eventually, as the iterations increase, you’ll have the documentation you have come to
expect from waterfall methods. This is just one way to control the scope. There’s more to be considered…

We have all become familiar with the interdependent factors of the classic software development ‘iron triangle’ shown in Figure 44. Change one side and the other two sides must also change because of the interconnections between the activities; often we are dismayed as unintended consequences of one domain decision impacts another and affects the overall quality delivered.

Managing scope is the first step in obtaining a measure of control over quality as market forces and budget limitations demand variations in resources and schedule. When you reach the buy/build decision point in your considerations for a new product this must be carefully thought out. Here are some of the parameters to consider:

You could easily add to the list. Even when you buy a product that appears to closely meet your needs, there will no doubt be configuration settings that you’ll have to manage and test. You are also dependent on the quality of testing performed by the vendor who provides the product. On the other hand, in house development creates many different issues that must be carefully managed.

One primary consideration in managing scope is to know that risk of failure is directly proportional to scope size; however, many times you have to build something really big or replace a legacy system that grew over decades. In these cases, where the scope can’t be easily reduced, management of an effective process becomes critical.

Adapt to Change
This certainly sounds like a platitude but because of the malleability of software – and often requirements – this is one cliché that should be revived and restored to its original meaning. If a team can’t adapt it is usually due to several factors, key among them can be that the leader at the top of the corporate development hierarchy doesn’t like change or thinks that their pet process should be good enough for everyone if they would only practice it in a disciplined manner. No doubt discipline is essential but so is recognizing that any process must fit the project at hand. At the risk of overloading you with clichés, one that comes to mind is this: if the only tool you know is a hammer then every problem looks like a nail.

Take a lesson from nature: adaptation has been the driving force behind life on earth for millions of years. As environmental forces worked on organisms with gradually changing responses to those forces, a selection process resulted in the survival of the fittest. These ‘gradually changing responses’ in nature are the result of apparently random genetic mutations. In the business world, your corporate DNA is mostly under your control. If it isn’t, then your instances of success may also be random.

In an analogous way, the marketplace and your corporate culture exert forces on your development process and if you don’t change when the forces become unbalanced, your project’s promise will be in danger of deflating like that shown in Figure 5.

Which of the following forces are acting on your projects that indicate a need for change?

If some of these real-world comments are familiar to you as a manager then you’ve got unbalanced forces acting within your organization. If these comments are not familiar then you should ask if you really know what your people are feeling about the development process. It’s very common for a manager to get out of touch with people on the ‘shop floor’ when too much time is spent courting those higher in the corporate hierarchy. Sure, politics is a reality and a necessity, but good politics means adapting, compromising and listening so that the best process solutions, reached by as large a consensus as possible, becomes the standard.

Prove the Process
In the early days of rapid software development methodological discussions, ‘iterations’ were viewed as the magic bullet for software shops. Today, all agile methods use iterative cycles of development, testing and stakeholder review as their core feature. It is true that small iterations can be helpful but so can having a carefully thought out roadmap to where you need the product to be, prior to any development activity, so that many design dead ends can be avoided; this is where waterfalls have an appeal. As mentioned under the heading of reviewing the scope, a versioned requirements plan that produces deliverable functionality with tested quality should be at the heart of any process you adapt. However, you shouldn’t believe this without proof derived from experience in your own organization.

A little history mixed with philosophy is now in order…

Most managers are familiar with the Agile Manifesto⁶, a document originating from the lessons learned from many failed software projects. In this manifesto value is placed on…

• Individuals and interactions over processes and tools.
• Working software over comprehensive documentation.
• Customer collaboration over contract negotiation.
• Responding to change over following a plan.

If you are a dyed-in-the-wool waterfall person, some of these phrases will chafe against the careful planning and control you feel are necessary for success. Certainly within the statements of the manifesto an implied bias against waterfall can be detected. The agile movement is aware of these reactions and some years after the manifesto was published, an ever enlarging group of agile supporters moved toward principles of interdependence⁷ with project management where leaders can say:

• We increase return on investment by making continuous flow of value our focus.
• We deliver reliable results by engaging customers in frequent interactions and shared ownership.
• We expect uncertainty and manage for it through iterations, anticipation, and adaptation.
• We unleash creativity and innovation by recognizing that individuals are the ultimate source of value, and creating an environment where they can make a difference.
• We boost performance through group accountability for results and shared responsibility for team effectiveness.
• We improve effectiveness and reliability through situational-specific strategies, processes and practices.

Now, in light of these last bullet points, how can anyone disagree with what the agile school is all about? Well, you can of course disagree, but that’s not the point. Can you learn from these principles and are you willing to tailor your own process based on good ideas derived from them? This is the point. Prove what works for you: have you been successful in the past with your processes? If not, consider the following practical ideas about process in the subsections below that while often attributed to the influence of the agile school, do have a debt to waterfall methods.

⁶ See agilemanifesto.org.
⁷ See pmdoi.org.

Requirements Gathering
In some shops gathering never ends. This isn’t surprising considering the pace of marketplace and technology changes in the past several decades. But, harkening back to matters concerning scope, you can’t build everything at once, but you can envision it. High level requirements can go a long way toward getting a project started, especially those that tie business needs to user expectations. Getting the ‘why’ and ‘what’ parts of a project correct facilitates correctly specifying the ‘how’. By contrast, some projects, often driven by a superstar developer can jump too quickly to ‘how’ and then you can end up with the dreaded ‘one owner pet’ application⁸ that may not really meet the long term business needs. This isn’t to say that developers shouldn’t have input into the requirements process, just be sure they are present for listening as much as talking, and offering correctives about impossible development scenarios that might be dreamed up by overly ambitious business analysts.

Sometimes the process is a simple as pulling out the user guide for a system to be replaced and simply telling a development team to rebuild the system with a new technology but be sure to keep the same functions. Well, that might sound simple but it can also be deadly since reconsidering the business processes that once served an organization should be part of creating requirements for any new product. Deconstructing the old system would be a better plan so that obsolete functionality can be discarded as well as knowledge gained about the data repositories and interfaces that will no doubt have to be migrated for any new system. Then reassemble the broken-out old requirements into a form that’s amenable to new development.

The use of index cards for requirements (often called user stories today) has often been recommended by some agile methods and this can work well for some projects. The main point of the index card idea is that you should be able to frame requirements in an atomic fashion so that specifications and design can be done more effectively. Nothing can keep you from putting these small scoped requirement fragments into a larger document, just be sure to add traceability nomenclature to the document so you can backtrack from later project phases as defects are found during testing.

Many tools have been created to facilitate the requirements gathering process and these should also be considered as ways to speed the work. Just be sure the tool doesn’t drive the process beyond what makes sense for your organization. We’ll discuss tools in a later section in more depth.

What about the role of QA during the requirement’s gathering? Well, we’ll have more to say in a later section about this, but in CTE’s experience the more QA is involved up front at the beginning of a project, the better everything will go.

Test Driven Development (TDD)
While the acronym TDD has only had widespread currency in recent years, the idea isn’t new. The formal incorporation of this method is somewhat new as can be seen from the fact that more and more integrated development environments (e.g. Visual Studio) are equipped out-of-the-box with templates for developers to test their code before they write it. What isn’t new, however, is that developers often resist the TDD approach.

This resistance can originate from several sources, primary among them is often the perceived ‘independent cast of mind’ that developers maintain. And, to the uninitiated, TDD might seem backwards, but the complaint of many QA teams that they don’t get adequate documentation on unit testing is often solved by embracing TDD.

⁸ This is code that no one other than the author can maintain. Version 1.0 seems to work well, but version 2.0 can prove problematic.

The core concept of TDD is shown in Figure 6. While this is an idealized representation of the approach, several variations might come to your mind as a manager that you could incorporate into your next project.

At the simplest level, all that is being asked of a coder following TDD is to at least think about the function that is to be accomplished in code by defining the initial and final states in such a way that a test can be written to exercise the code. Even better is if the test is written in close time proximity to the actual code. If the diagram of Figure 6 is too much for your team, how about something as simple as a spreadsheet with the following columns (shown here with example data):

You can see that the comment section leaves much out in terms of what QA would need to know, but at least it’s a start at defining a test case from the developer perspective who does know these boundary conditions.

In some cases, a formal developer-created test or test harness is not optional, especially when the code involves an interface between two systems that can’t be exercised without a complete integration test. You don’t want to have to perform an end-to-end test each time an interface changes: this makes it hard to pinpoint any issues. While this additional development time might be questioned by some, our experience at CTE is that it is worth the effort in terms of needlessly repeated test cycles that never quite get to the source of bugs.

Team Composition
If waterfall has a fault, it is the idea that separate teams hand-off their domain-specific knowledge to the next team in the development process and consider themselves finished after the hand-off. Classically, this looks
like Figure 7 drawn in such a fashion that ‘waterfall’ is an apt pictorial moniker.

The idea that this process could actually proceed downhill as shown is obviously not realistic since the minute a test fails you have to move back ‘uphill’ to correct the problem. What this approach does show is the mindset that certain groups have specialized knowledge and that they must focus single-mindedly on those things they know best. Of course, this is whole reason we have teams that work together. As a manager, the question you need to address is how well are these teams working together?

Business conditions often constrict the choices for the best team composition. When business analysts (BA) or subject matter experts (SME) are only available for part of a project lifetime then you’re often deprived of their expertise after the requirements phase. When development teams must serve several business units you may have coders with limited business knowledge and inadequate time or conflicting schedules. Then we come to testing, traditionally added near the end of a phase and often reusing the BA and SME resource pool. We’ll be addressing testing in more detail in the next section, but often reuse or re-tasking of personnel isn’t the best choice. All these considerations have led many organizations to adopt a silo approach to business units. This has strengths, but can also result in failing to take advantage of shared resources where appropriate. Again, what is working for you in terms of teams? Are they successful in your organization or do they need realignment?

Phases, Iterations and Other Magic Bullets
OK, they are not magic, but they do need to be considered since life by the mile is hard but by the inch it’s a cinch. This might be another cliché but again, clichés do exist for good reasons, they just get worn out with use and often don’t encapsulate or solve all the complexity the real world can throw at you. Nevertheless, breaking up work into small pieces is critically important in any SDLC methodology. You already know this. You’ve seen the flow charts from various SDLC schools so many times that they begin to blur together. Perhaps this blurring is a necessary step toward clarity for your development program. In other words, let’s take things apart again and try to put them back together in a more workable fashion.

For a brand new application that doesn’t have to replace a legacy system, agile methods are more easily fitted to a project’s needs as shown in Figure 8.

In the method shown in Figure 8, iterations are appropriate and allowed for by a skeletal application framework (wireframe) that instantiates the full architecture with placeholders for functionality added incrementally. The documents produced during iterations can be accumulated into artifacts that provide traceability during subsequent builds and will result in a full system description and test plan when the process is finished. Creating regression tests as the application grows is also facilitated by this method since your test cases grow with the scope of functionality built.

When it comes to replacing a legacy system, the creation of a controllable and predictable process becomes more challenging. Traditionally this is an area where aspects of waterfall methods have played a larger role than agile techniques. Figure 9 shows such a project actually managed by CTE.