Testing in the Real World

By Edward Kit

Chapter 9. Controlling validation costs

Chapter 11. Software testing tools.

Part IV. Managing test technology.

Presented by Ramunas Baltchetis

Edward Kit defines his six essentials of software testing as a necessary condition for any software organization seriously planning to compete in the future market. They are:

1) the quality of test process determines the success of the test effort

2) prevent defect migration by using early life-cycle testing techniques

3) the time for software tools is now

4) a real person must take responsibility for improving the testing process

5) testing is a professional discipline required trained, skilled people

6) cultivate a positive team attitude of creative destruction

It is hard to argue against any of the essentials. What, then, makes so difficult to

implement them? Edward Yordon in his gloomy book ¹ cited a 1990 study which provided some depressing statistics on the methods organizations used to pursue quality:

- 42 % of respondents answered “none”

- 24 % - “improved testing”

- 20 % - “focus on prevention”

- 9 % - “process improvement”

Yourdon blamed software shops for “lack of passion”. He insisted that “demonstrable commitment to quality” from management must be present to improve the quality. The structure of the organization should reflect the authority of the testing and QA groups, quality should be defined in quantifiable terms, rewarded.

Ten years later we see an improved situation. A survey, conducted annually by Quality Assurance Institute, provides rather optimistic status of software testing ². Majority of surveyed companies maintain test standards/procedures, have testing process improvement plans, train testers, collect defect data. More importantly, the majority of participants stated that testing is the responsibility of separate test group, users and testers decide when the system is ready for production.

The industry appears to be recognizing importance of concepts laid out by Edward Kit.

In the second part of his book Kit concentrates on such important aspects of testing process as cost control, testing tools, organizational approaches to testing. He also discusses current practices, trends and challenges. Let us take a closer look.

Controlling validation costs

One thing that is difficult about testing is knowing when to stop. In the real life time and money are always serious constraints. Kit provides the strategy for building the cost control into the testing process. The key concept is testware – a collection of work products of testing. The objective of testware is to maximize the testing yield by maximizing the potential for detecting errors and minimizing the number of tests required.

There other objectives for testware to consider, related to cost control. Cost of performing, maintaining and developing tests have to be minimized.

Only costs related to recurring tests ( as in regression testing ) are significant. Kit defines the following components of performing tests:

- pre-run setup costs

- execution costs

- post run costs

Controlling the amount of time and ( especially! ) skilled labor required to do the tasks related to the setup of testing environment ( configuring hardware/software, restoring files, etc. ) is a key to minimizing pre-run costs.

Minimizing execution time and required dedicated equipment – main goals of controlling execution costs. One aspect of this is to reduce the attended time when the intervention of the user or operator is required to run the tests. So the goal is a better test automation. Another side of cost control is the selection of the test to be performed. To run full regression testing every time the change introduced is costly. Two solutions are recommended:

- run all tests that would retest everything affected by the change

- run only the tests that would retest only things that are closely related to the change

First choice is less risky. Second choice requires good product knowledge to determine which parts are affected by the change and to select appropriate tests.

Minimizing post-run tests means reducing the amount of time and skilled labor required to analyze test results, documenting them, restoring the environment after the testing.

Kit recommends the use of dedicated hardware for testing. It is usually cheaper than reconfiguring. Another recommendation is the maximum automation of:

- configuration of software and environment

- identification and selection of tests to be run

- execution

- comparison of test results with expected results

Kit suggests the use of testing tools for automation of tests and their result analysis. Another candidate for automation is manual interactions, especially for products with highly interactive user interfaces, multi-user software.

The key to minimizing the costs of test maintenance is to periodically review tests and determine if they are still effective. Tests have to:

- provide value and not be redundant with other tests

- be executable ( if change introduced features that prevent test form executing – it is not effective anymore )

- be valid ( again, if change introduced functionality that made the results of the test misleading – it is not effective anymore )

This level of maintenance ( adding a task for each reported problem, adding tests for new changes, reviewing tests for effectiveness ) is nearly impossible to achieve without the configuration management system in place.

Most good software engineering ideas apply to the development of testware. Testware is software after all. Reuse of existing test cases should be attempted as much as possible. For some standard-based software buying testware is the most cost-effective solution. Some components lend themselves to testing using scripting. Tests and their expected outputs are stored in a text file. The simulator program then runs, executes each test and compares the actual result with expected result. A large number of tests can be collected in one file to reduce costs.

Finally, it is important to have easy-to-use testware library. It should be easy to select

and execute any subset of tests.

Software testing tools

Software testing tools are a powerful aid in achieving quality product. There are many quite mature tools on the market that can be categorized in a number of ways:

- by testing activity ( code verification, test execution, etc. )

- by functional keyword ( capture/playback , logic coverage, etc. )

- by major area of classification

Ed Kit argues for categorization of tools based on associated activities for two reasons: first, it is close to the view of the tester, second, it is consistent with testing standards.

He goes on to break tools into five broad categories and further into types:

1) tools for reviews and inspections

a) complexity analysis ( identifies complex areas; most tools work on source code only )

b) code comprehension (provides graphical representation of the program, identifies dead code, helps to trace logic )

c) syntax and semantic analysis ( static error checking, this is somewhat an extension of the compiler checking )

2) tools for test planning ( Kit is very vague on these except the suggestion to rely on IEEE/ANSI Standard for Software Test Documentation which helps in test plans design and also provides useful templates )

a) templates for test plan documentation

b) test schedule and staffing estimates

c) complexity analyzer

3) tools for test design and development ( again, I get the impression that not much help is offered by this category of tools to aid in test design yet. Obviously, test data generation tools are very useful in minimizing such a mundane task, however, tools based on requirements are yet to come. On the other hand, such tools may not be cost effective investment for a software shop with the less mature process )

a) test data generator

b) requirements-based test design tool

c) capture/playback

d) coverage analysis

4) tools for test execution and evaluation ( this is by far the richest category of tools; nearly every vendor in the Product Database distributed at QAI 1999 International Software Testing Conference ³ features tools in this category )

a) capture/playback ( capture user operations; captured tests then form a baseline that can be played and output can validate the changes )

b) coverage analysis

c) memory testing ( boundary check, leak detection, run-time error detection )

d) simulators and performance ( mostly used to test network, telecommunications, real-time software )

5) software testing support tools ( this is another fairly developed area; tools of this type are usually implemented in all software shops )

a) problem management

b) configuration management

A survey of the tools database mentioned above reveals that the absolute majority are more traditional code analysis tools. What is changing is the level of sophistication and variety of platforms and environments the tools are available for. A few exceptions are vendors like Rational Corp., McCabe & Associates, Inc. and Computer Associates which offer a more comprehensive set of tools.

Tools also vary greatly in price – from several hundred to tens of thousands of dollars. Tool acquisition becomes complicated task involving risk estimation, cost analysis. There many hidden costs to be considered such as installation, configuration, maintenance and, especially, training.

Bottom line is: if the organization does not have a fairly mature testing process, then tools of any sophistication will not help and will become a “shelfware”. That is why central argument if Kit’s book is – “start slowly, evaluate what is achieved, what state you are at, then develop a realistic plan of gradual improvement”. At some time in that process the time will come for tools. What is the best place to start for the immature software shop? Organization.

Organizational approaches to testing

Kit starts this chapter of the book by stating a key prerequisite for success – positive culture. The roles of the developer and the tester have to be equally rewarded. In fact, he observes, in most successful organizations best people freely migrate between development and testing groups. People who are good at both activities are considered the most valuable. Kit identifies the most frequent dangers testing manager faces:

- test independence is weak

- people in testing are not adequately rewarded

- testing is understaffed or staffed with junior people

- testing lacks authority

- management bandwidth problem

- quality is not emphasized

- test manager has concerns about his career growth

Organizational structures Kit proposes deal with one or several of these dangers. Lets consider approaches suggested by Kit. He intentionally presents them from simplest to more and more complex, arriving at better structural solutions and discussing advantages and disadvantages of each along the way.

Approach 1. Testing is each person’s responsibility

M

	T – tester P – developer M – manager T = P

P P P P P P

Advantages	Disadvantages
Most natural solution as an evolution from development team	Programmers are inherently incapable of testing their own programs.

Approach 2. Testing is each unit’s responsibility

M

	T -> P P -> T Developers test each others work.

P P P P P P

Advantages	Disadvantages
Solves the disadvantage of approach 1.	Programmer bandwidth, i.e. programmer is not only responsible for development of his modules, but also has to find time to test colleague’s work. Related to that is the extra training in testing methodologies, acquiring of testing skills.

Approach 3. Testing is performed by dedicated resource

M

	P T Test developer is a responsible solely for testing

P P P P P T

Advantages	Disadvantages
Solves the developer bandwidth problem. Single team	Management bandwidth, i.e. now the development manager has to understand how to manage different processes: development and testing.

We see that Kit guides us to understanding of the necessity of a separate testing group.

Approach 4. The test organization in QA

M M TM

P P P P P P P P P P P P T T T T

Advantages	Disadvantages
Solves the management problem of approach 3.	Where to put the group in the organization?

The disadvantage is resolved by making the test group a part of QA. This raises new concerns as illustrated below:

PDO QAO

PDG

TDG

PDO – Product Development Organization QAO – Quality Assurance Organization

PDG – Product Development Group TDG – Test Development Group

Advantages	Disadvantages
Solves the management problem of approach 4 by creating a formal test organization which is part of QA	Possible teamwork problems. TDG can get lost in QA group which has other responsibilities PDG is not accountable for producing quality. Who owns quality?

The problems of the approach 4 can be dealt by placing the test organization in the development. The success of this solution depends on the ability of the senior management to manage both development and testing organizations. However, approach 5 is a viable solution if the overall organization is not very big.

Approach 5. The test organization in Development

PDO

PDO – Product Development Organization TDG – Test Development Group

PDG – Product Development Group

Advantages	Disadvantages
Solves the management problem of approach 3 by creating a formal test organization which is part of development	Does the senior development manager meet the test management requirements?

The next 2 approaches are suitable for large organizations. They use a centralized test organization that is set up in the development division. As we have seen in the earlier approaches, the risk propagates up to higher-level management. In approaches 6 and 7 a lot depends on the ability of vice president who has the centralized power. With good senior management these structures perform extremely well, argues Kit.

Important issue that is resolved by this type of structure is the career path for test management.

Approach 6. Centralized test organization

VP

VP – Vice president TDG – Test Development Group

PDG – Product Development Group

Advantages	Disadvantages
Solves the senior management problem of approach 5 by centralizing the formal test organization which is part of development	Test organization live or dies by VP Potential lack of teamwork at individual/project level Possible lack of consistent test methodologies across VP organizations

The final approach is applicable to the organizations of even larger size. Consistency issue is addressed by creating independent Test Technology Group

Approach 7. Centralized test organization wit a Test Technology Center

VP VP SE

		TTG

TDG

TDG

PDG

TDG

TDG

PDG

PDG

VP – Vice president TDG – Test Development Group

SE – Software Engineering PDG – Product Development Group

TTG – Test Technology Group

Advantages	Disadvantages
Solves the consistency problems of approach 6 by centralizing a test technology group which is part of software engineering	Test organization live or dies by VP Potential lack of teamwork at individual/project level

To help in selecting the right approach Kit suggests to construct a decision matrix with approaches along one dimension and criteria for selection ( ranked according to some predefined scale ) along other dimension. Possible candidates for criteria are ownership of test technology, fast decision making, enhanced teamwork, career path, etc.

Summary: current practices and trends

Two major trends as observed by Kit are arrival of complex software systems which are using advanced GUI and Client/Server application architectures. Another major shift observed is increase of tester – to – developer ratios in favor of the tester. Companies like Microsoft and Lotus have a ratios of 2:3 and 2:1 respectively.

Kit, throughout his book, carries a central theme – centralization of the testing process in the hands of a separate testing organization. He also emphasizes that software testing has become a specialized profession. The developer, according to Kit, is inherently incapable of testing his own code ( except, maybe, at a unit level ).

Although this is generally true, especially for the big software companies, it is not necessarily universally true. Even such a giant as IBM recognized disadvantage of highly centralized and hierarchical structure of its organization and moved towards more independent groups granting them total ownership of the process.

In fact, Alistair Cockburn work with IBM on OO development methodology in 1990 may be considered as one of the earliest definitions of the “lightweight” methods ⁴. Lightweight methods are less document-oriented, more code-oriented, adaptive rather than predictive, people-oriented rather than process-oriented. These features can be viewed as advantages in a world where change is accelerating. Traditional, “heavyweight” methods, including the ideas presented by Kit, are too inert, bureaucratic and bulky to be implemented where the fast reaction to change is essential. Business software, e-commerce software are good examples of such domains.

Bertrand Meyer, the pioneering designer of Eiffel language, coined the term “design by contract” ⁵, which, in essence, means less exhaustive testing, since assertions ( which are built in Eiffel, as well as constructs like preconditions, postconditions and invariants ) make code “self – testing” in some sense. His idea was to provide developers with powerful tools for rapid development of industrial strength applications with less manpower and less testing as a separate process from the development.

“Extreme Programming” is another interesting recent movement, leaded by Kent Beck. It emphasizes continuous integration of the system, daily, even hourly build ( Microsoft is famous for its daily build strategy ), “test everything” and “test and then code” attitude, customer involvement to help steer the process, importance of acceptance testing ( giving more power to customer ) ⁶, concentration on the simplicity ( delivery of just what customer needs now ), collective ownership of the code.

References:

1. Edward Yourdon. “Decline & Fall of the American Programmer”, Yourdon Press, 1993.

2. Quality Assurance Institute. “Survey Results on Software Testing”, 1993. Copied from www.qaiusa.com

3. Tool Descriptions and Characteristics. This Product Database file can be downloaded from www.qaiusa.com. Database describes test tool activities, languages and environments each tool is designed for, and vendor info. The Quality Assurance Institute web site also contains very good downloadable summary articles on testing techniques and description of knowledge required of tester professionals. Another good source of detailed information on tools and testing practices is the site of Software Testing & Quality Engineering Magazine: www.stqemagazine.com

4. Martin Fowler provides excellent survey of “lightweight” methodologies in his article: www.martinfowler.com/articles/newMethodology.html. The article has links to resources describing A. Cockburn’s Crystal family of methodologies.

5. Bertrandt Meyer. “Reusable Software”, Prentice Hall, 1994. A good source of Meyers ideas on “development by contract” as well as Eiffel technology is the site www.eiffel.com.

6. The ideas of “Extreme Programming” as applied to testing are best expressed in Ronald E. Jeffries article “Extreme Testing” published in “Software Testing & Quality Engineering” magazine, March/April 1999. It can be downloaded from www.xprogramming.com/publications/software_testing.htm