CUNY B r o o k l y n  C o l l e g e            Department of Computer and Information Science
CIS 763X: Software Methodology
with Dr. D. Kopec                           Fall, 2000

Back tondividual Class Presentations

Software Testing in the Real World

by Edward Kit,

Chapter 1—Chapter 6: Testing process maturity and the framework for test process improvement

Presented by Yu Yang

The six essentials of software testing:

1. Quality—The quality of the test process determines the success of the test effort

 The quality of a software system is primarily determined by the quality of software process that produced it. Testing has its own cycle. The testing process begin with the product requirements phase and from there parallels the entire development process. In other words, for each phase of the development process there is an important testing activity. An immature test process within an immature development organization will result in an unproductive, chaotic, frustrating environment that produces low-quality results and unsatisfactory product.

2. Prevent defect migration—Prevent defect migration by using early life-cycle testing techniques

  From the chart 1, we can see that more than half the errors are usually introduced in the requirements (includes the functional design) phase. The cost of errors is minimized if they are detected in the same phase as the are introduce, and an effective test program prevents the migration of errors from any development phase to any subsequent phases. In other words, errors are not self-contained. As they migrate to the units downstream, they take on new forms.

Defect distribution
(Chart 1)

3. Time—The time for software testing tools is now

  After many years of software testing development, there is a wide variety of tool vendors to choose from, many of which have mature, healthy products. It is important to have a strategy for tool acquisition, and a proper procedure for handling tool selection. Tool acquisition is an area where there may be a strong case for seeking independent expert advice.

4. Responsibility—A real person must take responsibility for improving the testing process

  Management should appoint an architect or small core team to prioritize potential improvements and lead the testing improvement effort, and must make it clear that they will give their ongoing support. When all is said and done, software testing is a process that requires people who take responsibility for its improvement. For the testing process to improve, someone must plan and manage the progress.

5. Human resource—Testing is a professional discipline requiring trained, skilled people

  Testing is not an entry-level job. The software testing process has evolved considerably, and has reached the point where it is a discipline requiring trained professionals. To succeed today, an organization must be adequately staffed with skilled software testing professionals who get proper support from management. Testing should be independent, unbiased, and organized for the fair sharing of recognition and rewards for contributions made to product quality.

6. Attitude—Cultivate a positive team attitude of creative destruction

  Cultivate a positive team attitude of creative destruction is very important for the software development. Establishing the proper “test to break” mental attitude has a profound effect on testing success. Good testers must have testing attitude:
(1) Testers hunt errors, (2) testers are destructive-but creatively so, (3) testers pursue errors, not people, (4) testers add value.
 

Critical choice: what, when, and how to test

  Fifty percent or more of the development organization’s time is frequently devoted to error detection and removal. Risk management is very important for testing. We need to work out how risky they are before we do them, and take all reasonable precautions against their coming out badly.
(Chart 2) see presentation outline

  Chart 2 is a typical software life-cycle phases. It’s included concept, requirements, design, implementation, test, operation and maintenance. Testing also has a life cycle that parallels the software development cycle, one that includes both verification testing and validation testing. Testing processes should, as far as possible, be integrated at the most effective points in the development life cycle. From chart 1 we can see that more than 50% of all defects are usually introduce in the requirements stage alone. Test early prevent defect migration. Every time there is an opportunity to find a defect and we don’t find it, and it is allowed to migrate to next stage, it’s going to cost much more to fix when we do find it.

(Chart 3) see presentation outline

  The basic forms of testing are full testing, partial testing, endgame testing and audit-level testing. Effective testing is testing that is successful at detecting errors. Effective testing removes errors. But how do we know how much testing we should aim to do? Chart 3 shows the cost effectiveness. The more effective the error detection, the greater the savings in development and maintenance costs over the life of the product. Typically we use full testing for critical software or any software that will have heavy and diverse usage by a large user population, and we use partial testing for small, non-critical software products with a small, captive user population.
 

Framework for testing

    Any software engineering method that is common practice in software development projects is usually also applicable to software testing efforts. This means that the disciplines of planning and analysis, formal documents, configuration management, measurement, standards, tools, and procedures can be applied to testing.

1. Planing

  It is easy to see why medium-size software development project becomes out of control if the planning is not done properly. Planning is the first step of every testing activity. If testing is to be included as a serious part of software engineering, testers have to take time to analyze the product. Time spent on early planing is never wasted, and usually the total time cycle is significantly shorter. On the planning, we need to consider the issues both the verification planning and the validation planing.
For each type of verification, the issues to be addressed are as follows:

2. Software engineering maturity and the SEI

  The mission of the SEI is to provide leadership to advance the state-of-the-practice of software engineering to improve the quality of system that depends on software. The SEI defines five process maturity levels as part of a process model: (1) Initial, (2) Repeatable, (3) Defined, (4) Managed, (5) Optimizing. The SEI CMM describes goals and activities that are of vital importance to anyone striving to improve their everyday testing process.
   There are special consultancy service organizations trained in the SEI model which conduct software processes assessments (SPAs) on request. SPA is a method developed by the SEI to help organizations evaluate their software process maturity and identify key areas for improvement. They look at the practices an organization is using and recommend practices that should be implemented now to improve processes.

3. Configuration management

   Configuration management is a four-part discipline for: (1) Configuration identification, (2) Configuration and change control, (3) Configuration status accounting, (4) Configuration audit.
   It is essential for software project with more than a few people, and with more than a modest change volume. Configuration management problems come from confusion and lack of control. These problems can waste enormous amounts of time; they often happen at the worst possible moment; they are almost always very frustration, and they are totally unnecessary.

4. Standards

   Some standards are particularly important to the testing practitioner. They can provide a benchmark for writing documents like requirements, so that testers and others doing verification have a framework for what they can expect to find. It also tells us what to put into key test documents, such as a test plan. We can use some standards as guide to our software testing. such as IEEE standards, ANSI standards, ISO 9000 standards, SPICE and other standards.

5. Formal documents

A formal document is a document of which the form, general content, and timing have been agreed in advance. It is a deliverable. It has proven to be an absolutely essential element of any successful project. We need documents in order to put decisions in writing. These include requirement specification, functional design specification and internal design specification. A mature development organization will produce these documents and faithfully maintain them for the life of the product. Testing produces its own formal documents corresponding to its own life cycle.

6. Testware

   Testware is produce by both verification and validation testing methods. Testware includes verification checklists, verification error statistics, test data, and supporting documentation, like test plans, test specifications, test procedures, test cases, test data, and test reports.
   It is important to maintain testware, just as software and hardware are maintained. Testware management, though not directly related to error detection, has an enormous impact on the efficiency of the test effort.

7. Measurement

   Measurement provides an idea of the real program complexity. It helps in planning the test effort, and in prediction how many errors there are and where they occur. Measurement of the number and types of errors detected provides and accurate idea of our efficiency at verification and the weaknesses in our development process. Without measurements, we never know whether we’re moving forward or backward or standing still.

8. Tools

    There are a wide variety of tools available today to provide assistance in every phases of testing process. Like all testing activities, the use of tools needs to be integrated with the larger picture of our development process.

Summary

 Software testing is very important for the software product. It is professional, not an entry-level job. To understand testing and to improve testing practice, it is essential to see the software testing process in its broadest terms—as the means by which people, methodology, tools, measurement, and leadership are integrated to test a software product.
 
 
 
 
 

Comments and suggestions e-mail to Sergey D.