Unit 21 Software Applications Testing

Unit 21 Software Applications Testing


In this paper we will try to examine in details some of the fundamental aspects of software engineering in order to evaluate the vital part of testing early in the software lifecycle.

The topics covered are Software Engineering, Software Life Cycle, Software Testing, and Software Quality and of course examine the importance of Software Testing in the early stages of the life cycle.


Software Engineering

“Software engineering is the technological and managerial discipline concerned with systematic production and maintenance of software products that are developed and modified on time and within cost estimates.” [1] (Fairley, R.)

People involved in Software Engineering are concerned with the design, the construction and the testing of complex software systems. The integration of both hardware and human interfaces are included.

Software engineers are very much concerned with the development of quality software, on time within budget and meeting the design requirements.

Software Life Cycle

“Software Life Cycle is a description of the events that occur between the birth and death of a software project inclusively.” [2] (Succi G.) The waterfall model is a popular version of the systems development life cycle model for software engineering. The waterfall model describes a development method that is linear and sequential.

Waterfall development has distinct goals for each phase of development. Once a phase of development is completed, the development proceeds to the next phase and there is no turning back.

Development moves from concept, through design, implementation, testing, installation, troubleshooting, and ends up at operation and maintenance.

[pic] Figure 1.1 The waterfall model

“Software testing is the action of carrying out tests, where a test is a technical operation that determines one or more characteristics of a given software element or system, according to a specified procedure.” [3] (RTI, 2002)

Software Testing

Software testing is the process of testing the functionality and accuracy of software by running it. Software testing is usually performed mainly for two reasons: errors detection and reliability estimation.

When testing, we are seeking to improve the quality of the software under test. We do this by finding and removing errors that would, if permitted to remain, detract from the operational efficiency or accuracy of the product and thereby detract from the software’s quality.

According to Table 1 software developers in the 60-70’s focused eighty per cent in coding and unit testing phase spending very little time in the requirements analysis phase.

During the 80’s they reduced the time they spend on coding and unit testing phase and increased the time spend on the analysis of the requirements.

Since the 90’s, more and more software developers spend more time and resources trying to test software as a whole instead of independent entities. | |Requirements Analysis |Preliminary Design |Detailed Design|Coding and Unit |Integration and Test|System Test | | | | | |Testing | | | |1960-1970 |10% |80% |10% | |1980s |20% |60% |20% | |1990s |40% |30% |30% |

Table 1 (Institute of Technology, Lund, Sweden)

The amount of effort spent on determining the developmental requirements of a software solution has increased in importance. Forty percent of the software developer effort is now spent in the requirements analysis phase.

An improved software testing infrastructure would allow developers to find and correct more errors sooner with less cost.

The Importance of Testing

The ultimate concern of all those involved in the production of the code that implements a software system must be that the software produced is of high quality.

Testing cannot show if a piece of software is free from errors. Neither can it be guaranteed to eliminate all errors which are present.

A consideration of some statistics collected about the cost of testing is helpful in evolving attitudes to tests.

For example, Boehm (1976) notes that “on average 40% of the effort put into a programming project is devoted to the detection and eradication of errors.”

This measure can increase much if the cost for corrective maintenance, traditionally hidden in the maintenance phase, is included.

In some cases the cost has been seen to rise as high as 400% of the cost of the lifecycle up to implementation.

Figure 1 Boehm (1976)

The movement of the worldwide market for software testing tools shows the importance of software testing. “The worldwide market for software testing tools was $931 million in 1999 and is projected to grow to more than $2.6 billion by 2004” [4] (RTI, 2002)

The Application of Testing

When errors are detected later in the lifecycle, there is an increased amount of work that must be corrected and retested. Thus the cost of correcting errors at the later stages in the lifecycle will be much greater than in the early stages.

In programming, adequate testing throughout the development of the requirements, the design, and the implementation is vital.

The complexity of interactions between program and system elements is much increased. If testing is not adequately performed at the higher conceptual levels, essential problems which can not be removed without radical system redesign may not be detected until significant development effort has been spent. In the worst case, implementation of the requirements as they stand may be seen to be impossible.

The most damaging errors are those which are not discovered during the testing process. So they remain undiscovered until the system “goes to live” causing failures.

Software Quality

Software quality is defined as: “Conformance to explicitly state functional and performance requirements, explicitly documented development standards, and implicit characteristics that are expected of all professionally developed software.” [5] (Pressman R. 1987)

Software quality is among the most common topic in discussions about computers. Organizations, both public and private, have invested huge amounts of money in research to develop methodologies for producing software of quality. Schools have devoted large efforts to curricula in the engineering of software. Yet, many people remain dissatisfied with the progress of the field over the past years.

The methodologies that are currently employed to produce software quality are based on the concept that quality is strongly related to rigor in the specifications and texts that appear throughout the software design process.

Program construction techniques that maintain tight relationships between program structure and dynamic behavior are considered to be essential. Readability, modularity, modifiability, style, adequacy of comments, and good tests are considered to be important guidelines.

The importance of software quality can be viewed from the positive and negative impacts it has on an organization. The existence of better software quality performance will improve customer satisfaction and confidence leading to increased sales and profitability.

System failure and incident management are costly events in terms of both money and future impacts such as to recover and maintain market and consumer credibility.

Quality must be built in to a system for it to be effective. The only way to do so is to ensure that all activities of the development and maintenance process are organized to that end. Each technique has got its own merits and weaknesses. This must be recognized for the maximum benefits to be extracted from the technique.

Quality problems can never be eliminated, only reduced. Also the needs of the system users change constantly. Therefore, it is essential to continually repeat all the techniques within an environment where there is participation by everyone and support and commitment from management to achieve effectiveness.

Software Failures

Large organizations in the last decades are paying more attention and effort in testing activities. They are using procedures, techniques and methods in order to test the code and also to test other stages of the software life cycle such as analysis and design.

Software failures led organizations to do this. The number of project failed during the last years is gigantic. The result of these failures was not only great amounts of money wasted but most importantly human lives.

Some of these failures may have been avoided with the use of early testing methods.

For example, the explosion of Ariane 5 in 1996 was caused by a software error in the inertial reference system.

| | | | | | | | |Airbus A320 |Ariane 5 Galileo Poseidon |Lewis |Zenit 2 Delta 3 |DS-1 Orion 3 | | |(1993) |Flight 965 |Pathfinder |Near |Galileo Titan 4B | | | |(1996) |USAF Step |(1998) |(1999) | | | | |(1997) | | | |Aggregate cost | |$640 Million |$116.8 Million |$255 Million |$1.6 billion | |Loss of life |3 |160 | | | | |Loss of data | |Yes |Yes |Yes |Yes |

Table 2 (NASA IV&V Centre 2000)


In an ideal world, software development processes should be so advanced that no errors will enter in software during development.

Modern practices can only help to reduce the number of errors and not prevent all errors. However, even if the best practices were available, it would be risky to assume that no errors enter a system, especially if it is a system requiring high integrity.

The use of error analysis allows for early error detection and correction. When an error is made early in the lifecycle and it goes undetected, problems and costs can increase rapidly.

An incorrectly stated requirement may lead to incorrect assumptions in the design, which in turn can cause following errors in the code. Therefore, even a critical error may remain undetected and be delivered along with the final product.

This undetected error may subsequently cause a system failure, which results in costs not only to fix the error, but also for the system failure itself.

Sometimes the cost of fixing an error may affect a decision not to fix an error. This is particularly true if the error is found late in the lifecycle. For example, when an error has caused a failure during system test and the location of the error is found to be in the requirements or in the design, correcting that error can be expensive.

Figure 2 (University of Michigan)
Sometimes the error is allowed to remain and the fix of it is postponed until the next version of the software.

People responsible for these decisions may justify them simply on the basis of cost or on an analysis which shows that the error, even when exposed, will not cause a critical failure.

Decision makers must have confidence in the analyses used to identify the impact of the error, especially for software used in high integrity systems.

A strategy for avoiding the high costs of fixing errors late in the lifecycle is to prevent the situation from occurring altogether, by detecting and correcting errors as early as possible.

Studies have shown that it is much more expensive to correct software requirements deficiencies late in the development effort than it is to have correct requirements from the beginning.

In addition to the lower cost of fixing individual errors, another cost benefit of performing error analysis early in development is that the error spread rate will be lower, resulting in fewer errors to correct in later phases.

Thus, while error analysis at all phases is important, there is no better time, in terms of cost benefit, to conduct error analysis than during the software requirements phase.

Planning for error analysis should be part of the process of planning the software system, along with system hazard analysis and software criticality analysis.

System hazard analysis is used to identify potential events and circumstances that might lead to problems of varying degrees of severity, from critical failures resulting to serious malfunctions in the system.

Software hazard analysis focuses on the role of the software relative to the hazards. Software criticality analysis may use the results of system and software hazard analyses to identify the software requirements whose erroneous implementation would cause the most severe consequences.

Criticality analysis may also be used to identify project requirements that are essential to achieving a working software system. Critical software requirements are traced through the development process, so that developers can identify the software elements whose errors would be catastrophic.

The results of hazard analysis and criticality analysis can be used to build an effective error analysis strategy. They aid in choosing the most appropriate techniques to detect errors during the lifecycle.

They also aid in the planning of the error removal process. Lastly, they aid in the selection of metrics, statistical process control techniques, and software reliability estimation techniques.

Error analysis efforts and resources can be concentrated in the early stages. Error analysis techniques should be chosen according to which type of errors they are best at locating. The selection of techniques should take into account the error profile and the characteristics of the development methodology.

No project can afford to apply every technique, and no technique guarantees that every error will be caught. Instead, the most suitable combination of techniques should be selected to enable revealing of as many errors as possible in the earlier phases.

For example, assume that a team finds one coding error early in development, decides to prevent similar errors by implementing a team-wide coding standard which prohibits the error-prone coding construct responsible for that error, and then ends up preventing 100 errors throughout the project.

This error prevention effort would relieve the team from having to chase 100 errors; according to the industry averages this translates to a savings of $10,000 (assuming that each error was discovered early in development).

“Based on the software developer and user surveys, the national annual costs of an inadequate infrastructure for software testing is estimated to range from $22.2 to $59.5 billion” [6] (RTI, 2002)

Figure 3 (RTI, 2002)

Error prevention not only reduces the total time and money required to find and fix defects, but also reduces the expense of change.

Most applications need to change as the market and business requirements evolve.

However, each change typically introduces errors, and the more errors that are in the code prior to a change, the more additional errors that change is likely to introduce.

As a result, it is considerably more expensive to change a project containing many errors than it is to change a similarly-sized project that is virtually error-free.

Without error prevention, every change usually introduces a large number of errors that are costly to find and fix.

However, if a team proactively prevents errors and detects errors as soon as they are introduced, each change is less costly and problematic; fewer errors need to be detected and fixed, and any new errors can be removed immediately, before they generate additional errors.


Most of those who work in software development understand that the longer an error is allowed to remain in the software before it is discovered, the more expensive it becomes to fix

What many people do not realize is that it is also possible to prevent most errors completely. In addition to reducing development costs, error prevention can dramatically reduce software development and maintenance costs, increase team efficiency, and facilitate the software changes that are required to stay competitive in today's market.

By preventing errors, software teams reduce the total quantity of errors that would push the project costs.

As a result, the team has significantly fewer errors to find and fix which translates to reduced costs and a more rapid, efficient development process.


[1] Fairley, R. Software Engineering Concepts. New York: McGraw-Hill, 1985

[2] Giancarlo Succi, Jason Yip, 2000

[3] The economic impacts of inadequate infrastructure for software testing, RTI, 2002

[4] The economic impacts of inadequate infrastructure for software testing, RTI, 2002

[5] Roger Pressman, (1987) Software Engineering: A Practitioner's Approach (3rd Edition). McGraw-Hill.

[6] The economic impacts of inadequate infrastructure for software testing, RTI, 2002