chapter 6 information and software quality management

Chapter 6 Information and Software Quality Management

What is Software Quality Management?

Quality is a goal for an entire information system, not just the software portion, and it must be managed accordingly.

Effective quality management assures that a process is established and followed to build quality into Information Systems. It is NOT the purpose of quality management to "inspect in" quality after a product is built.

By the end of this lesson you will be able to identify commonly-used definitions of software quality and list software quality “ilities” most applicable to different categories of software-intensive systems.

Specifically, you will learn to:

Recognize common definitions of software quality

Define three different perspectives on software quality

Describe ways of determining software “size"

Define Error Density and its role as a software quality factor

Recognize typical software quality factors and “ilities”

Identify what influences the choice of software quality factors

Define Software Quality Assurance and outline its key processes

Describe methods and techniques that can influence software quality

Software Quality Management Program

A quality management program includes the following activities:

Monitor, measure, analyze, control and improve processes

Reduce product variation

Measure/Verify product conformity

Establish mechanisms for feedback on product performance

Implement an effective root cause analysis and corrective action system

The Quality Management Team

Many of the quality management techniques originated in software development, and were found to be of value throughout the acquisition of an information system.

As previously noted, it is not the purpose of quality management to "inspect in" quality; so, to enhance the success of obtaining high quality systems, quality management teams should be formed.

To be effective, a quality management team must have an independent reporting line to senior management.

The quality management team must be big enough to monitor the performance of all key planning, implementation, and verification activities. This generally requires a team of about 5% the size of the development team.

Software Quality

Engineering quality into a product requires an effective defect prevention program that engages accurate defect detection and analysis to determine how and why defects are inserted. In other words, it is not enough just to find defects. We need to fix the processes that generated these errors as well.

Software is a complex, changeable, conformable, and invisible product.

Because of this, trying to define software quality and measuring it accurately can be like nailing Jell-O® to a wall.

Nevertheless, the success of a software-intensive project is determined largely by the quality of its software

Poor Software Quality

Poor-quality software in the DoD:

Has caused cost overruns and late fielding of mission-critical systems

Can impact national security and the safety of operational personnel

This illustrates why defining software quality and understanding the issues associated with its definition and measurement are crucial software acquisition management skills

Perspectives on Software Quality part (1)

Perspectives on Software Quality

Software quality is a difficult thing to define and measure. What constitutes "quality" depends largely on your perspective, i.e., your relationship to the product in question. That perspective drives the particular quality attributes that are important to a project stakeholder.

Learning Objectives

This topic provides an overview of software quality and includes:

Common definitions of software quality

Different perspectives or viewpoints on what constitutes software quality

Once you have completed this topic, you will be able to apply different criteria and perspectives to define what constitutes quality in software.

Software Quality Challenges

Defining and measuring software quality can be difficult for Program Offices because:

Software is an invisible product—it is hard to measure what cannot be seen.

Quality is often an unknown commodity until the testing phase, but if you only measure quality at the end, it is often too costly to fix.

There are a number of different ways that "software quality" can be defined.

Definitions of Software Quality Some of the more common ways that "software quality" can be defined are shown below.

Five Definitions of Software Quality

1. User Need—Ability to satisfy given user needs. 2. Combination of Attributes—The degree to which the software possesses a desired

combination of quality attributes.

3. Conformance—Conformance to requirements.

4. Measures— A set of measurable characteristics that satisfy the buyer, the users, and the maintainers.

5. No Failures—The ability of a delivered product to support the user's needs without failure, problems, and errors.

Software Quality Perspectives

Software quality can be viewed from many different, equally valid perspectives. Three are shown on the cube below. Examine these perspectives to see how the definition of software quality is influenced. Click each side on the cube before proceeding

1. Contractual Perspective—The Contractual Perspective on software quality is sometimes referred to as a "buyer-seller" perspective. In this perspective of quality, if the developed software meets the "requirements" specified in your contract, it is by definition "Quality Software." The Defense Acquisition Guidebook (DAG) specifically states that "the supplier is responsible for the quality of its product. The government should allow the developer to define and use their preferred quality management system that meets required program support capabilities".

2. Attributes Perspective—The Attribute Perspective is a generic model of software quality that is based on a hierarchical, goal-oriented quality framework. This framework is used as a disciplined tool both to help to define the components of software quality, as well as to enable its measurement.

3. User-Focused Perspective—This perspective states that those software products that best satisfy user needs are quality products.

The Contractual Perspective

This perspective is embodied in many software development standards.

Such standards are used in a software acquisition environment. They help define how the "buyer" and "seller" interface, interact, and manage the project.

For example, some standards define software quality as "the ability of software to satisfy its specified requirements."

A key to success is to incorporate systems engineering/design quality into the product by defining the product or service quality requirements from the beginning and then providing the supplier with the maximum degree of flexibility to meet these requirements

The Attributes Perspective Framework This perspective allows software quality to be quantified and measured.

There are three levels in the framework.

At the top level are software quality factors. These are project-specific quality characteristics that are important to an acquirer or other customer of the system. For example, the guidance control software in the LRATS E-Sentinel missile system is both safety and mission-critical. Therefore RELIABILITY would be a key top-level software quality characteristic!

The middle level of the framework includes those specific software-oriented attributes, which are technically accepted as best supporting the top-level quality requirement. For example, Coding Simplicity, Module Consistency and Error Handling are criteria that when properly applied to software design and coding, are known to improve its overall reliability.

At the bottom are technical measures, some of which can be programming language-specific, that determine the degree to which the quality attributes are actually present. For example, Coding Simplicity depends on limited employment of branching and looping

commands, use of single entry and exit point to routines, etc. These can be measured in the software by automated tools and an assessment of the degree of Coding Simplicity made.

The User-Focused Perspective

In the User-Focused Perspective, "the user" is not strictly limited to the end operator of the delivered product. "Users" can be anyone with legitimate interests in the delivered software. As such, users can include buyers, operators, suppliers, maintainers, testers and other system stakeholders.

Quality Perspectives and Priorities What project stakeholder(s) hold an attributes perspective on software quality?LogisiticianSystems EngineerEnd User

Each quality perspective embraces a different viewpoint of what is most important for software quality. Each project stakeholder is likely to have different software quality priorities based on his or her perspective.

Logisticians, who are responsible for software life-cycle support, would likely have an Attributes perspective. Quality attributes, such as maintainability and supportability, would be among their top priorities.

Systems Engineers have concerns about the ability of the software to work on a wide variety of target hardware. From a systems perspective, they would be concerned about all aspects of software quality but would consider portability and transportability as particularly important.

End Users, who use the system in a combat environment, operate from the User-Focused perspective. For them, performance and reliability are key software quality attributes.

Software Suppliers work and are paid under the terms and conditions of a contract. Quality from their perspective means meeting contractual requirements

Information and Software Quality Management Part (2)

Software Size

Some software quality measurements are evaluated relative to the "size" of the

software product. It is, therefore, important to understand the concept of software size in order to make sense of those measurements.

Software size can be determined in several ways. The most commonly used methods are:

Source Lines of Code (SLOC) Counts

Function Points (FP) Counts

Learning Objectives

This topic examines the SLOC and FP methods of determining software size and discusses the issues associated with them.

Once you've completed this section, you will be able to describe these two ways of measuring software size.

Source Lines of Code (SLOC)

A large body of Software Engineering quality data exists based on using SLOC, or thousands of Source Lines of Code (KSLOC) to determine quality attributes, such as Error Density. An example of Error Density values expressed in KSLOC (E/KSLOC) for several types of generic systems are displayed in a separate chart.

The chart is adapted from: Donald J. Reifer, Reifer Consultants, Inc, Industry Software Cost, Quality and Productivity Benchmarks "Error Rates Upon Delivery by Application Domain."

Source Lines-of-Code (SLOC):

Is a low-level, programming language-dependent measure of software size.

May be automatically calculated using a variety of Computer-Aided Software Engineering (CASE) tools

Appears easy to calculate but in reality can be difficult to accurately and consistently determine

Computer-Aided Software Engineering (CASE)

Computer-Aided Software Engineering (CASE) is an acronym used to refer to the use of computer-based tools to do software engineering. It includes software tools that are used in any and all phases of developing an information system, including analysis, design, and programming.

https://learn.dau.mil/CourseWare/66_12/course/L06_T03/pdfs/SAM101_L06_T03_P015.pdf

https://learn.dau.mil/CourseWare/66_12/course/L06_T03/pdfs/SAM101_L06_T03_P015.pdf

Counting SLOC

Some important factors to consider when measuring software size by SLOC:

SLOC counts are dependent on the specific programming language being used.

Counting rules can vary by supplier, language used, and project.

Precautions must be taken if direct comparisons of SLOC measures are to be made across projects. Because of differences in SLOC counting rules, such comparisons may be very difficult to determine accurately.

In the English language, many rules exist for "proper usage." For example, the first sentence in a paragraph is indented, the first word in a sentence is capitalized, and a sentence ends with a punctuation mark.

Therefore, counting punctuation marks such as "." or "!" or "?" would give an accurate count of the number of sentences in a segment of text.

SLOC counting uses a similar approach.

Programming languages use rules of syntax analogous to those used in English. These are used to help count SLOC.

For example:

Many languages (e.g., C++, Pascal, Ada) use a ";" to end a single line-of-code.

Special characters identify "comments" (e.g., a leading "-"in Ada). Though ignored by the computer, comments help make programs understandable.

Programming language format characteristics such as these form the basis for SLOC counting. However, other considerations regarding how they are actually interpreted apply as well.

Estimating SLOC The following page contains an Ada code module (named "Ignition countdown") that might be used as part of the Long Range Acquisition & Targeting System (LRATS) E-SENTINEL missile system. This simplified module is part of a much larger program that activates the missile warhead.

SLOC counts can differ dramatically depending which counting criteria is used. Some (many other equally valid ones exist!) of these criteria and perspectives on their use include:

Counting logical lines: only code that has meaning (is “executable”) to the

computer is counted since that directly impacts performance, memory use and error density.

Counting all lines: includes physical lines, comments and blank lines. Since comments improve maintainability and blank lines improve legibility, the effort taken by programmers to do this is reflected in the SLOC count.

Counting physical lines: Comments and blank lines are excluded. However, programmers frequently use inactive “debug” code to improve later testability; using this criteria, they would not be counted.

These three counting perspectives are illustrated in the following example.

Counting Logical Lines of Code includes only the lines of code that have meaning to the computer. Counting Physical Lines of Code includes all lines in the code; even blank ones and comments in the count. Exclude Comments and Blank Lines. This method excludes Blank lines and Comments from the count.

Many Ways to Count

The previous example illustrated three different ways to count SLOC on a very small Ada code example. Even in this limited example, a variation of nearly two-to-one (9 lines versus 16 lines) in SLOC count is possible.

Research done by the Software Engineering Institute (SEI) has identified over 50 ways to count SLOC.

Software Engineering Institute (SEI)The Software Engineering Institute (SEI) is a federally funded research and development center sponsored by the DoD. It provides leadership in advancing the practice of software engineering to improve the quality of software-intensive systems. The SEI is located in Pittsburgh, PA, and staffed by personnel from industry, academia, and Government.

Choosing a Method

When choosing a SLOC counting method, it is best to remember:

There is no right way to count SLOC.

Program Management Offices (PMOs) and suppliers must agree on the counting method at the onset of a project and keep it constant.

Lack of clear definitions or enforcement of selected definitions will yield SLOC counts of little value and measures that depend on them.

Measurement of Error Density is critically dependent on the accuracy of the SLOC counts.

SLOC counts ultimately impact all three of the classic Cost, Schedule, and Performance measures commonly used to judge project success.

http://www.sei.cmu.edu/

Select each measure below to see how they are impacted by SLOC.

Cost -Software-intensive systems typically include some measure of cost expressed as dollar per Source Lines-of-Code or $/SLOC.

Schedule -Software-intensive systems usually include a productivity measure expressed in units of SLOC per Staff Month or SLOC/SM.

Performance -For software-intensive systems, product quality and performance are directly related because error-ridden software can directly impact a system's operational performance.

In this regard, software quality is measured by Error Density where Software Size is expressed in Thousands of SLOC (KSLOC) or more specifically,

ERROR DENSITY=ERRORS/KSLOC.

Functional Complexity

Suppose you counted the number of rivets in an aircraft. Would a greater number of rivets make a more complex aircraft? Select the more complex aircraft.

Two images of the side of an aircraft. One has more rivets than the other.

No matter which image you choose, the feedback says "The number of rivets does not necessarily correspond to the complexity of an aircraft! Similarly, a high SLOC count does not always indicate a more complex software package."

Function Points

If you want to correlate a system's functional complexity to the ultimate size of its software, another metric, called Function Points (FPs), can be used. As opposed to SLOC, which is a low-level measure of size, Function Points are based on what a system actually does.

Function Points (FPs):

Provide an alternative way to calculate software size

Estimate software size based on what the program does (analysis of its high-level requirements)

Increase in number as the software tasks increase in complexity

The Aircraft Example Using the previous aircraft example, a Function Point analysis would evaluate the aircraft's capability, not its rivet count. Now, select the more complex aircraft.

cropduster and an Air Superiority Fighter.

No matter which image you click the feedback says, "This approach would result in the air superiority fighter having a much higher measure of complexity over that of a crop-duster, even though both might have approximately the same number of rivets."

Measuring Function Points

Function Point measures:

Are based on a set of rules that determine a system's size and complexity

Use the types and amounts of system-level requirements as input to the calculation

Should be identical for two systems that perform the same tasks

Function Points are weighted sums of counts of different factors. These factors relate to overall system requirements and can include:

Number of Inputs

Number of Outputs

Logic (or Master) Files Used

Number of Inquiries

Number of Interfaces

These factors are estimated and then multiplied by different complexity-based weighted "adjustment" factors, resulting in an overall Function Point count for the system being measured.

Comparing SLOC to FPs

As an example comparison of Function Point versus SLOC counts, consider what you are doing right now.

You are probably running this courseware on a computer running an operating system, such as Microsoft Windows XP®.

Consider that Microsoft Windows XP®, written in the programming language

C++, has approximately:

40,000,000 SLOC

730,000 Function Poin

Even though SLOC and Function Points are based on radically different approaches, some researchers have been able to determine approximate equivalents for two counting methodologies based on the programming language used.

Function Point are based on empirical relationships discovered to exist between source code and Function Points in all known languages. This method is based on tables of average values. It is useful for doing retrospective studies of projects completed long ago, and for easing the transition to Function Point metrics for people who are familiar with lines-of-code metrics.

Click the ruler to see how many SLOCs in various programming lanquages it takes to be equivalent to a single Function Point.

Click the image of a ruler to see a chart showing the average number of statements required in each programming language to accomplish one function point. Assembler=320 C=128 COBOL=107 Ada=71 DB Languages=40 Object Oriented=29 Query Languages=25 Generators=16.

Advantages of Measuring FPs

Compared to SLOC, Function Points, which are manually computed as a weighted sum of factors, have the advantages of being:

Language Independent: FPs do not depend on the specific computer programming language used.

Solution Independent: FPs are not impacted by the technology used.

Determined Early: FPs are established early in the system life cycle and thus useful for cost/schedule estimating.

Understandable: FPs are an intuitively understandable concept.

Disadvantages of Measuring by FPs

Compared to SLOC, Function Points, which are manually computed as a weighted sum of factors, have the disadvantage of being:

Training: Measuring FPs requires that analysts be trained to exercise judgment.

Manual Calculations: FPs must be counted manually, as automatic Function Point counters do not exist.

Limited Application: Measuring FPs may not work well for computationally-intensive software as typically found in DoD embedded systems.

Lack of Tracking Support: Unlike SLOC, FP counts do not support detailed project tracking.

SLOC vs. FP: Which is Best?

Many heated technical arguments have occurred in trying to answer the question, "Should SLOC or Function Points be used?" The best answer, as illustrated below, is to use both

Two robots labeled SLOC and FP boxing. The FP boxer is getting the better of the fight.

Function Points are useful during the early stages of a project, when top-level requirements have been defined, but little is known about the detailed design. At this stage, SLOC takes a beating! Function Points can be accurately assessed, but SLOC estimates may be significantly in error.

The SLOC robot begins to fight back and overtake the FP robot.

SLOC makes a come back as the software design matures and more details are defined. As software modules are coded, estimated SLOC values can be compared to actual counts, and adjusted accordingly. Because SLOC is a low-level measure, it enables detailed project tracking as programming progresses.

What statement describes the SLOC method of determing software size?

Is primarily a 'low-level' measure

Error Density Part (3)

Error Density is the most commonly-used indicator of software quality. It is important to understand how Error Density is determined, because this quality measure is only as accurate as the values used to calculate it, and it can be biased in subtle ways.

Error Density measurements impact members of the Program Management

Office IPT in various ways. Error Density values are used as the basis for many important project decisions, such as assessing development risks, judging contract performance and estimating sustainment costs, to name a few.

Learning Objective

This section introduces the most commonly used software quality measure, Error Density.

Error Density is:

The ratio of total number of errors to software "size"

An indicator of software quality

Generated during software testing

Deceptively simple in concept

Once you have completed this topic, you will be able to define Error Density and its role as a software quality factor.

The Error Density Formula

Error Density is calculated by dividing the number of errors found during testing by the software size. Accurate and consistent measurements of the numerator (Number of Errors) and the denominator (Size) are:

Essential for obtaining valid Error Density values

Especially important to the Project Management Office (PMO) team because Error Density values can impact many functional disciplines

How is Error Density Used?

Listed below are examples of how the Project Management Office (PMO) can use Error Density values:

Contracting Officers: To establish criteria for an award fee or incentive contract.

Logisticians: To determine software support costs by estimating errors remaining in fielded software.

Testers: To determine readiness to start critical testing phases, such as Operational Testing.

Systems Engineers: To assess development schedule risks and to estimate software and system reliability.

How is Error Density Used?

Listed below are examples of how the Project Management Office (PMO) can use Error Density values:

Contracting Officers: To establish criteria for an award fee or incentive contract.

Logisticians: To determine software support costs by estimating errors remaining in fielded software.

Testers: To determine readiness to start critical testing phases, such as Operational Testing.

Systems Engineers: To assess development schedule risks and to estimate software and system reliability.

Counting Errors

Software errors can be counted in a variety of ways. In many Department of Defense (DoD) projects, a Software Problem Report (SPR) or a Software Test Report (STR) is generated to:

Formally document software errors discovered during testing

Rank problems by their severity into priority categories, usually from 1 (catastrophic errors) to 5 (cosmetic errors)

Error Counts and Quality

Using data from SPRs, software error counts can be plotted over time by error category (e.g., 1 through 5). These charts can:

Provide an overall insight into the current level of software quality

Show possible trends in software quality, by the slope of the line

Error Density=Number of Errors/Software Size

Summary

Error Density is the most commonly used measure of software quality. It is calculated by dividing the number of errors present by the software size. It is important to calculate Error Density values as accurately as possible, because the PMO uses it as the basis of many project decisions.

Additionally, Error Density:

Is a common software quality measure because it is intuitively understandable and related to the factor of "correctness."

Is expressed as "Errors per KSLOC" or "Errors per Function Points."

Tends to measure quality later in development when fixes are expensive.

Should be used with other software quality measures.

Software Quality Factors part (4)

Important research to identify dimensions of software quality beyond Error Density was sponsored by the U.S. Air Force's Rome Air Development Center, now the Information Directorate of Air Force Research Laboratory, Rome NY.

Their research discovered that there were many variables making up software quality. Most software-intensive projects do not have the schedule or budget to accommodate tracking all of these variables. It is, therefore, important to identify those that are critical to satisfy system requirements, and do a good job of measuring and tracking them.

Learning Objectives

This section discusses software quality factors and their attributes. Research has been able to identify many software quality attributes and link them to various programming techniques. By evaluating characteristics of the software itself, this linkage provides an indirect way to quantify software quality.

When you have completed this topic, you will be able to list and define typical information and software quality factors and ways that they are measured.

The Attribute Perspective

One way to approach software quality is from an Attribute Perspective. The Attribute Perspective:

Uses a framework of quality factors

Breaks factors into various combinations of relevant quality attributes

Measures each individual attribute

Yields an overall quality assessment by "rolling up" each attribute measure

The Attributes Perspective framework has three levels.

At the top level are project specific quality factors that are important to the acquirer.

The middle level includes software-oriented attributes, which technically support top-level quality requirements.

At the bottom are technical measures that determine the degree to which quality attributes are present.

Selecting Quality Factors

The challenge lies in determining which factors and attributes are relevant to a project. When selecting the quality attributes to measure, keep the following in mind:

It is better to assess a select few than to try to assess all attributes that have been defined.

Consider the tradeoffs associated with a given project. For example, cost and schedule pressures may keep you from using some attributes.

Software Quality Research

Many individuals from a variety of organizations and disciplines have conducted research on software quality attributes.

This research, which started in the late 1970's and still continues, focuses on answering the following questions:

What are the most relevant software quality attributes?

How should these software quality attributes be defined?

What is the best way to actually measure software quality?

Results of research on software quality factors and their attributes are described in a Framework Guidebook published by what was then the Rome Air Development Center. Currently an international standard, ISO 9126, also provides similar guidance.

Many of these quality factors have a suffix of "ility," so quality attributes are often referred to as the software quality "ilities."

Software Quality Factor Definintions

Correctness—Does the software do what I want?

Portability—Will I be able to use it on another machine? Relative effort to transfer software to another system environment.

Efficiency—Will the software run on my hardware as well as it can?

Reliability—Does the software accurately do what I want all of the time? Extent to which the software will perform without any failures within a specified time period

Expandability—Can I add new functions to the software?

Reusability—Will I be able to reuse some of the software? Relative effort to convert a software component for use in another application.

Flexibility—Can I change it?

Survivability—If some of the system breaks will the software continue to function?

Integrity—Is the software secure?

Testability—Can I test it?

Interoperability—Will I be able to interface the software with another system? Relative effort to couple one system with another.

Usability—Can I run the software?

Maintainability—Can I fix it?

Verifiability—Can I validate what the software does?

Embedded

Embedded software is specifically designed into (embedded) or dedicated to a weapon system. As an integrated part of the overall system it performs highly specific functions. Embedded software functions as an integral part of the weapon and cannot readily support other applications without some form of significant modification.

Efficiency

Survivability

Reliability

Automated Information Systems (AIS)

Automated Information Systems (AIS) are defined as a combination of computer hardware and software, data or telecommunications. They perform functions such as collecting, processing, transmitting and displaying information.

PortabilityMaintainability

Usability

Command, Control, Communications and Intelligence (C3I)

Command, Control, Communications and Intelligence (C3I) Systems encompass command, control, communications and intelligence software. These systems communicate, assimilate, coordinate, analyze and interpret information. They also provide decision support to military commanders.

Some software quality factors are "universal" and used across several categories of software-intensive systems. Other software quality factors tend to be emphasized more in one type of system because of the technical characteristics and operational requirements.

Integrity

expandability

interoperability

The graphic outlines possible responses to the previous KRs. Quality factors for Embedded Systems:• Efficiency• Reliability• Survivability

Quality factors for AIS:• Usability• Portability• Maintainability

Quality factors for C3I Systems:• Integrity• Expandability• Interoperability

Measuring Software Quality Attributes

After selecting key software quality factors, suppliers must:

Accurately measure the attributes that make up each factor (Unlike Error Density, determining values for those other quality attributes can be highly technical.)

Make these quality determinations part of their Software Quality Assurance Program

In many cases, the specific value of a software attribute is critically dependent

on the way:

The software module is designed

The programmer used (or chose not to use) technical features of the language when coding the module

To measure software quality, an analyst:

Identifies the design techniques and programming language features that contribute to the particular quality attribute

Determines the extent to which these features have been employed in the software

Establishes an estimated "value" for the particular software quality factor (value may combine other subjective evaluations)

Maintainability Example

Software design impacts quality factors. For example, the number of comments in the code influences the program’s maintainability: the more comments, the easier to maintain software. In this example, "maintainability" can be calculated as a percentage of commented lines-of-code and assigning a rating to it.

Style Guides

A supplier-produced Programming Style Guide is typically used by project programmers. Style Guides:

Provide details of recommended and allowable programming language usage

Should be based on coding styles that contribute best to quality attributes

Those statement are true.

Software quality factors are broken into attributes that are then measured.

Project tradeoffs such as cost and schedule drivers can influence choice of quality factors.

Unlike Error Density, determining other software quality attributes can be highly technical.

Programming Style Guides can help to foster software product quality.

Software Quality Assurance Part (5)

Software Quality Assurance (SQA) refers to the activities suppliers perform within their software development process to ensure that a quality product is delivered.

The Department of Defense (DoD) does not dictate any specific SQA system be used, however, there are some key activities and processes present in all effective Software Quality Management programs.

Learning Objectives

This section presents an overview of a Software Quality Assurance (SQA) program. It includes:

Some definitions of Software Quality Assurance

The key goals and activities of SQA

Guidelines for organizing an SQA team

An example SQA Plan format

Once you have completed this lesson, you will be able to define SQA and outline its key processes.

What is Software Quality Assurance?

Software Quality Assurance (SQA) is the term used to identify that portion of a developer's quality management process that incorporates key quality activities for the software development process.

An international standard, ISO/IEC 12207, on Information Technology-Software Life-Cycle Processes, states that:

"Software Quality Assurance (SQA) is a process that provides adequate assurance that the products and processes as used on the project conform to their specified requirements and adhere to their established plans. To be unbiased, quality assurance needs to have organizational freedom and authority from persons directly responsible for developing the software product or executing the project."

Key Goals and Tasks

The key goals and tasks of SQA are to:

Assure the quality of delivered software and its associated processes

Evaluate the adequacy of development and documentation processes

Assure that corrective actions are initiated

Maintain documented evidence that the software meets contractual requirements

Provide acquirer access to any review of products and activities

Ensure that noncompliance issues that cannot be resolved within the software project are addressed by senior management

DoD Quality Assurance Goal

DoD's goal is to prevent errors, not count them.

The Acquisition, Technology and Logistics (AT&L) Knowledge Sharing System, in discussing quality for DoD systems, states that it is much less costly to prevent defects than to "inspect in" quality later after most of the development is done.

Flexible QA Process

A Program Manager (PM) cannot mandate the specific quality processes used by a Supplier. The Defense Acquisition System Policies state that:

"A key to success is to incorporate systems engineer/design quality into the product by defining the product or service quality requirements from the beginning and then providing the contractor with the maximum degree of flexibility to meet these requirements....the PM shall allow contractors the flexibility to define and use their preferred quality management process that meets program objectives."

SQA Organizational Structure

Depending on the size of the project and the supplier's internal organization, SQA activities may be performed by:

A separate staff organization that performs SQA activities for several related projects- or -

An integral part of the design and development staff dedicated to a specific project

However they are organized, software standards recommend that the SQA group should have a formal reporting channel to senior management. This reporting channel:

Must be independent of the Project Manager and software engineering group

Is not used for routine matters

Should be used to report quality concerns

Software Quality Assurance Plan (SQAP)

SQA reporting channels and other specific details of the SQA processes for a particular project are typically documented in a Supplier's internal plans.

Sometimes suppliers will use a project-specific Software Quality Assurance Plan (SQAP) to document their SQA processes.

No matter how or where a supplier's SQA processes are documented, it is essential that they are:

Documented somewhere

Reviewed by the acquirer

Approved by management

Followed by the supplier

Organization

A plan should include information about the organization responsible for quality management program, the relationship of the quality management organization to other organizational entities (such as configuration management), and the number and skill levels of personnel who perform the software quality management activities.

Furnished Items

The Quality Assurance Plan should identify Government furnished facilities, equipment, software, and services (including manufacturer, model number, and equipment configuration) to be used in quality management.

Schedule

The Quality Assurance Plan should provide a detailed schedule for quality management activities. The schedule should include activity initiation, activity completion, and personnel responsible, as well as key development milestones such as formal reviews, audits, and delivery of items on the contract data requirements list.

Implementations

The plan addresses all tasks to be performed by the supplier in conducting a quality management program. It includes the procedures to be used in the quality management program and in conducting continuous assessments of the development process. It should also describe the tools and measures that will be used to conduct the program.

Records

The Quality Assurance Plan includes the supplier's plans for preparing, maintaining and making available for Government review, the records of each quality management program activity performed.

Resources

The Quality Assurance Plan will identify any subcontractors, vendors, or other resources to be used by the supplier to fulfill the development requirements of the prime contract.

SQAP Formats

Because DoD policy is to rely on a contractor's internal quality processes, acquirers do not specify the format of a supplier's SQA plans.

There are a variety of formats a supplier's SQAP can take; IEEE standard 730 provides one commonly used format.

IEEE standard 730SQA Plan Outline

I. Purpose

II. References

III. Management

IV. Documentation

V. Standards, Practices, Convention, and Metrics

VI. Reviews and Audits

VII. Testing

VIII. Problem Reporting and Corrective Actions

IX. Tools and Techniques

X. Code, Media, and Supplies Control

XI. Records Collection, Maintenance, Retention

XII. Training

XIII. Risk Management

Key Quality Assurance Activities

The Defense Acquisition System Policies lists key quality activities that make up a thorough quality management process. They include:

Monitor, measure, analyze, control and improve processes

Reduce product variation

Measure/Verify product conformity

Establish mechanisms for feedback on product performance

Implement an effective root cause analysis and corrective action system

These quality activities apply to any type of project, not only software-intensive ones

Software Quality and Process Maturity

The existence of a viable software quality program is one of the first criteria that must be met by a supplier to begin to improve software process maturity.

Better software quality is a natural by-product of higher levels of software process maturity, as illustrated by the graph on the right.

High values of Process Maturity are recommended as a "Best Practice" by acquisition policies. These publications emphasize “contracting with software suppliers that have domain experience in developing comparable software systems; with successful past performance; and with a mature software development capability and process.”

True

It is much less costly to prevent defects than to 'inspect quality in' after development is done.

A key quality activity is to monitor, measure, analyze, control and improve processes.

Better software quality is a natural by-product of higher levels of software process maturity.

Information and Software Quality Management: Methods and Techniques Introduction

A variety of management and technical methods can be used to help improve software quality. These range from low-cost, simple methods that are in universal use to special, highly-complex techniques used rarely.

The choice of methods employed on a specific project depends on the size and complexity of the software, the amount of new code developed, system and software risks, and available budget and time.

Many such methods are available that impact software quality. This lesson surveys some of them.

Information and Software Quality Management: Methods and Techniques

Learning Objectives

This section introduces methods and techniques that can be used by government and industry to help improve software quality. Once you have completed this topic, you will be able to describe them and how they can be used to improve software quality.

Once you have completed this topic, you will be able to identify and describe these techniques.


Eight Methods and Techniques

This section introduces eight methods used in industry and government to assure quality software products.

The eight methods we will discuss are listed below:

1.Peer Reviews

2.Walkthroughs

3.Formal Inspections

4.Cleanroom

5.Formal Specifications

6.Reviews and Audits

7.IV & V

8.Developmental and Operational Testing


Eight Methods and Techniques, Cont.

Method 1: Peer Reviews

Peer reviews are normally conducted by a supplier. Upon completion of a work product (requirements, design, code, test procedure), the author, after desk checking their work, asks one or more of her/his peers (other programmers with similar domain experience) to review it for errors. Changes made to the original product as a result of a peer review may or may not be subject to the configuration management process, depending on the supplier’s software development process.

Popup Text:

Method 1: Peer Reviews

1. Peer Reviews 2.Walkthroughs and Informal Inspections 3.Formal Inspections 4.Cleanroom 5.Formal Specifications 6.Reviews and Audits 7.IV & V 8.Developmental and Operational Testing



Method 2: Walkthroughs

Sometimes called an "informal inspection" a walkthrough is a SQA method conducted by the supplier. Walkthroughs are performed by a group of people who have no formal training in inspection

techniques. There are no formal participant roles, other than the product author, who drives the process. No formal checklists are maintained, and no follow-up is required to verify that fixes have been made. Informal inspections provide no guarantee of future quality or efficiency.

Popup Text:

Method 2: Walkthroughs

1.Peer Reviews 2. Walkthroughs and Informal Inspections 3.Formal Inspections 4.Cleanroom 5.Formal Specifications 6.Reviews and Audits 7.IV & V 8.Developmental and Operational Testing



Method 3: Formal Inspections

Formal inspections are performed on small units of requirements documentation, design documentation or actual code, prior to testing. Sometimes they are called "Fagan Inspections," named for Michael Fagan who first developed this process while at IBM.

The formal inspection process uses checklists and requires follow-up to ensure that the defects and errors found are corrected. This process assures fewer future defects and leads to improved process efficiency. Metrics on the number and type of defects found, and the time spent in inspection are collected and recorded.

Popup Text:

Method 3: Formal Inspections

1.Peer Reviews 2.Walkthroughs and Informal Inspections 3. Formal Inspections 4.Cleanroom 5.Formal Specifications 6.Reviews and Audits 7.IV & V 8.Developmental and Operational Testing



Method 3: Formal Inspections, Cont.

Inspectors are formally trained and should be neutral, not involved in the product development. The product author's role in the process is only to answer questions and provide clarification to the inspection team. Each member of the inspection team has a defined role.

The READER reads the material being inspected. The MODERATOR controls the process flow of the inspection. The AUTHOR answers technical questions. The RECORDER takes action notes.

Other participants ("Inspectors") assist in these roles and participate in reviews of materials being inspected.



Method 3: Formal Inspections, Cont.

Formal inspections are conducted by the supplier. When performed properly, a formal inspection can find 60-90% of defects prior to testing. These results are dependent on adherence to a strict process, formal training of the participants, and assignment of adequate resources to support the process.

Formal Inspections are effective in detecting errors early on in the software development process when they are the easiest and cheapest to remove. A general rule of thumb is that errors which cost $1 to fix at the requirements stage will cost approximately $10 at the design stage, $100 at the coding stage, and over $1,000 to fix if they remain undetected until Operational Testing.



Method 4: Cleanroom

The cleanroom is a theory-based, team-oriented approach to the development and verification of ultra-high reliability software systems. It is designed to improve productivity through statistical quality control. It combines practical new methods of specification, design, correctness verification, and statistical testing for certifying software quality using a process based on incremental development.

The goal of cleanroom software engineering is defect prevention, rather than defect removal. Proof of correctness is used to prevent defects. The emphasis shifts from removing defects from software products, to preventing the introduction of defects into the products.

Method 4: Cleanroom

1.Peer Reviews 2.Walkthroughs and Informal Inspections 3.Formal Inspections 4. Cleanroom 5.Formal Specifications 6.Reviews and Audits 7.IV & V 8.Developmental and Operational Testing



Method 4: Cleanroom, Cont.

The objectives of the cleanroom approach are to engineer software products under statistical quality control using mathematical verification, not debugging; and statistical certification of quality, through user testing at the system level and reliability predictions (e.g., mean time between failures).

Unit testing and debugging by programmers is not part of the cleanroom methodology, because it is claimed that they compromise the correctness of the original design and introduce complex software defects from the "tunnel vision" inherent in the debugging process. Under proper conditions, the cleanroom approach has been shown to remove as many as 90% of all defects prior to initial testing.



Method 5: Formal Specifications

Formal specifications use mathematical techniques to specify requirements, design, or code. This approach allows requirements to be verified with mathematical-based techniques such as proof of correctness. Additionally, such specifications can help in modeling and simulating key parts of the system.

The cleanroom approach is based on the use of such structured specifications. Because these specifications are algebraic in nature, early error detection is improved.

Because of cost, formal specifications are used rarely. Requirements must be well understood and the system must be critical enough to justify a substantial investment in time and resources. Some DoD nuclear-critical and cryptographic systems may fall into this category.

Popup Text:

Method 5: Formal Specifications

1.Peer Reviews 2.Walkthroughs and Informal Inspections 3.Formal Inspections 4.Cleanroom 5. Formal Specifications 6.Reviews and Audits 7.IV & V 8.Developmental and Operational Testing



Method 6: Reviews and Audits

Periodic reviews and audits provide another way to gain insight into the quality of a software product. They involve both the acquirer and the supplier in the process.

Audits are performed late in the development cycle, when there is an actual product to be evaluated. As such, they are more useful as a final quality check than as an error prevention technique.

Reviews may be formal, such as a Preliminary Design Review or Critical Design Review, or informal, such as Integrated Product Team (IPT) meetings and In-Process Reviews. Reviews provide an opportunity for the supplier to present the status of a program to the acquiring organization, and sometimes to the user community as well.

Popup Text:

Method 6: Reviews and Audits

1.Peer Reviews 2.Walkthroughs and Informal Inspections 3.Formal Inspections 4.Cleanroom 5.Formal Specifications 6. Reviews and Audits 7.IV & V 8.Developmental and Operational Testing

Preliminary Design Review

A Preliminary Design Review (PDR) is conducted to ascertain whether the preliminary design is to be committed to detailed design.

This review is conducted for each Configuration Item (CI) or aggregate of CIs. Risk management actions and the results of risk mitigation activities are evaluated. A system level PDR may be conducted upon completion of all CI PDRs.

Critical Design Review

A Critical Design Review (CDR) is conducted when detailed design is complete.

For a Supplier using a waterfall approach, this is the point when the detailed design documentation is released to fabricate, integrate and assemble hardware qualification units and to code and integrate the software qualification units.

A system level CDR may be conducted after the CI CDRs have been completed to review the progress of system development.



Method 6: Reviews and Audits, Cont.

Reviews and audits are agreed upon during contract negotiation. The use of Integrated Project Teams (IPTs) may eliminate the need for many traditional formal reviews.

Other categories of common reviews and audits, performed as part of the systems engineering process, are listed below.

Functional Configuration Audit

Physical Configuration Audit

Subsystem Reviews

Software Specification Reviews

Test Readiness Reviews

Functional Reviews

Support Reviews

Training Reviews

Manufacturing Reviews

Popup Text:

Functional Configuration Audit

Functional Configuration Audits (FCA) are conducted to verify a configuration item's performance against its approved and authenticated configuration documentation. This review is conducted in accordance with established configuration management procedures.

Functional Configuration Audits answer the question: Does the product do what it was intended to do?

Physical Configuration Audit

Physical Configuration Audits (PCA) are formal evaluations of the as-built version of a configuration item against its design documentation.

Physical Configuration Audits answer the question: Does the product as produced conform to the design?

Subsystem Reviews

Subsystem reviews are held to assure that all requirements, including interface requirements, for the

subsystem have been identified, balanced across prime mission products, and met.

These reviews allow subsystem review team members to address issues and assess progress of a subsystem or configuration item (CI).

Software Specification Reviews

A type of Subsystem Review, a Software Specification Review is conducted to evaluate Software Item (SI) requirements and operational concept. This review should have sufficient detail to ensure a complete understanding among participants on the software requirements specification and, if applicable, the completed interface requirements specification for the SI.

This review will determine whether the specifications form a satisfactory basis for proceeding into preliminary software design.

Test Readiness Reviews

Test Readiness Reviews are conducted to:

Evaluate completeness of the test procedures

Assure readiness for testing

Assure the supplier is prepared for formal testing

Test procedures are evaluated for compliance with test plans and descriptions, and for adequacy to accomplish testing requirements.

Functional Reviews

Functional reviews include:

Development (systems engineering)

Support (including disposal and deployment)

Training

Test Manufacturing

These reviews assess the functional area's status in satisfying the prime mission products, surface issues, and support the development of required functional plans and procedures.

Support Reviews

Support reviews are conducted to evaluate:

The completeness of logistics support requirements integration

Interface issues

Common vs. peculiar support equipment utilization

Integrated diagnostics requirements

Level of maintenance planned

Training Reviews

Training reviews can be held to evaluate training requirements and potential embedded training solutions to ensure they enhance the user’s capabilities, improve readiness, and reduce individual and collective training costs over the life of the system.

Manufacturing Reviews

Manufacturing reviews evaluate the development of manufacturing elements and processes related to the prime mission product(s).

These reviews assess manufacturing concerns, such as the need to identify high risk/low yield manufacturing processes or materials, and the manufacturing efforts necessary to satisfy design requirements.



Method 7: Independent Verification and Validation (IV&V)

Independent Verification and Validation (IV&V) was born during the early days of the space and missile programs. Both NASA and the Department of Defense (DoD) realized that the software developed for spacecraft and missile systems as well as other "safety-critical" systems which had to perform correctly the first time required special scrutiny.

For those system which are safety-critical (where failures can cause loss of life, significant damage or security compromises) resources and time are set aside for an organization or contractor, financially and managerially independent from the software supplier, to perform several key functions for the PMO. These are to:

Ensure that proper development steps are being followed

Determine if the developed software satisfies its requirements

Perform special oversight and evaluations of high-risk areas.

Popup Text:

Method 7: Independent Verification and Validation (IV&V)

1.Peer Reviews 2.Walkthroughs and Informal Inspections 3.Formal Inspections 4.Cleanroom 5.Formal Specifications 6.Reviews and Audits 7. IV & V 8.Developmental and Operational Testing



Method 7: IV&V, Cont.

Software IV&V is an important aspect of developing quality software. To be effective, IV&V must complement and reinforce the supplier's software engineering process, configuration management, and qualification test functions.

Verification is the iterative process of determining whether the product of certain steps in the software item (SI) development process fulfills the requirements levied by previous steps.

Validation comprises evaluation, integration and test activities carried out at the system level to ensure that the system developed satisfies the operational requirements of the system specification.



Method 8: Developmental and Operational Testing

Software is normally developed using a "top-down" approach. Testing and integration of software occurs from the bottom up. This approach is called the "Software V" Model.

Testing is the process of exercising or evaluating a system or system components by manual or automated means to verify that it satisfies specified requirements or to identify differences between expected and actual results.

Testing is typically the most labor-intensive activity performed during software development. Although it is listed here for completeness, testing alone cannot produce quality software, nor can it verify its correctness. Testing can only confirm the presence (as opposed to the absence) of software defects.

Popup Text:

Method 8: Developmental and Operational Testing

1.Peer Reviews 2.Walkthroughs and Informal Inspections 3.Formal Inspections 4.Cleanroom 5.Formal Specifications 6.Reviews and Audits 7.IV & V 8. Developmental and Operational Testing


Developmental Testing

The purpose of developmental testing is to ensure that the system and the software comprising it function in accordance with various technical requirements documents. These have a variety of names depending on the System Engineering approach being used. Some of these technical requirements documents typically include:

System Requirements Specification

System and Subsystem Specifications

System Interface Requirements Specifications

Software Requirements and Interface Specifications

There are various stages in the developmental test process. Some of the “lower level” tests are performed by the supplier or software developer. Depending on the test strategy and the type of system, as it is progressively integrated together in a “bottoms-up” process, a government-industry team may perform integrated developmental testing at the subsystem and system level. In other cases government agencies will perform some of these tests.


Developmental Testing, Cont.

Developmental Testing Includes:

Software Coding and Testing

Software Integration

Software Qualification Testing

System Integration

System Qualification Testing

On the following pages we will describe each of these tests in detail.

D-Link Text:

Long Description

Developmental Testing Diagram. Five rectangles labled from bottom to top, Software Coding and Testing, Software Integration, Software Qualification Testing, System Integration, System Qualification Testing. An arrow points from each rectangle to the one above it.

Close window to continue.




Once a software configuration item (SCI) has been designed, it enters into Software Coding and Testing.

During Software Coding and Testing, the Supplier evaluates and documents the software code and test results for each Software Unit (SU) comprising the SCI considering specific criteria.

The result of the Software Coding and Testing process is executable and tested code for all the Software Units comprising a given SCI.

D-Link Text:

Long Description

Developmental Testing Diagram. Five rectangles labled from bottom to top, Software Coding and Testing, Software Integration, Software Qualification Testing, System Integration, System Qualification Testing. An arrow points from each rectangle to the one above it. The Software Coding and Testing rectangle is highlighted.


Popup Text:


Software Coding and testing includes the following activities:

Implementing each software unit (SU) comprising the SCI by:

Developing the source code for each SU

Developing the data definitions for each SU

Preparing each SU for testing by:

Developing test cases for each SU

Developing bench mark test files

Performing SU testing

Revising and retesting each SU as required

specific criteria

Testers consider the following criteria when evaluating software code and test results:

Traceability to the requirements and design of the SCI

External consistency with the requirements and design of the SCI

Internal consistency with SU requirements

Test coverage of SUs

Appropriateness of coding methods and standards

Feasibility of software integration and testing

Feasibility of operation and maintenance



Software Integration

After coding and testing each SU comprising the SCI, the next step is Software Integration. The purpose of Software Integration is to ensure the SUs comprising the SCI work together as intended.

The supplier evaluates and documents the integration plan, design, code, test, test results, and user documentation using specific criteria.

The result of the Software Integration process is an SCI product baseline.

D-Link Text:

Long Description

Developmental Testing Diagram. Five rectangles labled from bottom to top, Software Coding and Testing, Software Integration, Software Qualification Testing, System Integration, System Qualification Testing. An arrow points from each rectangle to the one above it. The Software Integration rectangle is highlighted.


Popup Text:

specific criteria

Testers consider the following criteria when evaluating the integration plan, design, code, test, test results, and user documentation:

Traceability to the system requirements

External consistency with the system requirements

Internal consistency

Test coverage of the requirements of the SCI

Appropriateness of test standards and methods

Conformance to expected results

Feasibility of software qualification testing




Software Qualification Testing

The next step is to run each Software Configuration Item (SCI) through Software Qualification Testing. This type of testing demonstrates to the acquirer that the SCI meets the software requirements that have been allocated to it as part of the Systems Engineering process.

The supplier evaluates and documents the design, code, test, test results, and user documentation considering specific criteria.

Software Qualification Testing produces a qualified product baseline for each SCI.

D-Link Text:

Long Description

Developmental Testing Diagram. Five rectangles labled from bottom to top, Software Coding and Testing, Software Integration, Software Qualification Testing, System Integration, System Qualification Testing. An arrow points from each rectangle to the one above it. The Software Qualification Testing rectangle is highlighted.


Popup Text:

specific criteria

Testers consider the following criteria when evaluating the integration plan, design, code, test, test results, and user documentation:



Feasibility of system integration and testing




System Integration

Following software qualification testing, the next step required is System Integration. The purpose of System Integration is to ensure that the software and hardware items comprising the system work together as intended.

The supplier evaluates and documents the integrated system against specific criteria.

The result of System Integration is a product baseline for the system that is ready for system qualification testing.

D-Link Text:

Long Description

Developmental Testing Diagram. Five rectangles labled from bottom to top, Software Coding and Testing, Software Integration, Software Qualification Testing, System Integration, System Qualification Testing. An arrow points from each rectangle to the one above it. The System Integration rectangle is highlighted.


Popup Text:

specific criteria

Testers consider the following criteria when evaluating the integrated system:

Test coverage of system requirements

Appropriateness of test methods and standards


Feasibility of system qualification testing



System Qualification Testing

The purpose of System Qualification Testing (SQT) is to demonstrate to the acquirer that the product baseline produced during System Integration meets the performance and operational requirements defined in the System Specification. Independent testers and system users conduct the SQT on the target hardware using live data.

The system is evaluated and documented using specific criteria.

The result of the SQT is a qualified product baseline for the system. The system is now ready to go before the Milestone C Decision Review.

D-Link Text:

Long Description

Developmental Testing Diagram. Five rectangles labled from bottom to top, Software Coding and Testing, Software Integration, Software Qualification Testing, System Integration, System Qualification Testing. An arrow points from each rectangle to the one above it. The System Qualification Testing rectangle is highlighted.


Popup Text:

specific criteria

Testers consider the following criteria when evaluating system qualification testing results:



Feasibility of system integration and testing



Operational Testing

Operational testing is conducted by an independent testing agency on behalf of the system user. Each service has its own testing arm.

During operational testing, users exercise the system in a realistic environment to try to invoke defects. Operational testing may use both live and simulated scenarios to exercise the full range of the system. Results are then analyzed for conformance to the operational requirements and Key Performance

Parameters, such as the Net-Ready KPP, that are articulated in various user requrements and capabilities documents.

Popup Text:

Operational testing

Operational testing has five general objectives:

Usability

Effectiveness

Software maturity

Reliability (including system safety)

Supportability


Knowledge Review (Alternate)

Match each term with its definition.

Terms

1. Developmental and Operational Testing

2. Cleanroom 3. Formal Inspection 4. Reviews and Audits

Definitions

A. The PM needs a way to periodically check on the status of the development project.

B. A small, critical piece of software used in a DoD multi-level security system must be error-free. A proof of correctness of the software is desired. C. End-users of the system want to verify that the systm has appropriate functionality and is usable in its intended environment.

D. Done properly, can find 60-90% of defects prior to testing.

2. 1-C, 2-A, 3-D, 4-B


Knowledge Review (Alternate)

This type of testing is performed to demonstrate to the acquirer that the software configuration item's requirements have been met in accordance with its software requirements specification (SRS).

A. Software Coding and Testing B. System Qualification Testing C. Software Qualification Testing


Summary

This topic presented a range of methods and techniques that can be used to help improve software quality. Because in some cases their use depends on the type of system under development, not all these techniques are usable on every system.

Cleanroom techniques and Formal Specifications, in addition to their impact on software quality, are software design methodologies as well. But because of their cost and complexity, they are used on a very few, high-value critical DoD systems. IV&V is used more often, typically for safety-critical systems.

Peer Reviews and Walkthroughs are commonly used by nearly all software developers. Those with more mature development processes generally use the more effective Formal Inspections, which are rigorous

and require a trained cadre. All of these techniques emphasize the detection and elimination of efforts early in the lifecycle, when they are the easiest and cheapest to remove.

Reviews and audits should always be used in system and software development. The use of Developmental and Operational Testing is universal as well.

You have reached the end of this topic. To launch the next topic, click the topic title in the Table of Contents.

chapter 6 information and software quality management

Documents

definition of software

software quality assurance

overview of software

measure quality

list software quality

effective quality management

quality management techniques

purpose of quality management