103191733-notes-software-quality-management.pdf

Software Quality ManagementSoftware Quality Management

Software Quality IntroductionSoftware Quality Introduction

AgendaAgenda

DefinitionsDefinitions– Quality and software QualityQuality and software Quality

Views about qualityViews about quality Total quality managementTotal quality management

QualityQuality

Hard to define, impossible to measure and Hard to define, impossible to measure and easy to recogniseeasy to recognise

ExampleExample– Cars in India, Mercedes, Chevy, IndicaCars in India, Mercedes, Chevy, Indica

Quality is Quality is – not absolutenot absolute– multidimensionalmultidimensional– about acceptable compromisesabout acceptable compromises– criteria of quality interact with each othercriteria of quality interact with each other

Defining qualityDefining quality

Vague definitionsVague definitions– The degree of excellenceThe degree of excellence– Zero defectZero defect– Quality is when customer comes back but not Quality is when customer comes back but not

the productthe product Trying to be preciseTrying to be precise

– ISOISO The totality of features and characteristics of a The totality of features and characteristics of a

product or service that bear on its ability to satisfy product or service that bear on its ability to satisfy stated and implied needsstated and implied needs

Combining all the definitionsCombining all the definitions

It is fitness of needsIt is fitness of needs– Needs => specification conformanceNeeds => specification conformance– fitness => intended purposefitness => intended purpose

ExampleExample– Water fall model of developmentWater fall model of development

Requirements and design form the needRequirements and design form the need Verification and validation forms the fitnessVerification and validation forms the fitness

Issues in software qualityIssues in software quality

Software has no physical existenceSoftware has no physical existence Lack of knowledge of client needs at the Lack of knowledge of client needs at the

start of the projectstart of the project Rate of change of hardware / softwareRate of change of hardware / software High expectations from customer / High expectations from customer /

adaptabilityadaptability

Manufacturing vs softwareManufacturing vs software

No releasesNo releases No bug fixesNo bug fixes Purchase new model Purchase new model

for new specificationsfor new specifications

Patch releasesPatch releases Minor and Major Minor and Major

releasesreleases New features on top of New features on top of

old modelold model

Views on qualityViews on quality

Polyhedron metaphorPolyhedron metaphor– Timeliness, cost, functionality, correctness, reliability, Timeliness, cost, functionality, correctness, reliability,

maintainabilitymaintainability Project manager’s viewProject manager’s view

– Reliability, Timeliness, maintainability, correctnessReliability, Timeliness, maintainability, correctness Business analystBusiness analyst

– Functionality, correctness, timeliness, costFunctionality, correctness, timeliness, cost ProgrammerProgrammer

– Functionality, correctness, maintainabilityFunctionality, correctness, maintainability

Views on qualityViews on quality

Quality auditorQuality auditor– ??

End userEnd user– ??

Line ManagerLine Manager– ??

Project SponsorProject Sponsor– ??

Garvin’s views of qualityGarvin’s views of quality Transcendent viewTranscendent view

– EleganceElegance Product based viewProduct based view

– Higher the quality higher the costHigher the quality higher the cost User based viewUser based view

– Fitness of purposeFitness of purpose Manufacturing viewManufacturing view

– Conformance to requirementsConformance to requirements Value based viewValue based view

– Give what the customer requires at the price they wantGive what the customer requires at the price they want

Total Quality ManagementTotal Quality Management

Coined in 1985 by Naval Air Systems Coined in 1985 by Naval Air Systems CommandCommand

Links long term success with quality and Links long term success with quality and customer satisfactioncustomer satisfaction

Successful TQM examplesSuccessful TQM examples– HP – Total Quality ControlHP – Total Quality Control– Motorola – 6-Motorola – 6-– IBM – Market driven qualityIBM – Market driven quality

σ

Software TQM Contd.,Software TQM Contd., HPHP

– Focus on management commitment, leadership, Focus on management commitment, leadership, customer focus, total participation and systematic customer focus, total participation and systematic analysis leading to customer satisfactionanalysis leading to customer satisfaction

MotorolaMotorola– Stringent quality levels to obtain customer satisfaction Stringent quality levels to obtain customer satisfaction

through cycle time reduction and participative through cycle time reduction and participative managementmanagement

IBMIBM– Defect elimination, Cycle time reduction, customer and Defect elimination, Cycle time reduction, customer and

business partner satisfaction and adherence to business partner satisfaction and adherence to Baldridge assessment disciplineBaldridge assessment discipline

TQM systemTQM system

Customer focusCustomer focus– Total customer satisfactionTotal customer satisfaction

ProcessProcess– Continuous process improvementContinuous process improvement

Human side of qualityHuman side of quality– Company wide quality cultureCompany wide quality culture

Measurement and AnalysisMeasurement and Analysis– Continuous improvement with goal oriented Continuous improvement with goal oriented

measurementmeasurement

End of Session 1End of Session 1

Hierarchical models for quality – Hierarchical models for quality – Software Development processSoftware Development process

Session 2Session 2

AgendaAgenda

Definition of Hierarchial modelsDefinition of Hierarchial models Examples of Hierarchial modelsExamples of Hierarchial models Software Development Process modelsSoftware Development Process models Process maturity framework and quality Process maturity framework and quality

standardsstandards

Hierarchial ModelHierarchial Model

Need for a modelNeed for a model– To evaluate quality under different situationsTo evaluate quality under different situations

Model of BITS for assessmentModel of BITS for assessment– Subject – 4 tests – EC1, EC2, EC3, and EC4Subject – 4 tests – EC1, EC2, EC3, and EC4– EC2, EC4 is copy EC2, EC4 is copy open book open book– Marks for each testMarks for each test– GradingGrading– CGPACGPA

Structure of Hierarchial ModelStructure of Hierarchial Model

Quality Factor

Quality Criteria(Reliability)

Quality Criteria(Maintainability)

Quality Criteria(Usability)

Quality Metrics Quality MetricsQuality Metrics

(Accuracy, Consistency, Error Tolerance, Simplicity)

GE Model or McCall’s ModelGE Model or McCall’s Model

Product

Operation

Product

Transition

Product

Revision

Maintainability

Flexibility

Testability

Portability

Reusability

Interoperability

Correctness, Reliability, Efficiency, Integrity, Usability

Boehm ModelBoehm Model

General UtilityGeneral Utility– As is utilityAs is utility

EfficiencyEfficiency Human EngineeringHuman Engineering

– MaintainabilityMaintainability TestabilityTestability UnderstandabilityUnderstandability ModifiabilityModifiability

– PortabilityPortability

Base Factors of Boehm’s modelBase Factors of Boehm’s model Device independenceDevice independence Self ContainednessSelf Containedness AccuracyAccuracy CompletenessCompleteness Robustness / IntegrityRobustness / Integrity ConsistencyConsistency AccountabilityAccountability Device efficiencyDevice efficiency AccessibilityAccessibility CommunicativenessCommunicativeness Self DescriptivenessSelf Descriptiveness StructurednessStructuredness ConcisenessConciseness LegibilityLegibility AugmentabilityAugmentability

Common Factors of the modelCommon Factors of the model

Quality criteria is based on user viewQuality criteria is based on user view Models focus on parts that designers can Models focus on parts that designers can

analyseanalyse These models cannot be tested or validatedThese models cannot be tested or validated

– Then what’s the point in having this?Then what’s the point in having this? Measurement of quality is a weighted Measurement of quality is a weighted

summation of the characteristics – result is summation of the characteristics – result is of limited value but usefulof limited value but useful

Interrelation of quality criteriaInterrelation of quality criteria Integrity vs efficiency (I)Integrity vs efficiency (I) Maintainability vs flexibility (D)Maintainability vs flexibility (D) Usability vs efficiency ?Usability vs efficiency ? Portability vs efficiency ?Portability vs efficiency ? Portability vs reusability ?Portability vs reusability ? Correctness vs efficiency ?Correctness vs efficiency ?

These relationships may not be commutative / These relationships may not be commutative / some may defy classification !!some may defy classification !!

Gillies modelGillies model

Quality = correctnessQuality = correctness– Technical factors (conformance to spec)Technical factors (conformance to spec)

ReliabilityReliability MaintainabilityMaintainability IntegrityIntegrity EfficiencyEfficiency UsabilityUsability AdaptabilityAdaptability InteroperabilityInteroperability PortabilityPortability

Gillies modelGillies model

– Business Factors (fitness of purpose)Business Factors (fitness of purpose) Added valueAdded value CostCost Timeliness of deliveryTimeliness of delivery User SatisfactionUser Satisfaction Ease of TransitionEase of Transition

Software Development Software Development Process ModelsProcess Models

AgendaAgenda

WaterfallWaterfall PrototypingPrototyping SpiralSpiral Iterative developmentIterative development OODOOD Clean room methodologyClean room methodology Defect Prevention ProcessDefect Prevention Process

Water Fall Development ModelWater Fall Development ModelRequirements Gathering and Analysis

Architectural Design

HLD/Ins LLD/Ins Coding/Ins Unit Testing

Integration

Component and system test Alpha and

Beta Testing

Release

Water fallWater fall

Document driven approachDocument driven approach Risk takes a back seatRisk takes a back seat

PrototypingPrototyping

Useful when systems Useful when systems requirements change requirements change with timewith time

All external interfaces All external interfaces made availablemade available

Two modificationsTwo modifications– Throw away prototypingThrow away prototyping– Evolutionary prototypingEvolutionary prototyping

Requirements Gathering

and analysis

Quick Design

Building prototype, Customer evaluation,

refining design and prototype

Spiral ModelSpiral Model

Planning next phases

Review

Cumulative cost

Spiral ModelSpiral Model AdvantagesAdvantages

– Risk analysis and risk driven approachRisk analysis and risk driven approach– Prototyping is importantPrototyping is important– Uses simulations, models and benchmarksUses simulations, models and benchmarks– Reviews are doneReviews are done– Prepares for growth, evolution and changesPrepares for growth, evolution and changes

DisadvantagesDisadvantages– Matching to contract softwareMatching to contract software– Relying on risk management expertiseRelying on risk management expertise– Need for further elaboration of spiral stepsNeed for further elaboration of spiral steps

Iterative developmentIterative development

Iterative DevelopmentIterative Development

a.k.a iterative enhancement approacha.k.a iterative enhancement approach Prototype + Water fall + Spiral = IDPPrototype + Water fall + Spiral = IDP IBM OS/2 used this approachIBM OS/2 used this approach

OO DevelopmentOO Development

Model the essential systemModel the essential system– Use caseUse case

Derive candidate essential classesDerive candidate essential classes– Found in external entities, data stores, input flows and Found in external entities, data stores, input flows and

process specs.process specs. Constrain the essential modelConstrain the essential model

– Bring the target implementation environment into pictureBring the target implementation environment into picture Derive additional classesDerive additional classes

– Add classes specific to implementationAdd classes specific to implementation

OO DevelopmentOO Development

Synthesize classesSynthesize classes– Refinement of classes using OOPRefinement of classes using OOP

Define interfacesDefine interfaces Complete the designComplete the design

– Interaction, states etcInteraction, states etc Implement the solutionImplement the solution

– Classes coded and unit testedClasses coded and unit tested

Cleanroom MethodologyCleanroom Methodology

Defect Prevention ProcessDefect Prevention Process

Process Maturity Framework Process Maturity Framework and Quality Standardsand Quality Standards

AgendaAgenda

SEI AssessmentSEI Assessment SPR AssessmentSPR Assessment Malcolm Baldridge AssessmentMalcolm Baldridge Assessment ISOISO

SEI-CMMSEI-CMM

InitialInitial– Chaotic, unpredictable cost, schedule and Chaotic, unpredictable cost, schedule and

quality performancequality performance RepeatableRepeatable

– Requirements managementRequirements management– Software project planning and managementSoftware project planning and management– Software subcontract managementSoftware subcontract management– SQASQA– SCMSCM

SEI-CMMSEI-CMM

DefinedDefined– Organisational process improvementOrganisational process improvement– Organisation process definitionOrganisation process definition– Training programTraining program– Integrated software managementIntegrated software management– Software product engineeringSoftware product engineering– Intergroup coordinationIntergroup coordination– Peer reviewsPeer reviews

SEI-CMMSEI-CMM

ManagedManaged– Process measurement and analysisProcess measurement and analysis– Quality managementQuality management

OptimizingOptimizing– Defect PreventionDefect Prevention– Technology innovationTechnology innovation– Process change managementProcess change management

CMMICMMI InitialInitial

– Processes are adhoc and chaoticProcesses are adhoc and chaotic ManagedManaged

– Requirements managementRequirements management– Project planningProject planning– Project monitoring and controlProject monitoring and control– Supplier agreement managementSupplier agreement management– Measurement and analysisMeasurement and analysis– Process and product qaProcess and product qa– SCMSCM

CMMICMMI DefinedDefined

– Requirements developmentRequirements development– Technical solutionTechnical solution– Product integrationProduct integration– VerificationVerification– ValidationValidation– Organizational process focusOrganizational process focus– Organization process definitionOrganization process definition– Integrated product managementIntegrated product management– Risk managementRisk management– Decision analysis and resolutionDecision analysis and resolution– Organizational environment for integrationOrganizational environment for integration– Integrated teamingIntegrated teaming

CMMICMMI Qualitatively managedQualitatively managed

– Organization process performanceOrganization process performance– Quantitative project managementQuantitative project management

OptimizingOptimizing– Organizational innovation and deploymentOrganizational innovation and deployment– Causal analysis and resolutionCausal analysis and resolution

Capability level of individual process areasCapability level of individual process areas– L0 – In completeL0 – In complete– L1 – PerformedL1 – Performed– L2 – ManagedL2 – Managed– L3 – DefinedL3 – Defined– L4 – Quantitatively ManagedL4 – Quantitatively Managed– L5 - OptimizingL5 - Optimizing

Software Productivity Research Software Productivity Research AssessmentAssessment

5 point scale5 point scale– Excellent, Good, Average, Below Average, PoorExcellent, Good, Average, Below Average, Poor

Questions onQuestions on– Quality and productivity measurementsQuality and productivity measurements– Pretest defect removal experience among programmersPretest defect removal experience among programmers– Test defect removal experience among programmersTest defect removal experience among programmers– Project quality and reliability targetsProject quality and reliability targets– Pretest defect removal at project levelPretest defect removal at project level– Project testing defect removalProject testing defect removal– Post test defect removalPost test defect removal

Findings are grouped intoFindings are grouped into– Projects and software products being assessedProjects and software products being assessed– Software technologies usedSoftware technologies used– Software processes usedSoftware processes used– Ergonomics and work environment for staffErgonomics and work environment for staff– Personnel training for staff and managementPersonnel training for staff and management

Malcolm Baldridge AssessmentMalcolm Baldridge Assessment Most prestigious quality award in USMost prestigious quality award in US 7 Categories and 28 examination items7 Categories and 28 examination items CategoriesCategories

– LeadershipLeadership– Information and analysisInformation and analysis– Strategic quality planningStrategic quality planning– HR utilizationHR utilization– Quality Assurance of product and servicesQuality Assurance of product and services– Quality resultsQuality results– Customer SatisfactionCustomer Satisfaction

Scoring is based onScoring is based on– Approach, Deployment and resultsApproach, Deployment and results

Malcolm Baldridge AssessmentMalcolm Baldridge Assessment Purpose of assessmentPurpose of assessment

– Elevate quality standardsElevate quality standards– Faciliate sharing and communication within organizationFaciliate sharing and communication within organization– Serve as working tool for planning, training, assessment Serve as working tool for planning, training, assessment

and other usesand other uses– Provide basis for making awardProvide basis for making award– Provide feedback to applicantsProvide feedback to applicants

1000 points in award criteria1000 points in award criteria Each exam item has a % score >70% will be Each exam item has a % score >70% will be

considered for awardconsidered for award Feedback is very important not the awardFeedback is very important not the award

ISOISO

Evaluated on 20 elementsEvaluated on 20 elements Abroad about 60 – 70% organizations are rejectedAbroad about 60 – 70% organizations are rejected In India it is available at a cost!!!In India it is available at a cost!!! Documentation, Documentation , Documentation - Documentation, Documentation , Documentation -

all well trackedall well tracked Looks at both Looks at both

– Product metricsProduct metrics– Process metricsProcess metrics

NotesNotes

ISO and MB models are complementaryISO and MB models are complementary SEI and SPR looks at maturity of SEI and SPR looks at maturity of

organizationorganization ISO and MB assess the organization ISO and MB assess the organization

irrespective of the industryirrespective of the industry We will now start focusing on measurement We will now start focusing on measurement

and metrics which is used in all of the above and metrics which is used in all of the above !!!! from next class!!!! from next class

BITS PilaniPilani Campus

Software Quality Management

Lecture 1-2

BITS Pilani, Pilani Campus

Agenda

• Definitions– Quality and software Quality

• Views about quality• Total quality management


Quality

• Hard to define, impossible to measure and easy to recognise

• Example– Cars in India, Mercedes, Chevy, Indica

• Quality is – not absolute– multidimensional– about acceptable compromises– criteria of quality interact with each other


Defining quality

• Vague definitions– The degree of excellence– Zero defect– Quality is when customer comes back but not

the product• Trying to be precise

– ISO• The totality of features and characteristics of a

product or service that bear on its ability to satisfy stated and implied needs


Combining all the definitions

• It is fitness of needs– Needs => specification conformance– fitness => intended purpose

• Example– Water fall model of development

• Requirements and design form the need• Verification and validation forms the fitness


Issues in software quality

• Software has no physical existence• Lack of knowledge of client needs at the

start of the project• Rate of change of hardware / software• High expectations from customer /

adaptability


Manufacturing vs software

• No releases• No bug fixes• Purchase new model

for new specifications

• Patch releases• Minor and Major

releases• New features on top

of old model


Views on quality

• Polyhedron metaphor– Timeliness, cost, functionality, correctness, reliability,

maintainability• Project manager’s view

– Reliability, Timeliness, maintainability, correctness• Business analyst

– Functionality, correctness, timeliness, cost• Programmer

– Functionality, correctness, maintainability


Views on quality

• Quality auditor– ?

• End user– ?

• Line Manager– ?

• Project Sponsor– ?


Garvin’s views of quality• Transcendent view

– Elegance• Product based view

– Higher the quality higher the cost• User based view

– Fitness of purpose• Manufacturing view

– Conformance to requirements• Value based view

– Give what the customer requires at the price they want


Total Quality Management

• Coined in 1985 by Naval Air Systems Command

• Links long term success with quality and customer satisfaction

• Successful TQM examples– HP – Total Quality Control– Motorola – 6-– IBM – Market driven quality

σ


Software TQM Contd.,• HP

– Focus on management commitment, leadership, customer focus, total participation and systematic analysis leading to customer satisfaction

• Motorola– Stringent quality levels to obtain customer satisfaction

through cycle time reduction and participative management

• IBM– Defect elimination, Cycle time reduction, customer

and business partner satisfaction and adherence to Baldridge assessment discipline


TQM system

• Customer focus– Total customer satisfaction

• Process– Continuous process improvement

• Human side of quality– Company wide quality culture

• Measurement and Analysis– Continuous improvement with goal oriented

measurement


End of Session 1


Hierarchical models for quality – Software Development process

Session 2


Agenda

• Definition of Hierarchial models• Examples of Hierarchial models• Software Development Process models• Process maturity framework and quality

standards


Hierarchial Model

• Need for a model– To evaluate quality under different situations

• Model of BITS for assessment– Subject – 4 tests – EC1, EC2, EC3, and EC4– EC2, EC4 is copy open book– Marks for each test– Grading– CGPA


Structure of Hierarchial Model

Quality Factor

Quality Criteria(Reliability)

Quality Criteria(Maintainability)

Quality Criteria(Usability)

Quality Metrics Quality MetricsQuality Metrics

(Accuracy, Consistency, Error Tolerance, Simplicity)


GE Model or McCall’s Model

Product

Operation

Product

Transition

Product

Revision

Maintainability

Flexibility

Testability

Portability

Reusability

Interoperability

Correctness, Reliability, Efficiency, Integrity, Usability


Boehm Model

• General Utility– As is utility

• Efficiency• Human Engineering

– Maintainability• Testability• Understandability• Modifiability

– Portability


Base Factors of Boehm’s model• Device independence• Self Containedness• Accuracy• Completeness• Robustness / Integrity• Consistency• Accountability• Device efficiency• Accessibility• Communicativeness• Self Descriptiveness• Structuredness• Conciseness• Legibility• Augmentability


Common Factors of the model

• Quality criteria is based on user view• Models focus on parts that designers can

analyse• These models cannot be tested or

validated– Then what’s the point in having this?

• Measurement of quality is a weighted summation of the characteristics – result is of limited value but useful


Interrelation of quality criteria

• Integrity vs efficiency (I)• Maintainability vs flexibility (D)• Usability vs efficiency ?• Portability vs efficiency ?• Portability vs reusability ?• Correctness vs efficiency ?

• These relationships may not be commutative / some may defy classification !!


Gillies model

• Quality = correctness– Technical factors (conformance to spec)

• Reliability• Maintainability• Integrity• Efficiency• Usability• Adaptability• Interoperability• Portability


Gillies model

– Business Factors (fitness of purpose)• Added value• Cost• Timeliness of delivery• User Satisfaction• Ease of Transition


Software Development Process Models


Agenda

• Waterfall• Prototyping• Spiral• Iterative development• OOD• Clean room methodology• Defect Prevention Process


Water Fall Development ModelRequirements Gathering and Analysis

Architectural Design

HLD/Ins LLD/Ins Coding/Ins Unit Testing

Integration

Component and system test Alpha and

Beta Testing

Release


Water fall

• Document driven approach• Risk takes a back seat


Prototyping

• Useful when systems requirements change with time

• All external interfaces made available

• Two modifications– Throw away

prototyping– Evolutionary

prototyping

Requirements Gathering

and analysis

Quick Design

Building prototype, Customer evaluation,

refining design and prototype


Spiral Model

Planning next phases

Review

Cumulative cost


Spiral Model

• Advantages– Risk analysis and risk driven approach– Prototyping is important– Uses simulations, models and benchmarks– Reviews are done– Prepares for growth, evolution and changes

• Disadvantages– Matching to contract software– Relying on risk management expertise– Need for further elaboration of spiral steps


Iterative development


Iterative Development

• a.k.a iterative enhancement approach• Prototype + Water fall + Spiral = IDP• IBM OS/2 used this approach


OO Development

• Model the essential system– Use case

• Derive candidate essential classes– Found in external entities, data stores, input flows and

process specs.• Constrain the essential model

– Bring the target implementation environment into picture

• Derive additional classes– Add classes specific to implementation


OO Development

• Synthesize classes– Refinement of classes using OOP

• Define interfaces• Complete the design

– Interaction, states etc• Implement the solution

– Classes coded and unit tested


Cleanroom Methodology


Defect Prevention Process


Process Maturity Framework and Quality Standards


Agenda

• SEI Assessment• SPR Assessment• Malcolm Baldridge Assessment• ISO


SEI-CMM

• Initial– Chaotic, unpredictable cost, schedule and

quality performance• Repeatable

– Requirements management– Software project planning and management– Software subcontract management– SQA– SCM


SEI-CMM

• Defined– Organisational process improvement– Organisation process definition– Training program– Integrated software management– Software product engineering– Intergroup coordination– Peer reviews


SEI-CMM

• Managed– Process measurement and analysis– Quality management

• Optimizing– Defect Prevention– Technology innovation– Process change management


CMMI

• Initial– Processes are adhoc and chaotic

• Managed– Requirements management– Project planning– Project monitoring and control– Supplier agreement management– Measurement and analysis– Process and product qa– SCM


CMMI• Defined

– Requirements development– Technical solution– Product integration– Verification– Validation– Organizational process focus– Organization process definition– Integrated product management– Risk management– Decision analysis and resolution– Organizational environment for integration– Integrated teaming


CMMI• Qualitatively managed

– Organization process performance– Quantitative project management

• Optimizing– Organizational innovation and deployment– Causal analysis and resolution

• Capability level of individual process areas– L0 – In complete– L1 – Performed– L2 – Managed– L3 – Defined– L4 – Quantitatively Managed– L5 - Optimizing


Software Productivity Research Assessment• 5 point scale

– Excellent, Good, Average, Below Average, Poor• Questions on

– Quality and productivity measurements– Pretest defect removal experience among programmers– Test defect removal experience among programmers– Project quality and reliability targets– Pretest defect removal at project level– Project testing defect removal– Post test defect removal

• Findings are grouped into– Projects and software products being assessed– Software technologies used– Software processes used– Ergonomics and work environment for staff– Personnel training for staff and management


Malcolm Baldridge Assessment• Most prestigious quality award in US• 7 Categories and 28 examination items• Categories

– Leadership– Information and analysis– Strategic quality planning– HR utilization– Quality Assurance of product and services– Quality results– Customer Satisfaction

• Scoring is based on– Approach, Deployment and results


Malcolm Baldridge Assessment• Purpose of assessment

– Elevate quality standards– Faciliate sharing and communication within

organization– Serve as working tool for planning, training,

assessment and other uses– Provide basis for making award– Provide feedback to applicants

• 1000 points in award criteria• Each exam item has a % score >70% will be

considered for award• Feedback is very important not the award


ISO

• Evaluated on 20 elements• Abroad about 60 – 70% organizations are

rejected• In India it is available at a cost!!!• Documentation, Documentation , Documentation

- all well tracked• Looks at both

– Product metrics– Process metrics


Notes

• ISO and MB models are complementary• SEI and SPR looks at maturity of

organization• ISO and MB assess the organization

irrespective of the industry• We will now start focusing on

measurement and metrics which is used in all of the above !!!! from next class


Measurement and Metrics

Lecture 3-4

BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956



Agenda

• Metrics and Measures• Pitfalls• Some metrics for product, project and

process



• Why measure?• Metrics

– Indicator of one or more quality criteria and we seek to measure

• Metric Characteristic– Must be clearly linked to the quality criteria– be sensitive to the different degrees of criteria– provide objective determination of criteria

which can be mapped to a suitable scale


Example

• What are the ways to measure strength?• Metrics used can be

– Kg you can pull ?– Time you can run ?– Temperature you can bear ?

• How do we measure temperature?– Linear expansion – so ?


Metrics• Type

– Predictive• Used for prediction

– Descriptive• Talks about the system at that instant

• Good metrics– Objective– Reliable– Valid– Standard– Comparable– Economical– Useful


Common Metrics and Quality

• Readability, Complexity, Modularity, Testability of a document gives a measure for usability, maintainability

• Error prediction, detection gives a measure of correctness

• MTTF, Complexity gives a measure on reliability


Limitation of Common Metrics

• Relationships, cannot be validated• Generally not objective• Quality is a relative quantity – not absolute• Depend on small set of measurable

parameters• Complete set of quality criteria not

measured• Metrics measure more than one criteria


Simple Methods for calculating metrics• Simple scoring

– Scoring for each criteria – take average• Weighted scoring

– Same as above but add weights and divide by weight.

• Phased weighted score– Take different phases into account

• Polarity Profiling– Desirable vs achieved


Complex Metrics

• Metrics are got from measures• Example

– Prove, the more rigorously the front end of the development process is executed, the better the quality at the back end.

– Steps• What is front end of development process?• What is backend of development process?


Complex Metrics• Front end (rigorous)

– Inspection• Number of lines inspected • Final scoring of inspection on Liekert scale (very effective to poor inspection)

– Testing• Coverage• Defect rate in terms of KLOC

• Form Hypothesis– Higher the metrics during inspection, lower it will be during testing– Good final scoring leads to less defect rate– Good inspection and more coverage vs less defect

• Get Test data• Correlate with the hypothesis or do statistical analysis• If hypothesis confirms the data we are OK• If it refutes the data search for missing links


Two ways

• Theory• Proposition• Hypothesis• Data Analysis

• Concept• Definition• Operational Definition

– Software defect rate• Measurements in the

real world


Levels of measurement

• Nominal scale– Jointly exhaustive and mutually exclusive

• Example– Types of software process

• Ordinal scale– a > b > c

• Interval or Ratio scales– A/B or A:B


Basic Measures

• Ratio– a/b what if a > b

• Proportion– a + b + c = N => a/N + b/N + c/N = 1

• Percentage– Ratio / proportion expressed in terms of 100– Usually have a problem as % depends on base

• Rate– Dynamic measure – Associated with a changing quantity


Six Sigma

• Stringent quality measure• 3.4 defective parts per million• Area under a curve for a normal

distribution


Confidence Interval

σ 68.26894921371%

2σ 95.44997361036%

3σ 99.73002039367%

4σ 99.99366575163%

5σ 99.99994266969%

6σ 99.99999980268%

7σ 99.99999999974%


Reliable measurements

• Depends on situation• Usually

– Index of Variation = Standard Deviation / Mean

• Reliability vs Validity– 100 % of the time wrong result– 10% right but others < 50%– 100% right all the time


Measurement Errors

• Systematic measurement error– Related with validity– Can be overcome with averages

• Random measurement error– Related with reliability– Methods

• Test/retest• Alternative form• Split half method

• Finally Use Correlation but be carful while using it


Correct Measure

A B

1. Causal,

2. Correlated,

3. No spurious relationship


Metrics

• Product Metrics• Process Metrics • Project Metrics


Product Metrics

• Mean Time To Failure– Error = Human mistake and incorrect s/w– Fault = Due to error and unit fails – Failure = Cannot function as per requirements

• Defect Density– Defect = Anamoly in the product– Calculated in terms of shipped source instructions

and changed source instructions• Customer problems• Customer satisfaction


Function Points

• External inputs (3-6), outputs (4-7)• Internal(7-15), external files(5-10)• Queries (3-6)• Calculate weighted sum = FC• Take the 14 factors on 5 pt scale• Do Value adjustment

– VAF = 0.65 + 0.01 * sum (ci)• FP = FC * VAF


SEI - CMM

• Defect rate per function point• L1 = 0.75• L2 = 0.44• L3 = 0.27• L4 = 0.14• L5 = 0.05


Process Quality metrics

• Defect Density during machine testing• Defect arrival pattern• Phase based defect removal pattern• Defect removal effectiveness

– DRE = Defects removed during dev phase / Defect latent in the product expressed as percentage


Software Maintenance

• Fix backlog and backlog management index– BMI = Problems closed / problems arrived per month

in %• Fix response time and responsiveness

– Mean time for all problems closed• Precent delinquent fixes

– No of fixes that exceeded the response time criteria by severity level / Number of fixes delivered during that time as percentage

• Fix quality– Number of defective fixes / number of fixes as a

percentage


Collecting data

• Establish goal for data collection• Develop list of questions• Establish data categories• Design and test data collection forms• Collecte and Validate data• Analyse data


Case Study

• Read the metrics program of – Motorola – HP– IBM in your book


Summary

Thanks


Quality Management Systems

Lecture 5-6


History

• 3 Gurus– Deming

• Conformability and dependability• background in statistics• No posters

– Juran• whose idea of quality was fitness of purpose• No posters

– Crosby• Zero defects – book quality is free• Have poster campaign

• Edward Deming– Worked with Juran by accident


Deming’s 14 point for management

• Constancy of purpose• A new philosophy

– Single source of supply than tenders• Cease dependence on inspection• End lowest tender contracts• Improve every process• Institute training on the job• Institute leadership


Deming’s 14 point for management

• Drive out fear• Break down barriers• Eliminate exhortations• Eliminate targets• Permit pride of workmanship• Encourage education• Create top management structures


Juran’s 10 steps for quality

• Build awareness of need and opportunity for improvement

• Set goals for improvement• Organize to reach the goals• Provide training• Carry out projects to solve problems


Juran’s 10 steps for quality

• Report progress• Give recognition• Communicate results• Keep score• Maintain momemtum by making annual

improvement part of the regular process of the company


Crosby

• Aims at defect prevention• Suggest quality vaccine

– Determination, Education, Implementation• 4 absolutes of quality

– Definition : Conformance to requirements– System : Prevention– Performance Standard : zero defects– Measurement : the price of non-conformance


14 steps of Crosby

• Make it clear that management is committed to quality

• Form QMT with each department represented

• Determine where the current and potential problems lie


14 steps of Crosby

• Evaluate cost of quality and explain its use as a tool

• Raise the quality of awareness and concern of all employees

• Take actions to correct problems identified• Establish a committee for the zero defects

programme


14 steps of Crosby

• Train supervisors to actively carry out their role in quality improvement

• Hold a “zero defects day” for all employees to highlight the changes

• Encourage individuals to establish improvement goals


14 steps of Crosby

• Encourage communication with management about obstacles to improvement

• Recognize and appreciate participants• Establish quality councils to aid

communication• Do it all over again to show it never ends.


QMS

• The organizational structure, responsibilities, procedures, processes and resources for implementing quality -ISO Definition

• Best – Gives a frame work

• Worst– Bureaucratic framework

• Must include– QA and Quality improvement


TQM

• A method for ridding people’s lives of wasted effort by involving everybody in the process of improving the effectiveness of work so that results are achieved in less time

• Kanji’s definition– Quality is to satisfy customer’s requirements

continually– Total quality is to achieve quality at low cost– Total quality management is to obtain total quality by

involving every one’s daily commitment


QIP

• Used to refine QMS• Elements of QMS

– Organizational structure– Responsibility– Procedure– Processes– Resources

• Establishment of quality culture part of QIP


Monitoring quality

• Use the 7 tools described earlier


Human factors in QMS

• Staff acceptance of procedures, tools and techniques

• Needs – management committement– Team approach– organizational quality climate

• Quality circle or quality improvement team


External standards

• Most people are cynical • Quality needs to be forced• Quality is a personal factor and hence

conflicts arise• Groupisms

– More revenue vs. quality


Parts of QMS

• Development procedures– Methodology and use of tools, distribution of expertise

to maintain quality• Quality control

– Meetings, planning, user sign off, change control etc.• Quality improvement

– QIT, Quality circles• Quality Assurance

– Monitoring , assessment


Benefits

• Cost• Timeliness• Reliability• Functionality• Maintainability


Kaizen

• Emphasis is on improvement and inspection

• Quality must fully meet consumer needs• Customer centered view• Small cooperative groups and emphasis

on worker’s suggestions


Questions?


ISO 9000 Series of Quality Management Standards


Agenda

• Purpose of standard• Details of standard• Compliance • Audit• TickIT


Purpose

• A model of best practice that needs to be followed / compared against

• Not a better way of designing things• Conformance to standard way• Implementation meets the standard over a

continuous time


Types of accreditation

• First Party• Second Party

– Military suppliers• Third Party


Advantages of accreditation

• Self confidence• External credibility• Part of tenders• Inclusion in buyer’s guide compiled by

accreditation bodies and circulated to potential customers


Standards

• ISO 9000 – A guide to select appropriate standard

• 9001– Design, Development, Production, Installation and

Service• 9002

– Production and Installation• 9003

– Final inspection and testing• 9004

– Guidance for QMS to set up 1/2/3 standards


Fundamental principles of ISO 9001 Model• Do it right the first time• It should fit the purpose

• 20 sub clauses under clause 4 are used


Management Responsibility

• Provision of management representative• Responsible for quality and accountable to

senior management• Basic principles for establishing a quality

system are explained


Quality System

• Organizational Quality System• Documented• Quality plan and manual must be prepared


Contract Review

• Each customer order is a contract• Order entry procedures• Goals

– Customer requirements are given in writing– Highlight differences so that they are agreed– Ensure that requirements can be met


Design Control• Control and verify design activities

– Planning for R & D– Assignment to qualified staff– Identify interfaces between relevant groups– Preparation of design brief– Production of technical data– Verification that output of the design phase

meets the input requirements– Identification of all changes and modification

and documenting the same


Document Control

• Level 1 documents– Planning and policy documents

• Level 2 documents– Procedures

• Level 3 documents– Detailed instructions


Purchasing

• All purchased products conform to the standards and requirements of the organization

• If the supplying organization has conformance to ISO, then the procedures are simplified


Purchaser supplied product

• To trace all purchaser supplied product at every stage of the process as well as storage


Product Identification and traceability• Procedure to trace products between

inputs and outputs• Enables quality problems to be traced to

the root causes


Process Control

• Documented processes• Procedures for calibration• Documented instructions for staff


Inspection and Testing

• Done at three stages– Incoming– In process– Outgoing

• Should not reveal many issues / bugs


Inspection, measuring and testing equipment• Calibration of equipments• Maintenance of equipments at regular

intervals• Documentation of the above


Inspection and Testing Status

• Awaiting inspection or test• Finished inspection

– Passed– Failed


Control of nonconforming product

• Nonconformance– All products or services falling outside

tolerance limits agreed in advance with customer

• Identified, documented, physically separated

• Procedures needed for handling them• Selling of nonconforming products

allowed!!


Corrective Action

• Records kept so that future audits can investigate its effectiveness

• Should have systematic procedures and have duties of all parties


Handling storage packaging and delivery• All activities which are contractual

obligation to the customer• Subcontractors employed for transporation

subject to this


Quality records

• Records must be fit for intended purpose• No specific format / structure


Internal Quality Audits

• Documented• Internal auditors• Records for external auditors if needed• Corrective actions documented


Training

• Implemented and documented• Written procedures to

– Establish training needs– Carry out training activities– Record training requirements– Record completion requirements

• Also informal knowledge sharing


Service

• Servicing procedures are documented• Sufficient resources available• Set up good interfaces with the customer

for service


Statistical Techniques

• Use where appropriated• Used in process control


Seeking accredition

• Implement quality system with outside help

• Gain internal acceptance• Pre inspection quality audit by third party• Accredition body audit• Surprise inspection twice an year


ISO 9000-3 Notes for guidance of ISO9001 to s/w devSection 4 – Quality system frame work

Management responsibility, Quality system, internal audits, and corrective action

Section 5 – Quality system – lifecycle activitiesSection 6 – Quality system – supporting activities

Configuration management, measurement systems, practices and conventions, rules, tools and techniques


TickIT

• Boost awareness of certification and quality

• Survey results – read thebook• Third party accredition


Questions


Quality Tools

Lecture 5


S. Raman


Agenda

• Seven tools and their usage


Statistical Tools

• Proposed by Ishikawa• Seven basic tools• Useful for project managers and leaders• does not provide developers the way to

improve quality


Seven basic tools

Some books talks about check list instead of control charts


Histogram


Creating histogram


Use of histogram


Pareto Charts


Pareto Charts - Construction


Usage


Example


Cause Effect Diagram


Construction Cause Effect


Usage – Cause effect


Example


Scatter Diagrams


Construction


Usage


Example


Flow Charts


Creation


Use


Run charts


Creating Run charts


Example


Control Charts


Development of Control Charts


Usage


Summary


Models and Standards

for process improvements


Why not ISO?

More documentation Patch work for software Mostly for blue collar jobs More focus on processes Less focus on results or analysis


CMM Initial

Undefined controls and processes Repeatable

Standardized methods for repeatable processes Defined

Monitors and improves processes Managed

Advanced controls, metrics and feedback Optimizing

Metrics for optimization purposes


Level 1 - Initial

Chaotic processes Unpredictable cost, schedule and quality

performance Reasons

Unplanned commitments Gurus Magic Problems of scale


Repeatable

Management commitment Project management systems Quarterly reviews

• Projects reviews Milestone, financial status, issues, staff comments

• Computing support review Performance measures, Issues

• Process Status Assessment updates, Technology plan status, AI status

• Organizational performance Productivity, Quality and AI Summary


Repeatable

Project plan Planning

• Work break down structure Size and Estimation

• LoC, Function points Productivity and Scheduling

• User productivity• Project tracking and checkpoints


Repeatable

Software Configuration Management Baselines

• Configuration control• Change management• Revisions, Versions, Deltas, Conditional code

Tools• System libraries

Releases• Authorized releases• Change control boards


Repeatable

Software Quality Assurance SQA Plan

• To ensure appropriate development methodology• Standardize procedures and processes• Make projects auditable by external agencies• Change control measures

SQA People• Do not put fresh people

Independent Verification and Validation


Defined Process

Software Standards Software Inspection

Inspection training Reports and Tracking

Software Testing Different Testing


Defined Process

Advanced Configuration Management Design and Control Configuration during different phases Software Configuration Audit

Defining the software processes Process models


Defined ProcessSoftware Engineering Process Groups

Role• Identify key problems• Establish priorities• Define Action plans• Get professional and management agreement• Assign people• Provide guidance• Launch• Track progress• Fix problems

Standards establishment Process Databases Education and Training Processes Consultation and Assessments


Managed Processes

Data Gathering and Analysis Software measures and metrics Analysis

Managing Software Quality Measurement Criteria Estimating software quality Quality Goals Quality plans Tracking and controlling


Optimizing Processes Defect Prevention

Processes• Cause – effect categories

Automating the software process Organizational plans Technology transitions

Software Contracting Negotiation Process management Certification


Assessment Overview

Not an audit Review the organization’s software

capabilities and Advise the management and professionals

on how they can improve operations Identify high priority areas of improvement Only a guidance on improvement of

processes


Assessment Phases

Preparation Senior management approval on commitment

to quality Training programme for the assessment team

Assessment Questions and matching answers and proofs

Recommendations Local team for putting action items in place


Assessment Principles

Process model is the basis of assessment Confidentiality Senior management involvement Attitude of respect for the views of people

in the organization being assessed Action orientation


Assessment Process - I

Form an assessment team Train the assessment team Confidentiality of assessment process Site manager or head participates in the

opening and closing assessment A person responsible to implement Action

Items are assigned


Assessment Process - II

Day 1 Assessment Overview (Head, staff and

participants Briefing (participants) Questionnaire (project representatives)

• Completed for each project / project type Project discussions (project representatives)

• To clarify any questions and any evidence that is needed


Assessment Process - III Day 2

Functional areas interviews (functional rep) Preliminary findings (assess. team)

Day 3 Project discussions (project representatives) Findings formulation (assess. team)

Day 4 Findings Dry run (assess. team) Findings review (Project representatives) Findings presentation (Head, staff, participants) Senior management meeting (Head and assessment team

leader) Assessment postmortem (assessment team)

Conducting Project Assessments

Scope

• Assess end to end methodologies for the development and management of the project

• Overall development effectiveness and efficiency instead of in-process quality status and improvement opportunities

• Conducted by external people than in-project members

Audit and Assessment

• Audit– IEEE

• An independent examination of a work product or a set of work products to assess compliance with specifications, standards, contractual agreements and other criteria

– ISO• Certification, third party assessment is carried out by an

independent organization against a particular standard

• Outcome– In compliance or not in compliance – Pass or fail

Assessment

• A review of a software organization to advise its management and professionals on how they can improve their operation

• Assess maturity level vs. process roadmap• Identify current practices• Identify strength / weakness• find out causes for poor quality

DoD’s mechanisms

• Evaluates a third party vendor• Audits them rather than assess them• Done by a second party• Assessments are of 3 types

– First party– Second Party– Third Party

Assessment issues

• Organizational assessments– Sample project might lead to different

assessments– Organization of the company could be a factor

Assessment issues• Project level assessments do not suffer from the

above issues– Meaningful factors must be included– Degree of implementation and effectiveness must be

measured– Exploratory and In-depth probing is a characteristic of

this assessment– Independent assessments must be made to be

objective• Both these assessments should be

complementary

Software Process Assessment Cycle

• Select a team• Appraise the team• Expand on the assessment areas• Do a site visit• Find strengths and weaknesses• Create a KPA profile to present it to the

appropriate audience

Rules• Team should be lead by a Lead Assessor• Team should have 4 – 10 members• Atleast one team member from the organization• Must complete the CMM Based Approach for

Internal Process Improvement training• Must use

– Standard Questionnaire– Individual and Group Interviews– Document reviews– Feedback from the review of draft findings

Standard CMMI Assessment Method for Process Improvement

• Plan and preparation phase– Identify assessment scope– Develop assessment plan– Prepare and train assessment team– Make a brief assessment of participants– Administer the CMMI appraisal questionnaire– Examine Questionnaire responses– Conduct initial document review

SCAMPI• Online assessment phase

– Conduct an opening meeting– Conduct interviews– Consolidate information– Prepare presentation of draft findings– Prepare draft findings– Consolidate, rate and prepare final findings

• Reporting– Present final findings– Conduct executive session– Wrap up the assessment

Zahran’s Generic Process

Proposed SPA Method• It is project based• Facts Gathering

– Project methodology review– State of practices review– Questionairre completion and response validation

• Questionairre customization – No standardized questionairre

• Observations analysis and recommendations go hand in hand with development

• Direct input to the project teams given

Sample questionnaire

• What is the most common form of design reviews for this project?– On maturity

• To what extent were design reviews of the project conducted?– On project process

Successful Projects

• Effective– planning, cost estimating, measurements,

milestones tracking, quality control, change management, development processes, communication, project managers, technical personnel, specialists, substantial amount of reusable materials

• Failing Projects– None of the above

Example finding and recommendation

Assessment report

• Project information and basic project data• Assessment approach• Brief description and observation of project team

practices• Strengths and Weaknesses, gap analysis• Critical success factors and major project pitfalls• What would they do differently?• Recommendations

Question

• How is it different from audit report?

Sample project taken for assessment

Improvement plans for the project based on assessment

Project Y

Project Z

Summary

Dos and Don’ts of Software Process improvement

SCAMPI Model Advantages

Measuring Process capability

• 0 – 5 levels are used

• is this OK?

Note

• Measuring levels is not enough– The SEPG implementing practices in the

project but the organization does not reach level x

– The organization has reached level x but most of the projects are not following the practices

Note

• Alignment principle– Faster, Cheaper or better – pick one from

SEPG• Take time getting faster

– Be reliable rather than deliver on time

Notes

• Keep it simple • Measure the value of process

improvement• Measuring process adoption

Notes

• Measuring process compliance

• Celebrate the Journey Not the Destination

Summary

Thank you

Questions?

Defect Removal Defect Removal EffectivenessEffectiveness

Importance of this topicImportance of this topic

Defect Prevention in standardsDefect Prevention in standardsStatistical methodsStatistical methodsQuality improvementQuality improvementAffects schedulesAffects schedulesReduction in development timesReduction in development timesMeasure for effectiveness of defect Measure for effectiveness of defect removal processremoval process

GrowthGrowth

Earlier daysEarlier daysOnly defect removal method was testingOnly defect removal method was testing

LaterLaterFormal reviewsFormal reviewsInspectionsInspections

Initial formula for error detection efficiencyInitial formula for error detection efficiencyErrors Found by an inspection

--------------------------------------- * 100 %

Total error in the product

AnalysisAnalysis

If it is > 100 % then the inspection has If it is > 100 % then the inspection has found too many errors or the product was found too many errors or the product was not tested properly before inspection.not tested properly before inspection.

What should be done to improve What should be done to improve DRE?DRE?

Code / Design inspectionCode / Design inspectionRequirements analysis and inspection Requirements analysis and inspection

NASA NASA –– Houston centerHouston centerIBM Houston center has proven the IBM Houston center has proven the relationship between DRE with product relationship between DRE with product qualityquality

Other Definition of DREOther Definition of DRENumber of defects found in each phase

------------------------------------------------------------------------------------- * 100 %

Number of defects found in each phase + Number of defects found in subsequent activities (phases)

Total Defect Containment Effectiveness =

Number of pre release defects

========================================================

Number of prerelease defects + Number of post release defects

Phase Containment Effectiveness =

Number of phase i errors (found in dev phase)

======================================================

Number of phase i errors + Number of phase i defects (found later)

DRE formal developmentDRE formal development

PhasePhase Injected Injected RemovedRemovedRequirementsRequirements Requirements / Functional Requirements / Functional

specificationspecificationRequirements analysis and Requirements analysis and reviewreview

HLDHLD Design workDesign work High level design inspectionHigh level design inspection

LLDLLD Design workDesign work LLD InspectionLLD Inspection

Coding Coding CodingCoding Code inspectionCode inspection

IntegrationIntegration IntegrationIntegration Build verification testingBuild verification testing

Unit testingUnit testing Bad fixesBad fixes TestingTesting

Component TestingComponent Testing Bad fixesBad fixes TestingTesting

System TestingSystem Testing Bad fixesBad fixes TestingTesting

ProcessProcess

Defects existing Defects existing -- Defects injected Defects injected

Defects detected Defects detected –– Defects not detectedDefects not detected

Defects fixed Defects fixed –– Defects incorrectly fixedDefects incorrectly fixed

Remaining defectsRemaining defects

+

+

-

Defect Origin vs. place found

ReqReq HLDHLD LLDLLD CC UTUT CTCT STST CSCS Tot.Tot.

I0I0

I1I1

I2I2

UTUT

CTCT

STST

FF

Tot.Tot.

4949 681681

Use of the matrixUse of the matrix

Phase wise DRE can be foundPhase wise DRE can be found

Phase Defect Removal EffectivenessPhase Defect Removal EffectivenessOverall Inspection effectivenessOverall Inspection effectivenessTest effectivenessTest effectivenessRequirements effectivenessRequirements effectivenessDRE for development etc.DRE for development etc.

DRE and Quality PlanningDRE and Quality PlanningUse control charts for quality planningUse control charts for quality planningThis can be done for each phaseThis can be done for each phaseSee the + and See the + and –– in the figure.in the figure.

Use this for Phase Based Defect removal model Use this for Phase Based Defect removal model to address questions like,to address questions like,

if we increase the effectiveness of unit testing by if we increase the effectiveness of unit testing by 5%, then what is the gain in final quality of the 5%, then what is the gain in final quality of the product?product?

Cost EffectivenessCost Effectiveness

Defect removed at initial phases are less Defect removed at initial phases are less expensive than the ones removed at the expensive than the ones removed at the endendSo inverse of the absolute values in the So inverse of the absolute values in the matrix can be taken as matrix can be taken as weightageweightage!!!!!!Inspections can be modified to improve Inspections can be modified to improve effectiveness effectiveness

Having more than one inspectionHaving more than one inspection

DRE and Process MaturityDRE and Process Maturity

Level 1 = 85%Level 1 = 85%Level 2 = 89%Level 2 = 89%L3 = 91%L3 = 91%L4 = 93%L4 = 93%L5 = 95%L5 = 95%

L4 cumulative percentages of L4 cumulative percentages of Defects removed by phaseDefects removed by phase

Requirements 94%Requirements 94%HLD 95%HLD 95%LLD 96%LLD 96%Coding 94%Coding 94%IT = 75%IT = 75%ST = 70%ST = 70%AT = 70%AT = 70%

Questions?Questions?

In Process Software Testing Metrics

Agenda

• To discuss metrics that have been proven to be efficient in the industry (IBM)

• Purpose of the metric, data, interpretation, use and real life example

Test Progress S Curve

• Tests Planned for the week• Tests Attempted for the week• Actual Tests run for the week

Scores to test cases

• Test cases are assigned scores depending on their criticality

• Every week you need to run x test cases and score over y points for tracking

• The total points is calculated as • SUM(yi) where the summation is done

over all the test cases

Weekly activities

• Test Coverage weighing• Test score• Graph tracking• Find the disparity in testing between

planned and actual and look at schedule disparity ( > 15 % schedule skip by a week etc.)

Problems with S-Curve approach

• Module wise details are missing• Steep S-curve means more test cases in

less period of time • All the questions regarding the test cases

must be reviewed• Curve might shift to the right indicating

project delay• What are the solutions for above?

Other advantages

• For 2 consecutive releases overlapping the S curve gives more information on the front end / back end testing

Testing Defect Arrivals over Time

• Discuss the following test arrival rate

Testing Defect Arrivals

• Discuss the following

TDAR

• Discuss the following

Extrapolating Defect Arrival Pattern

• How to predict the defect volume• Lots of reliability models are available

Defect Backlog Metric

• Discuss

Product Size over time

• Discuss various boxes• Discuss the trend

CPU Utilization

• Discuss this metric

System Crashes and Hangs

• What does this tell?

In process metric and quality management

• Use calendar time• Shipped time should be the reference for

X axis and every week the metrics need to be tracked

• Metrics should indicate good or bad status• Bad status for a metric should force

management action• Metrics should drive process

improvements

The Effort Outcome model

• CELL 2 BEST-CASE SCENARIO.– GOOD QUALITY DESIGN / CODE– LOW ERROR INJECTION – EFFECTIVE TESTING.

• CELL 1 IS A GOOD/NOT BAD SCENARIO. – LATENT DEFECTS FOUND VIA EFFECTIVE TESTING.

• CELL 3 IS THE WORST-CASE SCENARIO. – BUGGY CODE AND PROBABLY PROBLEMATIC DESIGNS– HIGH ERROR INJECTION DURING THE DEVELOPMENT

PROCESS. • CELL 4 IS THE UNSURE SCENARIO.

– CANNOT ASCERTAIN WHETHER THE LOWER DEFECT RATE IS A RESULT OF GOOD CODE QUALITY OR INEFFECTIVE TESTING.

Merging S Curve and E/O Model

• Cell 2– Defect arrival lower, S curve ahead or same as

baseline• Cell 1

– Defect arrival higher, S curve ahead or same as baseline

• Cell 3– Late defect arrivals, S curve behind

• Cell 4– S curve behind, defect arrival lower in early part of the

curve

Metrics for Vendor developed software

• % of test cases attempted• % of defects per executed test case• No. of failing test cases without defect

records• Success rate, persistent failure rate, • Defect injection rate, code completeness

When is your product good to ship?

Thanks / Questions?

Complexity Metrics and Models

Need

• White box testing• More granular measurement• Done at program module level• Helps software engineers to improve the

quality of their work

Models and usage

• Reliability models– Researchers and software reliability practitioners – Need mathematics and statistics

• Quality Management models– Software quality professionals and project managers

• Software Complexity Research– Computer scientists and experimental software

engineers

Most familiar LoC• Originated from Assembly language• Inverse relation ship with respect to LoC exists

until 500 lines of code– Reason is interface errors

• With advancement – concave models seem to suit better and the tail raises– Increase in module size is proportional to

comprehension of the same• Optimum program size for each language /

project can be found

Halstead’s software science

• Given• n1 = number of distinct

operators in a program• n2 = number of distinct

operands in a program• N1 = number of operator

occurrences• N2 = number of operand

occurrence

• Vocabulary (n) = n1 + n2• Length (N) = N1 + N2

– = n1 log n1 + n2 log n2

• Volume (V) = N log n– = N log (n1+n2)

• Level (L) = V* / V• =(2/n1) x (n2 / N2)

• Difficulty (D) = V / V*– (n1 / 2) x (N2 / n2)

• Effort (E) = V / L• Faults (B) = V / S*

V* = minimum volume represented by a built-in function performing the task of the entire program

S* = Mean number of mental decisions

Cyclomatic complexity• Indicates program’s

testability and understandability

• Denotes number of regions in a graph

• Gives the linearly independent path that comprises of the program

• Effort required to test the program

• M = V(G) = e – n + 2p

• V(G) = Cyclomatic number of G

• e = number of edges• n = number of nodes• p = number of

unconnected parts of the graph

• Also equal to number of binary decision (if) + 1

Recommendation

• Cyclomatic complexity must be less than 10

• Has moderate to strong correlation with defect rates

Uses

• To help identify overly complex parts needing detailed inspections

• To help identify non complex parts likely to have a low defect rate and therefore candidates for development without detailed inspection

• To estimate programming and service effort, identify troublesome code and estimate testing effort

Syntactic Constructs

• Complexity takes into account only if then else's

• Studies were done taking into account different constructs

• Programmers found it difficult to understand the “do- while” construct

Structure metrics• Takes into account the interaction between the

modules of a product• Most important

– Fan-in = count of the modules that call a given module

– Fan-out = count of the modules that are called by a given module

• Modules with more Fan-in have insignificant correlation with defect levels

• Modules with more Fan-out have positive correlation with defects

Summary

• Two types of metrics– Syntactic constructs– Structure

• Measure both• Applied for software engineers• Other type of metrics

– Reliability– Project related

Measuring and Analyzing Customer Satisfaction

Need

• TQM point of view– Integrates product quality with customer

satisfaction• Enhancing CS is the bottom-line for any

business• Customer focus is the only way to retain

the business• News of dissatisfied customer goes out

twice as fast as a satisfied customer

Customer Satisfaction Surveys

• Source– telephone follow-up– customer complaint data– direct customer visits– customer advisory councils– user conferences

• Must cover the entire customer base

Customer Satisfaction Surveys• Methods

– Face to face interviews• prestructured questionnaire• limitation is cost factor concerning the interviewer• Biases can be introduced in the data by a wrong interviewer• Unbiased interviewer

– telephone interviews• should be short and impersonal• can be monitored for improvements• must be short• lack of direct interaction• lack of using exhibits• Limited group of respondents

Customer Satisfaction Surveys– mailed questionnaires

• Less expensive• less response rates• may introduce bias• more analysis needed• questions must be pre-tested

• Summarizing order– In person– phone – mail

Sampling Methods

• Use scientific probability sampling methods– simple random sampling

• Every sample can be selected with p = 1/n– systematic sampling

• Take every kth individual– stratified sampling

• group into overlapping “strata”– cluster sampling

• Group into clusters such as geographical unit

Sample Size

• Sample size depends on– confidence level– margin of error

• Statistical formulas are available for determining sample size

Analyzing Satisfaction Data

• 5 point scale• Confidence intervals can also be a part of

the bar or run charts which show the scale• Some choose customer dissatisfaction as

a metric

Analyzing Satisfaction Data

• Find specific attribute satisfaction and Overall satisfaction– Area of weakness NEED NOT be the priority

of improvement– E.g.. Documentation and reliability

• Correlate between attributes and overall satisfaction– For example 93.8 % reliability figure with high

correlation to CS

Satisfaction with company

• Overall satisfaction and Loyalty of a customer has been attributed to– set of common attributes of the company– satisfaction levels of dimensions of the entire

company• Company and Product = Customer

satisfaction• Examples

Example• Technical solutions

– R, A, Ease of use, pricing, installation• Support & Service

– Flexible, accessible, product knowledge• Marketing

– Solution, central point of contact, information• Administration

– purchasing, billing, warranty expiration, notification• Delivery

– on time, accurate, post delivery process• Company image

– technology leader, financial stability, executives image

How much CS is good?

• Long term goal = 100 %• Tradeoff between customer satisfaction

and market share• Babich formula

So what do you do?

• Also monitor your competitors CS• perform analyses on specific satisfaction

dimensions, quality attributes of products, strength, weakness, prioritization

• Do RCA to identify inhibitors in each dimension

• Set satisfaction targets• Formulate and implement action plans

Other than this do

• post purchase call back• Complaint management process

Summary

• Various methods for CS• Methods• Sampling• Analyzes• Beyond customer satisfaction

In process quality assessment

Need

• Your quality program is on track to satisfy the objectives

• Find out the current and future risks on the project

• Meet quality expectation of customers• Method

– Preparation, Evaluation, Summarization and Recommendations

Assessment and Audit

• Audit– Compares actual process with the defined

process• Assessment

– What is occurring?– What is the difference between right results

and results obtained?– What is the likely result?

In process assessment

• Should be integral to project management• Example

Preparation

• Depends on the development process– Expect only fewer metrics to be available

during initial phases– Concentrate on process steps being followed

during initial phases

Example

Caution

• Also look at qualitative data• Find out who you want to talk to?• Find out what you want to talk about?• Get variety of people including developers,

testers etc.• Frame questions

– Where are we?– What’s the outlook etc?

Evaluation

• Quantitative data– 7 tools– Get useful information

• Pay attention to things that are abnormal• Example

Evaluation

• Qualitative data– Use metrics

• Use expert opinion• Criteria

– Use expert judgment and cross validation for your judgment

– Use dashboards G,Y, R

Summarization

• Summarize using– Impact Ranking– Comparison with previous known issues– Identify what is done right

• Example

Overall Assessment Summarization

• Overall assessment is made the bottom line– Red – high probability of not meeting the

quality goals or customer quality expectations– Yellow – moderate risk of not meeting the

quality goals or customer quality expectations– Green – likely to meet product quality goals

and satisfy customer quality expectations

Desirable vs. undesirable scenario

• With every audit you move from Red to Green and not vice versa

R

Y

G

QA1 QA2 QA3 QA4 QA5 QA6 QA7 QA8

Recommendation

• Recommendations are part of assessment summary

• No magic formula• Various questions can be answered based

on the investigation findings

Risk Mitigation

Summary

• What is in-process quality audit?• What is a good assessment?• Store assessment records• Record lessons learned• Assessment depends on assessor• Manage risks based on assessments

Statistical Quality Control

Quality and Variability Variability is the enemy of quality Variability can be controlled using

Inspection Prevention

Use statistial process control

SPC Two type of variability

Random Variation Systematic Variation

Mean Control Charts Center line is the control line Lower Control Limit Upper Control Limit Process statistics must fall between LCL and

UCL Process is in Control if the values fall in

between LCL and UCL and are random Any pattern implies the process may be out of

control

Using LCL and UCL Once statistics shows that the process is out of

control, use the fish bone diagram or other means to find the root cause and get the process in control

Using Mean Control Charts Calculate grand mean given the data

Grand mean = sum of all observations ---------------------------------- (Number of samples * number of

observations in each sample) LCL = Grand mean - ((3 * mean range) / (d2 *

sqrt(n))) UCL = Grand mean + (( 3 * mean range)/ (d2 *

sqrt(n))) d2 will be given as a table for each sample size

Using Mean Control Charts Calculate Grand Mean Calculate RANGE MEAN Check value of d2 Get the LCL and UCL Study the pattern of the control chart Make conclusions

Decision Tree Analysis Define the problem in structured terms Model the decision process as a decision tree Apply the appropriate probability values and

financial data Solve the decision tree Perform Sensitivity Analysis List the underlying assumptions

Thank you

Software Quality Model Requirements for Software Quality Engineering

Marc-Alexis Côté M. Ing ([email protected])

Witold Suryn PhD ([email protected])

Elli Georgiadou ([email protected])

Keywords Software Quality Engineering, Software Quality Models, ISO/IEC 9126.

Abstract Software Quality Engineering is an emerging discipline that is concerned with improving the approach to software quality. It is important that this discipline be firmly rooted in a quality model satisfying its needs. In order to define the needs of this discipline, the meaning of quality is broadly defined by reviewing the literature on the subject. Software Quality Engineering needs a quality model that is usable throughout the software lifecycle and that it embraces all the perspectives of quality. The goal of this paper is to propose a quality model suitable for such a purpose, through the comparative evaluation of existing quality models and their respective support for Software Quality Engineering.

Introduction Over the last decade, the general focus of the software industry has shifted from providing ever more functionality to improving what has been coined as the user experience. The user experience refers to characteristics such as ease-of-use, security, stability and reliability. Improvements in such areas lead to an improved quality as perceived by the end users. Some software products, most notably Microsoft's next iteration of their Windows operating system, have been delayed by as much as two years in order to improve their quality. There is no doubt that software quality is becoming an increasingly important subject in software engineering.

Traditionally, software requirements have been classified either as functional or non-functional with eventual notions of quality hidden in the latter. As the industry focus is shifting from functionality to improving quality, a new category of requirements focused on quality is emerging. In order to define these new quality requirements, quality itself must be defined. A quality model provides the framework towards a definition of quality. Engineers have long recognised that in order for something to find its way in a product, it should be properly defined and specified. Unfortunately, the push towards software quality that can be observed in the industry today is lacking a solid foundation in the form of an agreed upon quality model that can be used not only to evaluate software quality, but also to specify it.

Bourque (2000) suggests that the implementation of quality in a software product is an effort that should be formally managed throughout the Software Engineering lifecycle. The implementation of quality should therefore begin with the specification of user quality requirements. Such an approach to the implementation of quality leads to Software Quality Engineering. Suryn (2003) has suggested that this discipline be defined as the application of a

continuous, systematic, disciplined, quantifiable approach to the development and maintenance of quality of software products and systems; that is, the application of quality engineering to software.

The objective of this paper is to identify the requirements for a software quality model to be used as a foundation to Software Quality Engineering.

Definition of Software Quality What exactly constitutes the quality of a product is often the subject of a hot debate. The reason the concept of quality is so controversial is that people fail to agree on what it means. For some it is “[the] degree to which a set of inherent characteristics fulfills requirements” (ISO/IEC 1999b) while for others it can be synonymous with “customer value” (Highsmith, 2002), or even “defect levels” (Highsmith, 2002). A possible explanation as to why any of these definitions fail to garner a consensus is that they generally fail to recognize the different perspectives of quality. Kitchenham and Pfleeger (1996), by reporting the teachings of David Garvin, report on the 5 different perspectives of quality:

• The transcendental perspective deals with the metaphysical aspect of quality. In this view of quality, it is “something toward which we strive as an ideal, but may never implement completely.” (Kitchenham & Pfleeger, 1996);

• The user perspective is concerned with the appropriateness of the product for a given context of use. Kitchenham and Pfleeger further note that “whereas the transcendental view is ethereal, the user view is more concrete, grounded in the product characteristics that meet user's needs”;

• The manufacturing perspective represents quality as conformance to requirements. This aspect of quality is stressed by standards such as ISO 9001, which defines quality as “[the] degree to which a set of inherent characteristics fulfills requirements” (ISO/IEC 1999b). Other models, like the Capability Maturity Model (CMM) state that the quality of a product is directly related to the quality of the engineering process, thus emphasising the need for a manufacturing-like process;

• The product perspective implies that quality can be appreciated by measuring the inherent characteristics of the product. Such an approach often leads to a bottom-up approach to software quality: by measuring some attributes of the different components composing a software product, a conclusion can be drawn as to the quality of the end product;

• The final perspective of quality is value-based. This perspective recognises that the different perspectives of quality may have a different importance, or value, to various stakeholders.

One could argue that in a world where conformance to ISO and IEEE standards is increasingly present in contractual agreements and used as a marketing tool (Adey & Hill, 2000), all the perspectives of quality are subordinate to the manufacturing view. This importance of the manufacturing perspective has increased throughout the years through works like Quality is Free (Crosby, 1979) and the popularity of movements like Six-Sigma (Biehl, 2001). The predominance of the manufacturing view in Software Engineering can be traced back to the 1960s, when the US Department of Defense and IBM gave birth to Software Quality Assurance (Voas, 2003). This has led to the belief that adherence to a development process, as in manufacturing, will lead to a quality product. The corollary to this belief is that process improvement will lead to improved product quality. According to many renowned researchers, this belief is false, or at least flawed. Geoff Dromey states:

“The flaw in this approach [that you need a quality process to produce a quality product] is that the emphasis on process usually comes at the expense of constructing, refining, and using adequate product quality models.” (Dromey, 1996)

Kitchenham and Pfleeger reinforce this opinion by stating:

“There is little evidence that conformance to process standards guarantees good products. In fact, the critics of this view suggest that process standards guarantee only uniformity of output [...]” (Kitchenham & Pfleeger, 1996)

Furthermore, data available from Agile (Highsmith, 2002) projects show that high quality is attainable without following a manufacturing-like approach.

However, recent studies conducted at Motorola (Eickelman, 2003; Diaz & Sligo, 1997) and Raytheon (Haley, 1996) show that there is indeed a correlation between the maturity level of an organization as measured by the Capability Maturity Model and the quality of the resulting product. These studies provide data on how a higher maturity level (as measured by the CMM) can lead to:

• Improved error/defect density (i.e. the error/defect density lowers as maturity improves)

• Lower error rate

• Lower cycle time (time to complete parts of the lifecycle)

• Better estimation capability

From these results, one could conclude quality can be improved by following a mature process. Georgiadou (2003a) studied the development of lifecycle models, and established that the maturity of the development process is reflected by the emphasis and location of testing and other quality assurance activities. Her study demonstrated that the more mature the process and its underlying lifecycle model the earlier the identification of errors in the deliverables. However, these measured improvements are directly related to the manufacturing perspective of quality. Therefore, such quality improvement efforts fail to address the other perspectives of quality. This might be one of the reasons that some observers of the software development scene perceive the “quality problem” as one of the main failings of the software engineering industry. Furthermore, studies show that improvement efforts grounded in the manufacturing perspective of quality are difficult to scale down to smaller projects and/or smaller teams (Laitinen, 2000; Boddie, 2000). Indeed, rather than being scaled down in smaller projects, these practices are simply not performed.

Over recent years, researchers have proposed new models that try to encompass more perspectives of quality than just the manufacturing view. Geoff Dromey (1995; 1996) proposed such model a model in which the quality of the end product is directly related to the quality of the artifacts that are a by-product of the process being followed. Therefore, he developed different models that can be used to evaluate the quality of the requirements model, the design model and the resulting software. The reasoning is that if quality artifacts are conceived and produced throughout the lifecycle, then the end product will manifest attributes of good quality. This approach can clearly be linked to the product perspective of quality with elements from the manufacturing view. This is certainly a step forward from the manufacturing-only approach described above, but it fails to view the engineering of quality as a process that covers all the perspectives of quality. Pfleeger (2001) warns against approaches that focus only on the product perspective of quality:

“This view [the product view] is the one often advocated by software metrics experts; they assume that good internal quality indicators will lead to good external ones, such as reliability and maintainability. However, more research is needed to verify these assumptions and to determine which aspects of quality affect the actual product's use.”

Georgiadou (2003b) developed a generic, customisable quality model (GEQUAMO) which enables any stakeholder to construct their own model depending on their requirements. In a further attempt to differentiate between stakeholders Siaka et al (1997) studied the viewpoints of users, sponsors and developers as three important constituencies/stakeholders and suggested attributes of interest to each constituency as well as level of interest. More recently, Siaka and Georgiadou (2005) reported the results of a survey amongst practitioners (from the UK, Greece, Egypt and Cyprus) on the importance placed on product quality characteristics. Using their empirical results they extended ISO 9126 by adding two new characteristics namely Extensibility and Security which have gained in importance in today’s global and inter-connected environment.

The above observations illustrate the main disagreements that exist in both the research community and the industry on the subject of software quality. The goal of a quality model is in essence to provide an operational definition of quality. While specific definitions have been established for given contexts, there is no consensus as to what constitutes quality in the general sense in software engineering. A first requirement for a software quality model to be useful as a foundation for Software Quality Engineering is thus to encompass all the perspectives of quality mentioned at the beginning of this section.

Specification and evaluation of quality Software Quality Engineering calls for a formal management of quality throughout the lifecycle. In order to support this requirement, a quality model should have the ability to support both the definition of quality requirements and their subsequent evaluation. This can be explained by referring to the manufacturing perspective of quality, which states that quality is conformance to requirements. A quality model that is to be used as the foundation for the definition of quality requirements should help in both the specification of quality requirements and the evaluation of software quality.

IEEE Std 1061-1998 (IEEE, 1998) defines this as a top to bottom and bottom to top approach to quality:

From a top down perspective the [quality] framework facilitates:

• Establishment of quality requirements factors, by customers and managers early in a system's life cycle;

• Communication of the established quality factors, in terms of quality sub-factors, to the technical personnel;

• Identification of metrics1 that are related to the established quality factors and quality sub-factors.

From a bottom up perspective the [quality] framework enables the managerial and technical personnel to obtain feedback by

1 In 2002, the ISO/IEC JTC1 sub-committee SC7 – Systems and Software Engineering – replaced the term “metric” by “measure” to align its vocabulary with the one used in metrology. This thesis will use the term measure whenever possible.

• Evaluating the software products and processes at the metrics level;

• Analysing the metric values to estimate and assess the quality factors.

A quality model that is to be used as the foundation for the definition of quality requirements should help in both the specification of quality requirements and the evaluation of software quality. In other words, it should be usable from the top of the development process to the bottom and from the bottom to the top.

Evaluation of quality models Three requirements that a quality model should possess to be a foundation for Software Quality Engineering have been identified:

• A quality model should support the 5 different perspectives of quality as defined by Kitchenham and Pfleeger (1996);

• A quality model should be usable from the top to the bottom of the lifecycle as defined by IEEE Std 1061-1998 (IEEE, 1998), i.e. should allow for defining quality requirements and their further decomposition into appropriate quality characteristics, subcharacteristics and measures;

• A quality model should be usable from the bottom to top of the lifecycle as defined by IEEE Std 1061-1998 (IEEE, 1998), i.e. should allow for required measurements and subsequent aggregation and evaluation of obtained results.

Four quality models will be evaluated with respect to these requirements.

McCall McCall (McCall, Richards & Walters, 1977) introduced his quality model in 1977. According to Pfleeger (2001), it was one of the first published quality models. Figure 1 presents this quality model. Each quality factor on the left hand side of the figure represents an aspect of quality that is not directly measurable. On the right hand side are the measurable properties that can be evaluated in order to quantify the quality in terms of the factors. McCall proposes a subjective grading scheme ranging from 0 (low) to 10 (high).

Regarding this model, Pressman notes that “unfortunately, many of the metrics defined by McCall et al. can be measured only subjectively” (Pressman, 2001). It is therefore difficult to use this framework to set precise and specific quality requirements. Furthermore, some of the factors and measurable properties, like traceability and self-documentation among others, are not really definable or even meaningful at an early stage for non-technical stakeholders. This model is not applicable with respect to the criteria outlined in the IEEE Standard for a Software Quality Metrics Methodology for a top to bottom approach to quality engineering. Furthermore, it emphasises the product perspective of quality to the detriment of the other perspectives. It is therefore not suited as a foundation for Software Quality Engineering according to the stated premises.

Figure 1: McCall’s Quality Model

Adapted from Pfleeger (2003) and McCall et al. (1977)

Boehm Boehm's quality model improves upon the work of McCall and his colleagues (Boehm, Brown, Kaspar, Lipow & MacCleod, 1978). As Figure 2 shows, this quality model loosely retains the factor-measurable property arrangement. However, for Boehm and his colleagues, the prime characteristic of quality is what they define as “general utility”. According to Pfleeger (2001), this is an assertion that first and foremost, a software system must be useful to be considered a quality system. For Boehm, general utility is composed of as-is utility, maintainability and portability (Boehm et al., 1976):

• How well (easily, reliably, efficiently) can I use it [software system] as-is?

• How easy is it to maintain (understand, modify, and retest)?

• Can I still use it if I change my environment?

If the semantics of McCall's model are used as a reference, the quality factors could be defined as: Portability, Reliability, Efficiency, Human Engineering, Testability, Understandability and Modifiability. These factors can be decomposed into measurable properties such as Device Independence, Accuracy, Completeness, etc. Portability is somewhat incoherent in this classification as it acts both as a top level component of general utility, and as a factor that possesses measurable attributes.

Figure 2: Boehm’s quality model

Adapted from Pfleeger (2003), Boehm et al. (1976; 1978)

It is interesting to note that in opposition to McCall's model, Boehm's model is decomposed in a hierarchy that at the top addresses the concerns of end-users while the bottom is of interest to technically inclined personnel. It is in effect the emergence of the user perspective of quality. However, this interest wanes when one reads Boehm's definition of the characteristics of software quality. Except for General Utility and As-is Utility, all definitions begin with “Code possesses the characteristic [...]”. The measurable properties and characteristics therefore concentrate on highly technical details of quality that are difficult to grasp for non-technical stakeholders that are typically involved early in the software lifecycle. The characteristics General Utility and As-is Utility are too generic and imprecise to be useful for defining verifiable requirements. Like the McCall model, this model is mostly useful for a bottom to top approach to software quality (i.e. it can effectively be used to define measures of software quality, but is more difficult to use to specify quality requirements).

While this model is a step forward in the sense that it provides basic support for a top to bottom approach to software quality, this support is too ephemeral to be considered as a solid foundation for quality engineering.

Dromey Dromey's (1995) model takes a different approach to software quality then the two previously presented models. For Dromey, a quality model should clearly be based upon the product perspective of quality:

“What must be recognized in any attempt to build a quality model is that software does not directly manifest quality attributes. Instead it exhibits product characteristic that imply or contribute to quality attributes and other characteristics (product defects) that detract from the quality attributes of a product. Most models of software quality fail to deal with the product characteristics side of the problem adequately and they also fail to make the

direct links between quality attributes and corresponding product characteristics.” (Dromey, 1995) (Emphasis added to support the argument)

Dromey has built a quality evaluation framework that analyzes the quality of software components through the measurement of tangible quality properties (Figure 3). Each artifact produced in the software lifecycle can be associated with a quality evaluation model. Dromey gives the following examples of what he means by software components for each of the different models:

• Variables, functions, statements, etc. can be considered components of the implementation model;

• A requirement can be considered a component of the requirements model;

• A module can be considered a component of the design model;

• Etc.

According to Dromey (1995), these components all possess intrinsic properties that can be classified into four categories:

• Correctness: Evaluates if some basic principles are violated.

• Internal: Measure how well a component has been deployed according to its intended use.

• Contextual: Deals with the external influences by and on the use of a component.

• Descriptive: Measure the descriptiveness of a component (for example, does it have a meaningful name?).

These properties are used to evaluate the quality of the components. This is illustrated in Figure 4 for a variable component present in the implementation model.

Figure 3 : Dromey’s Quality Model

Figure 4 : Quality evaluation of a variable component

It seems obvious from the inspection of the previous figures that Dromey's model is focused on the minute details of quality. This is stated explicitly:

“What we can do is identify and build in a consistent, harmonious, and complete set of product properties (such as modules without side effects) that result in manifestations of reliability and maintainability.” (Dromey, 1996)

For Dromey, the high level characteristics of quality will manifest themselves if the components of the software product, from the individual requirements to the programming language variables1, exhibit quality-carrying properties. Dromey's hypothesis should be questioned. If all the components of all the artifacts produced during the software lifecycle exhibit quality-carrying properties, will the resulting product manifest characteristics such as maintainability, functionality, and others?

The following analogy will be useful in answering this question:

If you buy the highest quality flour, along with the highest quality apples and the highest quality cinnamon, will you automatically produce an apple pie that is of the highest quality?

The answer is obviously negative. In addition to quality ingredients, at least three more things are needed in order to produce an apple pie of the highest quality:

• A recipe (i.e. an overall architecture and an execution process). Dromey acknowledges this by identifying process maturity as a desirable high level characteristic. However, it is only briefly mentioned in both his publications on the subject (Dromey, 1995; Dromey, 1996).

• The consumer's tastes must be taken into account. In order for the result to be considered of the highest quality by the consumer, it needs to be tuned to his tastes. This is akin to what is commonly called user needs in software engineering. User

needs are completely ignored by Dromey. However, as it was demonstrated in the introduction, they are an integral and non-negligible part of software quality.

• Someone with the qualifications and the tools to properly execute the recipe.

While Dromey's work is interesting from a technically inclined stakeholder's perspective, it is difficult to see how it could be used at the beginning of the lifecycle to determine user quality needs. Dromey (1995) states that software quality “must be considered in a systematic and structured way, from the tangible to the intangible”. By focusing too much on the tangible, Dromey fails to build a model that is meaningful for stakeholders typically involved at the beginning of the lifecycle. Do end users care about the variable naming convention or module coupling? In most cases, it is doubtful that this question can be answered affirmatively. Therefore, this model is rather unwieldy to specify user quality needs. This does not mean that it cannot be useful later on as a checklist for ensuring that product quality is up to standards. It can definitely be classified as a bottom to top approach to software quality.

Furthermore, as was illustrated at the beginning of this section, this quality model has its roots in the product perspective of quality, to the detriment of other perspectives. Therefore, it fails to qualify as a foundation for Software Quality Engineering according to the established requirements.

ISO/IEC 9126 In 1991, the International Organization for Standardization introduced a standard named ISO/IEC 9126 (1991): Software product evaluation - Quality characteristics and guidelines for their use. This standard aimed to define a quality model for software and a set of guidelines for measuring the characteristics associated with it. ISO/IEC 9126 quickly gained notoriety with IT specialists in Europe as the best way to interpret and measure quality (Bazzana, Anderson & Jokela, 1993). However, Pfleeger (2001) reports some important problems associated with the first release of ISO/IEC 9126:

• There are no guidelines on how to provide an overall assessment of quality.

• There are no indications on how to perform the measurements of the quality characteristics.

• Rather than focusing on the user view of software, the model's characteristics reflect a developer’s view of software.

According to Pfleeger, this first incarnation of ISO/IEC 9126 is not usable as a bottom up approach to quality engineering, and even less usable as a top down approach.

In order to address these concerns, an ISO committee began working on a revision of the standard. The results of this effort are the introduction of a revised version of ISO/IEC 9126 focusing on the quality model, and a new standard, ISO/IEC 14598 (ISO/IEC, 1999a) focusing on software product evaluation. ISO/IEC 14598 addresses Pfleeger's first concern while the revision to ISO/IEC 9126 aims to resolve the second and third issues. ISO/IEC 9126 is now a four part standard:

• ISO/IEC 9126-1 (ISO/IEC, 2001a) defines an updated quality model.

• ISO/IEC 9126-2 (ISO/IEC, 2003a) defines a set of external measures.

• ISO/IEC 9126-3 (ISO/IEC, 2003b) defines a set of internal measures.

• ISO/IEC 9126-4 (ISO/IEC, 2001b) defines a set of quality in use measures.

The new quality model defined in ISO/IEC 9126-1 recognises three aspects of software quality and defines them as follows: (the full definition is given as it is pertinent to the discussion that ensues).

• Quality in Use:

Quality in use is the user's view of the quality of the software product when it is used in a specific environment and a specific context of use. It measures the extent to which users can achieve their goals in a particular environment, rather than measuring the properties of the software itself. (ISO/IEC, 2001a)

• External quality:

External quality is the totality of characteristics of the software product from an external view. It is the quality when the software is executed, which is typically measured and evaluated while testing in a simulated environment with simulated data using external metrics. During testing, most faults should be discovered and eliminated. However, some faults may still remain after testing. As it is difficult to correct the software architecture or other fundamental design aspects of the software, the fundamental design remains unchanged throughout the testing. (ISO/IEC, 2001a)

• Internal Quality:

Internal quality is the totality of characteristics of the software product from an internal view. Internal quality is measured and evaluated against the Internal Quality requirements. Details of software product quality can be improved during code implementation, reviewing and testing, but the fundamental nature of the software product quality represented by the Internal Quality remains unchanged unless redesigned. (ISO/IEC, 2001a)

The Internal and External Quality model is inspired from McCall and Boehm's work. It is a three-layer model composed of quality characteristics, quality subcharacteristics and quality measures. Figure 5 illustrates this model. More than 100 measures of Internal and External Quality are proposed as part of the standard. It is important to note that the measures do not make an exhaustive set, which means that other measures can also be used.

Finally, Quality in Use is modeled in a different way than Internal and External Quality. Figure 6 illustrates the two-layer Quality in Use model composed of characteristics and quality measures. Table XV provides the definition of the characteristics.

Figure 5 : 3-layer model for internal and external quality. Adapted from (ISO/IEC, 2001a)

Figure 6 : Quality in use model. Adapted from (ISO/IEC, 2001a)

Theoretically, Internal Quality, External Quality and Quality in Use are linked together with a predictive mode1. This is illustrated in Figure 7.

Figure 7 : Relationships between the different aspects of quality. Adapted from (ISO/IEC,

2001a)

This prediction relationship states that user quality needs should first be established and specified using the Quality In Use model. From these requirements as well as other sources, External Quality requirements should be established using the External Quality model. Finally, the Internal Quality requirements should be constructed from the External Quality requirements and other sources. Once the requirements are established and software construction is under way, the quality model can be used to predict the overall quality. For example, measurement of Internal Quality can be useful in predicting External Quality. Likewise, measurement of External Quality can be useful in predicting Quality in Use.

The above paragraphs describe the ideal theoretical model that links these three aspects of quality. However, in reality, no model may claim to follow perfectly this prediction mechanism. Although the ISO/IEC 9126 model follows this approach closely, no claims are made as to the real predictive power of the model. While the links between Internal and External Quality seem rather obvious because the models are essentially the same, caution must be exercised. While the name of the characteristics and subcharacteristics are the same, the links between Internal and External Quality must be verified empirically. The same reasoning applies to the links between External Quality and Quality in Use.

The new version of ISO/IEC 9126 is gaining momentum in the industry. Some corporate quality models, for example MITRE's SQAE (Martin & Shaffer, 1996), are beginning a migration from a model based on McCall's and Boehm's research to one based on ISO/IEC 9126 (Côté, Suryn, Martin & Laporte, 2004a; Côté, Suryn, Martin & Laporte, 2004b; Côté, Suryn, Laporte & Martin, 2005). This new version of ISO/IEC 9126 is thus seen as an improvement upon the older quality models.

It is interesting to see how the three aspects of quality defined above can be directly linked to the perspectives of quality that were outlined previously. More specifically:

• ISO/IEC 9126-4, which defines Quality in Use, is directly related to the user and value-based perspectives. The definition of the user perspective of quality states that it is concerned with the appropriateness of a product for a given context of use. Quality in Use is defined as the capability of the software product to enable specified users to achieve specified goals in specified contexts of use. The relationship between the two is clear. Quality in Use and the value based perspective of quality are linked essentially through the Satisfaction characteristic. This characteristic inherently recognises that quality can have a different meaning and/or value for different stakeholders. Satisfaction levels can thus be set according to those levels of perception. This has been demonstrated by the study reported in Siakas and Georgiadou (2005).

• ISO/IEC 9126-3, which defines Internal Quality, and ISO/IEC 9126-2, which defines External Quality, are directly related to both the manufacturing and product perspectives. The definitions of the quality characteristics Functionality and Reliability can be linked with the manufacturing perspective of quality. Reliability, Usability, Efficiency, Maintainability and Portability are all inherent characteristics of the product and a manifestation of the product perspective of quality.

Figure 8 : Relationships between ISO/IEC 9126 and the perspectives of quality

From the review of the different quality models, one might point out that none seem to address the transcendental perspective of quality. One might even ask the following pertinent question: Does ISO/IEC 9126 address the transcendental perspective of quality? Recall that the transcendental perspective of quality relates to quality as something that is recognised but not defined. At this point, the following hypothesis will be made:

As the transcendental perspective of quality cannot be defined, it cannot be explicitly implemented in a software product. However, the transcendental aspect of quality will emerge when a holistic approach to quality engineering is taken.

This model seems to recognise all the perspectives of quality as important contributors to the overall assessment of quality. It takes an incremental approach to software quality that begins with Quality in Use, something that is easy to grasp for non-technical stakeholders, and ends with Internal Quality, something more technically inclined stakeholders will feel more comfortable with. Furthermore, there is a comprehensive set of suggested measures that allow for the assessment of software quality.

ISO/IEC 9126 is thus the only model that fulfills all the stated requirements for a model to be useful as a foundation to Software Quality Engineering. [NEED TO DISCUSS THIS -

Conclusion This paper has defined three requirements that a quality model should meet to serve as a foundation to Software Quality Engineering:

• A quality model should support the 5 different perspectives of quality as defined by Kitchenham and Pfleeger (1996).

• A quality model should be usable from the top to the bottom of the lifecycle as defined by IEEE Std 1061-1998 (IEEE, 1998).

• A quality model should be usable from the bottom to top of the lifecycle as defined by IEEE Std 1061-1998 (IEEE, 1998).

These criteria were applied to four quality models:

• It was found that the models proposed by McCall, Boehm and Dromey focus on the product perspective of quality to the detriment of other perspectives. Furthermore, they are primarily useful in a bottom up approach to quality that is not suitable for Software Quality Engineering.

• ISO/IEC 9126 is the only model that supports all the perspectives of quality (with the exception of the transcendental perspective as noted). Furthermore, its predictive framework clearly supports both the top down and bottom up approaches.

This paper has focused on analysing the semantics of the different models with respect to the stated requirements. In theory, ISO/IEC 9126 seems well suited for Software Quality Engineering. Further research is needed to see if the measures associated with ISO/IEC 9126 make this model usable for Software Quality Engineering in practice.

References

Adey, C. A. & Hill, G. K. (2000). Quality / ISO 9000 as a Marketing Tool, [En ligne]. http://www.smps.org/mrc/articles/0200qualityiso.pdf

Bazzana, G., Anderson, O., & Jokela, T. (1993). ISO 9126 and ISO 9000: Friends or foes? Presented at Software Engineering Standards Symposium.

Biehl, R. E. (2001). Six sigma for Software. IEEE Software, 21(2), 68-70.

Boddie, J. (2000). Do We Ever Really Scale Down?, IEEE Software,17(5), 79-81.

Boehm, B. W., Brown, J. R., Kaspar, J. R., Lipow, M. L. & MacCleod, G. (1978). Characteristics of Software Quality. New York: American Elsevier.

Boehm, B. W., Brown, J. R., Lipow, M. L. (1976). Quantitative Evaluation of Software Quality. Proceedings of the 2nd international conference on Software engineering, San Fransisco, California, United States, 592-605, IEEE Computer Society Press.

Bourque, P., R. Dupuis, A. Abran, J.W. Moore, L.L. Tripp et S. Wolff, (2000) Fundamental Principles of Software Engineering - A Journey, Journal of Systems and Software,

Côté, M.-A., Suryn, W., Martin, R. A., Laporte, C. Y. (2004a). Evolving a Corporate Software Quality Assessment Exercice: A Migration Path to ISO/IEC 9126, Software Quality Professional, 6(3), 4-17.

Côté, M.-A., Suryn, W., Martin, R. A., Laporte, C. Y. (2004b). The analysis of the industrial applicability of software product quality ISO standards: the context of MITRE's Software Quality Assessment exercise, in Proceedings of the 12th International Software Quality Management & INSPIRE Conference (BSI) 2004, Canterbury, Kent, United Kingdom.

Côté, M.-A., Suryn, W., Laporte, C. Y., Martin, R. A. (2005). The Evolution Path for Industrial Software Quality Evaluation Methods Applying ISO/IEC 9126:2001 Quality Model: Example of MITRE's SQAE Method, Software Quality Journal, vol. 13, 17-30.

Crosby, P.B. (1979). Quality is free: The art of making quality certain. New York : McGraw-Hill.

Diaz M. & Sligo, J. (1997). How Software Process Improvement Helped Motorola, IEEE Software, 17(5), 75-81.

Dromey, R. G. (1995). A model for software product quality. IEEE Transactions on Software Engineering 21, 146-162.

Dromey, R. G. (1996). Cornering the Chimera. IEEE Software, 13(1), 33-43.

Eickelman, N. (2003). An Insider's View of CMM Level 5, IEEE Software, 20(4), 79-81.

Haley, T. J. (1996). Software Process Improvement at Raytheon, IEEE Software, 13(6), 33-41.

Georgiadou E. (2003 a) Software Process and Product Improvement, A Historical Perspective, International Journal of Cybernetics, Volume 1, No1, Jan 2003 pp172-197

Georgiadou E.(2003b) GEQUAMO– A Generic, Multilayered, Customisable, Software Quality Model, Volume 11, Number 4 , 313-323 November 2003

Highsmith, J. (2002). Agile Software Development Ecosystems, Addison-Wesley Professional.

IEEE. 1998. Std. 1061-1998 IEEE Standard for a Software Quality Metrics Methodology.

ISO/IEC. 1999a. ISO/IEC 14598-1: Software product evaluation-Part 1 : General overview. Geneva, Switzerland: International Organization for Standardization.

ISO/IEC. 1999b. ISO/IEC 9000:2000 Quality management systems -- Fundamentals and vocabulary . Geneva, Switzerland: International Organization for Standardization.

ISO/IEC. 2000. ISO/IEC 15288: System Life Cycle Processes. Geneva, Switzerland: International Organization for Standardization.

ISO/IEC. 2001a. ISO/IEC 9126-1: Software Engineering-Software product quality-Part 1 : Quality model. Geneva, Switzerland: International Organization for Standardization.

ISO/IEC. 2001b. ISO/IEC DTR 9126-4: Software engineering-Software product quality-Part 4: Quality in use metrics. Geneva, Switzerland: International Organization for Standardization.

ISO/IEC. 2003a. ISO/IEC TR 9126-2: Software Engineering-Software product quality-Part 2 : External metrics. Geneva, Switzerland: International Organization for Standardization.

ISO/IEC. 2003b. ISO/IEC TR 9126-3: Software engineering-Software product quality-Part 3: Internal metrics. Geneva, Switzerland: International Organization for Standardization.

Kitchenham, S. L., Pfleeger (1996). Software Quality: The Elusive Target. IEEE Software,13(1), 12-21.

Laitinen, M. (2000). Scaling Down is Hard to Do, IEEE Software, 17(5), 78-80.

Leffingwell, D. & Widrig, D. (1999). Managing Software Requirements, A Unified Approach. Addison-Wesley Professional.

Martin, R. A. & Shaffer, L. (1996). Providing a framework for effective software quality assessment. Bedford, Mass : MITRE Corporation.

McCall, J. A., Richards, P. K., & Walters, G. F. (1977). Factors in software quality. Griffiths Air Force Base, N.Y. : Rome Air Development Center Air Force Systems Command.

Pfleeger, S. L. (2001). Software Engineering: Theory and practice (2nd ed.). Upper Saddle River, N.J. : Prentice Hall.

Pressman, R. S. (2001). Software Engineering: A practitioner's approach (5th ed.). Boston: McGraw-hill.

Siaka K V., Berki E, Georgiadou E, Sadler C (1997): The Complete Alphabet of Quality Software Systems: Conflicts and Compromises, 7th World Congress on Total Quality&Qualex 97, New Delhi, India, 17-19 February Siaka, K.V., Georgiadou, E. PERFUMES: A Scent of Product Quality Characteristics, SQM 2005, March 2005, UK

Suryn, W. (2003). Course notes SYS861. École de Technologie Supérieure, Montréal.

Voas, J. (2003). Assuring Software Quality Assurance. IEEE Software, 20(3), 48-49.

Toyota Plant in Japan

Production mix in the plant

Assembling with

white gloves.

Easy access to tools

Visual alarm that indicates problems.

Training material used to explain the process and teach people.

5S Chairs below table to save

space.

Device to clean foot. Before entering

the production hall, everybody must clean the foot

In the office area, the chairs stays besides and are used only for resting. People work in stand up position (the aim is genba approach). The furniture were elevate. All of them with wheel to facilitate moving..

5S – identification in chairs has the same identification in table, to indicates the right position of the chairs. It could look like excess, but it’s to show the 5S culture. That’s a

meeting room of company direction.

5S – all material identified and in its place. In the furniture, the wheels to facilitate moving.

5S. A training material folder to organize all procedures and OPLs. Most of them are made manually.

Floor identification. Each colour has a meaning, Very clear floor.

Board filled by hand. Checklist (beside) is also, made manually.

Skill matrix

Even here you find 5S.

Interest: They are already making plans for the car of the future. (for only 1 person)

Quality Criteria Relationship

Maintainability and testability vs. efficiency - (Inverse)

Optimized and compact code is not easy to maintain.

Portability vs. efficiency - (inverse)

The use of optimized software or system utilities will lead to decrease in probability.

Flexibility, reusability and interoperability vs. efficiency - (inverse)

The generally required for a flexible system, the use if interface routines and the modularity desirable

for reusability will all decrease efficiency

Flexibility and reusability vs. integrity - (inverse)

The general flexible data structures required for flexible and reusable software increase the security

and protection problem.

Interoperability vs. integrity - (inverse)

Coupled system allows more avenues of access to more and different users.

Reusability vs. reliability - (inverse)

Reusable software is required to be general:

Maintaining accuracy and error tolerance across all cases is difficult.

Maintainability vs. flexibility - (direct)

Maintainable code arises from code that is well structured.

Maintainability vs. reusability - (direct)

Well structured easily maintainable code is easier to reuse in other programs either as a library of

routines or as code placed directly within another program.

Portability vs. reusability - (direct)

Portable code is likely to be free of environment-specific features.

Correctness vs. efficiency - (neutral)

The correctness of code, i.e. its conformance to specification does not influence its efficiency

103191733-notes-software-quality-management.pdf

Documents