teaching software quality assurance in an undergraduate...
TRANSCRIPT
6/12/2007
1
Department of Software and IT Engineering
Teaching Software Quality Assurance in an Undergraduate Software Engineering Program
Claude Y Laporte, Alain April, Khaled Bencherif
Presented by Claude Y LaporteProfessor
Department of Software and IT EngineeringÉcole de technologie supérieure, Québec, Canada.
Fifth International Conference on INDUSTRIAL AUTOMATION
June 11-13th, 2007
2Department of Software and IT Engineering
Content• Introduction• Business Rationale for Software Quality Assurance• Cost of Poor Software Quality• Components of Software Quality Assurance Course• Conclusion
6/12/2007
2
3Department of Software and IT Engineering
Over 4500 students, 125 professors, 25 general senior lecturers and 200 lecturers.
2200 paid industrial internships in over 900 companies (2004).
Undergraduate Programs• Software Engineering• IT Engineering• Construction Engineering• Production Engineering • Electrical Engineering• Mechanical Engineering • Logistics and Operations Engineering
• Graduate Programs• Software Engineering• Information Technology• Other programs
École de technologie supérieure
• 700 students• 17 Professors in the department have a
mean industrial experience of 15 years.
www.etsmtl.ca
175 students.
4Department of Software and IT Engineering
Undergraduate Software Engineering Program
Quality Control and Metrics
Software DesignSoftware Architecture
Software Quality assurance
Analysis and Design of Telecommunications Software
Systems SecurityUser Interface Analysis and Design
Data Structures and Algorithms
Formal and Semi-Formal Languages
Advanced Object-Oriented Programming
Software Requirements
Capstone Project
Design of Real-Time Computer Systems
Telecommunication NetworksAlgorithms AnalysisInteractive Multimodal Systems
Introduction to Distributed SystemsDistributed Object-Oriented Architecture
Principles of Operating Systems and Systems Programming
High Performance DatabasesCompilation TechniquesIntroduction to Parallel ProcessingIntroduction to Databases
6/12/2007
3
5Department of Software and IT Engineering
Project Cost
Cost of Performance•Generation of plans•SW Development
Cost of Quality
Cost of Conformance Cost of NonConformance•Fixing defects•Updating code•Engineering changes•Re testing (time, equipment)•Re-reviews•Updating documentation•Patches to delivered code•External failures
Appraisal Costs•Reviews•Inspections•Testing•IV&V•Audits
Prevention Costs•Training•Methodologies•Tools•Data gatheringand Analysis
•Improvement projects•Fault analysis
6Department of Software and IT Engineering
Cost of Non Conformanceand Quality of Product
• Quiz– For your projects, what is the Cost on Non Conformance
• As a percentage (%) of the Project Cost ?
– What is the quality of your product (Defects/Unit of size) ?• e.g. Number of Defects /Thousand Line of Code (KLOC)
Development Integration and System Test (I&ST)
Software state of practice: “test in” qualitySTILL 60 - 80 % of effort and cost !
Standish Group, www.standishgroup.com, 1999
6/12/2007
4
7Department of Software and IT Engineering
0
5
10
15
20
25
30
35
40
45
Cost of Non Conformance (Rework)
Maturity Level
Appraisal & Prevention Costs
Start of Initiative
1 2 341988 1990
1992
41%
18%
11%
6%
% of TotalProject Cost
Cost of Quality
19961994
5%
Haley, T., ‘Software Process Improvement at Raytheon’,IEEE Software, November 1996.
8Department of Software and IT Engineering
Percentage of Rework on Projects
Note: Rework = Waste or Scrap
TRW 30% (Boehm ,1987)
NASA-SEL 40% (McGarry, 1987)
Hewlett-Packard 33% (Duncker, 1992)
Raytheon 41% (Dion, 1993)
6/12/2007
5
9Department of Software and IT Engineering
• What Do Management and Organisation Want ?• Predictable Content and Quality• Predictable Schedule• Predictable Cost
• Benefits of SQA to Management and Organisation– Provide with objective insight into processes and work products
(adapted from CMMI)– Provide proven industrial processes and practices to managers and
engineers– Provide real-time hard data to help decision making
• Completion of products - 90% completion syndrome• Measure of quality of products
– Provide information of difficulties and potential improvements to software processes and products.
Business Rationale for ImplementingSoftware Quality Assurance (SQA)
10Department of Software and IT Engineering
Typical Software Defect Profile
Requirements
Design
Code
Unit Test
Integration Test
System Test
Fielded defects
per KLOC
Rework Costs
10
20
50
100
40
20
KLOC = Thousand Lines of Code
Wheeler, D., Brykcynski, B., Meeson, R, Software Inspection – An Industry Best Practice, Institute of Electrical and Electronics Engineers (IEEE), 1996, p 10
6/12/2007
6
11Department of Software and IT Engineering
Insp.
Insp.
Insp.
Requirements
Design
Code
Unit Test
Integration Test
System Test
ReducedNumber of
Fielded Defects
Reduced Rework Costs
1 (10)
3 (20)
7 (50)
15 (100)
10 (40)
5 (20)
Typical Software Defect Profile
KLOC = Thousand Lines of Code
Wheeler, D., Brykcynski, B., Meeson, R, Software Inspection – An Industry Best Practice, Institute of Electrical and Electronics Engineers (IEEE), 1996, p 10
12Department of Software and IT Engineering
• Objectives– Characterize the content of the software engineering
discipline,– Promote a consistent view of software engineering worldwide,– Set the boundary of software engineering with respect to other
disciplines,– Provide a foundation for curriculum development and
individual licensing material
Guide to the Software Engineering Body of Knowledge (SWEBOK)
www.swebok.org
ISO/IEC TR 19559
6/12/2007
7
13Department of Software and IT Engineering
Software Engineering Knowledge Areas
1. Software Requirements2. Software Design3. Software Construction4. Software Quality (static techniques) 5. Software Testing6. Software Maintenance7. Software Configuration Management8. Software Eng. Management9. Software Eng. Tools & Methods10. Software Engineering Process
• Computer Engineering • Computer Science• Mathematics• Project Management• Management• Quality Management • Software Ergonomics• Systems Engineering
Related Disciplines
14Department of Software and IT Engineering
ISO/IEC Certification of Software Engineering Professionals
• To respond to the need for portability of software engineering professional certifications,
• To facilitate the exchange of professionals between different countries,
• To provide the processes needed to establish, administer, and maintain a certification scheme,
• Certification body will administer:– The certification activity, including all procedures and activities
intended to demonstrate the qualifications of software engineering professionals.
• SWEBOK will serve as a reference model for software engineering professional certifications.
ISO/IEC TR 19559
6/12/2007
8
15Department of Software and IT Engineering
Components of Software Quality Assurance Course
MEASUREMENTand DEFECTS
CODE of ETHICS
REVIEWSand
AUDITS
COST OF QUALITY
QUALITY and
CULTURE
STAFFTOOLS
CONTRACTSand
SUPPLIERS
VERIFICATIONand
VALIDATION
RISKSand SQA
CONFIGURATIONMANAGEMENT
PROCESSES,ACTIVITIES,
TASKS
STANDARDSand
MODELS
QUALITY ASSURANCE
PLAN
Note: Tests are covered in LOG 510
QUALITY FACTORS(ISO 9126)
16Department of Software and IT Engineering
Software Engineering Standards and SQA• Importance of Standards in Software Engineering
– No law of physics in software engineering !– Software Engineering Standards can be used in Court.
• Standards covered in class and laboratory– ISO/IEC
• ISO/IEC 9126-Software Engineering-Product Quality.• ISO/IEC 90003-Software Engineering: Guidelines for the
application of ISO 9001 to computer software– IEEE
• IEEE 730 - IEEE Standard for Software Quality Assurance Plans • IEEE 828 - IEEE Standard for Software Configuration Management
Plans• IEEE 1028- IEEE Standard for Software Reviews *• IEEE 1012 - Software Verification and Validation• IEEE 12207 - Software Lifecycle Processes.
– De Facto• Capability Maturity Model Integration (CMMI), Software
Engineering Institute (SEI).
6/12/2007
9
17Department of Software and IT Engineering
Types of Reviews
Audits
Walk-throughInspectionUnderstanding
DiscussFind Anomalies
EvaluateCompliance
FindAnomalies
• IEEE 1028 - Standard for Software Reviews
18Department of Software and IT Engineering
SQA Laboratory Topics• Code of Ethics - Robot killer Case Study• Team Project
• Plan, design and program new features, manage configuration, defects and changes, perform reviews and evaluate software quality.• Draft a quality assurance plan (IEEE-730), a project plan and a software
requirement specifications document (SRS).• Carry out a review (walk-through) (IEEE 1028)
• Perform Software Configuration Management and Traceability (IEEE 1012).• Write a CM procedure• Use Open Source tool (CVS)• Update effort estimation• Carry out a review (walk-through)
• Design, program and test.• Design and program additional features in existing software• Update effort estimation• Carry out a review (walk-through)
• Perform problem/change management and defect management• Log anomalies using Open Source tool – Issue tracking tool (Bugzilla) • Carry out a review (inspection) (IEEE 1028)
• Assess Product Quality.• Finalize/update quality assurance plan (IEEE 730)
6/12/2007
10
19Department of Software and IT Engineering
Conclusion• Our students learn how to engineer a software product and to implement
quality assurance practices and tools.
• Many changes have been made to the SQA course since its initial development in 2001.– Following the improvements, this course has scored 4.2/5.0
• An improvement of 0.6 on the previous format. – Adding the utilisation of tools has made the most significant difference
in the scores.
• The current SQA course lectures and laboratory sessions provide a solid foundation for future software engineers.
• SQA is still perceived as a low priority by most Small and Medium Enterprises (SMEs) and Very Small Enterprises (VSEs). – They have not calculated their cost of poor quality yet !
• The profession of software engineering is still young, and we know that Rome was not built in a day.
Department of Software and IT Engineering
Back-up Slides
6/12/2007
11
21Department of Software and IT Engineering
SQA Course Topics
Risk and Quality: Identification of risks related to SQA13
Supplier SQA: The activities relating to the management of the subcontractors with regard to the contribution of suppliers in providing a quality product.
12
Measurement: Product and Process Measures11
Software Configuration Management: Management of Change10
Verification and Validation: V&V practices according to the IEEE 1012 standard9
Software Quality Assurance Plan : According to standard IEEE 7308
Software Inspection: The inspection process and the cost and benefit.7
Software Reviews: Types of review, as defined by the IEEE 1028 standard6
Software Life Cycle Process: The SQA process and activities5
Quality Model: Software product quality requirements using the ISO/IEC 9126 quality model.4
Standards and Models: Key standards: IEEE 12207 and ISO/IEC 90003. Models such as the Capability Maturity Model®[1] Integration SM[2] (CMMI®)
3
Code of Ethics: IEEE/ACM Code of Ethics for software engineers.2
Introduction: Introduction to software quality, definitions and the cost of quality. 1
[1] Capability Maturity Model Integration is a service mark of Carnegie Mellon University.[2] CMMI is registered with the US Patents and Trademarks Office by Carnegie Mellon University.
22Department of Software and IT Engineering
Resources Available to Students
Contain standards, case studies, papers and some examples of plansCourse notes
CVS: Software Configuration Management repositoryBugzilla: Tracking and reporting bugs and change requestsCheckStyle: To verify the source code conformance to the programming
language standardEclipse: Development environment with a multitude of plug-ins.Logiscope: Product quality measurement
Tools
Galin, D., “Software Quality Assurance – From Theory to Implementation”Textbook
All the information concerning the course is available online. The course Web site contains the professor’s teaching materials
Web siteISO and IEEE standards Standards
The Capability and Maturity Model Integration is used during the lectures to illustrate SQA practices, such as peer review.
CMMI
Online library of all IEEE standards, papers and magazines.IEEExploreSoftware Engineering Body of KnowledgeSWEBOK
6/12/2007
12
23Department of Software and IT Engineering
SQA Laboratory Topics
Team project - Finalize/update quality assurance plan. 1) Finalize the plan according to IEEE standard 730; 2) Inspect the plan; 3) Carry out an evaluation of team members (peer evaluation); 4) Carry out a project postmortem; and 5) Use the effort estimation and project tracking data to:
Analyze the variations and the costs of quality, explain the variations, explain how, in similar projects, the tasks could be carried out to minimize the variations and the costs of an absence of quality (rework).
7
Team project - Product Quality Assessment. 1) Assess source code conformance to customer standards using CheckStyle and software complexity/quality using Logiscope.
6
Team project - Problem/change and defect management and inspection. 1) Complete section 4.8 of the IEEE 730 standard; 2) Document the change/problem and defect management procedure using the ETVX notation; 3) Read the documentation, configure and test the Bugzilla tool; 4) Print the statistics and management reports from Bugzilla; 5) Carry out a walkthrough; and 6) Update the project effort.
5
Team project - Programming and test. 1) Program additional features into existing software; 2) Test the software produced; 3) Update information on the IBM RequisitePro/Excel, Bugzilla and CVS tools; and 4) Update the project effort.
4
Team project - Software Configuration Management (SCM) and Traceability. 1) Implement the SCM plan using IEEE 730 and IEEE 828; 2) Document the SCM procedure using the Entry-Task-Verification-Exit notation (ETVX) (Rad85); 3) Update the project effort estimation; 4) Read documentation, and configure and test CVS tool for the following roles: system administrator, the individual responsible for configuration management and users; and 5) Carry out a walkthrough of the document produced.
3
Team project – Draft a project plan and a software requirement specifications document (SRS). 1) Software quality plan (IEEE standard 730); 2) Take into consideration the customer’s local Java programming rules; 3) Develop project plan and estimates by Cost of Quality items; 4) Use the IBM Rational SRS template (functional and nonfunctional requirements (use ISO/IEC 9126 for the nonfunctional requirements); 5) Carry out a walkthrough of the documents produced; and 6) Carry out a traceability analysis with the IBM Rational RequisitePro tool or Excel.
2
Code of Ethics: Robot killer case study; Find the violated clauses of the Code; and Determine the responsibilities.
1
24Department of Software and IT Engineering
Software Quality Knowledge Area
Software EngineeringCulture and Ethics
Value and Costsof Quality
Quality Improvement
Verification andValidation
Software QualityAssurance
Software Quality
Software QualityFundamentals
Software QualityManagementProcesses
PracticalConsiderations
Models andQualityCharacteristics
Reviews andAudits
Application QualityRequirements
DefectCharacterization
Software QualityMeasurement
Software QualityManagementTechniques
Note: Tests are covered in LOG 510
6/12/2007
13
25Department of Software and IT Engineering
Quality of Software
• Effort to find and fix– Finding and fixing a software problem after delivery is
often 100 times more expensive than finding and fixing it during the requirements and design phase.
• Amount of avoidable rework– Current software projects spend about 40 to 50 percent of
their effort on avoidable rework.• Software quality at delivery
– About 40 to 50 percent of user programs contain nontrivial defects
26Department of Software and IT Engineering
ISO Software & Systems Engineering Standards
SC7 Secretariat ReportMoscow, May 2007.
0
10
20
30
40
50
60
70
80
90
100
1987 1989 1991 1993 1995 1997 1999 2001 2003 2005 2007
Standards Published
Standards Maintained
6/12/2007
14
27Department of Software and IT Engineering
Inspection Process
110Conduct. KickoffMeeting
100PlanInspection
.. . . .
Source
120ConductDocumentChecking
130ConductLoggingMeeting
140EditDocument .
Rules
Checklists
Change requests
Productdocument
Processimprovements
150CompleteFollow-upExit &Release
Source: Holland, D., ‘Document Inspection: An Agent of Change’, November 1997.
InspectionRequest
28Department of Software and IT Engineering
Inspection Effectiveness
Less than 50%Level 1
50% - 65%Level 2
65% - 75%Level 3
75% - 90%Level 4
90% +Level 5
Defect RemovalEffectiveness
CMM Maturity Level
Radice, R., ‘High Quality Low Cost Software Inspections’, 2002.
6/12/2007
15
29Department of Software and IT Engineering
0
20
40
60
80
100
120
140
160
180
200
1 2 3 41988 19901992
19961994
170 %Increase
Productivity Increase
Reduced rework80% reduction of code fixing during integration50% reduction of cost of retesting
30Department of Software and IT Engineering
Why do software projects fail so often?• Among the most common factors:
– Unrealistic or unarticulated project goals– Inaccurate estimates of needed resources– Badly defined system requirements– Poor reporting of the project’s status– Unmanaged risks– Poor communication among customers, developers, and users– Use of immature technology– Inability to handle the project’s complexity– Sloppy development practices– Poor project management– Stakeholder politics– Commercial pressures
Charrette, R., Why Software Fails, IEEE Spectrum, Sep 2005.
6/12/2007
16
31Department of Software and IT Engineering
Defect Density
Jan Jul Jan July Jan July Jan July Jan July Jan July Jan July
0
2
4
6
8
10
12
14
16
18
1 23 41988
19901992
19951994
Defect Density = Defects/Thousand DSI
32Department of Software and IT Engineering
Cost of Quality
0
10
20
30
40
50
60
70
Per
cent
age
of to
tal p
roje
ct c
ost
Year
CMM level 3Start of intiative CMM level 1
TCoSQ
PreventionRework
Appraisal
Cost ofConformance
Rework
87 88 89 90 91 92 93 94 95 96