dynamic software tracking
DESCRIPTION
J. Erik Hemdal Robert L. Galen BOSCON 2004 – Track C4. Dynamic Software Tracking. Agenda. Introduction Project Goals Measures from Defect Data Triage and Workflow Conclusions Questions and Answers. Why are we here?. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Dynamic Software Tracking](https://reader036.vdocument.in/reader036/viewer/2022062423/56814385550346895db0021b/html5/thumbnails/1.jpg)
Dynamic Software Tracking
J. Erik HemdalRobert L. Galen
BOSCON 2004 – Track C4
![Page 2: Dynamic Software Tracking](https://reader036.vdocument.in/reader036/viewer/2022062423/56814385550346895db0021b/html5/thumbnails/2.jpg)
Agenda
● Introduction
● Project Goals
● Measures from Defect Data
● Triage and Workflow
● Conclusions
● Questions and Answers
![Page 3: Dynamic Software Tracking](https://reader036.vdocument.in/reader036/viewer/2022062423/56814385550346895db0021b/html5/thumbnails/3.jpg)
Why are we here?
● Discuss the Big Four elements you need to control for a successful project
● Examine a variety of defect-data measures– For what they tell us– For how they can be defeated
● Show how you can manage defects so that you can stay in control of your project and workload – especially in the endgame close to release points.
![Page 4: Dynamic Software Tracking](https://reader036.vdocument.in/reader036/viewer/2022062423/56814385550346895db0021b/html5/thumbnails/4.jpg)
The Big Four Elements
● Have stable, agreed-on goals
● Know the quality level and quality trends of your software
● Keep your team healthy
● Always know how much work remains and how much time is left
![Page 5: Dynamic Software Tracking](https://reader036.vdocument.in/reader036/viewer/2022062423/56814385550346895db0021b/html5/thumbnails/5.jpg)
Understanding & Influencing the Goals
● The formal requirements
● Unstated requirements
● Expectations from customer, sponsor, team
● Negotiation and balance are the key– Features– Specific delivery timing– Level of quality (Hot-button issues)– Essential vs. “nice-to-have” elements
![Page 6: Dynamic Software Tracking](https://reader036.vdocument.in/reader036/viewer/2022062423/56814385550346895db0021b/html5/thumbnails/6.jpg)
Understanding & Influencing the Goals (cont.)
Understanding the overall project plan– Development phasing (building to features, stability and
completeness)– Understanding the methodology (waterfall, incremental,
agile/extreme, RUP)– Test phasing (how many passes, reduction of change,
entrance & exit criteria)
● Key Milestones– Delivered phases– Code freeze / complete– Beta tests– Final release
![Page 7: Dynamic Software Tracking](https://reader036.vdocument.in/reader036/viewer/2022062423/56814385550346895db0021b/html5/thumbnails/7.jpg)
Understanding & Influencing the Goals (cont.)
● Release Criteria is the critical guide for releasing software. Should cover –– Scope– Quality– Time– Team
● Good release criteria should1. Define success2. Learn what’s important for the project3. Be Drafted and thoroughly reviewed4. Be “SMART” (specific, measurable, attainable, realistic
and trackable)5. Assist in gaining consensus6. Serve as a goal / guide throughout triage and release
![Page 8: Dynamic Software Tracking](https://reader036.vdocument.in/reader036/viewer/2022062423/56814385550346895db0021b/html5/thumbnails/8.jpg)
Software Measures from Defect Data
● Good defect tracking provides insight– Into the quality of the product– Into the performance and health of the team– Into the amount of remaining work and rework
● Can be a hot-button issue because of misuse
● New way of thinking: A “Defect” means “something to change”. More things to change means more work.
● Changes stand between you and Done!
![Page 9: Dynamic Software Tracking](https://reader036.vdocument.in/reader036/viewer/2022062423/56814385550346895db0021b/html5/thumbnails/9.jpg)
Common Defect Measures
Cost per defect, predict development repair efficiency
Fix hours / defect
Defect workflow in time, potential bottlenecks
Defects by state counts
Root cause, true cost of qualityPhase –containment of defects
Determine overall maturity - release / ship readiness
# high – medium – low severity defects
Determine overall maturity - release / ship readiness
# found, # fixed, # remaining over time
Time distribution of defects, looking for downward trends
Defect count over time
Testing workflow and productivityDefects / Unit of testing time
Defect density, predictor for future defect trends and overall product quality
Defect counts or defects / KLOC
IllustratesMeasure
![Page 10: Dynamic Software Tracking](https://reader036.vdocument.in/reader036/viewer/2022062423/56814385550346895db0021b/html5/thumbnails/10.jpg)
Defects/KLOC
● Uses: ● Indicate overall quality of
code● Guide to trouble spots
● Depends on:● Definition of a KLOC● Modular architecture with
visible granularity
● Defeat by:● Attacking the KLOC● Blame requirements● Writing longer code
E A C D B AVG0
20
40
60
80
100
120
140
Defects per KLOC by Module
Module ID
Def
ect
Den
sity
![Page 11: Dynamic Software Tracking](https://reader036.vdocument.in/reader036/viewer/2022062423/56814385550346895db0021b/html5/thumbnails/11.jpg)
Defects/Unit of Test Time
● Uses: ● Overall level of test “productivity”● Check test strategies or test “wearout”
● Depends on:● Factors testers usually don't control
● Defeat by:● Untestable/unreachable code● Cutting test time
● Graphical display – similar to Defects/KLOC
![Page 12: Dynamic Software Tracking](https://reader036.vdocument.in/reader036/viewer/2022062423/56814385550346895db0021b/html5/thumbnails/12.jpg)
Defect Counts over Time
● Uses:● Adjust test sequencing and
scheduling● Can indicate significant
problems in test
● Depends on:● Overall coordination
● Defeat by:● Interrupting testing● Finding significant defects● Missing functionality
1 2 3 4 5 6 7 80
5
10
15
20
25
30
35
40
45
New Defects by Week
Week Number
De
fect
Co
un
t
![Page 13: Dynamic Software Tracking](https://reader036.vdocument.in/reader036/viewer/2022062423/56814385550346895db0021b/html5/thumbnails/13.jpg)
# Found, # Fixed and # Open
● Uses: • Suggests when to ship
● Depends on:● Reliable, repeatable test
capability● Change control
● Defeat by:● Curtailing test● Many small changes● Late changes
1 2 3 4 5 6 7 8 9 10 11 120
20
40
60
80
100
120
Found vs. Fixed Defects
New
Closed
Cum-New
Cum-Closed
Week Number
De
fect
Co
un
t
![Page 14: Dynamic Software Tracking](https://reader036.vdocument.in/reader036/viewer/2022062423/56814385550346895db0021b/html5/thumbnails/14.jpg)
# High/Med/Low Severity Defects
● Use:● Indicates overall readiness of a
product
● Depends on:● Proper triage● Effective criteria
● Defeat by:● Management fiat● Subtle negotiation● Peer pressure
1 2 3 4 50
10
20
30
40
50
60
70
Defect Counts by Class
Class CClass BClass A
Week Number
De
fect
Co
un
t
![Page 15: Dynamic Software Tracking](https://reader036.vdocument.in/reader036/viewer/2022062423/56814385550346895db0021b/html5/thumbnails/15.jpg)
Phase-Containment of Defects
● Uses:● Identify process problems● Justify process improvement
● Depends on:● A defined phase-gate
process
● Defeat by:● Lack of commitment to the
process● Lack of time to analyze raw
defect data
Reqts Design Code Test Int'gn0
5
10
15
20
25
30
35
40
45
When Found vs. When Caused
Reqts.DesignCodeTestIntegration
Phase When Found
Def
ect C
ount
per
Pha
se W
hen
Cau
sed
![Page 16: Dynamic Software Tracking](https://reader036.vdocument.in/reader036/viewer/2022062423/56814385550346895db0021b/html5/thumbnails/16.jpg)
Average Fix Hours per Defect
● Uses:● Illustrate the cost of poor quality● Provide data for repair estimates
● Depends on:● Ability to capture data● High trust culture and within the development team
● Defeat by:● Misusing the data
● Similar measures possible for build, test, review time
![Page 17: Dynamic Software Tracking](https://reader036.vdocument.in/reader036/viewer/2022062423/56814385550346895db0021b/html5/thumbnails/17.jpg)
Defects By Status Histogram
● Uses:• Show team bottlenecks• Gauge project status and
product maturity
● Depends on:• Consistent and reliable state
update activity
● Defeat by:• Gamesmanship about defect
status
1 2 3 40
2
4
6
8
10
12
14
16
18
20
Defects by State
NewAssignedOpenClosed
Week Number
Co
un
t of D
efe
cts
![Page 18: Dynamic Software Tracking](https://reader036.vdocument.in/reader036/viewer/2022062423/56814385550346895db0021b/html5/thumbnails/18.jpg)
Dynamic Tracking
● Defect Triage
● Workflow Management
● Defect Packaging
![Page 19: Dynamic Software Tracking](https://reader036.vdocument.in/reader036/viewer/2022062423/56814385550346895db0021b/html5/thumbnails/19.jpg)
Defect Triage
● Manage three elements– Severity How serious is this defect?– Priority How soon should this be fixed?– Impact What is the effect on team and customer?
● Define a triage team and procedure early– Include QA, developers, project office, support– Set up the “drill”– Update defect reports with triage decisions – who, what,
why
● Map triage policy to the overall release plan
![Page 20: Dynamic Software Tracking](https://reader036.vdocument.in/reader036/viewer/2022062423/56814385550346895db0021b/html5/thumbnails/20.jpg)
Workflow Management
● Don’t just let work happen based on priority or chance - rather, proactively manage repairs!
● Pre analyze scope, level of difficulty and potential impact – prior to assigning major repairs
● Each engineer has an assigned “work queue” or a to-do list that is managed from the defect tracking system
● Leverage the DTS for reporting and work flow management
![Page 21: Dynamic Software Tracking](https://reader036.vdocument.in/reader036/viewer/2022062423/56814385550346895db0021b/html5/thumbnails/21.jpg)
Workflow Management (cont.)
● Create guidelines per engineer 5-10 work items– 1 or 2 high priority defects– 3-4 moderate priority defects– 1-2 defects to investigate/analyze
● Good idea: Focus on your “Top 10” issues, then stop and analyze
● Reallocate judiciously; avoid churning
● Create engineer profiles to help understand strengths and skills
![Page 22: Dynamic Software Tracking](https://reader036.vdocument.in/reader036/viewer/2022062423/56814385550346895db0021b/html5/thumbnails/22.jpg)
Defect Packaging
● Schedule a series of code drops or “packages” for testing– First release– Updates; new functions– “Critical fix” package– Release candidate– Final release
● Each should have a primary goal
● Testing involves high fixed costs; packages give you the most value (fixed defects) in each cycle
● Don't waste the “power of the package”
![Page 23: Dynamic Software Tracking](https://reader036.vdocument.in/reader036/viewer/2022062423/56814385550346895db0021b/html5/thumbnails/23.jpg)
Tying Things Together
● Supporting the Goals
● Maintaining the Level of Quality
● Watching the Health of the Team
● Knowing What's Left to be Done
![Page 24: Dynamic Software Tracking](https://reader036.vdocument.in/reader036/viewer/2022062423/56814385550346895db0021b/html5/thumbnails/24.jpg)
Supporting the Goals
● Effective defect triage helps you to maintain your focus on the goals of the project.
● Triage directs your team's effort to the most important work, balancing the demands of all stakeholders
● Defect data can flag roadblocks and slowdowns before they derail the project
![Page 25: Dynamic Software Tracking](https://reader036.vdocument.in/reader036/viewer/2022062423/56814385550346895db0021b/html5/thumbnails/25.jpg)
Maintaining the Level of Quality
● Defect measures and reports can give you a direct reading on the state of your project
– How many defects there are
– How defects affect the product and project
– Where extra design, test, or review effort is warranted
– When it's appropriate to release (or Not to release)
![Page 26: Dynamic Software Tracking](https://reader036.vdocument.in/reader036/viewer/2022062423/56814385550346895db0021b/html5/thumbnails/26.jpg)
Watching the Health of the Team
● Defect data helps here by -
– Signaling overloaded and frustrated team members
– Capturing and documenting key decisions and key changes
– Communicating changes that WILL NOT be made
– Preventing unnecessary work
![Page 27: Dynamic Software Tracking](https://reader036.vdocument.in/reader036/viewer/2022062423/56814385550346895db0021b/html5/thumbnails/27.jpg)
Knowing What's Left to be Done
● Defect tracking data helps here by indicating -
– How many items must be changed
– How long will these changes take
– How much will they cost
– What will be unfinished when we stop
![Page 28: Dynamic Software Tracking](https://reader036.vdocument.in/reader036/viewer/2022062423/56814385550346895db0021b/html5/thumbnails/28.jpg)
Questions?
![Page 29: Dynamic Software Tracking](https://reader036.vdocument.in/reader036/viewer/2022062423/56814385550346895db0021b/html5/thumbnails/29.jpg)
References
● A. Allison, "Meaningful Metrics" The Software Testing & Quality Engineering Magazine (May/Jun 2001)
● R. Black, Managing the Testing Process, 2’nd Edition, New York, NY: John Wiley & Sons Inc., 2002
● R. Galen, “Mastering the Software Project Endgame”, New York, NY: Dorset House, 2004 - forthcoming
● C. Necaise, "Managing the Endgame." The Software Testing & Quality Engineering Magazine (Jan/Feb 2000)
● J. Rothman, "Release Criteria: Is This Software Done?" The Software Testing & Quality Engineering Magazine (Mar/Apr 2002)
● J. Rothman, "Managing Projects: Release Criteria, or Is it Ready to Ship." Newsletter Vol. 1, No. 2 (1999)
● B. Schoor, “Managing Quality During the Endgame” Presentation from schoorconsulting.com website and PSQT/PSTT 2002 North conference – www.softdim.com (2002)
![Page 30: Dynamic Software Tracking](https://reader036.vdocument.in/reader036/viewer/2022062423/56814385550346895db0021b/html5/thumbnails/30.jpg)
Contact Information
Erik HemdalIndependent Consultantand Instructor
Bob GalenEMC2 Corp. & RGalen Consulting
Group, LLC