Download - Dynamic Software Tracking
Dynamic Software Tracking
J. Erik HemdalRobert L. Galen
BOSCON 2004 – Track C4
Agenda
● Introduction
● Project Goals
● Measures from Defect Data
● Triage and Workflow
● Conclusions
● Questions and Answers
Why are we here?
● Discuss the Big Four elements you need to control for a successful project
● Examine a variety of defect-data measures– For what they tell us– For how they can be defeated
● Show how you can manage defects so that you can stay in control of your project and workload – especially in the endgame close to release points.
The Big Four Elements
● Have stable, agreed-on goals
● Know the quality level and quality trends of your software
● Keep your team healthy
● Always know how much work remains and how much time is left
Understanding & Influencing the Goals
● The formal requirements
● Unstated requirements
● Expectations from customer, sponsor, team
● Negotiation and balance are the key– Features– Specific delivery timing– Level of quality (Hot-button issues)– Essential vs. “nice-to-have” elements
Understanding & Influencing the Goals (cont.)
Understanding the overall project plan– Development phasing (building to features, stability and
completeness)– Understanding the methodology (waterfall, incremental,
agile/extreme, RUP)– Test phasing (how many passes, reduction of change,
entrance & exit criteria)
● Key Milestones– Delivered phases– Code freeze / complete– Beta tests– Final release
Understanding & Influencing the Goals (cont.)
● Release Criteria is the critical guide for releasing software. Should cover –– Scope– Quality– Time– Team
● Good release criteria should1. Define success2. Learn what’s important for the project3. Be Drafted and thoroughly reviewed4. Be “SMART” (specific, measurable, attainable, realistic
and trackable)5. Assist in gaining consensus6. Serve as a goal / guide throughout triage and release
Software Measures from Defect Data
● Good defect tracking provides insight– Into the quality of the product– Into the performance and health of the team– Into the amount of remaining work and rework
● Can be a hot-button issue because of misuse
● New way of thinking: A “Defect” means “something to change”. More things to change means more work.
● Changes stand between you and Done!
Common Defect Measures
Cost per defect, predict development repair efficiency
Fix hours / defect
Defect workflow in time, potential bottlenecks
Defects by state counts
Root cause, true cost of qualityPhase –containment of defects
Determine overall maturity - release / ship readiness
# high – medium – low severity defects
Determine overall maturity - release / ship readiness
# found, # fixed, # remaining over time
Time distribution of defects, looking for downward trends
Defect count over time
Testing workflow and productivityDefects / Unit of testing time
Defect density, predictor for future defect trends and overall product quality
Defect counts or defects / KLOC
IllustratesMeasure
Defects/KLOC
● Uses: ● Indicate overall quality of
code● Guide to trouble spots
● Depends on:● Definition of a KLOC● Modular architecture with
visible granularity
● Defeat by:● Attacking the KLOC● Blame requirements● Writing longer code
E A C D B AVG0
20
40
60
80
100
120
140
Defects per KLOC by Module
Module ID
Def
ect
Den
sity
Defects/Unit of Test Time
● Uses: ● Overall level of test “productivity”● Check test strategies or test “wearout”
● Depends on:● Factors testers usually don't control
● Defeat by:● Untestable/unreachable code● Cutting test time
● Graphical display – similar to Defects/KLOC
Defect Counts over Time
● Uses:● Adjust test sequencing and
scheduling● Can indicate significant
problems in test
● Depends on:● Overall coordination
● Defeat by:● Interrupting testing● Finding significant defects● Missing functionality
1 2 3 4 5 6 7 80
5
10
15
20
25
30
35
40
45
New Defects by Week
Week Number
De
fect
Co
un
t
# Found, # Fixed and # Open
● Uses: • Suggests when to ship
● Depends on:● Reliable, repeatable test
capability● Change control
● Defeat by:● Curtailing test● Many small changes● Late changes
1 2 3 4 5 6 7 8 9 10 11 120
20
40
60
80
100
120
Found vs. Fixed Defects
New
Closed
Cum-New
Cum-Closed
Week Number
De
fect
Co
un
t
# High/Med/Low Severity Defects
● Use:● Indicates overall readiness of a
product
● Depends on:● Proper triage● Effective criteria
● Defeat by:● Management fiat● Subtle negotiation● Peer pressure
1 2 3 4 50
10
20
30
40
50
60
70
Defect Counts by Class
Class CClass BClass A
Week Number
De
fect
Co
un
t
Phase-Containment of Defects
● Uses:● Identify process problems● Justify process improvement
● Depends on:● A defined phase-gate
process
● Defeat by:● Lack of commitment to the
process● Lack of time to analyze raw
defect data
Reqts Design Code Test Int'gn0
5
10
15
20
25
30
35
40
45
When Found vs. When Caused
Reqts.DesignCodeTestIntegration
Phase When Found
Def
ect C
ount
per
Pha
se W
hen
Cau
sed
Average Fix Hours per Defect
● Uses:● Illustrate the cost of poor quality● Provide data for repair estimates
● Depends on:● Ability to capture data● High trust culture and within the development team
● Defeat by:● Misusing the data
● Similar measures possible for build, test, review time
Defects By Status Histogram
● Uses:• Show team bottlenecks• Gauge project status and
product maturity
● Depends on:• Consistent and reliable state
update activity
● Defeat by:• Gamesmanship about defect
status
1 2 3 40
2
4
6
8
10
12
14
16
18
20
Defects by State
NewAssignedOpenClosed
Week Number
Co
un
t of D
efe
cts
Dynamic Tracking
● Defect Triage
● Workflow Management
● Defect Packaging
Defect Triage
● Manage three elements– Severity How serious is this defect?– Priority How soon should this be fixed?– Impact What is the effect on team and customer?
● Define a triage team and procedure early– Include QA, developers, project office, support– Set up the “drill”– Update defect reports with triage decisions – who, what,
why
● Map triage policy to the overall release plan
Workflow Management
● Don’t just let work happen based on priority or chance - rather, proactively manage repairs!
● Pre analyze scope, level of difficulty and potential impact – prior to assigning major repairs
● Each engineer has an assigned “work queue” or a to-do list that is managed from the defect tracking system
● Leverage the DTS for reporting and work flow management
Workflow Management (cont.)
● Create guidelines per engineer 5-10 work items– 1 or 2 high priority defects– 3-4 moderate priority defects– 1-2 defects to investigate/analyze
● Good idea: Focus on your “Top 10” issues, then stop and analyze
● Reallocate judiciously; avoid churning
● Create engineer profiles to help understand strengths and skills
Defect Packaging
● Schedule a series of code drops or “packages” for testing– First release– Updates; new functions– “Critical fix” package– Release candidate– Final release
● Each should have a primary goal
● Testing involves high fixed costs; packages give you the most value (fixed defects) in each cycle
● Don't waste the “power of the package”
Tying Things Together
● Supporting the Goals
● Maintaining the Level of Quality
● Watching the Health of the Team
● Knowing What's Left to be Done
Supporting the Goals
● Effective defect triage helps you to maintain your focus on the goals of the project.
● Triage directs your team's effort to the most important work, balancing the demands of all stakeholders
● Defect data can flag roadblocks and slowdowns before they derail the project
Maintaining the Level of Quality
● Defect measures and reports can give you a direct reading on the state of your project
– How many defects there are
– How defects affect the product and project
– Where extra design, test, or review effort is warranted
– When it's appropriate to release (or Not to release)
Watching the Health of the Team
● Defect data helps here by -
– Signaling overloaded and frustrated team members
– Capturing and documenting key decisions and key changes
– Communicating changes that WILL NOT be made
– Preventing unnecessary work
Knowing What's Left to be Done
● Defect tracking data helps here by indicating -
– How many items must be changed
– How long will these changes take
– How much will they cost
– What will be unfinished when we stop
Questions?
References
● A. Allison, "Meaningful Metrics" The Software Testing & Quality Engineering Magazine (May/Jun 2001)
● R. Black, Managing the Testing Process, 2’nd Edition, New York, NY: John Wiley & Sons Inc., 2002
● R. Galen, “Mastering the Software Project Endgame”, New York, NY: Dorset House, 2004 - forthcoming
● C. Necaise, "Managing the Endgame." The Software Testing & Quality Engineering Magazine (Jan/Feb 2000)
● J. Rothman, "Release Criteria: Is This Software Done?" The Software Testing & Quality Engineering Magazine (Mar/Apr 2002)
● J. Rothman, "Managing Projects: Release Criteria, or Is it Ready to Ship." Newsletter Vol. 1, No. 2 (1999)
● B. Schoor, “Managing Quality During the Endgame” Presentation from schoorconsulting.com website and PSQT/PSTT 2002 North conference – www.softdim.com (2002)
Contact Information
Erik HemdalIndependent Consultantand Instructor
Bob GalenEMC2 Corp. & RGalen Consulting
Group, LLC