test metrics
DESCRIPTION
Deploying Test MetricTRANSCRIPT
The Metrics Lifecycle
Questions before starting to use metrics are:* Who will use these metrics?* What behavior are you trying to promote with these metrics?* What information is important to know across the project/enterprise?* What requires increased visibility or transparency?
Inception6/2010
Interim 1/2011
Comprehensive6/2011
Test Planning
Test Creation
Test Execution
Defect Management
Configuration Management
Cost – Project, BA, Dev, QA
Test Scripts Built
Pass, Failed, Risk*
# Scripts, Timeline
Status, Severity
Check Points, Test Plan,
Traceability Matrix
Trend, Block, No Runs
BA/Dev/Bus Accepted/Rejected
Test Automation
Discovery Rate, Removal Rate,
Trend, QA Impact (time, testers)
Uptime, Domain Version verification,
Release Notes
Removal Cost, Root Cause
BA/Dev/Bus Accepted/Rejected
Build Cost
# Scripts, Timeline
Pass, Failed, Usage
Artifact Lockdown
Sample Metrics
Stability Report
Automation Save
SCORECARD
Process Overview Review Data Points with teams*.
If Data are not captured, process is defined.
Data is saved to a shared space
On QC Data identity is defined (QC Project, Project ID, Date stamp, etc.)
Teams or process create sheets.
QM collects all reports, consolidates, reviews with teams and sends result to PGM.
Automated Process (QTP, Excel Marco, etc.) to upload sheets into Master sheet.
Macro/QM consolidates information
Naming Standards - Sheet Name (Team-Date-Test Metrics- Project)
Shared Space Location Name and Access
Initial
Not On QC
How Manual
Summary
Detail
Project View Execution/Defect Overview Report
Test Execution AE/PE < 10% AE/PE < 20% AE/PE > 30% ▲ ▬▬ ▼ Trend
Defects
defect/test executed <
10%defect/test
executed < 20%
defect/test executed >
30% Not In QC
Overall ProgressDF % + TEP % + Risk < 10%
DF % + TEP % + Risk < 20%
DF % + TEP % + Risk > 30% In QC
Sheet Key
FID Project View 4/1/2010
Total Tests
Planned To Date
Execution
Actual To Date
Execution Defects
Test Execution Progress Defects
Overall Test
Progress# of test cases % of Total % of Total % of Total
Commodities 55 24% 36% 15% ▲ ▲ ▲Credit 70 43% 36% 4% ▬▬ ▬▬ ▬▬GFX 120 64% 37% 57% ▼ ▬▬ ▲
Project ABC 245 49% 36% 33% ▼ ▬▬ ▲
Commodities 100 60% 54% 9% ▲ ▲ ▲Credit 83 53% 46% 18% ▬▬ ▬▬ ▬▬Rates 120 64% 45% 7% ▼ ▲ ▬▬
Project XYZ 303 60% 48% 11% ▼ ▲ ▬▬
FID Project View 4/1/2010
Total Tests
Planned To Date
Execution
Actual To Date
Execution Defects
Test Execution Progress Defects
Overall Test
Progress# of test cases % of Total % of Total % of Total
Project ABC 245 49% 36% 33% ▼ ▬▬ ▲
Project XYZ 303 60% 48% 11% ▼ ▲ ▬▬
Initial
Summary
DetailFID Product View 4/1/2010
Total TestsPlanned
ExecutionActual
Execution Defects
Test Execution Progress Defects
Open Defects
Overall Test
Progress # of test cases % of Total % of Total % of Total
ABC 55 24% 36% 10% ▲ ▲ ▬▬ ▲
XYZ 100 65% 52% 13% ▲ ▲ ▼ ▼Commodities 155 50% 46% 13% ▼ ▬▬ ▬▬ ▲
ABC 70 23% 36% 1% ▲ ▲ ▬▬ ▬▬
XYZ 200 48% 26% 18% ▲ ▲ ▬▬ ▬▬Credit 27000% 41% 29% 13% ▲ ▲ ▬▬ ▬▬
FID Product View 4/1/2010
Total TestsPlanned
ExecutionActual
Execution Defects
Test Execution Progress Defects
Open Defects
Overall Test
Progress # of test cases % of Total % of Total % of Total
Commodities 155 50% 46% 13% ▼ ▬▬ ▬▬ ▲
Credit 270 41% 29% 13% ▲ ▲ ▬▬ ▬▬
Product View Execution/Defect Overview Report
Process OverviewAll test teams on QC or compliant tool on an open architecture.
New metrics defined and prioritized.
QM teams generates report, verifies with team and sends to PGM.
HowUsed Defined Fields added and populated, workflows created, folder structures and added to QC project template.
New metrics may require a data pass or bridge to other tools. (Configuration Management, Project Management, Remedy, etc,)
Teams Trained*
Dry Run*
Interim Interim
June 1, 2009
Interim
Interim
Interim
April Release - ETL
Total Cases% Actual Executed
% Plan Executed # Passed
% Passed (not including Blocked
and No Run) # Failed # No RunNot
Executed # BlockedActive
DRS Total DRs
CIT
12/12 - 1/30 341 71% 100% 225 93% 18 73 25 0 29 82
SIT
2/2 - 2/27 385 6% 100% 18 75% 6 351 10 0 0 2
Show Stopper High Medium Low Total
DR Status Total 58 22 82
Active
New 2 2
Ready For Review 2 3 4 9
Ready For Development 1 3 4
In QA 8 5 13
Re-Open 1 1
Total 12 15 29
Closed
Withdrawn 10 4 14
Completed 36 3 39
Total 46 7 53
Traceability By Requirements
Interim
PCR-PCR TitleTotal
CasesNot
Completed Passed Failed N/ANo
RunTest
BlockedActive Dr's % Completed
% In Progress Comments
PCR88890 - Stage to OLTP ETL Interface 67 0.00% 0.00% PCR 88891 - Type Table Manager ETL Interface 1 0.00% 0.00% PCR 88892 - ADIM to OrgMart DDL & DML scripts 1 0.00% 0.00%
PCR88893 - OrgMart to BPS ETL Interface 12 0.00% 0.00% PCR88894 - OrgMart to Entitlements ETL Interface 13 0.00% 0.00% PCR88993 - ADIM to OrgMart Stage ETL Interface 140 0.00% 0.00% PCR89092 - Oracle Views for OMDS 29 0.00% 0.00% PCR89093 - Data Patches Phase 1 1 0.00% 0.00% PCR89094 - Data Patches Final non-HR Update 4 0.00% 0.00% PCR89095 - Export to ADIM ETL interface 100 0.00% 0.00% PCR89096 - CMD, FSL, B2C Exports ETL/Views impact 1 0.00% 0.00% PCR89629 - SSRS Reports impact 16 0.00% 0.00% Total 385 0 0 0 0 0 0 0 0.00% 0.00%
Process OverviewAll test teams on QC or compliant tool on an open architecture.
New metrics defined and prioritized span Test Management, time utilization of BA/Dev/QA and cost analysis.
HowData is identified, workflows created to capture and maintain the data. Applications may need to be modified or acquired to support metric.
Teams Trained*
Dry Run*
Comprehensive
Comprehensive
Mar-10 Feb-10 Jan-10 Dec-10BRDs/PCRs 39 28 23 43 Number of Defects in QA 118 76 30 54Number of Defects in UA 9 2 0 7Total Prod Fix BRDs in the Post Release 57 2 7
Total 135 32 68
Defect Efficiency (DR Hours/PCR Hours) 42% 25% 30% 21% Total Number of Builds In QA 3 5 4 10Total Number of Hot Fixes - Production 1 0 0
Severity Mar Feb Jan DecShowStopper 2 3 1 6
High 73 52 19 33Medium 33 15 7 13
Low 10 6 3 2Total 118 76 30 54
Release date Total Hours (PCR + DRs) 12127 5107 5644 5349 12603
Total Effort SA (Hrs) 886 156 321 474 894Total Effort Dev (Hrs) 3172 630 1578 481 1098Total Effort QA (Hrs) 6217 2820 2355 3155 8446
Total PCR Effort (Hrs) 10275 3606 4254 4110 10438 DRs Total Effort SA (Hrs) 135 267 321 244 200
Total Effort Dev (Hrs) 1353 893 771 703 1470Total Effort QA (Hrs) 364 341 298 292 495
Total DR Effort (Hrs) 1852 1501 1390 1239 2165 Defect Effort (Hrs) 42% 25% 30% 21%
Total Showstopper Defects 11 2 3 1 6Total High Defects 112 54 52 19 33Total Medium Defects 59 10 15 7 13Total Low Defects 16 7 6 3 2Deferred DRs 20 45 0 88
Total Overall Defects 218 118 76 118 54 Total Number of Prod Fix BRDs discovered during QA Cycle 31 35 4 17
Number of Defects Withdrawn 39 26 18 2 8 Number of Actual Defects Resolved in Pre Release 24 45 116 46
QA Defect Detection Duration (Hours per Defect) 32.72 34.39 37.98 70.11 183.61
Environment Issues (QA) 7 6 5 6 11Environment Issues (IT) 2 4 4 5 5
Total Environment Issues 9 10 9 11 16 Differed DRs converted in Prod Fix BRDs 19 6 10 1
Comprehensive
Pre Release Metrics Mar Feb Jan Dec NovRelease date
Defect Type 3rd party 1 3 2 2 3Architectural Design/Technical specs 3 0 2Business Requirement/Project scope 2 Coding 19 7 7 5 14db Patch issue/Incorrect Push 0 Existing production issue 0 Functional Specification 15 7 6 5 8Technical Specification 7 5 2 Other 136 74 98 44 19Build Process 0 Training 0 Configuration 2 1 1 Environment issue 0 BLANK 35 19
TOTAL 218 118 116 56 46 UAT
DRs Opened in Viper 0 0 2 0 7
Test Cases Executed 20 13 19 Passed 13 9 8 Failed 7 4 2 No Run 0 0 9
Functional Test Cases Reviewed 339 505 236 159 121
Accepted 336 478 229 129 72Rejected 3 27 7 30 46
Functional Test Results Reviewed 148 90 178 112 41Accepted 128 90 169 98 30Rejected 20 0 9 14 11
Integration Test Results Reviewed 35 51 178 112 41
Accepted 19 35 169 98 30Rejected 16 16 9 14 11
Pre Release Metrics Mar Feb Jan Dec NovRelease date
Comprehensive
A version collection tool is used to track all web service and UI versions in the QA, IT, and ST environments on a daily basis.
Comprehensive
TE 4
TE 4
TE 3
Kenneth J. Sweetman Environment
Environments "At a Glance"
June July Aug SeptMay7 14 21 28 2 9 16 23 30 2 9 17 24 31 6 13 20 273 10 17 24 31
TE 3MQA&B Dev
TE 1 13 States Q&R Backend
TE 2 DP Tri-state Backend
Project ARelease 4.2
Release 4.3.2Release 4.4
TE 4
TE 2
Migration Tools File Delivery
Project B
Project C Release 2.8
TE 1
TE 3TE 1
TE 3 TE 1
PTE 2
TE 1
Release 1.1Release 1.2 PTE 2
PTE2SS2ACA Backend
TE 4MQ Q&R Backend
PTE1SS2ZQ Backend
PTE1
GCX Change 5/13EAB Conflict -- ZQ set to 4/1
GCX Change 5/15
GCX Change 5/15
E24
Comprehensive
• Deployment requests are sent out to all necessary teams.
• Deployments are tracked as completed.
• All web service and UI version numbers are checked to make sure the correct version was deployed.
• All deployments are validated and issues raised as necessary.
Comprehensive
PCR # PCR Status PCR Title89833New NexOS - OrgMart 2.0 - Web Services Development94123New NexOS DB - OrgMart 2.0 - UI Oracle API Phase 2
91076In Analysis NexOS DB OrgMart 2.0 Config Data cleanup and Oracle API Changes
94216In Analysis NexOS Tools & Admin - OrgMart 2.0 Data Transformation and Services
89094Pending Estimation NexOS DB - OrgMart 2.0 - P - Data Patches Final non-HR Update
89282In DevelopmentNexOS DB - OrgMart 2.0 - P - Oracle Merge scripts to sync ORGMART_PROD to ORGMART (lower environments only)
90222In Development NexOS DB - OrgMart 2.0 - Oracle PL/SQL API Packages referencing HR schema impact
91080In Development NexOS DB OrgMart 2.0 Config to OrgMart Oracle API Changes
88890In QA NexOS DB - OrgMart 2.0 - P - Stage to OLTP ETL Interface
88891In QA NexOS DB - OrgMart 2.0 - P - Type Table Manager ETL Interface
88892In QA NexOS DB - OrgMart 2.0 - P - ADIM to OrgMart DDL & DML scripts
88893In QA NexOS DB - OrgMart 2.0 - P - OrgMart to BPS ETL Interface
88894In QA NexOS DB - OrgMart 2.0 - P - OrgMart to Entitlements ETL Interface
88993In QA NexOS DB - OrgMart 2.0 - P - ADIM to OrgMart Stage ETL Interface89092In QA NexOS DB - OrgMart 2.0 - P - Oracle Views for OMDS89093In QA NexOS DB - OrgMart 2.0 - P - Data Patches Phase 1
89095In QA NexOS DB - OrgMart 2.0 - P - Export to ADIM ETL interface
89096In QA NexOS DB - OrgMart 2.0 - P - CMD, FSL, B2C Exports ETL/Views impact
89097In QA NexOS DB - OrgMart 2.0 - P - Marketing ETL Export interface impact89247In QA NexOS DB - OrgMart 2.0 - P - UI Oracle API89248In QA NexOS DB - OrgMart 2.0 - P - SuperMarvin Oracle API89396In QA NexOS DB - OrgMart 2.0 - P - Oracle API for LandSafe89629In QA NexOS DB - OrgMart 2.0 - P - SSRS Reports impact
91069In QA NexOS DB OrgMart 2.0 Config Pipeline Delegation functionality XFER to iPipe
Requirements Tracking
Comprehensive
