project quality (and test process) metrics

© 2011 Cognifide Limited. In commercial confidence only.

Project quality and test process metrics

Zbigniew Moćkun, Tomasz Rękawek

Client/partner logo


Who are we?


Agenda

• Introduction to metrics• Code quality metrics

− Metrics− Sonar as example

• Project quality and testing process metrics− Metrics− Example quality report


What is it about?

Metrics

Client/partner logo


Why do we use metrics?

• Project overview• Control a process• Control risks• Force the use of good practices• Project audits• Projects / methodologies / ... comparison

Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore


I realized that metrics...

Encourage me to ask questions


Be careful with metrics!!


Processes

People


Type of measurements

• Trend• KPI (Key Performance Indicator)

− Lower threshold− Higher threshold− Value

• How to set correct KPI?



Not only about numbers



Qualitative vs. Quantitative- case study



Level of details

• You cann’t measure everything• Choose the most important metrics• Monitor and ask questions • Looking for details when you found a problem



Metrics gathering

• Analysis of live data (on request)• Analisis based on daily values (can be run as cron job)• Data vary in time manner• Froze data for analysis


Visualize metrics

• Use charts if possible• Use colors as status (green, amber, red)• Add description• Remember about recommendation (your feeling)\

− Subjective assessment is important too


© 2011 Cognifide Limited. In commercial confidence only.© 2011 Cognifide Limited. In commercial confidence only.

Quality is everywhere

Project Quality metrics

Client/partner logo


Code quaility metrics are covered by Tomek Rękawek presentation which can be found here:

http://www.slideshare.net/TomaszRkawek/code-metrics

Code quality metrics

Client/partner logo

http://www.slideshare.net/TomaszRkawek/code-metrics


QA vs. code quality metrics

• Part of assurance

• Define KPI with Technical Lead

• Understand what you measure and why

• Discuss trends, exception with TL

• Motivate developers to keep high coding standards


Project quality metrics

Client/partner logo


Why do we test?

Software testing is an investigation conducted to provide stakeholders with information about the quality of the product or service under test


Testing = measuring

Software testing is an investigation conducted to provide stakeholders with information about the quality of the product or service under test

Source: wikipedia


Dev – QA cooperation


Metric by actions


One measurement, two metrics

• Acceptance rate / Internal Quality− Acceptance actions/all qa actions

− All QA actions = accept or reject

− Measure quality of software send to QA

• QA velocity− Sum off all actions

− Divide by number of QA Engineers assigned to project

− QA velocity depend on software quality that comes to qa

− Why testing takes so long?


Test Case Execution metrics

• Planned test cases vs. written test case (Preparation, UAT)• Number / percentage of executed test cases (Acceptance, regression, UAT)• Number of passed/failed test cases (Acceptance, regression, UAT)• Test Case metrics mean nothing


Requirement coverage

• Requirement coverage− How many requirements have test cases?

− How many requirements were tested?

− How many requirements passed?

− Can we measure coverage if we use exploratory approach?

• Requirement traceability− Defect – test case - requirement


Bugs metrics – S curve

• S Curve

• Open against Resolved

• Wages − Blocker: 10

− Critical: 8

− Major: 6

• No Major or higher rule− 5 Trivials = 1 Minor

− 5 Minors = 1 Major , ...


External quality

External Quality

• All found bugs = internal + external

• Shouldn’t be higher than 0.15 (15%)

• Sometimes divided on two parts− Found during UAT (Used Acceptance Test phase)

− Found during Live (on production)

• Sometimes used wages

• Compared with internal shows our testing process quality


The importance of metrics

• Example A− Story comes to QA

− QA found 1 major issue and reject (1 day)

− Dev fix it (1 day)

− QA test it again, 1 major issue found and reject(1 day)


− QA accept (1 day)

− Sum: 5 days, 2, rejects, 2 issues only

• Example B− Story comes to QA

− QA found 1 critical, 1 major and 3 minor issues and reject story (1 day)


− QA test it again and accept (1 day)

− Sum: 4 days, 1 reject, 5 issues

• Acceptance rate against raised issues metric


Bug Taxonomy


Bug Taxonomy

• Phases− Project: Discovery, Development, UAT, Live

• Test Case lifecycle− Requirements/Documentation, Test Scenario, Test Execution

• Risk areas

• Application modules/components

• Functional/non-functional

• Product specific areas− CMS: author, publish, frontend, backend


Defect Density - components


Defect Density - Acceptance vs. Regression (automation vs. manual)


Defect DensityApplication specific


Defect Density – won’t fixDo we test in right way?


Automation - coverage

• Defects found (manual vs. automated)

• Coverage− Manual vs. automated test cases

− High priority

− Application specific Author vs. publish


Other metrics

• Performance monitoring trend as example

• Client side grades (webpagetest.org as example)

• QA queue

• Agile metrics− Lead Time (Open to Resolved time)

− Idle Time (time spend in queues)


Qualitative metrics

• Client feeling− Surveys

− Demo

− Just try to talk

• Tester / team subjective opinion− Recommendation

− Feeling

− Usability


Agile vs. Waterfall

• Quality metrics are independent against:− Methodology

− Approach

• Two main test approach can be used in both− Exploratory vs. scripting

• Agile put more emphasis on qualitative metrics


Example report


References

• http://www.kaner.com/pdfs/BugTaxonomies.pdf

• ISTQB Advance Test Manager Syllabus− http://www.istqb.org/downloads/viewcategory/46.html

• Michael Bolton blog post− http://www.developsense.com/blog/2009/01/meaningful-metrics/

• Douglas Hoffman− http://www.softwarequalitymethods.com/Papers/DarkMets%20Pape

r.pdf

http://www.kaner.com/pdfs/BugTaxonomies.pdf

http://www.kaner.com/pdfs/BugTaxonomies.pdf

http://www.istqb.org/downloads/viewcategory/46.html

http://www.istqb.org/downloads/viewcategory/46.html

http://www.developsense.com/blog/2009/01/meaningful-metrics/

http://www.developsense.com/blog/2009/01/meaningful-metrics/

http://www.softwarequalitymethods.com/Papers/DarkMets%20Paper.pdf

http://www.softwarequalitymethods.com/Papers/DarkMets%20Paper.pdf


Poznań Testing and Quality Group

• Local group

• Next meeting: 28th of November− Bogdan Bereza-Jarociński: Quality in Agile – quality at all

− Jakub Bryl: How to organize security testing

project quality (and test process) metrics

Technology

consectetur

qa test

project quality

investigation

dev fix

defect density

2011 cognifide

commercial