towards an effective process improvement platform: spago4q and the qest nd model

www.eng.it

EuroMed-SPI 2010 Conference

Sofia (Bulgaria), October 19-20 2010

Spago4Q and the QEST nD ModelTowards an Effective Process

Improvement Platform

Luigi Buglione Gabriele Ruffatti

Sergio OltolinaEngineering Group - Italy

Fulvio FratiErnesto Damiani

Mauro RegoliDipartimento di Tecnologie dell’Informazione

Università degli Studi di Milano – Italy

www.eng.it

Engineering Group At a glance

ERP ECMIT Security

Plant ManagementSystem

Broadband & MediaManaged Operations

System Int. & System Int. & ConsultancyConsultancy

OutsourcingOutsourcing

SoftwareSoftware

7070

1010

2020

8080

2020

5454

2727

1919

8080

1010

1010

FinanceFinance IndustryIndustry TELCOTELCO UtilitiesUtilities

%%

%%

%%

ResearchResearch and and DevelopmentDevelopment

3535

1919

4646

PA & HCPA & HC

_ The first Italian ICT player

_ more than 730 M/€ revenues_ 1000 clients_ 6,300 IT specialists

www.eng.it

2

http://www.eng.it/

www.eng.it3 EuroMed-SPI 2010 – Sofia, October 19-20, 2010

SESAR Lab Università degli Studi di Milano

• SErvice-oriented Software Architecture Research Lab within the Università degli Studi di Milano (Italy) headed by Prof. Ernesto Damiani

• The research activities mainly focused on: Software Engineering Knowledge Management Business Modeling Open Source Development paradigms Semantic Web Enhanced Security Architectures Mobile Nets interaction and Geolocation Distributed Systems Net Protocols

• The staff is composed by full time Professors, Researchers, Post-Docs, PhDs and Research Collaborators.

sesar.dti.unimi.it

http://sesar.dti.unimi.it/


SPAGO4Q && QEST nD Goals of the presentation

G1. Discuss the relevance of a SPI program properly set up G2. Propose an OSS based-solution joining a multidimensional performance management framework G3. Present a case study describing the implementation of a multidimensional performance model in a context of Application Management Services


Spago4Q && QEST nD Agenda

• Introduction– A bit of humor…

– Need for SPI tools

• Our Open Source Solution– QEST nD model

– Spago4Q platform

• Multidimensional Analysis Framework for SPI– Dimensions of Analysis

• Case Study– Description

– Results

• Conclusions & Perspectives

• Q & A


Introduction A bit of humour…

URL: www.dilbert.com

http://www.dilbert.com/


Introduction Some initial questions…

QQ: How (and from who) it has been perceived right now?

QQ: Is there a way to measure it?

QQ: Which possible improvements for achieving it?

QQ: What does it mean SPI in your ‘organizational language’?


Introduction Need for SPI tools

• The continuous improvement of software process has become a goal of paramount importance for organizations focused on software development

• Such a scenario of integrated information is known as Business Intelligenceand encloses all the business processes and tools used by organizations for data acquisition

• In order to achieve this goal, a number of commercial and open source frameworks has been developed and distributed implementing mathematical schema aimed at measuring process performance and suggesting process areas needing improvements.

• Tools have to be tightly coupled with formalized SPI framework to foster solid and reliable analyses


Introduction Related Works

• A fundamental problem facing software process engineering is verifying and ensuring that the mandated process is actually carried out and improved, if necessary Requires monitoring of performed process steps and produced work products

• Needed information can be obtained Monitoring actual projects as they are carried out Using a combination of manual and automatic recording Exploiting a post-mortem analysis of process enactment records

• Early efforts in this direction included TAME (Tools for a Measurement Environment), and SoftMan projects Focused on single-process, single-project data collection

• Holkar project produced a prototype quality monitoring system Used XML as means of both capturing data from sources and storing results in a

repository

• Commercial tools such as Polarion and 6th Sense Analytics do not support cross-project comparisons and are not meta-model driven

• Spago4Q fills the gap introducing a complete multi-process multi-project monitoring framework providing data for a complete process improvement analysis

http://www.cs.umd.edu/~basili/publications/journals/J35.pdf

http://www.ics.uci.edu/~wscacchi/Papers/Vintage/ieee92-process-integrated-CASE.pdf

http://www.polarion.com/

http://www.rallydev.com/company/news_events/press/2009-117-rally-software-acquires-6th-sense-analytics.html











– Results


• Q & A


Our OS Solution Introduction

Source: Buglione L. & Abran A., QEST nD: n-dimensional extension and generalisation of a Software Performance Measurement Model, International Journal of Advances in Engineering Software, Elsevier Science Publisher, Vol. 33, No. 1, January 2002, pp.1-7

• Spago4Q, the open source platform to measure, analyze and monitor quality of products, processes and services www.spago4q.org

Our work is aimed at showing the integration of • QEST nD model, a conceptual framework for measuring process

performance based on multiple analysis dimensions (e.g. economic, social, and technological dimensions) www.semq.eu/leng/modtechqlm.htm

http://www.spago4q.org/

http://www.semq.eu/leng/modtechqlm.htm



Our OS Solution QEST model

Source: Buglione L. & Abran A., QEST nD: n-dimensional extension and generalisation of a Software Performance Measurement Model, International Journal of Advances in Engineering Software, Elsevier Science Publisher, Vol. 33, No. 1, January 2002, pp.1-7

Method: Performance is expressed as the combination of the specific ratios selected for each of the 3 dimensions of the quantitative assessment (Productivity - PR) and the perceived product quality level of the qualitative assessment (Quality - Q)

Performance = PR + Q

Model: QEST (Quality factor + Economic, Social & Technical dimensions) is a “structured shell” to be filled according to management objectives in relation to a specific project

Such a model has the ability to handle independent sets of dimensions without predefined ratios and weights - referred to as an open model


Our OS Solution QEST model –– Geometrical Indicators

It is possible to measure performance considering at least 3 distinct geometrical concepts:• Distance between the tetrahedron base center of gravity and the center of the plane section along the tetrahedron height – the greater the distance from 0, the higher the performance level;• Area of the sloped plane section – the smaller the area, the higher the performance level;• Volume of the lowest part of the truncated tetrahedron – the greater the volume, the higher the performance level.

Target: measuring project performance (p) using 3 distinct viewpointsInput Data: list of weighted ratios for each dimension and quality questionnairesOutput Data: an integrated normalized value of performance


Our OS Solution QEST model –– Key Features

• Integrated quantitative and qualitative evaluation from 3 concurrent organisational viewpoints• A 3D geometrical representation at a single project phase (usually after the project is completed) • Use of de facto and de jure standards (e.g. ISO/IEC 9126 for the Quality Factor)• Performance Measurement Model to use for consolidating Balanced Scorecard (BSC) measurement outcomes• Extension of the original 3D model to n possible dimensions-perspectives QEST nD through the simplexsimplex as the mechanism to solve the problem from the 4th dimension on


Our OSS Solution Spago4Q Platform

Spago4Q (SpagoBI for Quality) is:• An open source platform for the continuous monitoring of software quality (

www.spago4q.org)• A vertical adaptation of SpagoBI, the Open Source Business Intelligence suite

(www.spagobi.org)• Pure open-source (no commercial version), with commercial support by

Engineering Group • Recently adopted by the European Commission –

Directorate General for Regional Policy (DG-REGIO)

Main features Multi-process multi-project monitoring Data Collection run in a fully, transparent wayEquipped with extractors specific of most-common software process environments (IDE, workflow management, text editing, …)


http://www.spagobi.org/



Our OSS Solution Spago4Q Structure



Our OSS Solution Spago4Q Metamodel



Our OSS Solution Definition of the Integrated Environment

• The definition of a QEST nD model is a multisteps procedure fully supported by Spago4Q

• The procedure is coherent with the PMAI (Plan-Measure-Assess-Improve) cycle: PLAN, defining a set of metrics, based on the GQM approach, and

possible dimensions of analysis (perspectives) characterizing the analysis MEASURE, including the collection of data, and the computation of

metric values and global performance value ASSESS, presenting results through dashboards and reports IMPROVE, analyzing in detail each value below expected thresholds in

order to find possible problems or bottlenecks from a process based viewpoint

Source: Buglione L., Misurare il Software. Quantità, qualità, standards e miglioramento di processo nell’Information & Communication Technology, FrancoAngeli, 3/ed, 2008, ISBN 978-88-464-9271-5



Our OSS Solution Step Step 11. Metrics & Model Definition

Declaration of a complete GQM, with the definition of the analysis dimensions

the concepts to measure

the metrics to apply to project’s workproducts



Our OSS Solution Step Step 22. Weights & Thresholds Definition

Couple each metric with the respective weight Indicates the importance that

such a concept plays in the dimension it belongs to

Define the specific thresholds Evaluates the value with respect

to organization policies

Assign (if provided) QF to each dimension Give to each dimension a quality

evaluation



Our OSS Solution Step Step 33. Data Gathering

Measures are taken directly from Spago4Q data warehouse The DB is filled by data automatically collected by extractors accessing process work-products (code package, text documents, project information, …) Supported RDBMS: MySQL, ProstgreSQL, Oracle 10g, etc…

Metrics are described in terms of: Name of the model to which the metric is assigned toDefault valueMinimum and maximum values (for normalization)KPI computation algorithm



Our OSS Solution Step Step 44. Overall Performance Calculation

• Overall and dimension-wise performance indexes are computed as KPIs that take in input configuration data and results of the metrics

• The performance value of each dimension is calculated as the weighted sum of each selected measure by its assigned weight for that dimension



Our OSS Solution Step Step 55. Reporting

• Sets of reports and dashboards could be defined and configured to satisfy reporting and managerial needs

• Spago4Q provides methods and interfaces to directly configure and create new reports using all the facilities provided by Eclipse BIRT open source reporting tool

http://www.eclipse.org/birt/phoenix/










– Results


• Q & A


Multidim. Analysis Dimensions of Analysis

• Four analysis dimensions:1. Economical (EE)

2. Technical (TT)

3. Resource Usage (RSRS)

4. Customer Satisfaction (CSCS)

• Each dimension is characterized by a specific metrics set for process evaluation

• Performance values for each dimension allow to identify process areas that need improvements









– Results


• Q & A


Case Study Project description

• The case study is based on a three-year project

• Application Management (AM) services

Software Maintenance (Corrective, Adaptive, Perfective, Preventive) for a large mission-critical system in a Finance Institute

• Services started in 2006

Analysis period : January 2008 – June 2010

• Data was collected from tools supporting development environment

Data gathered with the Polarion (www.polarion.com) Application Lifecycle Management (ALM) tool


Case Study Objectives

Verify QEST nD applicability and results in a context of AM Services

Define a QEST nD model aligned to the AM services goals

Monitor the effectiveness of improvement action with specific goals and metrics

Example of adopted goals and improvement actions

Goals EC-G3 Reduce the rework (intended as impact of defects in UAT or production environment) TE-G1 Improve the deploy process TE-G5 Improve effectiveness of peer reviews

Improvement actions Deploy process automation and automatic analysis of source code Progressively increasing of the number of peer reviews on critical work products Specific tasks were included in Impact analysis phase at the aim to:

Classify and identify critical Work Products to be reviewed Assign an owner to solve complex defects impacting on different development streams Root-cause analysis of the recurring defects


Case Study QEST Model - Dimensions && Metrics

The model defined for the case study take into account four analysis dimensions and goals as follows:

1. Economical (EE)E.G1 Reduce the effort of corrective maintenance E.G2 Improve the number of delayed deliverablesE.G3 Reduce the rework (intended as impact of defects in UAT or production environment)

1. Technical (TT)T.G1 Improve the deploy processT.G2 Reduce the resolution time for defects and technical issuesT.G3 Improve quality of documents and source codeT.G4 Reduce the rework (intended as impact of defects during development phase)T.G5 Effectiveness of peer reviewsT.G6 Improve non regression test

1. Resource Usage (RSRS)RS.G1 Reduce impact of human resource management issuesRS.G2 Improve hardware system availability

1 Customer Satisfaction (CSCS)CS.G1 Improve user satisfaction about training courses and application services


Case Study QEST Model - Dimensions && Metrics

Dimension Metric Description FormulaEconomical (EE) Incidence of Corrective Maintenance Effort w.r.t. maintained code size Corrective Maintenance Effort/ KLOC

Ratio Corrective Maintenance Effort - Adaptive Maintemance Effort Corrective Maintenance Effort/ Adaptive Maintenance Effort

Incidence of Delayed Deliverables w.r.t. total number of Deliverables no. Delayed Deliv. / no. Deliv.Incidence of Defects after system test w.r.t. total number of Defects no. Defects in UAT or production / total no. of

DefectsResource Usage (RSRS)

Human Resources management issues w.r.t. total number of issues admitted for working group size

no. HR issues / no. Issues for group size

Hardware System Availability Percentage System AvailabilityTechnical (TT) Technical management issues w.r.t. total number of issues admitted no. Technical issues / no.issues admitted

Issues Mean Resolution Time Total Res. Time / no. IssuesDocument quality: respect of document quality standard Percentage of positive response to a

checklist Software Complexity Results of automatic static code analysisCoding rules non-conformity level Results of automatic static code analysisSoftware Maintenability Results of automatic static code analysisIncidence of Peer Reviews w.r.t. total number of Deliverables no. Peer reviews / no. Deliverables

Number of Defects discovered by peer reviews w.r.t. total number of Defects no. Peer review defects / total no. defectsno. Defects / FP

Incidence of Defects Due to Design Phase w.r.t. total number of Defects no. Defects(Design phase) / Total no. Defectsfor any phase p

Test coverage w.r.t. Requirements no. Test Cases / no. Requirements

Production Defects Mean Resolution Time Total Res. Time / no. defects

Customer Satisfaction (CSCS)

Training Services Questionnaire resultsUser Satisfaction Questionnaire results


Case Study Results - QEST dashboard


Case Study Results – Dimensions trend analysis

Trend for each dimension

Last results for each dimension


Case Study Results – Global and technical % increasing


Case Study Detailed Analysis Results (exampleexample)

[AM-EC-M.04] Defects reduction in UAT and production environment [AM-TE-M.11] Defects mean resolution time reduction


Case Study

[AM-TE-M.01] Technical issues reduction: specifically related to deployment process

[AM-TE-M.02] Technical issues mean resolution time

Detailed Analysis Results (exampleexample)


Case Study

[AM-TE-M.07] Number of peer reviews actually executed vs. number of critical Work Products [AM-TE-M.08] Defects or potential defects discovered during peer reviews[AM-TE-M.09] Incidence of defects due to design phase

Detailed Analysis Results (exampleexample)







• Multidimensional Analysis Framework for SPI– Dimensions of analysis


– Results


• Q & A


Spago4Q && QEST nD Conclusions

• Need for SPI tools Huge attention from Management for low-cost technical solutions for a better monitoring

& control both at the project and organizational level OSS The right combination should include both methods and tools

• QEST nDQEST is multidimensional performance model taking into account several perspectives

from interested stakeholdersQEST can be exploited also at the SLC level with LIME (see www.semq.eu)

• SPAGO4QSpago4Q is a flexible platform for software process/product monitoring applicable to any

SPI activity Current version (v2.3.1) is freely downloadable @ www.spago4q.org

• Case study - Provided evidenceClear Benefits on measurement processGreater effectiveness of peer reviews

• Difficulties in achieving process improvements Time pressure Workload and urgent activities disqualify the team to provide sufficient effort to SPI

activities SPI is usually considered as a low priority process Continuous commitment is needed

http://www.semq.eu/



Spago4Q && QEST nD Perspectives

• Spago4Q new services will support the implementation measurement model based on standard models

Product level: ISO 9126-1 ( ISO 250xx), ISO 25012Process level: CMMI-DEV, CMMI-SVC, ISO 15504-2, ISO 20000-4, etc., …

• Spago4Q possible enhancements towards newest and more complete analysis frameworks

• Enlargement of the Spago4Q set of tool extractors to enlarge the set of extracted data

• Creation of a Correlation Matrix from base measures retrieved from the various systems to make more derived measures available to analyzers and include them in the project measurement plan (GQM-RGQM-R)

o Example: having the number of Unadjusted FP (UFP) and the project effort (in mm/h), the typical derived metrics are Productivity, defined as UFP/effort Productivity Delivery Rate, defined as effort/UFP

• Implementation of an Automatic/Dynamic (re)calculation of KPI thresholds from historical data


Q && A

Благодаря ви за вниманиетоБлагодаря ви за вниманието!! Thanks for your attentionThanks for your attention!!

Spago4Q && QEST nD


We care of your problems and we have in mind a solution

Thanks for your Attention !

Luigi Buglione, Gabriele Ruffatti, Sergio OltolinaEngineering Group

{luigi.buglione | gabriele.ruffatti | sergio.oltolina } @@eng.it

Ernesto Damiani, Fulvio Frati, Mauro RegoliUniversità degli Studi di Milano

{ernesto.damiani | fulvio.frati} @@unimi.it mauroregoli @ @tiscali.it

http://www.eng.it/

towards an effective process improvement platform: spago4q and the qest nd model

Technology

spi tools

spi program

formalized spi framework

software development

introduction need

spago4q qest nd goals

milano italy

revenues research