Spago4Q and the QEST nD Model:An Open Source Solution

for Software Performance MeasurementClaudio A. Ardagna, Ernesto Damiani, Fulvio Frati, Mauro Regoli

Dipartimento di Tecnologie dell’InformazioneUniversità degli Studi di Milano – Italy

Sergio Oltolina, Gabriele RuffattiEngineering Ingegneria Informatica - Italy

Outline

• QEST nD Model

• Spago4Q Open Source Platform

• Roadmap to build the Integrated Framework

• Case Study

2OSS 2010

Introduction

• The availability of detailed and updated information on development process is paramount– To maintain the competitiveness level – To operate in new and more challenging markets

• The scenario of integrated information is known as Business Intelligence, and enclose all the processes and tools for data acquisition and analysis

• A number of structured process models have been adopted depending on the domain and size– To collect specific knowledge about their development processes– To strengthen the know-how in terms of more efficiency and quality

3OSS 2010

Goals

• Deploy an environment that integrates models for software performance evaluation in a more complete and coherent framework– Non-invasive– Dynamic– Configurable– Scalable– Based on a formal mathematical model

• Provide a solution integrating– The QEST nD model for process performance evaluation– The Open Source Business Intelligence platform Spago4Q

4OSS 2010

The QEST nD Model - 1

• Several mono-dimensional performance models are available in SE context that integrates individual measurements into a single performance index– Models are oversimplified and unable to reflect different performance

aspects

• Traditional domains (e.g., Business Modeling) exploit multi-dimensional models – Data derived directly from accounting systems, implying multiple

analysis viewpoints

– Complex viewpoints analyze distinct aspects of the overall process performance

5OSS 2010

The QEST nD Model - 2

• The QEST nD model is aimed at measuring software project performances addressing multi-dimensionality and qualitative-quantitative assessment

• In the QEST model the quality can be viewed as the integration of at least three different viewpoints

– Economical

– Social

– Technical

• User-specific dimensions could be defined to evaluate ad-hoc analysis dimensions

L. Buglione, “Misurare il software. Quantità, Qualità, Standards eMiglioramento di processo nell'Information & Communication Technology,” 3rd Edition, Franco Angeli, 2008

OSS 2010 6

Mathematical Formalization

• QEST Performance value (P) given by the combination

– Quantitative measurements (RP-Rough Productivity)

– Qualitative measurements, calculated as a perception-based measurement of the overall product quality (QF-Quality Factor)

OSS 2010 7

Mathematical Formalization - 2

• The basic 3-D model QEST is represented as a geometrical regular tetrahedron– Corners E, S, T and the convergence

of the edges to the P vertex describe the performance level

– Points Qe, Qs, Qt represent the respective dimension values (Rough Productivity)

• Values calculated as the weighted sum of a list of n distinct measures

• Represented as the distance between points and corners

• If Quality factor is applied, points Q’e, Q’s, Q’t represent an upward or downward translation of Qe, Qs, Qt

OSS 2010 8

Mathematical Formalization - 3

• Performance P calculated as the distancebetween the center of gravity of the original tetrahedron and the center of the plane (Q’e, Q’s, Q’t) section along the tetrahedron height

• Through computational geometry, the multi-dimensional case is described with a simple formula for global performance value computation

OSS 2010 9

Spago4Q

• Spago4Q (SpagoBI for Quality) is an open source platform for the continuous monitoring of software quality

– Multi-process multi-project monitoring

– Collection of data executed in a fully-transparent way

– Equipped with extractors specific of most-common software process environments (IDE, workflow management, text editing, …)

• Spago4Q (www.spago4q.org) is a vertical adaptation of SpagoBI, the Open Source Business Intelligence suite (www.spagobi.org)

• Recently adopted by European Commission –Directorate General for Regional Policy

OSS 2010 10

Spago4Q Structure

OSS 2010 11

Spago4Q Metamodel

OSS 2010 12

An Integrated Environment

• The definition of a QEST nD model is a five-step procedure fully supported by Spago4Q

• The procedure is coherent with the PMAI (Plan-Measure-Assess-Improve) cycle:– PLAN, defining a set of KPIs, basing on the GQM approach, and

dimensions that characterize the analysis– MEASURE, which includes the collection of data, and the

computation of KPI values and global performance value– ASSESS, where results are presented in dashboards and

reports– IMPROVE, where every negative or low value is deeply

analyzed to find problems in the process

OSS 2010 13

1st Step: KPIs and Model Definition

• Declaration of a complete GQM, with the definition of – the analysis dimensions

– the concepts to measure

– the KPIs to apply to project work-products

OSS 2010 14

2nd Step: Weights and Thresholds Definition

• Couple each KPI with the respective weight– Indicates the importance that

such a concept plays in the dimension it belongs to

• Define the specific thresholds– Evaluates the value with

respect to organization policies

• Assign (if provided) the QF to each specific dimension

OSS 2010 15

3rd Step: Value Collection

• Measures are taken directly from Spago4Q data warehouse– The DB is filled by data automatically collected by

extractors accessing process work-products (code package, text documents, project information, …)

• KPIs are described in terms of – Default value

– Minimum and maximum values (for normalization)

– KPI computation algorithm

OSS 2010 16

4th Step: Global Performance Computation

• Global and dimension-wise performance indexes are computed as KPIs that take in input configuration data and results of the KPIs

• Dimension performance is calculated as the weighted sums of each specific KPI

OSS 2010 17

5th Step: Reports

• Sets of reports and dashboards could be defined and configured to satisfy reporting and managerial needs

• Spago4Q provides methods and interfaces to directly configure and create new reports using all the facilities provided by the open source reporting tool Eclipse BIRT

OSS 2010 18

Case Studies

• Case study based on the monitoring of the complex performance of three real projects with several measurements in a three-month time slot

• We use a QEST nD model called Business-Service Model and taking into consideration four specific analysis dimensions:1. QEST-EC: Economic performance indicator

2. QEST-RS: Resource performance indicator

3. QEST-TE: Technical performance indicator

4. QEST-CS: Customer Satisfaction performance indicator

OSS 2010 19

Case Studies: GQM

OSS 2010 20

Case Studies: KPIs

OSS 2010 21

Case Studies: Dashboards

• Spago4Q dashboard gives an immediate snapshot of the situation, highlighting problems and suggesting areas that need a process quality improvement

– Red squares indicate under-threshold performance values

OSS 2010 22

Spago4Q: Reports

• Reports indicate that all projects were concluded with an excellent global performance (close to one), showing some issues in the process that is worth analyzing– Project3 experienced bad performances in Technological and

Economic dimensions

OSS 2010 23

Conclusions

• We provide a complete multi-project multi-process performance evaluation environment combining– The mathematical formalization of the QEST nD model– The monitoring and reporting facilities offered by Spago4Q

• It gives the possibility to analyze the performance of the development process from different points of view integrating semantically different KPIs in a single indicator

• The described QEST nD model will be made available in the future version of Spago4Q (www.spago4q.org)

OSS 2010 24

Questions?

Thank you for the attention

OSS 2010 25