a quality certification model for grid research projects the etics feasibility study

www.eu-etics.org

INFSOM-RI-026753

A Quality Certification Model for A Quality Certification Model for Grid Research ProjectsGrid Research Projects

the ETICS feasibility Studythe ETICS feasibility Study

Adriano RippaAdriano Rippa((adriano.rippa@eng.itadriano.rippa@eng.it)) - Engineering Ingegneria Informatica s.p.a.- Engineering Ingegneria Informatica s.p.a.On behalf of the ETICS ProjectOn behalf of the ETICS Project

ETICS All Hands Meeting - Geneva 21-23/11/2007 2INFSOM-RI-026753

Summary

• Grid Quality Certification ModelGrid Quality Certification Model (GQCM) - UPDATE (GQCM) - UPDATE

• Comparing GQCM and other QA standardsComparing GQCM and other QA standards

• TimelineTimeline

• Gotten feedbacksGotten feedbacks

• PublicationsPublications

• A possible first implementationA possible first implementation

ETICS All Hands Meeting - Geneva 21-23/11/2007 3INFSOM-RI-026753

In Madison we left here…

• The Grid Quality Assurance The Grid Quality Assurance Certification Process Certification Process (GQACP) was organized in (GQACP) was organized in three perspectives looking three perspectives looking at Code, Platforms and at Code, Platforms and Standards quality aspects.Standards quality aspects.

• Now it asses the quality Now it asses the quality against the same macro-against the same macro-aspects but…aspects but…

CodeCode

PlatformsPlatforms

StandardsStandards

ETICS All Hands Meeting - Geneva 21-23/11/2007 4INFSOM-RI-026753

…Now we are here

• GQCM GQCM has been reorganized as follows:has been reorganized as follows:

– A new name!A new name!

– GQCMGQCM has been described has been described according to several ISO standardsaccording to several ISO standards (e.g. ISO/IEC (e.g. ISO/IEC 25000, 1459825000, 14598))

– GQCMGQCM has been restructured has been restructured according to ISO 25041according to ISO 25041

– Quality attributesQuality attributes has been named using the same terminology has been named using the same terminology as as ISO 9126ISO 9126

– More metrics More metrics has been added including bounds and target has been added including bounds and target valuesvalues

ETICS All Hands Meeting - Geneva 21-23/11/2007 5INFSOM-RI-026753

GQCM: the structure

• GQCM is structured in GQCM is structured in Evaluation Modules (EM)Evaluation Modules (EM) according to ISO 25041according to ISO 25041

• The set of evaluation techniques are grouped in The set of evaluation techniques are grouped in families. Every family is an EMfamilies. Every family is an EM

• 5 EMs:5 EMs:

– EM: Static analysisEM: Static analysis– EM: Coding styleEM: Coding style– EM: Structural testingEM: Structural testing– EM: Functional testingEM: Functional testing– EM: Standards complianceEM: Standards compliance

ETICS All Hands Meeting - Geneva 21-23/11/2007 6INFSOM-RI-026753

Evaluation Modules 1/5

• EM Static analysisEM Static analysis– Quality characteristics:Quality characteristics:

– Reliability – maturityReliability – maturity– Maintainability – analysabilityMaintainability – analysability– Maintainability – changeabilityMaintainability – changeability– Maintainability – testabilityMaintainability – testability

– Technique:Technique: – Static analysis of classes. Statistics on measures are used as Static analysis of classes. Statistics on measures are used as

predictor of quality characteristics. predictor of quality characteristics.

– Applicability:Applicability:– Object oriented programming languages. Object oriented programming languages.

– Input:Input:– source code, each class of the code is analysed source code, each class of the code is analysed

ETICS All Hands Meeting - Geneva 21-23/11/2007 7INFSOM-RI-026753

Evaluation Modules 2/5

• EM Coding styleEM Coding style– Quality characteristics:Quality characteristics:

– Maintainability – analysabilityMaintainability – analysability

– Technique: Technique: – Static analysis of the source code. Static analysis of the source code.

– Applicability:Applicability:– Most programming languageMost programming language

– Input:Input:– source codesource code

ETICS All Hands Meeting - Geneva 21-23/11/2007 8INFSOM-RI-026753

Evaluation Modules 3/5

• EM Structural testingEM Structural testing– Quality characteristics:Quality characteristics:

– Functionality – accuracyFunctionality – accuracy– Reliability – maturityReliability – maturity

– Technique:Technique:– Structural testing. The intention of this EM is to test specific classes Structural testing. The intention of this EM is to test specific classes

that are identified by static measures as being statistically more that are identified by static measures as being statistically more likely to have many errors. likely to have many errors.

– Applicability:Applicability:– Object oriented programming languagesObject oriented programming languages

– Input:Input:– source codesource code

ETICS All Hands Meeting - Geneva 21-23/11/2007 9INFSOM-RI-026753

Evaluation Modules 4/5

• EM Functional testingEM Functional testing– Quality characteristics:Quality characteristics:

– Functionality – accuracyFunctionality – accuracy– Functionality – interoperabilityFunctionality – interoperability– Reliability – maturityReliability – maturity– Portability – adaptabilityPortability – adaptability– Portability - installabilityPortability - installability

– Technique:Technique:– Functional testingFunctional testing

– The purpose of this EM is both to check platform compliance and The purpose of this EM is both to check platform compliance and to check to functional abilities of the software product.to check to functional abilities of the software product.

– Applicability:Applicability:– GeneralGeneral

– Input:Input:– compiled codecompiled code– user documentationuser documentation

ETICS All Hands Meeting - Geneva 21-23/11/2007 10INFSOM-RI-026753

Evaluation Modules 5/5

• EM Standards complianceEM Standards compliance– Quality characteristics:Quality characteristics:

– Functionality – standards complianceFunctionality – standards compliance

– Technique:Technique:– Functional testing Functional testing

– This EM has a good potential for automating the test. This EM has a good potential for automating the test.

– Applicability:Applicability:– Standard specificStandard specific

– Input:Input:– compiled codecompiled code

ETICS All Hands Meeting - Geneva 21-23/11/2007 11INFSOM-RI-026753

GQCM: Final Score

• Final score should be provided according to the following Final score should be provided according to the following schema. The items which should be available for the users are:schema. The items which should be available for the users are:– A A table summarizing the resultstable summarizing the results– A A list of passed and non passed testslist of passed and non passed tests– All the All the important informationimportant information as: as:

– Software product (e.g. name, version, executable code, documentation..)Software product (e.g. name, version, executable code, documentation..)

– Platform (name, version, date)Platform (name, version, date)

– Quality characteristics (name, evaluation result, evaluation module Quality characteristics (name, evaluation result, evaluation module identification)identification)

– Standard compliance (for each standard: name, version, date)Standard compliance (for each standard: name, version, date)

– Identification of evaluation report (organization, report number, date)Identification of evaluation report (organization, report number, date)

– Identification of certification body (organization, contact information)Identification of certification body (organization, contact information)

– Certification data (dates, certification number)Certification data (dates, certification number)

– Electronic signature of certification record Electronic signature of certification record

ETICS All Hands Meeting - Geneva 21-23/11/2007 12INFSOM-RI-026753

GQCM: Summarizing Table example

Consolidated eval. result

EM Static

EM Coding

EM structural

EM Functional

EM Std compliance

EM eval. result M M G G G Y

Functionality G G G

Accuracy G G G

Interoperability G G

Compliance Y (Y)

Reliability M M G G

Maturity M M G G

Maintainability M M G

Analyzability M M G

Changeability M M

Testability M M

Portability G G

Adaptability G G

Installability G G

G = GoodM = MediumP = Poor

ETICS All Hands Meeting - Geneva 21-23/11/2007 13INFSOM-RI-026753

GQCM integration in an ISO 9126 certified organization

• Facts:Facts:– ISO 9126 define quality attributes ISO 9126 define quality attributes – GQCM support the measure and evaluation of quality metricsGQCM support the measure and evaluation of quality metrics

• Why they are compatible each other?Why they are compatible each other?– Many ISO9126 aspects can be evaluated by GQCM Many ISO9126 aspects can be evaluated by GQCM GQCM GQCM

apply what ISO 9126 asksapply what ISO 9126 asks– Any tool implementing GQCM and can be integrated as Any tool implementing GQCM and can be integrated as

supporting ISO9126 adoptionsupporting ISO9126 adoption

• SoSo– GQCM can be both an independent quality model and a “tool” to GQCM can be both an independent quality model and a “tool” to

be easily compliant with ISO 9126be easily compliant with ISO 9126

ETICS All Hands Meeting - Geneva 21-23/11/2007 14INFSOM-RI-026753

GQCM integration in a CMMi certified organization

• FactsFacts– CMM is process oriented (it gives guidelines).CMM is process oriented (it gives guidelines).– GQCM is product oriented.GQCM is product oriented.

• Hypothesis:Hypothesis:– CMM2 CMM2

– asks to “control and value”asks to “control and value”– GQCM proposes the measures and metrics to be collected, hence GQCM proposes the measures and metrics to be collected, hence

controlling and evaluating specific sw related itemscontrolling and evaluating specific sw related items

– CMM3 CMM3 – asks to “use internal standards and have a reference model”asks to “use internal standards and have a reference model”– GQCM proposes code and implementation conventions (how to GQCM proposes code and implementation conventions (how to

organise and structure code, how to define tests, etc.)organise and structure code, how to define tests, etc.)

ETICS All Hands Meeting - Geneva 21-23/11/2007 15INFSOM-RI-026753

GQCM integration in a CMM certified organization

• CMM4 CMM4 – asks to “have measured and controlled processes using quantitative asks to “have measured and controlled processes using quantitative

and statistical techniques”and statistical techniques”– GQCM proposes the continuous collection of quality data to perform GQCM proposes the continuous collection of quality data to perform

trends analysis.trends analysis.

• CMM5 CMM5 – asks to have an “improvement process and quality based on asks to have an “improvement process and quality based on

measurement”measurement”– GQCM provides numerical reference to set the improvement GQCM provides numerical reference to set the improvement

processes.processes.

– Our ThesisOur Thesis– GQCM is not in contrast with CMMI and can be integrated in GQCM is not in contrast with CMMI and can be integrated in

organization as a tool to support CMMI adoptionorganization as a tool to support CMMI adoption

ETICS All Hands Meeting - Geneva 21-23/11/2007 16INFSOM-RI-026753

The timeline

OCTOBEROCTOBER DECEMBERDECEMBER

ECHOGRID/EUChinagridECHOGRID/EUChinagrid

Conference 24-25 April Conference 24-25 April

BeijingBeijing

MAYMAYFEBRUARYFEBRUARY

OGF 20/EGEE UFOGF 20/EGEE UF

7-11 May 7-11 May

ManchesterManchester

Belief ConferenceBelief Conference

25-28 June 25-28 June

Rio de JaneiroRio de Janeiro

OGF 21OGF 21

15-19 October15-19 October

SeattleSeattle

QUALIPSO QUALIPSO

ConferenceConference

16-17 January 200816-17 January 2008

RomeRome

NowNow

EGEE ’07EGEE ’07

1-5 October1-5 October

BudapestBudapest

EELA 3° EELA 3° Conference Conference

3-5 December3-5 December

CataniaCatania

AHMAHM

23-25 May 23-25 May

MadisonMadison

ETICS All Hands Meeting - Geneva 21-23/11/2007 17INFSOM-RI-026753

GQCM: publications

• 2 papers submitted2 papers submitted• 2 papers accepted2 papers accepted

• When/WhereWhen/Where• EELA third conferenceEELA third conference

• 3-5 December 20073-5 December 2007

• Catania/ItalyCatania/Italy

• QUALIPSO first conferenceQUALIPSO first conference• 16-17 January 200816-17 January 2008

• Rome/ItalyRome/Italy

• The first The first quality oriented conferencequality oriented conference!!

ETICS All Hands Meeting - Geneva 21-23/11/2007 18INFSOM-RI-026753

GQCM: feedbacks

• The model has got good feedbacks and The model has got good feedbacks and acknowledgementsacknowledgements

• People are asking for more informationPeople are asking for more information

• Some issues about the name of the model (solved)Some issues about the name of the model (solved)

• One issue about ISO9000 compliance (outside the One issue about ISO9000 compliance (outside the scope?)scope?)

• Requests for metrics about the process Requests for metrics about the process outside outside the scope?the scope?

ETICS All Hands Meeting - Geneva 21-23/11/2007 19INFSOM-RI-026753

GQCM: future and open issues

• Within ETICS 2Within ETICS 2• ImplementationImplementation• ImprovementsImprovements• Pilot projects to test itPilot projects to test it• ISO standardisationISO standardisation

• Open issueOpen issue• First implementation?First implementation?

ETICS All Hands Meeting - Geneva 21-23/11/2007 20INFSOM-RI-026753

What we have now?

• Programming languages Programming languages Java/Phyton Java/Phyton– Others?Others?

• JDepend can calculateJDepend can calculate– Number of classes indicator of the extensibility of the packageNumber of classes indicator of the extensibility of the package– Afferent Couplings indicator of the package’s responsibilityAfferent Couplings indicator of the package’s responsibility– Efferent Couplings … independenceEfferent Couplings … independence– Abstractness 0/1 Abstractness 0/1 package concrete – abstract package concrete – abstract– Instability 0/1 Instability 0/1 – Distance Distance indicator of the package bilance between abstractness and stability indicator of the package bilance between abstractness and stability

• PMD scans Java source code and looks for potential problems like:PMD scans Java source code and looks for potential problems like:– Possible bugs - empty try/catch/finally/switch statementsPossible bugs - empty try/catch/finally/switch statements– Dead code - unused local variables, parameters and private methodsDead code - unused local variables, parameters and private methods– Suboptimal code - wasteful String/StringBuffer usageSuboptimal code - wasteful String/StringBuffer usage– Overcomplicated expressions - unnecessary if statements, for loops that could be while loopsOvercomplicated expressions - unnecessary if statements, for loops that could be while loops– Duplicate code - copied/pasted code means copied/pasted bugsDuplicate code - copied/pasted code means copied/pasted bugs

• Cobertura/EmmaCobertura/Emma– Coverage: class, method, lines, blockCoverage: class, method, lines, block

• FindBugs reportsFindBugs reports– high/medium/low priority bugs for javahigh/medium/low priority bugs for java

• JUnitJUnit– Java unit testsJava unit tests

• PyUnitPyUnit– Python unit testsPython unit tests

• PyCyclomaticComplexityPyCyclomaticComplexity– Python Cyclomatic ComplexityPython Cyclomatic Complexity

ETICS All Hands Meeting - Geneva 21-23/11/2007 21INFSOM-RI-026753

A possible first “implementation”?

• We can implement partially three evaluation modulesWe can implement partially three evaluation modules– EM: Static Analysis (Only one measure..)EM: Static Analysis (Only one measure..)

– WMC: looks at the complexity of classesWMC: looks at the complexity of classes

– EM: Coding StyleEM: Coding Style– Only some measurement but we can have a realistic value from itOnly some measurement but we can have a realistic value from it

– EM: Structural TestingEM: Structural Testing – it is expected that a small part of the code contain a relative large it is expected that a small part of the code contain a relative large

number of the defects in the code. Some metrics are used to number of the defects in the code. Some metrics are used to identify potential problematic classes.identify potential problematic classes.

– Use other test cases than already used for statement coverage.Use other test cases than already used for statement coverage.

– The first implementation will aim at the programming languages The first implementation will aim at the programming languages Java and Python. […]Java and Python. […]

ETICS All Hands Meeting - Geneva 21-23/11/2007 22INFSOM-RI-026753

Q&A

http://www.eu-etics.org