criteria for the evaluation of implemented...

Arent Janszoon Ernststraat 595-HNL-1082 LD Amsterdaminfo@sig.nlwww.sig.nl

Criteria for the Evaluation of Implemented ArchitecturesEric Bouwers, Joost Visser, Arie van Deursen

2009/09/22

Criteria for the Evaluation of Implemented Architectures

Why evaluate software architectures?

‘Architectures allow or preclude nearly all of thesystem’s quality attributes’

-- from `Evaluating Software Architectures` by P. Clements, R. Kazman and M. Klein

Criteria for the Evaluation of Implemented Architectures

Why evaluate implemented architectures?

Criteria for the Evaluation of Implemented Architectures

Why evaluate implemented architectures?

Criteria for the Evaluation of Implemented Architectures

Why do we want criteria for the evaluation?

Maintenance Phase

Architecture Evaluation

Result

Architecture Evaluation

Result

Architecture Evaluation

Result

Architecture Evaluation

Result

Criteria for the Evaluation of Implemented Architectures

Literature reviews

A framework for classifying andcomparing software architecture

evaluation methods.

M. Babar, L. Zhu, and D. R. Jeffery

A survey on softwarearchitecture analysis

methods.

L. Dobrica and E. Niemelä.

Criteria for the Evaluation of Implemented Architectures

IEEE Computer, July 2009

‘Technology transfer remainsa major challenge. There is

limited out of the box processand tool support for

organizations that want tostart reviews.’

‘Software Architecture Review: The state of Practice’

by M. Babar, I. Gorton

Criteria for the Evaluation of Implemented Architectures

Software Risk Assessment (SRA)

A one-time investigation into software maintainabilityand related business risks

Criteria for the Evaluation of Implemented Architectures

Architectural properties

GUI

Data Access

Business Logic

High Level Design:

Saving Paying Stocks

Modularization:

if(hasPayed() && !isAdmin()){ showThankYou();}

Separation of Concerns:

Business logic/Security

Criteria for the Evaluation of Implemented Architectures

Research questions

Q1: Which system attributes do experts take into accountwhen evaluating architectural system properties?

Q2: How do these system attributes influence thearchitectural system properties?

Criteria for the Evaluation of Implemented Architectures

Empirical study resources

Criteria for the Evaluation of Implemented Architectures

A selection of our clients

Financials ITLogistics OtherPublic

Criteria for the Evaluation of Implemented Architectures

Evaluated systems characteristics

Criteria for the Evaluation of Implemented Architectures

Empirical study design overview

CreateAttribute List

Create Mapping to Properties

ValidateAttributes &Mapping

Criteria for the Evaluation of Implemented Architectures

Experimental design (1/3)Creation of the attribute list

1) Check table with property ratings for argumentsif this table does not exists

use specific paragraphs

For each report:

2) For all arguments founddetermine system attributeif the attribute is not on the list

add system attribute to the list

Criteria for the Evaluation of Implemented Architectures

Experimental design (2/3)Create mapping from attributes to properties

2) For all arguments founddetermine system attributeadd one to count of attribute for property

1) Extract arguments per property

For each report:

Criteria for the Evaluation of Implemented Architectures

Experimental design (3/3)Validation of Attributes and Mapping

Criteria for the Evaluation of Implemented Architectures

Results of Empirical Study (1/3)

CreateAttribute List

Create Mapping to Properties

ValidateAttributes &Mapping

Criteria for the Evaluation of Implemented Architectures

15 System attributes

• Abstraction• Layering• Logic in Database• Module Inconsistency• Module Size• Source Grouping• Technology Combination• Textual Duplication

• Functional Duplication• Libraries / Frameworks Usage• Module Dependencies• Module Functionality • Relation Documentation and Implementation• Technology Age• Technology Usage

Criteria for the Evaluation of Implemented Architectures

More details available in the paper

Inspecting the call-graph on modulelevel, matching thisagainst expecteddependencies

The staticdependencies (i.e.calls, includes)between modules

ModuleDependencies

Inspecting the listof imports andstructure of thesource- and build-files

The usage ofstandard librariesand frameworks

Libraries /Frameworks

AssessmentApproach

DescriptionName

Criteria for the Evaluation of Implemented Architectures

Results of empirical study (2/3)

CreateAttribute List

Create Mapping to Properties

ValidateAttributes &Mapping

Criteria for the Evaluation of Implemented Architectures

High Level Design

Modularization

Separation of Concerns

..... .....

Module Dependencies

Module Functionality

Relation Doc. / Impl.

Functional Duplication

Technology Combination

Technology Age

7 4 2 2 5 13

11 32 3 6 1 0

6 13 0 18 0 0

Attributes used for each property

Criteria for the Evaluation of Implemented Architectures

High Level Design

Modularization

Separation of Concerns

..... .....

Module Dependencies

Module Functionality

Relation Doc. / Impl.

Functional Duplication

Technology Combination

Technology Age

X X

X X X

X

X

Attributes used for each property

Criteria for the Evaluation of Implemented Architectures

Results of empirical study (3/3)

CreateAttribute List

Create Mapping to Properties

ValidateAttributes &Mapping

Criteria for the Evaluation of Implemented Architectures

Interview results

High Level Design

Modularization

Separation of Concerns

E1

E1, E2

E1

E2

E2

..... .....

Module Dependencies

Module Functionality

Relation Doc. / Impl.

Functional Duplication

Technology Combination

Technology Age

Criteria for the Evaluation of Implemented Architectures

Threats to validity

Representative data

Reliability of measurements

Generalization of the results

Criteria for the Evaluation of Implemented Architectures

‘What is in it for me?’-- you

Criteria for the Evaluation of Implemented Architectures

Our contributions

•A description of an empirical study using over 40 reports

•The identification of 15 system attributes that have an impact on the maintainability of an architecture

•An analysis of the projection of the found system attributes onto three architectural system properties

Criteria for the Evaluation of Implemented Architectures

Application of the results

Criteria for the Evaluation of Implemented Architectures

Future research opportunities

Criteria for the Evaluation of Implemented Architectures

Criteria for the Evaluation of Implemented Architectures

Criteria for the Evaluation of Implemented Architectures

Questions or comments?

Criteria for the Evaluation of Implemented Architectures

• A description of an empirical study using over 40 SRAreports• The identification of 15 system attributes that have an

impact on the maintainability of an implementedarchitecture• An analysis of the projection of the found system

attributes onto three architectural system properties

Feel free to contact me:e.bouwers@sig.eu