metaheuristic models for decision support in the software ... · dr. josé raúl romero salguero...

57
Metaheuristic models for decision support in the software construction process – Ph.D. Thesis – Aurora Ramírez Quesada Dept. Computer Science and Numerical Analysis University of Córdoba Supervisors: Dr. José Raúl Romero Salguero Dr. Sebastián Ventura Soto 24 th September 2018

Upload: others

Post on 05-Mar-2021

1 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Metaheuristic models for decision support in the software ... · Dr. José Raúl Romero Salguero Dr. Sebastián Ventura Soto 24th September 2018. Introduction Architecture discovery

Metaheuristic models for decision support in the softwareconstruction process

– Ph.D. Thesis –

Aurora Ramírez Quesada

Dept. Computer Science and Numerical AnalysisUniversity of Córdoba

Supervisors:Dr. José Raúl Romero Salguero

Dr. Sebastián Ventura Soto

24th September 2018

Page 2: Metaheuristic models for decision support in the software ... · Dr. José Raúl Romero Salguero Dr. Sebastián Ventura Soto 24th September 2018. Introduction Architecture discovery

Introduction Architecture discovery Multi/Many perspectives Human-in-the-loop approach Conclusions

1 Introduction

2 Evolutionary discovery of software architectures

3 The multi- and many-objective perspectives

4 The human-in-the-loop approach

5 Conclusions and future work

Aurora Ramírez Quesada Ph.D. Thesis 2 / 50

Page 3: Metaheuristic models for decision support in the software ... · Dr. José Raúl Romero Salguero Dr. Sebastián Ventura Soto 24th September 2018. Introduction Architecture discovery

Introduction Architecture discovery Multi/Many perspectives Human-in-the-loop approach Conclusions

Introduction

Aurora Ramírez Quesada Ph.D. Thesis 3 / 50

Page 4: Metaheuristic models for decision support in the software ... · Dr. José Raúl Romero Salguero Dr. Sebastián Ventura Soto 24th September 2018. Introduction Architecture discovery

Introduction Architecture discovery Multi/Many perspectives Human-in-the-loop approach Conclusions

Background - Software architectures

A software architecture represents “the fundamental concepts or properties of asystem in its environment embodied in its elements, relationships, and in the

principles of its design and evolution” (ISO/IEC Std. 42010)

A bridge between requirements and implementation

Essential in early software conception and its subsequent evolution

Special focus on non-functional requirements: availability, performance, security...

Software architectures are fundamental for:Understanding

Reuse

Construction

Evolution

Analysis

Management

Aurora Ramírez Quesada Ph.D. Thesis 4 / 50

Page 5: Metaheuristic models for decision support in the software ... · Dr. José Raúl Romero Salguero Dr. Sebastián Ventura Soto 24th September 2018. Introduction Architecture discovery

Introduction Architecture discovery Multi/Many perspectives Human-in-the-loop approach Conclusions

Background - Metaheuristics

“Intelligent” exploration of the search space

Optimality is not guaranteed

Intensification and diversificationProperties:

1 Iterative

2 Stochastic

3 Memory-based

4 Bio-inspired

Aurora Ramírez Quesada Ph.D. Thesis 5 / 50

Page 6: Metaheuristic models for decision support in the software ... · Dr. José Raúl Romero Salguero Dr. Sebastián Ventura Soto 24th September 2018. Introduction Architecture discovery

Introduction Architecture discovery Multi/Many perspectives Human-in-the-loop approach Conclusions

Background - Evolutionary computation

“It is not the strongest of the species that survives, nor the most intelligent thatsurvives. It is the one that is most adaptable to change.” (Charles Darwin)

Phenotype/genotype mapping

The fitness function

Aurora Ramírez Quesada Ph.D. Thesis 6 / 50

Page 7: Metaheuristic models for decision support in the software ... · Dr. José Raúl Romero Salguero Dr. Sebastián Ventura Soto 24th September 2018. Introduction Architecture discovery

Introduction Architecture discovery Multi/Many perspectives Human-in-the-loop approach Conclusions

Background - Optimisation with multiple objectives

Multi-objective optimisation problem:

x = [x1, x2, ..., xn] subject to gj (x) ≤ 0 j = 1, ..., pf (x) = [f1(x), f2(x), ..., fm(x)]

2-3 objectives, often in conflict (≥ 4→ many-objective)

The concept of Pareto dominance

No single optimal solution⇒ Pareto Set (PS)

In the objective space⇒ Pareto Front (PF)

The goals of a multi-/many-objective evolutionary algorithm (MOEA/MaOEA):

1 To find a good approximation of the true PF

2 To generate a well-spread and uniform PF

Aurora Ramírez Quesada Ph.D. Thesis 7 / 50

Page 8: Metaheuristic models for decision support in the software ... · Dr. José Raúl Romero Salguero Dr. Sebastián Ventura Soto 24th September 2018. Introduction Architecture discovery

Introduction Architecture discovery Multi/Many perspectives Human-in-the-loop approach Conclusions

Background - Interactive optimisation

Full automation is not always realistic:Uncertainty scenarios

Creative tasks

Any optimisation method, includingmetaheuristics, in which the human

actively participates in any step of theprocess to provide feedback

Human is put in the loop:To provide problem knowledge

To increase trust on automatic results

To meet user’s expectations

Design and implementation issues:Selection of solutions

Type of feedback

Frequency of interaction

Information lifetime

Aurora Ramírez Quesada Ph.D. Thesis 8 / 50

Page 9: Metaheuristic models for decision support in the software ... · Dr. José Raúl Romero Salguero Dr. Sebastián Ventura Soto 24th September 2018. Introduction Architecture discovery

Introduction Architecture discovery Multi/Many perspectives Human-in-the-loop approach Conclusions

Background - Search-based Software Engineering

Origin and characteristicsSBSE was formalised as research field by Harman and Jones in 2001

Software engineering activities can be reformulated as optimisation problems

Examples along the software lifecycle: project planning, requirement priorisation,selection of test cases, refactoring...

Software metrics are adopted to define the fitness function

Evolutionary computation (EC) stands out as the most popular technique

Search-based software design (SBSD)The need of creativity and intuition makes it particularly challenging

Applicable to different paradigms (object-oriented, services...)

Problem encoding and evaluation are hard to define

Aurora Ramírez Quesada Ph.D. Thesis 9 / 50

Page 10: Metaheuristic models for decision support in the software ... · Dr. José Raúl Romero Salguero Dr. Sebastián Ventura Soto 24th September 2018. Introduction Architecture discovery

Introduction Architecture discovery Multi/Many perspectives Human-in-the-loop approach Conclusions

Background - Software architecture optimisation

Scope and characteristicsAutomatic specification and improvement of architectural models

Often guided by more than one non-functional requirement

Problems can be defined at different levels of abstraction

Current approachesArchitecture synthesis from requirements or use cases

Selection and assembling of black-box components, e.g. COTS

Recovery of software components from code

Architecture deployment (reconfiguration and allocation)

Aurora Ramírez Quesada Ph.D. Thesis 10 / 50

Page 11: Metaheuristic models for decision support in the software ... · Dr. José Raúl Romero Salguero Dr. Sebastián Ventura Soto 24th September 2018. Introduction Architecture discovery

Introduction Architecture discovery Multi/Many perspectives Human-in-the-loop approach Conclusions

Objectives

General objective:

Development of search models based on metaheuristic algorithms to providesemi-automatic support to software engineers in early stages of the software

development process

Subobjectives:

O1: Analysis of the state of the art in SBSE, especially in the areas of SBSD andarchitecture optimisation

O2: Design and development of a metaheuristic model based on EC to supportthe discovery of component-based software architectures

O3: Design and development of multi-objective approaches and hybrid techniquesto study their suitability to the decision support process

O4: Design and development of an interactive approach to incorporate humanexpertise in the context of software architecture optimisation

Aurora Ramírez Quesada Ph.D. Thesis 11 / 50

Page 12: Metaheuristic models for decision support in the software ... · Dr. José Raúl Romero Salguero Dr. Sebastián Ventura Soto 24th September 2018. Introduction Architecture discovery

Introduction Architecture discovery Multi/Many perspectives Human-in-the-loop approach Conclusions

Work plan

1 Formulation of software architecture discovery as an optimisation problem

2 A first evolutionary model to validate the idea

3 Exploration of diverse search techniques to cope with additional decision factors

4 Participation of the engineer to complete the scenario

Aurora Ramírez Quesada Ph.D. Thesis 12 / 50

Page 13: Metaheuristic models for decision support in the software ... · Dr. José Raúl Romero Salguero Dr. Sebastián Ventura Soto 24th September 2018. Introduction Architecture discovery

Introduction Architecture discovery Multi/Many perspectives Human-in-the-loop approach Conclusions

Problem description

Software architecture discovery

To identify the underlying architecture of the system from the analysis informationcontained in a UML class diagram

The elements of the component-based software architecture are obtained as follows:ComponentA cohesive group of classes that work together to satisfy the expected behaviourof the componentInterfaceA directed relationship between classes in the analysis model belonging todifferent componentsConnectorThe linkage between a pair of required / provided interfaces interconnectingdifferent components

Aurora Ramírez Quesada Ph.D. Thesis 13 / 50

Page 14: Metaheuristic models for decision support in the software ... · Dr. José Raúl Romero Salguero Dr. Sebastián Ventura Soto 24th September 2018. Introduction Architecture discovery

Introduction Architecture discovery Multi/Many perspectives Human-in-the-loop approach Conclusions

Evolutionary discovery of software architectures

Aurora Ramírez Quesada Ph.D. Thesis 14 / 50

Page 15: Metaheuristic models for decision support in the software ... · Dr. José Raúl Romero Salguero Dr. Sebastián Ventura Soto 24th September 2018. Introduction Architecture discovery

Introduction Architecture discovery Multi/Many perspectives Human-in-the-loop approach Conclusions

Motivation

Current approaches recover components from low-level artefacts, e.g. code

The process cannot be performed until the system is built

The system structure could be different from that originally conceived

Analysis information is omitted

Aurora Ramírez Quesada Ph.D. Thesis 15 / 50

Page 16: Metaheuristic models for decision support in the software ... · Dr. José Raúl Romero Salguero Dr. Sebastián Ventura Soto 24th September 2018. Introduction Architecture discovery

Introduction Architecture discovery Multi/Many perspectives Human-in-the-loop approach Conclusions

Research questions

RQ: Can single-objective EAs help the software engineer to iden-tify an initial candidate architecture of a system at a high level ofabstraction?

RQ: How does the configuration of the algorithm influence both theevolutionary performance and the quality of the returned solution?

Aurora Ramírez Quesada Ph.D. Thesis 16 / 50

Page 17: Metaheuristic models for decision support in the software ... · Dr. José Raúl Romero Salguero Dr. Sebastián Ventura Soto 24th September 2018. Introduction Architecture discovery

Introduction Architecture discovery Multi/Many perspectives Human-in-the-loop approach Conclusions

Solution encoding

Solutions are encoded using tree structures:

Hierarchical decomposition

Comprehensive for software engineers

Architectural solutions of different size A_prov_E

E_req_A

Phenotype

Architecture

Components Connectors

Component_1 Component_2 Connector_1

Classes ClassesProvided

interfaces

Required

interfaces

Provided

interfaces

Required

interfaces

Provided

interface

Required

interface

B A_prov_E E_req_A A_prov_EE_req_AA C D FE G H

Genotype

Aurora Ramírez Quesada Ph.D. Thesis 17 / 50

Page 18: Metaheuristic models for decision support in the software ... · Dr. José Raúl Romero Salguero Dr. Sebastián Ventura Soto 24th September 2018. Introduction Architecture discovery

Introduction Architecture discovery Multi/Many perspectives Human-in-the-loop approach Conclusions

Fitness function

Intra-modular Coupling Density (ICD)

ICDi =#classestotal −#classesi

#classestotal·

CI ini

CI ini + CIout

iICD =

1n·

n∑i=1

ICDi

External Relations Penalty (ERP)

ERP =n∑

i=1

n∑j=i+1

(was · nasij + wag · nagij + wco · ncoij + wge · ngeij )

Groups/Components Ratio (GCR)

GCR =#cgroups

n

fitness(ind) =

{r(ICD(ind)) + r(ERP(ind)) + r(GCR(ind)) if ind is feasible#individuals ·#metrics + 1 if ind is infeasible

Aurora Ramírez Quesada Ph.D. Thesis 18 / 50

Page 19: Metaheuristic models for decision support in the software ... · Dr. José Raúl Romero Salguero Dr. Sebastián Ventura Soto 24th September 2018. Introduction Architecture discovery

Introduction Architecture discovery Multi/Many perspectives Human-in-the-loop approach Conclusions

Mutation operator

Algorithm 1: Add

1 Choose contributors;2 Extract small groups;3 Create new component;4 Add selected classes;5 Set interfaces;

Algorithm 2: Remove

1 Find highly-coupled components;2 Choose component;3 Distribute classes;4 Set interfaces;

Algorithm 3: Move

1 Choose origin;2 Choose class;3 Choose target;4 Move class;5 Set interfaces;

Algorithm 4: Merge

1 Find highly-coupled components;2 Choose candidate 1;3 if found>2 then4 Choose candidate 2;5 else6 Choose component 2;7 Join classes;8 Set interfaces;

Algorithm 5: Split

1 Find components with > 1 group;2 if found then3 Choose candidate;4 Distribute groups;5 else6 Choose component;7 Distribute classes;8 Set interfaces

Aurora Ramírez Quesada Ph.D. Thesis 19 / 50

Page 20: Metaheuristic models for decision support in the software ... · Dr. José Raúl Romero Salguero Dr. Sebastián Ventura Soto 24th September 2018. Introduction Architecture discovery

Introduction Architecture discovery Multi/Many perspectives Human-in-the-loop approach Conclusions

Illustrative example

(a) Input class diagram

Component_3

Component_2Component_1

E_req_A

G_req_H

A_prov_E

H_prov_G

ICD=0.17

GCR=2.00

Fitness=7.00

ERP=13.00

(b) Initial population

Fitness=5.00 ERP=5.00ICD=0.35 GCR=1.50

A_prov_E

E_req_A

Component_1 Component_2

(c) 5th Generation

Fitness=3.00 ERP=0.00ICD=0.38 GCR=1.00

A_prov_E

E_req_A

Component_1 Component_2

(d) 10th Generation

Aurora Ramírez Quesada Ph.D. Thesis 20 / 50

Page 21: Metaheuristic models for decision support in the software ... · Dr. José Raúl Romero Salguero Dr. Sebastián Ventura Soto 24th September 2018. Introduction Architecture discovery

Introduction Architecture discovery Multi/Many perspectives Human-in-the-loop approach Conclusions

Experimental framework

Implementation details

JCLEC as the metaheuristic framework to code the algorithm

SDMetrics Open Core to parse XMI files

Datapro4j to preprocess input data and generate reports

Problem instances

Problem #Classes #Relationships #InterfacesAs De Ag Co GeAqualush 58 69 6 0 0 20 74Borg 167 44 109 36 38 90 300Datapro4j 59 3 4 3 2 49 12Java2HTML 53 20 66 15 0 15 170JSapar 46 7 33 21 9 19 80Marvin 32 5 11 22 5 8 28NekoHTML 47 6 17 15 18 17 46

Aurora Ramírez Quesada Ph.D. Thesis 21 / 50

Page 22: Metaheuristic models for decision support in the software ... · Dr. José Raúl Romero Salguero Dr. Sebastián Ventura Soto 24th September 2018. Introduction Architecture discovery

Introduction Architecture discovery Multi/Many perspectives Human-in-the-loop approach Conclusions

Results

Parameter study

Selection and replacement strategies: Binary tournament, 10% elitism

Mutation weights (126 combinations):wadd = 0.2, wremove = 0.1, wmerge = 0.1, wsplit = 0.3, wmove = 0.3

Population size and number of evaluations: 150, 20,000-24,000

Final results

Problem ICD (↑) ERP (↓) GCR (↓) Time (ms)Aqualush 0.41240.0604 6.23333.8443 1.08330.1708 116.10198.4891Borg 0.28200.0689 3.93332.4626 1.16670.2274 2, 489.1223171.2028Datapro4j 0.64250.0356 33.600014.7436 2.39890.6744 37.11012.8308Java2HTML 0.25930.0000 0.00000.0000 1.00000.0000 250.45727.4674JSapar 0.37510.0307 9.00001.5492 1.16670.2981 94.88915.1418Marvin 0.50800.0187 3.16671.0980 1.67500.1146 14.11940.6885NekoHTML 0.45940.0345 3.26675.3037 1.23890.3768 57.11503.5524

Aurora Ramírez Quesada Ph.D. Thesis 22 / 50

Page 23: Metaheuristic models for decision support in the software ... · Dr. José Raúl Romero Salguero Dr. Sebastián Ventura Soto 24th September 2018. Introduction Architecture discovery

Introduction Architecture discovery Multi/Many perspectives Human-in-the-loop approach Conclusions

Lessons learned

I Lack of software metrics suitable for fitness function definition

I Need of domain-specific operators to improve performance

I Ability to obtain solutions close to manual designs

I Influence of the number and type of UML relations in the hardness of the problem

Aurora Ramírez Quesada Ph.D. Thesis 23 / 50

Page 24: Metaheuristic models for decision support in the software ... · Dr. José Raúl Romero Salguero Dr. Sebastián Ventura Soto 24th September 2018. Introduction Architecture discovery

Introduction Architecture discovery Multi/Many perspectives Human-in-the-loop approach Conclusions

Associated publications

Main publication:

A. Ramírez, J.R. Romero, S. Ventura. “An approach for the evolutionary discovery ofsoftware architectures”. Information Sciences, vol. 305, pp. 234-255, 2015.

Additional publications:

1 A. Ramírez, J.R. Romero, S. Ventura. “Análisis de la aplicabilidad de medidas software para eldiseño semi-automático de arquitecturas”. Proceedings of the XIX Jornadas en Ingeniería delSoftware y Bases de Datos (JISBD), pp. 307-320, 2014.

2 A. Ramírez, J.A. Molina, J.R. Romero, S. Ventura. “Estudio de mecanismos de hibridación parael descubrimiento evolutivo de arquitecturas”. Proceedings of the XXI Jornadas en Ingenieríadel Software y Bases de Datos (JISBD), pp. 481-494, 2016.

3 A. Ramírez, R. Barbudo, J.R. Romero, S. Ventura. “Memetic Algorithms for the Automatic Dis-covery of Software Architectures”. Proceedings of the 16th International Conference on Intelli-gent Systems Design and Applications (ISDA), vol. 557 AISC, pp. 437-447, 2016.

Aurora Ramírez Quesada Ph.D. Thesis 24 / 50

Page 25: Metaheuristic models for decision support in the software ... · Dr. José Raúl Romero Salguero Dr. Sebastián Ventura Soto 24th September 2018. Introduction Architecture discovery

Introduction Architecture discovery Multi/Many perspectives Human-in-the-loop approach Conclusions

The multi- and many-objective perspectives

Aurora Ramírez Quesada Ph.D. Thesis 25 / 50

Page 26: Metaheuristic models for decision support in the software ... · Dr. José Raúl Romero Salguero Dr. Sebastián Ventura Soto 24th September 2018. Introduction Architecture discovery

Introduction Architecture discovery Multi/Many perspectives Human-in-the-loop approach Conclusions

Motivation

Evidence of conflict between ICD and ERP

Additional quality criteria usually guide the discovery process

Each system might require the optimisation of a different number of metrics

The algorithm could provide a set of alternative solutions for engineer’s choice

Aurora Ramírez Quesada Ph.D. Thesis 26 / 50

Page 27: Metaheuristic models for decision support in the software ... · Dr. José Raúl Romero Salguero Dr. Sebastián Ventura Soto 24th September 2018. Introduction Architecture discovery

Introduction Architecture discovery Multi/Many perspectives Human-in-the-loop approach Conclusions

Research questions

RQ: How do a larger number and combination of metrics influencethe search-based discovery of software architectures?

RQ: How do state-of-the-art MOEAs and MaOEAs perform whenmultiple objectives guide the discovery process?

Aurora Ramírez Quesada Ph.D. Thesis 27 / 50

Page 28: Metaheuristic models for decision support in the software ... · Dr. José Raúl Romero Salguero Dr. Sebastián Ventura Soto 24th September 2018. Introduction Architecture discovery

Introduction Architecture discovery Multi/Many perspectives Human-in-the-loop approach Conclusions

Objective functions

Maintainability is “the degree to which the software product can be modified", wheremodifications “may include corrections, improvements or adaptation of the software to

changes in the environment, and in requirements and functional specifications"(ISO/IEC Std. 25000)

Modularity: “the degree to which a system is composed of discrete componentssuch that a change to one component has minimal impact on other components"

Reusability: “the degree to which an asset can be used in more than one softwaresystem or in building other assets"

Analysability: “the degree to which the parts of the software to be modified can beidentified"

Metric Min/Max Quality attribute Range of values Design goalsICD max modularity [0, 1] Small components with high cohesionERP min modularity [0, ∗] Large components with low couplingINS min modularity [0, 1] Components with few interactionsENC max modularity [0, 1] Components with hidden classesCS min modularity [0, n] Small or medium-sized componentsCL min modularity [0, n] Components with few provided interfaces

GCR min reusability [1, ∗] Connected classes within each componentABS max reusability [0, 1] Components with abstract classesCB max analisability [0, 1] Equal-sized components

Aurora Ramírez Quesada Ph.D. Thesis 28 / 50

Page 29: Metaheuristic models for decision support in the software ... · Dr. José Raúl Romero Salguero Dr. Sebastián Ventura Soto 24th September 2018. Introduction Architecture discovery

Introduction Architecture discovery Multi/Many perspectives Human-in-the-loop approach Conclusions

Multi/Many-objective evolutionary algorithms

1 Multi-objective algorithms (Pareto-based)SPEA2

NSGA-II

2 Decomposition algorithmsMOEA/D

3 Algorithms based on reference setNSGA-III

4 Algorithms based on landscape partitionε-MOEA

GrEA

5 Indicator-based algorithmsIBEA

HypE

Aurora Ramírez Quesada Ph.D. Thesis 29 / 50

Page 30: Metaheuristic models for decision support in the software ... · Dr. José Raúl Romero Salguero Dr. Sebastián Ventura Soto 24th September 2018. Introduction Architecture discovery

Introduction Architecture discovery Multi/Many perspectives Human-in-the-loop approach Conclusions

Experimental framework

Development of JCLEC-MOIndependence between the metaheuristic and the multi-objective techniques

Experimental utilities

Parameters and analysisAll possible combinations of 2, 4, 6, 8 and 9 metrics (256 in total)

The number of software systems is increased up to 10 (32 - 256 classes)

Quality indicators: hypervolume, spacing

Statistical analysis: Friedman&Holm, Cliff’s Delta (effect size)

Parameter ValuePopulation size 100, 120, 126, 330, 495Max. evaluations 10000, 15000, 20000, 66000, 99000Min-max. components 2-8Mutator weights wadd = 0.2, wremove = 0.3, wmerge = 0.2

wsplit = 0.1, wmove = 0.2ERP weights was = 2, wag = 3, wco = 3, wge = 5CS threshold 0.3CL threshold 8Algorithms configured following authors’ recommendations / preliminary experiments.

Aurora Ramírez Quesada Ph.D. Thesis 30 / 50

Page 31: Metaheuristic models for decision support in the software ... · Dr. José Raúl Romero Salguero Dr. Sebastián Ventura Soto 24th September 2018. Introduction Architecture discovery

Introduction Architecture discovery Multi/Many perspectives Human-in-the-loop approach Conclusions

Results

HypervolumeAlgorithm 2 4 6 8 9SPEA2 4.35± 0.78 4.66± 0.86 5.64± 0.29 6.33± 0.28 6.70± 0.00NSGA-II 2.02± 0.59 1.39± 0.40 1.92± 0.54 2.66± 0.46 2.90± 0.00MOEA/D 3.90± 0.81 4.41± 0.68 3.75± 0.47 3.48± 0.43 3.30± 0.00NSGA-III 6.75± 0.49 6.62± 0.21 6.49± 0.16 6.01± 0.29 5.90± 0.00ε-MOEA 3.88± 1.12 2.78± 1.15 1.51± 0.55 1.08± 0.09 1.00± 0.00GrEA 4.86± 0.79 5.30± 0.75 5.80± 0.41 5.83± 0.11 5.50± 0.00IBEA 7.21± 0.31 7.64± 0.15 7.76± 0.06 7.79± 0.05 7.90± 0.00HypE 3.04± 0.58 3.20± 0.75 3.20± 0.55 2.82± 0.31 2.80± 0.00

SpacingAlgorithm 2 4 6 8 9SPEA2 3.65± 1.25 2.09± 1.08 1.10± 0.17 1.37± 0.37 1.70± 0.00NSGA-II 4.00± 0.82 4.59± 0.91 2.77± 0.85 1.97± 0.34 1.80± 0.00MOEA/D 3.23± 1.10 3.16± 0.79 5.21± 0.50 5.69± 0.23 5.30± 0.00NSGA-III 4.35± 1.45 4.02± 1.09 3.53± 0.51 3.13± 0.16 3.20± 0.00ε-MOEA 5.58± 1.04 3.02± 1.49 3.41± 0.69 4.06± 0.37 4.60± 0.00GrEA 5.37± 0.52 6.78± 0.45 7.29± 0.17 7.28± 0.14 6.60± 0.00IBEA 5.73± 1.51 7.43± 0.42 7.60± 0.04 7.63± 0.07 7.80± 0.00HypE 4.09± 0.72 4.91± 0.95 5.11± 0.82 4.88± 0.57 5.00± 0.00

Aurora Ramírez Quesada Ph.D. Thesis 31 / 50

Page 32: Metaheuristic models for decision support in the software ... · Dr. José Raúl Romero Salguero Dr. Sebastián Ventura Soto 24th September 2018. Introduction Architecture discovery

Introduction Architecture discovery Multi/Many perspectives Human-in-the-loop approach Conclusions

Results

9-objective Pareto fronts

(a) SPEA2 (b) NSGA-II (c) MOEA/D (d) NSGA-III

(e) ε-MOEA (f) GrEA (g) IBEA (h) HypE

Aurora Ramírez Quesada Ph.D. Thesis 32 / 50

Page 33: Metaheuristic models for decision support in the software ... · Dr. José Raúl Romero Salguero Dr. Sebastián Ventura Soto 24th September 2018. Introduction Architecture discovery

Introduction Architecture discovery Multi/Many perspectives Human-in-the-loop approach Conclusions

Lessons learned

I Not all many-objective algorithms are effective in this context

I Some metrics do not present real conflict, others are highly opposed

I The number and combination of metrics might depend on the system under study

I Practical implications in decision-making: execution time, number of solutions

Aurora Ramírez Quesada Ph.D. Thesis 33 / 50

Page 34: Metaheuristic models for decision support in the software ... · Dr. José Raúl Romero Salguero Dr. Sebastián Ventura Soto 24th September 2018. Introduction Architecture discovery

Introduction Architecture discovery Multi/Many perspectives Human-in-the-loop approach Conclusions

Associated publications

Main publication:

A. Ramírez, J.R. Romero, S. Ventura. “A comparative study of many-objectiveevolutionary algorithms for the discovery of software architectures”. Empirical

Software Engineering, vol. 21, no. 6, pp. 2546-2600, 2016.

Additional publications:

1 A. Ramírez, J.R. Romero, S.Ventura. “On the Performance of Multiple Objective EvolutionaryAlgorithms for Software Architecture Discovery”. Proceedings of the 16th Genetic and Evolu-tionary Computation Conference (GECCO), pp. 1287-1294, 2014. Best paper of the SBSEtrack and nominated to best paper of GECCO’14.

2 A. Ramírez, J.R. Romero, S.Ventura. “Estudio preliminar del rendimiento de familias de algo-ritmos multiobjetivo en diseño arquitectónico”. Proceedings of the X Congreso Español sobreMetaheurísticas, Algoritmos Evolutivos y Bioinspirados (MAEB), pp. 173-180, 2015.

3 A. Ramírez, J.R. Romero, S.Ventura. “An Extensible JCLEC-based Solution for the Implemen-tation of Multi-Objective Evolutionary Algorithms”. Proceedings of the Companion Publicationof 17th Genetic and Evolutionary Computation Conference (GECCO Companion), pp. 1085-1092, 2015.

4 A. Ramírez, J.R. Romero, S. Ventura. “On the effect of local search in the multi-objective evolu-tionary discovery of software architectures”. Proceedings of the IEEE Congress on EvolutionaryComputation (IEEE CEC), pp. 2038-2045, 2017.

5 A. Ramírez, J.R. Romero, C. García-Martínez, S.Ventura. “JCLEC-MO: A Java suite for solvingmany-objective optimization engineering problems”. Under review, 2018. (Indexed journal, Q1).

6 A. Ramírez, J.R. Romero, S.Ventura. “A survey of many-objective optimisation in search-basedsoftware engineering”. Under review, 2018. (Indexed journal, Q1).

Aurora Ramírez Quesada Ph.D. Thesis 34 / 50

Page 35: Metaheuristic models for decision support in the software ... · Dr. José Raúl Romero Salguero Dr. Sebastián Ventura Soto 24th September 2018. Introduction Architecture discovery

Introduction Architecture discovery Multi/Many perspectives Human-in-the-loop approach Conclusions

The human-in-the-loop approach

Aurora Ramírez Quesada Ph.D. Thesis 35 / 50

Page 36: Metaheuristic models for decision support in the software ... · Dr. José Raúl Romero Salguero Dr. Sebastián Ventura Soto 24th September 2018. Introduction Architecture discovery

Introduction Architecture discovery Multi/Many perspectives Human-in-the-loop approach Conclusions

Motivation

Software design is a human-centred task

Qualitative aspects are difficult to quantify

The engineer should be further involved in the optimisation process

Sometimes it is easier to identify a bad solution rather than a good one

Aurora Ramírez Quesada Ph.D. Thesis 36 / 50

Page 37: Metaheuristic models for decision support in the software ... · Dr. José Raúl Romero Salguero Dr. Sebastián Ventura Soto 24th September 2018. Introduction Architecture discovery

Introduction Architecture discovery Multi/Many perspectives Human-in-the-loop approach Conclusions

Research questions

RQ: How can the qualitative judgement of the engineer be inte-grated into the evolutionary discovery of software architectures?

RQ: Does putting the human in the loop involve a significant im-provement compared with not considering him/her along the opti-misation process?

Aurora Ramírez Quesada Ph.D. Thesis 37 / 50

Page 38: Metaheuristic models for decision support in the software ... · Dr. José Raúl Romero Salguero Dr. Sebastián Ventura Soto 24th September 2018. Introduction Architecture discovery

Introduction Architecture discovery Multi/Many perspectives Human-in-the-loop approach Conclusions

The algorithm

Steady-state algorithm: two offspringare produced in each generation

Binary tournament selection frompopulation and archive

Offspring replace population memberswith worst fitness

Archive to store a small set ofrepresentative solutions

Aurora Ramírez Quesada Ph.D. Thesis 38 / 50

Page 39: Metaheuristic models for decision support in the software ... · Dr. José Raúl Romero Salguero Dr. Sebastián Ventura Soto 24th September 2018. Introduction Architecture discovery

Introduction Architecture discovery Multi/Many perspectives Human-in-the-loop approach Conclusions

The fitness function

A novel fitness function that combines objective and subjective evaluation criteria

fitness(s) = wobj · fobj (s) + wsub · fsub(s)

Quantitative: Maximin function to quantify bothdominance and diversity (software metrics)

fobj (s) =1 + maxz 6=s(mink (f s

k − f zk ))

2∀z ∈ Z

Qualitative: Engineer’s preferences onphenotypic aspects of the solution(architectural preferences)

fsub(s) = 1−1P·

P∑p=1

wp · prefp(s)

Aurora Ramírez Quesada Ph.D. Thesis 39 / 50

Page 40: Metaheuristic models for decision support in the software ... · Dr. José Raúl Romero Salguero Dr. Sebastián Ventura Soto 24th September 2018. Introduction Architecture discovery

Introduction Architecture discovery Multi/Many perspectives Human-in-the-loop approach Conclusions

Architectural preferences

Preference DescriptionBest component Similarity to the set of classesWorst component Dissimilarity to the set of classesBest provided interface Similarity to the set of operationsWorst provided interface Dissimilarity to the set of operationsNumber of components Distance to the preferred numberMetric in range Distance to the midrangeAspiration levels Weighted distance to the reference point

Aurora Ramírez Quesada Ph.D. Thesis 40 / 50

Page 41: Metaheuristic models for decision support in the software ... · Dr. José Raúl Romero Salguero Dr. Sebastián Ventura Soto 24th September 2018. Introduction Architecture discovery

Introduction Architecture discovery Multi/Many perspectives Human-in-the-loop approach Conclusions

Interaction mechanism

Interactions are scheduled at regularintervals

Solutions are selected from the currentpopulation using a clustering method

The software engineer rewards orpenalises some aspect of the solutionby choosing one preference

Additional actions:Freeze one component

Add to the archive

Remove from the population

Stop the search

Aurora Ramírez Quesada Ph.D. Thesis 41 / 50

Page 42: Metaheuristic models for decision support in the software ... · Dr. José Raúl Romero Salguero Dr. Sebastián Ventura Soto 24th September 2018. Introduction Architecture discovery

Introduction Architecture discovery Multi/Many perspectives Human-in-the-loop approach Conclusions

Empirical study

MethodologyCase study, 9 participants

3 interactions, 3 solutions each

Log files and questionnaires

● ● ●

●●

00

01

02

03

04

05

i1s1 i1s2 i1s3 i2s1 i2s2 i2s3 i3s1 i3s2 i3s3Step of interaction

Eva

luat

ion

time

(min

utes

)

Impact of interactionCohesion is indirectly improved

Different number of components

Components similar or equal tothe manual design

Actions (add, freeze) contribute tofind better solutions 0

10

20

30

40

50

60

70

80

2 3 4 5 6Number of components

Num

ber

of s

olut

ions

Without interaction With interaction

Aurora Ramírez Quesada Ph.D. Thesis 42 / 50

Page 43: Metaheuristic models for decision support in the software ... · Dr. José Raúl Romero Salguero Dr. Sebastián Ventura Soto 24th September 2018. Introduction Architecture discovery

Introduction Architecture discovery Multi/Many perspectives Human-in-the-loop approach Conclusions

Empirical study

Analysis of architectural preferencesMore interest in the internal structure

Negative preferences are morefrequently applied at the beginning

Use of “No preference”0

5

10

15

20

25

Freq

uenc

y

No prefere

nce

Best c

omponent

Best in

terfac

e

Worst co

mponent

Worst in

terfac

e

No. Componen

ts

Measu

re in ra

nge

Aspira

tion le

vels

Architectural Selected Usefulness IntuitivenessPreference (%) [1-8] [1-8]No preference 22.22 6.44 7.33Best component 29.63 7.44 7.44Worst component 23.46 7.22 7.33Best provided interface 2.47 5.29 6.38Worst provided interface 0.00 4.71 6.38Number of components 17.28 7.50 7.33Metric in range 2.47 4.17 5.44Aspiration levels 2.47 5.80 5.22

Aurora Ramírez Quesada Ph.D. Thesis 43 / 50

Page 44: Metaheuristic models for decision support in the software ... · Dr. José Raúl Romero Salguero Dr. Sebastián Ventura Soto 24th September 2018. Introduction Architecture discovery

Introduction Architecture discovery Multi/Many perspectives Human-in-the-loop approach Conclusions

Lessons learned

I Architecture preferences as a novel mechanism for subjective evaluation

I Design metrics are still needed at the beginning

I Negative opinions are useful too

I Humans are willing to participate (even more) in the optimisation

Aurora Ramírez Quesada Ph.D. Thesis 44 / 50

Page 45: Metaheuristic models for decision support in the software ... · Dr. José Raúl Romero Salguero Dr. Sebastián Ventura Soto 24th September 2018. Introduction Architecture discovery

Introduction Architecture discovery Multi/Many perspectives Human-in-the-loop approach Conclusions

Associated publications

Main publication:

A. Ramírez, J.R. Romero, and S. Ventura. “Interactive multi-objective optimization ofsoftware architectures”. Information Sciences, vol. 463-464, pp. 92-109, 2018.

Additional publications:

1 A. Ramírez, J.R. Romero, S.Ventura. “Interactividad en el descubrimiento evolutivo de arquitec-turas”. Proceedings of the XX Jornadas en Ingeniería del Software y Bases de Datos (JISBD),2015.

2 A. Ramírez, R. Barbudo, J.R. Romero, S. Ventura. “Herramienta basada en computación evo-lutiva interactiva para arquitectos software”. Proceedings of the XI Congreso Español sobreMetaheurísticas, Algoritmos Evolutivos y Bioinspirados (MAEB), pp. 387-396, 2016.

3 A. Ramírez, J.R. Romero, S. Ventura. “Búsqueda coevolutiva interactiva aplicada al diseñode software”. Proceedings of the XXII Jornadas en Ingeniería del Software y Bases de Datos(JISBD), 2017.

4 A. Ramírez, J.R. Romero, S. Ventura.“API para el desarrollo de algoritmos interactivos en inge-niería del software basada en búsqueda”. XXIII Jornadas en Ingeniería del Software y Basesde Datos (JISBD), 2018.

5 A. Ramírez, J.R. Romero, C. Simons. “A Systematic Review of Interaction in Search-BasedSoftware Engineering”. IEEE Transactions on Software Engineering, pp. 1-22. 2018. In press.

Aurora Ramírez Quesada Ph.D. Thesis 45 / 50

Page 46: Metaheuristic models for decision support in the software ... · Dr. José Raúl Romero Salguero Dr. Sebastián Ventura Soto 24th September 2018. Introduction Architecture discovery

Introduction Architecture discovery Multi/Many perspectives Human-in-the-loop approach Conclusions

Conclusions and future work

Aurora Ramírez Quesada Ph.D. Thesis 46 / 50

Page 47: Metaheuristic models for decision support in the software ... · Dr. José Raúl Romero Salguero Dr. Sebastián Ventura Soto 24th September 2018. Introduction Architecture discovery

Introduction Architecture discovery Multi/Many perspectives Human-in-the-loop approach Conclusions

Concluding remarks

1 Definition of software architecture discovery as an optimisation problem

2 Design of a comprehensible evolutionary model to discover architectures

3 Analysis of the influence of software metrics in a many-objective space

4 Exploration of hybrid approaches to improve the performance

5 Development of a general interactive model to integrate human expertise

Literature Problem Algorithm Experimental Modelanalysis formulation implementation validation integration

O1TSE

UnderReview1O2 JISBD’14 INFSCI

O3

GECCO’14, MAEB’15, EMSE GECCO’15JISBD’16, ISDA’16, CEC’17 UnderReview2

JISBD’16 ESwA

O4

JISBD’15INFSCI MAEB’16

JISBD’17 JISBD’18

Overview of publications derived from the Ph.D. Thesis

Aurora Ramírez Quesada Ph.D. Thesis 47 / 50

Page 48: Metaheuristic models for decision support in the software ... · Dr. José Raúl Romero Salguero Dr. Sebastián Ventura Soto 24th September 2018. Introduction Architecture discovery

Introduction Architecture discovery Multi/Many perspectives Human-in-the-loop approach Conclusions

Future lines of research

1 Extensions to the problem formulationDesign constraints

Evaluation and semantic aspects

2 Advanced interactive modelsCooperative coevolution + interaction

Learning from user interaction

3 New optimisation problems in SBSDService-oriented architectures

Cloud computing

4 Industrial acceptanceIntegrated process

Software tools

Aurora Ramírez Quesada Ph.D. Thesis 48 / 50

Page 49: Metaheuristic models for decision support in the software ... · Dr. José Raúl Romero Salguero Dr. Sebastián Ventura Soto 24th September 2018. Introduction Architecture discovery

Introduction Architecture discovery Multi/Many perspectives Human-in-the-loop approach Conclusions

Thesis quality indicators

Compendium of publications

A minimum of three articles publishedin journals ranked in the

Journal Citation Report (Q1)

International mention

Research stay (three months) in a foreigninstitution: University of the West ofEngland, under the supervision of

Dr. Christopher L. Simons

Positive evaluation reports by tworesearchers from a different institution

Additional international publications

Aurora Ramírez Quesada Ph.D. Thesis 49 / 50

Page 50: Metaheuristic models for decision support in the software ... · Dr. José Raúl Romero Salguero Dr. Sebastián Ventura Soto 24th September 2018. Introduction Architecture discovery

Metaheuristic models for decision support in the softwareconstruction process

– Ph.D. Thesis –

Aurora Ramírez Quesada

Dept. Computer Science and Numerical AnalysisUniversity of Córdoba

Supervisors:Dr. José Raúl Romero Salguero

Dr. Sebastián Ventura Soto

24th September 2018

Page 51: Metaheuristic models for decision support in the software ... · Dr. José Raúl Romero Salguero Dr. Sebastián Ventura Soto 24th September 2018. Introduction Architecture discovery

Mutation operatorAlgorithm 1 Add a component

Require:

Ensure:

1: �

2: for all component in do

3: candidates�

4: if (random(0,1)> Prob(component)) then

5: allGroups� getGroups(component)

6: if (size(groupsOf Classes) > 1) t hen

7: for all groupOf Classes in allGroups do

8: if (size(groupOf Classes) = min) then

9: � groupOf Classes

10: end if

11: end for

12: end if

13: end if

14: newComp � randomGroup( )

15: � component �

16: �

17: end for

18: if (newComp = ) then

19: �

20: for all component in do

21: for all class in component do

22: if (random(0,1) > 0.5) then

23: � class

24: end if

25: end for

26: � component �

27: newComp �

28: �

29: end for

30: end if

31: � newComp

32: )

33: return

offspring

offspring

offspring

offspring

offspring

offspring

candidates

candidates

candidates

candidates

candidates

candidates

candidates

candidates

offspring

parent

parent

parentoffspring

setInterfacesAndConnectors(

Algorithm 2 Split a component

Require:

Ensure:

1: �

2: �

3: for all component in do

4: if (numberOfGroups(component) > 1) then

5: � component

6: end if

7: end for

8: if (size( ) > 0) then

9: � randomComponent( )

10: for all in do

11: if (random(0,1) > 0.5) then

12: component1�

13: else

14: component2�

15: end if

16: end for

17: else

18: � randomComponent( )

19: for all class in do

20: if (random(0,1) > 0.5) then

21: component1� class

22: else

23: component2� class

24: end if

25: end for

26: end if

27: � �

28: � component1

29: � component2

30: )

31: return

parent

parent

parent

parent

offspring

offspring

offspring

offspringoffspring

offspring

offspring

groupOfClasses

groupOfClasses

groupOfClasses compToSplit

compToSplit

compToSplit

candidates

candidates

candidates

candidatescompToSplit

compToSplit

setInterfacesAndConnectors(

Aurora Ramírez Quesada Ph.D. Thesis 1 / 7

Page 52: Metaheuristic models for decision support in the software ... · Dr. José Raúl Romero Salguero Dr. Sebastián Ventura Soto 24th September 2018. Introduction Architecture discovery

Mutation operator

Algorithm 3 Remove a component

Require:

Ensure:

1: �

2: �

3: maxRel � maxNumExtRel( )

4: for all component in do

5: if (numExtRel(component)=maxRel) then

6: � component

7: end if

8: end for

9: compToRemove � randomComponent( )

10: � � compToRemove11: for all class in compToRemove do

12: randomComponent( )� class

13: end for

14: )

15: return

parent

parent

offspring

offspring

parent

offspring

offspring

offspring

candidates

candidates

candidates

parent

offspring

setInterfacesAndConnectors(

Algorithm 5 Move a class

Require:

Ensure:

1: �

2: � randomComponent( )

3: destination� randomComponent( )

4: � randomClass( )

5: removeClass( , )

6: addClass( , )

7: � �

8: �

9: �

10: )11: return

offspring

offspring

offspringoffspring

offspringoffspring

offspring

parent

parent

parent

parent

origin

origin

origin

(origin

origin

destinationdestination)

destination

class

class

class

setInterfacesAndConnectors(

Algorithm 4 Merge two components

Require:

Ensure:

1: �

2: �

3: maxRel � maxNumExtRel( )

4: for all component in do

5: if (numExtRel(component)=maxRel) then

6: � component

7: end if

8: end for

9: component1� randomComponent( )

10: if (size( ) > 1) then

11: component2� randomComponent( )

12: else

13: component2� randomComponent( )

14: end if

15: � � com ponent1 component2

16: � component1∪component2

17: )

18: return

offspring

offspring

offspringoffspring

offspring

offspring

parent

parent

parent

parent

parent

candidates

candidates

candidates

candidates

candidates

( ∪ )

setInterfacesAndConnectors(

Aurora Ramírez Quesada Ph.D. Thesis 2 / 7

Page 53: Metaheuristic models for decision support in the software ... · Dr. José Raúl Romero Salguero Dr. Sebastián Ventura Soto 24th September 2018. Introduction Architecture discovery

Mutation operator

Initialindividual

Architecture

Components Connectors

Component_1 Component_2 Connector_1

Classes ClassesProvided

interfaces

Required

interfaces

Provided

interfaces

Required

interfaces

Provided

interface

Required

interface

A_prov_E

E_req_A A_prov_EE_req_AA C

DE

GF

H

B

Provided

interface

Required

interface

Connector_2

D_prov_BB_req_D

D_prov_B B_req_D

Component_1 Component_2

A_prov_E

E_req_A

D_prov_B

B_req_D

Add acomponent

Architecture

Components Connectors

Component_1

Component_2

Connector_1

Classes

Classes

Provided

interfaces

Required

interfaces

Provided

interfaces

Required

interfacesProvided

interface

Required

interface

A_prov_E

E_req_A

A_prov_EE_req_AA

C D

E

G

F

H

B

Provided

interface

Required

interface

Connector_2

D_prov_BB_req_DD_prov_B

B_req_D

Component_3

Classes

Provided

interfaces

Required

interfaces

B F

B_req_D

A_prov_E

E_req_A

D_prov_B

B_req_D

Component_3

Component_1

Component_2

Move aclass

Architecture

Components Connectors

Component_1 Component_2 Connector_1

Classes ClassesProvided

interfaces

Required

interfaces

Provided

interfaces

Required

interfacesProvided

interface

Required

interface

A_prov_E

E_req_A A_prov_EE_req_AA

C

E

G

F H

B

Provided

interface

Required

interface

Connector_2

D_prov_BB_req_D

D_prov_B B_req_DB

D

Component_1 Component_2

A_prov_E

E_req_A

Aurora Ramírez Quesada Ph.D. Thesis 3 / 7

Page 54: Metaheuristic models for decision support in the software ... · Dr. José Raúl Romero Salguero Dr. Sebastián Ventura Soto 24th September 2018. Introduction Architecture discovery

Objective functions

Instability

INSi =ECi

ECi + ACiINS =

1n·

n∑i=1

INSi

Encapsulation

ENCi =#classesinner

#classestotalENC =

1n·

n∑i=1

ENCi

Abstractness

ABSi =#classesabstract

#classestotalABS =

1n·

n∑i=1

ABSi

Aurora Ramírez Quesada Ph.D. Thesis 4 / 7

Page 55: Metaheuristic models for decision support in the software ... · Dr. José Raúl Romero Salguero Dr. Sebastián Ventura Soto 24th September 2018. Introduction Architecture discovery

Objective functions

Critical size

CCsizei =

{1 if size(i) > threshold0 otherwise

CS =n∑

i=1

CCsizei

Critical link

CC linki =

{1 if #provided interfacesi > threshold0 otherwise

CL =n∑

i=1

CC linki

Component balance

SB(n) =

n−1µ−1 if n < µ

1− n−µω−µ if µ < n < ω

0 if n ≥ ω

CSU(C) = 1− Gini({volume(c) : c ∈ C})CB = SB(|C|) · CSU(C)

Aurora Ramírez Quesada Ph.D. Thesis 5 / 7

Page 56: Metaheuristic models for decision support in the software ... · Dr. José Raúl Romero Salguero Dr. Sebastián Ventura Soto 24th September 2018. Introduction Architecture discovery

Architectural preferences

Best component

prefbc = max{J(classes(c), classes(c+))} ∀c ∈ [1, n] J(A,B) = |A ∩ B|/|A ∪ B|

Worst component

prefwc = max{1− J(classes(c), classes(c−))} ∀c ∈ [1, n]

Best provided interface

prefbi = max{J(operations(interface(c)), operations(p+))} ∀c ∈ [1, n]

Worst provided interface

prefwi = max{1− J(operations(interface(c)), operations(p−))} ∀c ∈ [1, n]

Aurora Ramírez Quesada Ph.D. Thesis 6 / 7

Page 57: Metaheuristic models for decision support in the software ... · Dr. José Raúl Romero Salguero Dr. Sebastián Ventura Soto 24th September 2018. Introduction Architecture discovery

Architectural preferences

Number of components

prefnc =

{(n − nmin)/(n+ − nmin) if n < n+

1− ((n − n+)/(nmax − n)) if n ≥ n+

Metric in a range

prefmr =

0 if ms < mmin

1− (ms −mmid )/mmid if ms ∈ [mmin,mmax ]

0 if ms > mmax

mmid = (mmax −mmin)/2

Aspiration levels

prefal =

{1 if ASF ≤ 01− ASF if ASF > 0

ASF = max{wk · (f sk − z∗k )}

Aurora Ramírez Quesada Ph.D. Thesis 7 / 7