using interactive ga for requirements prioritization

Using Interactive GA for Requirements Prioritization

Paolo Tonella, Angelo Susi, Francis Palma

Fondazione Bruno Kessler Software Engineering Research Unit

Trento, Italy

{tonella | susi}@fbk.eu, [email protected]

Outline

• The problem of requirements prioritization• Approaches• Our Interactive Genetic Algorithm approach

– The inputs (requirements, Domain knowledge / constraints, feedback from human evaluators )

– The algorithm and evolution laws • Algorithm explained via an execution

• Exploiting the approach on the field• Conclusions

2 Benevento, Italy, September 7th, 2010

The problem

• Prioritization of requirements: the activity of finding an order relation on the set of requirements under analysis

• Prioritize according to information concerning the • available budget • time constraints • stakeholder expectations • technical constraints


Sorting approaches The rank of a requirement can be expressed as its relative position with respect to

the other requirements in the set, as in Bubble sort or Binary search procedures Or as an absolute measure of the evaluation criterion for the requirement, as in

Cumulative votingThe methods are highly sensible to errorsPairwise approaches

Analytic Hierarchy Process (AHP): stat-of-the-art pairwise comparison approach (considers only user feedback on the set of alternative requirements)

The method is not scalableCBRank: pairwise approach based on Machine Learning techniques that take into

account the user feedback and the domain knowledgeThis algorithm does not allow to represent some kinds of constraints categories (such as

dependencies)Genetic algorithm

Genetic Algorithm as NSGA- II algorithm It that does not consider the possibility of exploiting user feedback

Approaches


• Based on Interactive Genetic Algorithm (IGA)– aims at minimizing the disagreement between a total

order of prioritized requirements and the various constraints that are either encoded with the requirements or that are expressed iteratively by the user during the prioritization process

We try to overcame the limits of the other approaches: considering user knowledge, minimizing the requests to users to pay attention to scalability problems, considering the possibility of arbitrary constraints and assuring robustness to errors

Our Approach


• Set of Requirements

• Domain Knowledge: available as requirements documentation (e.g., cost of the implementation, value for the stakeholders, dependencies between requirements) that can be converted into total or partial rankings of the requirements

• Evaluation from users in terms of orderings between pairs of requirements

The Input


1. Acquisition and coding of set of Requirements and Domain Knowledge2. Interactive Genetic Algorithm: computation of solutions (individuals), also

exploiting evaluations from users3. Output of the ranking (the most promising individual)

The process


R3 R2

R1 R4

R1 R2

R3

R4

R5

Domain knowledge (constraints)

User feedback

R1 R2 R3 R4

Set of requirements

Interactive GeneticAlgorithm

R2

R1

R3

R4

2.1. computation of a first set of solutions (individuals)2.2. identification of conflicts (ties) between individuals and

constraints2.3. request of knowledge to users (to decide about conflicts)2.4. computation of new solutions

– via evolution rules

2.5. If max number of iterations – than exit – else 2.2

The IGA algorithm: step 2.


Executing the algorithm


Req Priorities Dependencies

R1 High R2, R3

R2 Low R3

R3 Low

R4 Medium R3

R5 Medium

Domain Knowledge coding into graphs


Transform the domain knowledge into graphs

R3R2

R1

R5R4

Priorities

R3 R2

R1 R4

Dependencies

Individual ID

Requirements rankings (Individual)

Disagree

Pr1 < R3,R2,R1,R4,R5 >

Pr2 < R3,R2,R1,R5,R4 >

Pr3 < R1,R3,R2,R4,R5 >

Pr4 < R2,R3,R1,R4,R5 >

Pr5 < R2,R3,R4,R5,R1 >

Pr6 < R2,R3,R5,R4,R1 >

Production of individuals and identification of conflicts


R3R2

R1

R5R4

Priorities

R3

R2

R1

R4

R5

Conflicts ={(R3, R1), (R3, R4), (R3, R5)}

dis(pr1, pr2) {(r,s) pr1* | (r,s) pr2*}

Individual ID

Requirements rankings (Individual)

Disagree

Pr1 < R3,R2,R1,R4,R5 >

Pr2 < R3,R2,R1,R5,R4 >

Pr3 < R1,R3,R2,R4,R5 >

Pr4 < R2,R3,R1,R4,R5 >

Pr5 < R2,R3,R4,R5,R1 >

Pr6 < R2,R3,R5,R4,R1 >

Production of individuals and identification of conflicts


R3R2

R1

R5R4

Priorities

R3

R2

R1

R4

R5

Conflicts ={(R2, R1), (R2, R4), (R2, R5),(R3, R1), (R3, R4), (R3, R5)}

dis(pr1, pr2) {(r,s) pr1* | (r,s) pr2*}… and so on …

6

6

7

9

9

6

TIE PAIRSPr1, Pr2, Pr3 (R4, R5), (R1, R2), (R1, R3)

Pr5,Pr6 (R4, R5)

Pairs to be evaluated to choosethe individual


Individual ID Requirements rankings (Individual)

Disagree

Pr1 < R3,R2,R1,R4,R5 > 6

Pr2 < R3,R2,R1,R5,R4 > 6

Pr3 < R1,R3,R2,R4,R5 > 6

Pr4 < R2,R3,R1,R4,R5 > 7

Pr5 < R2,R3,R4,R5,R1 > 9

Pr6 < R2,R3,R5,R4,R1 > 9

PR1 = < R3,R2,R1,R4,R5 >vsPR2 = < R3,R2,R1,R5,R4 >

(R4,R5)

Candidate pairs to be asked to decision maker

Ranked individuals with respect to disagreement

User feedback

14

Benevento, Italy, September 7th, 2010

TIE PAIRS

Pr1, Pr2, Pr3 (R4, R5), (R1, R2), (R1, R3)

Pr5,Pr6 (R4, R5)

R3R2

R1

R5R4

Priorities

R3 R2

R1 R4

Dependencies

Why (R4,R5) ? Nothing is said about (R4,R5) in the Priorities and Dependencies graphs

Why (R1,R2) and (R1,R3) ?

<>?

R1R2

R3R4

R5

User Preference Graph eliOrd

< or >

Contradiction(R1,R3)

Contradiction(R1,R2)

Population evolution: crossover


cut-head/fill-in-tail and cut-tail/fill-in-head

R3 R2 R1 R5 R4

R1 R3 R2 R4 R5

Positions 2-3 as cut points to cut-head/fill-in-tail

R3 R2 R1 R4 R5

Pr2

Pr3

Pr2’




R3 R2 R1 R5 R4

R1 R3 R2 R4 R5


R2 R5 R4R1 R3

Pr2

Pr3

Pr3’

Population evolution: mutation


swap operators

R3 R2 R1 R4 R5Pr1

R3 R2 R4 R1 R5Pr1’

R1 and R4 to swap

• The new evolved population

• is compared against the new set of constraints graphs

New round of the algorithm


R3R2

R1

R5R4

Priorities

R3 R2

R1 R4

Dependencies

R1R2

R3R4

R5

User Preference Graph eliOrd

R3 R2 R4 R1 R5Pr1’

R3 R2

R2 R5 R4R1 R3

R1 R4 R5

Pr3’

Pr2’

Evaluating the approach


• Prioritize requirements for a real software system, as part of the project ACube (Ambient Aware Assistance)– designing a highly technological monitoring environment to be deployed in

nursing homes to support medical and assistance staff• After user requirements analysis phase,

– 60 user requirements and 49 technical requirements– Four macro-scenarios have been identified.

Case Study


Id Macro-scenario # of requirements

FALL Monitoring falls 26

ESC Monitoring escapes 23

MON Monitoring dangerous behavior 21

ALL The three scenarios 49

• Two sets of technical constraints: – Priority: a function that associates technical requirement to a number

indicating the priority of the technical requirement with respect to the priority of the user requirements it is intended to address

– Dependency: is defined on the basis of the dependencies between requirements

• For each of the four macro-scenarios, we obtained the Gold Standard (GS) prioritization from the software architect of the ACube project– The GS prioritization is the ordering given by the software architect to

the requirements when planning their implementation during the ACube project

Ingredients


RQ1: convergence


The best individual in each population converges toward a low value of the final fitness function

RQ1 (Convergence) Can we observe convergence with respect to the finally elicited fitness function?

IGA converges even though the fitness function is known in its complete form only at the end of the elicitation process

RQ2: Role of interaction


RQ2 (Role of interaction) Does IGA produce improved prioritizations Compared to non-interactive requirement ordering?

IGA outperforms substantially GA (and RAND), especially when a higher number of pairwise comparisons can be carried out

RQ3: Role of initial precedence constraints


RQ3 (Role of initial precedence constraints) How does initial availability of precedence constraints affect the performance of IGA?

The improvement of IGA over GA is even higher when limited ranking information is available a-priori

RQ4: Robustness


RQ4 (Robustness) Is IGA robust with respect to errors committed by the user during the elicitation of pairwise comparisons?

It seems (more complex error models needed)

• Cost/benefit trade off offered by IGA as compared to AHP is extremely interesting – With an elicitation effort reduced to 10% of the one required by AHP,

IGA produces an approximate ordering which has a quite low average distance from the requirement positions in the GS

Threats• External validity (generalization of the findings):

– we used (only one) real project with four scenarios

• Construct Validity: – disagreement measure is used in other contexts – error model is simple

Discussion


• We proposed an interactive genetic algorithm to collect pairwise information useful to prioritize the requirements for a software system

• Our approach scaled to the size of the considered case study• We also verified the robustness of the algorithm with respect to increasing

user feedback errors• Finally we evaluated the approach in a real project

What’s next?• Algorithm:

– refining the algorithm – the evolution operators (e.g., introducing new heuristics)

• Experiment – off-line: comparisons with other approaches– on-line: controlled experiments on the field

Conclusion and Future


Thank you





R3 R2 R1 R4 R5Pr1

R3 R2 R1 R4 R5Pr1’

R1 and R4 to swap

Outline

• The problem• Research baseline

– GA and IGA approaches– Other approaches (?)

• Our approach– The data

• requirements • Domain knowledge / constraints (existing rankings and existing dependencies, e.g.

from human domain experts, domain laws, …)• Feedback from human evaluators (to solve ties)

– The algorithm and evolution laws (operators) [to let universe evolve towards the fitness specified by domain knowledge + human evaluators]

• Algorithm explained via an execution






R3 R2 R1 R5 R4

R1 R3 R2 R4 R5


R3 R2

R1 R5 R4R1 R3

R2 R4 R5

Pr2

Pr3

Pr3’

Pr2’




R3 R2 R1 R5 R4

R1 R3 R2 R4 R5


R3 R2 R1 R4 R5

Pr2

Pr3

Pr2’




R3 R2 R1 R5 R4

R1 R3 R2 R4 R5


R2 R5 R4R1 R3

Pr2

Pr3

Pr3’



cut-head/fill-in-tail and cut-tail/fill-in-head operators

R3 R2 R1 R5 R4

R1 R3 R2 R4 R5


R3 R2 R R5 R4

R1 R3 R2 R4 R5

Pr2

Pr3

Pr3’

Pr2’

Approaches

• process can be designed as an a-priori or as an a-posteriori process. – a-priori the preferences are formulated before the specification of the

set of requirements via predefined models (e.g., differential equations)

– a-posteriori approaches, the ranking is formulated on the basis of the characteristics of the set of requirements under analysis (e.g., via a process of pairwise comparison allowing to define at the same time which requirement and why it has to be preferred between two alternatives)


Four research questions:RQ1 (Convergence) Can we observe convergence with respect to the finally

elicited fitness function?RQ2 (Role of interaction) Does IGA produce improved prioritizations

compared to non-interactive requirement ordering?RQ3 (Role of initial precedence constraints) How does initial availability of

precedence constraints affect the performance of IGA?RQ4 (Robustness) Is IGA robust with respect to errors committed by the user

during the elicitation of pairwise comparisons?

Research Questions


Outline

• The problem• Research baseline

– GA and IGA approaches– Other approaches (?)

• Our approach– The data (requirements, Domain knowledge / constraints, Feedback from

human evaluators )– The algorithm and evolution laws (operators) [to let universe evolve towards

the fitness specified by domain knowledge + human evaluators]• Algorithm explained via an execution



using interactive ga for requirements prioritization

Technology