interactive requirements prioritization - francis palma,...
TRANSCRIPT
Interactive Requirements Prioritization Using Search Based Optimization Technique and Constraint Solver
Acknowledgement: Dr. Paolo Tonella and Dr. Angelo Susi
University of Trento, Italy Fondazione Bruno Kessler (FBK), Trento, Italy
Francis Palma [email protected]
Ecole Polytechnique de Montreal
PhD. Supervisor: Dr. Yann-Gaël Guéhéneuc, Ecole Polytechnique Co-supervisor: Dr. Naouel Moha, UQAM
Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
Problem: ‘Prioritization of Requirements’ To find the best ordering of requirements in each successive release to ensure quality & value of the delivered system, trade-off constraints & end-user satisfaction
Do Trade-off between user needs & real constraints
Now, highest quality & best valued system!
Extracted from Requirement Documents
Problem Description: An Example Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
2 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
q Problem description
q Related works
q Genetic Algorithm
q Approach A (Search Based Technique)
q Case study
q Results A
q Approach B (Constraint Solver: SMT)
q Results B
q Conclusions
Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
3 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
Requirements Prioritization?
A task performed in Software Project Management to determine which candidate requirements should be included in a certain release, negotiating all technical & non-technical constraints.
Requirements Prioritization matters, why?
Expectations ~ high; Timelines ~ short; Resources ~ limited, Make sure product contains the most essential functions, maximizing customer satisfaction and best resource utilization.
Prioritization dimensions may include:
Stakeholder expectations, business value, risk, cost, difficulty, time-constraints, dependencies among requirements etc.
Requirements Prioritization Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
4 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
(1) Acquisition and coding of set of Requirements and Domain Knowledge; (2) Apply A Prioritization Technique (3) Output of the ranking
Problem Description (cont.)
Find it! Why Ordered List? Order of implementation satisfy the developers’ constraints & delivers maximum value to the user.
Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
5 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
req/software analyst
Classification: State-of-the-Art approaches
Prioritization Approaches Classification
using user knowledge either performing pair-wise comparison or not
using domain knowledge
using both
ü Domain Knowledge: Encoded, reusable requirements details or requirement specifications ü User Knowledge: When Requirement Analyst aware of complete domain knowledge
Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
6 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
State-of-the-Art approaches (cont) Pairwise Comparison based approaches Analytic Hierarchy Process (AHP): comparing all unique pairs of requirements to determine which of the two is of higher priority, and to what extent [T.L. Saaty, ‘80]
Bubble Sort: compares two requirements & swap them if they are in the wrong order [Karlsson, ‘98] Cost-Value Approach: each requirement is determined on (i) the value to the users (ii) the cost of implementing, also uses the AHP technique [Karlsson & Ryan, ‘97] Case Based Ranking (CBRank): exploits a machine learning algorithm to guide the elicitation of user preferences during the prioritization process [Paolo Avesani et al ‘05]
Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
7 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
State-of-the-Art approaches (cont.)
Non-Pairwise comparison based approaches Numerical Assignment: grouping requirements in different priority groups [Bradner, ‘97] MoScoW: groups all requirements into four priority groups MUST have, SHOULD have, COULD have, and WON’T have [Tudor and Walter, ‘06] Simple Ranking: requirements are ranked from integer 1 to N [Berander & Andrews, ‘05] Binary Search Tree: each node represents a requirement, lower priority requirements placed in left subtree, higher priority ones in right subtree, than node priority [Karlsson, ‘98] $100 Method: each stakeholder is asked to assume having $100 to distribute over the requirements in a ratio scale [Berander & Andrews ‘05]
Combining Techniques based approaches Planning Game: combination of two prioritization techniques i.e. Numerical Assignment & Simple Ranking [Beck, ‘99]
Domain Knowledge based approaches Priority Groups: dividing requirements into separate groups. then groups are ranked by using AHP [Joachim Karlsson] Genetic Algorithm: optimization is an application of GA & used in the problem of requirements prioritization too; uses domain knowledge [John Holland]
Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
8 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
State-of-the-Art approaches (cont)
Summary Approach Cons
AHP Scalability Bubble Sort Scalability, time CBRank unable to accept constraints i.e. Dep Cost-Value Approach Time consuming
BST Sensitivity; a single error may build wrong tree GA Can’t resolve contradictory; $100 Method Longer; less confidence, biased MoScoW Ambiguous final ordering Simple Ranking unable handling complex scenarios
Scalability & handling of technical dependencies are the very common problems of all state-of-art approaches!
Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
9 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
Approach A
Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
10 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
Domain Knowledge: Prio & Dep
Domain knowledge includes two precedence graphs • Prio • Dep
How to build them? (encoded graph)
Prio: For 1st release, initial priorities Dep: For 1st release, dependencies
Prio
Dep
Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
11 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
Genetic Algorithm 1. Acquisition and coding of set of Requirements and Domain Knowledge 2. Apply Genetic Algorithm (no user knowledge) 3. Output of the ranking (the most promising individual)
Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
12 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
Genetic Algorithm Pseudo Code
The Canonical GA (a simple pseudo code is presented here):
1. choose initial population 2. evaluate each individual’s fitness REPEAT: 3. select best-ranking individuals to reproduce 4. apply crossover operator 5. apply mutation operator 6. evaluate each individual’s fitness until terminating condition (e.g. until at least one individual has the
desired fitness or enough generations have passed)
Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
13 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
What is Population & Individuals?
Indv. ID Individual Disagree Pr1 < R1, R3, R2, R4, R5, R6, R7, R8, R9 >
Pr2 < R2, R3, R4, R1, R5, R8, R6, R7, R9 >
Pr3 < R5, R2, R1, R3, R7, R8, R6, R9, R4 >
Pr4 < R4, R5, R6, R3, R2, R1, R8, R9, R7 >
Pr5 < R7, R8, R6, R5, R2, R3, R4, R9, R1 >
Pr6 < R5, R6, R7, R8, R9, R1, R2, R3, R4 >
Pr7 < R9, R8, R7, R6, R5, R4, R3, R2, R1 >
Pr8 < R8, R9, R6, R7, R4, R5, R2, R3, R1 >
Pr9 < R1, R3, R5, R7, R9, R2, R4, R6, R8 >
Pr10 < R1, R4, R2, R3, R5, R6, R9, R8, R7 >
R1 R3 R2 R4 R5 R6 R7 R8 R9
Population!
A set of solution candidates
Pr1
An individual, also a solution candidate A requirement as a gene
Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
14 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
Defining Individual’s Fitness?
Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
15 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
Domain Knowledge Encoding Transform the domain knowledge into graphs
Req Priorities Dependencies R1 10 R2, R3, R7 R2 20 R3 R3 10 R4 20 R5 30 R4, R8 R6 40 R7 30 R3 R8 20 R9 50 R8, R5
Prio
Dep
Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
16 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
Definition: Disagreement Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
Disagreement: Between a pair of ordering (i.e. Pr1 and Prio | Dep | Eli), disagreement is the count of pairs that are inverted in two orderings. § Lower disagreement defines higher fitness
Formally define:
17 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver !
dis(pr1, pr2) = {(r,s)" pr1* | (s,r)" pr2*}
of 63
Disagreement Calculation Indv. ID Individual Disagr
Pr1 < R1, R3, R2, R4, R5, R6, R7, R8, R9 > 4 Pr2 < R2, R3, R4, R1, R5, R8, R6, R7, R9 >
Pr3 < R5, R2, R1, R3, R7, R8, R6, R9, R4 >
Pr4 < R4, R5, R6, R3, R2, R1, R8, R9, R7 >
Pr5 < R7, R8, R6, R5, R2, R3, R4, R9, R1 >
Pr6 < R5, R6, R7, R8, R9, R1, R2, R3, R4 >
Pr7 < R9, R8, R7, R6, R5, R4, R3, R2, R1 >
Pr8 < R8, R9, R6, R7, R4, R5, R2, R3, R1 >
Pr9 < R1, R3, R5, R7, R9, R2, R4, R6, R8 >
Pr10 < R1, R4, R2, R3, R5, R6, R9, R8, R7 >
R1 R3 R2 R4 R5 R6 R7 R8 R9
Conflicts = {(R5, R8), (R6, R7), (R6, R8), (R7, R8)}
Prio
For Pr1
Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
18 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
Disagreement Calculation (cont) Indv. ID Individuals Disagr.
Pr1 < R1, R3, R2, R4, R5, R6, R7, R8, R9 > 4 Pr2 < R2, R3, R4, R1, R5, R8, R6, R7, R9 >
Pr3 < R5, R2, R1, R3, R7, R8, R6, R9, R4 >
Pr4 < R4, R5, R6, R3, R2, R1, R8, R9, R7 >
Pr5 < R7, R8, R6, R5, R2, R3, R4, R9, R1 >
Pr6 < R5, R6, R7, R8, R9, R1, R2, R3, R4 >
Pr7 < R9, R8, R7, R6, R5, R4, R3, R2, R1 >
Pr8 < R8, R9, R6, R7, R4, R5, R2, R3, R1 >
Pr9 < R1, R3, R5, R7, R9, R2, R4, R6, R8 >
Pr10 < R1, R4, R2, R3, R5, R6, R9, R8, R7 >
R1 R3 R2 R4 R5 R6 R7 R8 R9
Conflicts = {(R1, R3), (R1, R7), (R1, R2), (R5, R8)}
Dep
For Pr1
Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
19 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
Disagreement Calculation (cont)
Indv. ID Individuals Disagr Pr1 < R1, R3, R2, R4, R5, R6, R7, R8, R9 > 8 Pr2 < R2, R3, R4, R1, R5, R8, R6, R7, R9 > 8
Pr3 < R5, R2, R1, R3, R7, R8, R6, R9, R4 > 16
Pr4 < R4, R5, R6, R3, R2, R1, R8, R9, R7 > 15
Pr5 < R7, R8, R6, R5, R2, R3, R4, R9, R1 > 23
Pr6 < R5, R6, R7, R8, R9, R1, R2, R3, R4 > 29
Pr7 < R9, R8, R7, R6, R5, R4, R3, R2, R1 > 30
Pr8 < R8, R9, R6, R7, R4, R5, R2, R3, R1 > 29
Pr9 < R1, R3, R5, R7, R9, R2, R4, R6, R8 > 17
Pr10 < R1, R4, R2, R3, R5, R6, R9, R8, R7 > 13
Total Conflicts = 4+4 = 8 {(R5, R8), Prio (R6, R7), (R6, R8), (R7, R8), (R1, R3), Dep (R1, R7), (R1, R2), (R5, R8)}
and so on…
Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
20 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
GA Crossover Operator We used cut-head/fill-in-tail and cut-tail/fill-in-head…
R2 R3
Positions 5-6 as cut points
R8
Pr2
Pr3
Pr2’
R4 R1 R5 R8 R6 R7 R9
R5 R8 R1 R3 R7 R2 R9 R4 R6
R7 R9 R6 R2 R3 R4 R1 R5
- variation allows searching out different available niches, find better fitness values and subsequently better solutions - never produce chromosomes containing duplicate genes.
Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
21 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
GA Mutation Operator
Mutation is basically a swap operator, we used requirement-pair-swap
R2 R3
Pr2 R4 R1 R5 R8 R6 R7 R9
Pr2’ R2 R3 R6 R1 R5 R8 R4 R7 R9
- mutation causes movement in the search space - may produce a stronger chromosome.
Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
22 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
GA Selection Operator We used the Tournament Selection for our approach
- allows the selection pressure to be easily adjusted - faster than other selection operators i.e. Roulette Wheel Selection - better convergence
Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
23 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
Our IGA Approach 1. Acquisition and coding of set of Requirements and Domain Knowledge 2. Apply Interactive Genetic Algorithm (exploiting User Knowledge) 3. Output of the ranking (the most promising individual)
Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
24 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
Our Approach: Pairs to be Elicited Indv.
ID Individuals Disagree
Pr1 < R1, R3, R2, R4, R5, R6, R7, R8, R9 > 8 Pr2 < R2, R3, R4, R1, R5, R8, R6, R7, R9 > 8 Pr10 < R1, R4, R2, R3, R5, R6, R9, R8, R7 > 13
Pr4 < R4, R5, R6, R3, R2, R1, R8, R9, R7 > 15
Pr3 < R5, R2, R1, R3, R7, R8, R6, R9, R4 > 16
Pr9 < R1, R3, R5, R7, R9, R2, R4, R6, R8 > 17
Pr5 < R7, R8, R6, R5, R2, R3, R4, R9, R1 > 23
Pr8 < R8, R9, R6, R7, R4, R5, R2, R3, R1 > 29
Pr6 < R5, R6, R7, R8, R9, R1, R2, R3, R4 > 29
Pr7 < R9, R8, R7, R6, R5, R4, R3, R2, R1 > 30
Ranked individuals w. r. t. disagreement
Pr1 = < R1, R3, R2, R4, R5, R6, R7, R8, R9 > vs Pr2 = < R2, R3, R4, R1, R5, R8, R6, R7, R9 >
TIE PAIRS
Pr1, Pr2 (R1, R3), (R2, R3), (R6, R8), (R7, R8)
Candidate pairs to be asked to the decision maker
Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
25 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
Our Approach: User Feedback
Ties appears in population, - Contradictory information w.r.t. the initial constraints - Nothing is expressed explicitly in the initial constraints. etc... ... Simple example: Why (R7, R8)? Contradictory or Ambiguous w.r.t. Prio & Dep..
TIE PAIRS
Pr1, Pr2 (R1, R3), (R2, R3), (R6, R8), (R7, R8)
Prio
Dep
Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
26 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
Our Approach: User Feedback TIE PAIRS
Pr1, Pr2 (R1, R3), (R2, R3), (R6, R8), (R7, R8)
Eli
User Preference Graph eliOrd
Experience &
knowledge ‘<‘ or ‘>’
So, user knowledge is playing important role…
Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
27 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
Our Approach: New Round with New Constraints
§ The new evolved population after using GA operators on population:
ü Crossover ü Mutation ü Selection
§ Compared against the new set of constraints graphs
Eli Dep Prio
Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
28 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
Evaluation of the Approach A
Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
29 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
The Case Study: Target System (ACube) § Prioritize requirements for a real software system, as part of the project *ACube (Ambient Aware Assistance)
designing a highly technological monitoring environment to be deployed in nursing homes to support medical and assistance staff
§ After end-user requirements analysis phase, 60 user requirements and 49 technical requirements Four macro-scenarios have been identified.
ID Macro-Scenario # of requirements
FALL Monitoring falls 26
ESC Monitoring escapes 23
MON Monitoring dangerous behavior 21
ALL The three scenarios 49
*ACube is a social welfare project coordinated by Fondazione Bruno Kessler (FBK) and funded by Autonomous Province of Trento under Bando Grandi Progetti, 2006. *http://acube.fbk.eu/
Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
30 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
Case Study: The Overall Process Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
31 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
§ Collected user and technical requirements § Identified macro-scenarios § Tech. requirements has priority values (range 1-500) defined as a function that indicates priority-level of the tech. requirement w.r.t. the priority of the user requirements § Software architect analyzed the requirement documents & extracted requirement implementation dependencies § Software architect also defined a Gold Standard (GS) i.e. the ordering that would be followed at the end --- ---
§ With Prio info, Dep info and GS, we applied IGA Algorithm and generated Eli info during runtime, and evolve… § IGA returns an ordering applying GA operators with user’s (Requirements Analyst) assistance. § We then measure fitness against Gold Standard (GS)
of 63
Gold Standard (GS)
§ For each of the four macro-scenarios, we obtained the Gold Standard (GS) prioritization from the Software Architect of the ACube project
ü The GS prioritization is the ordering given by the software architect to the requirements when planning their implementation during the ACube project.
Why Gold Standard?
ü To measure disagreements with respect to GS. ü To evaluate our approach in terms of disagreement against other non-interactive approaches using the same GS.
Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
32 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
Research Questions
Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
33 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
Research Question - 1 RQ1 (Convergence) Can we observe convergence with respect to the finally elicited fitness function?
- Convergence is not obvious immediately, as Eli Graph is evolving at early stages. - Although the full fitness function is known only at end of elicitation process.
YES!
Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
34 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
Research Question - 2 RQ2 (Role of interaction) Does IGA produce improved prioritizations compared to non-interactive requirements ordering?
IGA outperforms substantially GA (and RAND), especially when a higher number of pairwise comparisons can be carried out
YES!
Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
35 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
Research Question - 3 RQ3 (Role of initial precedence constraints) How does initial availability of precedence constraints affect the performance of IGA?
- Different types of Domain Knowledge affect IGA significantly - The improvement of IGA over GA is even higher when limited ranking information is available
Improves!
Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
36 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
Research Question - 4 RQ4 (Robustness) Is IGA robust with respect to errors committed by the user during the elicitation of pairwise comparisons?
IGA can tolerate user errors up to 20%.
YES!
Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
37 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
General Discussion § Up to now we propose an Interactive approach for requirements prioritization
§ We experimented on a real project i.e. ACube
§ We also verified robustness w.r.t other non-interactive approaches Analytical Hierarchy Process (AHP):
- Most common approach for requirements prioritization… - Decision maker decides for every unique pairs based on different criteria… - For a system with N requirements and C criteria, AHP will have, total C × (C − 1) + C × N × (N − 1) / 2 comparisons (a huge manual task!) So, only with 25/50/100 (max) pairwise comparisons and acceptable performance IGA makes AHP obsolete
Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
38 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
Approach B Using Constraints Solver: SMT
Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
39 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
• Based on Sa$sfiability Modulo Theory* and on Interac9ve Pairwise User Feedback – aims at minimizing the disagreement between
• a total order of prioritized requirements (Gold Standard) • constraints that are either encoded with the requirements
or expressed iteratively by the user during prioritization process
* a decision problem for logical formulas expressed in classical
first-order logic with equality or inequality
We try to overcome some of the limits of other approaches: exploiting user knowledge, minimizing the requests to users to pay attention to scalability problems, considering the possibility of arbitrary constraints and assuring robustness to errors
Our Approach Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
40 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
• A set of Requirements
• Domain Knowledge: available as requirements documenta9on (e.g., cost of the implementa9on, value for the stakeholders, dependencies between requirements) that can be converted into total or par9al rankings of the requirements
Req Cost Dependencies
R1 High R2, R3 R2 Low R3 R3 Low
R4 Medium R3 R5 Medium
• Evaluation from users in terms of orderings between pairs of requirements
The Input Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
41 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
1. Acquisi9on and coding of Domain Knowledge for the set of requirements under analysis
2. Computa9on of solu9ons via SMT, also exploi9ng evalua9ons from decision makers
3. Output of the ranking
R3 R2
R1 R4
R1 R2
R3
R4
R5
Domain knowledge (constraints) User feedback
R1 R2 R3 R4
Set of requirements
Interac<ve SMT
R2
R1
R3
R4
<,> ?
The Main Approach Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
42 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
2.1. Computa9on of a set of solu9ons via SMT solver (Yices* tool) 2.2. If the number of solu9ons is grater than 1
– then: iden9fica9on of conflicts (9es) between the solu9ons and the constraints
– else: exit and return the solu9on 2.3. Request of knowledge to users to decide about conflicts 2.4. If max number of itera9ons / max pair comparison
– then: exit and return the solu9on – else: 2.1
The SMT Interactive Algorithm
*h^p://yices.csl.sri.com/
Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
43 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
Approach Details
Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
44 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
Req Value (Priori<es) Dependencies
R1 High R2, R3 R2 Low R3 R3 Low
R4 Medium R3 R5 Medium
Transform the domain knowledge into graphs
R3 R2
R1
R5 R4
Priorities
R3 R2
R1 R4
Dependencies
Domain Knowledge coding into Graphs Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
45 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
requirements and the various constraints that are either en-coded with the requirements or that are expressed itera-tively by the user during the prioritization process. Weuse SMT solvers to achieve such a minimization, takingadvantage of interactive input from the requirement engi-neer whenever the solver produces more than one prioritizedlist of requirements having the same disagreement with theavailable constraints. The prioritization process terminateswhen a unique solution at minimum disagreement is foundor the maximum allowed number of pairwise comparisons isreached.
Req Prio DepsR1 High R2, R3
R2 Low R3
R3 LowR4 Medium R2
R5 Medium
4 5
1
2 3
R
R R
R R
3 2
1 4
R R
R R
Prio Deps
Table 2: Requirements with priority and dependen-cies
Let us consider the five requirements listed in Table 2.For each requirement we consider the priority expressed bythe requirement engineer who collected them and the depen-dencies the requirement may have with other requirements.For conciseness, in Table 2 we omit other important ele-ments of a requirement (e.g., the textual description). Con-straints upon requirements can be represented by means ofa constraint graph. In a constraint graph, an edge betweentwo requirements indicates that, according to the relateddependency or priority, the requirement associated with thesource of the edge should be implemented before the targetrequirement. Edges may be weighted, to actually quantifythe strength or importance of each constraint. When en-coded in the input language of the SMT solver, such con-straints will be retractable constraints which are given theweight labeling the edge (or 1 if no weight is given). Aninfinite weight is used for constraints that must necessarilyhold in the final ordering of the requirements. These will benon-retractable constraints for the SMT solver.
At the right of Table 2, we show the constraint graphinduced by the priority (Prio) property of the requirementsand the constraint graph induced by the dependencies (Deps)between requirements. Requirements with High priority shouldprecede those with Medium priority. Hence the edges (R1, R4)and (R1, R5) in the graph Prio. Similarly, the precedence be-tween Medium and Low priority requirements induces thefour edges at the bottom of the constraint graph Prio. Re-quirement R1 depends on R2, R3, hence the implementationof R2, R3 should precede that of R1, which gives raise to theedges (R2, R1) and (R3, R1) in Deps.
It is possible to encode the requirements prioritizationproblem as a MAX-SAT2 problem. An SMT solver willfind an integer assignment which maximizes the weight ofthe retractable constraints that are satisfied by the solution(minimum cost of unsatisfied constraints). The encoding isdescribed formally in the next section. Intuitively, it con-sists of an assignment of positions (integers between 1 and2Technically we are exploiting a MAX-SMT solver.
(set-evidence! true)(define x::(-> nat nat))(define N::nat 5)
;; Only permutations of positions are allowed(assert (forall (i::(subrange 1 N)) (and (>= (x i) 1) (<= (x i) N))))(assert (forall (i::(subrange 1 (- N 1)))
(forall (j::(subrange (+ i 1) N)) (/= (x i) (x j)))))
;; Prio(assert+ (< (x 1) (x 4)) 1)(assert+ (< (x 1) (x 5)) 1)(assert+ (< (x 1) (x 2)) 1)(assert+ (< (x 1) (x 3)) 1)(assert+ (< (x 4) (x 2)) 1)(assert+ (< (x 4) (x 3)) 1)(assert+ (< (x 5) (x 2)) 1)(assert+ (< (x 5) (x 3)) 1)
;;Deps(assert+ (< (x 3) (x 2)) 1)(assert+ (< (x 3) (x 1)) 1)(assert+ (< (x 3) (x 4)) 1)(assert+ (< (x 2) (x 1)) 1)(assert+ (< (x 2) (x 4)) 1)
(max-sat)
Figure 1: Encoding of the constraints in Table 2 forthe SMT solver Yices
N , for N requirements) to the components of an integerarray x of size N . The assignment must define a permu-tation of the positions, hence there should be no repetitionof positions. This is easily encoded as a set of inequalitiesbetween pairs of positions (x[i] != x[j] " i != j). The otherconstraints (from the constraint graphs) are encoded as in-equalities (e.g., x[1] < x[4] and x[1] < x[5] for the two edgesat the top of the constraint graph Prio). The solutions tothe MAX-SAT problem produced by the SMT solver are allrequirement orderings that violate the minimum number ofretractable constraints (e.g., Deps and Prio), or have min-imum cost of violation, in case di!erent weights are givento di!erent constraints. The encoding of the requirementprioritization problem in Table 2 for the SMT solver Yices3
is shown in Figure 1.
Id Reqs DisagreePr1 < R1, R5, R4, R3, R2 > 4Pr2 < R1, R4, R5, R3, R2 > 4Pr3 < R1, R5, R3, R4, R2 > 4Pr4 < R1, R5, R3, R2, R4 > 4
Table 3: Prioritized requirements and related mini-mum disagreement
After encoding the constraint graphs in Table 2 as re-tractable assertions (assert+ in Figure 1) and running anSMT solver (e.g., Yices) on them, we obtain the four priori-tized lists of requirements shown in Table 3 (the solver mustbe run four times to obtain the complete set of prioritiza-tions, each time negating the previously found solutions).These are all prioritized lists having the same, minimumcost of retracted assertions (i.e., 4). Such a cost is called dis-agreement and it represents the number of constraints thatare not respected by the resulting requirement positions.The first two prioritizations Pr1 and Pr2 are in complete
3http://yices.csl.sri.com/
R3 R2
R1 R4
Dependencies
R3 R2
R1
R5 R4
Priorities
Retractable assertions
Problem SMT Encoding
Retractable assertions: assert+ can be retracted. (details on http://yices.csl.sri.com/)
Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
46 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
(assert+ [expr] [weight])
of 63
requirements and the various constraints that are either en-coded with the requirements or that are expressed itera-tively by the user during the prioritization process. Weuse SMT solvers to achieve such a minimization, takingadvantage of interactive input from the requirement engi-neer whenever the solver produces more than one prioritizedlist of requirements having the same disagreement with theavailable constraints. The prioritization process terminateswhen a unique solution at minimum disagreement is foundor the maximum allowed number of pairwise comparisons isreached.
Req Prio DepsR1 High R2, R3
R2 Low R3
R3 LowR4 Medium R2
R5 Medium
4 5
1
2 3
R
R R
R R
3 2
1 4
R R
R R
Prio Deps
Table 2: Requirements with priority and dependen-cies
Let us consider the five requirements listed in Table 2.For each requirement we consider the priority expressed bythe requirement engineer who collected them and the depen-dencies the requirement may have with other requirements.For conciseness, in Table 2 we omit other important ele-ments of a requirement (e.g., the textual description). Con-straints upon requirements can be represented by means ofa constraint graph. In a constraint graph, an edge betweentwo requirements indicates that, according to the relateddependency or priority, the requirement associated with thesource of the edge should be implemented before the targetrequirement. Edges may be weighted, to actually quantifythe strength or importance of each constraint. When en-coded in the input language of the SMT solver, such con-straints will be retractable constraints which are given theweight labeling the edge (or 1 if no weight is given). Aninfinite weight is used for constraints that must necessarilyhold in the final ordering of the requirements. These will benon-retractable constraints for the SMT solver.
At the right of Table 2, we show the constraint graphinduced by the priority (Prio) property of the requirementsand the constraint graph induced by the dependencies (Deps)between requirements. Requirements with High priority shouldprecede those with Medium priority. Hence the edges (R1, R4)and (R1, R5) in the graph Prio. Similarly, the precedence be-tween Medium and Low priority requirements induces thefour edges at the bottom of the constraint graph Prio. Re-quirement R1 depends on R2, R3, hence the implementationof R2, R3 should precede that of R1, which gives raise to theedges (R2, R1) and (R3, R1) in Deps.
It is possible to encode the requirements prioritizationproblem as a MAX-SAT2 problem. An SMT solver willfind an integer assignment which maximizes the weight ofthe retractable constraints that are satisfied by the solution(minimum cost of unsatisfied constraints). The encoding isdescribed formally in the next section. Intuitively, it con-sists of an assignment of positions (integers between 1 and2Technically we are exploiting a MAX-SMT solver.
(set-evidence! true)(define x::(-> nat nat))(define N::nat 5)
;; Only permutations of positions are allowed(assert (forall (i::(subrange 1 N)) (and (>= (x i) 1) (<= (x i) N))))(assert (forall (i::(subrange 1 (- N 1)))
(forall (j::(subrange (+ i 1) N)) (/= (x i) (x j)))))
;; Prio(assert+ (< (x 1) (x 4)) 1)(assert+ (< (x 1) (x 5)) 1)(assert+ (< (x 1) (x 2)) 1)(assert+ (< (x 1) (x 3)) 1)(assert+ (< (x 4) (x 2)) 1)(assert+ (< (x 4) (x 3)) 1)(assert+ (< (x 5) (x 2)) 1)(assert+ (< (x 5) (x 3)) 1)
;;Deps(assert+ (< (x 3) (x 2)) 1)(assert+ (< (x 3) (x 1)) 1)(assert+ (< (x 3) (x 4)) 1)(assert+ (< (x 2) (x 1)) 1)(assert+ (< (x 2) (x 4)) 1)
(max-sat)
Figure 1: Encoding of the constraints in Table 2 forthe SMT solver Yices
N , for N requirements) to the components of an integerarray x of size N . The assignment must define a permu-tation of the positions, hence there should be no repetitionof positions. This is easily encoded as a set of inequalitiesbetween pairs of positions (x[i] != x[j] " i != j). The otherconstraints (from the constraint graphs) are encoded as in-equalities (e.g., x[1] < x[4] and x[1] < x[5] for the two edgesat the top of the constraint graph Prio). The solutions tothe MAX-SAT problem produced by the SMT solver are allrequirement orderings that violate the minimum number ofretractable constraints (e.g., Deps and Prio), or have min-imum cost of violation, in case di!erent weights are givento di!erent constraints. The encoding of the requirementprioritization problem in Table 2 for the SMT solver Yices3
is shown in Figure 1.
Id Reqs DisagreePr1 < R1, R5, R4, R3, R2 > 4Pr2 < R1, R4, R5, R3, R2 > 4Pr3 < R1, R5, R3, R4, R2 > 4Pr4 < R1, R5, R3, R2, R4 > 4
Table 3: Prioritized requirements and related mini-mum disagreement
After encoding the constraint graphs in Table 2 as re-tractable assertions (assert+ in Figure 1) and running anSMT solver (e.g., Yices) on them, we obtain the four priori-tized lists of requirements shown in Table 3 (the solver mustbe run four times to obtain the complete set of prioritiza-tions, each time negating the previously found solutions).These are all prioritized lists having the same, minimumcost of retracted assertions (i.e., 4). Such a cost is called dis-agreement and it represents the number of constraints thatare not respected by the resulting requirement positions.The first two prioritizations Pr1 and Pr2 are in complete
3http://yices.csl.sri.com/
R3 R2
R1 R4
Dependencies
R3 R2
R1
R5 R4
Priorities
Completing the Encoding Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
47 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
solu<on Requirements rankings Disagree
Pr1 < R1,R5,R4,R3,R2 >
Pr2 < R1,R4,R5,R3,R2 >
Pr3 < R1,R5,R3,R4,R2 >
Pr4 < R1,R5,R3,R2,R4 >
R1
R5
R4
R3
R2
Conflicts = {(R3, R1), (R2, R1), (R2, R4), (R3, R4)}
!
dis(pr1, pr2) = {(r,s)" pr1* | (s,r)" pr2*}
R3 R2
R1 R4
Dep
R1
R5
R4
R3
R2
4
4
4
4
Production of Solutions & Identification of Conflicts via SMT
Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
48 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
Prio
of 63
solu<on Requirements rankings Disagree
Pr1 < R1,R5,R4,R3,R2 >
Pr2 < R1,R4,R5,R3,R2 >
Pr3 < R1,R5,R3,R4,R2 >
Pr4 < R1,R5,R3,R2,R4 >
TIE PAIRS
Pr1, Pr2, (R4, R5)
Pr1, Pr3 (R3, R4)
… …
PR1 = < R1,R5,R4,R3,R2 > vs PR2 = < R1,R4,R5,R3,R2 > (R4,R5)
Candidate pairs to be asked to decision maker
Solutions at minimum disagreement
4
4
4
4
Pairs to be Evaluated from the Decision Maker Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
49 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
R3 R2
R1
R5 R4
Priorities
R3 R2
R1 R4
Dependencies
Why (R4,R5) ? Nothing is said about (R4,R5) in the Priorities and Dependencies graphs
Why (R3,R4) ?
Contradiction in ordering (R3,R4)
TIE PAIRS
Pr1, Pr2, (R4, R5)
Pr1, Pr3 (R3, R4)
… … <>?
R3
R4 R4
R5
User Preference Graph eliOrd
< or >
User Feedback Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
50 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
R3 R2
R1
R5 R4
Priorities
R3 R2
R1 R4
Dependencies
R3
R4
R5
User Preference Graph eliOrd
New Graph eliOrd from User Feedback Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
51 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
Evaluation of Approach B
52
• Priori9ze requirements for a real soaware system, as part of the project ACube (Ambient Aware Assistance) – designing a highly technological monitoring environment to be deployed in nursing
homes to support medical and assistance staff • Aaer user requirements analysis phase,
– 60 user requirements where retrieved that produced 49 technical requirements – Four macro-‐scenarios have been iden9fied.
Id Macro-‐scenario # of requirements
FALL Monitoring falls of pa9ents 26
ESC Monitoring escapes of pa9ents 23
MON Monitoring dangerous behavior of pa9ents 21
ALL The three scenarios 49
Case Study: ACube Project Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
53 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
• Two sets of technical constraints: – Prio graph: a func9on that associates technical requirement to a number
indica9ng the priority of the technical requirement with respect to the priority of the user requirements it is intended to address
– Dep graph: is defined on the basis of the dependencies between technical requirements
• For each of the four macro-‐scenarios, we considered as the Gold Standard (GS) the priori9za9on from the soaware architect of the ACube project – The GS priori9za9on is the ordering given by the soaware architect to the
requirements when planning their implementa9on during the ACube project
• Comparison with “Incomplete Analy9c Hierarchy Process” and “Interac9ve Gene9c Algorithm”
Ingredients for the Evaluation Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
54 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
• Is a variant of the Analy9c Hierarchy Process (AHP) • Pairwise comparison itera9ve process in which the decision
maker is required to specify a preference between two requirements Ri and Rj as an integer value pij∈[1 . . . 9]
coming from other sources.We evaluated our SMT approach comparing it with other
state-of-the-art interactive prioritization techniques (IAHPand IGA in particular). Results indicate that SMT substan-tially outperforms them, while keeping the user e!ort (interms of number of elicited pairs) acceptable. Moreover, wecompared interactive and non-interactive prioritization, ob-serving an increased performance when interaction with thedecision maker takes place. We also evaluated the robust-ness of our method with respect to decision maker’s errors.
This paper is structured as follows: in Section 2 we de-scribe the background for our work, Section 3 presents theproposed approach and algorithm. In Section 4 we commenta set of experimental evaluations on e!ectiveness, robust-ness and e"ciency of the approach. The empirical assess-ment was conducted on a set of requirements from a realhealthcare project. Relevant related works are discussed inSection 5. Conclusions and future work are presented inSection 6.
2. BACKGROUNDHere we briefly introduce two quite successful, state-of-
the-art methods for interactive prioritization of requirements:Incomplete Analytic Hierarchy Process (IAHP) [4] and In-teractive Genetic Algorithm (IGA) [14]. Both methods havethe objective of synthesizing an approximation of the rank-ing of a set of N requirements, minimizing the e!ort asso-ciated with the interactive input required from the decisionmaker (consisting of pairwise comparisons).
2.1 Incomplete AHPIAHP [4] is a variant of the Analytic Hierarchy Process
(AHP) [12] that can deal with incomplete information (pair-wise comparisons) from the user. It was designed to supportscalability of AHP to requirements whose size make AHPprohibitively expensive (the number of pairs that must beelicited to apply AHP is quadratic with the number of re-quirements).
In particular, AHP implements a pairwise comparison it-erative process in which the decision maker is required tospecify a preference as an integer value pij ! [1 . . . 9], be-tween two requirements Ri and Rj . The value can be mappedto a qualitative measure of the preference relation, as sum-marized in Table 1. After assigning a preference value to oneof the two requirements (say, Ri) compared to the other one(Rj), the preference value for the least preferred requirement(Rj) is the reciprocal of pij , so, pji = 1/pij . When all thepairs have been elicited, a ranking is synthesized throughthe computation of the principal eigenvector of the matrixpij , the components of which determine the rank of eachrequirement.
Preference pij Definition
1 Ri Equally important to Rj
3 Ri Moderately more important than Rj
5 Ri Strongly more important than Rj
7 Ri Very strongly more important than Rj
9 Ri Extremely more important than Rj
2, 4, 6, 8 For compromise between the above values
Table 1: A possible fundamental scale in the interval[1 . . . 9] used with AHP
IAHP overcomes the AHP scalability problem by mini-mizing the number of pairs elicited from the decision maker,while maintaining a good trade-o! between the precision ofthe final solution and the e!ort of the decision maker. Thisis done by calculating, at each iteration of the elicitationprocess, a prediction of the next most promising pair to beasked to the decision maker in order to find a stable andgood approximation of the target ranking, early before elic-iting all the N(N "1)/2 pairs for the set of N requirements.
The high level steps of the IAHP process are: (i) thedecision maker provides N "1 judgments which form a con-nected graph; (ii) using the available pairwise comparisons,the missing comparisons are estimated by taking the geomet-ric mean of the intensities over a spanning tree (this resultsin a weight matrix); (iii) the derivatives of the weight matrixwith respect to the missing matrix elements are computedand the next pairwise comparison to ask is determined basedon such derivatives; (iv) if the stopping criteria are met (e.g.,maximum number of elicitations or convergence metrics be-low threshold), then the computation stops and a rank isproduced by calculating the principal eigenvector of the es-timated matrix; otherwise the selected comparison is elicitedand the algorithm iterates.
2.2 Interactive GAIn [14] we presented an Interactive Genetic Algorithm
(IGA) to achieve the minimization of the disagreement be-tween a total order of prioritized requirements and the var-ious constraints coming from the domain knowledge, thatare either encoded with the requirements or expressed iter-atively by the user during the prioritization process.
In the approach we exploit the interactive input from theuser whenever the fitness function cannot be computed pre-cisely based on the information available. Specifically, eachindividual in the population being evolved by the genetic al-gorithm represents an alternative prioritization Pri of theN requirements. Two examples of such individuals are:Pr1 = #R1, R5, R4, R3, R2$, or Pr2 = #R1, R4, R5, R3, R2$.When individuals having a high fitness (i.e., a low disagree-ment with the constraints) cannot be distinguished, sincetheir fitness function evaluates to a plateau, user input isrequested interactively, in the form of preferences betweenpairs of requirements, so specifying if Ri is before or afterRj in the user ideal ranking, so as to make the fitness func-tion landscape better suited for further minimization. Inthis way, knowledge about the constraints in the domaincan help the method to minimize the amount of knowledgeelicited from the decision maker, so decreasing her/his ef-fort. The prioritization process terminates when a low dis-agreement is reached, a time out is reached or the allocatedelicitation budget is over.
3. APPROACHThe prioritization approach we propose aims at minimiz-
ing the disagreement between a total order of prioritizedrequirements and the various constraints that are either en-coded with the requirements or that are expressed itera-tively by the user during the prioritization process. Weuse SMT solvers to achieve such a minimization, takingadvantage of interactive input from the requirement engi-neer whenever the solver produces more than one prioritizedlist of requirements having the same disagreement with theavailable constraints. The prioritization process terminates
**P. T. Harker. Incomplete Pairwise Comparisons in the Analytic Hierarchy Process. Math. Modelling, 9(11):837 – 848, 1987 **T. L. Saaty and L. G. Vargas. Models, Methods, Concepts & Applications of the Analytic Hierarchy Process. Kluwer Academic, 2000
I-AHP Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
55 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
• The IGA method consists in a Gene<c Algorithm associated to a Pairwise Interac<ve and Itera<ve process to acquire user preferences.
• Differently from IAHP, the priori9za9on elicited by the user is strict (Ri less or more important than Rj, no range as to be specified)
P. Tonella, A. Susi, and F. Palma. Using Interactive GA for Requirements Prioritization. In 2nd International Symposium on Search Based Software Engineering, pages 57–66. IEEE, 2010
Interactive Genetic Algorithm Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
56 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
RQ1 (Comparison) Does the SMT-based method produce improved prioritizations compared to IAHP and IGA?
Figure 2: Disagreement with GS after eliciting atmost 25 (top), 50 (middle) and 100 (bottom) pairsfrom the user in the ALL scenario
RQ3 (Robustness).In Figure 4, we show that even if the performance of the
SMT-based interactive approach degrades at increasing usererror rates, it always produces an improved prioritizationcompared to IGA and IAHP with a user who does not makeany mistake. We show this for 50 elicited pairs, but sim-ilar results are available for a di!erent number of elicitedpairs. With no error SMT produces a prioritization with adisagreement of 90. At 5%, 10% and 20% error rate, thedisagreement becomes 92, 95 and 98 respectively. On theother hand, even if the user makes no error (i.e. error-freeuser), the disagreement of the prioritizations produced byIGA and IAHP is 120 and 208. We obtained similar plotsfor the other three macro scenarios (MON, FALL, ESC).
In Figure 5 we compare the robustness of SMT with thetwo other approaches considered in this work, IGA and IAHP.Here we show the results obtained for ALL. Similar resultshave been obtained for the other macro scenarios. The SMT-based interactive approach is substantially more robust thanthe others. At increasing user error rates, the performanceof IGA and IAHP is degrading to a higher degree. SMT
Figure 3: Disagreement (top) and average dis-tance (bottom) with GS for interactive and non-interactive SMT after eliciting 25, 50, 100 pairs fromthe user in the ALL scenario
has also a degrading performance at increased error rates,but it outperforms IGA and IAHP to a large extent. For anerror rate of 5% SMT, IGA and IAHP have a disagreementof 92, 120 and 236 respectively. For a 10% error rate, theseare 95, 119 and 277. For an error rate of 20%, the disagree-ments with GS become 98, 123 and 331 respectively. So,clearly the SMT-based interactive approach is more robustthen IGA and IAHP.
RQ4 (Performance).Table 8 shows the comparison among execution times of
SMT, IGA and IAHP. All execution times in the table areobtained from the system time of each execution and aremeasured in seconds, with a very small observable variabil-
Figure 4: Robustness of SMT: Disagreement forSMT at di!erent user error rates; IGA and IAHPwith error-free user, after eliciting 50 pairs in theALL scenario
!
dis(pr1, pr2) = {(r,s)" pr1* | (s,r)" pr2*}
RQ1: Comparison SMT, IGA, IAHP Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
57 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
RQ2 (Role of interaction) Does SMT-based interactive prioritization produce improved prioritizations compared to non-interactive SMT-based prioritization?
Figure 2: Disagreement with GS after eliciting atmost 25 (top), 50 (middle) and 100 (bottom) pairsfrom the user in the ALL scenario
RQ3 (Robustness).In Figure 4, we show that even if the performance of the
SMT-based interactive approach degrades at increasing usererror rates, it always produces an improved prioritizationcompared to IGA and IAHP with a user who does not makeany mistake. We show this for 50 elicited pairs, but sim-ilar results are available for a di!erent number of elicitedpairs. With no error SMT produces a prioritization with adisagreement of 90. At 5%, 10% and 20% error rate, thedisagreement becomes 92, 95 and 98 respectively. On theother hand, even if the user makes no error (i.e. error-freeuser), the disagreement of the prioritizations produced byIGA and IAHP is 120 and 208. We obtained similar plotsfor the other three macro scenarios (MON, FALL, ESC).
In Figure 5 we compare the robustness of SMT with thetwo other approaches considered in this work, IGA and IAHP.Here we show the results obtained for ALL. Similar resultshave been obtained for the other macro scenarios. The SMT-based interactive approach is substantially more robust thanthe others. At increasing user error rates, the performanceof IGA and IAHP is degrading to a higher degree. SMT
Figure 3: Disagreement (top) and average dis-tance (bottom) with GS for interactive and non-interactive SMT after eliciting 25, 50, 100 pairs fromthe user in the ALL scenario
has also a degrading performance at increased error rates,but it outperforms IGA and IAHP to a large extent. For anerror rate of 5% SMT, IGA and IAHP have a disagreementof 92, 120 and 236 respectively. For a 10% error rate, theseare 95, 119 and 277. For an error rate of 20%, the disagree-ments with GS become 98, 123 and 331 respectively. So,clearly the SMT-based interactive approach is more robustthen IGA and IAHP.
RQ4 (Performance).Table 8 shows the comparison among execution times of
SMT, IGA and IAHP. All execution times in the table areobtained from the system time of each execution and aremeasured in seconds, with a very small observable variabil-
Figure 4: Robustness of SMT: Disagreement forSMT at di!erent user error rates; IGA and IAHPwith error-free user, after eliciting 50 pairs in theALL scenario
!
dis(pr1, pr2) = {(r,s)" pr1* | (s,r)" pr2*}
RQ2: Role of Interaction Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
58 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
RQ3 (Robustness) Is the SMT-based method more robust than IAHP and IGA with respect to errors committed by the user during the elicitation of pairwise comparisons?
Figure 2: Disagreement with GS after eliciting atmost 25 (top), 50 (middle) and 100 (bottom) pairsfrom the user in the ALL scenario
RQ3 (Robustness).In Figure 4, we show that even if the performance of the
SMT-based interactive approach degrades at increasing usererror rates, it always produces an improved prioritizationcompared to IGA and IAHP with a user who does not makeany mistake. We show this for 50 elicited pairs, but sim-ilar results are available for a di!erent number of elicitedpairs. With no error SMT produces a prioritization with adisagreement of 90. At 5%, 10% and 20% error rate, thedisagreement becomes 92, 95 and 98 respectively. On theother hand, even if the user makes no error (i.e. error-freeuser), the disagreement of the prioritizations produced byIGA and IAHP is 120 and 208. We obtained similar plotsfor the other three macro scenarios (MON, FALL, ESC).
In Figure 5 we compare the robustness of SMT with thetwo other approaches considered in this work, IGA and IAHP.Here we show the results obtained for ALL. Similar resultshave been obtained for the other macro scenarios. The SMT-based interactive approach is substantially more robust thanthe others. At increasing user error rates, the performanceof IGA and IAHP is degrading to a higher degree. SMT
Figure 3: Disagreement (top) and average dis-tance (bottom) with GS for interactive and non-interactive SMT after eliciting 25, 50, 100 pairs fromthe user in the ALL scenario
has also a degrading performance at increased error rates,but it outperforms IGA and IAHP to a large extent. For anerror rate of 5% SMT, IGA and IAHP have a disagreementof 92, 120 and 236 respectively. For a 10% error rate, theseare 95, 119 and 277. For an error rate of 20%, the disagree-ments with GS become 98, 123 and 331 respectively. So,clearly the SMT-based interactive approach is more robustthen IGA and IAHP.
RQ4 (Performance).Table 8 shows the comparison among execution times of
SMT, IGA and IAHP. All execution times in the table areobtained from the system time of each execution and aremeasured in seconds, with a very small observable variabil-
Figure 4: Robustness of SMT: Disagreement forSMT at di!erent user error rates; IGA and IAHPwith error-free user, after eliciting 50 pairs in theALL scenario
!
dis(pr1, pr2) = {(r,s)" pr1* | (s,r)" pr2*}
RQ3: Robustness Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
59 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
• Again cost/benefit trade off offered by Interac9ve SMT as compared to IAHP and IGA is more acceptable
Threats
• External validity (generaliza9on of the findings): – we used only 1 real project with 4 sub-‐scenarios
• Construct Validity: – Other performances measures should be considered (not only disagreement)
– More complex user error models should be considered
Discussion Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
60 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
Algorithm selection
Experiment parameters setting
Pairwise elicitation
The Prioritization Tool & Technical Report
TR-FBK-SE-2011-3: Francis Palma, Angelo Susi, Paolo Tonella: Using an SMT Solver for Interactive Requirements Prioritization. Tech. Rep. FBK-SE (Mar. 2011) 61 of 63
Conclusions & Future Works q We proposed an Interactive Genetic Algorithm & an Interactive SMT to collect pairwise information useful to prioritize the requirements
q We also verified the robustness of the algorithms with respect to increased error-prone responses from analysts
q We evaluated the approach in a real project (ACube)
q In summary, we contributed NOVEL APPROACH to prioritize requirements What’s Next? Experiment
- On-line: Empirical Study with real object (i.e. human/analyst)
Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
62 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
Publications
- Paolo Tonella, Angelo Susi, and Francis Palma. Using Interactive GA for Requirements Prioritization. In 2nd International Symposium on Search Based Software Engineering (SSBSE 2010), pages 57–66. IEEE, 2010 (Under minor review for the Journal of Information and Software Technology (IST)) - Francis Palma, Angelo Susi, Paolo Tonella, Using an SMT solver for interactive requirements prioritization, in Tibor Gyimothy and Andreas Zeller, SIGSOFT FSE, ACM, 2011, pp. 48-58 (19th ACM SIGSOFT Symposium on the Foundations of Software Engineering, Szeged, Hungary)da 09/05/2011 a 09/09/2011
Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
63 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver
of 63
Thank You! …and
Questions?
Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions
64