MCDA in E-democracy: Why Weight? Comparing Even Swaps and MAVT

Valerie Belton, University of Strathclyde

George Wright, Durham University

Gilberto Montibeller, Kingston University

TED Workshop, Helsinki, May 2005

Outline

Rationale for the talk and motivation for our studies

A brief introduction to the Even Swaps approach

Overview of the studies Findings Lessons for E-Democracy?

Rationale

Our extensive experience with the use of MAVT in practice and in teaching the approach to practicing managers continues to reveal issues in understanding criteria weights

In face to face situations a facilitator can be alert to such difficulties and try to alleviate them, but this is not the case in the context of e-democracy

The consequence may be that potential participants fail to engage, or that meaningless information is provided

The Holy Grail of MCDA?

An interpretation of the notion of “importance” or “weight” which is both psychologically meaningful and operationally well defined?

Or ….

A theoretically well founded and practically usable approach which does not depend on the specification of weights

Could Even Swaps be this approach?

The Even Swaps Approach

Starting point – A Consequences Table Conversion to a Ranking Table (not essential) Progressive simplification of the problem by:

Use of dominance or practical (near) dominance to eliminate alternatives

Use of swaps to equalise performances on a selected criterion allowing the elimination of that criterion

Until only one option remains

Even Swaps Example

Qualifications UG Dip UG 1st UG 2nd MBA MBA UG Dip Qualification

Marketing Experience

2 2 3 5 0 4 Years

Database Experience

2 1 1 2.5 3 3.5 Years

Web experience 0 1 2 2 5 2 Years

Team Working 2 2 3 4 5 25 Point Scale

(5=best)Communications skills

2 2 4 3 4 35 Point Scale

(5=best)

Salary 20 20 19 30 25 30 x £1000 pa

Step 1 – Practical Dominance

Qualifications UG Dip UG 1st UG 2nd MBA MBA UG Dip Qualification

Marketing Experience

2 2 3 5 0 4 Years

Database Experience

2 1 1 2.5 3 3.5 Years

Web experience 0 1 2 2 5 2 Years

Team Working 2 2 3 4 5 25 Point Scale

(5=best)Communications skills

2 2 4 3 4 35 Point Scale

(5=best)

Salary 20 20 19 30 25 30 x £1000 pa

Chris practically dominates

Angela

Step 1 – Practical Dominance

Qualifications UG Dip UG 1st UG 2nd MBA MBA UG Dip Qualification

Marketing Experience

2 2 3 5 0 4 Years

Database Experience

2 1 1 2.5 3 3.5 Years

Web experience 0 1 2 2 5 2 Years

Team Working 2 2 3 4 5 25 Point Scale

(5=best)Communications skills

2 2 4 3 4 35 Point Scale

(5=best)

Salary 20 20 19 30 25 30 x £1000 pa

Chris practically dominates Barry

David practically dominates Freda

Step 2 – 1st Swap

Chris David Ewan

Qualifications UG 2nd MBA MBA Qualification

Marketing Experience 3 5 0 Years

Database Experience 1 2.5 3 Years

Web experience 2 2 5 Years

Team Working 3 4 5 5 Point Scale (5=best)

Communications skills 4 3 4 5 Point Scale (5=best)

Salary 19 30 25 x £1000 pa

Swap 1 – Web Experience vs Marketing ExperienceEqualise scores on Web ExperienceCompensate on Marketing Experience

2

4

Eliminate Web Experience

Multi-Attribute Value Analysis using V.I.S.A

V(a) = i = 1 to N wi v i(a)

Even Swaps

Defining an Even Swap in MAVT terms

Despite the apparent complexity of this judgement it is one that intuitively seems to be psychologically meaningful.

wi (v i(a)new - v i(a)old ) =

wk (v k(a)old - v k(a)new)

Our initial thoughts about Even Swaps

Conceptually attractive Doesn’t require specification of weights Based on easy to understand principles Progressively simplifies the problem

Concerns Focused on micro judgements – loses holistic perspective Once swapping starts no longer comparing real options …

Curiosity Lack of attention in the decision making literature How do real managers / decision makers react?

Context for the studies

Involved full-time and part-time MBA students taking the 6 credit (of 180) core class, “Making Decisions”

Week 1 of a 5 week / 15 hour class How do people make decisions (1.5 hours) Introduce MAVT in class using worked example (1.5 hours)

Week 2 MAVT in practice (process and cases) Start to use MAVT for group project

Week 3 Introduced to Even Swaps in class using worked example Students work through personnel selection exercise using

paper-based pro-forma

Research methodology Study 1 – quantitative orientation

We looked at: Performance on the Even Swaps exercise Responses to questionnaires comparing MAVT and ES, distributed

immediately after the ES exercise Study 2 - more qualitative orientation

Class exercise was seen as an opportunity to practice ES Individual assignment:

Students to re-analyse their group decision task using ES & to “Compare and contrast the MAVT / ES analyses, commenting on

the relative strengths and weaknesses for individual and group decision making”

Comparative evaluations were analysed and categorised by two researchers to identify recurring themes

Performance on the Even Swaps exercise was assessed & recurring mistakes identified

Measure of performance

1st Study: Students did not perform well in the initial Even Swaps exercise Average score = 2.4 (on 1-5 scale), Average % of error- free swaps = 30%

2nd and 3rd studies Better performance, but still approx 30% showing poor understanding

05

1015202530354045

% o

f p

aper

s

Study 1

Study 2

Study 3

Nature of errors & “variations” on the method

Errors / shortcomings Misunderstanding / incorrect application of practical dominance Practical dominance not explained Use of Ranking Table for Swaps Swaps in the wrong direction Swaps not carried through to next step Didn’t equalise scores on base criterion Swapped “across” alternatives or other incorrect swaps Assume order of elimination relates to overall rank (inverse)

Variations Swapped against more than 1 criterion Focus swaps on 1 alternative (to create dominance) Defined “weighted value” table to equalise units

Study 1: Questionnaire

Please tick the box that best represents your opinion of the strengths of each of the two techniques for:

Improving your decision making:

17 questions

Smart/ Visa Poor Excellent

Even Swaps Poor Excellent

Results: Study 1 - Questionnaire

MAVT viewed as stronger than Even Swaps with respect to:

improving decision making quantifying decision making  providing insights into decision making justifying decisions to others challenging initial intuitive decisions documenting how a decision was made reconciling qualitative and quantitative aspects of a decision making decisions involving many attributes making decisions involving many alternatives trading-off costs against benefits enabling sensitivity analysis integrating objective measurement with value judgment.

All results significant at 1% using the Wilcoxon Z-test

Results: Study 1 – Questionnaire - continued

No significant difference was perceived between the two approaches with respect to:

making decisions involving few attributes making decisions involving few alternatives making tradeoffs.

In addition there was no significant difference in the students’ rating of the perceived ease / difficulty of the two approaches, or of their own understanding

Studies 2 & 3: Process of Analysis

Coding of 111 / 74 commentaries by two independent researchers

“The strength of the Even Swaps approach is that it is simple to use and can be carried out manually. However, it may not be suitable for complex decision problems. .. The MAVT approach provides structure for analysing complex problems in a systematic way that can be easily back-tracked, providing an audit trail, transparency and integrity”

Recorded in a spreadsheet

For each cohort the independent analyses were compared and reconciled – incorporating a process of cross-checking and merging concepts

Views of “competent” students retained Finally, concepts matched across studies (56 concepts cited by 10+%)

61 60 59 58 57 56 55 54 53 52 …

suitable for teams/groups/many stakeholders v,s v,xs v v,xs

quick s s s xs,v

Respondent

Qualitative Analysis: Top Twelve Comments

Stu

dy

1 -

No

Stu

dy

2 -

No

Ove

rall

%

MA

VT

/ V

ISA

NO

T M

AV

T /

VIS

A

EV

EN

S

WA

PS

NO

T E

VE

N

SW

AP

S

Ease of Usefast / quick 38 24 50 4 22 36 11 SWAPS not MAVT

simple/straightforward/easy to use 31 23 44 11 3 33 6 Both

Processhigh level of subjectivity 44 31 61 14 4 58 0 SWAPS

Analysis / outputsallows sensitivity analysis 39 30 56 54 0 1 24 MAVT not SWAPS

visual / graphical appeal 27 14 33 33 0 0 7 MAVT

ability to audit results 32 14 37 31 0 12 11MAVT, SWAPS and

not SWAPS

Understanding / insightsfacilitiates shared understanding/ownership/consensus 0 12 34 28 0 1 12 MAVT not SWAPS

When is the method suitable?good / better for simple problems 27 15 34 2 3 33 0 SWAPS

good for complex decisions 38 23 50 41 0 3 20 MAVT not SWAPS

suitable for individual DM 28 24 42 9 5 40 1 SWAPS

Suitable for groups/teams/many stakeholders40 26 54 54 0 2 15 MAVT not SWAPS

Summary Remarks

Although conceptually simple, the Even Swaps approach is not that easy to pick up – only 33% of students made a reasonable attempt in the 1st study, increasing to 70% and 79%, with the opportunity to practice and reflect, in the 2nd and 3rd studies

In the initial evaluation MAVT / V.I.S.A was rated equally or better with respect to all questions posed

In the 2nd and 3rd studies the students perceived the Even Swaps approach as being easy to use, well suited to simple/small scale (in terms of number of options and criteria) individual decisions, but highly subjective

They saw the MAVT/V.I.S.A approach as being better suited for complex decisions and for group working, being visual, permitting sensitivity analysis, facilitating shared understanding and generating ownership / consensus.

Both approaches were seen as simple / straightforward to use MAVT seen to provide an audit trail – views on Even Swaps divided

What can we learn for E-Democracy?

???? Initial thoughts … MAVT better suited for

working with groups …. therefore better suited to ED

BUT …. what is ED? (where does DGDSS stop and ED start?)

Is ED about sharing, comparing or aggregating?

A student assignment

I’m tired and ready for bedAll this swapping has messed up my head

I should have been prudentAnd been a SMART student

I now want that Hammond guy dead

Most Frequent Qualitative Comments (10+%)

Frequency of

com m ents

Nature of

com m ents (% of respondents)

Stu

dy

1: N

o. o

f ti

mes

co

nce

pt

Stu

dy

2: N

o. o

f ti

mes

co

nce

pt

Ove

rall

% o

f p

aper

s ci

tin

g

MA

VT

/ V

ISA

NO

T M

AV

T /

VIS

A

EV

EN

SW

AP

S

NO

T E

VE

N

SW

AP

S

Ease of Usefast / quick 38 24 50 4 22 36 11 SWAPS not MAVT

simple/straightforward/easy to use 31 23 44 11 3 33 6 Both

complicated (… easy to understand) 11 0 9 1 3 1 5 Neither

procedure is clear 19 0 15 8 0 13 0 Both

cumbersome/laborious (for complex decisions) 17 9 21 2 2 18 1 SWAPS

have to cope w ith a lot of information simultaneously 9 0 7 0 2 7 0 SWAPS

risk of manual errors 6 7 11 0 2 11 0 SWAPS

Most Frequent Qualitative Comments (10+%)

Stu

dy

1 -

No

Stu

dy

2 -

No

Ove

rall

%

MA

VT

/ V

ISA

NO

T M

AV

T /

VIS

A

EV

EN

SW

AP

S

NO

T E

VE

N

SW

AP

S

Processscientif ic / logical 11 7 15 14 0 3 3 MAVT

method is structured 18 12 24 20 0 13 1 Both

problem is broken up effectively 13 0 11 8 0 7 1 Both

intuitive 5 10 3 0 2 6 MAVT Not SWAPS

counterintuitive (... intuitive) 7

high level of subjectivity 44 31 61 14 4 58 0 SWAPS

handles subjectivity better / manages subjectivity 18 0 15 14 0 1 0 MAVT

decision gets simpler as you progress 8 3 9 0 0 9 0 SWAPS

quickly identify / eliminate w eaker alternatives 9 0 7 0 2 7 0 SWAPS

DM forced to address all criteria 9 0 7 6 0 5 1 Both

every step deliberated thoroughly 9 0 7 6 0 2 0 Both

makes you focus on specif ic choices 8 0 7 0 0 7 0 SWAPS

data can be manipulated for personal agenda 14 3 14 2 2 13 1 SWAPS

Most Frequent Qualitative Comments (10+%)

Stu

dy

1 -

No

Stu

dy

2 -

No

Ove

rall

%

MA

VT

/ V

ISA

NO

T M

AV

T /

VIS

A

EV

EN

SW

AP

S

NO

T E

VE

N

SW

AP

S

Problem structuringmethod gives good structure to problem 25 0 20 19 0 9 2 Both

method helps to identify criteria 8 0 7 5 0 2 2 MAVT

problem could be broken up effectively (in groups) 7 0 6 5 0 0 1 MAVT

Trade-offsmaking trade-offs/sw aps can be dif f icult 18 13 25 0 1 23 1 SWAPS

w eighting not allow ed/not explicit 9 11 16 0 7 14 0 SWAPS not MAVT

forced to assess trade-offs 16 6 18 6 2 15 1 Both

sw aps dif f icult in a group 0 13 11 0 0 11 0 SWAPS

Most Frequent Qualitative Comments (10+%)

Stu

dy

1 -

No

Stu

dy

2 -

No

Ove

rall

%

MA

VT

/ V

ISA

NO

T M

AV

T /

VIS

A

EV

EN

SW

AP

S

NO

T E

VE

N

SW

AP

S

Analysis / outputsallows sensitivity analysis 39 30 56 54 0 1 24 MAVT not SWAPS

flexible/ can rew ork 18 0 24 20 1 2 14 MAVT not SWAPS

Inflexible/difficult to rework 12

visual / graphical appeal 27 14 33 33 0 0 7 MAVT

good for presentation and communication 0 7 6 6 0 0 0 MAVT

ability to audit results 32 14 37 31 0 12 11MAVT, SWAPS and

not SWAPS

easy to justify decision to others 25 5 24 21 1 3 10 MAVT not SWAPS

result is robust 23 5 23 14 0 2 12 MAVT not SWAPS

no info about 2nd / other good alternatives 7 0 6 0 2 5 0 SWAPS not MAVT

more accompanying information 8 0 7 4 0 0 2 MAVT not SWAPS

Most Frequent Qualitative Comments (10+%)

Stu

dy

1 -

No

Stu

dy

2 -

No

Ove

rall

%

MA

VT

/ V

ISA

NO

T M

AV

T /

VIS

A

EV

EN

SW

AP

S

NO

T E

VE

N

SW

AP

S

Understanding / insightsmethod broadens your understanding of problem 14 0 9 7 0 1 1 MAVT

gain further insight during analysis 9 0 7 7 0 2 0 MAVT

facilitiates shared understanding/ownership/consensus 0 12 34 28 0 1 12 MAVT not SWAPS

promotes a common understanding of the problem 11

easier to gain consensus 19

provides a forum for negotiating / resolving conf lict 8 0 7 7 0 0 0 MAVT

Most Frequent Qualitative Comments (10+%)

Stu

dy

1 -

No

Stu

dy

2 -

No

Ove

rall

%

MA

VT

/ V

ISA

NO

T M

AV

T /

VIS

A

EV

EN

SW

AP

S

NO

T E

VE

N S

WA

PS

When is the method suitable?good / better for simple problems 27 15 34 2 3 33 0 SWAPS

good for complex decisions 38 23 50 41 0 3 20 MAVT not SWAPS

suitable for many options 0 9 7 3 0 0 7 MAVT not SWAPS

good if many crteria 0 20 16 11 0 0 10 MAVT not SWAPS

can handle complex criteria 15 0 12 7 0 1 9 MAVT not SWAPS

not good for big problems (many alternatives / criteria) 5 0 4 1 1 3 0 SWAPS

good for quantitative criteria 8 5 11 0 0 11 0 SWAPS

suitable for individual DM 28 24 42 9 5 40 1 SWAPS

Suitable for groups/teams/many stakeholders40 26 54 54 0 2 15 MAVT not SWAPS

inclusive process, takes account of all view s 9 15 20 15 0 1 5 MAVT

copes better w ith uncertainty / risk / missing information 8 0 7 5 1 0 4 MAVT not SWAPS

Most Frequent Qualitative Comments (10+%)

Stu

dy

1 -

No

Stu

dy

2 -

No

Ove

rall

%

MA

VT

/ V

ISA

NO

T M

AV

T /

VIS

A

EV

EN

SW

AP

S

NO

T E

VE

N S

WA

PS

Requirements 0 0 0 0Need training/experience 15 9 20 6 1 15 0 Both

need more technical know ledge / expertise 12 0 10 10 0 0 1 MAVT

needs good understanding of consequences 5 9 11 1 0 10 1 SWAPS

requires experienced facilitation 13 7 16 14 0 3 2 MAVT

need appropriate softw are 21 8 29 17 0 0 26 MAVT not SWAPS

only requires pen and paper 7