1

A Methodological Framework for the Construction of Composite Indicators

Michaela Saisana

michaela.saisana@jrc.it

European CommissionJoint Research Centre

Italy

COST 356 – EST Seminar

Oslo, Feb 20, 2008

2

"The mission of the JRC is to provide customer-driven scientific and technical support for the conception, development,

implementation and monitoring of EU policies. As a service of the European Commission, the JRC functions as a

reference centre of science and technology for the Union. Close to the policy-making process, it serves the

common interest of the Member States, while being independent of special

interests, whether private or national."

Roland SchenkelDirector General

DG JRC Robust Science for Policy Making

3

Technical definition: Composite indicators are mathematical combinations (or aggregations) of a set of indicators.

Conceptual definition: “Composite indicators are based on sub-indicators that have no common meaningful unit of measurement and there is no obvious way of weighting these sub-indicators” [*]

[*] Note on composite indicators, EC, Brussels, March 2002

What is a composite indicator?

4

A list of new “Structural Indicators” to be developed by the EC (Information Note to the College of ECFIN October 2005)

1. Price convergence between EU Members States 2. Healthy Life Years3. Biodiversity4. Urban population exposure to air pollution by ozone and 5. Urban population exposure to air pollution by particles (PM10) 6. Consumption of toxic chemicals 7. Generation of hazardous waste 8. Recycling rate of selected materials 9. Resource productivity10. E-business indicator

Can you guess how many of these are composite indicators?

5

ALL OF THEM! (One is a ratio of composites)1. Price convergence between EU Members States 2. Healthy Life Years3. Biodiversity4. Urban population exposure to air pollution by ozone and5. Urban population exposure to air pollution by particles (PM10) 6. Consumption of toxic chemicals 7. Generation of hazardous waste 8. Recycling rate of selected materials 9. Resource productivity: The definition of this indicator has now been

established as the ratio of Gross Domestic Product (GDP, at constant prices) over Domestic Material Consumption (DMC).

10. E-business indicator

… and yet their use within and outside the EC is controversial.

6

<< […] it is hard to imagine that debate on the use of composite indicators will ever be settled […] official statisticians may tend to resent composite indicators, whereby a lot of work in data collection and editing is “wasted” or “hidden” behind a single number of dubious significance.

On the other hand, the temptation of stakeholders and practitioners to summarise complex and sometime elusive processes (e.g. sustainability, single market policy, etc.) into a single figure to benchmark country performance for policy consumption seems likewise irresistible. >> [*]

[*] Saisana M., Saltelli A., Tarantola S. (2005) Uncertainty and Sensitivity analysis techniques as tools for the quality assessment of composite indicators, Journal of the Royal Statistical Society - A, 168(2), 307-323.

Composite indicators’ controversy

7

Handbook on Constructing Composite Indicators:

Methodology & User Guide

Nardo, Saisana, Saltelli and Tarantola (EC/JRC),

Hoffman and Giovannini (OECD), OECD Statistics Working Paper

JT00188147, STD/DOC(2005)3.

http://composite-indicators.jrc.ec.europa.eu/

OECD - JRC Handbook

8

Examples of Good Practices

The Alcohol Policy Index(New York Medical College)

Pilot 2006 EPI (Yale & Columbia univ.)

Trade & Development Index(UNCTAD)

http://farmweb.jrc.cec.eu.int/ci

Composite Learning Index(Canadian Council on Learning)

9

The Alcohol Policy Index(New York Medical College)

Framework(WHO report)

Results

Policy messageSensitivity analysis

10

y = 0.7691x + 20.249R2 = 0.6979

40

50

60

70

80

90

100

40 50 60 70 80 90 100

Composite Learning Index

Econ

omic

and

Soc

ial W

ell-b

eing

The Composite Learning Index(Canadian Council on Learning)

Results

Policy messageSensitivity analysis

Composite Learning Index(Canadian Council on Learning)

Framework

Scenario

Pillar Structure Normalisation Weighting Aggregation

CLI Preserved z-scores FA within pillar, Regression weights to Factors, FA pillars, Regression weights to pillars

Linear

S1 Preserved z-scores FA within pillar, FA pillars Linear S2 Preserved Min-max FA within pillar, FA pillars Linear S3 Not preserved z-scores FA all indicators Linear S4 Not preserved Min-max FA all indicators Linear S5 Preserved z-scores FA within pillar, EW pillars Linear S6 Preserved Min-max FA within pillar, EW pillars Linear S7 Not preserved z-scores EW all indicators Linear S8 Not preserved Min-max EW all indicators Linear S9 Preserved z-scores EW within pillar, EW pillars Linear S10 Preserved Min-max EW within pillar, EW pillars Linear S11 Preserved z-scores FA within pillar, FA pillars Geometric S12 Preserved Min-max FA within pillar, FA pillars Geometric S13 Not preserved z-scores FA all indicators Geometric S14 Not preserved Min-max FA all indicators Geometric S15 Preserved z-scores FA within pillar, EW pillars Geometric S16 Preserved Min-max FA within pillar, EW pillars Geometric S17 Not preserved z-scores EW all indicators Geometric S18 Not preserved Min-max EW all indicators Geometric S19 Preserved z-scores EW within pillar, EW pillars Geometric S20 Preserved Min-max EW within pillar, EW pillars Geometric S21 Preserved Raw data FA within pillar, FA pillars Multi-criteria S22 Not preserved Raw data FA all indicators Multi-criteria S23 Preserved Raw data FA within pillar, EW pillars Multi-criteria S24 Not preserved Raw data EW all indicators Multi-criteria S25 Preserved Raw data EW within pillar, EW pillars Multi-criteria

11

Results

Sensitivity analysis

Pilot 2006 EPI (Yale & Columbia univ.)

Framework

The Environmental Performance Index(Yale and Columbia University)

Policy message

12

stakeholder involvement

Steps in the Development of an Index

Step 1. Developing a theoretical framework

Step 2. Selecting indicators

Step 3. Multivariate analysis

Step 4. Imputation of missing data

Step 5. Normalisation of data

Step 6. Weighting and aggregation

Step 7. Robustness and sensitivity

Step 8. Links to other variables

Step 9. Back to the details

Step 10. Presentation and dissemination

13

Past efforts to develop quantitative environmental metrics, such as the 2005 Environmental Sustainability Index, have also been criticized for their lack of focus on the specific environmental choices for which governments can be held accountable.

The environmental dimension of the UN Millennium Development Goalshas been criticized in the past for being insufficiently defined and inadequately measured.

The Environmental Performance Index report attempts to address these criticisms and thus serves as a complement to the Environmental Sustainability Index.

Step 1. Theoretical framework

14

Formally released in Davos, at the annual meeting of the WEF, 26/01/06

Aim: It tracks actual results for a core set of 16 environmental issues forwhich governments can be held accountable.

Sensitivity Analysis by JRC-IPSC (M. Saisana, A. Saltelli)

15

?

Step 1. Theoretical frameworkExample: The 2006 Pilot EPI

16

A composite indicator is above all the sum of its parts…

Excerpt: The strengths and weaknesses of composite indicators largely derive from the quality of the underlying variables. […] While the choice of indicators must be guided by the theoretical framework for the composite,the data selection process can be quite subjective as there may be no single definitive set of indicators.

Step 2. Selecting variables

17

Step 2. Selecting variablesExample: The 2006 Pilot EPI

18

Multivariate statistic: suitability of dataset, understanding of implications of methodological choices (e.g. weighting, aggregation) during the construction phase of the composite indicator.

In the analysis, the statistical information inherent in the indicators’ set can be dealt with grouping information along the two dimensions of the dataset, i.e. along indicators and along constituencies (e.g. countries, regions, sectors, etc.), not independently of each other.

Step 3. Multivariate analysis

19

Step 4. Imputation of missing data

Missing data are present in almost all composite indicators

Dealing with missing data (3 generic approaches):

• case deletionremoves either country or indicator from the analysis single

imputation (e.g. Mean/Median/Mode substitution, Regression Imputation, Expectation-Maximisation Imputation, etc.)

multiple imputation (e.g. Markov Chain Monte Carlo algorithm).

20

RankingStandardizationRe-scalingDistance to reference countryCategorical scalesCyclical indicatorsBalance of opinions

Step 5. Normalisation of indicators

21

Select the proper method:

Take into account properties of dataAre there hard and/or soft data available?Does exceptional behaviour need to be rewarded or penalised?

Objectives of the composite indicatorDoes information on absolute levels matter?Is benchmarking against a reference country requested?Does the variance in the indicators need to be accountedfor?

Step 5. Normalisation of indicators

For example in presence of extreme values we shouldprefer normalisationmethods that are based on standard deviation or distance from the mean

22

Equal weightsWeights based on statistical models

Principal component/Factor analysis Data envelopment analysis Regression approach Unobserved components models

Weights based on opinions: participatory methodsBudget allocation Public opinionAnalytic hierarchy process Conjoint analysis

Step 6. Weighting (and aggregation)

23

Weight restrictions

Foster and Sen « On Economic Inequality »:« While the possibility of arriving at a unique set of weights is rather unlikely, that uniqueness is not really necessary to make agreed judgments in manysituations, and may indeed not even be required for a complete ordering »

Step 6. Weighting (and aggregation)

24

Equal weights

Proper normalisation of indicators is needed• Works well if all dimensions (e.g. economic, social, environmental, etc.) have the same number of indicators(e.g. TAI example)• Otherwise, equal weighting implies a higher weight to the dimension represented by the larger number of indicators. • Appealing when high correlation of indicators does not mean redundancy of information in the composite, i.e. when correlated components explain different aspects of the picture the composite aims to capture.

25

Two crucial problems:

1. weights assigned to sub-indicators are based on correlations which do not necessarily correspond to the underlying relationships between the indicators and the phenomena being measured.

[ confusion between correlation-redundancy: redundancy implies correlation but the reverse is not necessarily true]

2. disciplines homogeneity rather than representing plurality (PCA only applied when variables are correlated).

Weights based on Principal Component Analysis

26

Advantages:• Participatory techniqueLimitations:• Design of the survey - choice of experts (number, background)• Circular thinking• Up to 8-10 indicators• Not transferable from one area to another

Weights based on Budget Allocation

27

0.109 0.0980.0850.102

0.072

0.209

0.0450.063

0.178

0.107

0.063

0.1840.181

0.1100.148

0.246

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

Patents Royalties Internet Techexports

Telephones Electricity Schooling Universityst.

Budget AllocationAnalytic Hierarchy Process

Weights based on BAL & AHP

Budget allocation provides weights that are closer to the average than Analytic Hierarchy Process

28

Aggregation rules:

Linear aggregation implies full (and constant) compensability rewards sub-indicators proportionally to the weights

Geometric mean entails partial (non constant) compensability rewards more those countries with higher scores

The absence of synergy or conflict effects among the indicators is a necessary condition to admit either linear or geometric aggregation &weights express trade-offs between indicators

Multi-criteria analysisdifferent goals are equally legitimate and important (e.g. social,

economic dimensions), thus a non-compensatory logic

Step 6. (Weighting and) aggregation

29

∏=

=Q

1q

wc,qcqxCI

Geometric aggregation

A hypothetical composite: inequality, environmental degradation, GDP per capita and unemployment

Country A: 21, 1, 1, 1 2.14Country B: 6, 6, 6, 6 6

Country A: 21, 2, 1, 1 2.54 19% increase in the scoreCountry B: 6, 7, 6, 6 6.23 4% increase in the score

30

The absence of synergy or conflict effects

among the indicators & weights express trade-offs between indicators

are necessary conditions to admit

either linear or geometric aggregation

31

When different goals are equally legitimate and important, then a non compensatory logic may be necessary.

Example: physical, social and economic figures must be aggregated. If the analyst decides that an increase in economic performance can not compensate a loss in social cohesion or a worsening in environmental sustainability, then neither the linear nor the geometric aggregation are suitable.

Instead, a non-compensatory multicriteria approach will assure non compensability by formalizing the idea of finding a compromisebetween two or more legitimate goals.

+ does not reward outliers + different goals are equally legitimate and important + no normalisation is required

BUT- computational cost when the number of countries is high

Multi-criteria type of aggregation

32

The Computational problem

Moulin (1988, p. 312) clearly states that the Kemeny method is “the correct method” for ranking alternatives, and that the “only drawback of this aggregation method is the difficulty in computing it when the number of candidates grows”.

With only 10 countries 10! = 3,628,800 permutations

We have 133 countries 133! = 1,5 x 10226 permutations

Multi-criteria type of aggregation

33

Sensitivity analysis can be used to assess the robustness of composite indicators …

Step 7. Robustness and sensitivity

[The Economist, 1998] comments:

“Cynics say that models can be built to conclude anything provided that suitable assumptions are fed into them.”

34

• Composite indicators may send misleading, non-robust policy messages if they are poorly constructed or misinterpreted […] inviting politicians to draw simplistic policy conclusions […] or the press to communicate misleading information.

Note of the European Commission, 2002

• The construction of composite indicators involves stages where judgement has to be made: the selection of sub-indicators, choice of model, treatment of missing values and weighting schemes […]

Step 7. Robustness and sensitivity

35

Our tools for robustness assessment

free software

book (2000) primer (2004)

http://sensitivity-analysis.jrc.cec.eu.int

36

Our case study for sensitivity analysis

37

Total: 133 countries

95 countries (in blue) are correctly placed in the ranking (difference between the Original rank and the median rank under all simulated methodological scenarios ≤ 15 positions)

38 countries (in grey) are misplaced in the ranking

20406080

100120140

Cou

ntry

Ran

k

The ranks/scores of the Index are

displayed in the form of confidence bounds

How do the Original ranks compare to the median ranks?

Germany(22 40)

USA(28 44)

Slovenia(31 53)

38

Comparing effectively complex dimensions with other variables

Andre’ Sapir’s work (Globalisation and the Reform of European Social Models, 2005). Strictness of employment legislation versus % of unemployed people reporting benefits.

Step 8. Links to other variables

39

De-constructing composite indicators can help extend the analysis …

More disaggregated data are useful to bring out the variations among regions, social groups, gender

Step 9. Back to the details

40

41

The scissor diagram of Composite Learning Index and variability

40

50

60

70

80

90

100

Com

posi

te L

earn

ing

Inde

x (C

LI) S

core

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Coe

ffici

ent o

f var

iatio

n

CLI Variability

Success: simultaneous thrust on multiple goals within a coherentlifelong learning strategy, while emphasizing reduction of the existing gaps in areas where performance is lagging.

42

Step 10. Visualisation

• Composite indicators must be able to communicate the picture to decision-makers and users quickly, transparently and accurately.

• Visual tools should be able to highlight country pitfalls by providing warning signals or vice versa

• be aware of which tools to employ in order todisplay the message properly

• Each visual tool has its own pros and cons

43

A well-designed graph can speak louder than words …

The four-quadrant model of the Sustainable Project Appraisal Routine (SPeAR®), see EUR report 2005.

• ‘Transparency’ of the indicator – make composite indicators available via the web, along with the data, the weights and the documentation of the methodology

allow users to change variables, weights, etc. and to replicate sensitivity tests.

44

Dashboard

• What are specific strengths and weaknesses of my continent/my country

45

Dashboard

• What is the situation of my country compared to others

46Composite Indicators

Complexity

Policy messages

Process:TransparencyRobustness

47

Conclusions

• can provide a comprehensive vision of a multidimensional phenomenon

• allows for the setting of national benchmarks and for further international comparisons

• is a starting point for analysis and discussion

A well-designed Index…

Kill the messenger but listen to the message…

48

2002• Saisana M. and S. Tarantola (2002) State-of-the-Art Report on Current Methodologies and Practices for Composite Indicator Development, EUR 20408 EN

• Tarantola S., Saisana M., Saltelli A., Schmiedel F. and N. Leapman (2002) Statistical techniques and participatory approaches for the composition of the European Internal Market Index 1992-2001, EUR 20547 EN.

2003•Saisana, M., S. Tarantola, and A. Saltelli (2003) Exploratory Research Report: the Integration of Thematic Composite Indicators, EUR 20682 EN

2004•Tarantola S., R. Liska and A. Saltelli (DG JRC - Unit G09), M. Donnay (DG ECFIN -Unit E02) (2004) Structural Indicators of the Lisbon agenda: robustness analysis and construction of composite indicators, EUR 21287EN.

•Tarantola S., Liska R., Saltelli A., Leapman N., Grant C. (2004) The Internal Market Index 2004, EUR 21274EN

•Nardo M., S. Tarantola, A Saltelli, C. Andropoulos, R Buescher, G. Karageorgos, A. Latvala, F. Noel (2004) The e-business readiness composite indicator for 2003: a pilot study, EUR 21294EN

JRC references on composite indicators

49

2005• Saltelli A., Funtowicz A., Guimarães-Pereira A. and Malingreau J-P, Munda G., Giampietro M., (2005) Develping effective Lisbon Strategy Narratives, EUR 21644 EN.

•Nardo, M. M. Saisana, A. Saltelli and S. Tarantola (JRC), A. Hoffman and E. Giovannini (OECD) (2005), Handbook On Constructing Composite Indicators: Methodology And User Guide, OECD Statistics Working Paper JT00188147, STD/DOC(2005)3.

•Nardo M., Saisana M., Saltelli A. and Tarantola S. (2005) Tools for Composite Indicators Building. European Commission, EUR 21682 EN, JRC Ispra, Italy, pp. 131.

• Saisana M., Saltelli A., Tarantola S., 2005, Uncertainty and Sensitivity analysis techniques as tools for the quality assessment of composite indicators, J. R. Stat. Soc. A, 168(2), 1-17

• Saisana M., Nardo M., Saltelli A. (2005) Uncertainty and Senstivity Analysis for the Environmental Sustainability Index (in collaboration with the Yale Center for Environmental Law and Policy & the Center for International Earth Science Information Network at Columbia University), presented to the January 2005 conference in Davos, http://www.ciesin.columbia.edu/indicators/ESI/ .

JRC references on composite indicators

50

2005

•Sajeva M., Gatelli D., Tarantola S. (JRC) and Hollanders (MERIT) (2005) Methodology Report on European Innovation Scoreboard 2005, European Commission, Enterprise Directorate-General, A discussion paper from the Innovation/SMEs Programme.

•Saisana M., Saltelli A., Schulze N., Tarantola S., Duchene V. (2005) Uncertainty and Sensitivity Analysis for the Knowledge-based Economy Index, Conference on Medium-Term Economic Assessment (CMTEA), Sofia, September 29-30.

•Munda M. and Nardo M. (2005) Constructing Consistent Composite Indicators: the Issue of Weights, manuscript submitted to Economics Letters.

• Munda G. and Nardo M. (2005) Non-Compensatory Composite Indicators for Ranking Countries: A Defensible Setting, manuscript submitted to Economica.

• Munda G. (2005) Social Multi-Criteria Evaluation (SMCE): Methodological Foundations and Operational Consequences, forthcoming, J. of Operational Research.

• Pennoni F., Tarantola S. and Latvala A. (2005): The 2005 European e-Business Readiness Index. EUR 22155 EN.

[…]

JRC references on composite indicators

51

References –Recent papers

Saisana M., Saltelli A., Tarantola S. (2005) Uncertainty and Sensitivity analysis techniques as tools for the quality assessment of composite indicators, Journal of the Royal Statistical Society - A, 168(2), 307-323 (application to statistical indicators).

Zádor J., Zsély, I.G., Turányi, T., Ratto, M., Tarantola, S., and Saltelli, A., 2005, Local and Global Uncertainty Analyses of a Methane Flame Model, Journal Physical Chemistry A., November 2005 (application to chemistry).

Saltelli, A., M. Ratto, S. Tarantola and F. Campolongo (2005) Sensitivity Analysis for Chemical Models, Chemical Reviews, 105(7) pp 2811 – 2828 (concise review paper).

Hall, J.W., Tarantola, S., Bates, P.D. and Horritt M.S. (2005) Distributed sensitivity analysis of flood inundation model calibration, J. Hydraulic Engineering, ASCE, 131(2) 117-126 (application to geographically distributed output).

Selected References on SA