joint research centre - composite-indicators.jrc.ec.europa.eu · coin 2017 - 15th jrc annual...

The European Commission’s science and knowledge service

Joint Research Centre

Ten Steps Guide for Indices & Scoreboards

Michaela Saisana

COIN 2017 - 15th JRC Annual Training on Composite Indicators & Scoreboards 06-08/11/2017, Ispra (IT)

3 JRC-COIN © | Ten Steps Guide for Indices & Scoreboards

• 3 dimensions & 12 areas • 14 headline indicators • 28 secondary indicators • 93 indicators in total (gender/age)

https://composite-indicators.jrc.ec.europa.eu/social-scoreboard/

Example of a scoreboard Social Pillar Scoreboard for the European Pillar of Social Rights


• 3 dimensions & 12 areas • 14 headline indicators • 28 secondary indicators • 93 indicators in total (gender/age)

Example of a scoreboard Social Pillar Scoreboard for the European Pillar of Social Rights

https://composite-indicators.jrc.ec.europa.eu/social-scoreboard/


• 1 index • 2 sub-indices • 7 pillars • 21 sub-pillars • 80+ indicators

Example of a composite indicator Global Innovation Index


Why are they needed?

Globalization + Complexity

Who needs them?

International organizations, European Commission, governments, public

What are they?

com’-po-site: made of various parts or elements

in’-di-ca-tor: a device providing information on the state or condition

Need for indices and scoreboards


1987 First CI proposed by the EC Proposal for a Council Regulation establishing a Community system of aids to agricultural income [COM(87) 166 final] 1985 Second CI developed by a government German gov. to select areas eligible for investment aid in the North Rhine-Westphalia [85/12/EEC] 1982 First CI developed by a government Danish gov. to select areas eligible for regional aid [82/691/EEC] 1973 First reference within an EC document but … 14th rep. on the activities of the Monetary Committee [OJ 9.11.73 No C94]

Uptake of indices in the EC policies


Search for «composite indicator» OR «synthetic indicator» OR «aggregate indicator»

118 documents (most of them by the EC)

Uptake of indices in the EC policies


Last night: 14000

65 381

2630

5760

12400

17300

1950 1960 1970 1980 1990 2000 2010 2020

Directional radio receiver

Uptake of indices in the academia

More than ten-fold increase since 2000!


Uptake of indices in the media


Enthusiastic supporters, mostly

from advocacy groups developing their own indices to advance a

cause

Skeptical economists and official statisticians

concerned by the subjective nature of the selection of variables, weights and

aggregation

on the other skeptical economists and

official statisticians concerned by the

subjective nature of the variables’ and

weights’ selection and aggregation

procedure.

Polarized audience




cause



aggregation

on the other skeptical economists and

official statisticians concerned by the

subjective nature of the variables’ and

weights’ selection and aggregation

procedure.

Polarized audience


Indicators are powerful advocacy tools

2010

• A. Forrest decided to take up the cause of combatting human trafficking he established the Walk Free Foundation

• Bill Gates gave him some advice: “use a quantifiable metric. […] if you can’t measure it, it doesn’t exist.”

2012

• A. Forrest got Richard Branson on board, and by Dec. 2012, they had appealed to 25 large companies and governments to ban the use of forced labor

2013

• One year later, one activity got more attention than all of Walk Free’s efforts combined…

Andrew Forrest


Indicators are powerful advocacy tools

2013 • September: The Global Slavery Index

2013

• October: The index was covered in >100 newspaper stories around the world

2014

• July: In a global survey of NGOs that work on trafficking issues, over 40% had already heard of this new index

Andrew Forrest




cause



aggregation

Polarized audience


Conceptual Framework – World Health Organization

Unlike the Alcohol Policy Index, most composite indicators cannot be validated versus a ground truth [Alcohol Policy Index, 2007, PLoS Medicine, 4(4):752-759]

«Measurements without theory…»


…and on top of that indices have …

strong Political and Policy Implications

The Malaysian Industrial Development Authority insisted that Malaysia aims to

move from the 24th to top 10 on in the World Bank's `Doing Business' ranking list. “We continue to ask ourselves what it will take to reach the top 10, and are we willing to do what it takes to get there." [Asia in Focus, Jan. 8 2007]

The Minister of the Economic Development in Kyrgyzstan, expresses a hope (in 2008) that his country shall rank among top 20 countries in the Doing Business rating in three years.

[The WB Doing Business Report has long been credited with bringing about reforms in countries – as many as 2000 distinct reforms since its 2003 launch (Source: The Economist 2013)]


Step 10. Presentation & dissemination

Step 9. Association with other variables

Step 8. Back to the indicators

Step 7. Robustness & sensitivity

Step 6. Weighting & aggregation

Step 5. Normalisation of data

Step 4. Multivariate analysis

Step 3. Data treatment

Step 2. Selection of indicators

Step 1. Developing the framework

Finally endorsed (after 2 rounds of consultation) by the OECD high level statistical committee in March 2008

Ten Steps Guide for Composite Indicators


Step 10. Visualisation & Communication

Step 9. Back to the data

Step 8. Robustness & Sensitivity

Step 7. Statistical coherence

Step 6. Aggregation

Step 5. Weighting

Step 4. Normalisation

Step 3. Data treatment

Step 2. Selection of indicators

Step 1. Developing the framework

New version to be released end of 2018 (10 year anniversary)

Ten Steps Guide for Composite Indicators & Scoreboards

2018 version


Definition of the phenomenon

Added-value of index/scoreboard

Nested structure of the framework

Involve experts and stakeholders

Step 1 Theoretical/Conceptual framework


Selection criteria for indicators input, output, outcome, process

salience, credibility, legitimacy

data coverage

consider using proxy variables when official statistics are scarce

Summary statistics Source & data availability (countries, time)

type (hard, soft or input, output, process),

descriptive statistics (mean, median, skewness, kurtosis, min, max, variance, histogram)

Step 2

Data selection


…

Discuss strengths and limitations of each selected indicator

Involve experts and stakeholders

Step 2

Data selection


Make scale adjustments if necessary (e.g. divide by population, GDP, other)

Check for missing data and outliers

Treat outliers, if needed (so as to avoid that they become unintended benchmarks)

Estimate missing data, if appropriate (and estimate confidence interval for each imputed value to assess the impact of

imputation on the results)

Step 3

Data treatment


In arithmetic averages, no imputation of missing data for each unit only observed values are considered

unit Y1 Y2 Y3 Y4 mean

1 4 6 4 6 5.00

2 5 2 7 4.67

3 2 9 2 4.33

4 5 2 1 3 2.75

5 1 1 2 1.33

unit Y1 Y2 Y3 Y4 mean

1 4 6 4 6 5.00

2 5 4.67 2 7 4.67

3 2 9 4.33 2 4.33

4 5 2 1 3 2.75

5 1.33 1 1 2 1.33

variables

variables

based on observed values only


Make directional adjustments (so that higher scores correspond to better performance in all indicators or vice versa)

Select a suitable normalisation method that respects the conceptual framework and the data properties

Step 4

Normalisation

Does a large group of foreign-born population decrease social cohesion? (yes/no direction)

5 seconds rule!


Make directional adjustments (so that higher scores correspond to better performance in all indicators or vice versa)

Select a suitable normalisation method that respects the conceptual framework and the data properties

Step 4

Normalisation

ranking z-score Min-max

Distance to reference country

Categorical scales

Adjust for measurement unit

Y Y Y Y Y

Adjust for variance

Y Y N N N

Adjust for range

Y N Y N Y

Adjust for extreme values

Y N N N Y

Y/N

0.0

0.2

0.4

0.6

0.8

1.0

1.2

7.0 8.0 9.0 10.0 11.0 12.0 13.0

Raw indicator: Mean years of schooling (years)

No

rmali

sed

: m

in-m

ax

0

5

10

15

20

25

7.0 8.0 9.0 10.0 11.0 12.0 13.0

Raw indicator: Mean years of schooling (years)

No

rmali

sed

: R

an

k


Discuss if correlation among indicators should be accounted for in the weights If yes, then how (more correlated indicators more

or less weight?)

Select a suitable weighting method that respects the conceptual framework and the data properties

Step 5

Weighting

X2

X1 0


Discuss if compensability among indicators should be allowed (fully, partially, not at all)

Discuss up to which level to aggregate

Select a suitable aggregation method that respects the conceptual framework and the data properties

Step 6

Aggregation

Question 8+2 < = > 5+5

A B C D E

A 0 0.2 0.4 0.2 0.2

B 0.8 0 0.8 1 0.2

C 0.6 0.2 0 0.4 0.4

D 0.8 0 0.6 0 0.2

E 0.8 0.8 0.6 0.8 0

Outranking matrix


17.1) During the last 12 months, for how many months was your household’s main source

of water sufficient to meet your household’s drinking, cooking, bathing and cleaning

needs?

Months: Don’t remember (-1)

17.2) How often do you worry there will not be enough water from your household’s main

water source to satisfy your household’s drinking, cooking, bathing and cleaning needs?

Never (1) Rarely (2) Sometimes (3) Often (4) Always (5)

Example: Multidimensional Poverty Assessment Tool

Component: Domestic Water Supply, Subcomponent: Availability

104 HHs

8

3 HHs

4

Too worried? Careless?

Suggestion: With survey data, given some unavoidable inconsistencies (in part due to the way the human mind works), use a (weighted) arithmetic average (rule of thumb: 5-10 indicators) within a subcomponent.


Assess if few indicators dominate the framework

(rindicator,index>0.95)

Assess if indicators behave as “noise” in the framework

(-0.3<rindicator,index<0.3)

Assess if indicators are negatively related to the index

(rindicator,index<-0.3)

Step 7

Statistical coherence

Pearson Correlations

Ecosystem Vitality

Environmental Health

EPI 2010 0.29 0.90

Environmental Health

-0.08*

Standard deviation

10.8 24.8


…

Assess if indicators statistically fit better under different dimensions than those in the framework

Assess if dimensions should be merged or split

Assess if bias has been introduced in the index (e.g., due to population size, population density, GDP)

Step 7

Statistical coherence


Identify the sources of uncertainty in the index development

Assess the impact of the uncertainties to the index scores or ranks complement scores/ranks with confidence

intervals

Step 8

Robustness & Sensitivity Including/

excluding variables Normalisation

Missing data Weights

Aggregation

Country 1

10

20

30

40

50

60

Country 2 Country 3


Identify the sources of uncertainty in the index development

Assess the impact of the uncertainties to the index scores or ranks complement scores/ranks with confidence

intervals

Step 8

Robustness & Sensitivity


…

Identify which uncertainties are more crucial in determining the final classification

• Explain why certain countries notably improve or deteriorate their relative position given the assumptions

Robustness ≠Quality

Step 8

Robustness & Sensitivity


1

51

101

151

201

251

301

351

401

451

501Med

ian

rank

(and

99%

con

fiden

ce in

terv

al) a

ccou

ntin

g fo

r

met

hodo

logi

cal u

ncer

tain

ties

Seoul National University

University of Frankfurt

University of Hamburg

University of California-Davis

University of Alaska-

Fairbanks

Hanyang University

54 universities outside the interval (total of 503)

[43 universities in the Top 100]

1

51

101

151

201

251

301

351

401

Med

ian

rank

(and

99%

con

fiden

ce in

terv

al) a

ccou

ntin

g fo

r

met

hodo

logi

cal u

ncer

tain

ties

250 universities outside the interval (total of 400)

[61 universities in the Top 100]

University of California, Santa

Barbara

Stockholm School of Economics

University of st.

Gallen

University of Tokyo

University of

LeichesterUniversity La Sapienza,

Roma

ARWU 2008 ranking (more robust, higher correlations)

THES 2008 ranking

(less robust, lower correlations)

Robustness ≠Quality


Decompose performance at the indicator level (to reveal strengths and limitations for each country)

Correlate the index with relevant measurable phenomena and explain similarities or differences

Develop data-driven narratives on the results.

Perform causality tests (if time series data or microdata are available)

Step 9

Look back into the data & vis-à-vis other measures


Identify suitable visualisation tools for the targeted audience

Select the visualisation technique which communicates the most information without hiding vital information

Make your index/scoreboard EAST Easy, Attractive, Social and Timely

Step 10

Visualisation & Communication


Sound framework

Carefully selected indicators

Sound model

can help to depict reasonably reality

Prerequisites for any index


Sound framework

Carefully selected indicators

Sound model

can only offer an imperfect mirror of reality

Prerequisites for any index


Powerful evidence based narratives supported by good statistical measures and good analytic work are a possibility which should not be left untried

We need relevant and sound…

(Composite) Indicators


• Becker, W., M. Saisana, P. Paruolo, and I. Vandecasteele. 2017. ‘Weights and Importance in Composite Indicators:

Closing the Gap. Ecological Indicators 80: 12–22.

• Cohen, A., Saisana, M., 2014, Quantifying the Qualitative: Eliciting Expert Input to Develop the Multidimensional

Poverty Assessment Tool, Journal of Development Studies 50(1): 35-50.

• OECD/JRC, 2008, Handbook on Constructing Composite Indicators. Methodology and user Guide, OECD

Publishing, ISBN 978-92-64-04345-9.

• Paruolo P., Saisana M., Saltelli A., 2013, Ratings and Rankings: voodoo or science?. J Royal Statistical Society A

176(3), 609-634.

• Saisana, M., and Saltelli, A., 2011, Rankings and Ratings: Instructions for use, Hague Journal on the Rule of Law

3(2), 247-268.

• Saisana M., D’Hombres B., Saltelli A., 2011, Rickety Numbers: Volatility of university rankings and policy

implications. Research Policy 40, 165–177.

• Saisana, M., Saltelli, A., 2014, JRC Statistical Audit of the WJP Rule of Law Index® 2014 (p.188-198). In the World

Justice Project Rule of Law Index® Washington, D.C.: The World Justice Project.

References and related reading


• Saisana, M., Weziak-Bialowolska, D., 2013, JRC Statistical Audit on the Environment and Gender Index (p.143-

153), in IUCN The Environment and Gender Index (EGI) 2013 Pilot. Washington, D.C.: IUCN.

• Saisana, M., Saltelli A., 2012, Corruption Perceptions Index 2012. Statistical Assessment, EUR 25623, European

Commission, JRC-IPSC, Italy.

• Saisana M., 2010, ELLI-Index: a sound measure for lifelong learning in the EU, EUR 24529, European

Commission, JRC-IPSC, Italy.

• Saisana M., Saltelli A., 2010, The Multidimensional Poverty Assessment Tool (MPAT): Robustness issues and

Critical assessment, EUR 24310, European Commission, JRC-IPSC, Italy.

• Saisana M., 2008, The 2007 Composite Learning Index: Robustness Issues and Critical Assessment, EUR

23274, European Commission, JRC-IPSC, Italy.

• Saisana M., D’Hombres B., 2008, Higher Education Rankings: Robustness Issues and Critical Assessment, EUR

23487, European Commission, JRC-IPSC, Italy.

References and related reading

Welcome to email us at: [email protected]

THANK YOU

COIN in the EU Science Hub https://ec.europa.eu/jrc/en/coin

COIN tools are available at: https://composite-indicators.jrc.ec.europa.eu/

The European Commission’s

Competence Centre on Composite

Indicators and Scoreboards

mailto:[email protected]



https://ec.europa.eu/jrc/en/coin



https://composite-indicators.jrc.ec.europa.eu/





http://tiny.cc/d0bqoy

joint research centre - composite-indicators.jrc.ec.europa.eu · coin 2017 - 15th jrc annual...

Documents