a case study on composite indicators - european commission · a case study on composite indicators...

Summer School on Sensitivity Analysis

SAMO2014

Ranco (IT), 24-27/06/2014

1

Variance-based sensitivity methods

A case study on Composite Indicators

Michaela Saisana

[email protected]

European Commission

Joint Research Centre

Econometrics and Applied Statistics Unit


SAMO2014

Ranco (IT), 24-27/06/2014

2

Mostly based on …


SAMO2014

Ranco (IT), 24-27/06/2014

3

Outline

Composite indicators

Motivation and objective of the study

Statistical coherence of two international university rankings

ARWU ranking and THES ranking

Uncertainty analysis

Conclusions


SAMO2014

Ranco (IT), 24-27/06/2014

4

Composite indicators (1/2)

Composite indicators aggregate individual variables with the aim of capturing

relevant, possible latent dimensions of reality

Most composite indicators are linear, i.e. weighted arithmetic averages. Weights are

understood to reflect variables’ importance in the index (e.g. budget allocation)

Example: university rankings

Very appealing for capturing a university’s multiple missions in a single number

Allow one to situate a given university in the worldwide context

Can lead to misleading and/or simplistic policy conclusions

Caution: Nominal weights are not a measure of variables’ importance


SAMO2014

Ranco (IT), 24-27/06/2014

5


SAMO2014

Ranco (IT), 24-27/06/2014

6

Motivation of the study (1/3)

Q: Why nominal weights often do not coincide with the importance of variables?

Example: A dean wants to rank teachers based on ‘hours of teaching’ and

‘number of publications’, adding these two variables up

Assume:

X1 = Hours of teaching

X2 = Number of publications

Y = 0.5 X1 + 0.5 X2


SAMO2014

Ranco (IT), 24-27/06/2014

7

ρ(x1, x2) =−0.151, σ(x1) = 614, σ(x2) = 116, σ(y) = 162

Estimated 𝑅21= 0.075 and 𝑅2

2 = 0.826

Teachers’ ranking is mostly driven by the number of publications



SAMO2014

Ranco (IT), 24-27/06/2014

8


To obviate this, the dean substitutes the model

with

A professor comes by, looks at the last formula, and complains that publishing is

disregarded in the department …

Y = 0.5 X1 + 0.5 X2

Y = 0.7 X1 + 0.3 X2


SAMO2014

Ranco (IT), 24-27/06/2014

9

Objective of the study (1/3)

Our suggestion: to assess the quality of a composite indicator using – instead of 𝑅2i

(Pearson product moment correlation coefficient of the regression of y on xi) its

non-parametric equivalent:


SAMO2014

Ranco (IT), 24-27/06/2014

10


Appealing feature of the first order effect Si (main effect):

it offers a precise definition of importance, that is ‘the expected reduction in variance of the CI that

would be obtained if a variable could be fixed’

it can be used regardless of the degree of correlation between variables

it is model-free, in that it can be applied also in non-linear aggregations

it is not invasive, in that no changes are made to the CI or to the correlation structure of the

indicators

Nominal weights:

Ratios of weights: revealed target relative importance

w1/w2: substitution effect between x1 and x2


SAMO2014

Ranco (IT), 24-27/06/2014

11


One can hence compare the importance of an indicator as given by the nominal weight

(assigned by developers) with the importance as measured by the first order effect (Si) to test

the index for coherence.

Estimating first order effect:

Non-parametric local linear kernel regression

1) Cross-validation (CV) principle or 2) direct plug-in (DPI) method

Q: What are the limitations of non-parametric kernel regression in a CI context?

Case-studies:

University rankings: ARWU 2008 and THES 2008

Human development: HDI 2009 and HDI 2010

Index of African governance: IAG 2008

Sustainable society index: SSI 2008


SAMO2014

Ranco (IT), 24-27/06/2014

12

University rankings are used to judge about the performance of university systems, whether

intended or not on by their developers

University rankings: at the forefront of the policy debate


SAMO2014

Ranco (IT), 24-27/06/2014

13

France:

Creation of 10 centres of HE excellence

The minister of Education set a target to put at least 10

French universities among the top 100 in ARWU by 2012

President has put French standing in these international

ranking at the forefront of the policy debate (Le Monde,

2008).

Italy

No University in the top 100 of the ARWU ranking seen

as failure of the national educational system).

Spain

1 University in the top 200 of the ARWU hailed as a great

national achievement)



SAMO2014

Ranco (IT), 24-27/06/2014

14

An OECD study shows that worldwide university leaders are concerned about

ranking systems with consequences on the strategic and operational decisions they

take to improve their research performance.

(Hazelkorn, 2007)

There over 16,000 HEIs, yet some of the global rankings merely capture the top

100 universities – less than 1%.

(Hazelkorn, 2013)



SAMO2014

Ranco (IT), 24-27/06/2014

15

An extreme impact

What - 2005 THES created a major controversy in Malaysia: country’s top two

universities slipping by almost 100 places compared to 2004.

Why - change in the ranking methodology (not well known fact and of limited comfort)

Impact - Royal commission of inquiry to investigate the matter. A few weeks later, the

Vice-Chancellor of the University of Malaysia stepped down.



SAMO2014

Ranco (IT), 24-27/06/2014

16

University rankings: ARWU 2008

Methodology:

6 indicators; linear aggregation

Best performing institution = 100; score of other institutions calculated as percentage

Weighting scheme chosen by the developers

PROS and CONS

6 « objective » indicators

Focus on research

performance, overlooks

other university missions.

Biased towards hard sciences

intensive institutions

Favours large institutions


SAMO2014

Ranco (IT), 24-27/06/2014

17

University rankings: THES 2008

Methodology:

6 indicators; linear aggregation

z-score calculated for each indicator

Best performing institution = 100; score of other institutions calculated as %

Weighting scheme chosen by rankers

PROS and CONS Attempt to take into account teaching

quality

Two expert-based indicators: 50% of

total

Subjective indicators: lack of

transparency

Substantial yearly changes in

methodology

Measures research quantity


SAMO2014

Ranco (IT), 24-27/06/2014

18

Main effects and nominal weights: ARWU 2008

Si’s are more similar to each other than the nominal weights, i.e. ranging between 0.14

and 0.19 (normalized Si’s to unit sum; CV estimates) when weights should either be

0.10 or 0.20


SAMO2014

Ranco (IT), 24-27/06/2014

19

Main effects and nominal weights: ARWU 2008

Si’s are more similar to each other than the nominal weights, i.e. ranging between 0.14

and 0.19 (normalized Si’s to unit sum; CV estimates) when weights should either be

0.10 or 0.20

2008 ARWU Wi

Wi/Wacademic

performance Si,CV

Si/Sacademic

performance

Si

(unity sum)

Alumni winning Nobel Prize 0.1 1 0.65 0.9 0.14

Staff winning Nobel Prize 0.2 2 0.72 0.9 0.16

Highly cited researchers 0.2 2 0.85 1.1 0.19

Articles in Nature and Science 0.2 2 0.88 1.2 0.19

Articles in Science and Social Sciences Citation Index 0.2 2 0.7 0.9 0.15

Academic performance (size adjusted) 0.1 1 0.76 1.0 0.17


SAMO2014

Ranco (IT), 24-27/06/2014

20

Main effects and nominal weights: THES 2008

The combined importance of peer-review variables (recruiters and academia) appears larger

than stipulated by developers, indirectly supporting the hypothesis of linguistic bias

The teacher/student ratio, a key variable aimed at capturing the teaching dimension, is much

less important than it should be (normalized Si is 0.09, nominal weight is 0.20)


SAMO2014

Ranco (IT), 24-27/06/2014

21

Main effects and nominal weights: THES 2008

The combined importance of peer-review variables (recruiters and academia) appears larger

than stipulated by developers, indirectly supporting the hypothesis of linguistic bias

The teacher/student ratio, a key variable aimed at capturing the teaching dimension, is much

less important than it should be (normalized Si is 0.09, nominal weight is 0.20)

Q: Are the Si’s values rescaled to unity sum meaningful?

2008 THES Wi Wi/Wintstudents Si,CV Si/Sintstudents Si (unity sum)

Academic review 0.40 8 0.81 4.8 0.36

Recruiter review 0.10 2 0.54 3.2 0.24

Teacher/student ratio 0.20 4 0.20 1.2 0.09

Citations per faculty 0.20 4 0.41 2.4 0.18

International staff 0.05 1 0.12 0.7 0.05

International students 0.05 1 0.17 1.0 0.08


SAMO2014

Ranco (IT), 24-27/06/2014

22

Statistical coherence of university rankings

More discrepancy between the nominal weights and the main effects in

THES compared to ARWU

ARWU is less sensitive on the choice of bandwith h than THES

ARWU is more coherent than THES


SAMO2014

Ranco (IT), 24-27/06/2014

23

1

51

101

151

201

251

301

351

401

451

501Me

dia

n r

an

k (a

nd 9

9%

co

nfid

en

ce in

terv

al) a

cco

un

ting

for

meth

odolo

gic

al u

nce

rtain

ties

Seoul National University

University of Frankfurt

University of Hamburg

University of California-Davis

University of Alaska-

Fairbanks

Hanyang University

54 universities outside the interval (total of 503)

[43 universities in the Top 100]

Uncertainty analysis (ARWU)

Impact of uncertainties on the university rankings

Source: Saisana, D’Hombres,

Saltelli, 2011, Research Policy 40,

165–177


SAMO2014

Ranco (IT), 24-27/06/2014

24

Uncertainty analysis (THES)

Impact of uncertainties on the university rankings

Source: Saisana, D’Hombres,

Saltelli, 2011, Research Policy 40,

165–177

1

51

101

151

201

251

301

351

401

Me

dia

n r

an

k (a

nd 9

9%

co

nfid

en

ce in

terv

al) a

cco

un

ting

for

meth

odolo

gic

al u

nce

rtain

ties

250 universities outside the interval (total of 400)

[61 universities in the Top 100]

University of California, Santa

Barbara

Stockholm School of Economics

University of st.

Gallen

University of Tokyo

University of

LeichesterUniversity La Sapienza,

Roma


SAMO2014

Ranco (IT), 24-27/06/2014

25

ARWU versus THES (2008)

1 – Same top10

Universities

2 - Greater variations in

the middle to lower end

of the rankings

3 - Europe is lagging

behind USA

4 – THES favours UK

universities

THES

ranking


SAMO2014

Ranco (IT), 24-27/06/2014

26

Conclusions

Nominal weights of indicators within a composite do not necessarily coincide with

the importance of a variable

This paper provides a variance-based, non-invasive tool to measure the internal

discrepancy of a composite indicator between nominal weights and effective

importance

Effective tool to:

Avoid associating nominal weights with indicators’ importance

Assess different weighting strategies

Reconsider the aggregation scheme


SAMO2014

Ranco (IT), 24-27/06/2014

27

1. Paruolo P., Saisana M., Saltelli A., 2013, Ratings and Rankings: voodoo or science?. J Royal Statistical

Society A 176(2).

2. Saisana M., Saltelli A., 2012, JRC audit on the 2012 WJP Rule of Law Index, In Agrast, M., Botero, J.,

Martinez, J., Ponce, A., & Pratt, C. WJP Rule of Law Index® 2012. Washington, D.C.: The World

Justice Project.

3. Saisana M., Philippas D., 2012, Sustainable Society Index (SSI): Taking societies’ pulse along social,

environmental and economic issues, EUR 25578, Joint Research Centre, Publications Office of the

European Union, Italy.

4. Saisana M., D’Hombres B., Saltelli A., 2011, Rickety Numbers: Volatility of university rankings and

policy implications. Research Policy 40, 165–177.

5. Saisana M., Saltelli A., Tarantola S., 2005, Uncertainty and sensitivity analysis techniques as tools

for the analysis and validation of composite indicators. J Royal Statistical Society A 168(2), 307-323.

6. OECD/JRC, 2008, Handbook on Constructing Composite Indicators. Methodology and user Guide, OECD

Publishing, ISBN 978-92-64-04345-9.

References and Related reading


SAMO2014

Ranco (IT), 24-27/06/2014

28

Forthcoming:

12th JRC Annual Training on Composite Indicators

and Multi-criteria Analysis

22-26 September 2014

http://composite-indicators.jrc.ec.europa.eu

a case study on composite indicators - european commission · a case study on composite indicators...

Documents