1

The European Commission’sscience and knowledge service

Joint Research Centre

Auditing Composite Indicators of

Innovation

Dániel Vértesy, PhD

OECD Blue Sky III | Ghent, 19 Sep 2016

2

Name a single indicator that measures innovation!

Innovation is a complex process, involves multiple actors, with different functions, values & interests

Whether you assess the effort, the process, the outcomes, or the impacts of Innovation, you will likely need different information

= a multi-faceted phenomenon

?

Expenditure on innovation Interactions Introducing New-to-

the-world products

Competitiveness

3

Composite Indicators of InnovationComposite indicators – aggregate measures of multiple, individual indicators – are popular tools to measure innovation:

• The Global Innovation Index (GII) of Cornell/INSEAD/WIPO(since 2007; with WIPO since 2011);

• The European Commission’s European Innovation Scoreboard with its Summary Innovation Index (SII) (since 2000); andthe Innovation Output Indicator (since 2013)

Widespread; Newspapers report it;

Like university rankings: unstoppable?

4

Composite IndicatorsAdvantages:• Support decision makers by

summarizing complex or multi-dimensional issues

• Provide the “big picture”, highlight common trends

• Measure a latent phenomenon that is not directly measureable

• Attract public interest by benchmarking

Pitfalls:• Offer misleading, non-robust policy

messages if they are poorly constructed or misinterpreted

• May invite politicians to draw simplistic policy conclusions

• Easier to “manipulate” than individual indicators; the selection of sub-indicators and weights could be the target of political challenge

Assessing their quality and validity is particularly relevant

The development process helps• Better understand how a

system functions• Identify latent dimensions,

overlaps, redundancies ortrade-offs between components

5

The Quality of Composite Indicators“Composite indicators sit between analysis and advocacy, but quality discriminates the plausible from the rhetorical” (Saltelli, 2007)

Codified and continuously refined methodology• The OECD-JRC Handbook; Audits – robustness and

sensitivity analyses; etc.

Quantification (modelling) involves making normative choices• Indicators are intrinsically linked to policy priorities!

Normative choices affect:• the concept; • the operationalization; • accounting for information lossÞ Composite and individual

indicators are alike

What actors to include?

What indicators to include How to

address data quality

issues?

Which measure is

more relevant?

6

Conclusions – and questions1. The process of searching for ways to summarize multiple indicators

helps identify choices for policy

2. The most commonly used composite indicators of innovation include a few components that have [persistently] little or no impact on aggregate scores.• Innovation indicators advocate an incomplete picture of

innovation – typically [incremental] innovation in R&D- and patent-intensive industries.

• Do not aggregate all indicators, or use scoreboards, in case of significant loss of information

Is this sufficient to solve today’s societal challenges?

Is this aligned with policy needs?

Risks overlooking unintended consequences of innovations

I.e., The more a country’s firms spend on non-R&D innovation, the less they spend on business R&D

THANK YOU! daniel.vertesy@jrc.ec.europa.eu http://ec.europa.eu/jrc/en/coinCompetence Centre on Composite Indicators and Scoreboards (CC-COIN); Joint Research Centre (JRC)European Commission,