SAI’s role in development and use of key indicators for R&D evaluation: a quantitative example and some concluding remarks

INTOSAI Working Group on Key National Indicators

Ville Vehkasalo & Timo Oksanen, 23.4.2013, Krakow

2

Presentation outline

Our stance on indicator development

Example of how to use key indicators in quantitative R&D evaluation

Qualitative evaluation possibilities

Concluding remarks; incorporation into the White Paper on KNI

3

About SAI’s role in indicator development

Depending on the national mandates, the SAI’s role can be active or passive – or something in between

However, an active role in indicator development can endanger SAI’s independency and objectiveness

The NAO of Finland has not participated in Finland’s KNI development

Therefore, we have kept an outsider’s view to Finnish KNI-system

4

Example: how can we use key indicators in quantitative R&D evaluation?

EU’s Regional Development Fund (ERDF) aims to achieve the following objectives in 2007–2013:

1) to enhance regional R&D and innovation capacities

2) to stimulate innovation and entrepreneurship in all sectors of the regional and local economy

3) to promote entrepreneurship, in particular by facilitating the economic exploitation of new ideas and fostering the creation of new firms.Source: Regulation (EC) No 1080/2006

5

Example

Cost of the ERDF program in Finland, 2007–2013: 1,7 billion euros (EU funding)

The effects of ERDF program are monitored using these indicators:

1) number of new firms

2) number of jobs

3) unemployment rate

4) employment rate

6

Example

5) regional GDP increase relative to the whole economy

6) share of exports in firms’ turnover

7) share of R&D activities in GDP

8) average educational level.

Source: ERDF Program of Southern Finland 2007–2013

7

Example

The number of new firms is included in Finland in Figures, which contains key statistical data about Finland on 25 different statistical topics, produced by Statistics Finland

This statistic is not included in Findicator, the official key indicator compilation

8

Example

How can we measure the effects of the 2007–2013 ERDF program in Finland?

Counterfactual: what would have happened without the program?

We need a control group that was not subjected to the program

But in 2007–2013, the whole country is included in the ERDF program

9

Example

However, in the earlier ERDF program, 2000–2006, small parts of Southern Finland were not included in the program

Therefore, we can compare the development in these new municipalities to those in Southern Finland that had been included earlier (old municipalities), in order to control for economy-wide fluctuations that may also affect start-ups

Population changes can be accounted for by using per capita figures

10

Example

Straightforward comparison is out of the question, as old and new municipalities have systematic differences:

new firms per 1000 capita, population-weighted means

year 2005 year 2011

old municipalities 5,04 5,25

new municipalities 7,05 7,37

Even before joining the program, new areas had higher rates of firm creation

11

Example

In order to control for unobservable characteristics, we have to use panel data: the same municipalities before and after the policy change

Specifically, we use the number of new firms from 2005 (before) and 2011 (after) in each of these municipalities

Small sample: only 31 new municipalities vs. 34 old ones (N = 65)

12

Example

We use the difference from 2005 to 2011, Dy = y2011 – y2005, as the independent variable

Differencing wipes out time-invariant characteristics, such as proximity to a larger city

Regression Dy = a + b new_munic

Coefficient estimates are:

coef. robust s.e. p-value

new_munic -0,095 0,352 0,788

constant 0,150 0,200 0,455

13

Example

new_munic estimate has wrong sign but it is statistically insignificant

Previous estimates are unweighted, i.e. small and large municipalities get the same weight, or importance, in the results

Alternatively we can use weights that measure the size of the municipality, for instance population levels

14

Example

If we use 2005 population levels as weights, we get these estimates:

coef. robust s.e. p-value

new_munic 0,110 0,171 0,525

constant 0,205 0,123 0,101

Again, can not reject null hypothesis

15

Example

Average change of +0,31 in the intervention group differs from zero (p = 0,014) but it would be misleading to attribute this to the program

We had an average change of +0,2 in the municipalities that were included earlier, i.e. even without this “new” program

The ERDF program did not cause the observed increase of 0,31 in the number of new firms

16

Example

This example is a bit unrealistic (sample too small, etc.) but it illustrates the basic quantitative evaluation framework:

1) Gather relevant data on intervention and control groups, before and after the intervention

2) Use simple difference-in-differences regression or standard panel data methods

3) Present your results with careful interpretation

17

Qualitative methods

Quantitative methods are useful in assessing program effectiveness

In addition, there are various qualitative approaches to R&D evaluation, such as interviews and participant observation

Possible explanations to why or how something happened/did not happen as planned

General conclusions not possible

18

R&D subproject conclusions (1): Evaluating specific programs and interventions

Evaluation of R&D programs is difficult, but not impossible

Finding relevant data can be tricky

Not possible to evaluate all programs; must have control groups

Without proper analysis, indicators are of limited use in program evaluation

19

R&D subproject conclusions (2): Evaluating the whole R&D system as a part of modern society

Problems are threefold: normative, causative and conceptual

Lack of clear, strategic whole-of-society vision communicated by the government (normative)

Lack of understanding and knowledge about the general impacts of R&D system on modern economies (causative)

What would and could be the role of SAIs and Key National Indicators of R&D in all of this? (conceptual)

20

R&D subproject: Incorporation into the White Paper on KNI

WG Secretariat can freely use our reports in preparing/editing the White Paper on KNI

For instance, our reports could be useful in augmenting the section Principles and Guidelines, subsection Guidelines for knowledge-based economies, where the evaluation of R&D programs is already mentioned

21

R&D subproject: List of reports

Utilising R&D knowledge at R&D policymaking in Finland: problems and promises, Helsinki 2011 (.ppt)

SAI’s role in development and use of key indicators for R&D evaluation, Riga 2012 (.ppt)

SAI’s role in development and use of key indicators for research and development (R&D) evaluation, 2012 (.doc)

SAI’s role in development and use of key indicators for R&D evaluation: a quantitative example and some concluding remarks, Krakow 2013 (.ppt)

22

Thank you!

ville.vehkasalo@vtv.fi

timo.oksanen@vtv.fi

http://www.vtv.fi/en