Technopolis 1

Forskningsrådsevalueringen- metoder, utfordringer, resultat

Erik Arnold

Technopolis

EVA

September 5 2002

Technopolis 2

The evaluation team

• Panel members …

• Pieter Drenth, Amsterdam/ALLEA

• Anders Flodström, KTH

• Jecqueline Godet, CNRS

• Philippe Laredo, CSI Ecole des Mines

• Ben Martin, SPRU

• Frieder Meyer-Krahmer, FhG-ISI

• Erkki Ormala, Nokia

• Arie Rip, Twente (chair)

• The research team ...

• Erik Arnold, James Stroyan, Paul Simmonds, Sarah Teather, Ben Thuriaux, Alina Östling, Technopolis (leader)

• Johan Hauknes, Marianne Broch, Per Koch, Heidi Wiig, STEP

• Stefan Kuhlmann, Sybille Hinze, FhG-ISI

• Egil Kallerud, Randi Søgnen, Liv Langfeldt, Magnus Gulbrandsen, NIFU

• Barend van der Meulen, Frank van der Most, Twente

Technopolis 3

Contents …

• Why have research councils?

• Was the research council reform a good idea?

• Is RCN doing a good job?

• What are the research policy challenges?

• What next?

Technopolis 4

RCN’s mandate (§2) - and that’s just the written bit

• RCN shall develop its research policy and administer grants for research, based on guidelines drawn up by the government and parliament. RCN shall further provide advice as a basis for the development of the government’s general research policy. RCN shall

Initiate research, which promotes the development of Norwegian industry and society Contribute to the development of knowledge of humanity’s common problems, especially

problems related to the environment and development Support researcher-initiated research, among other things by acting as a supplementary

source of funding for the institutions, which perform basic research Work to achieve a good balance between long-term basic research and use-oriented

research Work to ensure that user concerns are taken into account in applied research Promote international research co-operation Promote quality, efficiency and relevance in the research system Initiate and use the results of evaluations of research and research institutions Take strategic responsibility for the research institute sector in Norway Promote the exploitation of the results of research by the state, industry and the general

public

Technopolis 5

Our mandate (leaving out the details …)

• The evaluation of the Research Council of Norway is to cover the period from its establishment on 1 January 1993 to 31 December 2000, both dates inclusive. It shall give an overall evaluation of the Research Council in the light of the principal objectives laid down in report to the Storting St. meld. nr. 43 (1991-92), recommendation to the Storting Innst. S. nr. 231 (1991-92), and the statutes (articles of association) of the Research Council.

• The evaluation shall analyse the connection between the Research Council’s framework conditions, organisation and instruments, and the objectives laid down for its activities. Assessments shall be empirically grounded, among other things in the experiences of central groups of actors in the Ministries, research institutions, the commercial sector and the Research Council itself. In the light of this analysis, the evaluation shall consider what the Research Council’s framework conditions should be, how the Research Council ought to be organised, and what steps the Council itself should take, so that one may be as well equipped as possible to meet the future challenges confronting Norwegian research.

Technopolis 6

Defining the field of observation

Collecting data in the field

Janus is the god of evaluation. He is a judge, not a researcher

Structuring Observing Analysing Judging

Choosing evaluation

questions, effects to be evaluated, defining criteria

Defining indicators or suitable alternatives

Analysis of data

Analysis of causality

and attribution of effects

Judging effects individually

Formulating a synthetic judgement

Technopolis 7

Given that evaluation tools are poor, we look for convergence

Preparation- Issues, context- Peer finalisation- Composition analysis- Publication mapping- Logistics

Synthesis Report- Results- Conclusions- Recommendations- Next Steps

Review findings with participants and

stakeholders

Structured Peer Review- Projects- JVG- Customer needs

Management- Governance- Strategic- Operational

Customer Interviews

JVG Member Interviews

KTH Interviews

Railway Research

Group, KTH

Technopolis 8

Detailed mapping of work steps onto terms of reference

Tasks IssuesPolicy Advice Research

QualityBasic

ResearchUse of

ResearchCo-operation Inter-national

RCNÕs role in the research sector

Survey of researchers Y Y Y Y Y

Bibliometric study Y Y Y Y Y

RCN in development and change in research Y Y Y Y Y Y

Interviews with universities, research institutes Y Y Y Y Y

Expert review of RCN Areas (Technopolis + domain experts) Y Y Y Y Y Y

RCN and international co-operation Y Y Y Y Y

RCNÕs internal functioning and organisation

Mapping of policy instruments and operations Y Y Y Y

RCN internal management and functioning Y Y Y Y Y

RCNÕs relations with government and innovation systems role

Main events and issues: Document study 1989 Š 1999 Y

Governance of RCN and national policy role Y Y

RCNÕs role in the Innovation System Y Y Y

RCNÕs role in innovation in industry and society

User surveys in industry and research institutes Y Y Y Y Y

Identifying needs and making strategy Y Y Y Y

Interviews with industry and governmental users of RCN Y Y Y Y Y Y

RCN in the Public Understanding of Science Y

Cross-cutting tasks

Composition analysis of RCN and predecessors Y Y

Review previous evaluations of RCN activities and programmes Y Y Y Y

International comparisons Y Y Y

Technopolis 9

Our evaluation leant heavily on domain theory. We made this explicit, focusing on the idea of innovation systems..

Framework ConditionsFinancial environment; taxation and

incentives; propensity to innovation and entrepreneurship; mobility

Education and Research System

Professional education, training

Higher education and research

Public sector research

Industrial System

Large companies

Mature SMEs

New, technology- based firms

IntermediariesInstitutes;Brokers

Consumers (final demand)Producers (intermediate demand)

Demand

Banking, venture capital

IPR & information

Innovation & business support

Standards and norms

Infrastructure

PoliticalSystem

Government

Governance

RTD policies

Technopolis 10

… and the Mode 1/2 discussion

Distinctions Between Modes 1 and 2

Problems set and solved in the context of the (academic) concerns of the research community

Disciplinary

Homogeneous

Hierarchical, tending to preserve existing forms of organisation

Internal quality control

Problems set and solved in the context of application

Transdisciplinary

Heterogeneous

Heterarchical, involving more transient forms of organisation

Quality control is more socially accountable

Mode 1 Mode 2

Technopolis 11

In the judging phase, we also tried to take account of current changes - some of which go beyond theory

Drivers

• Growth in Mode 2 production

• Changes in the nature of technologies• ‘Hyphen-technologies’

• Dematerialisation

• Appropriation via intellectual property

• Changed industrial organisation of knowledge production

• Globalisation

• De-integration

• Acceleration: concurrent science

• Changed social contract• Increased relevance

• The state as a growing user

Trends

• Global knowledge markets

• Emergence of superuniversities

• Reorganisation of the Research Institute sector

• End of the ‘3-hump model’

• Industrial ‘observatories’ and new PPPs at the university/industry interface

• IPR as a constraint as well as an income generator

Technopolis 12

Results. The research council reform was a good idea in 1993. It still is

The old structure was not only messy, but was unable to cope with the newer policy challenges, such as the Main Target Areas

The issues in the Norwegian research system are about adequate research funding and the role of research in economic and social development development

Changes in the way knowledge is produced and used require both holism and diversity in research policy

Some other countries are moving towards more integrated solutions - Denmark, England, Finland …

Technopolis 13

History matters. The merger was a triumph of political turf over policy vision. The forces that destroyed the action areas won again

NAVF

RS

F

RM

F

RH

F

RN

F

NM

F

RCN

KS

BF

NT IEMU

MH

NT

NF

NL

VF

NF

FR

NO

RA

S

Technopolis 14

RCN’s three ‘steering levels’ became a battlefield in the early days and continue to provide means to fragment the Council

Executive Board

Division Boards (6)

Programme & Discipline Boards

Director General

Division Directors (6)

Strategy, Admin

Divisional Staff

Government

Ministries

Technopolis 15

The organisation and its context lead to rigidities

• Vertical links are weak• No-one is present at more than one level

• Links to overall strategy are hard to forge

• Higher levels tend to bless what goes on below, rather than engaging strategically

• Structures are locked in, owing to their tight and detailed links with external funders

• Little or no slack is available for change-agent or arena roles

• There is quite a lot of administrative diversity, but some of it is in the wrong places

Technopolis 16

RCN does a solid job where it can, but has more responsibility than authority

Making progress towards its six (summarised) goals

Administration is cost-effective

Starvation rations do not encourage dynamism

The sector principle reinforces the barriers between the Divisions

RCN needs to change its character from planning-system to research and innovation policy arena

cp FUGE, Demo 2000, OG21 ...

Technopolis 17

Goal fulfilment is almost surprisingly good. Testimony to the power of muddling through?

• RCN shall produce useful national and sectoral research policy advice to the government, based on an holistic national perspective

• RCN shall fund research to meet social and industrial needs, taking account of users’ needs and promoting the uptake of results

• RCN shall fund the high-quality basic and applied research needed in the national system of knowledge production, seeking to integrate the two as far as is appropriate while securing the place of basic research

• RCN is tasked with strategic responsibility for the research institute sector in Norway

• RCN is tasked with promoting the interaction of Norwegian knowledge production with the international knowledge production system

• RCN shall use appropriate and efficient processes (including evaluation) and organisational structures in performing its tasks

Improving

Yes, but …

Q/A and procedures in place for both.Integration?Well done. Consequences?

Exploiting EU. Integration into world research

Use of diversity should be more strategic

Technopolis 18

RCN’s ability to act as an holistic council reflects its funding

0

500

1000

1500

2000

2500

3000

3500

1993 1994 1995 1996 1997 1998 1999 2000 2001

General Funds Institutes Research Fund Special funds Administration Sum i faste priser

Technopolis 19

Norwegian research policy faces major challenges

The ‘value-creation gap’ (cp Norman, Reve)

Raising national R&D/GDP, especially through the creation of new industry

Move from the so-called ‘three-hump model’ with a strict division of labour among universities, institutes and ‘users’ to one where different modes of knowledge production interpenetrate …

… and to take the consequences by extending and modernising the roles of the institutions

Find an appropriate balance between the number of researchers and the amount of ‘bottom-up’ funding available in the system. This is really a question about funding, not about basic research

Technopolis 20

R&D/GDP

0

0.5

1

1.5

2

2.5

3

3.5

4

1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997

Denmark Finland Norway Sweden Total OECD

Technopolis 21

Governments have cut SND and walked away from innovation policy. IE’s budget fell 16% 93-00, and has been cut by another 150 MNOK

SND Budget Development

0

1000

2000

3000

4000

5000

6000

1993 1994 1995 1996 1997 1998 1999

MNOK

National Regional Fishing Fleet

Technopolis 22

Norway needs a stronger Innovation Agency function, cp TEKES. Getting the needed political understanding is hard, in a rich country

Framework ConditionsFinancial environment; taxation andincentives; propensity to innovation

and entrepreneurship; mobility

Education andResearch System

Professionaleducation and

trainingHigher education

and research

Public sectorresearch

Industrial System

Large companies

Mature SMEs

New, technology-based firms

IntermediariesResearchinstitutes;Brokers

Consumers (final demand)Producers (intermediate demand)

Demand

Banking, venturecapital

IPR andinformation

systems

Innovation andbusiness support

system

Standards andnorms

Infrastructure

PoliticalSystem

Government

Governance

RTD policies

SIVASND

RCN

Gap?

Technopolis 23

What are the functions that would need to be encompassed in an holistic research council?

Strategic Programmes, Institutes and

Infrastructures

Strategic Innovation

Agency

Response mode/ ‘free’ research

Absorptive Capacity

Thematic R&D areas

Strategy and Foresight SND

Technopolis 24

The struggle to integrate disparate cultures will continue to be hard

Structural issues

• Cultures

• Contested nature of the social contract

• Sectoral principle and organisation

• Few convincing examples• Integrated approaches

• Change agencies

Capture of principal-agent systems by client communities

Research Community

Education Ministry

Research Council

Industry Ministry

Innovation Agency

Industrial Community

Technopolis 25

After the evaluation, political path-dependency re-asserts itself

Those who already ‘knew’ the answer before we started work carried on telling everyone what they already ‘knew’

The ‘basic research’ faction still wants NAVF back. (Perhaps the collective amnesia of this faction is liked with its average age…!)

Industry’s representatives want NTNF back My good friend Hans Skoie’e campaign against a single council continues. It’s about time he was

awarded a medal - at least for persistence!

The education minister seems to have blocked the idea of raising research and innovation to the national (prime ministerial) level

Politically understandable … … but this has important negative symbolic value, cp Finland, Sweden, Ireland

The idea of continuing with a single council is accepted … but what that actually means remains fluid