CCreating a reating a Culture of Evaluation – Culture of Evaluation – Getting from Getting from Rhetoric Rhetoric toto Possibility Possibility

Frances Ruane

FOCUS

Questions we need to answer to achieve “evaluation” rather than

“validation”

Outline

• Recent history and current context

• Why the concern with evaluation now?

• What questions need to be addressed if evaluation is to make a positive contribution to policy?

00.5

11.5

22.5

33.5

44.5

55.5

66.5

77.5

88.5

9

1970-1975 1975-1980 1980-1985 1985-1990 1990-1995 1995-2000

00.511.522.533.544.555.566.577.588.59

average GNP growth GNP 1970-2000 average

Irish Growth Rates 1970-2000

0

1

2

3

4

5

6

7

8

9

1970-1975 1975-1980 1980-1985 1985-1990 1990-1995 1995-20000

1

2

3

4

5

6

7

8

9

Irish GNP Growth EU15 GDP Growth Irish GNP 1970-200 Average EU15 GDP 1970-2000 Growth

Comparison of Irish and EU (15) Growth Rates 1970-2000

Current context

• weak history of planning => system-wide benchmark problems

• budgetary constraints => poor evaluation tradition

• inclusive, negotiated agreements => compromise solutions

• clientelist political tradition => bias against economic rationality

Who really wants evaluation?

Seen as “Raining on the parade”?

Recall the musical EvitaEvita“When the money keeps rolling out

you don't keep booksYou can tell you've done well by the

happy grateful looksAccountants only slow things down,

figures get in the way”

Why the new Concern

with Evaluation?

Why the new Concern

with Evaluation?

Accountability (C&AG, PAC, IA)

New Rhetoric (EBPM)

Efficiency

International Norms(EC)

Governance

Transparency(FOI)

Three key questions

• Who decides on the “evaluation context”?

• Who should undertake evaluations?

• Who should evaluate the evaluators?

Who decides on the “evaluation context”?

• Evaluation of what?• Evaluation for what purpose?• Evaluation with which tools?• Evaluation to what standards?• Evaluation over what timeframe?• Evaluation for whose eyes?• Implications of outturn for funding?

Who should undertake evaluations?

• Progamme promoters• Evaluation units within departments• Centralised evaluation units• Outside (national) evaluators• Outside (international) evaluators and to what template?

Who should evaluate the evaluators?

• Issue of standards/methodologies• Issue of professionalism/governance• Problem of small country …• Role of the network …• Need for a positive approach to evaluation• More than economics!• Reflect on insights of Charles M. Schultz