Reflections on Realist Evaluation:

Unpacking the Mechanisms of Change

Marnie CarterNed Hardie-Boys

Ella Spittle

What we will cover

• Overview of realist evaluation– Principles and approach

• Implementation of a realist evaluation

• Lessons from a realist evaluation

• Questions and discussion

What is realist evaluation?

• Realist evaluation is an approach that can be broadly applied to different types of evaluations of policies and programmes

• The term was drawn from Pawson & Tilley’s book, Realistic Evaluation (1997)

Why realist evaluation?

• It addresses the limits of traditional approaches, which– Only ask “does this programme work?”

– Assume that programmes can be credited with general cause-and-effect statements

• Realist evaluation focuses deeper on why a programme works– “What works for whom, in what contexts,

and how?”

How does realist evaluation work?

Realist evaluation focuses on

• Mechanisms: Why a programme ‘works’ (or does not work)

• Context: What conditions are required for the mechanisms to work

• Outcomes: The results of different mechanisms working in different contexts

Mechanism + Context = Outcomes

Principles of realist evaluation I1) Programmes, policies and

interventions address problems by influencing change

2) Change is instigated by allowing participants to make different choices

3) Making different choices requires changing resources and/or reasoning

4) The combination of resources and reasoning allow participants to change their behaviour (the mechanisms)

Principles of realist evaluation II5) Changes in behaviour will vary

depending on participants’ context

6) Context, therefore, affects which mechanisms are triggered and what outcomes are achieved

7) Outcomes are varied because different mechanisms are triggered in different contexts

8) Variation is why realist evaluation findings are always provisional: “What works for whom, in what contexts, and how?”

Realist evaluation in practiceEvaluation of changes to the Road User

Charges (RUC) system

• Examine initial impacts – What mechanisms have been triggered?

• Evaluate outcomes– Has activation of those mechanisms resulted in

the expected outcomes?

• Recommendations– How can the system better influence activation

of which mechanisms, by which stakeholders, in which contexts, to achieve desired outcomes?

RUC system – background

• RUC is a tax for road “wear and tear”

• A number of changes were made, but the main change was to reduce complexity and opportunities for evasion

– The previous RUC system charged road users based on a vehicle’s net weight

– The new RUC system charges road users based on the maximum potential net weight

Evaluation design

• Two stage– Framework– Implementation

• Why a realist approach?• Steps– Theory-building (outcomes and

mechanisms)– Questions, measures, methods and

sources

Efficiency I

Outcome

1. Reduced compliance costs for operators

2. Reduced administrative complexity for government

Mechanisms

• Behavioural response to simplify admin processes

• Organisational response by NZTA to simplify admin processes and streamline payment methods

• Organisational response by NZTA to reduce complexity of investigations to recover unpaid RUC

Efficiency II

Outcome

3. Improved efficiency in vehicle use

4. Simplified enforcement of RUC

5. Increased uptake of electronic systems

Mechanisms

• Incentive for operators to load vehicles up to their RUC weight

• Organisational response by NZ Police to reduce officer time spent on enforcement

• Simplified processes and reduced costs incentivise new providers to enter market

Integrity I

Outcome

6. Reduced RUC evasion

Mechanisms

• Deterrence: greater chance of being caught changes risk perception

• Deterrence: perception of consequence of being caught leads to change in behaviour

• Deterrence: requirement to produce EDR acts as a deterrent

Integrity II

Outcome

7. Increased understanding of the system

Mechanisms

• Clarification/ communication of changes to system

• Monitoring and evaluation of the system

Equity

Outcome

8. Revenue neutrality within groups of vehicles (operators pay fair share)

Mechanisms

• Organisational response by government to monitor impacts and ensure no disproportionate effects on operators

Cost recovery

Outcome

9. Reduced late payment

10. Improved recovery of unpaid RUC

Mechanisms

• Penalties are sufficient to incentivise people to pay on time

• Detection: NZTA receives info from WoF providers and is able to identify evasion

• Investigative: the issuing of binding assessments assists in recovery

Implementing the RUC evaluation

• Evaluation questions placed emphasis on behavioural change - What impact have the RUC changes had on the level of

evasion?

- To what extent have the RUC changes contributed to changes in operator perception and behaviour regarding evasion?

• Survey and interview tools were designed to seek evidence of behavioural change

• Analysis focused on- whether the expected mechanism(s) had been activated

- for whom, how and in what context the outcome had been achieved

Example: efficiency in vehicle use

• Previous system: transport operators purchase RUC for the exact weight they will carry

• Policy change: Fixed ‘RUC weight’ for vehicles

• Mechanism: Incentive for operators to load vehicles up to their RUC weight

• Expected outcome: Transport operators will use their vehicles more efficiently

• Required context: transport operators are able to, and do, change their business practices

Evaluation findings• Commercial transport operators were

already loading as efficiently as possible

• Other considerations (safety, customer needs) were more important than RUC in loading decisions

• Many loads ‘cubed out’ before reaching the maximum weight

• Vehicles tend to be load-specific making it difficult to ‘back load’

The mechanism was not activated

Example: Recovery of unpaid RUC

• Previous system: time-intensive manual process for NZTA to identify unpaid RUC amongst light diesel vehicles; opportunity cost

• Policy change: Odometer reading reported as part of Warrant of Fitness inspection

• Mechanism: NZTA is better able to detect evasion

• Expected outcome: Unpaid RUC will be identified and recovered

• Required context: NZTA develops a system to receive data from WoF providers, and uses the information to recover debt

Evaluation findings• NZTA developed an automatic

reconciliation and invoicing system

0

1000

2000

3000

4000

5000

6000

Jul-1

1Au

g-11

Sep-

11O

ct-1

1No

v-11

Dec-

11Ja

n-12

Feb-

12M

ar-1

2Ap

r-12

May

-12

Jun-

12Ju

l-12

Aug-

12Se

p-12

Oct

-12

Nov-

12De

c-12

Jan-

13Fe

b-13

Mar

-13

Apr-

13

Thou

sand

s ($)

• The amount of invoiced debt has increased substantially

• Only 14% of invoiced debt has been recovered

The mechanism was partially activated

Reflections

It can be easy to get bogged down labelling mechanisms

Lesson one

Focus on describing mechanisms and what is needed to activate them – not naming them

The perils of tunnel vision

Avoiding tunnel vision

• Expected outcome of reduced compliance costs for transport operators

• Transport operators were expected to modify their administrative processes in response to a simpler system

• An online survey and case studies were used to seek evidence of behavioural change

• No evidence that operators had changed their behaviour was found

Initial conclusion: outcome was not achieved

However…

• Changes were made to the NZTA RUC administration fee structure

• 30% drop in NZTA admin fees (although this was a small portion of overall RUC compliance costs)

• Transport operators had therefore achieved a small reduction in cost, despite not changing their behaviours

Revised conclusion: the outcome was partially achieved

Lesson two

Behavioural change of the intended recipients is not the only means through which outcomes can be activated

Reduced evasion of RUC• Previous system: opportunity for operators to under-

nominate weight; outdated penalties for other forms of evasion

• Policy change: Fixed RUC weight; updated offences and penalties regime

• Mechanism: The removal of weight-based evasion is built into the system (no mechanism); updated penalties act as a deterrent to other forms of evasion

• Expected outcome: Evasion will reduce

• Required context: Transport operators are aware of the new penalties, which causes their risk perception to change resulting in modified behaviour

Evaluation findings

• Police Heavy Vehicle Operation data showed that evasion had reduced from 4% in 2012 to 1.2% in 2013

• No transport operators we interviewed admitted that they had previously evaded RUC and now were not

• We were therefore unable to identify how and why this effect had occurred (other than through speculation)

Lesson three

For sensitive or morally questionable topics, a focus only on what (rather than how and why) is appropriate.

Overall reflections

• The realist evaluation approach led us to lines of enquiry we may not have otherwise considered

• Clearly defining the mechanism of change assisted us to frame what evidence we were looking for

• Provided a means to demonstrate how and why the changes to the RUC system ‘worked’

• Helped to get inside the black box

• A useful tool for the evaluation kete

mcarter@allenandclarke.co.nz

nhardie-boys@allenandclarke.co.nz

espittle@allenandclarke.co.nz