The Comparative Method for Assessing Impacts and Cost

Effectiveness of Development Assistance Programs

Paul ClementsWestern Michigan University

It is possible to substantially enhance the impacts of development assistance with

evaluations employing the comparative method.

This depends on two, related arguments:

1.An analytic argument

2. An organizational or structural argument

2

The analytic argument - overviewLinear analysis, the method for rigorous impact evaluations (e.g.

with randomized controlled trials) is grounded in situational analysis.

Situational analysis can be rendered more rigorous through the comparative method with synthesis anchored in cost effectiveness.

Applying lessons from evaluations always requires (an act of) synthesis.

Weak synthesis in development practice is due about equally to analytic shortcomings and to weak incentives (and the melding of these in organizational cultures).

Stronger synthesis (hence more cost effective development interventions) can be achieved through a particular organization of evaluation based on the comparative method.

3

The organizational/structural argument - overview

(See “Reducing World Poverty by Improving Evaluation of Development Aid,” by Paul Clements, Thomaz Chianca and Ryoh Sasaki, American Journal of Evaluation, June 2008, pp.195-214.)

Aid evaluation today, e.g. under the DAC criteria or results-based approaches, has uneven quality and inconsistent units and evaluative judgments even for similar projects.

It is controlled by donor and implementing agencies with incentives for positive bias, so evaluative conclusions tend to be too narrow (not addressing cost effectiveness) or positively biased, and both of these amount to positive bias.

Evaluations are usually framed in individual, project specific terms.Together these factors undermine learning and accountability in the

development assistance community.

4

The proposal - overviewIt is possible to dramatically improve the quality of

evaluations, the discourse of the development community, and the impacts of aid interventions, by professionalizing aid evaluation.

A professional association (Evaluation International) with guidelines and standards for evaluation based on impacts and cost effectiveness.

The association’s evaluations also: - explain impacts in terms of project design and

implementation,- address strategic issues in the sector, and- employ the comparative method. Units and evaluative judgments are consistent between

evaluations.5

The Comparative MethodIntroduction

Cost effectiveness is the correct analytic anchor for the discourse of development management (we want to maximize impacts given our resources).

Linear AnalysisRigorous impact estimates based on the well known formula: (Ben t1 – Ben t0) – (Control t1 – Control t0)Sets of statistical procedures and analytic requirements for

valid and precise results.Linear analysis depends on situational or constructive analysis

for setting up the design and for applying the results.The comparative method builds on and adds rigor to

situational or constructive analysis6

The Comparative Method:situational/constructive analysis

Integrates sense of mechanism, of causal relations, vis a vis an objective

Involves our sense of the relative importance of several contributing factors

Involves triangulation

7

The Comparative Method:Forms of Reasoning (a)

Given similar known complex phenomena a, b, c and a new one, x

a > b > c ; where does x fit in this range?

8

The Comparative Method:Forms of Reasoning (b)

Consider two microfinance programs in environments with:

Program A Program B

High and variable inflation Low and steady inflation

Scattered, sparse population Dense population

Largely subsistence-based Fully monetized societysociety

Low GDP growth High GDP growth

9

The Comparative Method:Forms of Reasoning (c)

Two microfinance programs:

Program A Program B

Weak training Strong trainingprogram program

Weak sense of Strong sense of missionmission

10

The Comparative Method:Forms of Reasoning (d)

Estimating future impacts

Evaluation of condition C at time t1

Ct1 – Ct0 = x

what is Ct2 – Ct1?11

The Comparative Method:Forms of Reasoning (e)

Routine, every-day forms of reasoning

Employed in complex integrated judgments (hence constructive)

When grounded in a single value – cost effectiveness – exceed lay person’s normal analytic capacity

Can be systematized

12

The Comparative MethodIntegrated

To evaluate a primary health care project:Assume 20 evaluations of completed primary health care projects,

more and less rigorous, all estimate impacts and cost effectivenessAll explain impacts and c. e. in terms of the project’s design and

implementation and characteristics of the target population and of the context or environment.

- Evaluator locates the project on the multi-dimensional spectrum established by these earlier evaluations

- Evaluator investigates hypotheses based on this placement- Analysis of baseline and counterfactual- Sample survey to estimate changes in conditions- Estimates future impacts and costs- Inferences informed by technical literature and prior evaluations as

well as survey results, project records, etc.

13

The Comparative MethodHow to Develop and Apply it

This can be achieved, rendered rigorous, and generalized through the establishment of a professional association along the lines of associations of accountants and auditors.

Proceeding on a sector basis:Consistent units, consistent evaluative judgments given similar evidence.

14

The Comparative MethodSupports Learning and Accountability

Given consistent estimates of cost effectiveness

- Better portfolio management by donor and implementing agencies

- Stronger within- and across-sector learning across the development community

- New basis for personnel management- More cost effective resource allocation

15