gysen. 2006. the modus narrandi

8/20/2019 Gysen. 2006. the Modus Narrandi

1/25

http://evi.sagepub.comEvaluation

DOI: 10.1177/13563890060641762006; 12; 95Evaluation

Joos Gysen, Hans Bruyninckx and Kris BachusPolicyThe Modus Narrandi: A Methodology for Evaluating Effects of Environmental

http://evi.sagepub.com/cgi/content/abstract/12/1/95 The online version of this article can be found at:

Published by:

http://www.sagepublications.com

On behalf of:

The Tavistock Institute

can be found at:EvaluationAdditional services and information for

http://evi.sagepub.com/cgi/alertsEmail Alerts:

http://evi.sagepub.com/subscriptionsSubscriptions:

http://www.sagepub.com/journalsReprints.navReprints:

http://www.sagepub.com/journalsPermissions.navPermissions:

© 2006 SAGE Publications. All rights reserved. Not for commercial use or unauthorized distribution. at UNIVERSITAT AUTÒNOMA DE BARCELONA on March 14, 2007http://evi.sagepub.comDownloaded from

http://www.tavinstitute.org/index.phphttp://evi.sagepub.com/cgi/alertshttp://evi.sagepub.com/cgi/alertshttp://evi.sagepub.com/subscriptionshttp://evi.sagepub.com/subscriptionshttp://evi.sagepub.com/subscriptionshttp://www.sagepub.com/journalsReprints.navhttp://www.sagepub.com/journalsReprints.navhttp://www.sagepub.com/journalsReprints.navhttp://www.sagepub.com/journalsPermissions.navhttp://www.sagepub.com/journalsPermissions.navhttp://evi.sagepub.com/http://evi.sagepub.com/http://evi.sagepub.com/http://evi.sagepub.com/http://www.sagepub.com/journalsPermissions.navhttp://www.sagepub.com/journalsReprints.navhttp://evi.sagepub.com/subscriptionshttp://evi.sagepub.com/cgi/alertshttp://www.tavinstitute.org/index.php


2/25

95

The Modus Narrandi A Methodology for Evaluating Effects of Environmental Policy

J O O S G YS E NCatholic University of Leuven, Belgium

H A N S B R U Y N I N C K XCatholic University of Leuven, Belgium and Wageningen University, The Netherlands

K R I S B A C H U SCatholic University of Leuven, Belgium

‘Are we being effective?’ has become an increasingly central question inpolicy evaluation research. This article defends the position that this

question can only be answered by employing a methodology that is adaptedand can stand scrutiny in the policy field in which the evaluation occurs. Inour experience the evaluation of environmental policy requires a renewedeffort to develop methodologies that take into account the keycharacteristics of that policy area. The ‘modus narrandi’, based on causalnarrative story reconstruction, is our contribution to these efforts. It is amethodology for effect evaluation that looks at effectiveness and side effectswithin contexts characterized by causal uncertainty.

KEYWO RDS: causality; effectiveness; environmental policy evaluation;modus narrandi methodology; side effects

Introduction

This article aims to address some central issues in the evaluation of the effects of environmental policy. Implementation and effectiveness of policy have recentlybeen subject to inquiry at different policy levels and in different policy fields. Atthe European level, the European Environmental Agency (EEA) has beenparticularly active in trying to answer the question ‘are we being effective?’ (EEA,2001) and aims to develop appropriate methodologies in order to do this. Others

including scholars and practitioners (e.g. administrations) have engaged inanswering the effectiveness question. Most initiatives have been designed around

EvaluationCopyright © 2006

SAGE Publications (London,Thousand Oaks and New Delhi)

DOI: 10.1177/1356389006064176Vol 12(1): 95–118


http://evi.sagepub.com/http://evi.sagepub.com/http://evi.sagepub.com/http://evi.sagepub.com/


3/25

specific environmental subfields and concrete evaluation exercises, e.g. the evalu-ation of specific water treatment policies, air pollution abatement policies. In mostinstances, a case study is at the core of the research and evaluation methodology.

This article aims to contribute to a number of approaches that provide

elements of more universally applicable methodologies to measure the effec-tiveness and side effects of environmental policy (Bressers and Klok, 1991;Mickwitz, 2000). In addition, it presents ‘effect evaluation’, a methodology thattakes into account both effectiveness and side effects. ‘Effect evaluation’combines both approaches into one evaluative statement. We emphasize someconceptual and methodological issues within the field of environmental policyevaluation, not only because of our personal interest in specific methodologicaland conceptual challenges in this subfield, but also because environmental policyevaluation is of increasing importance. Environmental issues have achieved a‘sense of urgency and seriousness’. This has been accompanied by an increase in

both public and political interest. In addition, the amount of financial and otherresources allocated to both environmental policy-making and environmentalpolicy evaluation has increased significantly. Yet environmental policy is a rela-tively ‘young’ policy field. The first serious policy initiatives date from only a fewdecades ago. Their evaluation and, more specifically, the link between policy-making and environmental outcomes have only relatively recently becomeserious issues for evaluation scholars and professionals.

This article first addresses the central concepts of effectiveness and causalityand their role in environmental policy evaluation. Second, some of the keycharacteristics of environmental policy and their influence on subsequent evalu-

ative efforts are discussed. Third, a methodology is designed fitted to the realityof environmental policy.

Conceptualizing Effects and ‘Effectiveness’

Our central goal is the development of a methodology that allows claims to bemade about the ‘effectiveness’ of environmental policies. This is an inherentlynormative question because it involves an appreciation of whether and to whatextent a policy ‘works’. In order to answer this type of evaluation question, atleast three different evaluative subquestions have to be taken into account.

(1) Questions can be descriptive. These questions try to address the ‘what hashappened?’. question.

(2) Questions can refer to causal linkages. Causal questions do not simply ask‘what has happened?’ They try to link changes with the policy.

(3) Questions can be normative. These questions evolve around the centralquestion, ‘Are the results satisfactory?’ According to the EEA, thisquestion can be answered from the perspectives of effectiveness, relevance,efficiency or utility. We are especially interested in effectiveness. Althoughwe recognize the importance of relevance, efficiency and utility, the scope

of this article is limited to effectiveness, and more precisely, in relation toenvironmental policy.

Evaluation 12(1)

96




4/25

The effectiveness question cannot be separated from the two former types of evaluation questions (the descriptive and causality questions), if it is to beanswered in a realistic and satisfactory way. In fact, the methodology proposedstrongly emphasizes the fact that effectiveness evaluation builds on well-

documented and developed answers to the descriptive and causal questions.First, to further refine the conceptual framework, an important but often ignoreddistinction is made between main effects (intended) and side effects (un-intended).

Main EffectsThe main effects are those intended by a policy/intervention. They were antici-pated and wanted and can be divided into three categories: output effects,outcome effects and impact effects.

The output is defined as the tangible results of a measure (EEA, 2001). Output

has a rather short-term dimension. Output becomes evident shortly after theimplementation process of the policy instrument or even during the process, e.g.when a new type of permit becomes compulsory, the amount of permits grantedfrom the first day will be an output effect. These output effects are not necess-arily part of the actual policy objectives. They do not necessarily have an auto-matic or direct relationship with policy performance. For example, the numberof environmental permits granted, or increasing the number of field inspectors,can be policy objectives,but they are not a direct indication that the environmentis improving. They can only give an indirect indication of the intended policyresults. However, in practice, output effects are often the only effects that are

monitored.The outcome of the policy, defined as the response of the target groups to theoutput (EEA, 2001), corresponds in principle with the policy objectives. Wherethe output effects usually occur in the short term, outcome effects are most likelyto occur in the mid to long term. A change in environmental infringements, forexample, is a possible outcome of the number of extra field inspectors. Otherexamples of environmental policy outcome effects are reductions in emissions,increased recycling rates or shifts in the use of different modes of transport.1

Any effect that influences the state of the policy issue is defined here as animpact effect. Mickwitz (2000) would call this effect the ‘ultimate outcome’. We

choose not to use this term because of the significant distinction we makebetween outcome and impact. The impact effects or impact (EEA, 2001) of apolicy are often only visible in the long term. In the example of environmentalpolicy, impact effects are related to actual changes in the state of the environ-ment. Often these effects are expressed in terms of quality. Better air qualitycould be a possible impact effect if the policy was aimed at reducing the un-authorized emission of air pollutants.

Side EffectsSide effects are defined here as the unintended effects of a policy or intervention.

The difference between side effects and main effects is based on the criterion of ‘intention’. Intention was chosen because it was felt that other criteria fail in

Gysen et al.: The Modus Narrandi

97




5/25

Evaluation 12(1)

98

terms of clarity and consistency. Vedung (1997) uses ‘target area’ as a distinctcriterion. In his reasoning all effects within the target area are main effects. Thosewhich fall outside of the target area of the policy are side effects. This conceptu-alization is not necessarily contrary to our idea of intention,as long as target area

is defined as the part of reality that the actors try to shape (Van de Graaf andHoppe, 2000).2 Following from this definition we argue that ‘trying to shape’involves at least a passive intention. To avoid vagueness we suggest using inten-tion as the distinguishing criterion. It is important here to stress the differencebetween the criteria intention and anticipation (see Box 1). An effect can beanticipated without being intended: therefore these concepts should not be usedinterchangeably. Since intention is the defining feature, side effects can be antic-ipated or unanticipated. As will be demonstrated, whether they are anticipatedcan make a significant difference in many ways, but they remain side effects.

Further distinctions are drawn here between direct, indirect and derived side

effects. First, direct side effects are visible and measurable first-instance effects.These effects result directly from the policy, even though they are not intended.For example, the Flemish Law on Soil Pollution and Sanitation (1994) had adirect side effect, i.e. the establishment of an economic industry of professionalsin this field that did not previously exist. This led to the creation of at least 1000 jobs in the sector (Bachus and Bruyninckx, 1998). Second, indirect side effectsresult from direct side effects. They are mostly visible and measurable. They donot result directly from the policy but from another side effect. Drawing on thesame example, the creation of this economic sector has meant that additionaltraining and educational programmes are required and several of these now exist.

They are an indirect side effect. Third, derived side effects are difficult to separatefrom indirect effects. The difference between the two is that a derived side effectrequires an ‘extra effect’ or catalyst to occur. It doesn’t automatically follow froma direct or indirect effect. A good example of an ‘extra effect’ is the multipliereffect, a chain reaction of additional income and purchases that results in a finalincrease in total purchases that is greater than the initial increase in purchases.3

For example, because of the new market for soil protection and sanitation, strate-gic choices within the environmental consulting and services sector can beconsidered as derived effects.

Box 1. Example of a Side Effect of a Policy to Reduce Pollution

The Council Directive 91/676/EEC concerning the Protection of Waters against Pollu-

tion caused by Nitrates from Agricultural Sources was implemented in Flanders with a

strong emphasis on changes in the intensive pig-farming sector.The two main instruments

were a reduction in the number of pigs being farmed, and a collection of dung in special-

ized installations to prevent dumping in the environment. One side effect of this policy

– i.e. unintended – is a reduction in employment in the pig-farming sector. Clearly this

side effect was anticipated, demonstrated by the fact that measures are taken in the policy

to soften this negative side effect. Additionally it is clearly not a main effect because the

policy aims to reduce the level of nitrates not the employment rate.




6/25


99

Based on these conceptual distinctions, five types of ‘effectiveness’ can beidentified: (1) institutional effectiveness, (2) target group effectiveness, (3) impacteffectiveness, (4) societal effectiveness and (5) side effectiveness. Table 1 providesan overview.

Institutional EffectivenessInstitutional effectiveness is the ‘match of the output of a given policy or programto the output objectives of that policy or program’. We have chosen to attributethis particular term to this phenomenon because the term ‘institutional’ clearlyincorporates the link to the performance of those institutions or organizationsthat are expected to initiate policy actions. This can include government bureau-cracies at different policy levels, compliance agencies, etc. It is the performance

Table 1. Overview of Effects and Effectiveness

Effect Definition of effect Type of Definition of EffectivenessEffectiveness

Output E The tangible results Institutional The match of the output of

of a measure a given policy or programme

with the ‘output objective’

of that policy/programme

Outcome E The response of the Target group The degree to which the

target groups to the target group responds to

output the policy, due to the policy,

as aimed for by that policy

Impact The degree to which the

change of the policy issue,

Impact E The influences on the caused by the policy, is in

policy issue line with the policy goals

Societal The extent to which the

effect is in line with broader

societal objectives

Direct SE Visible and measurable

first instance efects.

These effects followdirectly from the policy,

even though they areThe extent to which a policy

not intendedhas produced unintended

Indirect SE Does not directly follow Side effects which are in line with

from the policy but from related policy objectives and

another side effect public interest

Derived SE The result of the

interaction of another

SE and an external

factor




7/25

of the institutions or organizations that is the focus of the evaluation, not whatfollows from their actions.

Target Group Effectiveness

‘Target group effectiveness’ means the degree to which the target group respondsto the policy, due to the policy and as intended by that policy (Gysen et al., 2002).It clearly leaves the institutional boundaries of the policy, and instead focuses onchanges to actor behaviour. For example, an evaluation would evaluate changesin emissions from a certain company, not changes in the water quality of the riverin which the emissions are deposited. It is important to acknowledge that targetgroup effectiveness measures outcome effects with reference to outcome objec-tives, not as proxies to impact effects.

Impact Effectiveness

A third type of effectiveness is ‘impact effectiveness’. This is defined as the degreeto which the change of the policy issue, caused by the policy, is in line with thepolicy goals (Gysen et al., 2002). Bruyninckx and Cioppa (2000) define this effec-tiveness (based on applying it to international environmental regimes) as ‘anenvironmentally effective institution eradicates or alleviates anthropogenicdeductions from and/or deposits to an ecological system or systems in balancewith the system’s natural regenerative processes’.

This definition provides a basis for evaluation, independent from the immanentobjectives of the institution or policy. However, we do not go as far as Bruyn-inckx and Cioppa. In our view the intended impact – or environmental objec-

tives – as formulated in the policy remains central to setting standards for theevaluation. Impact effectiveness is not goal-free. For example, environmentalquality cannot be judged apart from specific policy goals on environmentalquality. It is, to the contrary, a strictly goal-based evaluation.

Societal EffectivenessWithin the EEA framework (EEA, 2001) societal effectiveness corresponds withboth the relevance and utility questions. Societal effectiveness addresses thequestion whether or not the impact (or impact effects) satisfies societal needs: ‘Isthe effect a contribution to broader societal objectives?’ This is the point where

we switch from our dominant goal-based design to a goal-free design (Scriven,1991).4 Societal effectiveness is not measured by the direct objectives of thepolicy. It is goal-free in the sense that the evaluator will define goals on whichthe policy will be judged in terms of societal effectiveness. The evaluator will haveto choose the goals and the set of criteria based on a set of widely acceptedsocietal principles and goals. Sustainable development is such a concept andnational sustainable development strategies, which exist in almost every countrynowadays, are an expression of such broader principles and goals. This type of effectiveness measurement necessarily broadens the scope of evaluation. Itallows the evaluator to look further than what may be somewhat narrow policy

goals. However, it is not a substitute for one of the other effectiveness evaluationsbased on clear policy objectives.

Evaluation 12(1)

100




8/25

These types of effectiveness (institutional, target group, impact and societal)all remain part of what we have called ‘main’ types of effectiveness. The commondenominator is that they are all based on main – i.e. intended – effects of aparticular policy.

Side EffectivenessSide effectiveness is the fifth type of effectiveness within an integral approach toeffect evaluation. Side effectiveness is partly based on a goal-free approach. Thecriteria of merit and the standards of performance are based on the goals andobjectives of the stakeholders and the objectives of other, related policy fields.For example, if the focus is on employment side effects, employment policies willbe analysed to find relevant policy objectives. Side effectiveness expresses theextent to which a policy has produced unintended effects which are, or are not,in line with related policy objectives and stakeholder interests. A set of merit

criteria and standards of performance are constructed and the side effects aremeasured against these. These effects can be direct, indirect or derived effects.Contrary to our approach on main effects and effectiveness, a further distinctionis not drawn between various types of side effectiveness. Essentially, the distinc-tion between direct, indirect and derived side effects is only valuable for bridgingthe causality gap between policy and effect. In the case of main effects the distinc-tion is important for the same reason. In addition it is essential to be able todistinguish between the various types of effectiveness because of possibleconfusion of users of policy evaluation results, for example, to avoid a situationwhere findings of institutional effectiveness are mistaken for results of impact

effectiveness. Obviously, the findings of these two do not have the same status.In the case of side effects, by definition they are not linked with policy objectives,and hence, further distinctions based on stated policy goals are irrelevant.

The result will be a statement on the side effects of a policy, which adds valueto the more mainstream effectiveness evaluation, because it is broader than onlylooking at the policy goals. Yet, it must be emphasized that the evaluation of sideeffects is also valuable if carried out by itself, as for example with environmentaland social impact assessments.

Particular Difficulties Associated with the Evaluation of

Environmental Policies

Evaluating the effectiveness of any field of policy-making cannot be disconnectedfrom the typical features of the issue at stake and the characteristics of the policyfield. The following paragraphs elaborate this argument in relation to environ-mental policies and their effects.

Characteristics of Environmental PhenomenaEnvironmental phenomena are characterized by some special features, whichtaken together distinguish them to a certain extent from purely social phenom-

ena. The term ‘environmental phenomena’ is used in this article to include bothnatural phenomena, with no direct human intervention,and social–environmental


101




9/25

interactions. First these characteristics will be listed and grouped. Subsequentlythe consequences of these features for environmental policy-making and evalu-ation are discussed.

Complexity Environmental phenomena combine complex patterns of impactand causation. Often there are serious knowledge deficits, uncertainty and a lackof data and to a great extent the phenomena rely on expert knowledge and scien-tific rationality.

(a) Complex patterns of impact and causation. The diffuse character of environmental issues adds to the difficulty in establishing clear-cut causalrelationships. A typical example is air quality. It is influenced by many factors.Even if we would focus on CO2 emissions, they are still manifold, including traffic,household consumption, industrial emissions, power plants. This makes anyevaluation based on the links between policy objectives and effects inherently

difficult.5

In addition, environmental phenomena typically cross sovereign andadministrative boundaries. Events in one place can lead to effects elsewhere. Awell-known example is the acid rain in Scandinavia that is partly caused by indus-trial emissions in Central and Western Europe. Phenomena also tend not to belimited by policy-makers’ time-frames. Lafferty and Meadowcroft (1996) refer tothe combination of these characteristics as the ‘time-scale effect’. As today’sevents can cause effects many years from now, this adds to the difficulty in deter-mining clear-cut causal relationships.

(b) Knowledge deficit. Lafferty and Meadowcroft (1996) consider knowledgedeficits in this field as closely related to the uncertainty, complexity and techni-

cal difficulty of the environmental phenomena as they are very difficult tocomprehend and therefore knowledge about them is lacking. The complex inter-action of physical, social and biological factors requires advanced scientificrefinement. Global warming is a good example. We still lack consensus on itseffects and, without consensus, uncertainty dominates and complicates policydiscussions.

(c) Lack of data. Even if the detailed physical, social and biological relation-ships and interactions were understood, an important problem would still remain:monitoring and gathering data. Because of other features related to complexity,such as technical complexity and diffuse sources, monitoring and data gathering

can be very difficult and therefore very expensive. Next to these physicalobstacles, environmental policy is a relatively young policy field which means thatgood longitudinal data are often scarce.6

(d) Scientific rationality. Environmental phenomena depend heavily onscience to be explained. Most such phenomena cannot be understood by non-experts. It is clear that because of the complexity of the environmental phenom-ena that are increasingly important, such as global warming, biodiversity orgenetic modification, the reliance on science has intensified.7

Irreversibility and thresholds Irreversibility is defined here as when recovery,

whether natural or man-made, is no longer feasible even within the longest knowntime-frame. Irreversibility also means that there is a sharp and abrupt cut-off

Evaluation 12(1)

102




10/25


103

point beyond which effective environmental policy-making is impossible.Examples of irreversibility include degenerated topsoil, for instance, in the desertarea of Africa; species becoming extinct; coral reefs dying.

Closely linked to irreversibility is the existence of thresholds. Thresholds refer

to certain points at which processes or situations change dramatically. Thischange is often preceded by a period marked by the absence of any clearly visibleeffect. The more a threshold is passed, the more difficult it becomes in generalto reverse the process. Passing the ‘final threshold’ leads to irreversibility.

The environment as a public good The natural environment is – in principle –an example of a public good. Public goods and externalities both pose specificdifficulties for policy-making and for evaluation. For example, the costs of main-taining public goods are too high for any individual and nobody can be excludedfrom the advantages (Mueller, 1979).

Closely related is the concept of externalities. An externality occurs when anindividual of a group imposes high costs to society (STOHO, 1996). Weale (1992),while discussing pollution, observed that the distribution of costs and benefitsthat are typically present when actors behave and organize around public goodsand cause externalities will not necessarily be equal. Some will suffer more fromsome externalities while others will benefit more from the effects of the publicgood.8

Both the character of public goods and the distributional aspects of externali-ties further complicate environmental issues as policy ‘objects’. Figure 1 summar-izes the different features of environmental phenomena.

Influence on Environmental Policy-Making and EvaluationEnvironmental policy is partly determined by the features of the socioenviron-mental phenomenon it aims to deal with. The more these characteristics areevident in the subject of the policy, the more difficult it is to formulate and

Figure 1. Features of Environmental Phenomena

Complexity Complex patterns of impact and causation

Lack of data

Knowledge def icit

Science rationality

Irreversibility and threshold

The environment as a public good




11/25

implement good policy. The same reasoning can be applied to the evaluation of environmental policies.

Policy development and implementation is a process of searching for accept-able compromise and feasibility given specific contexts. This is based on social,

political and economic dynamics more than on the purely ‘objective’ scientificfeatures of the problem at hand. As long as parties involved do not agree on thenature and the effects of a certain natural phenomenon, controversy will persist.This means that, in addition to the characteristics described above, a process of social mediation determines the specific characteristics of the policy outcome.The results of this process are often ‘below’ the expectations of the rational,scientific advice formulated regarding the policy. This is important to bear inmind when evaluating the effectiveness of any given environmental policy.

Uncertainty, for example, is (in principle) an ‘enemy’ of the policy-maker (andthe evaluator for that matter). Uncertainty brings ‘guestimation’ and risk taking

to the policy arena. The uncertainty in policy-making is a function of the uncer-tainty in science. Limited knowledge influences the possibilities of the policy-maker. Sometimes the policy-maker will have no other option than to makepolicy on a subject that is not fully understood. In such a case the policy has blindspots. The problem of lacking data aggravates the uncertainty problem. Incorrectpolicy theories can be the result of policy formulated on the basis of incompleteinformation. This problem finds its way into evaluation: without data, evaluationsare typified by guesses and estimates.

Thresholds are also subject to different interpretations within the politicalarena. Whereas scientists regard them as a matter of determining non-linear

processes and their – mostly environmental – consequences, thresholds are lessdefinite and are subject to social debate and construction in policy processes.Other factors – such as the cost of the process, social considerations like thevariable effects of a threshold in different locations or distributional aspects –come into play. This will result in political thresholds that can also be the subjectof an evaluation.

The previous two points also have an effect on two of the crucial politicalelements of policy-making. Environmental policies have to be responsive toconcerns about accountability and liability. An event today can result in effectsmany years from now, and in another place. If the ‘other place’ is in another

country, national legislation is no longer sufficient. Supranational structures arerequired to deal with this challenge but these structures (or regimes) are char-acterized by great complexity and inertia (Keohane et al., 1993; Kutting, 1998;Levy and Young, 1995; Peterson, 1997a, 1997b; Young, 1997).

Policy-making and evaluation are also affected by time-lag effects. Policy-makers and evaluators are sometimes confronted with a policy the effects of which are not all noticeable yet. Second, as Weale (1992) stated, future gener-ations are not well represented in relation to contemporary decisions. This resultsin problems with the value system on which the evaluation criteria are based, aswell as problems of uncertainty.

To conclude, the inherent or specific characteristics of environmental problemsare accentuated and take on additional meaning in a policy-making process. This

Evaluation 12(1)

104




12/25

process is bounded by additional limits because of social and political bargain-ing. The result is often a policy that is only indirectly linked to presumed eco-logical objectives. These elements put stress on the evaluation of environmentalpolicies. The next section presents a methodology that aims to answer relevant

evaluation questions in these kinds of environmental policy setting.

Introducing the ‘Modus Narrandi’ as an ‘Effect EvaluationMethod’ for Environmental Policy

The ‘modus narrandi’, as we have named this methodology, responds to twofundamental evaluative needs: 1) assessing the legitimacy of a policy (the legiti-macy argument) and 2) addressing the features of environmental policy whenaiming to ‘prove’ the causal relationship between the policy and effects (thecausality argument).

Before exploring these arguments more deeply it is necessary to clarify whatis meant by ‘effect evaluation’: an effect evaluation brings together the findingsof an effectiveness evaluation and a side effects evaluation.

Assessing LegitimacyEffectiveness evaluation aims to evaluate the intended effects of a policy inter-vention. These effects are usually evaluated in light of the goals of the interven-tion. Four types of effectiveness evaluation have been distinguished: outputeffectiveness, target group effectiveness, impact effectiveness and societal effec-tiveness. These all have in common that the criteria used to make evaluative

statements are the goals or objectives of the intervention. Therefore there is noneed, within an effectiveness evaluation, to weight these effects in relation toother elements. Scriven (1991) refers to these as autonomous criteria as opposedto compensatory criteria. The former stand alone, while the latter are traded off:the performance of one evaluand is compared or traded off against the perform-ance of another evaluand.

In addition to these types of effectiveness, a fifth type of ‘effectiveness’ wasdefined: ‘side effectiveness’. Intuitively one might think that the criterion used inside effect evaluation is more of a compensatory criterion, as Scriven suggests,but this is not the case. Side effects are evaluated using criteria derived from the

context of the evaluand, without trade-offs. This methodology uses goals andobjectives of other related policy fields and also stakeholders to assess sideeffects. Each of these is considered, in the first instance,as a stand-alone criterion.

The study of all effects is important because we strongly believe that the legit-imacy of an intervention is based on its effectiveness and on its side effects, butmost of all, on the synthesis of the assessments of both effectiveness and sideeffects. This synthesis is more than the mere combination of the findings of bothevaluations. It also comprises trading off the effects and the side effects. Thissynthesis is the result of bringing the effects and side effects together into acompensatory evaluative frame.9

The best way to assess the legitimacy of a policy is by carrying out a compre-hensive assessment of all effects. An effectiveness evaluation runs the risk of


105




13/25

missing important side effects that could be so detrimental that a highly effec-tive intervention does more harm than good or is perceived as such. Examplesinclude employment or economic side effects of environmental policies, or effectson social inequality. A side effect evaluation, on the other hand, ignores the main

aim or intended effects of the intervention. Indeed, positive side effects mighteven be achieved more efficiently via interventions in other policy fields or viaother instruments in the same policy field. Either way, both an effectivenessevaluation and a side effect evaluation, conducted independently of one another,offer only a partial view on the intervention and therefore contribute onlypartially to an understanding of the impacts and the legitimacy of an interven-tion. The best argument for legitimizing the intervention can therefore be foundin a combined, compensatory assessment. Evidence for the importance of asynthesis approach is provided by authors within the impact assessment school(Barrow, 1997; Burdge, 1994). Both social and environmental impact assessments

aim to involve effects other than the intended effects on policy decision. Themethodological aspect of this will be discussed under the heading ‘The Steps of the Modus Narrandi’.

‘Proving’ CausalityThe key challenge for effect evaluation is ‘proving’ the causal relationshipbetween the effect and the policy. In theory, an experimental design offers thebest solution to this challenge. However, the practice of policy-making in general,and environmental policy-making in particular, is very different from the ‘clinical’approach assumed in experiments. In addition, the characteristics of environ-

mental phenomena and their impact on environmental policy-making and evalu-ation, are significant and create a difficult set of challenges for effect evaluation.The essence of this challenge lies to a great extent in the issue of causality.Whether side effects or effectiveness are studied, the problem of causality has tobe tackled by linking interventions to (side) effects in order to make causalclaims, rather than claims about observed correlations. This approach opposesthe Humean model of constant conjunction. Mahoney (2004) discusses thisapproach as follows:

According to Hume (1748/1788), we infer causation when we repeatedly observeputative causes followed effects; that is, two events are ‘constantly conjoined’ in our

experience. However, we can never actually confirm the existence of causation becausethe imagined ‘necessary connections’ linking the two events cannot themselves beobserved or known. In contrast to this Humean model, scholars concerned with causalmechanisms argue that exploration of the ‘black box’ connecting independent anddependent variables is essential to good causal research. On this view, ‘we are not satis-fied with merely establishing systematic covariation between variables or events; asatisfactory explanation requires that we are also able to specify the social “cogs andwheels” that have brought the relationship into existence’.

Mahoney addresses the problem of the ‘black box’ within the framework of social science. He claims that, unless causal mechanisms are proposed, the

problem of the black box will not be resolved. Causal mechanisms are empiri-cally underspecified properties of particular causal agents. Related to causality

Evaluation 12(1)

106




14/25

in evaluations, this means that, in order to explain the causal links between anintervention and an effect, one must investigate the black box that will containexplanations as to why the effect exists or what influenced its observable features.Uncovering the black box provides insights that will allow us to move from corre-

lation relations to causal relations. In our experience, opening one black box willoften, certainly in complex settings, lead to a new black box. Uncovering theselayers of black boxes is fundamental for a more solid understanding of the causal-ity behind the correlation. Causal proof lies in uncovering the causal mechan-isms. Causality, in this approach, can be seen as the ‘sum’ of two elements: oneis causal proximity (Homer-Dixon, 1995), the other is causal contribution. Causalproximity expresses both the strength of the link between intervention and effectand variable and hence, second, the validity of the established relationship.Strong proximity signifies a direct link between intervention and effect. Weakproximity means that several other factors in a causal chain link the intervention

to the effect. Causal contribution (Gysen and Bachus, 2003) indicates the import-ance of the ‘correlation’ (or in the case of strong proximity, causality) of onevariable as opposed to the other variables in a multi-correlation relationship withan event. This is an addition to Homer-Dixon’s ‘causal proximity’. This refine-ment is necessary in order to fully grasp the relationships and especially to framethe causal relationship within a context characterized by many influencing factorsand multiple black boxes.

This approach to causality will be central to the methodology for ‘effect evalu-ation’. Two groups of evaluation methodologies are commonly used in the litera-ture, each based on a different conceptualization of causality. The first group,

referred to here as the rationalistic methods, is based on a causality concept thatleans in the direction of the Popperian falsification approach (Popper, 1968).Policy-making is evaluated as a social activity based on certain theoreticalassumptions that can be falsified by setting up policy experiments or by search-ing for ‘failed’ or ‘falsified’ policies. The methodological translation of thisapproach is found in evaluations that are based on purely quantitativeapproaches, modelling and on experiments. It is clear to us that, in terms of thesocial sciences in general and evaluation methodologies for environmental policyin particular, a rationalistic or Popperian view is untenable. First, propositionsunderlying policy-making are generally not ‘falsifiable’. Second, for the over-

whelming majority of social theory and policy-making, finding ‘the case that falsi-fies the theory’ or the policy case that doesn’t work, is rather easy as universalsocial scientific laws are almost non-existent.

There is a set of alternative methodologies that has the potential to getaround this problem. These could form the foundation for a more realistic andworkable environmental policy evaluation methodology. This second group of methodologies is referred to here as the hermeneutic methods. First, one couldbase the evaluation methodology on the causal claims embedded in a varietyof grand policy theory. However, this option is confronted with the problem of a lack of an accepted general theory: no single grand theory can explain all

causal links in social science, and hence in policy contexts. A second option isworking in an inductive way. The researcher starts with observation of reality


107




15/25

Evaluation 12(1)

108

and uses observation to explore the issue, generate hypotheses about the policyprocess, and the possible effects. The third option focuses on middle-level policytheories. With this option a shift towards a lower level of abstraction (middle-level) occurs and the policy is approached from a more context-specific point of

view. These middle-level theories can address the general characteristics of politi-cal or bureaucratic policy-making, theoretical insights regarding the environ-mental policy-making dynamics, or other more specific and partial approaches tothe policy-making context. The fourth and last option implies a change inthinking about the causality question. This option is to be used where no directlink or relationship can be established or tested. The goal is to reconstruct thepolicy process with regard to the effects based on a ‘probable causal explanatorymodus’. It uses middle-level theories, inductive claims and where possible quan-titative data or ‘numbers’ to explain and illustrate a causal story. By doing thisin a thorough and comprehensive way, the burden of proof shifts from the evalu-

ator to his critics. Others will have to demonstrate that the causal story of howa policy is (or is not) linked to certain outputs, outcomes or impacts is incoher-ent, incomplete or false by demonstrating a more probable causal narrative.Figure 2 illustrates the two frameworks presented here.

Boskma and Herweijer (1988) argued that ‘qualitative case-study designsfocused on causal analyses can be fitted into the rationalist explanatory model’,i.e. hermeneutic methods can lead to results similar to those of rationalist results.

Figure 2. Conceptualization–Method Relationships

Conceptualization of Causality Methods

Methods

Hard theory

cf. ‘Falsifaction’ (Popper)

Rationalistic methods

Focused on ‘proving’ causality

through experi mental research

designs

Hermeneutic methods

Focused on ‘understanding’

causality through qualitative

research

Prove

Understanding

Conceptualization of Causality

Soft theory

Causal narrative story

Deductive analysis

Inductive analysis

Midlevel analysis




16/25

Robert Yin (2003) adds to this discussion that if a chain of evidence is estab-lished, the use of case studies can lead to general explanations. The principle isto allow an external observer to follow the derivation of any evidence (Yin, 2003).Even the methods that are traditionally linked to ‘explaining’ can support claims

about effectiveness.The EEA (2001) has followed similar reasoning in an attempt to design a ‘path

of choice’ based on evaluative situations and various methodological options.First,where causal links are few and predictable, standard models may be derivedby examining a small number of case studies. Second, where the application of ameasure is differentiated in space or by target sector, comparative case studiescan be used to help identify causal relationships. Third, where the target groupis relatively small, in-depth interviews can be used. Fourth (and this will be ourmethodological starting point ), when the link between a policy and its impact onthe environment is too diffuse or extended and therefore the effectiveness of the

impact effects not directly measurable, the focus of the evaluation can shift toimmediate outcomes and outputs as a rough proxy for the impact effects. Allthese methods largely adhere to the hermeneutic group of methods, but we wouldargue they should be acceptable as methods for identifying a causal relationship.

In an effort to tackle these problems, we suggest a methodology based on thedistinctions between the different effects. Because of the difficulty that lies inestablishing the effectiveness of a policy, we have chosen to focus on causal argu-mentation and assessment, rather than focusing on establishing hard proof. Werefer to this approach as the ‘modus narrandi’10 because this term emphasizes boththe method (modus and the grammatical construction which implies ‘the way it

should be told’) and the causal story (narrare). It is based on a careful explorationof the policy and its various components (e.g. policy process, policy implementa-tion, organization, actors, means). These ‘policy and contextual elements’ will helpus to gain an understanding of the causal relationships and this understanding willbe the basis of a reasonable explanation of the effectiveness. Finally, a ‘reasonablecausal chain which is contextually constructed’ (Homer-Dixon, 1995) will be theresult. The chain brings together the various elements which make up the pathfrom policy through context elements to output, outcome and impact effects.

The Steps of the Modus Narrandi

The point of departure for the application of the modus narrandi is the conceptof ‘formal evaluation’ as introduced by Dunn (1994) and further elaborated byBressers (1998). Formal evaluation distinguishes four phases: explaining thepolicy theory, estimating the goal attainment, establishing the effectiveness andexplaining the effectiveness. Based on the previous sections, we make a numberof amendments. First, we rename the first step ‘reconstruction of the effectsspectrum’. This step consists of the reconstruction of the policy theory supple-mented with the reconstruction of any ‘blind spots’. Next, we change ‘estimatingthe goal attainment’ to ‘valuing of effects’. Finally, we combine the last two stepsand call this ‘establishing causality’. In this step we make use of the two measure-

ments, causal proximity and causal contribution. Figure 3 illustrates the differentsteps, which are then elaborated.


109




17/25

Evaluation 12(1)

110

Step 1: Reconstruction of the effects spectrum Step 1 contains two phases.Phase one is the ‘reconstruction of the policy theory’ which Blommestein et al.(1984) define as ‘all the assumptions that form the basis of the policy’,11 i.e. thepolicy-makers’ interpretation of reality. Van de Graaf and Hoppe add to this thenotion of the arguments and elements of knowledge on which the policy theoryis based. They summarize the concept as follows:

The policy theory indicates why, in the judgment of policymakers, in a specific policyfield they have acted in a specific way. The policy theory is a reflection of the ration-ality of the policy maker, behind the policy. (2000: 70)

This rationale is at the core of the choice of policy instruments and otherelements of implementation. When reconstructing the policy theory, the aim is

Figure 3. The Modus Narrandi

Reconstruction

of the eff ects

spectrum

Empirical dataReconstructionof the causalnarrative story

Illustrate causallinks

Reconstructionof the causal and

impact model

Triangulation

Causalproximity

Judging the meritof the policy theory

Policy theory

Causalcontribution

Valuing the

eff ects

Establishing

causality

Effect

assessment

Establishingthe goals

underlying principlesto achieve the goal

Establishing the




18/25

to identify the relationships that make up the fundamentals of this rationality. InAnglo-Saxon literature this approach is called ‘policy theory based evaluation’(e.g. Posavec and Carey, 2003; Rossi et al., 2004). This first step provides an insightinto the main intended effects and the anticipated side effects.12 Logically it does

not cover the unanticipated side effects.Therefore phase two is necessary: the ‘reconstruction of the blind spot’. This

blind spot refers directly to the policy theory and hence to the assumptions of the policy-maker. It is the part of the policy that is not covered by the policytheory and therefore leaves room for the existence of unanticipated side effects.Van de Graaf and Hoppe (2000) suggest reconstructing (the policy theory) by aclose reading of policy text(s). We suggest supplementing this with several quali-tative interviews with those involved in writing the policy text(s) and those withrelevant high-level knowledge about the choices that were made. These inter-views can add important information, in addition to what are sometimes rather

concise policy text(s), especially on implicit policy goals and assumptions. Usinga combination of methods or at least checking information based on differentsources (triangulation) is of great value when constructing an effect ‘spectrum’.13

At this point in the process, a draft causal story can be designed by the evalu-ator in terms of the expectations embedded in the policy. However, it must beemphasized that at this stage it is a draft causal story only.

Step 2: Valuing the effects The second step in formal evaluation is measuringthe effects and attributing a value to them. At this stage, separate approaches arerequired for main and side effects.

(a) Main effects. The aim is to find out to what extent the policy goals havebeen met. Therefore measurements14 of the effect are required that allow theircomparison with stated policy goals. After the comparison, a certain value canbe attributed to the effect in terms of goal attainment. We emphasize the import-ance of making a clear distinction between goal attainment – valued at this stage– and effectiveness, which will be judged at the end of the evaluation. In contrastto what other scholars such as Scriven (1991) and Mickwitz (2003) claim, webelieve it is important to draw a distinction between those two concepts.15 Toemphasize this difference we explicitly use the term goal attainment as opposedto the more widely used and differently interpreted ‘goal achievement’. In our

view, goal attainment as such does not say anything about the causal relationshipbetween intervention and effects. It looks only at the effects and compares themto the policy goals, which is exactly what we intend to do at this stage. Forinstance, if a policy on water quality aims at a concentration x1 of a substance Xin the water then the concentration level of X (x2) is compared to x1. If the actualstate does not meet the objective then goal attainment is negative but if theobjective is met then goal attainment is positive. So this phase of the evaluationonly measures the effects and compares them to the policy objectives.

(b) Side effects. The methodology for side effects is more complicated. Theinitial step is to define the set of side effects that will be included in the evalu-

ation. For anticipated side effects we can use the same method as for main effects.For unanticipated side effects we need to use several other methods, including:


111




19/25

analysis of environmental policy literature on a specific issue; results of previousevaluations; ex ante estimations of possible costs and benefits; and finally, andcrucial in our opinion, expert interviews with a variety of actors, who have beenclose to implementation of the policy. After a short list of side effects has been

created, they need to measured. This is no different from measuring main effects.The main difference lies in the valuing of side effects. In practice the twoapproaches will be used simultaneously. First, the policy objectives of the relevantpolicy fields to which the side effects could potentially belong will be looked at.These are the policy fields in which policy objectives with regard to these effectsare formulated. For instance, if the side effect is a given concentration of parti-cles of nitrogen oxide (NOx) then the air quality policy objectives for NOx maybe investigated. The second approach places the stakeholder perspectivecentrally in order to (1) reflect the public interest and (2) emphasize the centralposition of those who are unintentionally affected by the policy. It implies a focus

on those who are affected by the policy, while using their perspective as a refer-ence point for valuing the effect. The challenge is to move beyond thepositive–negative dichotomy and look for gradations for the valuation of sideeffects. For example, if employment in the pig-farming sector has decreasedbecause of nitrate reduction policies, this can be considered as a negative sideeffect. However, this becomes interesting information only if we can place itwithin the further policy objectives of the agricultural sector as a whole, orpossibly in light of stated employment policy goals. It also depends on the situ-ation of those who have lost their jobs. In other words, valuing side effects is adifficult task which requires clear distinctions that are known to (and accepted

by) all those involved in the exercise.The reconstruction of the effect spectrum provides us with a list of effects.Measuring the effects is important in terms of attributing influencing factors tothe effects. Policy evaluators mostly have to rely on existing datasets about thedifferent types of effects in order to measure them. Gathering extra or new datanormally falls outside the policy evaluator’s tasks. We visualize goal attainmentby using indicators like initial situation, actual measurement and distance totarget. We do not want to limit our appraisal to binary outcomes such as goalattainment ‘yes’ or ‘no’.

Step 3: Establishing causality Step 3 is to establish and explain the causalrelationships between the policy and the effects. We draw no distinction betweenthe two (establish/explain), because the aim is to establish the causality througha methodology that is based on uncovering and explaining cause–effect relation-ships. It is at this stage of the methodology that we need to offer a solution tocounter the inherent features of environmental policy evaluation. Assessingcausality and related concepts relevant for evaluation purposes have already beendiscussed. This third step in the methodology bridges the gap between the diffi-cult and complex causal setting of environmental policy evaluation and the ambi-tion to make strong and causal evaluative statements.

This third step is also necessary to link the measured effects (the result of thegoal attainment analysis) to the policy. Only by doing this can we distinguish the

Evaluation 12(1)

112




20/25

effects of the policy from the potential effects of other explanatory factors. Onlythe effects caused by the policy or intervention can be labelled as policy effects,whether they are main or side effects.

The first two steps provide the material on which the causal chains can be

constructed. Crucially important for evaluation purposes are the linkagesbetween the elements of the chain. These linkages are the expression of causal-ity. Causal proximity and causal contribution are used to refine these expressionsto the level where scientific reliability and real life utility and credibility meet.At this point the meaning or value of the elements that make up the links in thechain should to be clear.

The core activity at this stage is to build a causal narrative ‘story’. This story ispresented as a chain of events/elements. Using causal contribution and causalproximity, the real life linkages between the elements are reconstructed. Thisshould be done using as many information sources as possible. Data from exper-

iments are preferred, but given the characteristics of environmental policy this islikely to be an ideal rather than reality. We suggest using surveys, focusing onboth qualitative and quantitative data, expert interviews and document analyses.These sources should provide correlations or confirmations on which to buildusing qualitative data. The crux of the methodology is that all these data areorganized in a structure (the chain) that offers a clear view of the causal relation-ships between the first (policy) and the last elements (effect) in the chain. Theaim is not to establish statistically significant results, but to explain why and howcertain effects are present or not. If the causal story line is carefully constructed,it will provide far more information about why, how and with what sorts of

expectations and results policies have been made and therefore how they can beimproved.

Effect assessment – a combined appraisal of the effects Once the causal narra-tive stories have been constructed in terms of both effectiveness and side effects,these separate stories need to be combined into one effects evaluation statement.This statement brings together the results of the effectiveness evaluation,whetherit was an institutional, target group or impact effectiveness evaluation, and theoutcome of the side effects evaluation. This exercise should be done in the mostopen and transparent way. At this stage, the evaluator has to weight the different

elements – that is results – of the evaluation exercise. The weighting of the differ-ent elements is the core of the effect evaluation’s final statement and can beapproached from a descriptive or a prescriptive angle. Although we are generallynot very keen on using the prescriptive approach, we believe that in this situationonly a prescriptive statement can provide additional value. A descriptive approachat this point in the process will be limited to an echo of the outcome of step 3. Inother words, the evaluator has to make some kind of bad/good/better/ best state-ment at this stage. The problem with the prescriptive approach – namely what‘yardstick’ to use to make such claims – can be partially solved by a high level of transparency. This transparency invites critics to scrutinize this statement. As long

as a ‘yardstick’ has been explicitly applied, it should be made clear that judgementstatements are the view of the evaluator.


113




21/25

The core of the statement lies in valuing the effects in a compensatory way.The outcome of the effectiveness and side effects evaluations will be comparedwith each other and a value statement will be made on the whole, which considersvarious positive and negative effects. Only by balancing out the intended and

unintended effects can one make a statement on the legitimacy of the givenintervention.

This modus narrandi provides a statement on effectiveness, on side effects andon the overall impact of the intervention. In practice all identified (side) effectsare listed and fed into a valuation matrix. The compensatory valuation uses theelements of the matrix and builds a comprehensive story. This story is the evalu-ative effect statement based on which the legitimacy of the policy is supportedor criticized and also its effects – both intended and unintended – are explained.

Conclusion

Policy evaluators are confronted with a dilemma. First they acknowledge theneed for good and solid evaluations, preferably based on hard proof. However,they are also aware of the context in which they try to accomplish this objective.In environmental policy this context is inherently not ‘hard-proof friendly’. Nextto this methodological discussion, evaluations are often used to make statementson the legitimacy of a policy. This article claimed that a side effect evaluationand/or an effectiveness evaluation, conducted independently of each other, donot provide a sufficient basis upon which to draw conclusions about the legiti-macy of a policy. The legitimacy of a policy can only be found in the synthesis of

the two.For effect evaluation the article proposed a methodology that takes intoaccount the specific conditions and characteristics of environmental policy,without compromising on the expectations attached to causality-based evalu-ations. In fact, the modus narrandi is a methodology that builds explicitly on acausal narrative story that looks into the relationships between the variouscomponents of a policy. To do this, it makes a distinction between main (output,outcome and impact) effects and side (direct, indirect and derived) effects.Furthermore it looks into the different components of the policy and the immedi-ate context. These elements are the fundaments of a causal chain. The connec-

tion between the elements of the chain is indicated through two measurements:causal proximity and causal contribution. These connections are the heart of theanswer on causality.

In sum, we hope that this methodology opens the door for effect evaluationsin policy fields such as environmental policy where context and inherent charac-teristics have been important obstacles. By levelling the playing field for effectevaluations we hope to make a modest contribution to better evaluations andhence better policies.

Evaluation 12(1)

114




22/25

Notes

1. European Commission, ‘Towards a methodology for evaluating the effects of measures taken to implement EU environmental legislation’, first draft.

2. The original, Dutch definition of beleidsveld is: ‘het beleidsveld is dit deel van de

werkelijkheid dat de actoren naar hun hand proberen te zetten’.3. Thomson elearning: http://lms.thomsonelearning.com/hbcp/glossary/glossary.taf?gid=

39&start=m4. The importance of his plea is underlined here but we are also sensitive to Patton’s

(1997) remark as to the dangers of this approach. Patton warns about the risk of unuseful evaluation findings when goal-free evaluation is used. In Patton’s viewScriven leaves out the evaluation needs of the local programme staff. Additionally,Patton claims that ‘goal free evaluation carries the danger of substituting the evalu-ator’s goals for those of the project’ (1997).

5. In most cases in the literature (e.g. Weale, 1992) the diffuse character is referred towithin the framework of pollution, but it also holds for non-pollution related fields of

environmental policy-making such as biodiversity or soil degradation.6. For example, in Flanders we have only had a qualitative and quantitative develop-

ment environmental policy since the 1980s and this in turn affects the state of the artof monitoring. Lack of data still tends to be a serious problem, not only in Flandersbut in general. Indicator exercises such as the ‘Environmental sustainability index’(ESI, Yale and Columbia) encounter major data problems. These affect the quality of the exercise.

7. We rely on scientists not only to explain these phenomena but also to discover andname them. If it wasn’t for scientists the hole in the ozone layer would never havebeen discovered (Hannigan, 1995: 236).

8. Mining natural resources in third-world countries is a good example. In most cases

the cost of the mining is borne by local people. Not only environmental (e.g. adamaged environment) but also economic (e.g. low wages and expensive finalproducts) and social (poor general living conditions) costs are often involved. Thebenefits of these mining operations go to the owners of the mining companies, oftenmultinational companies. Closer to home there are numerous examples from heavyindustries. Often, the members of the boards of large polluting companies do not livein the vicinity of the plants. The poorer, often uneducated workers are the ones thatinhabit the areas around the plants and suffer as a result of the pollution. These peopleare not empowered to change the balance between costs and benefits. Even if theywere well informed about the situation, they would still experience difficulties ascer-taining the best way to address these issues.

9. To evaluate the ‘effect’ of a policy we use the compensatory approach. The final assess-ment of the ‘effect’ of a policy involves a trade-off between the different (side) effects.This is in contrast to the separate evaluation of effectiveness and side effects, whichis done using stand-alone criteria.

10. We deliberately chose this name to indicate the relationship to Scriven’s modusoperandi method (Scriven, 1991), a procedure for linking cause and effect by thereconstruction of a chain of events preceding the effect and the ambient conditions.Even though some similarities seem to exist at first, there is an essential differencebetween the modus narrandi and the modus operandi method. This difference lies inthe focus of the modus narrandi on the narrative reconstruction of the causal story.The modus operandi offers a way of dealing with situations where it is impossible to

use a control group. The modus narrandi tells us more precisely how.


115




23/25

11. Translation by the authors.12. These anticipated side effects can be identified through methods such as environ-

mental impact assessment, social impact assessment, cost–benefit analysis, etc.13. The next logical phase, after identifying the effects and therefore also reconstructing

the policy theory, is to evaluate the policy theory. Although this is an interesting evalu-ation question, it is beyond the scope of this article.

14. These measurements can come from official statistics, other existing sources, or some-times have to be gathered by the evaluator.

15. Here, goal achievement is synonymous with goal attainment.

References

Bachus, K. and H. Bruyninckx (1998) Tewerkstellingseffecten en economische impact vanhet bodemsaneringsdecreet . Leuven-Mechelen: HIVA-OVAM.

Barrow, C. (1997) Environmental and Societal Impact Assessment: An Introduction.

London: Arnold.Blommestein, H., J. Bressers and A. Hoogerwerf (1984) Handboek beleidsevaluatie: Een

multi-disciplinaire benadering. Alphen aan den Rijn: Samsom.Boskma, A. F. and M. Herweijer (1988) ‘Beleidseffectiviteit en case-studies’, Beleids-

wetenschap 2(1): 52–69.Bressers, J. (1998) ‘De evaluatie van beleid’, in A. Hoogerwerf and M. Herweijer (eds)

Overheidsbeleid: Een inleiding in de beleidswetenschap, pp. 119–41. Alphen aan de Rijn:Samsom.

Bressers, J. and P. Klok (1991) ‘Hoe valt effectiviteit van beleid te verklaren? Deel 2:Instrumententheorie’, in J. Bressers and A. Hoogerwerf (eds) Beleidsevaluatie ,pp. 136–54. Alphen aan de Rijn: Samsom.

Bruyninckx, H. and T. Cioppa (2000) ‘The Effectiveness of International EnvironmentalRegimes: What about the Environment?’, paper presented at the 41st annual conven-tion of the International Studies Association, Los Angeles, CA, 14–18 March.

Burdge, R. (1994) A Conceptual Approach to Social Impact Assessment . Middleton, WI:Social Ecology Press.

Dunn, W. (1994) Public Policy Analysis: An Introduction. Englewood Cliffs, NJ: Prentice-Hall.

EEA (2001) Reporting on Environmental Measures: Are we being Effective? Copenhagen:EEA.

Gysen, J. and K. Bachus (2003) Sectorale uitvoeringsplannen in het Vlaamse afvalbeleid:Effectiviteitsanalyse van een planningsinstrument . Leuven-Antwerpen: HIVA-Steunpunt

Milieubeleidswetenschappen.Gysen, J., K. Bachus and H. Bruyninckx (2002) Evaluating the Effectiveness of Environ-

mental Policy: An Analysis of Conceptual and Methodological Issues. Leuven-Antwerpen:HIVA-Steunpunt Milieubeleidswetenschappen.

Hannigan, J. (1995) Environmental Sociology: A Social Constructionist Perspective.London and New York: Routledge.

Homer-Dixon,T. (1995) Strategies for Studying Causation in Complex Ecological Political Systems. Washington, DC: American Association for the Advancement of Science andthe University of Toronto. Found on http://www.library.utoronto.ca/pcs/eps/method/methods1htm (accessed 28 Sept. 2005).

Keohane, R., P. Haas and M. Levy (1993) ‘The Effectiveness of International Environ-

mental Institutions’, in P. Haas,R. Keohane and M. Levy (eds) Institutions for the Earth:Sources of Effective Environmental Protection, pp. 3–24. Cambridge, MA: MIT Press.

Evaluation 12(1)

116




24/25


117

Kutting, G. (1998) ‘Assessing the Effectiveness of International Environmental Agree-ments: New Dimensions of Analysis’, paper presented at the 39th International StudiesAssociation Convention, Minneapolis, 17–21 March.

Lafferty, W. and J. Meadowcroft, eds (1996) ‘Democracy and the Environment: Congru-ence and Conflict – Preliminary Reflections’, in W. Lafferty and J. Meadowcroft (eds)Democracy and the Environment: Problems and Prospects, pp. 1–17. Cheltenham:Edward Elgar.

Levy, M. and O. Young, eds (1995) ‘The Effectiveness of International Regimes’, unpub-lished typescript.

Mahoney, J. (2004) ‘Tentative Answers to Questions about Causal Mechanisms’, unpub-lished typescript.

Mickwitz,P. (2000) ‘Evaluating Environmental Policy Instruments’,paper presented at theAmerican Evaluation society conference, Waikiki, USA, 1–4 Nov.

Mickwitz, P. (2003) ‘A Framework for Evaluating Environmental Policy Instruments:Context and Key Concepts’, Evaluation 9(4): 415–36.

Mueller, D. (1979) Public Choice. Cambridge: Cambridge University Press.Patton, M. (1997) Utilization-Focus Evaluation: The New Century Text . Thousand Oaks,

CA: SAGE.Peterson, M. (1997a) ‘Regimes as Governance Systems’, in O. Young (ed.) Global Govern-

ance: Drawing Insights from the Environmental Experience, pp. 27–63. Cambridge, MA:MIT Press.

Peterson, M. (1997b) ‘International Organizations and the Implementation of Environ-mental Regimes’, in O. Young (ed.) Global Governance: Drawing Insights from theEnvironmental Experience, pp. 115–51. Cambridge, MA: MIT Press.

Popper, K. (1968) The Logic of Scientific Discovery. New York: Harper & Row.Posavec, E. and R. Carey (2003) Program Evaluation: Methods and Case Studies. Upper

Saddle River, NJ: Prentice-Hall.Rossi, P., M. Lipsey and H. Freeman (2004) Evaluation: A Systematic Approach. Thousand

Oaks, CA: SAGE.Scriven, M. (1991) Evaluation Thesaurus. London: SAGE.STOHO (1996) Inleiding tot de bestuurskunde. Brussels: STOHO.Van de Graaf, H. and R. Hoppe (2000) Beleid en politiek: Een inleiding tot de beleids-

wetenschap en beleidskunde. Bussum: Coutinho.Vedung, E. (1997) Public Policy and Program Evaluation. New Brunswick, NJ: Trans-

action Publishers.Weale,A. (1992) The New Politics of Pollution: Issues in Environmental Politics. Manches-

ter: Manchester University Press.Yin, R. (2003) Case Study Research: Design and Method. Thousand Oaks, CA: SAGE.Young, O. (1997) ‘Rights,Rules, and Resources in World Affairs’, in O. Young (ed.) Global

Governance: Drawing Insights from the Environmental Experience, pp. 1–23. Cambridge,MA: MIT Press.

JOOS GYSEN, senior researcher at the environmental policy research group,

Catholic University of Leuven and the Flemish Centre of Expertise for

Environmental Policy Sciences. Please address correspondence to: Parkstraat 47,

B-3000 Leuven, Belgium. [email: [email protected]]




25/25

118

Evaluation 12(1)

HA NS BRU YNINCKX, Ph. D. environmental politics, Colorado State

University. Professor of environmental politics, Catholic University of Leuven,

Belgium and Wageningen University, The Netherlands. Co-promotor Flemish

Centre of Expertise for Environmental Policy Sciences. Please address

correspondence to: E. Van Evenstraat 2, B-3000 Leuven, Belgium. [email:

[email protected]]

KRI S BAC HUS , research co-ordinator at the environmental policy research

group, Catholic University of Leuven and the Flemish Centre of Expertise for

Environmental Policy Sciences. Please address correspondence to: Parkstraat 47,

B-3000 Leuven, Belgium. [email: [email protected]]

gysen. 2006. the modus narrandi

Documents