178 ALITRNATIVE APPROACHES TO PROGRAM EVALUATION

of their evaluation designs against Stufflebeam's (1973b) list of steps in designingevaluations. In similar ways, most of the evaluation approaches summarized inprior chapters influence in important ways the practice of evaIuation.

Thinking back to evaluations we have done, our colleagues' work was usedin this way in almost every study. As noted earlier,

Although i have developed some preferences of my own in doing evalua-tions, probably 75 percent of what I do is application of what i havedistilled from others' ideas. Doubtlessly, all who have been repeatedlyexposed to the evaluation literature have absorbed much "through theperes," as it were, and now reapply it without cognizance of its source.Although few of us may conduct our evaluations in strict adherence toany "model" of evaluation, few of us conduct evaluations which are notenormously influenced by the impact of our colleagues' thinking on ourown preferences and actions. (Worthen, 1977, p. 12)

The alternative conceptions about how evaluation should be conducted-the accompanying sets of categories, lists of things to think about, descriptionsof different strategies, and exhortations to heed-influence the practice ofprogram evaIuation in sometimes subtle, sometimes direct, but always significantways. Some evaIuation designs adopt or adapt proposed approaches. Manyevaluators, however, conduct evaluations without strict adherence (or evenpurposeful attention) to any "model; yet draw unconscioıısly in their philosophy,plans, and procedures on what they have internalized through exposure to theliterature. So the value of the alternative approaches lies in their capacity to helpus think, to present and provoke new ideas and techniques, and to serve asmental checklists of things we ought to consider, remember, or worry about.Their heuristic value is very high; their prescriptive value seems much less.

COMPARATIVE ANALYSIS OF CHARACTERlSllCSOF ALTERNATIVE EVALUAll0N APPROACHES

So many new concepts have been presented in Chapters 5 through 10 that thereader might be feeling challenged to assimilate all of it. The matrix in Figure11.1-a comparative analysis of the characteristics, strengths, and limitations ofthe six approaches-should help. The aspects of each approach that we havechosen to highlight are as follows:

1. Proponents-Individuals who have written about the approach2. Purpose of evaluation- The intended useıs) of evaluation proposed

by writers advocating each particular approach or the purposes thatmay be inferred from their writings

3. Distinguisbing characteristics-Key descriptors associated with eachapproach

'\

c:"8 ~c: w:J "'o..:. '"C-oCc

c: _ ""o ~ c o I'J~ Q:: c c ro.!: vi

.~ - C rtJ >. 0° ~ "E .s••.•• ~ (v o tt ~ QJ n:ı viS 'ı: ~ ~ -g .9- ~ ~ i ~Q;OVl"-lJ",~,,- U

-g QJO'~ C c 'r:ıo>:~t'J..c ii) ro ro v'ı .2 -!. ;,n U ı:: ••• cl. OJoo-IlJEJı c;a~QJ:J'-QJx.!::.s bO ~ E ~ B ..vi QJ E E E "o g ~ QJ \i)

"'tJ C X ro QO QJ E ı.. ttı .~ vi u "'o cc 's, ~ CO >. c g QJ -2 ~ OJ '+- ~ .~ c o~ ~ cl. e .~~ .~ '5.E '':; ı.. o ı.. "'o 1 OJ

~o§o.·eg-gCTı.. ~-3.e:~-gEg-g o. u ro ro c, ro ~.E ~ ';: ::J"Z n:ı;,: .!!':ı '"

'o

t-ı:ı >-~.!: vi QJ ~

;~-~c~~'t: ci ~ 'S :ı~O~O~~

"1:>-..c'1J ı~~oo

c '+- .~ i:. - ı: iii u .- ro c c iii° o vıooO1'0 QJ.:.:: >.ı:ıQJ.-CıOro "'c ::: t :E ~ '" ~.D .. 00 'o E ~ .E00 QJ ~ cl > oo...c QJ ::J ~ C C::J '"o "'c

.E g 'E :2 g CJ'.§ ~ ~ ~.§ .~ ~ :~ ~"E ~~ ~ ~ ~ g a ~ ~ ~ o ~ ~'B ~ c: ~ ee .n cv iv u .~!: vi 3: ~..c. ° 0."'0 c ıc o.

"- :ı

<ilc1is:uroecl.cl.roc.gro:ı

tO>c1ic1i

·5roEs

tO

c:.Q

.~] .~ ~_ - '"O:J

t ~ a; ~ eQ.'ı:: ~ ~ (J'::Oiü<

-'o'"~ ~._ c:

COLLLQ)>.

~.!~=.- 0'"0 iv). •.. :J :Je 0.'-' o-n,

c:~'" o'E Eu oVi:.<

<il <il<il c: '"Ü o vi:J .- iv III

c: "'o '5"5 g.Q e cu :;''':;

oo;Gc..'1Ja.a..s E 5 ~ 5.g"o o o t'W o 1'0·>'C.Do.1:Jı..0·_ 1'0 .•••• ct:ı o~

~ 1=.. -E ı:ı '"~~~c.o- Q) vi

vı ~.~:=.S~.- ~ lo. :J....!ıı:: o~ ~OV;<~

<tlcroc1i

·5

:2BooQJ c: .S<il 0-'"~'; E c~ E.S·~._ o '-). -"'O u~.~.~ ~

~o

s:QJ .~ cv..ı::...c~

1:> _ ~ '"c: '" <il~ ~ r:ı _ '"o .=: 3 g:ı '"o

eti ~ "tl 'i cv c: C:',= cucl. ".::,! vi ~ 1 o E ~'E a:; .~E ~ ı::: ı.. :J ~.~ E 1 E ra ı.. '-..cO ';:'OI"'>-~Ea."'OE~ .!luU .ı::ı,'l:: - O cv .ır:: ra o.D O _ Q) O iv

OOç~~ I"-~iö~ O,...,...,.....,cı::ıoLi:

<il

;:••c:O

g ~v) ct,.J

••...O c:(lt .e<il -

O '"ct-2S ~o.. ••

'"

<il 1:> '"c _ c: ~QJ ttı ttı -' viE:ı ,,:ı

'"o cu CLJ vi.gı:~.gıg ~4!!! c.a-g~.~ c ;n~.Q~.- ~ ~ e -g E 2._ c: c x _ ttı l'O'iii~ O....!ıı:: a.ı o ti ~ .;;

<xl

ÔOc:c: o ',=o - vi

.~:: v;~oo~·~·~QJüü·SE

b:g1--5-5E~OOQJlVooOc:

.~ -5 ~ a. a. E 8:ı

; -;c: -o '" -'c; c: ~..... ~~ .0 ôO'= o 1= E~.~.=:~Vliiig.

.~~~ ~ ~ ~~J:1JEQJ~a.~

~o 05 '" ~c:: tO·... c:oo..c~..c:·;~:~tı

.S 5 ~-c e- '"Et:'" .!,! ~.!!~.s·qaLu ..:ıtt!

O~ı..e-'iii o:..::c:: ı.. iii o

.2 ~ E ct .,;;e~;,;.~.S c g> ~ .~E o ı.. ~ Q.I1'0 U a. ......c:ıll

dı~" ... .6:ı '" c:_ c: o c:-:':~';.g ~:>.. ::: 1'0 _-g § "2 .s .." ~V;.1:JbEE·ü~:g~~~:;J: CLJu u

~:;

i_ ı.. c:ı.. E o-E.g

E ~..g §-.Q e .~:ı01:>Qj-~Eca.o>~"oQJo~a.Q)~e.;u "'o III 0."0

6. 6 ...o , 00 vi ).Qj3",E E~.~ ıi ~ "~

1:>.fE~E~cE :..;.- ..... ~'c ::ıttlC~cooc:o

~E,gEK~~a':

ı!. v; " ~ ~:J Q) ~ o u a.'

CQJ; .~ ''':;- a::.~ cu..Q B QJ

ClJ-UVll\lı..'_.1:J UQJ,_ C:~

E ~ .~ c ;;.!2 ~ o 1J ~ ~ g rE il) ~",il.!l '" 1:> '" c: c: E 1:>E"'''' Ec: o o E ı.. co a. QJ 1'0 ı.. .9- E III

.- 00 :J QJ .:: ~ cu o E;". ...u o vi

~~'~~-="3~!Qj't ~~.~~~~ ~ ~ .s ~.2 i5 ~ 'E ~ ~ E ~ 5 ~cı. ~

vi "-

"";::~ .~.~ 't:::ı ••

.~~~ '"a-5'"

<ile••

FlGURE 11.1

5. Contributionsto the corıcep-ıuelizetion ofan evslueıion

6. Criteria (orjudgingeva/ualions

(continued)

Objectives-OrientedPre-post mea-

surement ofperformance;clarification ofgoals; use ofobjective testsand measure-ments that aretechnicallysound

Measurability ofobjectives;measurementreliability andvalidity

Managemenl-OrientedIdentify and eval-

uate needs andobjectives; con-sider alternativeprogram designsand evaluatethem; watchthe implemen-tatian of a pro-gram; look forbugs and explainoutcornes: seeif needs havebeen reducedor eliminated;metaevaluation;guidelines forinstitutionalizingevaluation

Utility; feasibility;propriety; tech-nical soundness

Consumer-OrientedLists of criteria

for evaluatingeducationalproducts andactivities; archi-val referencesfor completedreviews;formative-summatlveroles of eval-uatiorı: biascontrol

Freedom frombias; technicalsoundness;defensiblecriteria used todraw conclu-sions and makerecommenda-tlorıs: evidenceof need andeffectivenessrequired

Expertise-Orien/edLegitimation of

subjectivecriticism; self-study withoutsideverification;standards

Use of recog-nized stand-ards; qualifica-tions of experts

Adversary-Orien/edUse of forensic

and judicialforms of publichearing; cross-examination ofevidence;thoroughpresentation ofmultiple per-spectives; focuson and darifi-catian of issues

Balance; fairness;publicness;opportunity forcross-examination

Participant-Orien/edEmergent evalua-

tion designs;use of induc-tive reasoning;recognitian ofmultiple reali-ties; impor-tance of study-ing cantext;criteria forjudging therigor ofnaturalisticinquiry

Credibility; fit;auditability;confirmability

7. Benetiıs

8. Limiıeıiorıs

Ease of use; sim-plicity; focuson ouleames;high accept-ability; forcesobjectives tobe set

Oversimplicationof evaluationand programs;outcomes-onlyortentation,reductionistic;linear; over-emphasis onouıeames

Comprehensiveness;sensitivity toinformationneeds of thosein aleadershippasitian; syste-matic approachto evaluation;use of evalua-tion throughoutthe process ofprogram develop-ment; welloperationalizedwith detailedguidelines forimplementation;use of a widevariety ofinformation

Emphasis onorganizationalefficiency andproductionmodel; assunıp-tion of orderli-ness and pre-dictabifity indecision making;can be expen-sive to adminis-ter and maintain;narrow focuson the concernsof leaders

Emphasis on co n-sumer informa-tion needs;influence onproduct devel-opers; concernwith cost-effectivenessand utility;availability ofchecklists

Cost and lack ofsponsorship;mav suppresscreativity orinnovalion; notopen to debateor cross-examination

Broad caverage;efficiency (easeof implementa-tion, timing);capitalizes onhumanjudgment

Replicability;vulnerability topersonal bias;scarcity ofsupporting

documentationto supportconclusions;open lo conflictof interest;superficial lookat context:overuse of intu-ition; relianeeon qualificationsof the "experts"

Braad caverage;dose examina-tion of daims;aimed towardclosure or reso-lutian; Hlumina-tion of differenısides of issues;impact onaudience; useof a widevariety ofinformation

Fallible arbiters orjudges; highpotential costsand consump-tion of time;reliance oninvestigatoryand communi-catton skills ofpresenters;potentialirrelevancies orar tiftcial polari-zation; limitedto informationthat is presented

Focus on descrip-tion and judg-ment; concernwith context,openness toevolve eva/ua-tion plan; plural-istic; use ofinductive reason-ing; use of awide variety ofinformation;emphasis onunderstanding

Nondirective;tendeney to beattracted bythe bizarre oratypical; poten-tially high

labor-intensityand cost; hypo-thesis genera-ting; potentialfor failure toreach dosure