178 ALITRNATIVE APPROACHES TO PROGRAM EVALUATION
of their evaluation designs against Stufflebeam's (1973b) list of steps in designingevaluations. In similar ways, most of the evaluation approaches summarized inprior chapters influence in important ways the practice of evaIuation.
Thinking back to evaluations we have done, our colleagues' work was usedin this way in almost every study. As noted earlier,
Although i have developed some preferences of my own in doing evalua-tions, probably 75 percent of what I do is application of what i havedistilled from others' ideas. Doubtlessly, all who have been repeatedlyexposed to the evaluation literature have absorbed much "through theperes," as it were, and now reapply it without cognizance of its source.Although few of us may conduct our evaluations in strict adherence toany "model" of evaluation, few of us conduct evaluations which are notenormously influenced by the impact of our colleagues' thinking on ourown preferences and actions. (Worthen, 1977, p. 12)
The alternative conceptions about how evaluation should be conducted-the accompanying sets of categories, lists of things to think about, descriptionsof different strategies, and exhortations to heed-influence the practice ofprogram evaIuation in sometimes subtle, sometimes direct, but always significantways. Some evaIuation designs adopt or adapt proposed approaches. Manyevaluators, however, conduct evaluations without strict adherence (or evenpurposeful attention) to any "model; yet draw unconscioıısly in their philosophy,plans, and procedures on what they have internalized through exposure to theliterature. So the value of the alternative approaches lies in their capacity to helpus think, to present and provoke new ideas and techniques, and to serve asmental checklists of things we ought to consider, remember, or worry about.Their heuristic value is very high; their prescriptive value seems much less.
COMPARATIVE ANALYSIS OF CHARACTERlSllCSOF ALTERNATIVE EVALUAll0N APPROACHES
So many new concepts have been presented in Chapters 5 through 10 that thereader might be feeling challenged to assimilate all of it. The matrix in Figure11.1-a comparative analysis of the characteristics, strengths, and limitations ofthe six approaches-should help. The aspects of each approach that we havechosen to highlight are as follows:
1. Proponents-Individuals who have written about the approach2. Purpose of evaluation- The intended useıs) of evaluation proposed
by writers advocating each particular approach or the purposes thatmay be inferred from their writings
3. Distinguisbing characteristics-Key descriptors associated with eachapproach
'\
c:"8 ~c: w:J "'o..:. '"C-oCc
c: _ ""o ~ c o I'J~ Q:: c c ro.!: vi
.~ - C rtJ >. 0° ~ "E .s••.•• ~ (v o tt ~ QJ n:ı viS 'ı: ~ ~ -g .9- ~ ~ i ~Q;OVl"-lJ",~,,- U
-g QJO'~ C c 'r:ıo>:~t'J..c ii) ro ro v'ı .2 -!. ;,n U ı:: ••• cl. OJoo-IlJEJı c;a~QJ:J'-QJx.!::.s bO ~ E ~ B ..vi QJ E E E "o g ~ QJ \i)
"'tJ C X ro QO QJ E ı.. ttı .~ vi u "'o cc 's, ~ CO >. c g QJ -2 ~ OJ '+- ~ .~ c o~ ~ cl. e .~~ .~ '5.E '':; ı.. o ı.. "'o 1 OJ
~o§o.·eg-gCTı.. ~-3.e:~-gEg-g o. u ro ro c, ro ~.E ~ ';: ::J"Z n:ı;,: .!!':ı '"
'o
t-ı:ı >-~.!: vi QJ ~
;~-~c~~'t: ci ~ 'S :ı~O~O~~
"1:>-..c'1J ı~~oo
c '+- .~ i:. - ı: iii u .- ro c c iii° o vıooO1'0 QJ.:.:: >.ı:ıQJ.-CıOro "'c ::: t :E ~ '" ~.D .. 00 'o E ~ .E00 QJ ~ cl > oo...c QJ ::J ~ C C::J '"o "'c
.E g 'E :2 g CJ'.§ ~ ~ ~.§ .~ ~ :~ ~"E ~~ ~ ~ ~ g a ~ ~ ~ o ~ ~'B ~ c: ~ ee .n cv iv u .~!: vi 3: ~..c. ° 0."'0 c ıc o.
"- :ı
<ilc1is:uroecl.cl.roc.gro:ı
tO>c1ic1i
·5roEs
tO
c:.Q
.~] .~ ~_ - '"O:J
t ~ a; ~ eQ.'ı:: ~ ~ (J'::Oiü<
-'o'"~ ~._ c:
COLLLQ)>.
~.!~=.- 0'"0 iv). •.. :J :Je 0.'-' o-n,
c:~'" o'E Eu oVi:.<
<il <il<il c: '"Ü o vi:J .- iv III
c: "'o '5"5 g.Q e cu :;''':;
oo;Gc..'1Ja.a..s E 5 ~ 5.g"o o o t'W o 1'0·>'C.Do.1:Jı..0·_ 1'0 .•••• ct:ı o~
~ 1=.. -E ı:ı '"~~~c.o- Q) vi
vı ~.~:=.S~.- ~ lo. :J....!ıı:: o~ ~OV;<~
<tlcroc1i
·5
:2BooQJ c: .S<il 0-'"~'; E c~ E.S·~._ o '-). -"'O u~.~.~ ~
~o
s:QJ .~ cv..ı::...c~
1:> _ ~ '"c: '" <il~ ~ r:ı _ '"o .=: 3 g:ı '"o
eti ~ "tl 'i cv c: C:',= cucl. ".::,! vi ~ 1 o E ~'E a:; .~E ~ ı::: ı.. :J ~.~ E 1 E ra ı.. '-..cO ';:'OI"'>-~Ea."'OE~ .!luU .ı::ı,'l:: - O cv .ır:: ra o.D O _ Q) O iv
OOç~~ I"-~iö~ O,...,...,.....,cı::ıoLi:
<il
;:••c:O
g ~v) ct,.J
••...O c:(lt .e<il -
O '"ct-2S ~o.. ••
'"
<il 1:> '"c _ c: ~QJ ttı ttı -' viE:ı ,,:ı
'"o cu CLJ vi.gı:~.gıg ~4!!! c.a-g~.~ c ;n~.Q~.- ~ ~ e -g E 2._ c: c x _ ttı l'O'iii~ O....!ıı:: a.ı o ti ~ .;;
<xl
ÔOc:c: o ',=o - vi
.~:: v;~oo~·~·~QJüü·SE
b:g1--5-5E~OOQJlVooOc:
.~ -5 ~ a. a. E 8:ı
; -;c: -o '" -'c; c: ~..... ~~ .0 ôO'= o 1= E~.~.=:~Vliiig.
.~~~ ~ ~ ~~J:1JEQJ~a.~
~o 05 '" ~c:: tO·... c:oo..c~..c:·;~:~tı
.S 5 ~-c e- '"Et:'" .!,! ~.!!~.s·qaLu ..:ıtt!
O~ı..e-'iii o:..::c:: ı.. iii o
.2 ~ E ct .,;;e~;,;.~.S c g> ~ .~E o ı.. ~ Q.I1'0 U a. ......c:ıll
dı~" ... .6:ı '" c:_ c: o c:-:':~';.g ~:>.. ::: 1'0 _-g § "2 .s .." ~V;.1:JbEE·ü~:g~~~:;J: CLJu u
~:;
i_ ı.. c:ı.. E o-E.g
E ~..g §-.Q e .~:ı01:>Qj-~Eca.o>~"oQJo~a.Q)~e.;u "'o III 0."0
6. 6 ...o , 00 vi ).Qj3",E E~.~ ıi ~ "~
1:>.fE~E~cE :..;.- ..... ~'c ::ıttlC~cooc:o
~E,gEK~~a':
ı!. v; " ~ ~:J Q) ~ o u a.'
CQJ; .~ ''':;- a::.~ cu..Q B QJ
ClJ-UVll\lı..'_.1:J UQJ,_ C:~
E ~ .~ c ;;.!2 ~ o 1J ~ ~ g rE il) ~",il.!l '" 1:> '" c: c: E 1:>E"'''' Ec: o o E ı.. co a. QJ 1'0 ı.. .9- E III
.- 00 :J QJ .:: ~ cu o E;". ...u o vi
~~'~~-="3~!Qj't ~~.~~~~ ~ ~ .s ~.2 i5 ~ 'E ~ ~ E ~ 5 ~cı. ~
vi "-
"";::~ .~.~ 't:::ı ••
.~~~ '"a-5'"
<ile••
FlGURE 11.1
5. Contributionsto the corıcep-ıuelizetion ofan evslueıion
6. Criteria (orjudgingeva/ualions
(continued)
Objectives-OrientedPre-post mea-
surement ofperformance;clarification ofgoals; use ofobjective testsand measure-ments that aretechnicallysound
Measurability ofobjectives;measurementreliability andvalidity
Managemenl-OrientedIdentify and eval-
uate needs andobjectives; con-sider alternativeprogram designsand evaluatethem; watchthe implemen-tatian of a pro-gram; look forbugs and explainoutcornes: seeif needs havebeen reducedor eliminated;metaevaluation;guidelines forinstitutionalizingevaluation
Utility; feasibility;propriety; tech-nical soundness
Consumer-OrientedLists of criteria
for evaluatingeducationalproducts andactivities; archi-val referencesfor completedreviews;formative-summatlveroles of eval-uatiorı: biascontrol
Freedom frombias; technicalsoundness;defensiblecriteria used todraw conclu-sions and makerecommenda-tlorıs: evidenceof need andeffectivenessrequired
Expertise-Orien/edLegitimation of
subjectivecriticism; self-study withoutsideverification;standards
Use of recog-nized stand-ards; qualifica-tions of experts
Adversary-Orien/edUse of forensic
and judicialforms of publichearing; cross-examination ofevidence;thoroughpresentation ofmultiple per-spectives; focuson and darifi-catian of issues
Balance; fairness;publicness;opportunity forcross-examination
Participant-Orien/edEmergent evalua-
tion designs;use of induc-tive reasoning;recognitian ofmultiple reali-ties; impor-tance of study-ing cantext;criteria forjudging therigor ofnaturalisticinquiry
Credibility; fit;auditability;confirmability
7. Benetiıs
8. Limiıeıiorıs
Ease of use; sim-plicity; focuson ouleames;high accept-ability; forcesobjectives tobe set
Oversimplicationof evaluationand programs;outcomes-onlyortentation,reductionistic;linear; over-emphasis onouıeames
Comprehensiveness;sensitivity toinformationneeds of thosein aleadershippasitian; syste-matic approachto evaluation;use of evalua-tion throughoutthe process ofprogram develop-ment; welloperationalizedwith detailedguidelines forimplementation;use of a widevariety ofinformation
Emphasis onorganizationalefficiency andproductionmodel; assunıp-tion of orderli-ness and pre-dictabifity indecision making;can be expen-sive to adminis-ter and maintain;narrow focuson the concernsof leaders
Emphasis on co n-sumer informa-tion needs;influence onproduct devel-opers; concernwith cost-effectivenessand utility;availability ofchecklists
Cost and lack ofsponsorship;mav suppresscreativity orinnovalion; notopen to debateor cross-examination
Broad caverage;efficiency (easeof implementa-tion, timing);capitalizes onhumanjudgment
Replicability;vulnerability topersonal bias;scarcity ofsupporting
documentationto supportconclusions;open lo conflictof interest;superficial lookat context:overuse of intu-ition; relianeeon qualificationsof the "experts"
Braad caverage;dose examina-tion of daims;aimed towardclosure or reso-lutian; Hlumina-tion of differenısides of issues;impact onaudience; useof a widevariety ofinformation
Fallible arbiters orjudges; highpotential costsand consump-tion of time;reliance oninvestigatoryand communi-catton skills ofpresenters;potentialirrelevancies orar tiftcial polari-zation; limitedto informationthat is presented
Focus on descrip-tion and judg-ment; concernwith context,openness toevolve eva/ua-tion plan; plural-istic; use ofinductive reason-ing; use of awide variety ofinformation;emphasis onunderstanding
Nondirective;tendeney to beattracted bythe bizarre oratypical; poten-tially high
labor-intensityand cost; hypo-thesis genera-ting; potentialfor failure toreach dosure