Drawingaboveusedwithpermissionoftheartist,RebeccaGraceJones.

OnEvaluatingtheSecondaryImpactof

LeadershipPrograms:AWhitePaper

JustusBairdandAaronDorfman

Processesandmethodstoevaluatetheimpact

ofleadershipprogramsrunbynonprofitagencies,

particularlyofprogramsdesignedtoleverageleaders

forimpactontheircommunitiesandsociety.

ProducedwiththesupportofaWexnerGraduateFellowshipAlumniCollaborationGrant

1

Summary

Thisreportoutlinesanapproachandsomespecificmethodsforevaluatingaparticular,andoften

neglected,aspectofnonprofit leadershiptraininganddevelopmentprograms.Whiletheprimary

(or direct) impacts of these programs on the participating leaders are often evaluated and

understood, it’s less clear how leadership programs ought to think about and evaluate their

secondary impact on the communities inwhich their programalumniwork. This report offers a

conceptualframeworkandtoolsforstaffofleadershiptraininganddevelopmentprogramstouse

inevaluatingthatsecondaryimpact.

INTRODUCTION

There’sanapocryphalstoryinwhichaHarvardBusinessSchoolstudentvisitsthecoursematerials

officeatHBSandisdisgruntledbythelackadaisicalservice.Fedupwithwaiting,hefinallyrapshis

knucklesonthecounterandbarks,“Hey!Canyouhurryitup?I’mthecustomer!”Inresponse,the

clerkcasuallylowersthecrosswordpuzzleinwhichshe’sbeenengrossedandremarks,“You’renot

thecustomer.You’retheproduct.”Thisjokeencapsulatesaperceptionthatisoftenheldamong

those who manage programs focused on leadership training and development: that the

participantsarethe“product”ofthelearningprocess.

Many nonprofits run programs to invest in future

skilled leaders and professionals. While we are

deeply committed to the intrinsic value that these

programsoffertothosewhoparticipateinthem,we

should be equally committed to understanding the

impactthattheleadersinwhomwe’veinvestedhave

on their communities and networks, what are

referredto in this reportas“secondary impacts.” In

fact, the real measure of our success is not the

leader’s experiencealone, but also the change they

catalyze in their communities, their networks, and

thelargerworld.Inthiscontext,ourparticipantsare

alsoameanstosupportourlargervisionsforchange.

Our implied theory of change is that investing in

leaders translates to better-equipped leaders who

canmoreeffectivelypursue thechangeweallneed

to see in theworld. Themeasure of our success is

howmuchmoreeffectivelytheyrealizethatchange.

This report was born out of a shared need of its

authors, staff leaders at two different nonprofits,

American Jewish World Service and Auburn

Seminary.Wewerebothoverseeingmajorprograms

Case study from the Talmud: Training

leaderstoachievesecondaryimpacts

(Thisstoryabout leadership inamomentof

crisis comes from the Babylonian Talmud,

BavaMetzia85b.)

OnedayRabbiHaninaandRabbiHiyyawere

arguing, and Rabbi Hanina said to Rabbi

Hiyya: ‘How can you argue with me? If,

Heaven forbid, the Torah were forgotten in

Israel, Iwould [be able to] restore it bymy

argumentative powers.' To which Rabbi

Hiyyarejoined:'Howcanyouarguewithme,

whoactuallypreventedtheTorahfrombeing

forgotten in Israel? How did I do it? I went

and sowed flax, made nets [from the flax],

trappeddeerandgavethemeattoorphans,

andpreparedscrolls[fromtheirskins],upon

whichIwrotetheFiveBooks[ofTorah].Then

I went to a town [where there were no

teachers] and taught the Five Books to five

children,andthesixorders[oftheMishnah]

to six children. And I told them: "Until I

return, teacheachother theFiveBooksand

the Mishnah;" and thus I preserved the

Torahfrombeingforgottenin Israel.'This is

whatRabbi[YehudahaNasimeantwhenhe]

said,'HowgreataretheworksofHiyya!'

Background

2

thataspiretostrengthentheleadershipofacohortofleaders.Forbothofus,theinvestmentin

leaderswasclearlyameanstoanend:therealdesiredoutcomewasmakinganimpactonsociety,

andtheleadersweweretrainingservedasvehiclesforachievingthatimpact.Wewereoperating

under a hypothesis about social change that presumes that great leaders make great things

happen, so investing inpeople tohelp thembecomegreat leaders—inspiring,empowering,and

trainingthemtocatalyzecommunaltransformation—wouldleadtogreatchange.

Ourbare-bonestheoryofchangelookedsomethinglikethis:

Whilewehadmanytoolsforevaluatingourprogram’simpactontheleadersthemselves(A->Bor

“primaryimpact”),whenitcametounderstandingormeasuringhoweachleader’sexperiencein

ourprogramcontributedtoanimpactonthecommunitiestheyled(A->Cor“secondaryimpact”),

wewereataloss.

Stirredbythisfrustration,weproducedatwo-dayseminarfornonprofitstafftostudyapproaches

to evaluating the secondary impacts of leadership programs. We are grateful to the Wexner

Foundation for underwriting the costs associated with the seminar (through an alumni

collaborationgrant),heldinApril2015attheNewYorkofficesofAmericanJewishWorldService.

Twenty-two colleagues who lead similar leadership programs (mostly in the Jewish nonprofit

sector)participatedintheseminar.Wehiredtwoexpertsinevaluationtoleadusinourlearning

andreflection:Dr.JewlyaLynnfromtheSparkPolicyInstituteinDenverandTanyaBeerfromthe

CenterforEvaluationInnovationinWashington,D.C.Thereportsharesthetoolsandinsightswe

gained through the seminar, in the hopes that other nonprofit staff and funders designing and

supportingleadershipeffortscanbetterassesstheimpacttheirleadershaveonthecommunities

theyserve, inorderto learnandimprovethoseprogramstoeffectyetmorepositivechangefor

ourworld.

AOrganizabonrecruitspeoplewithgreatleadership

potenbal,andinspires,

empowers,andtrainsthemtoadvancetheorganizabon's

mission.

BLeadersemergefromleadershipprogramandbegintouse

theirnewskillstocatalyze

changeamongpeopleintheirnetworksandcommunibes.

CChangesinthe

adtudes,knowledge,andbehaviorwithin

leaders'communibessupporttheimpactthe

organizabonisseekingtomake.

Background

3

EVALUATINGIMPACTINACOMPLEXSYSTEM

Theleadershipprogramsatissueaddresswhatareoftencalledcomplexproblemsorsystems.As

opposed to simple problems (baking a cake) and complicated problems (sending a rocket into

space), complex problems or systems—such as raising a child, changing immigration policy, or

solving climate change—are characterized by a multiplicity of interconnected factors, some of

whichareunknownorinvisible;aninabilitytopredictconsequences;andunstablecontexts.

When evaluating impact in the context of a complex system, experts encourage evaluators to

develop a plausible case for how a program contributed to the outcomes, as opposed to

overstatingthecase.TheSparkPolicyInstituteencouragesevaluatorsto:

• Aimforcontributionratherthanattribution(i.e.don’tassumethatanyparticularoutcome

isentirelyattributabletoasingleintervention);

• Lookforemergentoutcomes,especiallythoseyoudidn’tanticipateorexpect;

• Keepoutcomesrealistic;

• Differentiatebetweenoutcomesthatyou“expecttosee,”“wouldliketosee,”and“would

lovetosee”;and

• Use a “golden spike”1to link to other evidence: connect your impact to already proven

causalrelationships,focusingyourevaluationonthegapsornewlearning.

Perhapsthemostimportantapproachtoevaluatingcomplexsystemsistofocusmoreon“learning

andadaptation”thanontraditional“accountabilityforimpact.”

Evaluations that focus on learning and adaptation encourage developmental learning and risk

taking. The complex problems our programs are addressing are never entirely solved; what

worked last timemaynotwork in the future; andwhatworks in one contextmaynotwork in

4

another.Developmentalevaluationmethodstrackeffectsastheyunfold,notexpectingmethods

orevenintendedoutcomestoremainstable.Inthiscontext,evaluationencouragesstafftoadapt

towhatislearned,acknowledgewhatisn’tworking,andusethenewknowledgetoinformfuture

decisions.IntheirbookGettingtoMaybe:HowtheWorldisChanged,Westley,Zimmerman,and

Pattoncall thisa“safe-fail”approach,asopposedtoa“fail-safe”approach2.Thepoint is that in

thecontextof complexsystems, spending lotsof timeand resourcesonmeasuring theoriginal,

predictedimpactmaynotbeaneffectivewaytogettotheultimateoutcome.

THREE-STEPCYCLEFORBETTEREVALUATION

Movingfromtheideaofmoreeffectivelyevaluatingleadershipprogramsinacomplexsystemto

actuallydoingitcanbedaunting.Wehavefoundinourownworkthatittakessignificanttimeand

energytogetinstitutionalbuy-inandinvolvecolleaguesinnewbehaviors.WhetheryoutryaDIY

(“doityourself”)approachorinvolveaconsultant,thesethreestepsformthecoreelementsofan

evaluationandlearningcycle.

Thefirststepistoreflectonwhatyouaretryingtoachieve(outcomes)andhowyouaretrying

toachieveit(changepathways).Itisimpossibletoeffectivelymeasureorlearnaboutimpactuntil

youhavefirstarticulatedasclearlyaspossiblewhatthedesiredoutcomesareandhowyouthink

theywillcomeabout.Thesecondstepisto identifyandutilizemethods to collectandanalyze

datathatcanmeasurethedesiredoutcomes.Evaluationexpertshavedevelopedawiderangeof

creativemethodsand tools for this step,wellbeyond the surveys, interviews, case studies, and

focusgroupsthatwearemostfamiliarwith.Andevenforsomeofthesetried-and-truemethods,

expertsproposerefinementsthatmakethemmoreeffective inthecontextofcomplexsystems.

Thethirdstepistofeedthelearningbackintotheongoingprogramdesignandimplementation.

Step1:ArticulateMeaningfulOutcomesandChangePathways

Mostprogramstreatevaluationasanexerciseincountingheads(e.g.,Howmanypeopleapplied

to/participatedin/graduatedfrommyprogram?)and/orcustomersatisfaction(e.g.,Howwould

you rateyourexperience inmyprogramona scaleof1-5?).Wedesign leadershipprograms to

leverage leaders to effect meaningful change in the world. The first step in evaluating these

programsistoarticulatethechangeweseektoproduceandthewaywethinkitwillmaterialize:

What dowewant to see happen in the communities our leaders serve?Willmore people join

thosecommunities?Willcommunitymembersengageinmore—orbetter—politicalactivism?Will

they engagemore productively in civil discourse around contentious issues?Will theymanage

theirinstitutionsmoresustainably?

Onewaytobeginarticulatingtheoutcomesandchangepathwaysofa leadershipprogram is to

developapersona,aniconic/archetypalstoryofaprogramparticipantwhoexemplifieswhatthe

program is trying to accomplish. Using personas can help concretize program design in the

observedinterests,experiences,andperspectivesofrealpeople.Developingapersonaconsistsof

thefollowingkeysteps:

5

1. Identifytargetusers

2. Developaresearchprocess(interviewandobservationprotocol)

3. Learnfromtargetusers(throughinterviews,focusgroups,observation,etc.)

4. Analyzeandsynthesizelearningintooneormorepersonas

5. Produceawrittenprofileforeachpersona

ShlomoGoltzhaswrittenahelpfulguide fordeveloping robustprogrampersonas.Below,you’ll

also find a case study from one of the program directors who participated in our evaluation

seminar. While this is a description of an actual program participant and not a synthesized

persona, it conveys the value of having an individual story to capture the kind of change your

programisdesignedtoproduce.

Anothervaluabletoolforarticulatingoutcomesandchangepathwaysisthelogicmodel.(SeeW.K.

Kellogg Foundation’s logic model development guide.) A logic model is a flow-chart diagram

depictingthecausalrelationshipsamongthevariouselementsofprogramdesign.Whilethereare

manylogic-modelformats,mostincludeanddescriberelationshipsamongthefollowingelements.

• Plannedwork:

§ Inputs: resources invested in the program (e.g., funding, staff time, materials,

politicalcapital,etc.)

§ Activities: activities undertaken using inputs (e.g., training programs, political

advocacy,directservice,etc.)

• Intendedresults:

§ Outputs:directandusuallyquantifiableproductsresultingfromactivities(e.g.,#of

people served, # of lobbying visits or petition signatures, customer satisfaction

targets,etc.)

§ Outcomes: specific changes in program participants’ attitudes, behaviors,

knowledge, skills, etc., that result from the program (e.g., reduction in child

malnutrition,passageofspecificlegislation,increasein#ofskilledvolunteers,etc.)

§ Impact: systemic transformation supported by the program’s outcomes (e.g.,

healthier familiesandchildren, reduction incrimerates/violence, increase incivic

engagement,etc.)

6

Some logicmodels also include articulations of the problem the program is designed to solve,

underlying assumptions that inform the logic model design, and critical factors in the external

operatingenvironment.

Logicmodelscanbeusedformanypurposesinbothprogramdesignandevaluation,buttheircore

utilityisinmakingexplicitandtransparentthecausalrelationshipsamongthevariouselementsof

aprogram.

3

7

AnExampleinPractice:BronfmanYouthFellowships(BYFI)

Julie(notherrealname)grewupinapartoftheU.S.withasmallJewishcommunity.Shebelongedto

a Reform Jewish synagogue, attended Hebrew School, and participated in Reform Jewish summer

programsgrowingup.Her limitedexposure to Jewishdiversity resulted invery few friendshipswith

JewsoutsideoftheReformmovement.

JulietraveledtoIsraelwithBYFIthesummerbeforehersenioryearofhighschoolandcontinuedwith

BYFIseminarsduringher senioryear.Theprogramexposedher to Jewishpluralism,text study,and

Israel in a multifaceted way. She built close relationships with Jews different from her (e.g., she

emergedfromtheprogrambestfriendswithanobservantOrthodoxyoungwoman).

Juliewentontoattendan IvyLeagueschool.Becauseofherexperience inBYFI, shedecided toget

involved and joined a prayer community, but came to see that the Hillel—dominated by Orthodox

students on her campus—seemed to alienate Jews frommore secular backgrounds and thosewith

left-leaningperspectivesonIsrael.

As a junior, Julie decided to run forHillel president against a politically conservative and religiously

Orthodoxfellowstudent.Shewastheunderdog,butspenta lotoftimereachingouttostudentson

thefringesandthoseinthecenter.HerBYFIexperiencegavehervaluabletoolsforhowtospeakwith

Jewsfromdiversebackgroundsandhelpedherhonoreachperson’spointofviewasequallyvalid.She

gainedpeople’strustandwontheelection.

AmonthpriortotheelectionshewasfeelingdisheartenedaboutpolarizedattitudestowardIsraelon

campus.SheattendedaBYFIseminarforcollegealumnidesignedtocreateasafespacetoexplorethe

Israel discourse on campus, provide discussion tools, and build community for students who care

about Israelacrossdifferentpoliticalperspectives.Shebrokedown in tearsat theseminar,buthad

severalmentoringconversationswithstaffandpeersthatbolsteredher.Aftertheseminarshewrote

to staff saying the retreathad rejuvenated her and given her increased confidenceand strength to

keep her campaign going. The sense of being part of a community that believed in her and the

reminderthatthereareJewishspacesthatcanenableheartfelt-yet-civildiscussionaboutIsraelgave

herhopeandenergy.

AsHillel president she focused onmaking those on themargins feelwelcome.She hosted the first

everpre-Passoverbagelbrunch,withover250studentsattending(mostofwhomwerenot“theusual

suspects”).ShehasalsoconvincedtheHillelprofessionalstafftodevelopprogramsthatprovidesmall-

groupsettingsforIsraelconversationtobringpeopletogether.

Onelastpoint:Juliewenttocollegeintendingtomajorinpoliticalscience.Becauseofherexposureto

arangeofJewishtextsandideasthroughBYFI,shedecidedtotryYiddishandisnowaYiddishmajor.

~BeckyVoorwinde,ExecutiveDirectorofTheBronfmanYouthFellowships

8

Step2:OpentheEvaluationToolbox:UtilizeaRangeofMethodstoCollectand

AnalyzeData

Ifthefirststepisarticulatingmeaningfuloutcomesandthemeansbywhichweintendtoachieve

them,thenextstepistoopentheevaluationtoolboxtofindmethodsandtoolsthatcaptureand

analyzedatarelevanttothoseoutcomes.

Thecentraltaskofthisstepistomatchoneormoreevaluationtoolstoyourdesiredoutcome(s).

Todothat,youneedtoexploretheevaluationtoolbox,andthenreflectonhowyouwouldusethe

datathataspecifictoolwouldgenerate.Sofirstwe’llintroduceyoutothetoolbox,andthenwe’ll

offerasetofquestionstoreflectonasyouconsiderpotentialevaluationmethods.

Oneofthemostexcitingmomentsoftheseminarwaswhenourtwoevaluationexpertswalkedus

throughasetof15evaluationtoolstheyandotherexpertshavedevelopedforprogramproviders.

Whileitisdifficulttorecreatethatexperienceinawrittenreport,webelievethattastingthe

varietyofevaluationmethodsthatareavailableisanimportantexperienceintheprocessof

improvingourapproachtoevaluating/assessingourprograms.

Belowarebriefdescriptionsofahandfuloftheseevaluationtools.Attheendofthisreport,you’ll

findabriefsummaryofall15differentapproaches,achartyoucanusetoidentifyadditional

approachesthatyoumaywishtoexplore.Dependingontheevaluationexperienceofyourstaff,

youmaydecidetohireanevaluationconsultanttoidentify—andsupportyouinusing—themost

relevanttoolstocollectdatarelatedtoyourdesiredoutcomes.

IdentityLeadershipInventory

UsetheIdentityLeadershipInventorytomeasurealeader’sabilitytomobilizeanddirectfollowers’

energies.

The Identity Leadership Inventory (ILI) is a simple, validated3survey instrument designed to

measure a leader’s influence on her or his community vis-a-vis four interconnected

relationships:

• Theleader’sabilityto“beoneofus”

• Theleader’sabilityto“doitforus”

• Theleader’sabilityto“craftasenseofus”

9

• Theleader’sabilityto“embedasenseofus”

By deploying the ILI in a leader’s community before, during, and after a leadership-training

intervention, you can infer some of the intervention’s value in supporting the leader’s

influence in her/his community. You can find the ILI survey instrument and guidance for

administeringitinNiklasK.Steffen,etal’s

“Identity Leadership Inventory (ILI) Instrument and Scoring Guide (ILI Version 1.0),”

downloadablehere:https://www.researchgate.net/publication/271071707_ILI_Guide

OutcomeHarvesting

UseOutcomeHarvestingtoidentifychangesinacomplexsystemandworkbackwardsto

determinewhetherandhowaninterventioncontributedtothosechanges.

Theoutcomeharvestingmethodologyisunusualinthatitdoesnotseektomeasureprogress

towardapredeterminedsetofoutcomes.Instead,itstartsbyidentifyingchanges(e.g.,shifts

in communal behavior, adoption of new regulations/legislation, emergence of new

memes/messages,etc.)withinasystemaffectedbyanintervention(e.g.,acommunitywhose

leaderhasexperiencedaleadership-trainingprogram)andthenworksbackwardtodetermine

whetherthere’scredibleevidencethattheinterventioncontributedtothechanges.Itseeksto

answerthefollowingquestions:

• Whathappened?

• Whodidit(orcontributedtoit)?

• Howdoweknowthis?Istherecorroboratingevidence?

• Whyisthisimportant?Whatdowedowithwhatwefoundout?

It’sparticularlyapplicableincontextsinwhichtheconnectionsbetweencauseandeffectare

ambiguous. You can find detailed guidance on the outcomes harvesting methodology in

Ricardo Wilson-Grau and Heather Britt’s “Outcome Harvesting,” downloadable here:

http://www.outcomemapping.ca/download/wilsongrau_en_Outome%20Harvesting%20Brief_

revised%20Nov%202013.pdf

10

MostSignificantChange

UseMostSignificantChangetoassessnotonlywhatoutcomesaprogramhasachieved,butalso

howvariousstakeholdersvaluethoseoutcomesindifferentways.

TheMostSignificantChange(MSC)techniqueisahighlyparticipatorymethodologydesigned

toengageabroadcross-sectionof stakeholders in theevaluationofagiven intervention. In

thecontextofa leadership-trainingprogram,membersof the leader’s communitywouldbe

invitedtosharestoriesofwhattheyconsidertobesignificantchangesresultingfromleader’s

participation.Thisprocessofsharingnotonlyprovidesusefulevidencefortheimpactsofthe

intervention, but also surfaces valuable insights into what community members consider

“significant.” Once these stories have been collected, a group of select stakeholders (staff,

advisors,etc.)discussthestoriesanddeterminewhichtheyconsider“mostsignificant,”along

withdescribingarationale fortheirdeterminations.Theseanalysesarethenfedbacktothe

community to focus its attention on impact, shared with leaders, and used to refine the

trainingprogram.Youcanfinddetailedbackgroundinformationandimplementationguidance

inRickDaviesand JessDart’sThe ‘MostSignificantChange’ (MSC)Technique:AGuide to Its

Use,downloadablehere:http://www.mande.co.uk/docs/MSCGuide.pdf

CollectiveEfficacyScale

UseaCollectiveEfficacyScaletoassessagroup’ssharedbeliefinitsjointcapabilitytoorganize

andexecuteacourseofactioninpursuitofitsdesiredresults.

TheCollective Efficacy Scale emerged from criminal-justice research intohow communities’

sharedvaluesandwillingnesstocooperatecouldexert influenceoncommunitymembersto

11

reduceviolenceandcrime.Ithasbeenadaptedforuseinschoolsandmanyothercontextsin

whichcommunitymembers’senseofthemselvesasamutuallysupportivecommunitycapable

of collective action is important. By deploying the Collective Efficacy Scale in a leader’s

communitybefore,during,andafteraleadership-trainingprogram,youcaninfersomeofthe

program’svalueinbuildingthecommunity’ssenseofefficacy.Youcanfindmoreinformation

on the Collective Efficacy Scale in John M. Carroll, Mary Beth Rosson, and Jingying Zhou’s

“Collective Efficacy as a Measure of Community,” downloadable here:

https://www.researchgate.net/publication/221519069_Collective_efficacy_as_a_measure_of_

community.

RippleEffectsMapping

UseRippleEffectsMappingtoreflectuponandvisuallymaptheintendedandunintendedchanges

producedbyacomplexprogramorcollaboration.

Ripple Effects Mapping (REM) is a participatory qualitative evaluation method in which

stakeholders collectively and visually map the interconnected outcomes resulting from a

programorintervention.InanREMprocess,participantscometogethertoreflectontheways

in which a program has affected their lives and the lives of others. Individual impacts are

captured using mind-mapping software or on a large white board, and the group works

together to identify and map connections among these impacts to generate a collective

representationoftherippleeffectsoftheprogramorintervention.Formoreinformationand

a practical guide for usingREM, seeWashington StateUniversity-Extension’s “Ripple Effects

Mapping for Evaluation,” downloadable at: http://extension.wsu.edu/stevens/wp-

content/uploads/sites/19/2013/12/REM.Complete.pdf.

More details and more tools are included in the section titled “15 Evaluation Tools to Consider”

near the end of the report.

12

TESTTHETOOLSONYOUROUTCOMES

Onceyouhaveidentifiedoneortwoevaluationtoolsof interest,youcantestthemoutonyour

outcomes.Dothisbyreflectingonthefollowingquestions:

1. Whatoutcome(s)doyouwanttomeasureusingthisevaluationmethod?

2. How are you planning to use the approach?Whowill be the informants? Howwill you

collectthedata?

3. Imaginea setofdatayoumight receive fromthismethod.Whatwouldyoudowith the

data?Howwoulditaffectthedecisionsyoumakeabouttheprogram?

Notethateachofthesemethodsalsointroducesaframeyoucanapplytoyourevaluationevenif

youaren’tusingthefullmethod.Forexample,youcanlearnaboutthetypesofquestionsyou

mightasktocaptureemergentoutcomesbybecomingfamiliarwithoutcomeharvesting,evenif

youdon’tplantodothefullverificationprocessthatispartofthattechnique.

Evaluation experts offer various guidelines for selecting appropriate evaluation tools.No tool is

perfect, comprehensive, or appropriate for all contexts, so deciding upfront what is most

important about the tool can help. Some of the key criteria to consider in choosing evaluation

toolsinclude:

• Accuracy: The accuracy of your evaluation data is highly dependent on the constraints

underwhich you are operating (e.g., having access to relevant respondents at the right

time).

• Timeliness:Datathatwon’tbeavailableintimetomeaningfullyinformdecisionswon’tbe

thathelpful.

• Actionability:Weoftencollectdatathatareinterestinginsteadofdatathatareactionable.

Consideradressrehearsaltotestouttheusefulnessofyourexpecteddata.Imagineresults

fromamethod, and then imaginewhat youwoulddodifferentlybasedonhaving those

data.Ifyoucan’tfigureoutwhatyouwouldchange,thosedatamaynotbeuseful.

• Impartiality:Avoidaskingleadingquestionsorquestionsthatassumepositiveorplanned

outcomes.

• Representative of multiple perspectives: Identify key perspectives you need to further

develop theprogram (e.g.,endusers, funders, community stakeholders,etc.),and finda

toolthatwilldeliverthoseperspectives.

Notethatthesearecriteriatobalance—yourtoolsneedtomeetallofthemtoacertaindegree,

butyouwillalsohavetobalancehowfartogo(e.g.themosttimelydatacollectionmaynotbethe

most accurate or represent themost perspectives, but you don’t want to go so far into being

accurateandrepresentativethatthedatadoesn’tshowupuntilit’stoolatetouseit).

STEP3:APPLYLEARNINGSFORCONTINUOUSPROGRAMIMPROVEMENT

Thisisperhapsthesimplestandyetmostoverlookedstepintheprogramevaluationprocess.The

learninggeneratedthroughevaluationprocessesshouldbefedbackintotheprogramdesignand

processtoensureacontinuouscycleofprogramimprovement.There’sno“magicbullet”

methodologyforthis—it’smostlyafunctionofsettingasidetimeforaplanningteamtoreflecton

13

theevaluationandidentifywaysinwhichtheprogramcanberefinedtoimproveeffectiveness.

Thisshouldbeanongoingcycleinwhichtheprogramdesign(outcomesandchangepathways),

andlearningandevaluationmethodsareregularlyinterrogatedandrefinedtoensurethatthe

overallinterventioniscontinuallyimproving.

Wehopethisreporthasstimulatedyourthinking,andyourpractices,aroundevaluatingyourown

leadershipprogram.

As your evaluation practices evolve, you may wish to deepen your learning about evaluation

methods. In the addenda you’ll find a short list of resources to consider. Youmay alsowish to

partnerwithotherorganizationsandpoolresourcesorexplorejointevaluationefforts.Couldyou

design or use an instrument together?Workwith a consultant together? Share data you have

gatheredandthelearningithasoffered?

Ultimately,evaluationisacorepartoflearningandimprovement.Goodevaluationgivesusdata

toreflecton,helpsusimaginebetterwaystomakeanimpact,andguidesourongoingeffortsat

improvement.

Developoutcomesandchangepathways

Implementprogram

Evaluateandlearn

14

NOTETOFUNDERS

Donorsandfundershaveasignificantinfluenceontheevaluationpracticesofnonprofits.

Inmostcases,thisinfluenceisverypositive.Thequestionsembeddedinagrantapplicationand

theinquiriesofaprogramofficeraboutevaluationpracticesforceprogramdesignerstoreflecton

theoutcomestheyhopetoachieveandarticulatehowtheyhopetomeasureprogresstoward

thoseoutcomes.Butsometimesafunder’srequestsforevaluationleadtoacultureinwhich

programdirectorsfeelpressuretoshowhowapre-determinedtargetwasachieved.Whena

programissituatedinacomplexsystem,suchalinearapproachtoevaluationcanshutdown

importantlearningandreflectionthatwillleadtobreakthroughslateron.Weencouragefunders

andnonprofitstafftoengageinopenconversationsaboutapproachestoevaluationthatinvite

curiosityandexcitementandratherthancompliance-drivenfear.Bycommittingtooutcomes,

independentofthesuccessofanyparticularprogrammaticapproach,andencouraginghonesty,

collaboration,andinnovation,funderscanshapealargerfieldofpractice.Somefunderswework

withtakealearning-focusedapproachanduselanguagelike:“Wecaremoreaboutmakingan

impactthantheparticularsuccessorfailureofyourprogram.Wewanttolearnwhatyouare

learningandfindouthowtomakeadifference.Ifthisprogramdoesn’tworkwell,weknowwe

willalllearnalotfromtheexperience,andthenwecantrysomethingelsetogether.”

15

ACKNOWLEDGEMENTS

ThisreportwasproducedwiththesupportofaWexnerGraduateFellowshipAlumniCollaboration

Grant.

TheauthorsexpressgratitudetotheWexnerFoundationfortheirinvestmentinleadership

developmentandforthefinancialsupportthatmadethisinitiativepossible.

Authorsofthisreport:

• RabbiJustusBairdisDeanofAuburnSeminary.

• AaronDorfmanisPresidentofLippmanKanferFoundationforLivingTorah.HewasVice

PresidentforNationalProgramsatAmericanJewishWorldServiceduringphase1ofthis

project.

Weexpressappreciationtothealmosttwodozenstaffwhooverseeleadershipdevelopment

programswhocametotheApril2015seminarthatledtothedevelopmentofthisreport:

1. SarraAlpert,AVODAH:TheJewishServiceCorps

2. JustusBaird,AuburnSeminary

3. LaurenBerkun,ShalomHartmanInstitute

4. ShifraBroznick,AdvancingWomenProfessionals

5. AaronDorfman,AmericanJewishWorldService

6. MireleGoldsmith,Hazon

7. IritHouvras,AmericanJewishWorldService

8. LeoraIsaacs,SYNERGY

9. StevenJacobson,Dorot

10. JasonKimelman-Block,BendtheArc

11. KateLauzar,SYNERGY

12. SharonMiller,AuburnSeminary

13. LevNelson,T’ruah:TheRabbinicCallforHumanRights

14. HeidiRosbe,Encounter

15. NadiaRoumani,AmericanMuslimCivicLeadershipInitiative

16. ShaunaRuda,JDCEntwine

17. RachelJacobyRosenfield,AmericanJewishWorldService

18. JudithSamuels,SYNERGY

19. ChrisScharen,AuburnSeminary

20. RebeccaSirbu,CLAL:TheNationalJewishCenterforLearningandLeadership

21. EmilySoloff,AmericanJewishCommittee

22. BeckyVoorwinde,TheBronfmanYouthFellowships

ThismaterialisadaptedfromdatacollectiontoolspresentedaspartofSparkPolicyInstitute’sadvocacyevaluationtoolkit.

15EvaluationToolsToConsiderUsethis2-pagechartofEvaluationToolstoexploremoremethodologiesforevaluatingthe

secondaryimpactofyourleadershipdevelopmentprogram.

Tool Setting #ofInformants TypeofInformationtoCollect

Survey

Onpaper,by

email,orby

phone.

Typicallyalargernumber

ofparticipants(20+).

Close-endedquestionsandopen-

endedquestionstailoredtocapture

datatolearnaboutoneormore

outcomes.

InterviewByphoneor

inperson.

Typicallyasmallnumber

ofparticipants(around5

–15).

Open-endedquestionsthatelicit

moreexplanationandinformation

thansurveysusuallyprovide,

tailoredtocapturedatatolearn

aboutoneormoreoutcomes.

FocusGroup Inperson.Typicallyasmallnumber

ofparticipants(5–10).

Smallnumberofopen-ended

questionsdesignedtoelicit

dialogueamonginformants.

Providesveryin-depthinformation

tolearnaboutoneormore

outcomes.

Document

ReviewInyouroffice!

Noparticipants,butdatais

usuallyformaldocuments,

includingorganization

plans,earnedmedia,

policies,etc.

Canbeanalyzedbycountingkey

piecesofinformationand/or

summarizingthenarrative,tailored

tocapturedatatolearnaboutone

ormoreoutcomes

Observation

Inrealtime,

asanaction

isbeing

taken.

Oneortwoobservers,

collectingdataabout

everyonebeingobserved

(orlistenedto).

Countsandqualitativenarrative

regardingeasilyobservableactions

duringagroupeventoractivityof

anytype,tailoredtocapturedatato

learnaboutoneormoreoutcomes.

CaseStudyMixof

settings.

Typicallyincludesamixof

documentreview,

observation,interviews,

and/orfocusgroups.

Indepthlearningfrommultiple

perspectivesaboutaspecificexample

ofchangehappening(e.g.asubsetof

leaderswhoworkedtogetherora

specificleader),tailoredtocapture

datatolearnaboutoneormore

outcomes.

Outcome

Harvesting

Group

dialogue.

Typicallytheteamof

programstrategistswith

inviteeswhohaveagood

perspectiveontheprogram

Participatorytechniquetoidentify,

formulate,verify,andmakesenseof

outcomeswhenrelationshipsof

causeandeffectareunknown.Good

forsurfacingunplannedoutcomes.

ThismaterialisadaptedfromdatacollectiontoolspresentedaspartofSparkPolicyInstitute’sadvocacyevaluationtoolkit.

Tool Setting #ofInformants TypeofInformationtoCollect

RippleEffect

Mapping

Paired

interviews

andgroup

dialogue

Typicallyalargegroupof

programparticipantsand

membersofthelarger

network.

Appreciativeinquiry-styleinterviews

toelicitstoriesofsuccessfollowedbya

groupmappingof“ripples”ofchange.

Capturesawiderangeofoutcomesthat

mayhavehappenedindifferent

locations,aswellasgrouphypotheses

aboutwhatlikelysecondaryeffects

thoseoutcomeswillhave.

SocialNetwork

Analysis

Onpaper,by

email,orby

phone.

Canbesurveysor

interviews,dependingon

useofinformation.

Helpstounderstandrelationships

betweenindividualsororganizations,

theresourcesleveragedinthose

relationships,andcanalsoprovide

insightintothewhyandhowbehind

therelationshipswhendone

qualitatively.

360s(e.g.

Servant

Leadership

Questionnaire)

Onpaper,by

email,orby

phone.

Apre-definedinterview

instrumentsdeployedto

multipleaudiencesofthe

leader.

Gathersperspectivesonleader

characterandperformancefromthose

indirectcontactwiththeleaderona

numberofdifferentdimensions,aswell

asfromtheleaderherorhimself.

Identity

Leadership

Inventory

Onpaper,by

email,orby

phone.

Apre-definedinterview

instrumentsdeployedto

multipleaudiencesofthe

leader.

Aleadershipinventorybasedonsocial

identitytheorythatcaptureshowa

leaderaffectsgroupcohesionand

identity,embodiesthegroup’ssenseof

“we”,andcreatesplatformsorvehicles

forthegrouptoactonitsprinciples

andinterests.

Collective

EfficacyScale

Onpaper,by

email,orby

phone.

Apre-definedsurvey

instrumentdeployedto

multipleaudiencesofthe

leader.

Demonstratestheextenttowhicha

grouphasconfidencethatitis

equippedtoachieveitsgoalstogether.

Collectiveefficacyispredictiveof

successbecauseitincreasesagroup’s

abilitytoovercomeadversityand

maintaincommitment.

Appreciative

Inquiry

Interviews

Byphoneor

inperson.

Interviewsconductedwith

audiencesoftheleader.

Focusesonsurfacingpastexperiences,

inthiscasewiththeleader,thatare

importantrepeatinthefuture(very

focusedonstrengths).

Discourse/

FrameAnalysis

Onpaper,by

email,orby

phone.

Interviews,surveysor

focusgroups

Explorestheframesornarratives

peopleusetounderstandanissue,a

group,orthemselves,and/ortheextent

towhichnewframesandnarratives

resonatewiththeirexistingvalues.

ThismaterialisadaptedfromdatacollectiontoolspresentedaspartofSparkPolicyInstitute’sadvocacyevaluationtoolkit.

RecommendedResources

AmericanEvaluationAssociation(http://www.eval.org/)

TheAmerican EvaluationAssociation’smission is to improve evaluation practices andmethods,

increase evaluation use, promote evaluation as a profession, and support the contribution of

evaluationtothegenerationoftheoryandknowledgeabouteffectivehumanaction.

CenterforEvaluationInnovation(http://www.evaluationinnovation.org/)

Ouraimistopushphilanthropicandnonprofitevaluationpracticeinnewdirectionsandintonew

arenas. We specialize in areas that are challenging to assess, such as advocacy and systems

change.

FreeResourcesforProgramEvaluationandSocialResearchMethods

(http://gsociology.icaap.org/methods/)

ThispagelistsfreeresourcesforProgramEvaluationandSocialResearchMethods.Thefocusison

"how-to"doprogramevaluationandsocialresearch:surveys,focusgroups,sampling,interviews,

andothermethods.Mostoftheselinksaretoresourcesthatcanbereadovertheweb.

SparkPolicyInstitute(http://sparkpolicy.com/)

Spark Policy Institute partnerswith stakeholders throughout the country to develop innovative,

research-based solutions to complex societal problems. Spark combines community and

stakeholder-driven researchwithpractical,hands-onexperienceandbestpractices, allowing for

solutionsthatbridgesectors,issues,beliefs,andvalues.Byintegratingdiversepolicysystems,the

teamatSparkidentifiesanddevelopsthebestsolutionsforallstakeholders.

19

Notes

1There’satendencyinprogramevaluationcirclestothinkthat,inordertobecredible,aprogramneedsto

demonstrateandaccountforthefullchainofcausationfromactivitythroughoutputs/outcomestoimpactforeach

discreteprogramorintervention.The“goldenspike”analogy(evokingthefinalconnectionbetweenCentralPacific

andUnionPacificRailroadstocompletetheFirstTranscontinentalRailroadacrosstheUnitedStates)shortcircuitsthis

presumptionbyarguingthatanevaluationcan(andshould)drawonexistingresearchforanalogousproofpointsto

connectactivitiestoultimateimpactwithouthavingtoprovethefullcausalchainforeachdiscreteintervention.Say,

forexample,thatyou’reconcernedaboutchildhoodobesity.Youdesignaninterventionthatreplacesprocessed

snackswithfreshfruitandvegetablesnacksinalocalmiddleschoolcafeteria.Thegoldenspikeconceptsuggeststhat

ifyoucandemonstratethatstudentsareconsumingmorefruitsandvegetablesandfewerprocessedsnacksasa

resultofyourintervention,andyoucancitetoresearchthatdemonstratesthatchildrenwhoincreasetheir

consumptionoffruitsandvegetablesanddecreasetheirconsumptionofprocessedsnacksexperiencereductionsin

obesity,youdon’tneedtoprovethatyourinterventionreducesobesity.Youcansimplyrelyonthe“goldenspike”

connectionbetweenwhatyou’redoingandwhattheresearchindicateswillhappenasaresult.Thewillingnesstouse

the“goldenspike”hasimportantimplicationsforevaluationbudgets,timelines,participantresources,andabilityto

produceresultsthatwillbeseenascredibleanduseful.

2FrancesWestley,BrendaZimmerman,andMichaelQ.Patton,GettingtoMaybe:HowtheWorldisChanged

(Toronto:VintageCanada,2007).