integrating gender into project-level evaluation
TRANSCRIPT
ECGReferenceDocument–IntegratingGenderintoProject-levelEvaluation–Appendixes ii
AcknowledgmentsThemethodologicalguidelinesaretheproductofacollectiveworkofanECGTaskForcecomposedofthefollowinginstitutionsandmembers,ledbytheAfricanDevelopmentBank(AfDB),andsupportedbyMichaelJ.Bamberger,ExpertConsultant.
TaskForcemembers
IndependentDevelopmentEvaluation(IDEV)AfricanDevelopmentBankGroup(AfDB)
SamerHachem,DivisionManagerSohnaNgum,Consultant(OfficeoftheSpecialEnvoyonGender)JessicaHarris,Consultant(OfficeoftheSpecialEnvoyonGender)
IndependentEvaluationDepartment(IED)AsianDevelopmentBank(AsDB)
FarzanaAhmed,LeadEvaluationSpecialistHyunSon,PrincipalEvaluationSpecialist
EvaluationDepartment(EvD)EuropeanBankforReconstructionandDevelopment(EBRD)
ShireenEl-Wahab,PrincipalEvaluationManagerBeatrizPerez-Timermans,Principal,EvaluationManager
OperationsEvaluation(EV)EuropeanInvestmentBank(EIB)
EmmanuelPondard,EvaluationSpecialist
IndependentEvaluationOffice(IEO)GlobalEnvironmentFacility(GEF)
GeetaBatra,Deputy-DirectorAnnaViggh,SeniorEvaluationOfficer
IndependentEvaluationOffice(IEO)IFAD
CatrinaPerch,EvaluationSpecialistMarkKeating,EvaluationOfficer
IndependentEvaluationOffice(IEO)UNWomen
ShravantiReddy,EvaluationSpecialistSabrinaEvangelista,EvaluationSpecialist
IndependentEvaluationGroup(IEG)WorldBankGroup(WBG)
GiselaGarcia,EvaluationOfficerElenaBardasi,SeniorEconomist
ECGReferenceDocument–IntegratingGenderintoProject-levelEvaluation–Appendixes iii
TableofcontentsAcknowledgments..........................................................................................................................................................ii
Tableofcontents...........................................................................................................................................................iii
Appendix1:Therationaleforintegratinggenderintoprojectandprogramevaluations...................4
Appendix2:ThespecialchallengesofgenderevaluationwithintheIEOcontext...................................6
Appendix3:OverviewofGREEvaluationDesigns..............................................................................................81.ThemainevaluationdesignsthatcanbeusedforstandardGRE.................................................................................................................82.Morein-depthmethodsforspecialGREevaluations.......................................................................................................................................10
Appendix4:Toolsforcollectingdataforgenderevaluations......................................................................121.Mixedandmulti-methodapproaches.....................................................................................................................................................................122.Secondarydatasources................................................................................................................................................................................................133.Theorybasedevaluation(TheoryofChange).....................................................................................................................................................134.Reconstructingbaselinedata.....................................................................................................................................................................................135.Surveys.................................................................................................................................................................................................................................146.Qualitativemethods.......................................................................................................................................................................................................147.Broaderapplicationsofqualitativemethodsusedinspecial,in-depthGREs.......................................................................................15
Appendix5:AdatacollectionplanningmatrixforaGRE:ApplicationtoahypotheticalVillageDevelopmentProjectinCentralAsia.....................................................................................................................21
Appendix6:ExampleofaGREdesignmatrix:EvaluatingahypotheticalvillagedevelopmentprojectwithdefinedgenderobjectivesinCentralAsia...................................................................................23
1.Theproject’sgenderobjectives.................................................................................................................................................................................233.Theproject’sgenderresponsiveinterventions..................................................................................................................................................244.Definitionofthegenderquestionstobeaddressed.........................................................................................................................................255.Initialproposalfortheevaluationmethodology...............................................................................................................................................26
Appendix7:ExampleofaGREdesignmatrix:EvaluatingahypotheticalpublictransportprojectinEurasiawithdefinedgenderobjectives................................................................................................................33
1.Frameworkoftheevaluationreport.......................................................................................................................................................................332.Theproject’sgenderobjectives.................................................................................................................................................................................333.Theproject’sgenderresponsiveinterventions..................................................................................................................................................344.Definitionofthegenderquestionstobeaddressedintheevaluation.....................................................................................................365.Initialproposalforthegenderevaluationmethodology...............................................................................................................................38
Appendix8:Linkstothegenderindexes.............................................................................................................48
Appendix9:Strengtheningthedisseminationanduseofgender-evaluationfindings.......................491.Theunderutilizationofevaluations........................................................................................................................................................................492.StrategiesforpromotingtheutilizationofGREevaluations........................................................................................................................494.Buildinggenderindicatorsandfindingsintokeyagencyreports.............................................................................................................50
ECGReferenceDocument–IntegratingGenderintoProject-levelEvaluation–Appendixes
4
4
Appendix 1: The rationale for integrating gender into project and program evaluations Thefollowingparagraphs(adaptedfromBamberger,2013)presentarationalethatcanbeadaptedbydifferentagenciestostresstheimportanceofincorporatingagenderfocusintoagencyevaluationprograms.Inthiscase,theexamplesaretakenfromcountry-levelimpactsbutsectorexamplescanalsobeincluded.ForamorecomprehensivereviewseetheOverviewofthe2012WorldDevelopmentReportonGenderandDevelopment.
WhyisitimportanttoincorporategenderintoM&E?
Ineverysociety,therearerulesgoverningappropriatebehaviorformenandwomenandgirlsandboys,inthehome,thecommunity,thelabormarket,schools,andinpolitics.Someoftheserulesareregulatedbysocialcustoms,othersbylawsortheoperationofthelabormarket.Sometimestheformsofcontrolaresubtle,whileothersmaybeenforcedbylegalsanctionsorthethreatofviolence.Whilesomesectorsofsocietymaybelievetheserulestobebasedon“natural”differencesbetweenmenandwomen,therulesare,infact,sociallyconstructedandvaryfromonesocietytoanotherandovertime.However,despitedifferencesacrosssocieties,ineverycountrythathasbeenstudied,theserulesplacewomenatadisadvantagewithrespecttokeydimensionsofdevelopment.
Thepersistenceofsignificantgenderinequalitiesinallregionsnegatesfundamentalhumanrightsandtheexpansionofhumanfreedoms.Inaddition,genderinequalitiesareseriousbarrierstotheachievementofdevelopmentobjectives(BoxA1-1).
InGenderEqualityandDevelopment(WorldBank,2012),itisarguedthatpromotinggenderequitycanmakeamajorcontributiontodevelopment:first,byfullyutilizingthecapacitiesofbothwomenandmen;second,throughimproveddevelopmentoutcomesforthenextgeneration;andthird,bymakinginstitutionsmorerepresentative.Genderequitywillopenthedoorstomorepolicychoicesandinstitutionswillbecomemorerepresentative.
ECGReferenceDocument–IntegratingGenderintoProject-levelEvaluation–Appendixes
5
5
Inlightofthiscompellingevidence,manygovernmentsandinternationaldevelopmentagencieshaveprioritizedgenderequalityasoneoftheirtopdevelopmentobjectives.Achievinggenderequalityrequiresintegratinggenderintoallaspectsofprogramming,budgeting,implementation,monitoring,andevaluation.Manyorganizationshaveoperationalizedtheirgenderequalitystrategiesthroughgenderactionplans(GAP)thatstressthecriticalroleofappropriategendertoolsfordatacollectionandanalysisformonitoringandevaluation(M&E).WhileaGAPprovidesausefulframeworkforanintegratedapproachtogenderequality,itisnotessentialandmanyagenciesbeginbybuildinggenderintotheirexistingM&Esystems.TheymaythendevelopabroadergenderframeworkaftergainingexperiencewithgenderM&E.
Source:M.Bamberger(2013),EngenderingM&EPREMSeriesonNutsandBoltsofM&Esystems.NoteNo.27.WorldBank.
BoxA1-1:EstimatedEconomicCostsofGenderInequality:SomeexamplesfromAfrica,theMiddleEastandAsia.
• IntheMiddleEastandNorthAfrica,ifwomen’slaborforceparticipationhadincreasedinthe1990satthesamerateaswomen’seducation,theaveragehouseholdincomewouldhavebeen25percenthigher.
• Tanzaniacouldincreasegrowthbyonepercentbyremovingbarrierstowomenentrepreneurs.• IfIndiaincreaseditsratiooffemaletomaleworkersbytenpercent,grossdomesticproduct
wouldincreasebyeightpercent.• TotalagriculturaloutputinSub-SaharanAfricacouldincreasebysixto20percentifwomen’s
accesstoagriculturalinputswereequaltomen’s.• AsiaislosingbetweenUS$42billionandUS$47billionperyearduetowomen’slimitedaccessto
employmentopportunities.• AsiaislosingbetweenUS$16billionandUS$30billionperyearasaresultofgirls’limitedaccess
toeducation.
Source:DFID(2008);UNESCAP(2007).
ECGReferenceDocument–IntegratingGenderintoProject-levelEvaluation–Appendixes
6
6
Appendix 2: The special challenges of gender evaluation within the IEO context [ThisisanexpandedversionofthediscussioninPartI-ChapterOne]
ThechallengesandopportunitiesresultingfromtheIEOmandate
TheReferenceDocumentrecognizesthatIEOsoperatewithinamandatethatdefinesthescopeoftheirevaluations,thetime-frameswithinwhichtheyoperateandtheevaluationmethodologiesthattheyuse.Giventheseparameters,theIEOapproachtoevaluationingeneral,andGREinparticular,isdifferentfromtheevaluationscenariosandmethodologiesdiscussedinmostevaluationtextbooks.IEOswereestablishedtoensuretheirindependencesothattheycanprovideanobjectiveassessmenttotheBoardofDirectorsandotherkeystakeholdersontheextenttowhichprojectsandotherinterventionshaveachievedtheirobjectives,whethertheyhaveusedtheirresourcesinthemostefficientway,andwhethertheyhavecompliedwiththeadministrativeproceduresandcodesofconductdefinedbytheirrespectiveagencies.
ThismandateprovidesbothuniqueopportunitiesandchallengesforconductingGREs.OpportunitiesarisebecauseIEOsreportdirectlytotheBoardofDirectors,meaningthereareinstitutionallydefinedmechanismsforthedisseminationanduseoftheevaluation,andprocedurestoensuretheobjectivityandindependenceoftheevaluations.ChallengesarisebecauseIEOevaluationsareconductedex-postafterprojectshaveclosed,sotypicallytheycannotinfluencethekindsofbaselineorimplementationdatacollectedontheprojectsbeingevaluated.Consequently,itisnotalwayspossibletoapplymostofthepre-test:post-testexperimentalandquasi-experimentalevaluationdesignsadvocatedbyevaluationtextbooks.ThesechallengesareparticularlyimportantforGREwheretheprocessesofwomen’sempowermentareregulatedthroughsocialmechanismsandprocessesofbehavioralchange,whichideallyshouldbeobservedovertimeratherthanassessedthroughrecallattheendoftheprocess.However,thisreferencedocumentreferstosomeofthenewinformationtechnologies(smartphonesandbigdata)thatofferthepossibilitytoreconstructbaselineandlongitudinaldatafromexistingdatasets,suchastwitterandothersocialmedia,satellitetimeseries,phonerecordsandelectronicfinancialtransactiondatasuchasATMs.Thesesourcesopen-upthepossibilitytobroadentherangeofevaluationmethodologies.
Specialchallengesfacinggenderevaluation
Inadditiontochallengesthatalldevelopmentevaluationsface,GREsfaceanumberoftheirownspecialchallenges:
a. Relevance:Manyagencystaffarenotconvincedthatgenderissuesarerelevantinallsectors.Forexample,somestaffworkingininfrastructure,financeortrademayarguethattheirsectorsare“genderneutral”andthatbothmenandwomenhavethesameneedsandwillbenefit(orbeaffectednegatively)equally.
b. Costandtime:GREfrequentlyinvolveadditionalcostsasmoredatahastobecollected,anddatamaybemoreexpensiveortime-consumingtocollect.Giventhedemandsonevaluationresources,theseconsiderationscanbeaseriousconstraint.
c. Dataisnotavailable:Whereanagencyhasnotcollectgenderdata,thislackofdataisgivenasthereasonfornotincludinggenderinanevaluation,creatinga‘chicken-and-egg’situation.Asgenderissuescannotbeanalyzed,itissometimesarguedthatwedonotknowifgenderissuesarerelevant–sodataisnotcollected.GREdatacanbemoredifficulttocollect,soitmaybehardtomakethecaseforitscollectionwithoutevidenceastoitsrelevance.
d. GRErequirestheuseofnewandunfamiliarmethodologies:Someresearcherswhohaveestablishedtheirprofessionalreputationsbyconductingcertainkindsofevaluation(e.g.,
ECGReferenceDocument–IntegratingGenderintoProject-levelEvaluation–Appendixes
7
7
collectionandanalysisofconventionalquantitativedata)maybereluctanttoincorporatenewkindsofdataandmethodswithwhichtheyarenotfamiliar.Forsomeresearchers,GREcanalsoseemthreateningasitmayquestionthevalidityofsomeofthetraditionalevaluationmethodsi.Somestaff,particularlythosetrainedineconomicsandquantitativemethods,maynotconsiderthemorequalitativelyorientedGREasbeing“professional”evaluations.
e. Theremaybeconcernsabouttheperceivedpoliticalideologyoffeministresearchers.Althoughthiswillfrequentlynotbethecase,insomecountriesfeministsareperceivedashavingapoliticalagendathatmaybedisruptiveorthatmaydivertevaluationsawayfromwhatareperceivedtobetheirintendedpurposefortheagency.
ECGReferenceDocument–IntegratingGenderintoProject-levelEvaluation–Appendixes
8
8
Appendix 3: Overview of GRE Evaluation Designs
1.EvaluationdesignsusedforstandardGREsThefollowingarethemostcommonevaluationdesignsusedforstandardGREs:
A.Descriptivepost-projectfieldvisits.ThisisprobablythemostwidelyusedGRE.DeskresearchandpersonalorphoneinterviewswithkeyinformantsarecombinedwithcountryvisitsbyIEOstaffand/orconsultants.Rapidprojectvisitsandindividualorgroupinterviewswithcommunitygroups,arecombinedwithmeetingswithimplementingagenciesandlocalgovernmentofficials,civilsocietyandotherkeyinformantinterviews.Focusgroupdiscussions(FGDs)withcommunitygroups,civilsocietyorimplementingagenciesmayalsobeincluded.
B.Quasi-experimentaldesigns.Whentimeandresourcespermitasurveymaybeconducted.Thismayuseapre-test:post-testcomparisonwithasampleoftheprojectpopulationbutwherepossibleacomparisongroupshouldalsobeincluded.GiventhatIEOevaluationsareconductedretrospectively,baseline(pre-test)datawillnormallyusetechniqueforreconstructingbaselinedata.
C.Theorybasedevaluation.Atheorybasedevaluation,suchasatheoryofchange(TOC),canprovideausefulframeworkforthedesignandimplementationofanevaluation.Frequently,thestandardGREwillincludearelativelysimpleformofTOCwhilespecial,in-depthevaluationsmayuseamoreelaborateform.Consequently,notalloftheelementsdescribedherewillbeincorporatedintoeverystandardGRE.
ATOCcanbeparticularlyusefulforGREasitcanhelpidentifythemanysubtleanddifficult-to-measurefactorsthatcanconstrainsuccessfuloutcomesofgenderinterventions.Manyofthesefactorsarenotnormallyaddressedinconventionalevaluations,sotheTOCcanalsoserveasachecklisttoensurethatimportantquestionsandindicatorsarenotoverlooked.Ideally,aTOCforaGREshouldincludethefollowingelements:
a. Problemdiagnostic:Descriptionoftheproblembeingaddressedandsomeofitscauses.Wherepossiblethisshouldincludeahistoricalanalysistoidentifyhowpastexperienceswithinterventionsinthisareamayaffectattitudestothepresentproject.
b. ProjectIntervention:Identificationofthegender-responsivecomponentsandinterventions.c. Implementationprocesses:Theprocessesthroughwhichthedifferentcomponents/serviceswill
bedelivered.d. Outputs:Thegender-responsiveoutputsthateachcomponentisexpectedtoproduce.e. Outcomes:Theintendedgender-responsiveoutcomesandthecombinationofoutputsthatare
expectedtocontributetoeachoutcome.Thereisrarelyaone-to-onerelationshipbetweenasingleoutputandaparticularoutcome,soitisimportanttoidentifythedifferentoutputsandexternalfactorsthatcancontributetoeachone.
f. Impactsorgoals:Someagenciesbreak-downimpactsintoshort,mediumandlong-termwhileothersdistinguishbetweenimpacts(thatareclearlylinkedtotheproject)andbroaderdevelopmentgoalstowhichprojectimpactsareonlyonecontributingfactor.
Inadditiontotheaboveelements,thatareusuallypresentedinalinearfigurewitheachelementdirectlylinkedtothenextlevelup,thereareotherimportantelementsthatshouldbeaddressedintheGRE.
• Contextualfactors(e.g.economic,legal,organizational,political,cultural,climaticandhistoricalfactorsthatcanaffecthowtheprojectisdesigned,implementedandthegender-responsiveoutputs,outcomesitproduces.Thisshouldbelinkedtofactorsaffectingparticularoutcomeand
ECGReferenceDocument–IntegratingGenderintoProject-levelEvaluation–Appendixes
9
9
impacts.Thesefactorsarecloselylinkedtothechallengesofaddressingcomplexityintheevaluationdesign(Bamberger,VaessenandRaimondo,2016;FunnellandRogers,2011;Patton2011).
• TheTOCmustbefalsifiable.Atheorymustbetestable(otherwiseitisjustastatementofbelieforhope),whichmeansthatitmustbepossibletostatethatithasnotworkedorthatthereisnocredibleevidencetoshowthatitdidwork.ThisrequiresmorerigorinthearticulationoftheTOCthanisusuallythecase.RequirementsofarobustTOCinclude:
o Atimelineoverwhichoutputs,outcomesandimpactsaretobeachieved.AcriticismofmanyTOCapproachesisthattheydonotincludethistimeline,sothatifexpectedoutcomeshavenotbeenachieveddefendersoftheprojectcanalwaysarguethat“ourtheoryofchangeisvalid,itisjustthatmoretimeisneededtoproduceresults.”Ideally,thereshouldbearelativelylongtimelineforGREasmanyoutcomes,particularlythoseinvolvingbehavioralororganizationalchange,evolveslowlyovertime.Also,theprocessesoftransformativechangeareusuallynotlinear,andtheremaybebacklashandresistancewhichisrecognizedbyfeministresearchersinthesaying“twostepsforward–onestepback”.
o Clearlydefinedandmeasurableindicators.
o Asetofalternative(rival)hypothesesthatcouldexplainhowtheexpectedoutcomescouldhavebeenachieved.Soeveniftheexpectedoutcomesareachieved,itisnotpossibletoconcludethatthesechangeswereduetotheprojectinterventionunlesstherivalhypotheseshavebeentestedandfoundnottobecredible.
g. Emergence:AcriticismofmostTOCsisthattheyarestaticandimplicitlyassumethattheenvironmentinwhichtheprogramoperateswillnotchangeoverthelife-timeoftheproject.However,thisstaticassumptionisrarelytrue.Governmentschange,thelocalandnationaleconomychange,andothercomplementaryorcompetingprogramsarelaunched.Equallyimportantisthefactthatthenatureoftheprojectandhowitoperateswillalsochange.Realistevaluation(Pawson,2013)hasshownthatprojectschangeinresponsetointeractionswithaffectedcommunitiesandotherstakeholders.Servicesanddeliverystylesthatbeneficiarieslikewillcontinueandbestrengthened,whilethosethatpeopledonotlikewilloftenchangeorwilldieoutifno-oneusesthem.Furthermore,programswilloftenevolveinnewdirectionssothattherewilloftenbeimportantunanticipated(positiveandnegative)outcomes.TheTOCmusthavetheflexibletoadapttotheemergentenvironment.Emergencepresentsmanychallengesfortheevaluator.Inadditiontothemethodologicalchallenges,thereisthefactthatmanyprojectswillbeheldaccountableonthebasisofthe(almostalways)relativelystaticresultsframework.Whilethisframeworkcanrevisetheoriginalnumericaltargets,mostresultsframeworksdonothavetheflexibilitytoincorporatecompletelynewandunplannedoutcomesorimpacts,andimplementingagenciesarefrequentlynotassessedontheirflexibilitytoadapttochangingcircumstances.
D.Objectives-basedevaluation(results-basedmanagement).Manyagenciesincorporateintotheirprojectdesignsaresultsframeworkthatdefinesasetofoutput,andoutcomeindicatorsthatdefinetheintendedresults.Usuallytheseincludebaselinemeasures,initialintendedtargetstobeachievedoveragivenperiodoftimeandtheactualvaluesachievedii.Theresultsframeworkisagoodstartingpointfortheevaluation,bothbecauseitprovidesapreciseandcomprehensivedefinitionofintendedresults,andbecauseprojectsarerequiredtocollectinformationonallofthesefactors.
However,thepracticallimitationforGREisthatmostprojectshaveonlyafewobjectivesthataredisaggregatedbysex,andevenfewerspecificgenderobjectives.So,whiletheresultsframework
ECGReferenceDocument–IntegratingGenderintoProject-levelEvaluation–Appendixes
10
10
providesausefulstartingpoint,itwillalmostalwayshavetobecomplementedbyothersourcesofinformation.Insomecaseitmaybepossibletoreconstructapproximategenderobjectivesby,forexample,askingstaffhowimportanttheinclusionofwomenincommitteesandleadershippositionsorinaccesstotrainingandprojectbenefitswasintheprojectdesign?Importancecanberatedas“veryimportant”,“quiteimportant”,“notimportant”and“notconsidered”.ManyIEOsincludetheseratingsinthegenderflags(seeChapter2,Section2.4)thatareusedtorateprojectdesignandimplementationintermsofhowwellgenderissuesareaddressed.Women’sactualparticipationoraccesstobenefitsindifferentactivitiescanthenbeassessedintheevaluation.Ofcourse,incaseswheretheprojecthadgender-specificobjectives,amorepreciseassessmentcanbemade.
E.Casestudies.Casestudydesignsarebasedontheselectionofcases(households,communities,schoolsandsoforth)thatarebroadlyrepresentativeofthetotalprojectpopulation.Casescanbedescriptive(illustratingthedifferenttypes),illustrative(describingthedifferenttypologiesfoundinsurveysorotherpartsoftheevaluation)oranalytic(seekingtoidentifyorexplaindifferentbehavior,processesofchangeoroutcomes).Casestudiesareaverypowerfulformofevaluationastheycandigdeeperandhelpexplainthelivedexperiencesofdifferentsectorsofthetargetpopulation.Forgenderevaluation,casescanoftenuncoverandhelpunderstandsubtleprocessesthataredifficulttocaptureininterviews.Thelatersectiononspecial,in-depthevaluationswilldiscussthegrowinguseofQualitativeComparativeAnalysis(QCA)casestudies.RobertYin(2003,2004,2012)providesausefulintroductiontocasestudyresearch,whileByrneandRagin(2009)provideacomprehensivereviewofcasestudymethods.However,noneofthesetextsfocusdirectlyongender.
F.Qualitativemethods.AlmostallGREincorporatequalitative(QUAL)methodsintotheevaluationdesign.Thesemethodsincludeunstructuredandsemi-structuredindividualinterviews,groupinterviewsanddiscussions(includingfocusgroups),andobservation(participantandnon-participant).Therearealsoawiderangeofparticipatorygroupdiscussiontechniques(PRA)thatincludemanymethodsdesignedforgroupswithlowliteracy.Onesetofveryusefultechniquesusesocialmappingtohelpunderstandthesocial,economicandpowerstructureofacommunity,andanothersetoftechniquestraceshistoricaltimelinesandthemajoreventsinthehistoryofthecommunity.
2.Morein-depthmethodsforspecialGREevaluationsParticularprojectsorthemesaresometimesselectedformorein-depthevaluation.Thefollowingaresomeofthewiderrangeofevaluationtoolsthatitmaybepossibletouseforthesestudieswhenmoretimeandresourcesareavailableandwherethemandatemaybebroader.Asindicatedearlier,thereisnoclearlinebetweenstandardandin-depthevaluations.
Theorybasedevaluation.In-depthevaluationsareoftenabletousesomeofthemoreadvancedapproachestotheory-basedevaluation,including:
a. Contributionanalysis
Experimentaldesignsarerarelypossible,particularlyforretrospectiveevaluations.Contributionanalysis(Mayne,2011)recognizesthatitisrarelypossibletoassessthedirecteffectofaparticularinterventionasmostprojectsareimplementedincontextswherethereareotheragencies,otherprojects,newgovernmentpoliciesandexternalfactors,allofwhichcancontributetoobservedchangesintheareawhereaprojectisoperating.Consequently,thepurposeistodefineandassessthemostplausible“contributionstory”.Theanalysisusuallyinvolvessixsteps:
• Definehowtheprojectisintendedtocontributetoasetofoutcomes(thecauseandeffectrelationship).
• Layouttheprogramtheory(developthe“programstory”).• Gatherallofthesupportingevidence.• Assessandchallengethecontributionclaim(identifyandtestrivalhypotheses).
ECGReferenceDocument–IntegratingGenderintoProject-levelEvaluation–Appendixes
11
11
• Findadditionalevidence.• Strengthenthecontributionclaim.
WhilemanyTOCsonlydescribeandassesstheproject’stheoryofchange,contributionanalysisarguesthatitisessentialtoidentifyandtestplausiblerivalhypotheses.Italsostressestheimportanceofcontinuallyseekingnewevidencetosupportorchallengetheevolvingtheory.
b.Outcomeharvesting
Theapproachissimilartocontributionanalysis,butwhiletheformerisoftenimplementedatthestartofaprojectandcontinuesthroughoutitsimplementation,outcomeharvestingisusedattheendofaprojectandisconsequentlyveryrelevantforIEOevaluations(WilsonGrau,2012).Outcomeharvestingcollectsstoriesfromprojectbeneficiariesandotherstakeholdersonchangesthathaveoccurredoverthelifeoftheproject.Itisquitecommonforhundredsofstoriestobecollected.Theyareorganizedintogroupsandtheircredibility,andtheirlinkstotheproject,arethenassessedbytheresearchteam.Manyevaluatorsfindthisapproachusefulbecausethestories(potentialoutcomes)aregeneratedbythestakeholdersthemselvesandnotbytheresearchteam.Similarapproachestocontributionanalysisareusedtotestthecredibilityoftheprogramstory(theory).
c.Realistevaluation
Realistevaluation(Pawson,2013)believesthattheexperimentaldesignquestion“Didtheinterventionworkinthiscontext,withthispopulationandatthispointintime?”istoonarrowtobeofmuchpracticalvalue.Insteadrealistevaluationasks:“Whatworks,forwhom,inwhatrespects,towhatextent,inwhatcontexts,andhow?”Thesequestionsareaddressedbydefining“generativemechanisms”thatexplain“how”theoutcomeswerecausedandtheinfluenceofcontext.Thebasicmodelisdefinedas:C(context),M(mechanisms)O(outcomes).
Theuniquefeatureoftheapproachisthebeliefthatthe“reasoning”oftheactorsinresponsetotheresourcesoropportunitiesprovidedbytheprojectiswhatcausestheoutcomes.Consequently,theprocessofunderstandingthisreasoningandthefactorsthataffectitimpliesaverydifferentresearchapproachfromconventionalevaluationdesignsthatassumetheoutcomesarecausedbytheprojectinterventions.TheapproachispotentiallyveryvaluableforGREaswomen’sreasoninginresponsetoprojectopportunitiesisinfluencedbyawiderangeofsocial,economic,cultural,political,legalandhistoricalfactors–whichiswhyprojectoutcomescanbeverydifferentindifferentcommunitiesandcontexts.TheapproachisalsoconsistentwiththeGREapproachofconsideringwomen’sagency.
d.Broaderapplicationsofqualitativemethods.
Thenatureofmostqualitativemethodsisthattheyrelyonbuildingtrustwiththecommunitiesbeingstudied,andconsequentlytheirapplicationideallyrequiresaconsiderableinvestmentoftime.So,althoughqualitativemethodsarewidelyusedinstandardevaluations,timeconstraintslimittheirfullapplication.ThisisdetailedinAppendix4below.
C
M________O
ECGReferenceDocument–IntegratingGenderintoProject-levelEvaluation–Appendixes
12
12
Appendix 4: Tools for collecting data for gender evaluations ThisAppendixdescribesthemostwidelyuseddatacollectionmethodsthatcanbeusedfortheGREevaluationdesignsdescribedinChapter3andAppendix3.Manyofthedatacollectionmethodsareusedinseveraloftheevaluations,andTable4.1oftheGenderNoteliststhemethodsmostcommonlyusedforeachdesign.ManyofthedatacollectionmethodscaneitherbeusedinarelativelysimplewayforstandardGREorinmoredepthforspecialevaluations.Thefollowingdiscussiondoesnottrytodistinguishbetweenstandardandmorein-depthdatacollectionapplicationsasthereisnoclear-cutlinebetweenthetwo.
1.Mixedandmulti-methodapproachesAlmostallIEOevaluationsrequireanassessmentofbothquantitative(howmuch?howmany?whoisincludedandexcluded?)andqualitativedimensions(understandingtheimplementationprocesses,thelivedexperiencesofdifferentgroups,andmechanismsofsocialcontrol).Consequently,allGREshouldincorporateamixed/multi-methodapproachiii.QUANTandQUALmethodsareoftencombinedinasomewhatadhocway(e.g.conductingnot-very-wellselectedfocusgroupsattheendofasurvey,orcommissioningafewcasestudiesthatareconductedwithverylittlecoordinationwithaQUANTsurvey).However,mixedmethodevaluationshouldbeconsideredasanintegratedevaluationstrategythatcancombineQUANTandQUALapproachesatallstagesoftheevaluation(Bamberger,RughandMabry,2012;Bamberger,2016).
ThereareseveraladvantagesofamixedmethodapproachforGREevaluations.First,triangulation(comparingindependentestimatesofakeyindicator)canincreasethereliabilityandvalidityofevaluationfindings.Second,combiningdifferentmethodsmayincreasethevalidityoffindingswhenworkingunderbudgetconstraintsiv.Third,mixedmethodshelpdescribeandunderstandinteractionsamongdifferentorganizationsandactorsandtoobserveprocessesofbehavioralchangetocomplementandhelpinterprettheQUANTfindings.
Thereareseveralguidelinestokeepinmindwhenusingmixedmethods:
• Mixedmethodsofteninvolveprofessionalsfromdifferentsocialsciencedisciplineswhohavedifferentwaysofworking.Consequently,itisimportanttoallowmoretimeforteam-buildingandplanninginordertointegratethedifferentapproachesandtofullybenefitfromthewiderrangeoffindingsandanalyticalmethods.TeambuildingisparticularlyimportantforGREasgenderandfeministresearchersoftenbringapproachesthatareunfamiliartomanyQUANTresearchers,whichmayinitiallycreatesomeresistanceorquestioningastowhethersomeoftheQUALmethodsreallymeet“professional”researchstandards.
• Thereshouldbeclosecoordinationwithrespecttodesignandapplicationofalldatacollectionmethods.Thisisessentialtopermittriangulationsothatonemethodcanbeusedtovalidatedatacollectedusingadifferentmethod.Thisisonlypossibleifallinstrumentsarecollectingcomparabledata.Thesituationoftenariseswhereestimatesofhouseholdincomeobtainedfromasurveyareinconsistentwithestimatesobtainedfromin-depthQUALinterviews.Oftenthequestionsareaskedindifferentwayssothatitisnotpossibletodeterminewhetherthedifferencesareduetothewaythequestionwasaskedorwhetherinfactonemethodiscollectingmorereliableinformationv.
• Thetimingofthecollectionofdifferentkindsofdatamustbecoordinatedsothattheinitialfindingsfromonemethodcanbeusedtocorrectanyissueswithanothermethod.Oftendiscrepanciesareonlyfoundlateintheresearchwhenitistoolatetomakeanycorrections.
ECGReferenceDocument–IntegratingGenderintoProject-levelEvaluation–Appendixes
13
13
2.SecondarydatasourcesAlmostallevaluationsbeginwithareviewofavailablesecondarysources.Thesecaninclude:
a. Projectdocuments.Whilemanyoftheseareincludedinprojectfilesandeasilyaccessible,oftenprojectstaffhavetheirownfileswithadditionalusefulreports.Forolderprojects,someofthesedocumentsmaynothavebeendigitalizedandhardertolocate.
b. Reportsfromotheragencies.Again,itmaybeusefultocontactagencystafftorequestdocumentsfromtheirpersonalfiles.
c. Systematicreviewsprovideasynthesisofresearchandevaluationfindings,thatmeetcertainmethodologicalstandards,fromallstudiesconductedonaparticulartypeofintervention(e.g.off-gridruralelectrification,villagewatersupply).Theseprovideausefulreferencepointtodeterminethemaximumimpactsimilarinterventionshaveachieved.Itisimportanttobeawarethatsystematicreviewsoftenhavequiteselectivescreeningcriteria,suchasonlyincludingrandomizedcontroltrials,soinmanyareasofgenderresearchitmaybethecasethatthevastmajorityofstudieswereexcludedbecausetheyusedqualitativemethods.
3.Theorybasedevaluation(theoryofchange)Theorybasedevaluationsusedamixedmethodsapproachfordatacollectionastheydrawonallavailablesourcesofdata.Forexample,contributionanalysisseeksatvariouspointsintheanalysistoidentifyanynewsourcesofdatathatmayberelevanttomakingthe“projectstory”morecredible,oralternatively,seekingevidencethatcouldchallengethecredibilityoftheprojecttheory.Thiswillofteninvolveseekingoutsourcesofdatathatconventionalevaluationswouldnotnormallyuse.
Therearetwokindsoftheory-basedevaluationthathavetheirownsourcesofdataanddatacollectionmethods:
a. Outcomeharvesting(Wilson-GrauandBritt,2012):Outcomeharvestingcollectsstoriesfromprojectbeneficiariesandotherstakeholdersonchangesthathaveoccurredoverthelifeoftheproject.Itisquitecommonforhundredsofstoriestobecollected.Theyarethenorganizedintogroupsandtheircredibility;theirlinkstotheprojectarethenassessedbytheresearchteam.Manyevaluatorsfindthisapproachusefulbecausethestories(potentialoutcomes)aregeneratedbythestakeholdersthemselvesandnotbytheresearchteam.Similarapproachestocontributionanalysisareusedtotestthecredibilityoftheprogramstory(theory).
b. Realistevaluation(Pawson,2013)seeksinformationtoanswerthequestions:“Whatworks,forwhom,inwhatrespects,towhatextent,inwhatcontexts,andhow?”ThisrequiresacreativeuseofmixedmethodsoftenwithastrongrelianceonQUALmethodstounderstanddifficult-to-measureconcepts,suchasbehavioralchangeandhowmechanismsofsocialcontrolinfluencedecisionsandactionsofindividualsandgroups.Realistevaluationalsoexamineshowthecontextaffectsprogramoutcomesandconsequentlydrawsonsomeofthekindsofdatausedincomplexityevaluationandsystemsanalysis.Processanalysisisalsousedtounderstandhowthegenerativemechanismsoperatevi.Thesearethecontextualfactorsthatcausepeopletoreasoninacertainwayaboutaprojectandwhichresultsintheiractinginaspecificwayinresponsetotheproject.
4.ReconstructingbaselinedataAsalmostallIEOevaluationsareconductedretrospectivelyaftertheprojecthasbeencompleted,itisnotpossibletoconductabaselinestudytocollectdataforapre-test:post-testcomparison.However,thereareanumberofwaysthatbaselinedatacanbe“reconstructed”:
ECGReferenceDocument–IntegratingGenderintoProject-levelEvaluation–Appendixes
14
14
a. Usingdatacollectedbytheprojectforprojectselectionanddesign.Forsomeprojects,suchaslow-costhousingormicro-credit,applicantshavetocompletequestionnaireswhichmayincludequitedetailedsocio-economicdata(Bamberger,RughandMabry,2012,Chapter5).
b. Usesecondarysurveydata.c. Conductingretrospectivesurveyswhererespondentsareaskedtorecalltheirsituationatthe
timetheprojectbegan.d. Keyinformantinterviews.e. PRAandotherqualitativemethods.f. GISdataandsatelliteimages.
5.Surveys.Forsomelargerevaluations,itmaybepossibletoconductsurveys.Theseincludearangeofapproaches:short,rapidsurveys,unstructuredorsemi-structuredinterviewsorlarge-scalestructuredsamplesurveys.Experienceshowsthatiftheevaluatorshavelocalcounterpartsitisoftenpossibletoconductashortsurveyofseveralhundredhouseholdsinonetotwoweeksandatamodestcost.Sometimesinterviewscanbeconductedbylocalteachersorstudentnurses,orinsomecasesevenhigh-schoolstudents.
IntegratinggenderintostandardQUANTsurveyswilloftenrequirecarefulcoordination.Inmanycasescollectingtherequiredsex-disaggregateddatamaybemoredifficultthanitmightseem.Whenhouseholdinterviewsareonlyconductedwiththe“headofhousehold”,inmanyculturesthemajoritywillbemen.Oftenthemanmaynotknowabouttheactivitiesoffemalemembers(forexample,howmanyhourstheyspendeachdayonhouseholdchoresandcollectingwaterandfuel?)Studieshaveshownthatmenoftenunderestimatethetimeburdenofsuchactivitiesontheirwivesanddaughters.Inmanyculturesmenalsohavelittleinformationontheirchildren’seducationbutareunwillingtoacknowledgethisfact.Anumberofstudieshavealsofoundthatmendonotmentiontheseveralhourstheirwivesspendeachdaycollectingwaterorfuelasaproblemvii.Forallofthesereasonsitisimportanttoplanhowreliablesex-disaggregateddatacanbecollected.Insomecases,itmightrequireamaleandfemaleinterviewertoworktogetherasateam,oritmaymeanarrangingafollow-upinterviewwithoneormorefemalemembersofthehousehold.
Observationisanimportanttooltocomplementsurveys.Forexample,whenaskedwhomakesmajordecisionsonhouseholdpurchasesandchildren’seducation,thewifewilloftensayitisherhusband.However,oncetheevaluatorhasgainedtheconfidenceofthewifeandisinvitedintothehouse,shewilloftenobservethatinfactthewifeisactivelyinvolvedinthesedecisions.
6.Qualitativemethods.Thereareawiderangeofqualitativetechniquesthatevaluatorscandrawon.Manyofthesehavetheadvantageofbeingrelativelyeconomicalandhavetheflexibilitytoadapttolocalconditions.Theyarealsousefulfortheanalysisoftheprocessofprojectimplementation,relationshipsamongdifferentagencies,andforunderstandingprocessesofbehavioralchange.HowardWhiteusedtheterm“ethnographiceconomics”torefertothecollectionofinsightsonhouseholdorcommunitydynamicsfromconversationswithstaff,communitymembersandcasualobservationthatcanprovidebackgroundforunderstandingfactorsthatinfluencebehaviorandprojectoutcomesthattendtogetmissedbyconventionalsurveysandprojectvisitsviii.Someofthemostwidelyusedtechniquesinclude:
a. Keyinformantinterviews.b. Focusgroups.c. Observationchecklists(forexample,forstudyinghowwomenparticipateinmeetings
andotherprojectactivities).
ECGReferenceDocument–IntegratingGenderintoProject-levelEvaluation–Appendixes
15
15
d. In-depthinterviews.e. PRAandotherparticipatorygroupconsultationmethods.f. Diaries:participantsorprojectstaffareaskedtokeepdetailedrecordsoftheiractivities,
suchastimeuseduringatypicalday,useofdifferentsourcesoffuelandpower,expenditures.
g. Photographsandartefacts(suchasornamentsandreligiousrelics,furnitureandhouseholdpossessions,photographsandgraffiti.
h. Audioandvideorecordings.
7.Broaderapplicationsofqualitativemethodsusedinspecial,in-depthGREsThefollowingillustratesomeofthewaysthatapplicationofqualitativetoolscanbeimprovedwhentimeandresourcespermit.
Initialdiagnosticstudies
Forcommunitybasedprojects,itisveryusefultoassignaresearchertospendseveralweeksinthecommunityinordertohelpunderstandthedailylives,problemsandconcernsandattitudestotheproposedinterventions(PillowandMayo,2012;Clarke,2012;Salmen,1987).Thegoalistobeabletoobserveandtoexperiencecommunityliferatherthantoconductinterviews.Itcanalsobehelpfultounderstandthelanguageandconceptsusedtothinkaboutanddescribekeyconceptssuchaswellnessandsickness,povertyandvulnerability,publicagenciesandtheservicestheyprovide,andpeople’saspirationsandfears.Theresearcherwillalsoexplorelocalhistoryandhowthisaffectsattitudestotheproposedproject.
Itisimportanttonotethatinmanycasesthestudycanbeconductedbyalocalresearcher(withappropriateguidance)sothestudydoesnotnecessarilyhavetobeveryexpensive.
Keyinformantsandinformalpanelstudies
Theluxuryofmoretimemakesitpossibletobetterselectkeyinformantsandtobuildconfidencewiththem.Manyofthemostvaluableinformantsprovenottobethepeopletheevaluatormetduringthetypicalonetotwodayvisittoacommunity.
Iftheevaluatorisabletomakeseveralvisitstothecommunityoveraperiodoftime,itisoftenusefultodevelopanetworkofinformantswhocanbevisitedperiodicallytoreportontheactualprogressoftheprojectontheground,andwhatpeoplearesayingaboutit.Forexample,peopleoftenhavedifferentperceptionsandexpectationsthatwhattheprojectbelievestheyareexpecting.
Participatoryplanningandgroupconsultationtechniques
Participatorymethodscanbeusedbothtoinvolvecommunitiesintheplanning,design,analysisanddisseminationofsurveysorasaparticipatoryevaluationtooltoobtaincommunityperspectivesonpriorityissues,thesocialstructureofthecommunityandthechangingcontributionmadebytheprojectovertime.
TheWorldBankSocialObservatoryprojectinIndiahasmadeextensiveuseofparticipatorytechniquesintheirresearchprojects.Forexample,villagewomendesignedasurveyinstrumenttoidentifyfamilyneedsinpoorcommunitiesinIndia.ThesurveywasthenadministeredtoalmostonemillionpeopleandtheresultswereanalyzedincooperationbetweenthewomenandtheSocialObservatoryteam.Thewomenthendesignedandimplementedparticipatorydisseminationstrategies,usingpictures,marchesandtalks(WorldBank,SocialObservatoryblog).
Withrespecttoparticipatorygroupconsultationmethodsforevaluation,thereisawidearsenaloftoolsandtechniquesthatbaseplanningandevaluationstudiesonfeedbackobtainedthroughgroup
ECGReferenceDocument–IntegratingGenderintoProject-levelEvaluation–Appendixes
16
16
consultations.Participatoryruralappraisal(PRA)isonewidelyusedsetofapproaches(LykesandHerschberg,2012;Kumar,2002).Thesetechniqueselicitperceptionsofthecommunityanditssurroundings,powerstructures,constraintsonwomenandothervulnerablegroups,andahistoricalperspectiveontheevolutionofthecommunity.Whiletherearerisksofbias,ofcooptionoftheprocessbyasmallgroupofmorepowerfulpeople,orinterpretationsimposed(intentionallyorinadvertently)bytheresearchers,theseapproachesareaveryvaluabletool.However,theyrequireconsiderabletimefortheirproperuseinordertopreparethegroup,gainanunderstandingofthecontext,conductthemeetings(whichcanoftentakethreetofives)andworkwiththecommunitytointerpretthefindings.Thesetechniqueshavebeenusedextensivelyingenderresearch(Cornwall,2003,2008;Kumar,2002).
Story-tellingandsense-making
Inrecentyearstherehasbeenanincreasedinterestinstory-telling.Oftenbeneficiariesareaskedtotellastoryaboutsignificantrecenteventsinthecommunityorwhattheyhaveexperiencedorheardabouttheproject.Oftenthestoriesareanalyzedusing“Sense-makingsoftware”.Thisapproachcanprovideadifferentperspectivebycapturingandanalyzingtheperspectiveofthecommunityratherthanbyaskingthemtorespondtoquestionsdevelopedbytheevaluator.DevaultandGross(2012)provideanoverviewoffeministapproachestoqualitativeinterviewing,listeningandstory-telling.
Anotherapproachtostory-tellingistomakeaudioorvideorecordingswhichcanthenbeanalyzedusingsomeofthenewdataanalyticssoftware.
Case-basedmethods.
Casestudieshavealwaysplayedanimportantroleinprogramevaluation.Theyprovidevaluablewaystoexplainquantitativefindings(particularlyunexpectedfindings),andtoprovidein-depthinsightsintolivedexperiences,projectimplementationprocessesandbehavioralchange.Anumberofdifferentcasestudyapproaches,allofwhichareusefulforGREcanbeidentified:
a. Exploratoryanddescriptivecasestudies.b. Illustrativecasestudies:Theseareoftenusedasafollow-uptoaquantitativesurvey.For
example,surveyanalysismaycreateatypologyofoutcomesandcasescanbepreparedtoillustrateorexplaineachtype.
c. Analytical:thepastdecadehasseenanincreasinguseofQualitativeComparativeAnalysis(QCA).Theunitofanalysisisacasewhichcanbeaslargeasacountryorassmallasanindividual.Casescanalsobehouseholdsororganizations.Thismethodidentifiestheconfigurationoffactors(forexample,householdattributesorcommunitycharacteristics,suchasaccesstoinfrastructure)foreachcase,thatarepresentwhentheoutcomeispresent.QCAisconsideredausefulwaytoprovideanapproximateestimateofcausalitywhenexperimentaldesignsarenotpossible.QCAalsohastheadvantagethatitcanbeusedwithsmallsamples(50orlesscases),anditisalsousefulfortheanalysisofcomplexprogramsasitidentifiesconfigurations(combinations)offactorsthatareassociatedwithanoutcome,ratherthananalyzingasinglefactor(asisthecasewithexperimentaldesigns).ForanoverviewofQCAmethodologiesseeByrneandRagin(eds)2009.UNWomen(2014)illustrateshowQCAwasusedtoevaluatetheimpactsofUNWomen’scountrystrategiesforpromotingwomen’empowerment.
ExperimentalandQuasi-experimentaldesigns.ItisnormallynotpossibletoconductanexperimentaldesignforIEOevaluationsastheyareconductedretrospectively.Yetitmayoccasionallybepossibletousethefindingsofexperimentaldesignsconductedbyotherpartsoftheorganization.Forexample,theWorldBank’sAfricaGenderInnovationLabconductsRCTsandquasi-experimentaldesignstotestinnovativeapproachesforthedesignandimplementationofgenderprojects(AfricaGenderInnovationLab,2016).
However,itmaybepossibletouseaquasi-experimentaldesign(QED).WhilethesedonothavethestatisticalrigorofRCTswithrespecttointernaldesignvalidity,QEDsoftenprovideuseful
ECGReferenceDocument–IntegratingGenderintoProject-levelEvaluation–Appendixes
17
17
approximationsforcausalanalysis,andsomewouldarguethatawell-designedQEDmaybeabletoaddressexternalvalidityissuesbetterthananRCT.ThefollowingareexamplesofQEDsthatcouldbeconsideredforGREwhenresourcesandtimepermit:
a. Post-testcomparisondesign.Forexample,manyevaluationsofthegenderimpactofmicrocreditprogramsusecross-sectionalstudieswherewomenwhousedavillagebankarematchedwiththosewhodidnot,wherepossibleusingpropensityscorematching.Thechallengewiththesedesignsishowtoaddressinitialdifferencesbetweenthetwogroups(beforethevillagebankbegantooperate)thatmightexplainsomeofthedifferencesinoutcomeindicators(expenditureonfood,educationandotherhouseholdessentials};investmentinhousing;profitsgeneratedbysmallbusinesses;women’sroleinhouseholddecision-making.Forexample,thewomenwhotookoutloansmighthavehadmoresmallbusinessexperience,ortheymightcomefromfamiliesthatweremoresupportiveofwomenrunningasmallbusiness.Post-testevaluationsfinditdifficulttocontrolfortheseinitialdifferences.However,ifamixedmethodsdesignisusedtherearemanyqualitativetechniquesthatcanexplorepossibleinitialdifferencesandhowtheycouldaffectoutcomes.
b. Naturalexperimentsandpipelinedesigns.Sometimesaprojectorpolicyisintendedtoreachallofthepopulationofadistrict,provinceorthewholecountry.However,duetoadministrativeproblems,budgetcutsorproblemssuchasflooding,somesectorsofthetargetpopulationmaynotbereachedormaybesubjecttolongdelays.Inthesecases,theoutcomesforthegroupsthatreceivedtheintendedservicescanbecomparedwiththosethatdidnot,thusprovidinganapproximateestimateofprojectoutcomes.Whilethisapproachisuseful,andquitewidelyused,itmustbeinterpretedwithcareastheremaybesystematicdifferencesbetweenbeneficiariesandnon-beneficiaries.Forexample,administrativeproblemsmaybemorelikelytoaffectpoorerormoreremoteareas.
Asimilarlogiccanbeusedincaseswhereprojectsareimplementedinphasesoveraperiodoftime.Anexamplewouldbetheinstallationofwaterandsanitationinalargeurbanslum.Theprojectwillusuallybeimplementedinphases(startingatoneendofthecommunityandmovingtotheother)overaperiodofyears.Anotherexampleistheconstructionofaroadwhichagaincantakeseveralyears.ThesectorsofthecommunitythatwillnotreceivethewaterandsanitationuntilYear2or3canbeusedasacomparisongroupforthePhase1areasthatwillreceivetheservicesinYear1.Assumingthatthecharacteristicsofhouseholdsineachphaseoftheprojectaresimilar(whichisnotalwaysthecase),thenabaselinesurveycanbeconductedinbothareasatthestartofYear1andthenrepeatedattheendofYear1.Comparingchangesinthetwogroupscanprovideanapproximateestimateofprojectimpacts.Theanalysiscanberefinedtocomparethechangesforwomenandmen.
c. Reconstructingbaselinedata.Asdiscussedearlier,itmaybepossibletofindbaselinedatasothatapre-test:post-testcomparisondesigncanbeused.
Bamberger,RughandMabry(2012)Chapter11andAppendixFreview,withexamples,allofthemostcommonexperimentalandquasi-experimentaldesigns.
Systemsandcomplexityscience-basedapproaches(WilliamsandHummelbrunner,2011,reviewthemainsystemsanalysisapproaches).ThereareanumberofnewevaluationapproachesdevelopedbycomplexitysciencethatcouldbeappliedinGRE.Alloftheseidentifythemainstakeholderoractors,thelinkagesbetweenthem,howinformationflows,leadershippatternsandhowdecisionsaremade.Systemsmapsandmodelsidentifythelinkagesbetweenorganizationsthatfacilitateorconstraindesiredprocessesofchange.Whilesomeoftheapproaches,suchassociometricanalysisrequirethecollectionofconsiderableamountsofsurveydata,othertechniquesarevisualizationtoolstohelpconceptualizethenatureofthesystemwithinwhichaparticularprogramoperates.Manyoftheapproachescanbelinkedtoatheoryofchange.Systemsapproachesareparticularlyvaluableforgender
ECGReferenceDocument–IntegratingGenderintoProject-levelEvaluation–Appendixes
18
18
analysisastheyhelpunderstandthecomplexityofthesocialsystemwithinwhichaprojectinterventionisseekingtopromotesocialchange.Theseareusuallytoolstomakeclients(aswellasevaluatorsthemselves)awarethatoutcomesareinfluencedbymanymorefactorsthanareusuallytakenintoconsiderationinprogramdesignandevaluation.Thefollowingaresomeofthemostwidelyusedapproaches,allofwhicharedescribedwithexamplesinWilliamsandHummelbrunner(2011).
a. Systemsmapping:Thiscoversanumberofapproachesthatprovideavisualrepresentationofasystem.Systemsmappinghelpsidentifythedifferentpartsofasystemandthelinkagesbetweenthedifferentpartsthatarelikelytochange.Systemsmapscanalsoidentifypositiveandnegativelinkagesbetweendifferentpartsofthesystemandthestrengthofthelinkages.Theapproachiscloselylinkedtoatheoryofchangeandprovidesatoolforexamininginmoredetailthedifferentpartsofthesystemthataffectedtheintendedprocessesofchange.Asystemsmapcouldalsobeusedtomodelalloftheinstitutionsandprocessesthatcomprisethesystemofsocialcontrolthatconstrainsprocessesofwomen’sempowerment.
b. Socialnetworkanalysis.Theapproachisusefulformodellingstakeholderrelationshipsanddescribinghowinformationflowsthroughthesystemanddecisionsaremade.Theapproachisusuallybasedonsurveysorobservationsthatcalculateinteractionsbetweendifferentindividualsorgroupsandthestructureofpoweranddecision-makingwithinagrouporamongdifferentgroups.
c. Systemdynamics:Thisdealwithinterconnectednessanddynamicrelationshipsamongdifferentpartsofaprogramsystem.Theapproachdrawsonengineeringandmanagementandinvolvesdevelopingafiguretorepresentstocksandflowvariables.Theapproachcanbeusedinevaluationtocapturehowcomplexsystemsareaffectedby,andrespondto,developmentinterventions.Forexample,WilliamsandHummelbrunner(2011)usedSystemsDynamicstoevaluatetheeffectsofmicroloanprogramstargetingsexworkersinWestAfricaaspartofalargerprogramtocombatHIV/AIDS.Theanalysiswasabletoidentifyandhelpexplainviolentswingsinthepopularityofthemicroloanprogramwithsexworkers.
d. Criticalsystemsheuristics:Thisapproachfocusesonunderstandingthefactorsthatdeterminewhatisconsideredtoliewithinthesystembeingstudiedthatiswhatgetsevaluated?Thisisprincipallyanethicaldecisionbasedonavaluejudgment.ThisisimportantforGREbecauseinmanycasesgenderissuesarenotconsideredtoliewithinthe(project)systemsogenderisnotincludedintheevaluation.
Conceptmapping(KaneandTrochim,2007).Conceptmappingisatechniquethatusesinterviewswithstakeholdersorexpertstoobtainanapproximateestimateofprogrameffectiveness,outcomesorimpacts.Agroupofexpertsorstakeholdersareaskedtolistthecharacteristicsofasuccessfulprogram(e.g.topromotewomen’sempowerment).Thelisteditemsaresortedintogroups(manuallyorbycomputer)andtheseareorganizedintoasetofdimensions.Thesameordifferentgroupisthenaskedtorate(usuallyona1-5scale)actualprogramsonthesedimensions.Theratingscaneitherbeproducedatonepointintime(aswouldusuallybethecaseforanIEOevaluation,ortheratingscanbeproducedatthestartandendofaprojecttomeasurechange.Conceptmappinghasseveraladvantages.First,thedimensionsonwhichaprogramisevaluatedaredefinedbystakeholders(orexperts)andnotbytheevaluationteam.Thisisparticularlyimportantforevaluatingmulti-dimensional,anddifficulttodefine,conceptssuchasempowerment.Second,softwareisavailabletoconducttheconceptdevelopmentandevaluationonlinesoitcanbemuchmoreeconomicalandcaninvolveawiderrangeofstakeholdersorraters.Theprocesscanalsobecarriedoutrelativelyquickly.UsingnewinformationtechnologyforGRE(Bamberger,2017).Thepastfewyearshaveseenarapiddevelopmentofexcitingnewsourcesfordatacollectionandanalysisthatareopeningupnewapproachestoprogramevaluationthatitwouldhavebeendifficulttohaveimaginedeventenyearsago.ThesearebasedonICTs(smartphones,internetandotherportabledevicesthatcanbeusedinthefieldandbyeventhepooresthouseholdslivinginremoteregions),andonbigdata(generatedfromsatellite
ECGReferenceDocument–IntegratingGenderintoProject-levelEvaluation–Appendixes
19
19
images,fromtwitterandothersocialmedia,electronictransfers,remotesensorsandtheinternetofthings).Thesearecomplementedbypowerfulnewdataanalyticstoolsthatcananalyzevastquantitiesofdatafarbeyondthecapacityofofficecomputers.Mostoftheseinvolvenewkindsofdata,muchofitcollectedforacompletelydifferentpurpose(suchasanATMtransactionorasocialmediadiscussion),aredistinctfromthekindsofinformationcurrentlyusebyevaluators.Infact,evaluatorshavebeenmuchslowertoadoptnewinformationtechnologiesthanmarketresearchers,medicalresearchers,programplanners,socialmarketers,andagenciesworkinginemergencyrelief.
Despiteevaluatorshavingbeenmuchslowerontheuptake,therearenowexamplesfromallsectorsandregionsontheimmensepotentialofnewinformationtechnologyfordevelopmentevaluation.ManyofthenewtechnologiesareofparticularinteresttoIEOs,includingforGREs,astheycanovercomemanyoftheconstraintsdiscussedearlier.Someofthepotentialapplicationsandadvantagesinclude:
a. Muchoftheinformationisveryfastandcheaptocollectandanalyzeasitisderivedfromexistingdatasourcesandtheevaluationdoesnothavetopayforcollection.
b. Economicalaccessmeansthatevaluationsnolongerhavetorelyonrelativelysmallsamples,butdatacanoftencoverthetotalpopulation.Thismakesitpossibletoconductkindsofdatadisaggregationthatwerenotpreviouslypossible,asthesmallsamplesizedidnotprovidesufficientnumbersfordisaggregateddataanalysis.
c. Easeofdataaccessalsomakesitpossibletoincorporatemanycontextualfactorswhichwerepreviouslyinaccessibleortoextensivetouse.Thispermitsplacingprogramsintheirbroadercontexts,whichisveryimportantforunderstandingthemultiplefactorsthataffectorconstrainsocialchange.
d. Longitudinaldatasetsarestartingtobecomeavailablethatofferthepotentialtoovercomethemajorconstraintonretrospectiveevaluations,namelythelackofbaselinedata.Examplesoflongitudinaldatasetsinclude:ananalysisofchangingattitudesandinformationonmajorsocialissuesasreflectedontwitter(whichnowcoveranumberofyears);datastreamsfromsatelliteimagesthatcapturepopulationmovements,indicatorsofpovertyandeconomicgrowth(suchasvehiculartraffic,typesofhouseconstruction,nocturnallightemissionsinpoorcommunities,areasundercultivation).
e. Dataanalyticsmakesitpossibletoconstructanintegrateddataplatformthatbringstogethermanydifferentsourcesofdatausingacommonmetric.Thismakesitpossibletoidentifypatternsofrelationshipsbetweendifferentdatasetsthatwerepreviouslydifficulttodetect.
f. Real-timedatafeedbackmakesitpossibletodetectchangesandtrendseveninpost-projectdata
Portfolioanalysis.PortfolioanalysiswasdescribedinSection2.4B.Dataiscollectedfromthefollowingsources:
• Policy,planningandprojectdocumentsfromcountryprograms.Documentsarereviewedtodeterminewhetherandhowgenderissuesareaddressed.Sometimestheassessmentsimplyindicateswhetherthereisareferencetogender,butinothercasesthereisaratingofwhethergenderwasacentralpriorityorhowthoroughlyitwasaddressed.
• Keyinformantinterviewsmaybeconductedbyphone,e-mailorinpersontoobtainopinionsonhowgenderwasapproachedindifferentpartsoftheprogram.
• Focusgroupinterviewsmaybeusedinasmallnumberofcountriestodigdeeper.• Afewcountriesmaybeselectedtoconductfieldstudiesthatmayincludeprojectsurveys,site
visitsandinterviewswithdifferentstakeholdergroups.Asmallnumberofrepresentativeprojectsmayalsobeselectedformorein-depthanalysis.Forexample,theWorldBankassessmentoftheimplementationoftheirglobalgenderpolicy(WorldBank,2009)beganwiththeassessmentofhowwellgenderwasincorporatedintopoliciesandprojectsin93countrieswheretheBankhadactiveprograms.Ratingscaleswereusedand1,153projectswereanalyzed.Basedontheseratings,12countrieswereselectedformorein-depthanalysis(e.g.stakeholder
ECGReferenceDocument–IntegratingGenderintoProject-levelEvaluation–Appendixes
20
20
interviews),ofwhichthreewereselectedforfocusgroupinterviewsandtwoforintensivecountrystudies.
Genderflags.Manyagencieshavedevelopedchecklistsforassessingwhetherandhowwellgenderissueshavebeenaddressedincountryprogramframeworks,sectorprogramsorprojectsix.Theindicatorseitherusea“Yes/No”formatorratehowwelltheissuewasaddressed.Box2.3inChapter2illustratesthequestionsincludedintheWorldBankIEGGenderFlagcountryevaluationtemplate.Checklistscaneitherbeusedforself-assessmentbyoperationsstafforforexternalassessmentbyIEOsorconsultants.
Datacollectionforcasestudies.DescriptivecasestudiesmainlyuseQUALmethodstodescribethecontextinwhichtheprojectoperates.Illustrativecasestudies,ontheotherhandnormallyusedmixedmethodsdatacollectionassurveysareanalyzedtoidentifyatypologyofsubgroupsforexample,farmers,villagebankmembers,mothersusingachildcareclinic.CasesarethenselectedwithineachtypologyandarestudiedusingQUALmethods.
QualitativeComparativeAnalysis(QCA)casestudiesuseaspecializeddatacollectionmethod(Byrne,2009;ByrneandRagin,2009).Amatrixisconstructedwhereeachcaseisarowandeachcolumnrepresentseitheranattributeofthecase,orofthecontextinwhichitislocated,ortheoutcomevariable.InthesimplestformofQCA,allattributesandoutcomesaredichotomized.Forexample,QCAmightbeusedtoassesshowgirlsenrolmentinsecondaryschool(theoutcome)isaffectedby:(i)whetherthemotherhascompletedsecondaryschool;(ii)whetherthegirlliveslessthan1,000metresfromtheschool;and(iii)whethertheschoolhassatisfactorytoilets(asdefinedbytheMinistryofEducation)forgirls.Thecolumnswouldbedefinedasfollows:
• Column1:Casenumber.• Column2:Motherhascompletedsecondaryschool[Yes=1,No=0].• Column3:Girlliveslessthan1,000metersfromtheschool[Yes=1,No=0].• Column4:Theschoolhassatisfactory(asdefinedbytheMinistryofEducation)toiletforgirls
[Yes=1,No=0].• Column5[outcomevariable]:Thegirlisenrolledinsecondaryschool[Yes=1,No=0].
Thedatamaybecollectedinaspecialsurveyoritmaybeavailablefromprevioussurveysorrecords.Oftenthevariablesareselectedtoreducethetimeandcostofdatacollection.
ECGReferenceDocument–IntegratingGenderintoProject-levelEvaluation–Appendixes
21
21
Appendix 5: A data collection planning matrix for a GRE: Application to a hypothetical Village Development Project in Central Asia Note:Thisisahypotheticalexampletoillustratetheuseofthedatacollectionplanningmatrix.Thisexampleonlyillustratesafewofthepossibleindicatorsanddatacollectionmethodsthatcouldbeconsidered.ThismatrixcouldbeusedinthedesignofallthreecasestudiesdiscussedintheReferenceDocument.
• • Evaluationdesign • Indicators • Datasources• Genderquestion1:Didtheprojectaddresswomen’sneeds?• 1.Didtheprojectstaffbelieve
thatwomen’sneedswereaddressed?
• a.Deskreviewcombinedwithquantitativeorqualitativesurveys.
• b.Ifpossiblebaselinedatawillbereconstructedtopermitapre-test:post-testdesign.
• a.Didtheprojectstaffbelievewomen’sneedswereaddressed?
• b.Whichneedswereaddressed?
• Addressingbothquestions• i.Interviewswithprojectstaff.• ii.Monitoringandotherproject
reports.• 2.Didwomenbelievethat
theirneedswereaddressed?• a.Theproportionofwomenwhosay
projects:• Respondeddirectlytotheir
needs.• Respondedsomewhattotheir
needs.• Didnotrespondtotheirneeds.
• i.Samplesurveywithwomen.• ii.Focusgroups.
• Notesonthefeasibilityoftheproposeddatacollectionmethods:• 1.Reviewmonitoringandotherprojectreportstocheckthekindsofinformationincludedonprojectsperceivedtosatisfywomen’sneeds.• 2.Aretheoriginalprojectstaffstillavailabletobeinterviewed?• 3.Willtimeandresourcepermittheapplicationofasurvey?• 4.Whathasbeentheexperienceinterviewingwomen?Isthisfeasible?Dosurveysorfocusgroupstendtoworkbetterwithwomen?• Genderquestion2:Howdidtheprojectaffectwomen’sempowerment?• 1.Participationinproject-
relatedcommunityorganizations.
• a.Deskreviewscombinedwithquantitativeorqualitativesurveys.
• b.Reconstructingbaselinedataifpossible.
a.Numberofwomenandmenparticipatingindifferentprojectorganizationsforeachyearoftheproject.b.Regularityofattendance.
• a-i.Projectmonitoringreports.• a-ii.Interviewswithproject
staff.• a-iii.Projectprofiles.• • b-i.Monitoringreports.
ECGReferenceDocument–IntegratingGenderintoProject-levelEvaluation–Appendixes
22
22
• b-ii.Attendancerecords• b-iii.Visitstomeetingstocheck
records• 2.Participationindecision-
makingincommunityorganizations.
• a.Numberofwomenandmenonleadershipcommitteesinprojectorganizations.
• b.Levelofparticipationofwomencomparedtomenmeasuredby:
i. Frequencyofspeaking.ii. Numberofwomen’sideas
approved.
• a-i.Projectmonitoringreports.• b-i.Observationchecklistsused
incommitteemeetings.• b-ii.Audioandvideorecordings
(ifpermitted).
• 3.Impactsonwomen’sstatusinthehousehold• a.Mobility. • • Placesvisited:(adaptlisttocontext):
[1pointforeach]• a.Market.• b.Clinic.• c.Movies.• d.Outsidethevillage.
• Coveringallitems:• iRapidsamplesurveywith
women.• ii.Keyinformants.• iii.Focusgroups.
• b.Abilitytomakesmallpurchases.
• Thingspurchased(adaptlist)[1pointforeach]
• a.Itemsfordailyuse.• b.Itemsforself.• c.Treatsforchildren.
• Coveringallitems.• i.Observationduringvisitsto
families.• ii.Rapidsamplesurveywith
women.• iii.Requestingagroupof
womentocompleteanexpenditurediary.
• c.Involvementinmajorhouseholddecisions.
• (Adaptlist)[pointstobedecided].Maygivedifferentscoresfordecisionsmadeonownormadewithhusband.
• a.Purchaseofgoatorsmallanimal• b.Householdrepair.• c.Leasingland.• d.Purchaseoflandorequipmentfor
farmorbusiness.
•
• Notesonthefeasibilityofdatacollectionmethods:• 1-4Asforquestion1.• 5.Isthereanyexperiencewiththeapplicationofmultiplechoicequestions?Howwelldotheywork?
ECGReferenceDocument–IntegratingGenderintoProject-levelEvaluation–Appendixes
23
23
Appendix 6: Example of a GRE design matrix: Evaluating a hypothetical village development project with defined gender objectives in Central Asia.
1.Theproject’sgenderobjectivesItwillbenecessarytoconfirmwhatthestatedgenderobjectivesoftheprojectare,andwhetherthereareadditionalgenderobjectivesthatareimplicitbutnotspecificallystatedwhichcouldbeincludedintheassessment.BoxA6-1identifiesthegenderobjectivesthatmightbeincluded.Intheboxthesearedividedintooutputsandoutcomes(followingthecategoriesusedinmostTOCs)butthedistinctionbetweenthetwocategoriesisnotveryclear.Forexample,participationinagroupisconsideredanoutput,whereasassumingapositionofleadershipisconsideredanoutcome.Theprimaryoutcomesareconsideredastheproject’sdefinedgenderobjectives.Whilesomeoftheseareclearlystatedintheprojectdocument,othersareinferredaspossiblegenderoutcomesthatmightbeproducedandwhichgobeyondthedefinedprojectobjectives.Thesecouldbeclassifiedasprimary(defined)genderobjectivesandsecondaryoutcomes(otherpotentialeffectsonwomenintheproject)andtertiarygendereffects(onwomeninothercommunities).So,inthedesignoftheprojectevaluationitwillbeimportanttomakethisdistinctionandalsotoagreewithmanagementwhichsecondaryoutcomesshouldbeassessed.Itwillbeimportantforpoliticalreasonstoclarifythatmanagersarenotbeingassessedonwhethersecondaryandtertiarygenderoutcomesareachieved,butthatthesearebeingreviewedtobetterunderstandthebroaderpotentialoutcomesthatfutureCommunityDrivenDevelopmentprojectscan(andcannot)beexpectedtoachieve.
ECGReferenceDocument–IntegratingGenderintoProject-levelEvaluation–Appendixes
24
24
3.Theproject’sgenderresponsiveinterventionsTheprojecthasanumberofcomponents/interventionsthatarespecificallyintendedtopromotetheequalparticipationofwomenandtopromotetheireconomicorsocialwell-being,andanumberofothersthathavethepotentialtoadvancethestatusofwomen(seeBoxA6-2).
BoxA6-1:Possiblegenderobjectivestobeincludedintheassessment
Outputs
a. Women’sactiveparticipationinthedifferentprojectcommittees.b. Women’sactiveparticipationintheprivategroupenterprises.c. Women’sequalaccesstoprojectservicesandbenefits.
Outcomes
Primarygenderoutcomes(definedintheprojectdesign)
a. Organizationandleadership• Womenassumeleadershippositionsindifferentcommitteesandgroups.• Involvementindecisionsontheselectionofprojects.• Infrastructureprojectsareselectedthatdirectlybenefitwomen.
b. Accessto,andcontrolofeconomicandproductiveresources• Involvementindecisionsoncontrolofprojectresources.
c. Equalaccessto,andcontrolof,resourcesforprivategroupenterprises.Secondaryoutcomes(effectsonwomenintheprojectthatarenotidentifiedintheprojectdesign)andtertiaryoutcomes(effectsonwomeninothercommunities)
a. Women’spositionstrengthenedwithinthehousehold• Women’srolestrengthenedincontrolofhouseholdresources.• Womenenjoygreaterindependence(forexamplegeographicalmobility).
b. Economic• Increasedemploymentopportunities.• Increasedincomeandearningsopportunities.
c. Social• Increasedaccesstoeducationforwomenandgirls.• Greaterpersonalsecurity(forexample,reduceddomesticviolenceandsexual
harassmentoutsidethehome).
ECGReferenceDocument–IntegratingGenderintoProject-levelEvaluation–Appendixes
25
25
4.DefinitionofthegenderquestionstobeaddressedBoxA6-3identifiesaninitiallistofgender-relatedquestionsrelatingtofivedimensions:
a. Historyandcontextandhowtheseaffecttheimplementationandlikelygenderoutcomesoftheproject.
b. Relevanceoftheprojectdesigntotheachievementofimportantgenderobjectives.c. Efficiencyofprojectimplementationwithrespecttotheachievementofgenderobjectives.d. Achievementofgenderobjectives(efficacy).Objectivesaredefinedintermsofoutputsand
outcomes.e. Sustainabilityandresilience.WhiletheAppraisalReportdiscussessustainability,itdoesnot
discussresilience–whichisaconceptthathasbeenintroducedsincethisprojectwaslaunched,butwhichisnowfrequentlydiscussedtogetherwithsustainability.
BoxA6-2:Projectcomponentsandinterventionsdesignedtopromotegenderequality,andtheeconomicandsocialwell-beingofwomen
1. Socialinclusionstrategya. Socialmobilization:
i. Womenaredefinedasoneoftheprioritygroupstoreceivetrainingandotherkindsofsupporttoensuretheirparticipationinprojectactivities.
ii. Annualmeetingswillbeheldwithallpartneragenciestoreviewprogressinachievingtheparticipationofwomenandothertargetgroups.
iii. Effortswillbemadetorecruitwomenassocialmobilizers.b. Villageprofiles:
i. Willfocusonissuesofexclusion,includingexclusionofwomenandwillexaminesocial,political,legal,economicandotherfactorsthatcauseorsustainexclusion.
ii. Asocialcapitalanalysiswillexaminedifferencesinsocialcapitalforwomenandmen.
c. Villagemeetings:i. Itisrequiredthatatleasttwoofthesixcommitteemembersshouldbe
women.ii. Aseriesofstructuredmeetingsareplannedandtheparticipationof
womenandmencanbemonitored.d. Preparationofgroupsandstatementsofneeds:effortswillbemadetoensurethat
womenmemberswillhavetheopportunitytostatetheirneeds.2. Identificationofprioritysocialinfrastructure
a. Itwillbeimportanttomonitor/assesswhetherwomen’sprioritiesarereflectedintheselectionofinfrastructure[note:thePADdoesnotstatewhetherthisisapriorityandhowitwillbeachieved].
ECGReferenceDocument–IntegratingGenderintoProject-levelEvaluation–Appendixes
26
26
5.Initialproposalfortheevaluationmethodology5.1. TableA6-1presentsinitialideasfortheevaluationdesign.Thisonlyprovidesaframeworkthat
willneedtobeelaboratedonceagreementhasbeenreachonthegeneralapproach.Thetableincludesthreecolumns:
• ThequestionscoveringeachofthefivedimensionsoftheevaluationthatarelistedinBox4.• Theindicatorsusedtoaddresseachquestion.• Thedatacollectionmethods.
BoxA6-3:Gender-relatedquestionstobeaddressed
1.Historyandcontext
a. Howwaswomen’sparticipationandaccesstobenefitsaffectedbythetransitionfromtheSovietera?
b. Howdidhighunemploymentaffectwomen’sparticipationandaccesstoprogrambenefits?2.Relevance
a. Howdoestheprogramexpecttostrengthenwomen’sempowermentandensurewomenhaveaccesstoprogrambenefits?
b. Aretheinterventionsrelevanttotheachievementoftheseobjectives?c. Howrelevantwereinclusion,socialmobilizationandprovisionofinfrastructureto
achievingobjectives?d. Howrelevantwassmallscaleprivategroupenterprises?e. Howrelevantarethegenderobjectivesandtheirimplementationsstrategiestothe
achievementofoverallprojectobjectives?3.Efficiency[notallagenciesincludeefficiencyintheevaluation]
a. Werewomenconsultedandinvolvedinprojectdesign?b. Didtheselectedprojectsreflecttheprioritiesofwomen?c. Wereprojectimplementationstrategiesconducivetotheparticipationofwomen?d. Howdidwomen’sparticipationindesignandimplementationaffectoverallproject
outcomes?e. Theeffectivenessofthesocialmobilizationstrategy.
4.Achievementofgenderobjectives(efficacy)a. Outputs
i. Women’sparticipationinprojectleadershipandmanagementii. Women’sroleindecision-making
b. PrimaryOutcomesi. Projecteffectsonwomen’sempowerment.ii. Assessingsocialandeconomicoutcomesforwomen.iii. Didprojectsrespondtowomen’sneedsandpriorities?
c.Secondaryandtertiaryoutcomesi. Weretherepositiveand/ornegativeunintendedprojectoutcomesforwomen?ii. Whateffectsdidtheprojecthaveonwomen’empowerment?
5.Sustainabilityandresiliencea. Whatevidenceistherethatthedifferentgenderoutcomeswillbesustainable?b. Doindividualwomen,andwomen’sgroupsdevelopstrategiestoadapttostressandshocks
anddotheylearnfromexperienceandimprovehowtheyadapt?
ECGReferenceDocument–IntegratingGenderintoProject-levelEvaluation–Appendixes
27
27
Theframeworkidentifiesthefollowingpossibledatacollectionmethods.Theapplicationofeachofthesedatacollectiontoolswillbeexplainedoncethereisaninitialagreementontherangeoffeasibleandappropriatemethodsforthisevaluation.Itisrecognizedthatnotallofthelistedmethodswillbeusedinthisevaluation.Theinitiallistofdatacollectionandanalyticalmethodsincludes:1. ConstructingagenderTOCthatcanhelpidentifykeyevaluationquestions,indicatorsandthe
pathwaysthroughwhichitisexpectedthatgenderoutcomeswillbeachieved.2. Constructingadatacollectionplanningmatrix.Appendix5presentsanexampleofadatacollection
planningmatrixthatcouldbeusedinthedesignofthepresentevaluation.Thematrixlistsallofthekeygender(andother)questions,theproposedevaluationdesigns,theindicatorsandthedatacollectionmethods.Importantly,italsoassessesthefeasibilityofcollectingtheproposedinformationwithinthebudget,time,methodologicalandorganizationalconstraintswithinwhichtheevaluationwillbeconducted.
3. Secondarysources:a. TheAppraisalReport.b. TheCompletionReport.c. Projectmonitoringreports.d. Projectprofiles.e. Governmentreports.f. Partnerreports.g. Civilsocietyreports.
4. Consultations:a. Expertandkeyinformants.b. Focusgroups.
5. Surveysandquestionnaires:a. Rapid,shortsamplesurvey.b. Ratingscalestobecompletedbybeneficiaries,projectstafforpartners.
6. Casestudiesa. Descriptivecasestudies.b. Analyticalcasestudies(QCA).
7. Qualitativefieldwork:a. Projectvisitsb. Informalinterviews.c. In-depthinterviews.d. Observation.
8. Socialmediaanalysis(ifthisisfeasible):a. Socialmediaanalytics(forexample,analysisoftwitterandothersocialmedia).
9. Pipelinedesignstoconstructacounterfactual.5.2.Evaluatingprojectoutcomespromotingwomen’sempowerment.Theprojectreferstosocialmobilizationasamechanismforstrengtheningtheparticipationofwomenincommunityorganizationspromotedbytheprojectandinthemanagementoftheseorganizations.Italsopromotestheparticipationofwomenintheprivategroupenterprises.Althoughtheprojectdocumentsdonotreferdirectlytogenderempowerment,theseobjectivescouldbeconsideredimportantelementsinwhatmanyagencieswouldcalltheeconomicandsocialempowermentofwomen.Chapter8oftheReferenceDocumentdiscusseshowtheseobjectivescouldbeincorporatedintoagenderempowermentframeworkandillustrateshowanempowermentframeworkcouldbedevelopmenttoassesstheeffectsoftheprojectonpromotingthedifferentdimensionsofwomen’sempowerment.
ECGReferenceDocument–IntegratingGenderintoProject-levelEvaluation–Appendixes
28
28
• TableA6-1:Keygender-responsivequestions,indicatorsanddatacollectionmethods:
PARTI:SOMEGENERALAPPROACHESTHATCANBEUSEDTHRUGHOUTTHEEVALUATION
1.Gendertheoryofchange:Developingagendertheoryofchangethatisusedtoidentifythekeyevaluationquestionsandtodefinetheprocessesthroughwhichoutputsandoutcomesaretobeachieved.Thisalsodefineskeyassumptionstobetested.
• 2.Resultsframework(ifithasbeenusedintheproject):Thisdefinestheintendedgenderobjectivestobeassessed.
• 3.Descriptivecasestudies:Usuallyarelativelysmallnumberofcasestudiesthatarebroadlyrepresentativeofthemainprojectscenariosandwhichareusedtoillustratehowtheprojectsevolvedandthelivedexperienceoftheprojectpopulations.Casestudiescanbelongitudinal(conductedoverarelativelylongperiodoftime),or(asisusuallythecasewithex-postevaluations)conductedatonepointintime,andrelyingextensivelyonrecall.
• 4.Analyticalcasestudies:TheseusetechniquessuchasQCA(qualitativecaseanalysis)toidentifythenecessaryandsufficientconditionsforprojectoutcomestooccur,andalsothenecessaryandsufficientconditionswhereprojectoutcomesdoNOToccur.Normallyatleast30casesarerequired.WhileQCAarerarely(ifever)usedbyIEG,theycouldbeapotentiallypowerfulanalyticaltoolthatpermitsattributionanalysistobeused.
• 5.Counterfactualanalysis:Thiscomparesprovincesorregionswheretheprojecthasbeenimplementedwithregionswheretheprojecthasyetbeenimplemented.Manyprojectsarerolled-outinphasessothatapipelinedesigncanbeusedwhereregionswheretheprojecthasnotyetbeenimplementedarecomparedwithareaswhereithasbeenimplemented.Sometimesthedesignisbasedonplannedphasing-inofdifferentregionswhileinmanycasesitisbasedonnaturalexperimentswheredelaysarecausedbyunplannedcircumstances,suchasdelaysinfundingoradministrativeproblems.Whiletherearemethodologicallimitationsintheuseofpipelinedesigns,theyofferausefultoolforassessingattributionwhereexperimentalandquasi-experimentaldesignsarenotfeasible.
•
• PARTII:DESIGNSFORASSESSINGEACHDIMENSIONOFTHEEVALUATION
•
Dimension/Question Indicators Datacollectionmethods
• 1.HISTORYANDCONTEXT
1. Howwaswomen’sparticipationandaccesstobenefitsaffectedbythetransitionfromtheSovietera?
• Opinionsfromexpertsandkeyinformantsontheeffectsofhighunemployment.
a. Expertandkeyinformantinterviews.
2. Howdidhighunemploymentaffectwomen’seffectiveparticipation?
• Opinionsfromexpertsandkeyinformantsontheeffectsofhighunemployment.
a. ExpertandKeyinformantinterviews.b. AppraisalReportandICR.
c. • 2.RELEVANCE:Therelevanceoftheprojectconceptanddesignforpromotingwomen’sempowermentandsocialandeconomicbenefits
1. Howdoestheprogramexpectto a. DevelopaTheoryofChange(TOC)thatidentifiesthe a. ThedraftTOCwouldbedevelopedby
ECGReferenceDocument–IntegratingGenderintoProject-levelEvaluation–Appendixes
29
29
strengthenwomen’sempowermentandensurewomenhaveequalaccesstoservices?
intendedgenderoutputsandoutcomesandtheprocesses/stepsthroughwhichthesearetobeachieved.Someoftheoutputsandoutcomesmayinclude:
• Outputs.
a. Womenparticipatinginthedifferentcommittees.b. Womenparticipatingintheprivategroupenterprises.c. Women’saccesstobenefitsandservicesprovided
throughtheproject.Outcomes.
a. Womeninleadershippositionsinthedifferentgroups.b. Womenowningorgainingmanagementpositionsinthe
privategroupenterprises.c. Women’saccessto,andcontrolofeconomicand
productiveresourcesatthelevelofthehousehold,communityandenterprise.
d. Women’sandgirl’sincreasedaccesstoeducationalopportunities.
e. Women’sincreasedgeographicalmobility.
consultantsonthebasisofprojectdocumentsandinterviews.Feedbackwouldbeobtainedfromprojectstaff,beneficiariesandcivilsociety.
b. ICR.c. Interviewswithprojectstaffandpartner
implementingagencies.d. ICRAnnexes11and12.
2. Howrelevantaretheseintendedoutputsandoutcomesforwomen’sempowerment?
a. Women’sopinionsontherelevanceofeachoutputandoutcome(seepoint1above)totheirlivesandtotheirfeelingofempowerment[***Note:anappropriateandunderstandabletermfor“empowerment”mustbeidentified].
b. Theopinionofkeyinformantsandwomen’sorganizationsontherelevanceoftheoutputsandoutcomesfordifferentgroupsofwomen.
a. Reviewtheoryofchange.b. Focusgroupswithbeneficiaries,project
staffandcivilsociety.c. Anappropriatelocallyunderstood
translationof“empowerment”wouldbedevelopedthroughin-depthinterviewsandpossiblydiscussiongroupswithbeneficiaries.
d. Oneapproachusedfortheevaluationoftheimpactsofmicrocreditonwomen’sempowermentinBangladeshwastomeetwithlocalwomeningroupsandtoworkwiththemtoidentifywhatforthemwouldbekeydimensionsofempowerment(forexample,beingabletotraveloutsidethefamilycompound,tobeinvolvedindecisionsonpurchaseofschooluniforms,beingabletowalkthroughricepaddieswithoutbeing
ECGReferenceDocument–IntegratingGenderintoProject-levelEvaluation–Appendixes
30
30
requiredtostepintothewatertoallowmentopassonthenarrowpathway).Eachitemwasthenputintoascaleandwomenwereaskedtolocatetheirpresentpositiononeachdimension1.Theadvantageofthisapproachisthatthewomenthemselvesdefinedthedimensionsthatwereimportanttothem,andtheadvances(usuallyverymodest)thattheythoughtwerepossible.
3. Howrelevantaretheinclusionstrategy,socialmobilizationandprovidingessentialinfrastructure,towomen’sempowerment?
a. Women’sopinionsontherelevanceofinclusion,socialmobilizationandprovisionofinfrastructurefortheirlivesandtheirfeelingofempowerment.
b. Theopinionofkeyinformantsandwomen’sorganizationsontherelevanceoftheseprojectcomponentsforthedifferentgroupsofwomen.
a. Expertandkeyinformantinterviews.b. Interviewswithprojectstaff.c. InformationfromthePADandICR.d. Focusgroupsandin-depthinterviews.e. Observationofprojectactivities(for
example,dosocialmobilizationtrainingactivitieshelpwomentoparticipatemoreactively?
f. Audioandvideorecordingsofgroupactivities.
4. Howrelevantaresmallscaleprivategroupenterprisesforwomen’sempowerment?
a. Women’sopinionsontherelevanceofprivategroupenterprisesfortheirlivesandtheirfeelingofempowerment.
b. Theopinionsofkeyinformantsandwomen’sorganizationsontherelevanceoftheseenterprisesfordifferentgroupsofwomen.
a. Asforpoint3.b. Reviewofgrouprecords(forexample,
loansizesforwomenandmen).
5. Howrelevantwerethegenderobjectivesandtheirimplementationfortheachievementofoverallprojectobjectives?
a. Opinionsofprojectstaffontherelevanceofthegenderobjectivesfortheoverallachievementofprojectobjectives.
b. Opinionsofkeyinformantandwomen’sorganizations.
a. Asforpoint3.b. Reviewoftheoryofchangetocheckon
criticalassumptions.
• 3.EFFICIENCY:Gender-responsivenessofprojectimplementation
1. Werewomenconsultedandinvolvedinprojectdesign?
a. Proportionofwomenondifferentplanninggroups.b. Proportionofwomenwhowereawareofthe
proposedprojects.c. Proportionofwomenwhosaytheywereconsulted.d. Proportionofwomenwhowereinvolvedinproject
1Hashemi,SchulerandRiley(1996)“Ruralcreditprogramsandwomen’sempowermentinBangladesh,”WorldDevelopment24(4):635-53.
ECGReferenceDocument–IntegratingGenderintoProject-levelEvaluation–Appendixes
31
31
implementation. Anappropriatecombinationofthefollowingtoolsandtechniqueswillbeusedtoaddresseachofthesequestions.
Projectmonitoringreports.
a. Projectprofiles.b. Focusgroups.c. Interviewswithprojectstaff.d. Socialmediaanalysis(ifappropriate).e. Rapidsurveys(iffeasible).f. ICR.
2. Didtheprojectsselectedreflecttheprioritiesofwomen?
i. Proportionofwomenwhosayprojects:o Respondeddirectlytotheirneedso Respondedsomewhattotheirneedso Didnotrespondtotheirneeds
3. Wasthewaytheprojectwasimplementedconducivetotheparticipationofwomen?
a. Werethereguidelinesonhowtoinvolvewomeninprojects?
b. Howactivelywerewomeninvolved?4. Howdidtheparticipationof
womeninprojectdesignandimplementationaffecttheachievementoftheoverallprojectobjectives?
a. Howactivelywerewomeninvolvedinprojectdesignandimplementation?
b. Werethereanydifferencesinhowwellprojectsachievedtheirobjectivedependingonhowactivelywomenwereinvolved?
5. Theeffectivenessofthemobilizationstrategy.
a. Weretheredifferencesinhoweffectivelymobilizationstrategiesinvolvedwomenandmen?
b. Whatwerewomen’viewsontheeffectivenessofthemobilizationstrategies?
• 4.OUTPUTSANDOUTCOMES:Achievementofgenderresponsiveprojectobjectives{efficacy]
A.OUTPUTS: • •
1. Women’sparticipationinleadershipandmanagementofprojects.
a. Theproportionofprojectorganizationleadersandmanagerswhoarewomen.
a. Projectmonitoringreports.b. Interviewswithprojectstaff.c. Projectprofiles.d. Audioandvideorecordings.e. Observation.f. Expertandkeyinformantinterviews.
2. Women’sroleinprojectdecision-making.
g. Whatisthelevelofcontributionofwomentoprojectdecisionmaking:
o Womenplayamajorrole.o Womenplaysomerole.o Womenplayverylittlerole.
B.OUTCOMES:Assessinggenderresponsiveoutcomes.
• •
1. Projecteffectsonwomen’sempowerment.
a. Participationincommunityorganizations.b. Participationindecision-makingincommunity
organizations.c. Didtheselectionofvillageprojectsreflectwomen’s
needsandpriorities?d. Impactsonwomen’sstatusinthehousehold:
• Accesstoandcontroloverresources.• Participationindecision-making.
a. Theoryofchange.b. Monitoringreports.c. ICR.d. Expertsandkeyinformants.e. Casestudies.f. Analysisofsocialmedia.g. Audioandvideorecordings.h. Focusgroups.
ECGReferenceDocument–IntegratingGenderintoProject-levelEvaluation–Appendixes
32
32
e. Geographicalmobility.
f. f.Strengtheningsocialcapital.
i. Observation.j. Rapidsurvey.k. Projectprofiles.l. Self-reportedratingscales.m. Harvard-typetimeuseandaccesstoand
controlofresourceschecklists.
2. Assessingsocialandeconomicoutcomesforwomen.
a. Girl’sandwomen’seducation.b. Personalsecurity.c. Geographicalmobility.d. Strengtheningsocialnetworksandsocialcapital.e. Accesstoinformationabouttheoutsideworldand
aboutthecommunity.f. Reducingtimeburdens.
3. Doprojectsaddresswomen’sneedsandpriorities[***Note:thismaybeincludedunderefficiency].
a.Proportionofwomenwhosayprojects:
• Respondeddirectlytotheirneeds.• Respondedsomewhattotheirneeds.• Didnotrespondtotheirneeds
4. Identifyingunintendedoutcomes. a. Didtheprojectshaveanyunintended(unanticipated)positiveoutcomesforwomen?
b. Didtheprojectshaveanyunintended(unanticipated)negativeoutcomesforwomen?
• SUSTAINABIITYANDRESILIENCE:
1. Whatevidenceistherethatdifferentgenderoutcomeswillbesustainable?
a. Haveprojectscontinuedtodeliverthesamelevelofservicesoverthelifeoftheproject(up-tototimeoftheevaluation?)
a. Sustainabilitychecklists.b. Casestudies.c. Observation.
2. Doindividualwomen,andwomen’sgroupsdevelopstrategiestoadapttostressandshocksanddotheylearnfromexperienceandimprovehowtheyadapt?
a. Whatlessonshavebeenlearnedwithrespecttowaystoanticipateandadapttostressesandshocks?
ECGReferenceDocument–IntegratingGenderintoProject-levelEvaluation–Appendixes
33
33
Appendix 7: Example of a GRE design matrix: Evaluating a hypothetical public transport project in Eurasia with defined gender objectives.
1.FrameworkoftheevaluationreportItisproposedthattheframeworkoftheevaluationshouldfollowthestructureofatypicalIndependentEvaluationGroupprojectassessmentreport.Thegenderassessmentwillbebuiltintotherespectivesectionsoftheoverallevaluation.TheproposedoutlinepresentedinBoxA7-1alsosuggestssomeadditionalsectionsthatmightbeincludedinapublictransportproject.
2.Theproject’sgenderobjectives.2.1. PrimaryobjectivesTheProjectAppraisalDocumentidentifiesseveralobjectivesrelatingtopromotingequalityofopportunityintheprocessesofrecruitment,trainingandcareeradvancementwithinthepublictransportcompany.Thesewillbedefinedasthedirectgenderobjectives.Followingthepracticeofmosttheoriesofchange,theseobjectivesaredividedintooutputsandoutcomes(seeBoxA7-2).However,thereisextensiveevidencethatwomenandmenhavedifferenttravelandtransportneeds.Furthermore,manyprojectstendtofocusmoreonmen’stransportneeds(gettingquicklyfromhometo
BoxA7-1:ProposedoutlineforthegenderassessmentcomponentofthePublicTransportProjectinEurasia.
[Possibleadditionalsectionsnotnormallyincludedinthistypeofevaluationareindicatedby***].1. Historyandcontext:
a. ***Theproject’sgenderobjectivesandtheprojectdesignthroughwhichtheobjectiveswillbeachieved.Thismightincludeboththeexplicitgenderobjectivesandsomesecondarygenderoutcomes[objectives]thatmightbeachieved.
b. TheGenderTheoryofChange.TheIEOwilldecideifthisisrequired.2. Relevance:Therelevanceoftheprojectconceptanddesignforpromotingwomen’s
empowermentandaccesstotheprojectsocialandeconomicbenefits.3. ***Efficiency:Thegender-responsivenessofprojectdesignandimplementation.4. Achievementoftheprojectgenderobjectives[efficacy].
a. Outputs.b. Outcomes:primary(projectgenderobjectives),secondary(otherpotentialbenefitsfor
womenparticipantsnotincludedinprojectdesign),andtertiary(effectsonotherwomennotinvolvedintheproject).
5. ***Sustainabilityandresilience:Thelikelihoodthatgenderoutputsandoutcomeswillbesustainedandthebuscompanyandotherstakeholderswillhavetheresiliencetolearnfromexperienceandtoadapttotheevolvingcontextwithinwhichtheprojectoperates.
6. Lessonslearned:howtodesignfutureprojectstostrengthenwomen’sempowermentandgenderequality.
ECGReferenceDocument–IntegratingGenderintoProject-levelEvaluation–Appendixes
34
34
work)andtooverlookwomen’smorecomplexneedstocombinetraveltoworkwithtakingchildrentoschoolandthedoctorandshopping(WorldBank,2010)2.Thesemulti-chainingneedsareignoredbymanytransportprojects,resultinginthelossofmanypotentialwomenpassengerswhohavetorelyoninformaltransportservices.Therearealsoissuesofsecurityasmanystudieshavefoundthathighproportionsofwomenhaveexperiencedsexualharassmentonpublictransport.TheAsianDevelopmentBank(AsDB)GenderandTransportToolkitreportsthattherearegenderdifferencesintravelpatterns,useoftransportmodes,timeuseandtimepoverty,accesstoresourcesfortravel,mobilityandsafetyandthatthereareanumberofgenderbarriersanddifferencesinbenefits3.Asanobjectiveoftheprojectistoprovidehighqualityservicetopassengers,BoxA7-2identifiesanumberofsecondaryandtertiaryoutcomesthatcouldbeconsideredforinclusion.Ifanyoftheseoutcomesareincludedintheevaluation,itwillbeimportanttoclarifythatprojectmanagersandstaffarenotbeingassessedonwhethersecondaryoutcomesareachievedasthesewerenotincludedintheprojectobjectives.ThereasonfortheirinclusionistohelptheEuropeanBankforReconstructionandDevelopment(EBRD)learnlessonstoimprovethedesignandgender-responsivenessoffutureprojects.2.2. DefiningsecondaryandtertiaryoutcomesChapter9discussesinmoredetailhowsecondaryandtertiaryoutcomescouldbedefinedandmeasuredforthisproject.Theproposedapproachisstrategicallyhelpfulbecauseitshowsthatmanyoftheseoutcomesarepotentiallypositiveandcouldsignificantlyincreasetheestimatedprojectbenefits(ratesofreturn).Thisisimportantbecausesecondaryandtertiaryoutcomesareoftendiscussedintermsof“unintendedoutcomes”wherethefocushasusuallybeenonthenegativeoutcomes,suchasincreasesindomesticviolenceorincreaseddemandsonwomen’stime.
3.Theproject’sgenderresponsiveinterventionsTheprojecthasseveralcomponents/interventionsthatarespecificallyintendedtopromotetheequalparticipationofwomeninrecruitment,trainingandjobadvancementwithinthepublictransportcompany(seeBoxA7-3).ThereisalsoareferencetoensuringtheproposedspecificationsforthebusessatisfyEUrequirementsforenvironmentalandsocialimpact.Thiscouldbeinterpretedtoincludestrategiestopreventsexualharassment(acommonproblemonpublictransport)andpossiblyalsotoconsidersafetyandconvenienceconsiderationswithrespecttothelocationofbusstops.Oneimportantsetofissuesthatdonotseemtobeaddressedconcernsthedifferenttransportandtravelneedsofwomenandmen(discussedintheprevioussection).However,asthereisnoreferencetotheseissuesintheprojectdocument,thesequestionscanpresumablynotbeaddressedwhenassessingprojectinterventions.
2WorldBank(2010).Mainstreaminggenderinroadtransport.Chapter2Gendertrippatternsandmobilityconstraints.3ADB(2013)Gendertoolkit:Transport.Maximizingthebenefitsofimprovedmobilityforall.
ECGReferenceDocument–IntegratingGenderintoProject-levelEvaluation–Appendixes
35
35
BoxA7-2:Identifyinggenderobjectives:Directprojectgenderoutputsandoutcomes,andsecondaryandtertiarygenderoutcomes.
GenderOutputs
DirectGenderOutputs
a. Equalityof:o Recruitmentopportunities.o Careerprogression.o Accesstotraining.
b. Increasethenumberofwomenhired.c. Workshopwithstakeholderstodisseminatelessonslearnedonhowtopromotegender
equality.d. EnsurebusesmeetEuropeanUnionstandards,includingongenderconsiderations(suchas
securityandcomfort).
Secondarygenderoutputs[notdefinedintheprojectdesign]
a. Routeplanningandservicefrequencytakesintoconsiderationwomen’sspecifictransportneeds.
b. Serviceplanningtakesintoconsiderationgender-relatedsecurityandsafety.Forexample:o Driversandconductorsaretrainedhowtoaddresssexualharassmentonthebuses.o Busstopsarewelllitarenotlocatedclosetobarsorotherareaswithgreaterriskfor
women.o Additionalsecurityisprovidedforwomentravellingtoandfromworkatnight.o Driversandstaffaretrainedtorespectwomenpedestrians[studiesinanumberof
countrieshavefoundthatdriversdonotslowdowntoallowwomenpedestrianstosafelycrosstheroad,andsometimesdriverswillnotcompletelystopsothatwomenhavetogetoffwhenthebusisstillmoving].
c. Numberofwomenpassengerssteadilyincreases.
PrimaryGenderOutcomes(programobjectives)
Directgenderoutcomes
a. Morewomenarehired.b. Morewomenreceivetraining.c. Morewomenarepromoted.d. Women’sjobstabilityincreases.e. Morewomenbecomedriversandmechanics.f. Gendergoodpracticelessonsaredisseminatedto,andadoptedbyothertransport
companies.Secondarygenderoutcomes
a. Women’ssafetyimprovescomparedtootherbuscompaniesandformsoftransport.b. Genderresponsiverouteplanningcontributestoreducingwomen’stimeburden.c. Women’incomeincreases(duetogreateraccesstojobopportunitiesthroughmore
convenienttransport].d. Children’shealthimprovesasgender-responsivetransportplanningmakesiteasierfor
workingmotherstotakechildrentothedoctor.e. Children’sschoolattendanceimprovesasitiseasierforworkingmotherstotakechildrento
school.
ECGReferenceDocument–IntegratingGenderintoProject-levelEvaluation–Appendixes
36
36
4.DefinitionofthegenderquestionstobeaddressedintheevaluationBoxA7-4identifiesaninitiallistofpossiblegender-relatedquestionsrelatingtodimensionslistedinTableA7-1.Thesecoverfivedimensions:
a. Historyandcontextandhowtheseaffecttheimplementationandlikelygenderoutcomesoftheproject.Thisincludesinformationonearlierprogramstoincorporatewomenintothetransportsectorandbroadereffortstobroadenwomen’saccesstolabormarkets.Relevantlegislationandgovernmentpolicieswillalsobedescribed.
b. Relevanceoftheprojectdesigntotheachievementofimportantgenderobjectives.c. Efficiencyofprojectimplementationwithrespecttotheachievementofgenderobjectives.d. Achievementofgenderobjectives(outcomes/efficacy).Objectivesaredefinedintermsof
outputsandoutcomes.e. Sustainabilityandresilience.Thereisnodirectreferencetosustainabilityorresiliencein
theprojectdocument.However,theseareimportantconsiderationsasmanyinitiativestopromotewomen’seconomicequalitybeginwellbutoftenencounterorganizational,cultural,politicalandeconomicchallengesthatreducetheirlonger-termimpact.So,adecisionmustbemadeastowhetheritisappropriatetoincludethisdimensionintheevaluation.
BoxA7-3Projectcomponentsandinterventionsdesignedtopromotewomen’saccesstoemploymentopportunitiesandattentiontowomen’sconcernsinthedesignofbuses(andpossiblybusstops).
1.Interventionstopromotewomen’saccesstoemploymentopportunitiesinthebuscompany.
a. Promotinggender-responsiverecruitmentpolicies.b. Puttinginplacecareeradvancementpoliciesforwomen.c. Trainingforwomencandidatestopromotecareeradvancementandincreasing
thenumberofwomenemployedatthecompany.2.Gender-sensitiveservicedesign.
a. EnsuringbusdesignrespondstoEUrequirementsonsocialandenvironmentalimpacts.
b. Thisrequirementmightalsoincludelocatingbusstopsandtheirdesign(forexample,adequatelighting)totakeintoconsiderationwomen’ssafetyandconvenience.
c. Note:theredoesseemtobeanyreferencetoplanningtransportroutestorespondtowomen’smulti-chainingneedstocombinetraveltoworkwiththeneedtotakechildrentoschoolandtothedoctorandtogoshopping.
3.Disseminatinglessonsonhowtomaketransportprogramsmoregender-responsive.Workshopspromotingknowledgesharingacrossthecountryandthewiderregion.
ECGReferenceDocument–IntegratingGenderintoProject-levelEvaluation–Appendixes
37
37
BoxA7-4:Gender-relatedquestionstobeaddressedintheevaluation
1.Historyandcontexta. Havetherebeenearlierinitiativestopromotewomen’semploymentinthetransport
sector?Whatweretheresults?b. Havetherebeeninitiativesinothersectorsandwhatweretheresults?c. Isthereanylegislationorgovernmentpoliciesaffectingwomen’slabormarketaccess?
Whathavebeentheeffectsofthesemeasures?d. Howdidtheseearlierexperiencesaffectthedesignofthepresentproject?
2.Relevancea. Howdoestheprogramexpecttostrengthenwomen’saccesstoemploymentandjob
advancementinthebuscompany?b. Aretheinterventionsrelevanttotheachievementofgenderequalityinthebuscompany
andperhapsmorewidely?c. Howrelevantarethegenderobjectivesandtheirimplementationsstrategiestothe
achievementofoverallprojectobjectives?3.Efficiency
a. Werewomenconsultedandinvolvedinprojectdesign?b. Dotheselectedinterventionreflecttheprioritiesofwomen?c. Arethereanyadditionaldesignorimplementationelementsthatshouldhavebeen
included?d. Wereprojectimplementationstrategiesconducivetotheparticipationofwomen?
4.Achievementofgenderobjectives(efficacy)a. Outputs:
i. Increasednumberofwomenrecruited.ii. Increasednumberofwomentrained.iii. Rateofwomen’sadvancementincreases.iv. Consultationmechanismswithwomenareputinplace.v. Busandservicedesigntakesgenderissuesintoconsideration.vi. Genderissuesareincludedinworkshopstodisseminatelessonsfromtheproject.
b. Primarygenderoutcomes.i. Increaseinproportionofwomenemployedbythebuscompany.Proportion
increaseoverthelifeoftheproject.ii. Increaseinproportionofwomeninnon-administrativepositions(drivers,
mechanics).iii. Increaseinwomen’searnings.iv. Increaseinproportionofwomeninmanagerialpositions.
c.Secondarygenderoutcomes.
i. Improvedwork-lifebalance.ii. Enhancedwomen’sempowerment(seeChapter9).
d.Tertiarygenderoutcomes
i. Improvedcomfortandsafetyforwomenpassengers.ii. Timesaving.iii. Increasedaccesstourbanservicesandentertainment.
5.Sustainabilityandresiliencea. Whatevidenceistherethatwomen’sadvancesinthecompanyarelikelytobesustained?b. Havewomeninthecompanylearnedcopingmechanismsforworkingandadvancingina
male-dominatedworkenvironment?
ECGReferenceDocument–IntegratingGenderintoProject-levelEvaluation–Appendixes
38
38
5.Initialproposalforthegenderevaluationmethodology.5.1.TableA7-1presentsinitialideasfortheevaluationdesign.Thisprovidesaninitialframeworkthatwillneedtoberefinedonceagreementhasbeenreachedontheevaluationapproach.Thetableincludesthreecolumns:
• ThequestionscoveringeachofthefivedimensionsoftheevaluationlistedinBoxA7-4.• Theindicatorsusedtoaddresseachquestion.• Thedatacollectionmethods.
Theframeworkidentifiesarangeofpossibledatacollectionmethodsfromwhichanappropriatesetwillbeselectedforeachstageoftheevaluation.AllofthesemethodsaredescribedintheECGReferenceDocument.Itislikelythatnotallofthelistedmethodswillbeusedinthepresentevaluation.Theinitiallistofdatacollectionandanalyticalmethodsincludes:1. Constructingagendertheoryofchange[TOC]thatcanhelpidentifykeyevaluationquestions,
indicatorsandthepathwaysthroughwhichitisexpectedthatgenderoutcomeswillbeachieved.NotallgenderevaluationsuseaTOCsoadecisionwillbeneededastowhetheritisappropriateforthepresentevaluation.
2. Constructingadatacollectionplanningmatrix.Appendix5presentsanexampleofadatacollectionplanningmatrixthatcouldbeusedinthedesignofthepresentevaluation.Thematrixlistsallofthekeygender(andother)questions,theproposedevaluationdesigns,theindicatorsandthedatacollectionmethods.Importantly,italsoassessesthefeasibilityofcollectingtheproposedinformationwithinthebudget,time,methodologicalandorganizationalconstraintswithinwhichtheevaluationwillbeconducted.
3. Secondarysources.a. Theprojectdocument.b. Projectmonitoringandprogressreports.c. Governmentreports.d. Partnerreports.e. Civilsocietyreports.f. Academicresearchandpublications.g. Systematicreviews.
4. Consultations.a. Expertandkeyinformants.b. Civilsocietyorganizations.c. Otherfundingagencies.d. Focusgroups.
5. Surveys,ratingscalesandchecklists.a. Rapid,shortsamplesurvey.b. Trafficandpassengersurveys.c. Travelsafetyaudits.d. Ratingscalestobecompletedbybeneficiaries,projectstafforpartners.e. Checklists.
6. Casestudies.a. Descriptivecasestudies.b. Analyticalcasestudies(QCA).
7. Qualitativefieldwork.a. Projectvisits.b. Informalinterviews.c. In-depthinterviews.d. Observation.e. Participantobservation.
ECGReferenceDocument–IntegratingGenderintoProject-levelEvaluation–Appendixes
39
39
f. Traveldiaries.8. Socialmediaanalysis(ifthisisfeasible).
a. Socialmediaanalytics(forexample,analysisoftwitterandothersocialmedia).b. Internetsurveys.
9. Pipelinedesignstoconstructacounterfactual.5.2.MeasuringsecondaryandtertiaryoutcomesChapter9discussesinmoredetailhowsecondaryandtertiaryoutcomescanbedefinedandmeasured.
ECGReferenceDocument–IntegratingGenderintoProject-levelEvaluation–Appendixes
40
40
• TableA7-1:Keygender-responsivequestions,indicatorsanddatacollectionmethods
•
• PARTI:SOMEGENERALAPPROACHESTHATCANBEUSEDTHRUGHOUTTHEEVALUATION
1.GenderTOC:DevelopingagenderTOCthatisusedtoidentifythekeyevaluationquestionsandtodefinetheprocessesthroughwhichoutputsandoutcomesaretobeachieved.Thisalsodefineskeyassumptionstobetested.
• 2.Resultsframework(ifithasbeenusedintheproject):Thisdefinestheintendedgenderobjectivestobeassessed.
• 3.Descriptivecasestudies:Usuallyarelativelysmallnumberofcasestudiesthatarebroadlyrepresentativeofthemainprojectscenariosandwhichareusedtoillustratehowtheprojectsevolvedandthelivedexperienceoftheprojectpopulations.Casestudiescanbelongitudinal(conductedover
arelativelylongperiodoftime),or(asisusuallythecasewithex-postevaluations)conductedatonepointintime,andrelyingextensivelyonrecall.
• 4.Analyticalcasestudies:TheseusetechniquessuchasQCA(qualitativecaseanalysis)toidentifythenecessaryandsufficientconditionsforprojectoutcomestooccur,andalsothenecessaryandsufficientconditionswhereprojectoutcomesdoNOToccur.Normallyatleast30casesarerequired.
WhileQCAarerarely(ifever)usedbyIEOs,theycouldbeapotentiallypowerfulanalyticaltoolthatpermitsattributionanalysistobeused.
• 5.Counterfactualanalysis:Thiscanbeusedatthenationallevelforprogramsintendedtocoverthewholecountry,orattheprojectlevel.FortheevaluationoftheTransportProject,counterfactualwouldprobablycomparetheprojectwithotherbuscompanies(assumingthereanyother
companiesthatareofasimilarsizewithcommoncharacteristics).Theanalysiswouldrequireabeforeandaftercomparison.
•
• PARTII:DESIGNSFORASSESSINGEACHDIMENSIONOFTHEEVALUATION
•
• Dimension/Question • Indicators • Datacollectionmethods
• [appropriatetoolswillbeselectedforeachphaseoftheevaluationfromthislistof
options]
• 1.HISTORYANDCONTEXT
a. Havetherebeenearlier
initiativestopromotewomen’s
employmentinthetransport
sector?Whatweretheresults?
a. Transportsectorprojectswithgender
components/objectives.
b. Genderoutcomes.
c. Transportprojectsthathadunintendedgender
outcomes.
a. Expertandkeyinformantsinterviews.
b. Governmentreports.
c. Civilsocietyreports.
d. Donorreports.
e. Academicresearch.
b. Howdidtheseearlier
experiencesaffectthedesignof
thepresentproject?
a. Referencesintheprojectdocumentsonprevious
genderinitiatives.
ECGReferenceDocument–IntegratingGenderintoProject-levelEvaluation–Appendixes
41
41
c. Havetherebeeninitiativesin
othersectorsandwhatwerethe
results?
a. Othersectorprojectswithgenderemployment
components.
b. Projectgenderoutcomes.
d. Isthereanylegislationor
governmentpoliciesaffecting
women’slabormarketaccess?
Whathavebeentheeffectsof
thesemeasures?
•
a. Legislationconcerningwomen’slabormarketaccess.
b. Opinionsonpolicyoutcomes.
e. Arethereanyimportant
economic,politicalorcultural
contextualfactorsthathave
affectedhowthegender
componentsweredesignedor
implementedorthataffected
theiroutcomes?
a. Reportsandindicatorsoncontextualfactorsaffecting
women’slabormarketaccess.•
• 2.RELEVANCE:Therelevanceoftheprojectconceptanddesignforpromotingwomen’sempowermentandsocialandeconomicbenefits
a.Howdoestheprojectexpectto
promotewomen’saccessto
employmentandtoimprovethe
qualityoftransportservicesfor
women?
a. DevelopaTOCthatidentifiestheintendedgender
outputsandoutcomesandtheprocesses/steps
throughwhichthesearetobeachieved.Someofthe
outputsandoutcomesmayinclude:
i. DirectGenderOutputs[seeSection4A].
ii. Secondarygenderoutputs[seeSection4B].
iii. Directgenderoutcomes[seeSection4C].
iv. Secondarygenderoutputs[seeSection4D.]
a. ThedraftTOCwouldbedeveloped
byconsultantsonthebasisofproject
documentsandinterviews.Feedback
wouldbeobtainedfromprojectstaff,
beneficiariesandcivilsociety.
b. Projectdocument.
c. Interviewswithprojectstaffand
partnerimplementingagencies.
d. Expertsandkeyinformants.
e. Householdincomeandexpenditure
surveys.
f. Passengerandtransportsurveys.
g. Socialmediaanalysis(twitter,and
suchlike)iffeasible.
h. Participantobservation(observer
travellingonbuses).
ECGReferenceDocument–IntegratingGenderintoProject-levelEvaluation–Appendixes
42
42
b.Howrelevantaretheseintended
outputsandoutcomesforwomen
workersandpassengers?
a. Women’sopinionsontherelevanceofeachoutputand
outcome(seepoint1above)totheirlives[***Note:an
appropriateandunderstandabletermfor
“empowerment”mustbeidentified].
b. Theopinionofkeyinformantsandwomen’s
organizationsontherelevanceoftheoutputsand
outcomesfordifferentgroupsofwomen.
a. Reviewtheoryofchange.
b. Focusgroupswithbeneficiaries,project
staffandcivilsociety.
c. Expertandkeyinformantinterviews.
d. Interviewswithprojectstaff.
e. Informationfromtheprojectdocument.
f. Audioandvideorecordingsofproject
activities.
g. Projectmonitoringandprogressreports.
h. ApplicationofOECD-typeratingscale.
c.Howrelevantwerethegender
objectivesandtheirimplementation
fortheachievementofoverallproject
objectives?
a. Opinionsofprojectstaffontherelevanceofthegender
objectivesfortheoverallachievementofproject
objectives.
b. Opinionsofkeyinformantandwomen’sorganizations.
ECGReferenceDocument–IntegratingGenderintoProject-levelEvaluation–Appendixes
43
43
• 3.EFFICIENCY:Gender-responsivenessofprojectimplementation
a.Werewomenconsultedand
involvedinprojectdesign?a.Werestakeholdersconsultedonprojectdesign?
b.Werethereprovisionstoinvolvewomeninthe
consultations?
c.Whichgroupsofwomenwereconsulted?
-Actualorpotentialemployees
-Civilsociety
f.Howmanywomenwereactivelyinvolvedthe
consultations?
•
a. Projectmonitoringreports
b. Projectprofiles
c. Focusgroups
d. Interviewswithprojectstaff
e. Socialmediaanalysis(ifappropriate)
f. Rapidsurveys(iffeasible)
g. Observation
b.Didtheprojectsselectedreflect
theprioritiesofwomen
Proportionofwomenwhosayprojects:
o Respondeddirectlytotheirneeds.
o Respondedsomewhattotheirneeds.
o Didnotrespondtotheirneeds.
c.Wasthewaytheprojectwas
implementedconducivetothe
participationofwomen?
a. Werethereguidelinesonhowtoinvolve
womeninprojects?
b. Howactivelywerewomeninvolved?
d.Howdidtheparticipationof
womeninprojectdesignand
implementationaffectthe
achievementoftheoverallproject
objectives?
a. Howactivelywerewomeninvolvedinproject
designandimplementation?
b. Werethereanychangesindesigninresponse
tofeedbackfromwomen?
c. Howdidthesechangesaffecttheoverall
efficiencyofthedifferentprojectcomponents?
• 4.OUTPUTSANDOUTCOMES(IMPACTS):Achievementofgenderresponsiveprojectobjectives
A.DIRECTGENDER-RESPONSIVEOUTPUTS:
a.DirectGenderOutputs
i. Increasednumberofwomenrecruited.
ii. Increasednumberofwomentrained.
iii. Rateofwomen’sadvancementincreases.
iv. Consultationmechanismswithwomenareput
inplace.
v. Busandservicedesigntakesgenderissuesinto
consideration.
vi. Genderissuesareincludedinworkshopsto
disseminatelessonsfromtheproject.
a. Theprojectdocument.
b. Interviewswithprojectstaffandpartnerimplementingagencies.
c. Expertsandkeyinformants.d. Householdincomeandexpenditure
surveys.
e. Passengerandtransportsurveys.f. Socialmediaanalysis(twitterandsuch
like)iffeasible.
g. Participantobservation(theresearcherspendstimetravellingonbusesto
ECGReferenceDocument–IntegratingGenderintoProject-levelEvaluation–Appendixes
44
44
• observesexualharassmentandother
issuesaffectingwomenpassengersand
drivers/conductors.
h. Participantobservation(observertravellingonbuses).
i. Focusgroupswithbeneficiaries,
projectstaffandcivilsociety.
j. Audioandvideorecordingsofproject
activities.
k. Projectmonitoringandprogress
reports.
l. ApplicationofOECD-typeratingscale.
•
B.SECONDARYGENDER-RESPONSIVEOUTPUTS
b.Secondarygenderoutputs.
i. Routeplanningandservicefrequencytakes
intoconsiderationwomen’sspecifictransport
needs.
ii. Serviceplanningtakesintoconsideration
gender-relatedsecurityandsafety.For
example:
o Driversandconductorsaretrainedhowto
addresssexualharassmentonthebuses.
o Busstopsarewelllightedandarenot
locatedclosetobarsorotherareaswith
greaterriskforwomen.
o Additionalsecurityisprovidedforwomen
travellingtoandfromworkatnight.
o Driversandstaffaretrainedtorespect
womenpedestrians[studiesinanumberof
countrieshavefoundthatdriversdonot
slowdowntoallowwomenpedestriansto
safelycrosstheroad,andsometimes
driverswillnotcompletelystopsothat
womenhavetogetoffwhenthebusisstill
moving].
•
ECGReferenceDocument–IntegratingGenderintoProject-levelEvaluation–Appendixes
45
45
iii. Numberofwomenpassengerssteadily
increases.
C.DIRECTGENDERRESPONSIVEOUTCOMES
c.Directgenderoutcomes
i. Increaseinproportionofwomenemployedby
thebuscompany.Proportionincreaseoverthe
lifeoftheproject.
ii. Increaseinproportionofwomeninnon-
administrativepositions(drivers,mechanics).
iii. Increaseinwomen’searnings.
iv. Increaseinproportionofwomeninmanagerial
positions.
v. Hasthework/livebalanceimprovedfor
womenworkers?D.SECONDARYGENDERRESPONSIVEOUTCOMES
d.Secondarygenderoutcomes
i. Women’ssafetyimprovescomparedtoother
buscompaniesandformsoftransport.
ii. Genderresponsiverouteplanningcontributes
toreducingwomen’stimeburden.
iii. Women’incomeincreases(duetogreater
accesstojobopportunitiesthroughmore
convenienttransport.
iv. Children’shealthimprovesasgender-
responsivetransportplanningmakesiteasier
forworkingmotherstotakechildrentothe
doctor.
v. Children’sschoolattendanceimprovesasitis
easierforworkingmotherstotakechildrento
school.
•
3. Women’sparticipationin
leadershipandmanagementof
projects
a. Theproportionofprojectorganizationleadersand
managerswhoarewomen
a. Projectmonitoringreports.
b. Interviewswithprojectstaff.
c. Projectprofiles.
d. Audioandvideorecordings.
e. Observation.
f. Expertandkeyinformantinterviews.
4. Women’sroleinproject
decision-making
a.Whatisthelevelofcontributionofwomentoproject
decisionmaking:
i. Womenplayamajorrole.
ECGReferenceDocument–IntegratingGenderintoProject-levelEvaluation–Appendixes
46
46
ii. Womenplaysomerole.
iii. Womenplayverylittlerole.
B.OUTCOMES:Assessinggenderresponsiveoutcomes.
• •
5. Projecteffectsonwomen’s
empowerment
a. Participationincommunityorganizations.
b. Participationindecision-makingincommunity
organizations
c. Impactsonwomen’sstatusinthehousehold.
d. Accesstoandcontroloverresources.
e. Participationindecision-making.
f. Geographicalmobility.
g. Strengtheningsocialcapital.
a. Theoryofchange.
b. Monitoringreports.
c. Projectdocument.
d. Expertsandkeyinformants.
e. Casestudies.
f. Analysisofsocialmedia.
g. Audioandvideorecordings.
h. Focusgroups.
i. Observation.
j. Rapidsurveys.
k. Projectprofiles.
l. Self-reportedratingscales.
m. Harvard-typetimeuseandaccesstoandcontrolofresources
checklists.
6. Assessingsocialandeconomic
outcomesforwomen
i. Girl’sandwomen’seducation.
ii. Personalsecurity.
iii. Geographicalmobility.
iv. Strengtheningsocialnetworksandsocial
capital.
v. Accesstoinformationabouttheoutsideworld
andaboutthecommunity.
vi. Reducingtimeburdens.
7. Doprojectsaddresswomen’s
needsandpriorities[***Note:
thismaybeincludedunder
efficiency].
a.Proportionofwomenwhosayprojects:
i. Respondeddirectlytotheirneeds.
ii. Respondedsomewhattotheirneeds.
iii. Didnotrespondtotheirneeds.
8. Identifyingunintended
outcomes.
a. Didtheprojectshaveanyunintended
(unanticipated)positiveoutcomesforwomen?
b. Didtheprojectshaveanyunintended
(unanticipated)negativeoutcomesforwomen?
ECGReferenceDocument–IntegratingGenderintoProject-levelEvaluation–Appendixes
47
47
• SUSTAINABIITYANDRESILIENCE:
3. Whatevidenceistherethat
women’sadvancesinthe
companyarelikelytobe
sustained?
a. Didtheproportionofwomenrecruitedincrease,
anddidtheincreasecontinuethroughoutthelifeof
theproject?
b. Didtheproportionofwomenwhoreceived
advancements(promotion)increaseanddidthese
increasescontinuethroughoutthelifeofthe
project?
c. Didthenumber/proportionofwomenreceiving
trainingincrease,anddidtheseincreasescontinue
throughoutthelifeoftheproject?
d. Opinionsofdifferentstakeholdergroupsonthe
likelihoodthattheimprovementsinpointsa-cwill
besustained.
a. Sustainabilitychecklists.
b. Casestudies.
c. Observation.
d. Focusgroups.
e. Individualinterviews.
4. Havewomeninthecompany
learnedcopingmechanismsfor
workingandadvancingina
male-dominatedwork
environment?
a. Whatwerethechallengesfacingwomenworkersto
sustaintheirprogress?
b. Whatweretheopinionsofotherstakeholder
groups?
c. Didwomenreportthattheyhadlearnedanycoping
mechanismstodealwithproblemsidentifiedin
pointb?
ECGReferenceDocument–IntegratingGenderintoProject-levelEvaluation–Appendixes
48
48
Appendix 8: Links to the gender indexes
1.AfricaGenderEqualityIndex
https://www.afdb.org/en/topics-and-sectors/topics/quality-assurance-results/gender-equality-index/
2.SDGIndicators
https://sustainabledevelopment.un.org/content/documents/21252030percent20Agendapercent20forpercent20Sustainablepercent20Develop
mentpercent20web.pdf
3.TheGender-RelatedDevelopmentIndex(GDI)
http://hdr.undp.org/en/content/gender-development-index-gdi
4.GenderEmpowermentMeasure(GEM)
https://en.wikipedia.org/wiki/Gender_Empowerment_Measure
5.SocialWatchGenderEquityIndex(GGI)
http://www.socialwatch.org/taxonomy/term/527
6.WorldEconomicForumGenderGapIndex(GGI)
https://www.weforum.org/reports/the-global-gender-gap-report-2016
7.AfricaWomen’sProgressScorecard
http://www.uneca.org/sites/default/files/PublicationFiles/agdi_2011_eng_fin.pdf
8.ThematicIndicators
FAOhttp://www.fao.org/fileadmin/templates/ess/ess_test_folder/Workshops_Events/AFCAS_19/AFCAS_05_7_2_b.pdf
UNESCOhttp://uis.unesco.org/en/glossary-term/gender-parity-index-gpi
ECGReferenceDocument–IntegratingGenderintoProject-levelEvaluation–Appendixes
49
49
Appendix 9: Strengthening the dissemination and use of gender-evaluation findings.
1.TheunderutilizationofevaluationsThereisextensiveevidencethatevenmethodologicallysoundevaluationsarefrequentlyunder-utilized
(BoxA9.1).MostofthesefindingsareequallyapplicabletoGREbutthereareadditionalfactorsthatcan
alsoconstraintheutilizationofgender-responsivefindings.First,manyprojectsdonothavespecific
genderobjectives,orgenderobjectivesaredefinedverynarrowly.Inbothcases,findingsrelatingto
potentiallyimportantgender-relatedsecondaryandtertiaryoutcomesmaynotbeconsideredrelevantif
theywerenotincludedintheresultsframework.ThecasestudiespresentedinPartIIprovidemultiple
examplesofexcludedgenderissues.Second,projectstaffmayberesistanttoacceptingthefindingsof
genderanalysiswhichtheyfeelunfairlycriticizethemfornothavingaddressedgenderissuesthatwere
notincludedintheprojectdesign.Third,thewayinwhichGREfindingsarepresentedisoftennotlinked
directlytodevelopmentobjectives,butaddressesbroaderconcernssuchaswomen’sempowermentor
humanrights.Fourth,GREoftenneedtomakeastrongercaseforthevalueaddedofthetimeand
resourcesinvestedinaddressinggenderissues.Fifth,IEOsmaynothaveadisseminationstrategythat
ensuresthefindingsareaccessibletothewiderrangeofstakeholders(forexample,civilsocietyand
women’sorganizations)thatarethepotentialadvocatesandimplementersoftheevaluationfindings.
Finally,stakeholdersoftenarguethattheGREfindingsarenotbasedonafullunderstandingofthelocal
contextandthatrecommendationsarenotrealisticwithinthelocalpoliticalandculturalscenario.
2.StrategiesforpromotingtheutilizationofGREevaluationsGiventhenatureoftheIEOmandate,notalloftheutilizationstrategiesdiscussedintheliteratureare
applicabletoIEOs.Inparticular,therequirementforindependenceandthefactthatevaluationsare
conductedex-postmeansthatmanyoftheutilization-focusedevaluationstrategies(Patton,2008)that
BoxA9.1Reasonswhyevaluationfindingsareoftenunderutilized
• Badtiming:thereportistoolatetocontributetopolicydecisionsoritcomestooearlybefore
agenciesarefocusingontheseissues.
• Lackofflexibilitytorespondtotheinformationneedsofstakeholders.
• Wrongquestionsareaskedandfindingsnotconsideredusefulorrelevant.
• Evaluationsaretooexpensiveandmaketoomanydemandsonagencystaffandresources.
• Simplisticanswersaregiventocomplexissues.
• Evaluatorsdonotunderstandthecomplexitiesofthelocalcontext.
• Localexpertsarenotconsulted/involvedintheevaluation.
• Findingsarenotpresentedinawaythatiseasilyaccessibletodifferentstakeholders.
Source:Bamberger,SegoneandTateossian(2016)
ECGReferenceDocument–IntegratingGenderintoProject-levelEvaluation–Appendixes
50
50
requireregularinteractionwithimplementingagenciesarenotdirectlyapplicable.Thefollowingare
someofthestrategiesforpromotingGREutilizationapplicabletoIEOs:
a. Developadisseminationstrategythataddressesalloftheissuesidentifiedintheprevious
section,includingthespecificgender-relatedconstraintsandopportunities.
b. Ensurefindingsandrecommendationsarealignedwithprojectdevelopmentobjectives.Many
GREfindingsaddressbroaderissuessomeprojectstaffmaynotfindrelevantorcannot
implement.Thispresentsastrategicchallengeasmanyoftheseissuesrelatingtoempowerment,
humanrightsandinclusionarecriticaltodevelopment.Findingsmustbeintegratedwitha
capacity-buildingstrategythatcanhelpagenciesreviewandbroadentheirapproachestosocial
aswellaseconomicdevelopment.
c. Relatedtothepreviouspoint,utilizationofGREevaluationfindingscanonlybeachievedifthey
arebasedonafullunderstandingofthelocalpolitical,economicandsocialcontext.Changesin
deeplyengrainedsystemsofsocialcontrolrequirethestrategicidentificationofpotential
interventionpointsthatarerealisticwithineachlocalcontext.
d. Using“carrots”(incentives),“sticks”(sanctionsandpunishments)and“sermons”(showof
supportfromrespectedfigures)toencourageutilization(MacKay,2007).Anexampleofan
incentivemightbetheavailabilityofagenderfundthatcouldbedrawnontosupport
implementationofsomeoftherecommendations.
e. IEOsmayneedtodevelopinnovativecommunicationmechanisms(suchasuseofsocialmedia,
smartphones,collaborationwithcivilsociety,shorterpublicationstargetedtoparticular
audiences).
f. Ensuringthatagreedactionsongenderfindingsareincludedinthemanagementactionplans
thatmostIEOsusetomonitorimplementationofthemanagementagreementsand
commitmentsonevaluationrecommendations.
3.DevelopinganorganizationallearningstrategyIEOsshouldcoordinatewithagencycapacitydevelopmentunitstostrengthenstaffunderstandingof
genderissuesandoutcomesandhowtheyshouldbeaddressedinthedesignoffutureprojects
(programsandpolicies).GREreportsprovidevaluableteachingmaterialastheyillustratehowagency
genderpoliciesandstrategiesactuallyoperateinthefield.Thiscanalsohelpbuildawarenessofthe
importanceofbuildingthecollectionofdataongenderindictorsintoprojectM&Eandmanagement
systems.
4.BuildinggenderindicatorsandfindingsintokeyagencyreportsIEOshouldworktoensurethatgenderindicatorsareincorporatedintokeyagencyreportsandpolicy
documentsaswellasintomostIEGproducts.Chapter2describedthedifferentkindsofindicatorsthat
couldbemainstreamed(seeSection2.7).Thefollowingaresomeoftheindicatorsandevidencethat
couldbeincorporatedintodifferentagencyreports.
BoxA9.2UsefulreferencesonutilizationofGREevaluations
UNWomenIndependentEvaluationOffice(2015)HowtoManageGender-ResponsiveEvaluation:EvaluationHandbook.Chapter7Useandfollow-up.
Bamberger,M,M.Segone&S.Reddy(2014).Nationalpoliciesforsustainableandequitabledevelopment:Howtointegrategenderequalityandsocialequityinnationalevaluationpoliciesandsystems.EvalPartners,UNWomenandIOCE.
KarkaraN(undated).“Advocatingforevaluation:Atoolkittodevelopadvocacy
strategiestostrengthenanenablingenvironmentforevaluation”.
ECGReferenceDocument–IntegratingGenderintoProject-levelEvaluation–Appendixes
51
51
a. Disaggregatingstandardsocio-economicindicatorsbysex.
b. Presentinggenderchecklists.
c. IncorporatinggenderintotheOECD/DACstandardratingscales.
d. Shortillustrativecasestudies.
ECGReferenceDocument–IntegratingGenderintoProject-levelEvaluation–Appendixes
52
52
iManyevaluationsrelyondatacollectedatthelevelofthehousehold.GREevaluatorsarguethathouseholdleveldata
ignoresimportantdifferencesinhowfoodandotherresourcesaredistributedamonghouseholdmembers.Disaggregated
analysisiscriticalforGREaswomenandgirlsoftenreceiveasmallershareoffoodsothatmalnutritionratescanbe
higher.
iiForexample,CaseStudyNo.1(theVillageInvestmentProject)includesoutputindicatorsforeachoftheProject
Components(capacitybuildingandempowerment,villageinvestmentsandmanagement).Forthevillageinvestments
component,tensetsofoutputindicatorsaremeasuredforcoresectorsincluding:peopleattheprojectlevelwith
improvedwatersupply,constructionofwatersupplypoints,directfemaleandmaleprojectbeneficiaries.
iiiMixedmethodsareusuallyunderstoodtocombinequantitativeandqualitativemethodsfromtwodifferentsocial
sciencedisciplineswhilemulti-methodapproachesinvolvecombiningdifferentresearchandevaluationmethodsfrom
withinthesamediscipline.However,thedistinctionisnotveryclearandsomeoffersprefertocombinethetwo
approaches.
ivOftenmostoftheevaluationbudgetmightbeinvestedinaquantitativesurveysoastoachievemaximumstatistical
power.Anotheroptionmightbetoreducethesamplesizeandtoinvestsomeoftheresourcesinqualitativemethodssuch
asfocusgroups,keyinformantinterviewsandsoforthCombiningtheseindependentestimatesmightproducemore
usefulandmeaningfulresults.However,increasingvalidityrequiresthatQUALdataisselectedtoensureitis
representativeandcanbecomparedwiththeQUANTfindings,
vAnexampleofacommondifferenceiswhenaQUANTsurveyisaskingaboutchangesinincomesincethestartofthe
projectwhileQUALinterviewsarefocusingmoreonfeelingsofeconomicsecurityandvulnerabilitytoeconomiccrises.
Sometimessurveysfindthatincomehasgoneupbutthatmanypeoplefeelmorevulnerableandinsecure.Itoftentakes
theresearcherssometimethatthesequestionsareexploringdifferentissues.vi BetterEvaluation(http://www.betterevaluation.org/en/approach/realist_evaluation)definesagenerativemechanismsasfollows:Strictlyspeaking,theterm‘generativemechanism’referstotheunderlyingsocialorpsychological
driversthat‘cause’thereasoningofactors.Forexample,aparentingskillsprogrammayhaveachieveddifferent
outcomesforfathersandmothers.Themechanismgeneratingdifferent‘reasoning’bymothersandfathersmayrelateto
dominantsocialnormsabouttherolesandresponsibilitiesofmothersandfathers.Additionalmechanismsmaybe
situatedinpsychological,socialorotherspheres.Contextmatters:first,itinfluences‘reasoning’and,secondly,generative
mechanismscanonlyworkifthecircumstancesareright.Goingbacktoourexample,theremaydifferentsocialbeliefs
abouttherolesandresponsibilitiesofmothersandfathersindifferentcultures,whichmayaffecthowparentsrespondto
theparentingprogram.Whetherparentscanputtheirnewlearningintopracticewilldependonarangeoffactors–
perhapsthetimetheyhaveavailable,theirownbeliefsaboutparenting,ortheirmentalhealth.Finally,thecontextmay
providealternativeexplanationsoftheobservedoutcomes,andtheseneedtobetakenintoaccountduringtheanalysis.
viiAtypicalresponseisthatwomenactuallyenjoythistraveltime(andcarryingheavyburdensontheirhead)time
becausetheysingandchatwiththeirfriendsastheywalk.
viiiAfrequentlycitedexampleconcernstheroleofthemother-in-lawonchildnutrition.Themother-in-lawisfrequently
nottargetedinnutritioneducationprograms,butshecanpreventherdaughter-in-lawfromapplyingthelessonlearned
innutritioneducationprograms.
ixAnumberoforganizationssuchasUNWomen,USAID,DIFDandCAREinternationalhavedevelopedchecklistsforstaff
toassesshowwellgenderissueshavebeenaddressedinthedesignandimplementationoftheirprograms.Forexample,
DFIDstaffareaskedquestionssuchas:“Havewecountedallwomenandgirls?”,“Havebothwomenandmenbeen
consulted?”,“Haveweinvestedequallyinwomenandmen?”,“Dowomenandgirlsreceiveafairshare(ofprogram
resources?)”.