using perceptions data to monitor the post-2015 … · using perceptions data to monitor the ......

January 2015

developmentprogress.org

• Thecallfora‘datarevolution’hasspurreddebatearoundtheinclusionofnewdataandindicatorstomeasureprogresstowardsdevelopmentgoals.Indicatorsofperceptions–basedonaskingpeoplewhatmatterstothemmostandtheiropinionsofchange–couldhelptostimulatepublicdebateandholdpolicy-makersaccountable.

• Keystrengthsofperceptionsdataaretheirtimelinessandfrequency–suchattributescouldmakethemveryusefulaswarningsignalsforpolicyintervention.

• Weillustratethepotentialofperceptionindicatorsinthreepost-2015areas:socialnormsrelatedtogender,violenceandsecurity,andgovernance.Perceptionsandso-called‘objective’datacanmeasurecomplementaryaspectsoftheseareas.Analysinggapsbetweenperceptionsandobjectiveindicatorscanimproveunderstandingofhowpeoplearedissatisfied,orwhenthereareimplementationgapsinthepoliciesintendedtotackletheseareas.

• Mainlimitationsofthisdataarethechallengeofensuringthereliabilityoftheinformationobtained,anddifficultiesinmakingmeaningfulcomparisonsacrossgroupsofpeople.Wesuggestthatperceptionsdatawouldbemoreusefultomonitorchangingsituationsovertimewithincountries,ratherthantoestablishcomparisonsacrossthem.

Key messages

Working paper 413

Shaping policy for development odi.org

Asking people what they think Using perceptions data to monitor the post-2015 agendaLaura Rodríguez Takeuchi and Sébastien Hine with Cirenia Chávez

developmentprogress.org

odi.org

Contents

Acronyms and abbreviations 5

Acknowledgements 5

Introduction and context 6

The case for perceptions data 7

a) Perceptions data for guiding policy trade-offs 8

b) Perceptions data for accountability 10

Filling measurement gaps 14

a) Gender and social norms 14

b) Governance 15

c) Violence and security 17

Why not perceptions data? Common but differentiated challenges of perception surveys 19

a) Representation: whose responses are represented? 19

b) Reliability: how accurate are people’s responses? 21

c) Adaptation: have poor people come to terms with living in poverty? 22

d) Objective and subjective mismatch 23

Perceptions data in monitoring the post-2015 agenda: implementation issues 26

Conclusions 28

References 29

2 ODI Report

List of tables, figures and boxes

Tables

Table 1: Selected gender and social norms indicators 16

Table 2: Selected governance and effective institutions indicators 17

Table 3: Selected indicators to measure violence and insecurity 18

Table 4: Selected indicators to measure progress on eliminating violence against women and girls 18

Figures

Figure 1: Errors in the process of obtaining estimates from surveys 20

Boxes

Box 1: Income perception questions 8

Box 2: Adaptation, income poverty and subjective wellbeing 24

Asking people what they think 3

Acronyms and abbreviationsDHS Demographic and Health Surveys

FAO Food and Agriculture Organization

FGM female genital mutilation

GWP Gallup World Poll

HDI Human Development Index

HLP High Level Panel of Eminent Persons on the Post-2015 Development Agenda (UN)

IAEG-GS Inter-Agency and Expert Group on Gender Statistics (UN)

IEAG Independent Expert Advisory Group on a Data Revolution for Sustainable Development

IPFRI International Food Policy Research Institute

LSMS Living Standard Measurement Survey

MDG Millennium Development Goals

MICS Multiple Indicator Cluster Surveys

NSO National Statistical Office

OECD Organisation for Economic Co-operation and Development

OPHI Oxford Poverty & Human Development Initiative

OWG Open Working Group

SDG Sustainable Development Goals

SIGI Social Institutions and Gender Index (OECD)

SWB subjective wellbeing

UNDP United Nations Development Programme

WEIA Women’s Empowerment in Agriculture Index (IPFRI)

WVS World Values Surveys

Acknowledgements

WearegratefulforhelpfulcommentsandinputsfromEmmaSamman,JessicaHagen-ZankerandClaireMelamedfromODIaswellasfromLauraCamfield(UEA),HaykGyuzalyan(IpsosMori)andEnricoGiovanini

(UniversityofRome‘TorVergata’).NinaBehrmanprovidededitorialsupport.Theusualdisclaimersapply.

4 ODI Report

Introduction and context

TheprocessofdevelopingandassessingprogressagainsttheMillenniumDevelopmentGoals(MDG)hasbeencriticisedastechnocraticandtoomuch‘behindcloseddoors’.Incontrast,negotiationsaroundthesuccessorstotheMDGhavebeenwide-ranging,seekingtobeconsultativeandincorporatingtheviewsofallgovernmentsandofcitizensglobally.Thisspiritofmakingthepost-2015processmoreinclusiveandresponsivetocitizen’sviewsanddemandsshouldbecarriedintothemonitoringofthenewdevelopmentgoals.

Perceptionsdataareincreasinglywidelyavailable,andcouldbeusefulinmonitoringprogresstowardsnewdevelopmentgoals.Availabledataincludethosefrom:

• theWorldValueSurveys(WVS)conductedinover80countriesandtheGallupWorldPoll(GWP)coveringover160countries

• the‘barometers’(LatinAmerica,Asia,ArabregionandAfrica),theLatinAmericaPublicOpinionProjectandtheEuropeanSocialSurvey,whichhaveamoreregionalfocus

• questionsinLivingStandardMeasurementSurveys(LSMS),whichareinternationallycomparablemulti-topichouseholdsurveysconductedindevelopingcountries

• subjectivewellbeingdata(SWB)collectedbyNationalStatisticalOffices(NSOs)insomecountries1.

Whileexistingdatacouldbeusedascriteriafornarrowingdowntheplethoraofproposedgoalsandtargets,thevastinterestinexaminingnewformsofmeasuring–whattomeasure,andhowtomeasureit–providesagoodopportunitytoexaminenewpossibilities.TheUNHighLevelPanelofEminentPersonsonthePost-2015DevelopmentAgenda(HLP)hascalledfora‘datarevolution’,andtherecentreportoftheUN’sExpertGroupontheDataRevolution(IEAG,2014)promptedwidedebateonpossiblenewapproachesandincorporatingnewmethodsofcollectingandusingdifferenttypesofdata.

‘[A data revolution is] an explosion in the volume of data, the speed with which data are produced, the number of producers of data, the dissemination of data, and the range of things on which there is data, coming from new technologies such as mobile phones and the “internet of things”, and from other sources, such as qualitative data, citizen-generated data and perceptions data.’ (IEAG, 2014)

TheSecretary-General’ssynthesisreport(UN,2014b)alsoclearlyunderlinestheimportanceofsubjectivedatainmeasuringwellbeing.AsarguedbytheStiglitz,Sen,FitoussiCommission(2009),perceptionsdatacouldshiftthemeasurementdebatetowardsimprovingwellbeingofcitizensaroundtheworldastheyunderstandit.Nevertheless,toourknowledge,thereisnospecificassessmentofthepotentialofsubjectivedata,orthechallengesinvolved,inthiscontext.Thispaperaddressesthisissue,framedinthecontextofthemonitoringofthenewsetofdevelopmentgoals.Italsooutlinessomemethodologicalissuesmorebroadly.

Section2definesperceptionsdataandpresentsgeneralinitialthoughtsontheirpotential.Section3considersthepotentialofperceptionsdataformonitoringtheevolutionofgendernorms,violenceandsecurity,andgovernance.Section4concentratesonthechallenges,andinSection5wefocusonpracticalconsiderationsfortheinclusionofperceptionsdatainthemonitoringofpost-2015developmentgoals.

1 NationalStatisticalOfficesincludingSWBmeasuresare:Canada,France,Italy,Mexico,Morocco,NewZealand,UK,US,andmostEUcountries(O’Donnelletal.,2014).


The case for perceptions data

Perceptionsdata,oftencalledsubjectivedata,concerntheexpressedviews(opinionsorperceptions)2ofpeopleonaparticulartopic.3Incontrast,so-called‘objectivedata’arereportsofmaterialpossessions,eventsorsituationscollectedthroughdirectphysicalexaminationorobservation.Werefertothissecondkindofdataas‘objective’inthispaper,althoughwerecognisethattheycanalsobesubjecttobiasesanddifferentinterpretations.

Thedefinitionweadoptalsoexcludesself-reportedobjectivedata,orwhatsurveyresearcherscallanswersto‘factual’questions.Theserefertoinformationonpastactionsorobjectivelivingconditions,forexamplevotingbehaviourorself-reportedincome,althoughbothmaybecollectedthroughthesamemeansofsurveyquestionnaire.Thisdistinctionisimportant.Takeforexample,aquestionaskinganindividualtoreporttheirhouseholdincome(eitherasanexactfigureorwithinasetrange).Thistypeofquestionisself-reported,ratherthancollectedforexamplethroughtaxrecords,butitisnotaboutperceptionsasdefinedhere.

Whileclearlytherearesomeobjectivelymeasurablethingssuchasanamountoffood(orthenecessaryincometopurchasethatfood),thatpeopleneedtoleadfullandhealthylives,peoplecanalsomakesubjectiveassessmentsoftheseobjectiveconditions.Themainadvantageofperceptionsdataisthattheyprovide‘therespondents’ownviewsdirectly…perceptionsofthemselvesandtheirworld,whichareunobtainableinanyotherway’(Bakeretal.,2002).Forexample,aquestionabouttheperceptionofthehouseholdincomesufficiencywouldbeincludedinourdefinitionofperceptionsdata(Box1).Thesesubjectivejudgementsmatterregardlessoftheobjectivecircumstancesbecausepeoplemayreacttoitiftheycan–withconsequencesforhowtheyrespondtoincentives,andtopoliticalsystemsandprocesses,andforhowpeople

relatetoeachother(Coulthardetal.,2014).Moreover,thereareotherthingsthatarebestassessedonlyinsubjectivetermsbecausewhatreallymattersishowpeopleexperiencethese(Veenhoven,2004;GoughandMcGregor,2009;Coulthardetal.,2014).Veenhoven(2004)usestheexampleofxenophobia.Toadegree,thiscanbemeasuredbyareductionofracistattacksortheratesofinterethnicmarriage,forinstance,butperceptionsareneededtocapturethisconceptfully.Similarargumentsapplyfortheconceptsofhumandignityandsecurity(Coulthardetal.,2014).Perceptiondataarecollectedmainlythroughhouseholdsurveys.ThispaperfocusesoninternationallycomparabledatathatcouldbeusedformonitoringtheagreedSustainableDevelopmentGoals(SDG)/post-2015targets.Somemulti-topichouseholdsurveysincludeselectedquestionstoascertainperceptionsaroundspecificissues.Subjectivemodulescanbeappendedtobroadhouseholdsurveys,asinthe‘missingdimensions’projectoftheOxfordPoverty&HumanDevelopmentInitiative(OPHI).4Therearesomespecialisedsurveysfocusingexclusivelyoncapturingperceptionsdata.TwoofthelargestofthesearetheWVSandtheGWP,conductedinawidenumberofcountriesaroundtheworldfollowingastandardmethodologyandquestionnaire.Theyaskarepresentativesampleofparticipatingcountryresidentsabouttheirperceptionsontopicsrangingfromgovernmentandpolitics,tofamily,religion,ethicsandwellbeing.Theregionalbarometersurveysaresimilarbutthespecifictopicsandquestionsmayvaryintheregionalsurveys.

TheUNMyWorldsurveyisanotherlarge-scaleperceptionsurvey.Resemblingmoreashortopinionpollthanahouseholdsurvey,itconsistsofonlyonecorequestion(‘Which of these are most important for you and your family?’5),aimedatgatheringpeople’sprioritiesforglobaldevelopmentpost-2015.Inaddition,althoughthere

2 Evenwithinthedefinitionadopted,therecanbevarioustypesofdata.Adistinctioncouldbemadebetweenquestionsthataskaboutperceptions(‘howdoyoufeelabout…?’),opinions(‘doyouthinkthat…?’)andchoices(‘shouldthegovernmentdo...or…?’).Themainemphasisisonthefirsttwoofthesetypes,especiallysincethethirdmayhavedifferentmethodologicalproblems.Thatsaid,someissuesarecommontoallperceptionsdata,andevensometoallestimatesbasedonsurveymethods.

3 Experts’valuationscouldbeclassifiedasperceptionsdata,insofarastheyrefertoopinions/perceptions.Althoughexpertviewshavebeenwidelyused,forexampleinthemeasurementofgovernance,weexcludethemfromthedefinitionusedinthispaperasourmaininteresthereisperceptionsdatadrawnfrom(mostlyrepresentative)samplesofthegeneralpopulationratherthanfromselectedgroupsofexperts.

4 http://www.ophi.org.uk/research/missing-dimensions/

5 Thesurveyasksrespondentstochoose6issuesthatmattermostoutof16possibilities.

6 ODI Report

http://www.ophi.org.uk/research/missing-dimensions/

aresomenationallyrepresentativeMyWorldsurveys,formostpartthelargesampleofMyWorld–over7million–hasbeenpossiblebecauseithassoughttobeatrulyworldwidesurveyandcoverawiderangeofcountries,ratherthantobenationallyrepresentative.

a) Perceptions data for guiding policy trade-offs Perceptionsdatacanplaymultiplerolesinpolicyprocesses.Seaford(2013)definessixrolesforsubjectivewellbeingdata:tostimulatepublicdebate,todrawattentiontoimportantissueswhichmayotherwisebeignored,toinformthedevelopmentofeconomicmodels,toinfluencethechoiceofotherobjectiveindicators,toinputintoanewformofcost–benefitanalysis,6andtoholdpoliticiansaccountable.Theserolesapplymorebroadly


Box 1: Income perception questions

Householdorpersonalincomeisoftenusedasameasureofwelfareandsubsequentestimatesofmonetarypovertyandinequality.Itisalsopossibletoderivepovertyandinequalitymeasuresfromsubjectiveperceptionsofrelativewealth/povertyratherthanactualincome.Somehouseholdsurveyshaveincorporatedthistypeofquestion,forexample:

• Howdoesyourhouseholdincomecomparewithotherhouseholdsinyourvillage/neighbourhood?MuchaboveaverageincomeAboveaverageincomeAverageincomeBelowaverageincomeMuchbelowaverageincome(Donotknow)

• Whatmonthlyincomeleveldoyouconsidertobeminimalforyourhousehold,i.e.yourhouseholdcouldnotmakeendsmeetwithless?

• Isthetotalmonthlyincomeofyourhouseholdhigher,lowerormoreorlessthesameasthisfigure?MuchhigherHigherMoreorlessthesameLowerMuchlower

(SouthAfricanSocialAttitudesSurvey,2007)

• Yourhouseholdincome:IsnotenoughtomeettheminimumexpenditureIsjustenoughtomeettheminimumexpenditureIsabovetheminimumexpenditureDon’tknow/NA

(EncuestadeCalidaddeVidadeBogotá,2007)

• Isyourpayjust?Wearenotaskingabouthowmuchyouwouldliketoearn–butwhatyoufeelisjustgivenyourskillsandeffort.Ifyouarenotworkingnow,pleasetellaboutyourlastjob.

MuchlessthanisjustAlittlelessthanisjustAboutjustformeAlittlemorethanisjustMuchmorethanisjustNeverhadajobCan’tchoose

(ISSPSocialinequality-IV,2009)

• Concerningyourfamily’stotalincomeoverthepastonemonth,whichofthefollowingistrue?Itwaslessthanadequateforyourfamily’sneedsItwasjustadequateforyourfamily’sneedsItwasmorethanadequateforyourfamily’sneeds(TanzaniaKageraHealthandDevelopmentSurvey(LSMS),2004)

toothertypesofperceptionsdata.Welookindetailatthefirsttwointhissection,andtouchonsomeoftheotherrolesinSection2b.

Inthecontextofthepost-2015goals,therehasbeenasystematicattemptincludeassessmentsandperceptionsofdifferentgroupsintheprocessofdefiningthenewgoals.Butbeyondthisparticipatorydecision-makingprocess,thereisgreaterscopeforperceptionsindicatorstoguidepolicy.Whilewellbeingdataindevelopedcountrieshavebeenusedincreasinglytosupportpolicies,theyhavebeenlessusedtodrivethepolicyagenda(Seaford,2013).

Perceptionsdatacanhelptodrivethepolicyagendabyprovidinginformationonpeople’spriorities.Thiscanhelpanalysingtheinteractionsbetweendifferentpathwaysfrompoliciestowellbeing.Forexample,thisdatacanhelptomakevisiblehowpoliciesinhealth,forexample,arecomplementarywiththoseineducationorinsecurityandstimulatepublicdebateontherightmixofpoliciesappropriateinagivencountrycontext.Thisalsopointstothetrade-offsbetweenconflictingobjectivesoramultiplicityofobjectivesthatneedtobeprioritised.

Examplesinthepost-2015contextandbeyondshowhowsubjectiveindicatorscanfulfilthisrole.TheOECD‘How’slife’initiativeandtheBetterLifeIndex7areexamplesofthelatter.Theyidentify11dimensionsofmaterialwellbeingandqualityoflife8forOECDcountriesandothermajoreconomies.Theindexisaninteractiveweb-basedtoolthatinvitespeopletoexpresstheimportancetheygivetodifferentdimensionsofwellbeing.9Thekeyisthatthisexerciseallowsrespondentstocontrastcountriesaccordingtotherelativevaluetheyplaceoneachdimension,ratherthanbyhavingastaticpredeterminedcombination.Moreover,itallowsforanillustrativecomparativeanalysisofhowtheseprioritiesdifferacrosspeoplefromdifferentcountries,ageandgender.10People’sresponsesmakeitpossibletounderstandhowthehighimportanceattributedtohealth,forexample,isrelatedtootherdimensionsofwellbeingandtotheobjectivequalityofhealthcaresystemsinparticularcountries.Inthissense,theworkonnewmeasuresofprogress,suchastheBetterLifeIndex,isusefulinconsideringpublicprioritiesandstrategiesinpolicy-making(UNTaskTeam,2013),

identifyingpoliciesthatwouldcontributemoretooverallwellbeing.

Similarexercisescanbecarriedoutwithmoretraditionalmeasuresofwellbeing,toindicatewhatpeoplevalueandtowhatextent,andtoassesshowwellgovernmentsarerespondingtothosetrade-offsandfulfillingcitizens’expectationsofwellbeing.Forexample,Kroll(2013)proposesameasureofcountryperformanceinhumandevelopmentthatcombinestheobjectiveindicatorsoftheHumanDevelopmentIndex(HDI)butweightedaccordingtotheircontributiontoSWB.ThismeasureassessesthecountrycorrelationofthethreeHDIcomponents(materialconditions,healthandeducation)andpeople’sresponsesinalife-satisfactionquestion.BymeasuringwhichofthethreeHDIelementsaremorestronglyrelatedtolifesatisfaction,itbecomesclearwhichcountriesaremoresuccessfulingeneratingthegoodsthattrulymattertopeople’swellbeing.Krollfindssignificantheterogeneityintheimportanceofthesethreefactorsacross70countries.Forexample,inMoldova,thereisthehighestcorrelationbetweenSBWandincome.Incontrast,inFinlandorTurkey,thecontributionofincometoSBWislow.

Inhealth,useoftheQualityAdjustedLifeYears(QALY)andofPatientReportedOutcomeMeasures(PROMs)isapracticalexampleofhowtheseperceptionsontrade-offscanbeincorporatedinthedecision-makingprocessofpolicy(Melamedetal.,2012).PioneeredintheUK,theexerciseinvolveselicitingthe‘patient’sperspective’throughthousandsofsurveys(knownastheEQ5D11)askingpeopleabouttheirhealth.Foreachofthefivehealthdimensions,individualsratetheirhealthonthedaywhenthequestionnaireiscompleted–noproblems,someproblemsorsevereproblems–andvalueeachofthepossiblehealthoutcomes.12Foranygiventreatment,policy-makersneedtoknowhowmuchvalueisgained,forhowlong,andatwhatcost.OneQALYisequaltooneyearlivedinfullhealth.Ayearinanythinglessthanfullhealthisvaluedatlessthanone,dependingonthesocialvalue(derivedfromthePROMSprocessdescribedabove)attachedtothatparticularoutcome.Throughthisexercise,itispossibletoassessthetrade-offbetweenhealthtreatmentsthatimprovedifferentaspectsofhealth.Foranygiventreatment,thenumberofQALYsgained(eitherthroughimprovementsin

6 TheUKNationalHealthService(NHS)useofPatientReportedOutcomeMeasures(PROMS)asoneinputintomeasuringtheperformanceofdifferenthealthproviders/serviceswouldbepartofthiscost–benefitanalysis.

7 Seehttp://www.oecd.org/statistics/howslife.htmandhttp://www.oecdbetterlifeindex.org/

8 SelectedtomatchthoseproposedbyStiglitzetal.(2009),agreedbyOECDcountriesandreflectingdimensionsofwellbeingthatareuniversalandrelevantforallhumanbeingsratherthanspecificforaparticularcountry.Theyare:community,education,environment,civicengagement,health,housing,income,jobs,lifesatisfaction,safetyandwork–lifebalance.

9 Usershavetorateeachofthe11topicsfrom0(‘notimportant’)to5(‘veryimportant’).

10 Althoughtobeprecise,representativesamplesofthepopulationwouldberequired.

11 Eachofitsfivesectionscorrespondstoadifferenthealthoutcome:mobility(abilitytowalkaboutnormally);self-care(abilitytolookafteroneself);usualactivities(abilitytoperformusualactivities);painordiscomfort;andanxietyordepression.

12 Thereare243differentpossibleoutcomestothequestionnaire(threepossibleoutcomesoneachoffivedimensionsofhealth).

8 ODI Report

http://www.oecd.org/statistics/howslife.htm

http://www.oecdbetterlifeindex.org/

qualityorlengthoflife,orboth)canbecomparedwiththetotalcostsofthetreatment(Melamedetal.,2012).

Trade-offsinbudgetallocationsareespeciallyvisible.OneofthesuccessesoftheMDGwastoincreasedevelopmentaidfromdevelopedtodevelopingcountries(Clemensetal.,2007)buttherearedifferencesinwhatdifferentgroupsofpeopleperceivetobethebestuseforthoseresources.Usingperceptionsdatatoillustratehowpeopleperceivetrade-offscanbeparticularlyusefulininformingandmonitoringbudgetdecisionsofdonorcountries,acomparisonthatisinterestinginthepost-2015contextwhereasimilarexerciseofassigningbudgetprioritiesmaybeneeded.Leo(2013)investigateswhetherUSaidisprioritisingthoseissuesthatmattermosttopeopleinaid-receivingcountriesinAfricaandLatinAmerica.RegionalBarometersaskwhatpeoplebelievearethemostpressingproblemsfacingtheircountry.Only16%ofUSassistancehasbeenfocusedonwhatAfricansciteastheirtopthreemostpressingproblems(jobs,incomeandinfrastructure)since2002;nearly60%hasbeentargetedtowardssecondaryortertiaryconcerns(Leo,2013).Bycontrast,Prizzon(2014)findsthat,inEthiopia,CambodiaandZambia,citizens’preferencesmaynotbenecessarilyalignedtothetopprioritiesoftheirgovernments(infrastructure,energyandgrowthcreation)butthattheallocationofaidflowsfromdonorsdoesappearinthosecountriestobecloselyalignedtocitizens’preferences.

Inthecontextofdefiningthenewsetofdevelopmentgoals,informationonpeople’spreferencescouldbeextremelyhelpfulinestablishingprioritiesforimplementingthe17goalssofarproposedbytheOpenWorkingGroup(OWG).BothGallupandtheWVSaskedpeopletoprioritisetheMDGgoalsmostimportanttothem.In26countriesofsub-SaharanAfrica,accordingtothe2006Galluppoll,reducingpovertyandhungerwerethehighestranked,followedbyreducingthespreadofHIV,andprovidingjobs,withaveryconsistentpatternbypopulationsubgroupsbutsomevariationbycountry,particularlyinWestAfrica(Tortora,2009).

Otherinterestingmessagesemergefromperceptionsdatathataskpeopletoprioritiseglobalandcountrygoals.IntheWVS,amongotherproblems,‘peoplelivinginpoverty’wasoverwhelminglyrankedtopacrosscountriesasthemostseriousproblemfortheworld.However,alowershareofpeopleconsidereditthetopcountrypriority.13Moreover,whenaskedaboutwhetherthecountryshouldfocusonsolvingitsownproblemsorhelpingtoreducingpovertyintheworld,mostpeoplethinktheircountry’sproblemscomefirst.14Eveninthemostaltruisticcountries,15themajorityofpeoplewouldprefertoconcentrateontheirowncountryprioritiesrather

thanonglobalones.Theinclusionofsustainability,theenvironmentandclimateinthenewsetofdevelopmentgoalshasbroadenedthescopeoftheSDGbeyondthenationalspectrum.TheinformationpresentedintheWVSalignswiththeexperienceofMDG8‘globalpartnership’(KennyandDykstra,2013),suggestingthatitisgoingtobehardtoensureglobalcommitmenttoissuesthatareglobalinnature,orthatrequirecoordinationandcooperationfromvariouscountries.Apartfromthecoordinationissuesandthealignmentofpoliticalviewsbygovernmentsofdifferentcountries,ensuringpublicsupportwithincountriestotacklethoseissueswillnotbeeasy.

Insum,thevalueofsubjectiveindicatorsisthattheycanmakeexplicittheimplicitassumptionsthattypicallyguidepublicdebate,mainlyaroundtheissuesthatpeopleconsiderimportantinimprovingtheirlives(Seaford,2013;RodriguezTakeuchi,2014).Thiscanbeveryhelpfulininformingpolicydebatesbutmaynotsettlethem(Seaford,2013).Moreover,whileitisimportanttoknowtheseoverallpreferences,perceptionsdatacanalsobeusedtoassesstrade-offsfordifferentgroupsofpeople,forexampletoseethedifferencesbetweenwhatmenandwomen,oryoungerandolderpeople,thinkcouldmostimprovetheirwellbeing.Inthissense,perceptionsdatacandrawattentiontoimportantissuesorgroupsofpeoplewhichotherwisemaybeignored,becausetheywerenotpreviouslyidentifiedasaconcern(Seaford,2013).

b) Perceptions data for accountability‘To ask the people, with regularity, for their own thoughts strikes us as being both useful and a check on the claims of those in power’ (Dionne and Mann, 2003)Thereisnosingleagreeddefinitionofaccountability.Itinvolvesbothanswerability–theresponsibilityofduty-bearerstoprovideinformationaboutandjustificationoftheiractions–andenforceability–thepossibilityofpenaltiesorconsequencesforfailingtoansweraccountabilityclaims,althoughthissecondaspectisoftenleftasideintheunderstandingofaccountability(McGeeandGaventa,2011).Thereisstillmuchdebateabouthowtotranslateinformation,transparencyandparticipationintoaccountabilityandimprovedoutcomes,andwhether,infact,thereisaclearpositiveconnection(McGeeandGaventa,2011).Nevertheless,theverynatureofperceptionsdata–askingpeoplewhattheythinkonarangeofdimensionsoftheirwellbeing–hasthepotential

13 63%consideritaglobalpriorityand58%anationalone.

14 Inascalefrom1to10,theaveragescorewas7.8.About39%ofrespondentsmarked10(‘itisatopprioritytosolvemyowncountry’sproblems’).

15 Finland,Iran,Italy,Malaysia,Norway,Mali,Mexico,Spain,SwedenandSwitzerland,accordingtothesurveyresults.


togenerategreateraccountability.Thisworksintwoways:first,byopeningthedebatearoundmeasuringandactingonwhatistrulyimportanttopeople;and,second,byincreasingtheavailabilityofinformationandhence,opportunitiestotrackprogress.

Accountabilitystartswiththecreationofsystemsthatallowit.Makingsuretomeasurethingsthatpeopleactuallythinkareimportantandrelevantisafirststep.Thereisawidedebateonwhethersocialaccountabilitycanoccurbymakingrules,proceduresandplanstransparentandopenfordiscussionbeforetheyareimplemented(McGeeandGaventa,2011).Inthissense,perceptionsdatacouldbeusedtoopenthisdiscussionandeveninfluencethechoiceofindicatorsfortrackingoutcomesthatgovernmentswillbeheldaccountableforuntil2030.EvenifperceptionsdataarenotdirectlyincorporatedinthemonitoringindicatorsoftheSDG,theycaninfluencethechoiceanddesignofotherindicatorsandhighlightimportantissuesexpressedbypeoplewhichmayotherwisebeignored.

Thisexercisemayalsorevealthatcommonlyusedobjectiveindicatorsmaynotbereflectingtheappropriateissues.Forinstance,thereoftenseemstobeamarkeddistancebetweenstandardmeasuresofimportantsocioeconomicvariableslikeeconomicgrowth,inflationandunemployment,andwidespreadperceptionsofcountries’developmentprogress(Stiglitzetal.,2009).Insomecountries,thesegapshaveunderminedconfidenceinofficialstatistics,16andinotherstheyhaveresultedinapartialifnotmisguidedanalysisofcountries’andpeople’ssituations.

‘All institutes worldwide knew GDP was rising in Tunisia and Egypt. They knew what 11 million Tunisians and 80 million Egyptians were buying and selling — but they didn’t know what they were thinking. As a result, revolutions in those countries came as a shock. The UN didn’t see those revolutions coming, neither did the WEF nor the World Bank. The U.S. spends tens of billions on intelligence — and it missed those revolutions too.’ (Gallup, 2012)

Perceptionsdatacanthuscontributetothesocialrelevanceofindicatorsbyhelpingtodiscernthemostrelevantissuesforpolicy-makerstoaddress–forexample,byidentifyingobjectiveindicatorsthatbettercorrelatewithpeople’ssubjectivewellbeing.Seaford(2013)points

outthattomeasureacountry’seconomicprosperity,forexample,GDPpercapita(meanGDP)couldbereplacedbymedianincome,whichtakesintoaccountdistributionalissuesandhasagreaterimpactonwellbeingthanaverage(mean)income.Similarly,lookingattherelationshipbetween,say,perceptionsofsafetyandarangeofindicatorsonsafetyandsecuritymaybeusedtoinformtheselectionofanobjectiveindicatortomonitorapost-2015targetonsecurity.Moreover,perceptionsindicatorsaresociallyrelevantbecausetheycollectvaluableinformationondomainsofwellbeingthatareintrinsicallysubjective(GoughandMacGregor,2009)andthusdifficulttomeasurewithobjectivedata.

Inthisway,perceptionsdatacanhelptomakegovernmentsaccountableforsocialprogress,definedintermsofcontributorstomultidimensionalwellbeing,ratherthananarrowsetofoutcomes.Usingsubjectivedatathenmightimprovetheaccountabilityofglobalandnationaldecision-makerswithrespecttotheoverallaimofthepost-2015goals:toimprovepeople’slives.

Inthespecificcaseofindicatorsofgovernance,perceptionsdatacanbeusedtoholdgovernmentsaccountablefordemocraticprogress.Whilemanyobjectivemeasuresofgovernancetendtofocusonformalproceduresorinstitutions(e.g.whetherandhowoftenelectionsareheld,theformalseparationofpowers,theexistenceofananticorruptioncommission),governanceprogressrequiresmeasuresalongvariousdimensions(Forestietal.,2014)thatunderpintherelationshipsbetweenthestateandsociety.Perceptionsdatamayaddtheperspectiveofthosepeoplewhomgovernanceissupposedtobenefit.

Corruptionisakeyexample.Askingpeoplewhethertheythinkofficialswhocommitcrimesgounpunished,orfindingoutthenumberoffirmsthatreportthatinformalgiftsorpaymentsareexpectedtoobtainservices(UNDP,2014),providesinformationonhowpeopleexperiencecorruptionratherthanjustaboutprevalencelevels.Otherperceptionmeasuresimportantintermsofaccountabilityandgovernanceincludetheextenttowhichcitizensfeeltheyareabletoparticipateintheselectionoftheirgovernments,havefreedomofexpressionandliveinacountrywheremediaarefreeandofferavarietyofviews(Thomas,2009).Ifthesemeasuresareproperlyintegrated,theygiveacriticalvoicetopeoplewhopreviouslyhavehadlimitedornovoiceandhadthereforebeenexcludedfromaccountabilitymechanisms.Theimportantpointhereishow,andperhapsif,thedataareused.Dataalonearenotanaccountabilitymechanism.Butifthedataareappropriatelyusedby,forexample,freemediatoraiseissueshighlightedinpeople’sperceptionsthenthiscanmakeadifferencetohowpoliticiansactandrespondtopeople’sneeds.

16 Forexample,inFranceandintheUnitedKingdom.onlyonethirdofcitizenstrustofficialfigures,andthesecountriesarenotexceptions(Stiglitzetal.,2009).

10 ODI Report

Perceptionsdata,especiallyincombinationwithnewmethodsofdatacollection,canalsofillanotherimportantgapinmeasuringoutcomestotrackprogresstowardsthepost-2015targets:theavailabilityandtimelinessofdata.Increasingthefrequencyandcoverageofdataisakeycomponentofthedatarevolution.Thisiscrucialnotonlytoensureaccuracyofthestatisticsusedtoshowprogress(orlackofit),butalsotounderstandtheimpactsofshocks,suddenchangesinwellbeingandinstabilitysituations.Inparticular,perceptionsdatacanactassocialmonitorsandpointtochangesintrends,beforethosearereflectedinothertypesofstatistics.Forinstance,thesubjectivefoodinsecurityscaleoftheFoodandAgricultureOrganization(FAO)showsthepotentialofsuchindicatorstodetectearlyonsetofmalnutritioncrises.Unlikeameasureoffoodavailabilityfromagriculturalproductionortrade,oranthropometricindicatorssuchasunderweightorstuntingprevalence,whichevolveonlyaftermalnutritionbecomesmanifest,aperception-basedfoodinsecurityindexcanbeusedtoidentifyvulnerablepopulationsinatimelyway(Ballardetal.,2013)andevenhelptoidentifyseasonalitypatterns(HeadeyandEcker,2013).

ThissameideawasbehindtheSocialWeatherStation(SWS)project,oneofthefirstperceptionssurveysinthedevelopingworld–theideathat‘surveyscanservelikeobservationpoststomonitorsocialconditions,muchasmeteorologicalstationsmonitorweatherconditions’(Mangahas,2013).Assuch,perceptionsdataofferauniqueopportunitytocapturetheevolutioninpeople’sperceptionsandpotentiallyserveasan‘earlywarningsystem’(AlkireandSamman,2014)17fortheneedforpolicyintervention.Infact,anexplorationoftheSWSself-reportedpovertydataandtheofficialpovertystatisticsofthePhilippinesrevealthat,whileself-reportedpovertyandofficialpovertystatisticsdiffergreatlyintermsoflevels,thetrendstheydescribearebroadlyconsistent(AlkireandSamman,2014).

AlthoughsomeheadlineobjectiveindicatorssuchasGDParetrackedveryregularlyinmoredevelopedcountries,insomeothercountriesequivalentdatacanbeunreliable(seeforexampleJerven’s(2103)argumentaboutcorestatisticsinAfrica)ormuchdelayed.WithrespecttoMDGindicators,forexample,themostrecentglobalreportincludesdatauptofouryearsold:‘Thepovertyfiguresgobackto2010.We’renowinthemiddleof2014,sothat’squitefrustrating’(KeikoOsaki-Tomita18inSciDev,2014).

IncreasingthefrequencyofsurveyscanbecostlyandhighlydemandingforNSOswithlimitedcapacity.Morecompletesourcesofdatasuchascensusestendtobeveryinfrequent,oftenconductedonceeverydecade.Household

surveyshavebeenrisinginnumberbutarestillrelativelylimited,especiallyinlow-incomecountries.Chandy(2013)forexamplefindsthat,inlow-incomecountries,householdsurveysusedtomeasuremonetarypovertyhavebeencarriedoutonaveragejustfourtimessince1980.Similarly,DemographicandHealthSurveys(DHS),keysurveysusedfortrackingMDG,havebeencarriedoutin89countriesfromthelate1980stodate.However,ofthose89countries,26haveonlybeensurveyedonce.19Incontrast,currentperceptionsdatasurveysarecollectedmorefrequentlyandthushavethepotentialtobecollectedinatimelymannerandfrequentlyinawiderangeofcountries.TheSWSprojecthastrackedFilipinos’perceptionsatquarterlyintervalsforover30years.TheGWP,forexample,hasacoverageof160countriesandisfieldedannuallyin93ofthem(AlkireandSamman,2014).Thismaybepartlybecausecollectingperceptionsdataissimplerandlesscostly(seeBallardetal.,2013;HeadeyandEcker,2013,inthecontextoffoodsecuritydata).However,addressingsomeofthereliabilityissues(asdiscussedinSection4)requirestheapplicationofmethodologicalrigour,interviewertrainingandensuringtheappropriatesurveyprotocols,allofwhichcanincreasethelengthandcostofperceptionsurveysinthefuture.Thecostandtimeinvolvedincollectingperceptionsdataisalsocontingentinthespecificityoftheinstrumentsandwaysofdatacollection.20

Finally,acombinationofbigdataandsubjectivedatacanbeusedtoextrapolateharddata,for‘nowcasting’.Thisisimmediate-term(‘now’)forecastingonthebasisofreal-timedataflow(WorldBankandSecondMuse,2014).Thistechniqueisusedbystatisticalofficestofillinthegapsofserieswithlowfrequency,suchasinforecastsoffoodpricesordiseaseonsetusingGoogletrendsanalysis.Forexample,inthecontextofperceptionsdata,ajointWorldBankandUNGlobalPulsepilotexploredthepotentialofsocialnetworkcontent(fromTwitter)toanalysepublicperceptionsofagassubsidyreforminElSalvador.Thestudyusedtextanalyticstoseeiftheresultsfromthesocialmediaanalysis,whichcanbetrackedwithhighfrequency,matchedpublicopinionasmeasuredthroughhouseholdsurveysconductedbeforeandafterthereform.PreliminaryresultsconfirmedthatTwitterdataprovidedausefulcomplementtoanalysethepublicperceptionofapolicyreform(WorldBankandSecondMuse,2014)andthuscanbeusedtocapturetheevolutioninpeople’sperceptions,potentiallyservingasawarningsystem,asmentionedabove.Thiscouldpointtotheneedforpolicyinterventionbeforethefullhouseholdsurveycomesintoplace.AsimilarcasecouldbeenvisionedforexampleforaUNICEF

17 Cummings(2009)suggestedasimilarargumentforsubjectivewellbeingdata.

18 ChiefofthedemographicandsocialstatisticsbranchoftheUN’sStatisticsDivisionandleadauthorofthe2014UNMDGreport.

19 http://dhsprogram.com/data/available-datasets.cfm

20 Theuseofnewtechnologiesmayhelptoreducecostsandtime,althoughthisdiscussionfallsbeyondthescopeofthispaper.


http://dhsprogram.com/data/available-datasets.cfm

projectmonitoringsocialmediaandpublicblogpoststotrackparents’attitudestowardsvaccination,particularlyvaccinehesitancyinEasternEurope.High-frequencydataontheseattitudeswouldbeveryvaluableinplanning

interventionbeforetheprogrammecomestoanend,whenverylittlecouldbedonetochangeadverseperceptionsofchildvaccination.

12 ODI Report

Filling measurement gaps

Thissectionconcentratesonthepotentialofperceptionsdatatoguidepolicyandmonitorprogressforparticularpost-2015targetswhereexistingdatagapsaresizeable.Basedonananalysisoftheexistingproposalsforpost-2015,21whereuseofperceptionsdatahasbeensuggestedaspartofthemeasurementstrategy,andtheassessmentoftheUNtaskteamonindicators(UNTaskTeam,2013:ix),weidentifiedthreeareaswherethistypeofindicatorcanbeofparticularrelevancetosupplementobjectiveindicatorsandprovideafullerpicture.Theseare:a)genderandsocialnorms,b)governanceandservicedelivery,andc)violenceandsecurity.Thissectionthisfocusesonhowperceptionsdatacanexpandthepossibilitiesformeasurement.

a) Gender and social norms Goal3oftheMDGisto‘Promotegenderequalityandempowerwomen’.Themaintargetfocusesoneliminatinggenderdisparitiesineducation,withcomplementarytargetsongapsinwageemploymentinthenon-agriculturalsectorandonpoliticalrepresentation.Traditionalgenderindicatorsmeasureprogressonoutcomesofparticularrelevanceforwomenoroutcomegapsbetweenmenandwomen.Examplesarematernalmortality,earlymarriageandfemalegenitalmutilation(FGM)rates.Thesecondtypeofindicatorinvolvesdisaggregatingexistingindicatorsbygender.MDGgenderindicators,forexample,areofthissecondtype:theshareofwomeninnationalparliaments,genderparityineducation,andtheshareofwomeninnon-agriculturalwageemployment.Athirdtypeofindicatorrelatestothecommitmentofnationalgovernmentstoensuringgenderequality.TheInter-AgencyandExpertGrouponGenderStatistics(IAEG-GS)measures11ofthosecommitmentsthroughtheexistenceoftherelevantlegislationorratificationofinternationalconventionsandcommitments.

Mosttraditionalgenderindicators,whetheroutcome-basedorprocedural,arebasedonobjectivedata.ThesetofminimumgenderindicatorsrecommendedbytheIAEG-GSincludes52indicators;22noneisbasedonperceptionsdata.Nevertheless,inthecontextofthenewsetofdevelopmentgoals,achievinggenderequalityandempowermentmayrequirenotonlytrackinggendergapsinoutcomesin2030,

butalsomonitoringtheevolutionofsocialnorms.Socialnormsaredefinedas‘informalandformallaws,beliefsandpracticesthathelptodeterminecollectiveunderstandingofwhatareacceptableattitudesandbehaviours’(Harperetal.,2014:2),andareakeycomponentofprogresstowardsgenderequalityandempowerment.

Inthepost-2015discussions,genderhasfeaturedstronglyingoalsproposedbyboththeHLPandtheOWG.IntheOWGproposal,thegoalisstatedasto‘Endallformsofdiscriminationagainstallwomenandgirlseverywhere’,andtheHLPreportstatesthat‘thesebarriers(toopportunitiesforwomenandgirls)canonlyberemovedwhenthereiszerotoleranceofviolenceagainstandexploitationofwomenandgirls’(HLP,2013:17,italicsadded).Someoftheseformsofdiscriminationwillbereflectedinobjectiveindicators,andratiosbetweenmenandwomenareusefultocapturethem.Butsomeotherformsofdiscrimination,andthezerotolerancedescribedbytheHLP,arehardertocapture.

Perceptionsdatacanhelptomeasuretheevolutionofsocialnormsthatdriveunequaloutcomesforwomenthroughtheirlife-cycleindomainssuchaseducation,health,politicalrepresentationandlabourmarkets(Branisaetal.,2009andOECD,2010(Atlas of Gender and Development)citedinHarperetal.,2012).Forexample,studiesbyHussein(2010)intheUK,orYirgaetal.(2012)anddeCaoandLutz(2014)inEthiopia, trytounderstandthebeliefsbehindthepracticeofFGMratherthanitsprevalence.TheyfoundthatFGMcanoftenbeunquestionedbywomenthemselves,andperceivedasatraditionalrequirement,despitethepaininvolved,toavoidsocialisolation.Thistypeofinformationmaybehelpfultounderstandhow,beyondlegalframeworks,informalsocialnormsunderpinunequalordiscriminatoryoutcomesforwomen.

Indicatorstomeasuresocialnormsusingperceptionsdatahavestartedtoappear.TheOECD’sSIGI(SocialInstitutionsandGenderIndex)23launchedin2009isthefirstcross-countrymeasureofsocialnorms,lookingatthede jure(legal)aswellasthede facto(actual)situationofdiscriminatorysocialinstitutionsinfivecategories:familycode,civilliberties,physicalintegrity,sonpreference,andownershiprights.InSIGI,bothattitudinalandprevalencedataareessentialtocapturethede factosituationof

21 Seehttp://tracker.post2015.org/

22 Seehttp://genderstats.org/

23 http://genderindex.org/


http://tracker.post2015.org/

http://genderstats.org/

http://genderindex.org/

women.BasedonSIGI,Harperetal.(2014)proposepost-2015targetsmeasuredbychangesinsocialnormsassignalsforgrowingempowermentofwomenandgirls.TheWomen’sEmpowermentinAgricultureIndex(WEIA),24developedbytheInternationalFoodPolicyResearchInstitute (IPFRI),isanotherpioneeringgender-relatedindexthatincludesperceptionsdataatthecoreofthemeasurementofseveraloftheindicatorscomprisingthe‘Fivedomainsofempowerment(5DE)’index.25AsinSIGI,alargenumberofindicatorsthatcomprisetheindexarebasedonperceptionsdata(seeAlkireetal.,2013).

Indicatorsbasedonsocialperceptions(heldbybothmenandwomen)areatthebaseofvariousproposalstomeasureagendergoalpost-2015.Table1presentsselectedexamplesshowinghowbothtypesofindicatorscouldbeusedinvarioussub-categoriesofwomen’sempowerment.Forinstance,measuringatargetonsexualandreproductivehealthcanbedonethroughprevalenceratesofunmetneedforcontraception,butalsothroughresponsestoquestionsaboutperceptionstowardssexualattitudes(inparticularabouttherighttorefusesexwithapartner).TheseareasarenotnecessarilycoveredintheMDGframeworkorallfeaturedintheSDG.However,mostoftheseexampleshavebeenproposedformonitoringthepost-2015goals.Afewadditionalindicatorscurrentlyusedformeasurementwereaddedtothetableforindicativepurpose.Thereisnotnecessarilyaone-to-onecorrespondencebetweentheperceptionandtheobjectiveindicators,butthecontrastisusefultonotewiththedifferentinformationthatperceptionsorobjectiveindicatorsprovide.

b) Governance GovernancewasakeyarealeftoutoftheMDG.Incontrast,theimportanceofgoodgovernanceinitselfandtowardsthefulfilmentofgoalsforothersectorshasbeenrecognisedinproposalsforpost-2015targets.TheHLP’sreport(HLP,2013)hasoneofits13goalsas‘Ensuregoodgovernanceandeffectiveinstitutions’andproposesthefollowingfivetargets:

1) Providefreeanduniversallegalidentity,suchasbirthregistrations.

2) Ensurethatpeopleenjoyfreedomofspeech,association,peacefulprotestandaccesstoindependentmediaandinformation.

3) Increasepublicparticipationinpoliticalprocessesandcivicengagementatalllevels.

4) Guaranteethepublic’srighttoinformationandaccesstogovernmentdata.

5) Reducebriberyandcorruptionandensureofficialscanbeheldaccountable.

TheOpenWorkingGroup’slatestproposal(2014)hasanarrowerfocus,withgoal16includingsomeaspectssuchasbuilding‘effectiveaccountableandinclusiveinstitutionsatalllevels’.Althougheachofthesegoalsisaccompaniedbymorevarioustargets,thedetailofthespecificindicatorsandmeasuresthatwouldbeusedtotrackthesetargetshasnotbeenannounced.Measuringtheseissuesappropriatelywillbekeytounderstandingtheextenttowhichcountriesareprogressing(orregressing)relativetotheirtargets.

Todate,measuresofgovernanceoftenfocusontheformofgovernmentratherthanitsperformance,forexample,ontheexistenceofparticularlegislationorpolicyandtheratificationofconventions(Forestietal.,2014).Governanceisalsocloselyconnectedwithservicedelivery.Whilethequantityaspectofservicedeliveryiswellsuitedtoobjectivemeasures,issuestodowiththequalityandaccesstotheseservicesareoftenbettersuitedtosubjectivemeasures.Thepost-2015discussionhasfocusedonrefiningandexpandingthesemeasuresofservicedeliverybutstillalongobjectivelines.Anoptimumsituationwouldbetheuseofbothobjectiveandsubjectivemeasurestounderstandboththeformandfunctionofgovernance,andwhetheritisdeliveringwhatpeoplewant.Table2showssomeofthedifferentpossibleperceptionmeasuresthatcouldbeused,comparedtoobjectivemeasuresinsimilarareas.

Therearehowever,afewconsiderationswhenusingperceptionsdataonservicedelivery.Perceptionmeasuresaroundeducation,forexample,maybeunreliableifwearedealingwithfirst-generationlearners.Parentswhoarethemselvesnon-literateorwhohavenotbeentoschoolwillfinditdifficulttopassqualitativejudgementonthekindofschoolingbeingprovidedtotheirchildren.Themediacanplayanimportantroleinshapingperceptions.Ifmassmediaarerelativelyfreefromstatemanipulation,theycanactasanaccountabilitysystembothtoinformthepublicofimportantissuesandformembersofthepublictoraisetheseissuesthemselves.Wherethestatehascontrolofthemedia,theycanbeusedtomanipulatepeople’sperceptionsofthequalityofgovernance.Itshouldbenotedthatobjectivemeasuresareasmuchatriskofgovernmentmanipulation,forexamplebyadoptingcertain‘forms’thatleadto‘better’scoresonanindicator,butwithoutrealchangeintheunderlyingaspectsofgovernance(alsoknownasisomorphicmimicry)(Forestiatal.,2014).

Framingeffectsmaybeimportantinevaluatingperceptionsofservicedelivery.Thephrasingofsinglequestionscanhaveasignificantimpactonpeople’sresponses.ThisisdemonstratedinperceptionsdataaroundtheUK’sNationalHealthService(NHS),orinuseofwordssuchas‘assistancetothepoor’versus‘welfare’(Bartels(2003);asdiscussedinSection4bbelow).IntheUK,whenaskedin2011abouttheirlocalNHSprovider,

A

24 http://www.ifpri.org/publication/womens-empowerment-agriculture-index

25 Thefivedomainsare:production,resources,income,leadershipandtime.

14 ODI Report

http://www.ifpri.org/publication/womens-empowerment-agriculture-index


Table 1: Selected gender and social norms indicators

Target: ‘End all forms of discrimination against all women and girls everywhere’

Area Perception indicators Objective indicators/self-reported objective proxy

Sexual and reproductive integrity/autonomy Percentage of women and men who think a woman can refuse sex with her husband/partner under any circumstance.U

Percentage of married women with an unmet need*** for family planning.H

Freedom from violence Percentage of women and men who think that a husband/partner is never justified in beating his wife/partner.H,U

Lifetime prevalence of domestic violence.H,S,U

Percentage of women who have undergone female genital mutilation.S,U

Decision-making ability about land, assets and resources

Extent to which the individual feels he or she can make his or her own personal decisions regarding the aspects of household life* if he or she wanted to.W

Relative Autonomy Indicator.** W

Percentage of people who think important decisions in the household should be made by both men and women, by sex.U

Self-reported questions on decision-making and women’s participation in decision-making and control over earnings in households. For example: who decides how to spend money (the wife? the husband? both?)H Self-reported questions on decision-making and woman’s participation in decision making regarding economic activities. For example: If an individual participated in the activity,**** how much input did the individual have in making decisions?W

Proportion of adult population owning land, by sex.U

Proportion of population with access to institutional credit (other than microfinance), by sex.U

Participation in political and civic life Proportion of those of voting age who agree or strongly agree that, on the whole, men make better political leaders than women.H

Score based on legal restrictions or discriminatory practices affecting women’s access to public space.*****

Percentage of women in national parliament.U

Equal value to boys and girls Percentage of respondents who agree that ‘education is more relevant for boys’.H

Missing Women indicator (gender bias in mortality due to sex-selective abortions, female infanticide or insufficient care given to baby girls).H,S

Time Rank their level of satisfaction from 1 = not satisfied to 10 = very satisfied, with the time available for leisure activities such as visiting neighbours, watching TV, listening to the radio, seeing movies, or participating in sports.W

Female-to-male ratio of average time devoted to household activities/unpaid domestic work.H,U

The female-to-male ratio of total workload (both paid and unpaid work).H

Support services and justice for women and girls Proportion of the population who feel safe walking alone at night in the area where they live, by sex.U

Proportion of women over 15 years old subjected to physical or sexual violence in the past 12 months who reported it to the justice system.U

Proportion of law enforcement professionals who are women (including judges and the police).U

Sources: H - Harper et al (2014), W - WEIA, S - SIGI, U - UNWomen (2013)* Activity areas: (a) which inputs to buy, (b) which types of crops to grow for agricultural production, (c) when to take or who should take crops

to market, and (d) whether to engage in livestock-raising.** Formed of three questions: 1. My actions in [activity area] are partly because I will get in trouble with someone if I act differently; 2.

Regarding [activity area] I do what I do so others don’t think poorly of me; and 3. Regarding [activity area] I do what I do because I personally

think it is the right thing to do. *** Women who are fecund and sexually active but are not using any method of contraception, and report not wanting any more children. This

definition points to the gap between women’s reproductive intentions and their contraceptive behaviour and thus it incorporates a perceptions

component (‘not wanting more children’). We place this under the self-reported proxy category.**** About food crop farming, cash crop farming, livestock raising, and fish culture.***** For example, the restrictions on women’s choice of domicile, restricted ability to visit family and friends, requirements for husband’s

approval to apply for a passport or widespread threats of political violence.

77%ofrespondentsagreedthattheywerebeingprovidedwithagoodservice.WhenaskedaboutwhethertheNHSdeliversagoodservicenationally,only63%ofthesamerespondentsagreedthattheservicewasgood.Whenaskedaboutthegovernment’sNHSpolicies,approvallevelsdroppedto28%.Thistrendhasbeenmaintainedsincedatawerefirstcollectedin2003whenthefigureswere76%,48%and30%,respectively(IpsosMori,2012).

c) Violence and security TheMDGdidnotincludeanygoalsortargetsaroundviolenceandinsecurity,althoughmomentumisbuildingfortheirinclusioninthepost-2015agenda.Freedomfromfear,conflictandviolencewasoneofthefivetransformativeshiftstodrivetheuniversaldevelopmentagendaoutlinedbytheHLP(2013).Itisalsoabasichumanrightandacoreelementofwellbeing.TheOWG’sproposal(UN,2014a)includestargetsoneliminatingallformsofviolenceagainstgirlsandwomen,significantlyreducingallformsofviolenceandrelateddeathrates,andendingallformsofviolenceandtortureagainstchildren.

Clearly,someoftheseissuescanbemeasuredwithobjectiveindicators.Traditionalindicatorsincludehomicideandcrimeratesandclashesbetweenarmedgroups(legalorillegal).Thesearebroadlycomparableacrosscountriesandofferinsightsintothelevelofsecurityriskposedtothegeneralpopulation.However,people’sperceptionsofviolenceandinsecuritydonotalwayscorrelatewithobjectivemeasures.Stiglitzetal.(2009)reportthattherecanbesignificantdifferencesbetweenactualandperceivedratesofviolenceandinsecurity,withthemediapotentiallyplayingasignificantroleindistortingpeople’sviews.WhenanalysingthevictimisationratecomparedwiththefearofcrimeinOECDcountriesin2005,nocorrelationwasfoundbetweenthetwo.Similarly,GarrettandAhmed(2004)foundthatinthecityofDinajpurinBangladesh,

despitethehighcrimerate,only10%ofmenandwomenreportedthattheydidnotfeelsafe.

Insituationsinwhichsocialnormsdictatethatcertaintypesofviolenceareacceptable,peoplemayunderreportit.AstrongexampleofthisisaroundFGM.Incountrieswherethispracticeisacceptedandevendesired,thistypeofviolencewillbeunderreportedbygirls(Hussein,2010;Yirgaetal.,2012).Similarly,somepeoplemaybelievethatcertainlevelsofdomesticviolencearetobeexpectedandthereforedonotperceivethemasharmful.Thissuggeststhatweshouldnotrelysolelyonperceptionsdatabutshouldusetheminconjunctionwithobjectivedata.

Large-scaleobjectivemeasureswillgiveonlyasurface-levelunderstandingofthevolumeofviolenceatbest,whereassubjectivemeasurescanhelpelucidatetheunderlyingcausesofthisviolencebyaskingpeoplemorespecificquestionsaroundhowthisviolenceisexperienced.Subjectiveindicatorsaremorenuancedandofferadeeperunderstandingofhowviolenceandinsecurityareperceivedbypeopleandcanhelptounderstandunderlyingdynamicsinfragileandconflict-affectedsituations(Mallett,2012).IntheBangladeshcaseabove,theroleofmastaan(localstrongmen)whoofferpeople‘protection’,wasapossiblecauseforthemismatchbetweenobjectiveandsubjectivemeasures.Nevertheless,thiscameatthecostofthosemostmarginalised,ornotinthefavourofthesestrongmenfigures,pointingtosomeinequalitiesintheprovisionofsecurity.Significantmismatchbetweenperceptionsandobjectivemeasurescanhighlightsignificantproblemsinasociety,forexamplelackofaccesstojusticeortotheinstitutionsinchargeofprotectingciviliansanddeliveringsecurity.

Akeycomponentoftheconceptofsecurityishowsafepeopleperceivetheirenvironmenttobe,andhowtheirwellbeinginteractswiththisperception.Forexample,Møller(2005)findsthattheperceivedlikelihoodofvictimisationandconcernaboutpersonalsafetyhasastrongernegativeimpactonlifesatisfactionthanactuallevelsofvictimisation.

16 ODI Report

Table 2: Selected governance and effective institutions indicators

Target: ‘Ensure good governance and effective institutions’


Government effectiveness Satisfaction with education system Quality of public schools Increase in tax revenue as a proportion of GDP

Control of corruption How many government officials do you think are involved in corruption? Proportion of public who believe they can receive timely services without paying a bribe

Government efforts to tackle corruption Reduction in number of people who report paying a bribe

Voice and accountability Satisfaction with democracy Freedom of the press Open Budget Index score

Sources: Kaufmann et al. (2009) based on various sources; Foresti et al. (2014) based on various sources

Thereareavarietyofproposalsforsubjectiveindicatorsthatcouldbeintroducedinthepost-2015framework.Saferworld(2013),whichhasoneofthemostdetailedproposalsconcerningviolenceandsecurity,givesseveralexamplesofexistingsubjectiveindicatorsthatcouldbeincorporatedtotracktwoproposedtargets:ageneraltargetaroundviolenceandinsecurity(Table3),andspecificallygender-basedviolence(Table4).TheSecureLivelihoodsResearchConsortium(SLRC)26surveysinconflict-affectedcountriesalsopresentssomevaluableexamples.Again,thereisnotaone-to-onecorrespondencebetweentheobjectiveandperceptionsindicatorsinthetables,whichisindicativeofthedifferentaspectscapturedbyeachtypeofdata.

Insum,thereviewofthesethreeareasshowsthatperceptionsdatacouldserveasausefulcomplementto

objectiveindicatorsofprogress.Objectiveindicatorsareusefulbuttheyhavetheirlimits,andsometimesadditionalperceptionsindicatorsarerequired(Veenhoven,2004;GoughandMcGregor,2009).Forexample,inthecaseofgender,thereductioningapsbetweenmenandwomenmanifestsonlypartlyimprovementstogenderempowermentandequalopportunities.Datareflectingsocialbeliefsongenderrolesandacceptedbehaviourcanalsopointtoimportantadvancementsinachievinggenderequality.Inthissense,subjectiveperceptionsareveryusefultoassesspolicysuccess(Veenhoven,2004)butlesseffectiveforassessingwhyaspecificpolicyworkedandhowtogeneratesuchchanges(Hulmeetal.,2014).

26 http://www.securelivelihoods.org/


Table 3: Selected indicators to measure violence and insecurity

Target 1: ‘All social groups are free from violence and insecurity’


Equal justice Do the police treat people equally? (piloted by the Vera Institute of Justice)

Percentage of police, prosecutors and judges that are women

Safety Do you feel safe walking alone at night in the city or area where you live?How safe do you feel now moving to other places e.g. markets or town?

Political stability and absence of violence score

Trust in police How much do you trust the police?

Police capabilities How easy or difficult is it to get help from the police? Number of convictions over number of police

Sources: Saferworld (2013) based on various sources; SLRC surveys

Table 4: Selected indicators to measure progress on eliminating violence against women and girls

Target 2: ‘Violence against women and girls is eliminated’


Violence against women Percentage of population believing that a husband is justified in hitting or beating his wife/partnerPercentage of women vs. men who believe that the police would respond if they reported a crime (piloted by Vera Institute of Justice)

Number of recorded rapes per 100,000 women and girlsHomicides of females per 100,000 females

Source: Saferworld (2013) based on various sources

http://www.securelivelihoods.org/

Why not perceptions data? Common but differentiated challenges of perception surveys

Interestisincreasingfromgovernmentsinunderstanding,collectingandusingperceptionsdata.Awiderrangeofresearchhelpstounderstandthemainconcernsandissuestoconsiderhere.Thedataalsohavesomelimitations,whichwediscussinthissection.Thus,itmaybeessentialtocomplementperceptionsdatawithobjectivemeasuresoradministrativerecordsfromothersources.

Theissuesdiscussedinthissectionarenotspecifictothepost-2015context,butratherrelevantingeneraltothecollectionofperceptions(andothersurvey)data.Becauseperceptionsdataarecollectedthroughindividualorhouseholdsurveys,keyissuesinusingthistypeofdatarelatetosurveymethodsandtheirinherentlimitations.Errormayarisefromwell-knownflawsaffectingsurveys.Thisdoesnotimplymistakes,inthecolloquialsenseoftheword,butratherdeviationsfromwhatisdesiredinthesurveyprocess(Grovesetal.,2009).Inbroadterms,surveyerrorscanbedividedintotwogroups:errorsofobservationormeasurement;anderrorsofnon-observation,whichrelatetotheprocessofobtainingstatisticsfromasampleofthepopulation(Figure1).Wetouchonsomeoftheissuesofrepresentationfirst,thenfocusonaselectionofmeasurementerrorsourcesthatcouldbeimportantinthecontextofcollectingperceptionsdata,particularlybecauseofthenatureofaskingpeopletoself-reporttheiropinions.Thisreviewisnotintendedtobe

exhaustivebutshouldindicatethemainissuesofreliabilityofperceptionsdata.27

a) Representation: whose responses are represented?First,asinanysurvey,possibleconcernsinclude:therepresentativenessofthesample,samplenumbers,lowresponseratesandaccessibilityissueswheninfrastructureislimited.28Adoptingarigoroussamplingmechanismisanecessarystepforanysurvey,asisreportingonsamplingerrors,tohaveasenseofhowclosethesampleistothepopulationofreference.29Nevertheless,incomparisonwithsomelarge-scalemulti-topichouseholdsurveyscurrentlyusedinthemonitoringoftheMDG,existingperceptionsurveystendtohavemuchsmallersamplesizes.Forexample,thesamplesizeintheGallupWorldPollisaround1,000peopleandupto2,000inlargecountriessuchasChinaandRussia.30Thesurveysalsohavesamplesizesofabout1,000peopleineachcountry,althoughinsomeexceptionalcaseslargersampleshavebeendrawn.31Incontrast,astandardDHShasasampleofbetween5,000and30,000households.32MultipleIndicatorClusterSurveys(MICS),asusedbyUnicef,alsotendtohavelargesamples;themostrecentMICSsurveyforVietnamforexample,interviewedcloseto10,000householdsand10,000women.33Ifperceptionssurveysbecamepartofthe

27 Foradetailedreview,seeGrovesetal.(2009)orKrosnikandPresser(2009),amongothers.

28 See,forexample,Traugott(2003)forsomeoftheissuesconcerningelectoralpollsintheUS,andHerbert(2013)onperceptionsurveysinconflict-affectedcountries.

29 Seehttp://www.aapor.org/Best_Practices1.htm#.U9Z1_PldU9Yforbestpracticesinperceptionsurveys.

30 http://www.gallup.com/poll/105226/world-poll-methodology.aspx

31 http://www.hks.harvard.edu/fs/pnorris/Acrobat/Global%20Communications/Technical%20Appendix%20%20B%20List%20of%20Countries.pdf

32 http://dhsprogram.com/What-We-Do/Survey-Types/DHS.cfm

33 http://www.childinfo.org/files/Viet_Nam_2013-14_MICS_KFR.pdf

18 ODI Report

http://www.aapor.org/Best_Practices1.htm%22 \l %22.U9Z1_PldU9Y

http://www.gallup.com/poll/105226/world-poll-methodology.aspx

http://www.hks.harvard.edu/fs/pnorris/Acrobat/Global%20Communications/Technical%20Appendix%20%20B%20List%20of%20Countries.pdf

http://dhsprogram.com/What-We-Do/Survey-Types/DHS.cfm

http://www.childinfo.org/files/Viet_Nam_2013-14_MICS_KFR.pdf

SDGmonitoring,orperceptionsmoduleswereconsistentlyaddedtoexistinglargerhouseholdsurveys,perceptionsdatamightbecomemuchmorerepresentative.

Evenwiththebiggersamplesizesoftraditionalhouseholdsurveys,theremaybeissueswhentryingtogatherdataforspecificpopulationgroupslikelytobeunrepresented.Moreover,thereisabiggerpointrelatedtoakeyfocusoftheSDGinfinishingtheMDGtaskbyreachingthehard-to-reachand‘leavingnoonebehind’.Aresurveysreallygoingtobethebestwaytogathertheexperiencesofthesegroups?Thishasbeenhighlightedasproblematicforinstanceinthecaseofthoselivingininformalsettlementsofurbanareas,olderpeopleandpeoplewithdisabilities(Carr-Hill,2013;LucciandBhatkal,2014;SammanandRodriguezTakeuchi,2014).Moreover,thepossibilityofproducingrepresentativeresultsforpopulationsubgroupsbeyondgender,placeofresidence(rural/urban),andsometimesageandnationalsub-region,getssmallerwithlowersamplesizes.Surveysneedtobespecificwhenreportingresultsabouttherepresentativenessoftheirdata,sothatitisclearwhatlevelofdisaggregationispossible.Itmaybenecessarytoincreasesamplesizes

tobeabletoprovidesufficientinformationthatdoesnotmasktheheterogeneityofpointsofview.

Insomecases,actingonaggregatepreferencesmaynotbedesiredforequityreasons.Theanalysisofsubjectivedatamaycontributetopost-2015equityobjectivesbyhighlightinghowpopulationsubgroupsmaybenefitfromthedifferentgoalsandmaybeunevenlyaffectedbydifferentpolicies.However,thevariableeffectofdifferentprioritiesandtrade-offsondifferentsubgroupsofpopulation,accordingtoage,gender,socioeconomicposition,culture,andmanyothercharacteristics(Bellanietal.,2013),34mayalsoposeanimportantchallengeinusingthesedata.Inparticular,therecouldbesystematicbiasesunderlyingperceptionsdata.Forexample,thewell-known‘U’relationshipbetweenlifesatisfactionandage(BlanchflowerandOswald,2011),thegenderdifferencesinreportedlifeevaluations(e.g.Boarinietal.,2012),orthehigherdissatisfactionwithdemocraticinstitutionsreportedbyyoungpeople(Berghetal.,2014)maypointtoanimportantpolicyissuetobesolved.Further,unlessweknowmoreaboutthereasonsforthesystematicdifferences,thereistheriskofinformingpolicyonthebasisofincorrectinformation.

34 Inprinciple,‘theycanvarybetweenindividualsalmostcontinuously,thoughforpolicypurposestheidentificationofdiscretetargetgroupsmaybeeasiertointerpretandmoreusefulforthedevelopmentofpolicy’(Bellanietal.,2013).


Figure 1: Errors in the process of obtaining estimates from surveys

Measurement/ observation

Design

Measurement

Response

Adjustments to data

Representation/ sample selection

Target population

Sampling frame

Sample

Respondents

Coverage error

Sampling error

Non-response error

Validity

Measurement error

Processing error

Source: Adapted from Groves et al. (2009)

Aseparateissueconcernsgatheringperceptionsinareaswheredatacollectionisdifficultbecauseittouchesonsensitiveissues.Manyofthequestionsproposedtodatefortheareasofgendernorms,violenceandsecurityandgovernancemaybedifficulttogatherforthisreason.Responseratesappearhighforsomeperception-basedquestions.IntheGWPandtheWVSforexample,(objective)incomequestionshadbetween10and100timeshighernon-responseratesthandidsubjectivewellbeingquestions,dependingonthecountry(Smith,2013,citedinOECD,2013).Forothertypesofperceptions,itemnon-responsemaybeabiggerproblem,althoughtheevidenceislimited.Forexample,non-responsecouldbehigherforquestionsoritemsthataredifficulttocomprehend,offensivetoparticipants,orconcernsensitiveissues,andalsoinsurveysthatrequirelongerinterviewtimes.Non-responseinself-reportedquestionsaboutlivingconditionshasbeenanalysed(forexample,RiphahnandSerflinf(2002),forincome),anddirectsurveyquestionsonsensitiveissuesarelikelytoresultinlowparticipationandmisreportingofanswers,particularlyforthosewithsomethingconsidered‘embarrassing’toreport(TourangeauandYan,2007;Tourangeauetal.,2010).However,thatevidenceislimitedtoself-reportedproxiesofobjectiveoutcomes(forexampledruguse,voting,income),whichfallbeyondourdefinitionofperceptionsdata.Theextentofitemnon-responseforotherperceptionquestionsneedstobeexaminedfurther.

Guidanceandbestprocedurestominimisequestionandsurveynon-responsehavebeendevelopedforsectionsofhouseholdsurveystouchingonsensitiveissues,andcouldbeadaptedtosurveysthatareexclusivelydedicatedtoperceptions.Forinstance,since1998-99,DHSsetthetaskofdevelopingastandardisedapproachtomeasuredomesticviolence,accompaniedbyguidelinesontheethicalimplementationofthequestionnaire.35Someofthecriteriaare,forexample,toensureprivacyandanonymityoftheresponses,havinganinterviewerofthesamesexastherespondent,askingsensitivequestionstowardstheendofthesurveywhenrapporthasbeenbuiltup,ormakingavailableinformationonorganisationsthatprovideservicesorreferralstovictimsofdomesticviolence.Thesemeasuresarealsohelpfultosolvesomeofreliabilityissuesdiscussedbelow.

b) Reliability: how accurate are people’s responses?Thenextissueistoensurethattheresponsesproducereliableinformation,indispensablefortrackingthepost-2015targets.Pollsandpollstersoftenfacealackoftrustandareespeciallycriticisedwhenopinionmovesinthe‘wrong’direction(DionneandMann,2003),

whichcouldbecontrarytoeitherindividualbeliefsorthepredominantsociallyacceptedview.Questionsabouttheindependenceofthepollstersareoftenraised,butamorefundamentalquestionconcernsthereliabilityofself-reportedinformationandtheunderlyingquestionnaireusedtoobtainsuchinformation.Peoplemayunder-orover-statetheirresponsesorreportfalseinformation,eitherintentionallyorunintentionally,andsuchmisreportingmaybiassubjectiveindicators.Unlikewithobjectiveindicators,cross-validationisnotpossiblebysimpleobservationoflivingconditionsoradministrativerecords.Thewayinwhichquestionsareasked,andthequestionnaireitself,areperhapsthemostimportantconsiderationsinreducingmeasurementerror(KrosnikandPresser,2009).

Wehighlightfourkeyproblemsthatcanleadtoinaccurateinformationonperceptions:

• Recall bias:thisoccurswhenindividualsmakemistakesinrecallingpastevents.Forexample,whenaskedabouthowmanytimesduringthepastweek/monththeyexperiencedacertainfeeling(happiness,anxiety,etc.).Thismaylimitreliabilityofanswersrequiringparticipantstocomparecurrentandpastperceptions.

• Social desirability bias:thisoccurswhenparticipantsrespondwithanswersthoughttobemoresociallyacceptableordesirable,orwhentheyrestricttheiranswersforfearofrepercussions.Forexample,inconflictsituations,peoplemayfearreportingtheiractualperceptionsofthestate(Herbert,2013).Socialdesirabilitymayalsobepresentwhendiscussingillegaltopics(e.g.relatedtosexualbehaviourordruguse).Thistypeofbiasmaybeparticularlyrelevantinhighlypoliticisedsocietiesorwherefreedomofexpressionislimited.

• Framing effects:thisoccurswhenthewayinwhichquestionsareaskedaffectstheanswersgiven.Thistypeofbiascaninteractwithsocialdesirabilitybias.TakeforinstancetheUSGeneralSocialSurvey(1984,1985and1986):respondentswereaskedwhethertheythoughttheircountrywasspending‘toomuch,toolittle,orabouttherightamount’oneachofavarietyofgovernmentprogrammes.Whileonlyaquarteroftherespondentseachyearsaidthattoolittlewasbeingspentonwelfare,closeto65%saidthattoolittlewasbeingspentonassistance to the poor.Thesedifferencesareattributedtothedifferentconnotationsoftheterms‘welfare’and‘assistancetothepoor’intheUS(Bartels,2003).Questionsthat‘lead’theanswermaysufferfrombothframingeffectsandsocialdesirabilityissues.Forinstance,questionssuchas‘Doyouthinkthat...?’and‘Don’tyouthinkthat...?’pull‘yes’forananswer(Barkeretal.,2002).Ontheotherhand,insomecasesitmaybeusefultoincludesuchimplicitpremisesofacceptedbehaviourtoencouragetherespondent

35 Giventhattheyarepartofthesamesurvey,asisoftenthecasewithquestionsonexperienceofdomesticviolenceandperceptionsofdomesticviolence,thisguidanceisalsoappliedtotheperception-questionssectionofthesurvey.

20 ODI Report

tospeakhonestly.Forexample,studiesonsexualbehaviouroftenask‘Howoldwhereyouwhen…?’ratherthan‘Didyouever…?’(Barkeretal.,2002).Theorderofquestionsorevenwhethertherespondentisaloneorincompanymayleadtodifferentresults.Sensitivequestionstendtobeleftforlastwhentherehasbeentimetobuildrapportbetweenandtheintervieweefeelsmorepositiveaboutexpressinghertrueopinion.

• Proxy respondent:whenaquestionisaskednotdirectlytothepersoninvolved,buttoanominatedhouseholdmember,forexampletheheadofhousehold,themother(ofachild),oranyavailableadultinthehouseholdatthetimeofinterview.Bardasietal.(2011)showthatthiscangreatlyaffectestimatesderivedfromhouseholdsurveys.

Measurementerrormaybeminimisedthroughgoodevidence-basedquestionnairedesign.Forexample,thiscaninclude:usingrecordingsheets–formswhererespondentswritedowntheinformationoverthecourseofaperiod–toreducerecallbias;havinganinterviewerofthesamegendertobuildrapport;usinganonymousresponses,asforexampleinMyWorld,toencouragehonestanswersaboutsensitiveissues;introducingprobingquestionsandusingplainlanguagetoavoidconfusionandmisreporting.

Thereareparticularproblemswithself-reportedproxiesofobjectiveindicators,likelytoresultinunderestimatesofprevalenceofsensitiveissuessuchasFGM,domesticviolenceordruguse.Inamoredetailedreview,AlkireandSamman(2014)concludethatitisproblematictouseperceptualdataasproxyforobjectivedeprivations.

Perceptionquestions,asdefinedinthisreport,arenotconcernedwiththeindividualexperienceorconditionbutratherwiththeperception.Assuch,framingquestionsinsuchawaythattheydonotimplydirectdisclosureofasociallyundesirablebehaviourmayreducethereliabilitybias,althoughitmaynotcompletelyeliminateit.Forexample,inastudyonperceptionsofFGMinEthiopia,deCaoandLutz(2014)usealistexperiment36asanalternativefordirectquestioningtoobtainperceptionsaboutFGM.Thisindirectmethodresultedinhigherdisclosureofacceptanceofthepractice,comparedtodirectquestioning.Thismethodhasalsobeenusedtoaskaboutracialattitudes,voterbehaviour,thesocialacceptabilityofsexualattitudesandillegalmigration.

Thepotentialtoreducebiasbyusingindirectquestioningispromising,butadeeperissueisthatperceptionsfluctuateacrossgroupsofsocietyandovertime,andresponsesshowsystematicdifferences.Thesefluctuationsneedtobeseparatedfromreliabilityissues.37

Distinguishingbetweentruechangesinperceptions,variabilityacrosstypesofrespondentsorsometypeofbiasremainsacontestedissue,anditisasyetunclearwhatthegoalpostshouldbeforcorrectingthepotentialbiases.Forexample,inthestudyofFGMattitudesinEthiopiaforexample,theauthorsattempttofindbiasesbycomparingtheanswersfromdirectandindirectquestions;findingthatuneducatedwomenwhohadpreviouscontactwithalocalNGOweremorelikelyshowasocialdesirabilitymisreportingbias(deCaoandLutz,2014).Incomparison,IvanynaandShah(2009)comparecountryrankingsofstatecapacityusingcitizenorexpertperceptions.Theyadjustthecitizen-basedanswersforthepossibilitythatrespondentsareafraidtotellthetruthabouttheirgovernments(measuredbythefreedomofexpressionintheircountry),areindoctrinated(whenrespondentsthinkwelloftheirgovernmentsbecauseofexposuretofavourablemass-mediacoverage)orareexcessivelyoptimisticorcritical.Thesecorrectionsareveryinteresting,yetdepartfromthepremisethatsuchbiasesaredrivingthedifferencesbetweentheexpert-basedandcitizen-basedperceptionsofthestate.Inthiscase,theexistenceofreliabilitybiasmaybehardertoprovethantocorrect.

c) Adaptation: have poor people come to terms with living in poverty?Anotheraspectthataffectsthereliabilityofperceptiondataisadaptation,particularlyrelevantwhenconsideringthat,unliketheMDG,manyifnotalloftheSDGtargetsarelikelytobeappliedtoalltypesofcountriesandpeoples–poorandrich,andfromavarietyofbackgroundsandcultures.Unlikethereliabilityissues,theadaptationproblemdoesnotimplythatpeoplemisreportanswers.Infact,peoplereporttheiractualperception,buttheinterpretationofresultsbecomesdifficult.Whilethisdoesnotunderminethevalueofperceptionsdata,itaffectshowtheyshouldbeusedbypolicy-makers–forexample,tounderstandhowtoapproachpeoplewithideasandpotentialpoliciestomakeimprovementstotheirlives,ratherthantodecidewhetherornottomakesuchimprovements.

Therearetwotypesofadaptation.Thefirstresultsfromhabituationtoone’slivingconditionsandtheconsequentabandonmentofwants,aspirationsandexpectations(Neff,2012).Ifadaptationhastakenplace,theassessmentindividualsmakeoftheirwellbeing(orotheraspectsoftheirlives)maybelimitedbytheirlifecircumstances.Sen(1999)forexample,arguedthat:

36 Thisisanindirectwaytoaskaquestion.Themethodpresentsrespondentswithalistofstatementsandasksthemindicatethetotalnumberofstatementswithwhichtheyagree.Therespondentsarerandomlydividedintoacontrolandatreatmentgroup.Thecontrol-grouprespondentsreceivealistofnon-sensitiveitems.Thetreatment-grouprespondentsreceivethesamelistofnon-sensitiveitemsplusonesensitiveitem,forexampleaboutFGM(e.g.,‘Agirlshouldbecircumcised’).Thedifferenceinthetotalnumberofitemsbetweencontrolandtreatmentgroupidentifiestheproportionofpeopleinthepopulationthatagreeswiththesensitiveitem.

37 Forstandardsurveys,acommonwaytoassessreliabilityistoapplyquestionnairesvariousseveraltimestosimilarsubjectsandinsimilarconditionsandcomparetheresponses.Reliablequestionswouldresultinthesameanswersreportedeachtimethequestionnaireisapplied.


‘The deprived people tend to come in terms with their deprivation because of their sheer necessity of survival, and they may, as a result, lack the courage to desire any radical change, and may even adjust their desires and expectations to what they unambitiously see as feasible.’ (Sen, 1999: 63)

Adaptationmaychangetheplaceinthereferencescalewhereindividualsplacethemselveswhenansweringperceptionquestions.Forexample,inthequestionsaboutincomesufficiencypresentedinBox1,whatconstitutes‘sufficient’maybedeterminedbyone’sactuallivingconditions.Inthisscenario,peoplelivinginpovertymaysaythattheirincomeissufficient,whilesomeonewithhigherincomesmaysayitisnot.Inconsequence,ifsuchadaptationexists,itlimitsthecomparabilityoftheanswersforthesetwotypesofpersons.Thisisalsoproblematicintermsofcomparabilityacrossgroupsbecauseitmakesithardtodeterminewhetherthereasonsbehindthevariabilityofresponsesaretruedifferencesinperceptionsoraretheproductofadaptation.

Thesecondtypeofadaptationoccurswhenextraordinaryeventstemporarilychangepeople’sperceptionsandbiassubjectiveresponses,particularlyaroundsubjectivewellbeing.However,researchhasshownthatsuchbiasestendtobeonlyshort-livedandsubjectivewellbeingreturnstoastablelevelaftertheevents.Thisiscalledhedonicadaptation.Forexample,thechangeinreportedsatisfactionthathappensafterimportantlifeeventssuchasgettingmarried,havingchildrenorwinningthelotteryisoftentemporaryandtheoveralleffectonlifesatisfactionorlong-termhappinessmeasuresisminor(KahnemanandKrueger,2006;Gilbert,2004).However,theevidenceseemsmixedonthedegreeofhedonicadaptation.Somehavefoundthatpeopleadaptmoretopositivethantonegativeshocks(Burchardt,2005);othershavefoundthatforhealthshockssuchasdisabilityonset,fullhedonicadaptationdoesnottakeplace(FafchampsandKebede,2007;OswaldandPowdthavee,2008).Largeandrandomlychosensampleswouldhelptocontrolforunobservedeffectswhenthoseareunsystematicallypresent(ArnesenandTrommald,2005);usingasampledrawnfromagroupwithenoughvariation,onecanexpectthat,inaggregate,indicatorswouldbemorereliable(RodriguezTakeuchi,2014).

Itisimportanttohighlightthatempiricalevidencesuggeststhattheissueofadaptation,atleastinthecaseofincomepovertyandsubjectivewellbeing,islessstrongthaninitiallysupposed(Box2).Infact,recentresearchshowsthatdeprivationdoesnotneedtotranslateintoadaptation.Findingssuggestthatsubjectivewellbeingisdependentonthefulfilmentofbasicneeds(Veenhoven,1988),butthatincomeandbasicneedsfulfilmentarenot

thesoledeterminantsofhighersubjectivewellbeing.Forexample,Burchardt(2005)findsevidenceofadaptationamongthetopincomegrouponly,suggestingthatthoseonconstanthighincomeshaveadaptedtothatlevel.Similarly,thefamousEasterlinparadox(1974)alsoshowsthatwhilecountryincomehasapositivecorrelationwithsubjectivewellbeing,thereseemstobeathresholdabovewhichthereturnsfromincomearelower.ThiswascorroboratedbyCamfieldetal.(2009)atthemicrolevel.

Adaptationcanbeseeninmanyotherareas.Forexample,researchonwomen’sdemandsforgenderequalityshowthatingender-unequalcountries(measuredbyobjectiveindicators),thedemandforgenderequalityisstronger(Bhatkal,2014).Paradoxically,whilethesewomendemandequality,theyalsocontinuetoholdviewswhichreinforcegenderinequality.Forexample,withrespecttotherighttoeducationandworktheymaystillexpressagreementwiththestatementthatboyshavemorerighttoeducationthangirls.Thishintsatthepresenceofadaptationbecauseitsuggeststhatwomeninmoreunequalsocietiestendtoundervaluewomen’srights.Inthissense,peoples’framesofreferenceareaffectednotjustbytheirown(absolute)livingconditions,butalsobycomparisontothosearoundthem.Thissituationmaystillpointtothepresenceofverystrongnormsonacceptedsocialattitudes,andthuswouldremainagoodindicatoroftheevolution(orlackofit)ofinformalgendernormsinagivencountry,asdiscussedinSection3a.

However,iftheadaptationbiasissystematicallypresentforonesubgroup(i.e.women,thepoororthosemoreexposedtomassmedia,asinthecasepresentedbyIvanyaandShah(2009),thiscouldbecorrectedforusingeconometrictechniques,forexample(seeDecancqetal.,2009).Asdiscussed,themainissueisthatidentifyingthepresenceofabiasisnotstraightforward,asthereisnogoldstandardorcleargoalpostofcomparison.Moreover,discardingsystematicvariationsinresponsesacrossgroupsisnotdesirable.Rather,oneadvantageidentifiedforperceptionsdataistohighlightthosedifferences,aswellastheopinionsofgroupsthatotherwisecouldbemadeinvisiblebyusingaverages.Cross-validationofindicatorswithothermethodologiesisalwaysdesirable,andnecessarywhenusinganykindofindicatorsfornationalpolicydiagnosis.

d) Objective and subjective mismatchInlinewithStiglitzetal.(2009),wearguethatobjectiveandsubjectiveindicatorscouldbecomplementaryratherthanalternatives,astheymightbemeasuringdifferentaspectsofaphenomenon.Inthatcase,thereisnoreasontoexpectperfectcoincidencebetweenthetwotypesofdata.Ratherthanbeingaproblemwithperceptionsdata,amismatchbetweenthetwotypesshowsthattheyeachcontributedifferentinformationthatisusefulandnecessary.Ifthetwoweregivingthesameinformation,

22 ODI Report

therewouldnotbeaneedforusingperceptionsdata,butrathereffortswouldneedtobefocusedonperfectingtheexistingindicatorsandmethodsofdatacollection.

Stiglitzetal.(2009)pointoutthatareasonbehindthemismatchbetweensubjectivewellbeingandtraditionaleconomicindicatorsofcountryprogress(i.e.GDP)istheincompletenessofthelatterintermsofcapturingsomephenomenawhichhavealargeimpactonwellbeing.WhileGDPpercapitamaybeagoodindicatorofeconomicactivity,itisinsufficienttomeasureacountry’sprogressinwellbeing,aconceptthatencompassesmuchmorethannationaleconomicprogress.Moreover,therelationshipbetweenobjectiveandsubjectivewellbeingmaybenon-linear,thatis,itchangesdependingonthespecificlevelsoftheindicators.AsdiscussedinthecontextofadaptationandSWBmeasures(seeBox2),materiallivingconditions,includingincome,areimportantdeterminantsofsubjectivewellbeingatlowlevels.Butoncethemostbasicmaterialneedsarecovered,therelationshipmaybecomemoreindirect,andotheraspects–suchasmentalhealth,communityandfamilyrelationshipsorenvironmentalconditions–canplayagreaterroleindrivingwellbeing.

Themismatchmayalsoariseinothertypesofindicators.HeadeyandEcker(2013)findaverylowcorrelationbetweenself-reportedandothertypesofmeasuresoffoodsecurity.Thisispartlybecausethedifferentindicatorsaremeasuringdifferentaspectsoffoodsecurity.Forinstance,the2010FAOStateofFoodInsecurityreportuseddatamainlyonfoodavailabilityandaveragecaloricrequirements.Thisplacedalargeemphasisonfoodimporting,andunderestimatedtheroleofnationalfoodproductionanddistributionwithincountries.ThefoodinsecurityexperiencescalefortheVoicesoftheHungryproject38doesnotprovidespecificinformationonactualfoodavailability,butisusefultomeasurethepsychologicaldimensionoffoodsecurity(HeadeyandEcker,2013),focusingonthesocio-culturalaspectsoftheexperienceofhunger(Ballardetal.,2013).39Nevertheless,becauseoftheissuesaroundadaptationandcross-culturalunderstandingsofhungerandfoodadequacy,thesetypesofindicatorsmaypresentproblemsintheidentificationofneeds,forexampletotargetprogrammesoffoodrelieforfoodtransfer.

Anotherinterestingcaseisthemeasurementofgovernance.Forestietal.(2014)andBerghetal.(2014)

38 http://www.fao.org/fileadmin/templates/ess/voh/FIES_062014.pdf

39 Thereisalsoanissuearoundcapturinginequality.Whilehungerismoreanissueofaccessthanofavailability,foodavailabilityandcaloricconsumptioncanindicateonlytheupperboundofchronicnutritioninacountry(forexample,thesituationwhereallfoodavailableinthecountrywasequallydistributed,andallindividualshadsimilarcaloricrequirements).


Box 2: Adaptation, income poverty and subjective wellbeing

Someresearchonsubjectivewellbeinghasfoundapossibleimpactofadaptationintheinterpersonalcomparisonofsubjectivewellbeingresponses.Inparticular,theconcernrelatestopoorerpeoplereportinghighlevelsofsubjectivewellbeing,despitegrimlivingconditions.Contrarytopopularbelief,thereisnowidespreadevidenceofadaptationamongthepoorest(Clark,2012).Forexample:

• Neff(2012)showsthatintwovillagesinAndhraPradesh,themajorityofthepoorarelesssatisfiedwiththeirlivesonaveragecomparedtotheirnon-poorcounterparts,asarelowercastesandMuslimscomparedtohighercastes.Healsofindshighlysignificantstatisticaldifferencesregardingthemeansatisfactionlevelbetweenexpenditurequintilegroups.Thelowestexpenditurequintilegrouphasthelowestmeansatisfaction.

• InBangladesh,Camfieldetal.(2009)findthat,althoughthemajorityofpeoplearesatisfiedwiththeirlives,therearesignificantdifferencesbetweenricherandpoorergroups.Oftherich,31.5%ofpeoplereportbeing‘veryhappy’,comparedto4.7%ofthepoor.

• Mexico’sfirstnationalsubjectivewellbeingsurvey(BIARE),conductedin2013,showsthatindividualsfromthelowestincomedecilearelesssatisfiedwiththeirlivesthanindividualsfromthehighestincomequintiles(Chávez,forthcoming).

• InSouthAfrica,astudyonsocialperceptionsofmaterialnecessities(Wright,2008)foundthatpeoplelackingessentialitemsdefinedbythestudy(suchashavingabathorshowerinthehouse)areawareoftheirshortagesandlistedthemasessentialneeds.Thestudyalsofindsconsensusbetweendifferentgroupsaboutwhattheydefineasessentialneedsandwhatastandardoflivingcomprises.

ThesefindingsinIndia,Bangladesh,MexicoandSouthAfricaindicatethatpeopleinlowestincomegroupsreportlowerlevelsofsubjectivewellbeingandlifesatisfactionthanthoseinhigherincomegroups.Atthecountrylevel,thisalsoseemstohold.TheLegatumCommission(O’Donnelletal.,2014)analysedGWPdataandfindthatmoredevelopedNordiccountriesregularlyleadtheworldintermsoflifeevaluationorsatisfaction,whilelesseconomicallydevelopedcountriessuchasTogo,SierraLeone,andZimbabwe,areconsistentlyatthebottom.

http://www.fao.org/fileadmin/templates/ess/voh/FIES_062014.pdf

havepointedtothedifferencebetweenindicatorslookingatforms ofgovernanceversusindicatorslookingatways ofgovernance,adiscussionthathasbeenprominentinthepost-2015governancedebates.Forexample,indicatorsmeasuringregimetypeortheexistenceofcertainlegislationorinstitutions(e.g.ananticorruptioncommission)maybeproneto‘isomorphicmimicry’ortheadoptionofformstomeetthetargetswithoutrealchangeinperformance(Forestietal.,2014).Thus,thegapbetweenperceptionsandobjectiveindictorsofgovernancemaysuggestagapbetweenanidealformandthefunctioningofgovernanceinstitutionsandcansignalperceivedcitizendissatisfactionsandgrievances.Berghetal.(2014)argueforexamplethatcitizendissatisfactionwiththelackofperceivedresponsivenessandfunctioningofdemocraticgovernmentsmayhavebeenbehindmanyoftherecentpopularuprisingsaroundtheworld.

Intheareaofgovernance,populationsurveyshavetheadvantageofcapturingtheviewsofthosedirectlyinvolvedintheinstitutionsofthecountry,butcouldbelessaptforcross-countrycomparisonsthanothersourcesofdatasuchasexperts’perceptionsorobjectivedata(Hulmeetal.,2014).However,theseauthorsarguethatperceptionsdataarealsolesssuitablefordistinguishingbetweenthedifferentattributesofgovernanceandthushardertouseforidentifyingrelevantpolicyinterventions.Forexample,itwouldnotbeclearwhatisbehindachangeinreporteddissatisfactionwiththepolice:anincreaseincrime,oranincreaseinpoliceeffectivenessincapturingcriminalsandthusthevisibilityofcrime.ThissuggeststhatperceptionindicatorsofgovernancewouldnotmeetForestietal.’scriterionofshowingaclear‘theoryofchange’,thatis,a‘clearpathwayofchangeunderpinningtheproposed

indicator,specificallyconsideringhowinpracticeitcouldleadtobettermonitoringandmeasurement,andhencegreateractionandresourcemobilisationtoimprovegovernanceoutcomes’(Forestietal.,2014:18).

Tosummarise,inthissectionwehaveshowedthat,whilebothtypesofdataoftensufferfromsimilarmethodologicaldifficultiesrelatedtothedesignandexecutionofsurveys,perceptionsdatahaveadditionalproblemswhichstillrequiremoreresearch.Inparticular,intermsofensuringthereliabilityofthedataandtheindicatorsandensuringcross-countryandcross-groupcomparability,thereisnotyetaclearconsensusonhowtoidentifyandcontrolforpotentialbiases.

Finally,moreresearchisalsoneededtoclarifythesourceofdifferencesbetweenobjectiveandperceptionsindicatorsandwhatthegapsbetweenthetwocantellus.Theymaybothcapturedifferentaspectsofasinglephenomenon(Stiglitzetal.,2009)–forexample,foodsecurityorgovernance–andthusbotharestilluseful,dependingonthepurposesofmeasurement.Inparticular,perceptionindicatorsappearusefulinidentifyinggapsandcriticalpointsforintervention,forexamplebeforemalnutritionbecomesmanifestorwhencitizendissatisfactionsignalsafailureofdemocraticgovernance.However,thesegapshighlightthedifficultiesofelicitingtheappropriatetheoryofchangebehindtheindicators.Determininghowtoreacttotheinformationpresentedbyperceptionsdata,oridentifyingtheexactpolicyinterventionsbehindachangeinperceptions,isahardertask.Thischallengealsooccurswithothertypesofdataandmethods;mixed-methodsapproacheshaveemergedasanattempttocapturethecomplexityofhowpoliciesandprogrammesoperate.40

40 Seehttp://betterevaluation.org/resources/guides/intro_mixed-methods_impact-evaluationandhttp://betterevaluation.org/blog/mixed_methods_part1

24 ODI Report

http://betterevaluation.org/resources/guides/intro_mixed-methods_impact-evaluation

http://betterevaluation.org/blog/mixed_methods_part1

Perceptions data in monitoring the post-2015 agenda: implementation issues

Howcanperceptionsdatabeincorporatedinthemonitoringofkeyissuesinthepost-2015agenda?Thissectionoutlinessomepracticalissuestoconsiderforimplementation.

Thefirstissuetoconsideriswhether,inthecontextofthepost-2015targets,perceptionsdatawouldbemoreusefulfornationalorinternationalmonitoringandtrackingofthetargets.Whileperceptionsindicatorscouldbepotentiallygoodmarkersofprogressincertainareas,therearelimitstotheircomparability,especiallywhenconsideringresponsesgivenbypeoplelivingincountriesatverydifferentlevelsofdevelopment(e.g.becauseoftheadaptationproblem).Internationalrankingshaveemergedinnumerousareasandarebecominganincreasinglyacceptedpolicytoolintheinternationalarena,butnationalreactionstorankingsarenotyetwidelyunderstood(Cooley,forthcoming).

Rankingsinherentlycompareonecountrywithothers,andthuscanconferstatustocountriesrankedhighlyandstigmatisethoserankedlower(Cooley,forthcoming).WhiletheMDGandtheirSDGsuccessorsdonotestablishexplicitcountryrankings,theydoinvolvealargedegreeofcomparisonofprogressacrosscountries.Moreover,whilethemeasurementandcomparisonofcountriesmayhaveincentivisedcountriestryingtoreachtheMDGtargets(UNDP,2010),ithasbeenshownthatthismeasurementwasunfair,especiallyforthepoorestcountriesforwhichsomegoalswereunattainable(e.g.Easterly,2009).Thelessonhereisthattakingintoaccountcountries’heterogeneityandstartingpositionswasimportant.Rankingorcomparingcountriesbasedontheirperformanceonperceptionindicatorswouldlikelybeproblematicforthepost-2015targets.Moreover,asdiscussedinprevioussections,oneofthegreatadvantagesofperceptionsdataaretheirpotentialtomonitorchangingconditionsandtoactaspointersofchangesintrendsovertime.Becauseofthedifficultyofdiscerningwhatisbehindachangeinaperceptionindicator,triangulationwithothersourcesofdataisnecessaryto

determinetheappropriatepolicyresponse.Thus,wesuggestthatperceptionsdatawouldbemoreusefultomonitorchangingsituationsovertimewithincountries,ratherthantoestablishcomparisonsacrossthem.

Asecond,relatedissueistheproductionofperceptionsdata.Despitetheincreaseinavailabilityofperceptionsdata,fewNSOswereconvinceduntilrecentlyoftheirpotentialtoinformpolicyandthusfewofthemconsidereditworthinvestinginmeasuringsuchindicators.Thus,particularlyindevelopingcountries,surveysofpopulationattitudes,expectationsandsatisfactionarenotyetwellestablished(UNTaskTeam,2013).

Ontheotherhand,privatecompaniesusingperceptionsdataformarketresearchhavelongrealisedtheirpotential.Furthermore,privatepollingcompaniessuchasGallupandIpsosMoriarestillamongtheleadingcollectorsofsubjectivewellbeingdata.ResearchconsortiasuchastheEuropeanValueSurveysandtheWorldValuesSurveysarealsoleadingonthecollectionofperceptionsdata.Thefactthat,currently,perceptionsdataarestronglyassociatedwiththeseinstitutionsposesadditionalquestionsofdatagovernanceandaccountability.Theuseofprivatesourcesasofficialmeasuresfortarget-settingraisesquestionsofdataquality,ownershipandaccountability.

Dataqualitymaynotdependonwhethertheorganisationispublic,privateorresearch-based.Infact,whenprivatecompaniesarebeingcommissionedorworkinpartnershipwithpublicinstitutions,includinginternationalones,tocollectperceptionsdata,theboundariesofdataownershiparehardtodraw.Furthermore,inthecaseofresearchconsortia,thepresenceofacademicswithstrongresearchprofilesmayhelptoimprovethequalityofthedatacollected.However,itmaybemoredifficulttoassessthequalityofthedataiftheyarenotsubjecttoappropriateaccountability.

Forexample,GallupWorldPolldatahavebeenusedininternationalprojectssuchastheWorldBankGlobal


Findex,41UNDP’sHumanDevelopmentReportsince2010,42theOECD’sBetterLifeindex,43ILO’sSocialUnrestindex44andWorldofWorkReport,2013andtheLegatumProsperityIndex.45

Whilesomedataaremadepublicwhentheyarepartofsuchprojects,themicrodatabehindtheaggregatenumbersarenotreleased,andsometimesnoteventoapartnerinstitution.Conductingfurtheranalysis,verifyingthesamplingproceduresandrepresentativenessofthesurveys,checkingforomissions,recordingmistakesandotherdataqualityissuesisnotpossiblewithoutaccesstothemicrodata.Itisalsonotpossibletodisaggregatethedatabeyondwhatthepartnerinstitutionpresents,forexample,iftheyreleaseresultsatonlythenationallevel.Whilethepartnerinstitutionscouldensuredataqualityandaccountability,thismaynotalwaysbethecase.Beyondthat,thisalsoimpliesachangeinthewaydataaretraditionallyreportedtotrackprogressintheMDG.ItwillnotbenationalgovernmentsreportingtotheUN-systeminstitutions,butsuchorganisationscollectingdatafromprivateorganisationsandthenreportingbacktocountriesontheirprogress.

ExperiencessuchasthatofthePhilippinesSWSdemonstratethatthereisvalueinprivateinstitutionsgeneratingsocialstatisticsforpublicuse(GuerreroandMangahas,461989,citedinGuerrero,2004),especiallyinacontextofarestricteddemocracywheregovernmentsmaytrytoconcealinformation.TheSWSwasbannedin1982bytheauthoritarianrulingregimeforbeing‘toorisky’topublishonlywhentheSWSbecameindependent;theresultswerewidelydisseminatedacrossthegeneralpopulation(Guerrero,2004).Counterexperiencescanalsobefound,especiallyinthecaseofpoliticalormarketingpollscommissionedbyinterestgroupstochangepublicopinion,camouflagecontroversialpolicies,orgather‘proof’tosustaincertaindemandsorconcessionsfromthegovernment(DionneandMann,2003).

AnincreasingnumberofNSOsarestartingtocollectandreleaseperception-baseddata,particularlyonsubjectivewellbeing.Totheextentthatthesedatabecomesuseful,thereiscertainlyconsensusthatSWBmeasuresshouldbeviewedasonesetinthemuchbroaderarrayofindicatorsthroughwhichpopulationsaremonitoredandpoliciesinformed(seeOECD,2013).Thekeyissueseemsto

bewhetherthedataaremadeavailableandcommunicatedwidely,ratherthanwhethertheyarepublicorprivate.Availabilityalsoenablestheimprovementofdataquality,ascheckscanbeconductedbydatausers,andthestrengthsandlimitsofparticularsurveyscanbebetterunderstood.

Further,astheOECDhasdoneforindicatorsofSWB,acommonsetofstandardswillneedtobederivedforotherperception-basedindicators.Whileawiderrangeofperceptionsdataiscollectedongendernorms,violenceandsecurity,andgovernance,differentsurveysoftenaskdifferentquestionsandacompilationandreviewofbestpracticesisnecessary.Thereviewpresentedhere,onindicatorsofgendersocialnorms,violenceandgovernmentaccountabilityshouldserveasastartingpoint.Projectsfocusedonperceptionsdataarealsogrowingandincreasinglylookingtostandardisetheirprocedures.TheWVS,anditsprecursor,theEuropeanValueSurveys,aswellastheregionalbarometernetworkareexampleswherejointeffortshaveincreasinglyledtoacommonsetofprocedures.However,questionsremainaboutthevalueaddedofaglobalperceptionsurveythatisfullycomparableacrosscountries,especiallygiventhat,fortheSDG,perceptionsdatawouldbemostusefulatthenationallevel.

Whilesomeconsensusisstartingtoemergeoncommonstandardsfordataqualityandforcollectingandanalysingperceptionsdata,somedegreeofflexibilityinhowquestionsareaskedisneededtobeabletoaccountforthecountry-specificityofperceptions.Forexample,toensurethattheframingofquestiondoesnotresultinbiasedestimatesforsensitiveissues,thespecifictechniquesandtermsusedforthequestionsmayneedtobeadapteddependingonthecountry,asasimpletranslatingofthequestionnairesmaynotbesufficient.

Finally,itisworthnotingthattheapplicationofstandardsofgood-practiceprocedures,aswellasthedesiredincreaseinthesamplesizetotrytoaddressrepresentativenessandinclusionofmarginalisedgroups,mayalsoincreasethecostofperceptionsurveys.Thismayriskoneoftheirkeysellingpoints–frequencyandcountrycoverage.Althoughadetailedreviewisbeyondthescopeofthispaper,exploringthemixofnewmethodsandnewtechnologiesinthecollectionofperceptionsdatacouldbeawaytobringthosecostsdown.Moreexplorationofthisaspectisrequired.

41 WorldBankarticle:http://bit.ly/OGopsB

42 AlthoughperceptionsdataarenotincludedintheHDI

43 http://www.oecdbetterlifeindex.org/

44 http://www.ilo.org/newyork/voices-at-work/WCMS_217280/lang--en/index.htm

45 http://www.prosperity.com/

46 NotethatMangahasisthefounderoftheSWS.

26 ODI Report

http://bit.ly/OGopsB

http://www.ilo.org/newyork/voices-at-work/WCMS_217280/lang--en/index.htm

http://www.prosperity.com/

ConclusionsThepost-2015negotiationsandinparticularthecallfora‘datarevolution’havespurreddebatearoundtheinclusionofnewindicatorsofprogress.Inthiscontext,indicatorsbasedonperceptionsdatacouldshiftthemeasurementdebateandhelptofocusonwhatisimportanttoimprovewellbeingofcitizensaroundtheworld,inbothobjectiveandsubjectivedimensions.

Themainadvantageofperceptionsdataisthattheycanpresenttherespondents’ownviewsdirectly.Thisisimportanttostimulatepublicdebatearoundthetrade-offsofcompetingpoliciesandtoholdpolicy-makerstoaccountforwhatmattersmosttopeople.Toselecttargetsandindicatorstomeasurethepost-2015goals,thisprocesscanhelptovoicetocitizen’sperspectivesonwhattheyperceiveastheconstituentelementsofthegoals.Moreover,someperceptionssurveyssuchasMyWorldcanbeusedtoprioritiseimplementationamongtheplethoraofgoalsandtargetsandtocomparethepreferencesofpeopleindifferentcountriesorbelongingtospecificsocialgroups.

Theincreaseintheavailabilityofperceptionsdataandtheresearcharoundsubjectivedimensionsofwellbeingpresentagreatopportunity.Inparticular,comparedtohouseholdsurveysandothersourcesofinformationsuchascensuses,perceptionsurveyshavegreatpotentialtoproducetimelyandfrequentdataformonitoringsocial

conditions,andtospotchangesintrendsthatcouldendangertheachievementofthepost-2015targets.

WehaveidentifiedthreeareaswhereperceptionsdatahavefeaturedstronglyinproposalsformonitoringtheSDG:genderandsocialnorms,violenceandsecurity,andgovernance.Ineachcase,perceptionsdatacouldprovidecomplementaryinformationinconjunctionwithotherdata.Whileperceptionsurveyscouldbedifficulttouseontheirowntomeasuretheachievementofatargetinanyofthesethreeareas,differentconsiderationsneedtobeincorporatedinthedecision-makingprocesswithperceptionsinformationasaconsistentinput.Moreover,adisaggregatedanalysisofperceptionsmayhelptomakeexplicitthevarietyofpreferencesinagivensociety,whichneedstobeunderstoodinthecontextofthepost-2015targetstoaddressinequalitiesinorderto‘leavenoonebehind’.

However,perceptionsdataarelessusefulindeterminingtheappropriatepolicychangerequiredwhensuchanalarmisraised.Themainlimitationsare:1)representativenessatthesub-nationallevel,particularlygivenlowsamplesizes;2)reliabilityoftheanswers;and3)theissuesaroundadaptationandhowtoseparatethisfromheterogeneityinpointsofview.Thesechallengesarestillunresolved,andmaylimitthecomparabilityofanswers,especiallyofthoselivinginextremedeprivationorwithrestrictedfreedomsandrights.


References Alkire,S.andSamman,E.(2014)Mobilizing the household data required to progress toward the SDGs.Sustainable

DevelopmentSolutionsNetwork.(http://bit.ly/1DpxLQq)Alkire,S.,Malapit,H.,Meinzen-Dick,R.,Peterman,A.,Quisumbing,A.,Seymour,G.andA.Vaz.(2013)‘Instructional

guideontheWomen’sEmpowermentinAgricultureIndex’.Arnesen,T.andTrommald,M.(2005)‘AreQALYsbasedontimetrade-offcomparable?AsystematicreviewofTTO

methodologies’,Health Economics14(1):39-53.Ballard,T.,Kepple,A.andCafiero,C.(2013)TheFoodInsecurityExperienceScale.Developmentofaglobalstandardfor

monitoringhungerworldwide.Rome:FAO.Bardasi,E.,Beegle,K.,Dillon,A.andSerneels,P.(2011)‘Dolaborstatisticsdependonhowandtowhomthequestions

areasked?ResultsfromasurveyexperimentinTanzania’,World Bank Economic Review25(3):418-447.Barker,C.,Pistrang,N.andElliott,R.(2002)‘Self-reportedmethods’,inResearch methods in clinical psychology. An

introduction for students and practitioners(2ndedn).Wiley.Bartels,L.M.(2003)‘Is“popularrule”possible?Polls,politicalpsychology,anddemocracy’,The Brookings Review,Summer.Bellani,L.,Hunter,G.andAnand,P.(2013)Multidimensionalwelfare:dogroupsvaryintheirprioritiesandbehaviours?

Fiscal Studies34(3):333-354.Bergh,G.,RochaMenocal,A.andRodriguezTakeuchi,L.(2014)What’s behind the demand for governance? An

assessment of people’s views.London:ODI.Bhatkal,T.(2014)Whatdowomenwant?Gender,perceptionsdataanddevelopmentpriorities.London:ODI.Blanchflower,D.andOswald,A.(2011),‘Internationalhappiness’.NBERWorkingPaper16668.NationalBureauof

EconomicResearch.Boarini,R.,Comola,M.,Smith,C.,Manchin,R.andDeKeulenaer,F.(2012)Whatmakesforabetterlife?The

determinantsofsubjectivewell-beinginOECDcountries:evidencefromtheGallupWorldPoll.Paris:OECD.Burchardt,T.(2005)‘Areoneman’sragsanotherman’sriches?Identifyingadaptiveexpectationsusingpaneldata’,Social

Indicators Research74:57-102.Camfield,L.,Choudhury,K.andDevine,J.(2009)‘Well-being,happinessandwhyrelationshipsmatter’,Journal of

Happiness Studies10:71-91.deCao,E.andLutz,C.(2014)Sensitive survey questions: Measuring attitudes regarding female circumcision through a

list experiment (http://bit.ly/145E2S5).Carr-Hill,R.(2013)‘Missingmillionsandmeasuringdevelopmentprogress’, World Development46:30-44.Chandy,L.(2013)‘Countingthepoor.Methods,problemsandsolutionsbehindthe$1.25adayglobalpoverty

estimates’. InvestmentstoEndPovertyWorkingPaper. DevelopmentInitiatives.Chávez,C.(forthcoming)‘YouthDelinquencyinMexico:Anexplorationofcontributingfactors’.PhDdiss.,UniversityofCambridge.Clark,D.(2012)‘Adaptationanddevelopment–issues,evidenceandpolicyrelevance’,inD.Clark(ed.),Adaptation,

poverty and development: the dynamics of subjective well-being.London:PalgraveMacmillan.Clemens,M.,Kenny,C.andMoss,T.(2007)‘ThetroublewiththeMDG:confrontingexpectationsofaidand

developmentsuccess’,World Development35(5):735-751.Cooley,A.(forthcoming)‘Theemergingpoliticsofinternationalrankingsandratings:aframeworkforanalysis’,inA.Cooley

andJ.Syder(eds),Ranking the world: the politics of international rankings and ratings.CambridgeUniversityPress.Coulthard,S.Camfield,L.andMcGregor,A.(2014)‘Valuingdevelopmentandsocialprogress:practicalapproachesto

measurewhatmattersindevelopment’.London:ODI(mimeo).Cummings,R.(2009)‘Measuringpopulationhappinesstoinformpublicpolicy’.The3rdOECDWorldForumon‘Statistics,

KnowledgeandPolicy’ChartingProgress,BuildingVisions,ImprovingLife,Busan,Korea,27–30October2009.Decancq,K.,Fleurbaey,M.andSchokkaert,E.(2009)‘Whatgoodishappiness?’Ecorediscussionpaper(http://www.

ecore.be/DPs/dp_1238577455.pdf).Dionne,E.J.andMann,T.E.(2003)‘Polling&publicopinion:thegood,thebad,andtheugly‘,The Brookings Review.O’Donnell,G.,Deaton,A.,Durand,M.,Halpern,D.andLayard,R.(2014)Report of the Commission on Wellbeing and

Policy.London:LegatumInstitute.Easterlin,R.(1974)‘Doeseconomicgrowthimprovethehumanlot?Someempiricalevidence’,inPaulA.DavidandMelvinW.

Reder(eds),Nations and households in economic growth: essays in honor of Moses Abramovitz.NewYork:AcademicPress.Easterly,W.(2009)‘HowtheMillenniumDevelopmentGoalsareunfairtoAfrica’,World Development37(1):26-35.Fafchamps,M.andKebede,B.(2008)‘Subjective wellbeing, disability and adaptation: A case study from rural Ethiopia’.

WorkingPaper283.Oxford:TheCentrefortheStudyofAfricanEconomies(http://www.bepress.com/csae/paper283).

28 ODI Report

http://www.ecore.be/DPs/dp_1238577455.pdf

http://www.ecore.be/DPs/dp_1238577455.pdf

Foresti,M.Wild,L.withNorton,A.andRodriguezTakeuchi,L.(2014)Governance targets and indicators for post 2015. An initial assessment.London:ODI.

Gallup(2012)Globalstatesofmind.Newmetricsforworldleaders.WashingtonD.C.:Gallup.Garrett,J.andAhmed,A.(2004)‘Incorporatingcrimeinhouseholdsurveys:aresearchnote’, Environment and

Urbanization16:139-151.Gilbert,D.(2004)‘Thesurprisingscienceofhappiness’. TEDTalks(http://www.ted.com/talks/dan_gilbert_asks_why_

are_we_happy.html).Gough,I.andMcGregor,J.A.(2009)‘After2015:“3Dhumanwellbeing”.’IDSinFocus,PolicyBriefIssue09(http://

www.ids.ac.uk/files/dmfile/IF9.2.pdf).Groves,R.M.etal.(2009)Survey methodology (2ndedn).NewJersey:Wiley.Guerrero,L.L.B.(2004)Social weather stations: Asia’s oldest barometer.WSWOccasionalPaper(June).Harper,C.,Nowacka,K.,Alder,H.andFerrant,G.(2014)Measuringwomen’sempowermentandsocialtransformation

inthepost-2015agenda.London:ODI.Harper,C.,Jones,N.andWatson,C.(2012)Genderjusticeforadolescentgirls:tacklingdiscriminatorysocialnorms.

Towardsaconceptualframework.London:ODI.Headey,D.andEcker,O.(2013)‘Rethinkingthemeasurementoffoodsecurity:fromfirstprinciplestobestpractice’,

FoodSecurity5(3):327-343.Herbert,S.(2013)Perceptionsurveysinfragileandconflict-affectedstates(GSDRCHelpdeskResearchReport910).

Birmingham,UK:GSDRC,UniversityofBirmingham(http://www.gsdrc.org/docs/open/HDQ910.pdf).HLP(HighLevelPanelofEminentPersonsonthePost-2015DevelopmentAgenda)(2013)Anewglobalpartnership:

eradicatepovertyandtransformeconomiesthroughsustainabledevelopment.NewYork:UN.Hulme,D.,Savoia,A.andSen,K.(2014)Governanceasaglobaldevelopmentgoal?Setting,measuringandmonitoringthe

Post-2015DevelopmentAgenda.ESIDWorkingPaper32.Manchester:EffectiveStatesandInclusiveDevelopment(ESID).Hussein,E.(2010)Women’sexperiences,perceptionsandattitudesoffemalegenitalmutilation.TheBristolPEERStudy.

FoundationforWomen’sHealthResearchandDevelopment(FORWARD).IEAG(IndependentExpertAdvisoryGrouponaDataRevolutionforSustainableDevelopment)(2014)Aworldthat

counts(http://bit.ly/1wApBjP).IpsosMori(2012)PublicperceptionsoftheNHSandsocialcare:anongoingtrackingstudyconductedforthe

DepartmentofHealth,December2011Wave.London:IpsosMori.Ivanyna,M.andShah,A.(2009)Citizen-centricgovernanceindicators:measuringandmonitoringgovernanceby

listeningtothepeopleandnottheinterestgroups.EconomicsDiscussionPaper2009-27,KielInstitutefortheWorldEconomy(http://bit.ly/1wwvGLh).

Jerven,M.(2103)Poornumbers:howwearemisledbyAfricandevelopmentstatisticsandwhattodoaboutit.IthacaandLondon:CornellUniversityPress.

Kahneman,D.andKrueger,A.B.(2006)‘Developmentsinthemeasurementofsubjectivewell-being‘,JournalofEconomicPerspectives20(1):3-24.

Kaufmann,D.,Kraay,A.andMastruzzi,M.(2009)GovernancemattersVIIIaggregateandindividualgovernanceindicators1996-2008.PolicyResearchWorkingPaper4978.WashingtonDC:TheWorldBank.

Kenny,C.andDykstra,S.(2013)TheGlobalPartnershipforDevelopment:areviewofMDG8andproposalsforthepost-2015developmentagenda.WashingtonDC:CDG.

Kroll,C.(2013)Globaldevelopmentandhappiness:howcandataonsubjectivewellbeinginformdevelopmenttheoryandpractice?IDSworkingpaper432.Brighton:InstituteofDevelopmentStudies.

Krosnik,J.A.andPresser,S.(2009)‘Questionandquestionnairedesign’,inJ.D.WrightandP.V.Marsden(eds),Handbookofsurveyresearch(2ndedn).SanDiego,CA:Elsevier.

Leo,B.(2013)Isanyonelistening?DoesUSforeignassistancetargetpeople’stoppriorities?CDGWorkingPaper348(http://bit.ly/19KhMtx).

Lucci,P.andBhatkal,T.(2014)Monitoringprogressonurbanpoverty:arecurrentdataandindicatorsfitforpurpose?London:ODI.Mallett,R.(2012)Howfardoperceptionsurveystakeusinfragileandconflict-affectedsituations?TheAsiaFoundation

(http://bit.ly/1tQGAA0),accessed28November2014.Mangahas,M.(2013)Thevalueofstatisticalcompetition.12thNationalConventiononStatistics,MandaluyongCity,

October1-2,2013.McGee,R.andGaventa,J.(2011)Synthesisreport:reviewofimpactandeffectivenessoftransparencyandaccountability

initiatives.London:TransparencyandAccountabilityInitiative.Melamed,C.(2012)Doesdevelopmentgivepoorpeoplewhattheywant?London:ODI.Møller,V.(2005)‘Resilientorresigned?CriminalvictimisationandqualityoflifeinSouthAfrica’,SocialIndicators

Research72(3):263-317.KluwerAcademicPublishers.


http://www.ted.com/talks/dan_gilbert_asks_why_are_we_happy.html

http://www.ted.com/talks/dan_gilbert_asks_why_are_we_happy.html

http://www.ids.ac.uk/files/dmfile/IF9.2.pdf

http://www.ids.ac.uk/files/dmfile/IF9.2.pdf

http://www.gsdrc.org/docs/open/HDQ910.pdf

Neff,D.(2012)‘Adaptation,povertyandsubjectivewell-being:evidencefromSouthIndia’,inD.Clark(ed.),Adaptation,povertyanddevelopment:thedynamicsofsubjectivewell-being.London:PalgraveMacmillan.

OECD(2013)OECDguidelinesonmeasuringsubjectivewell-being.Paris:OECD.Oswald,A.J.andPowdthavee,N.(2008)‘Doeshappinessadapt?Alongitudinalstudyofdisabilitywithimplicationsfor

economistsandjudges’,JournalofPublicEconomics92(5-6):1061-1077.Prizzon,A.(2014)Whatdocountrieswantfromaid?Somereflectionsfromthefield.ValuingProgressblogseries(http://

bit.ly/1HUa1nj).Riphahn,R.T.andSerfling,O.(2002)Itemnon-responseonincomeandwealthquestions.IZAdiscussionpaperN576.Bonn:IZA.RodriguezTakeuchi,L.(2014)Incorporatingpeople’svaluesindevelopment:weightingalternatives.London:ODI.Saferworld(2013)Addressingconflictandviolencefrom2015:avisionofgoals,targetsandindicators.February2013

Briefing.London:Saferworld.Samman,E.andRodriguezTakeuchi,L.(2014)Oldage,disabilityandmentalhealth:dataissuesforapost-2015

framework.London:ODI.SciDev(2014)‘Frustration’overoldormissingdatainMDGreport,8July2014(http://bit.ly/1xISWtj).Seaford,C.(2013)‘Themultipleusesofsubjectivewellbeingindicators’,SocialIndicatorsResearch114:29-43.Sen,A.(1999)Developmentasfreedom.Oxford:OxfordUniversityPress.Stiglitz,J.E.,Sen,A.andFitoussi,J.-P.(2009)ReportbytheCommissionontheMeasurementofEconomicPerformance

andSocialProgress(http://www.stiglitz-sen-fitoussi.fr/documents/rapport_anglais.pdf).Thomas,M.A.(2009)‘WhatdotheWorldwideGovernanceIndicatorsmeasure?’, EuropeanJournalofDevelopment

Research,16July.Tortora,R.D.(2009)Sub-SaharanAfricansranktheMillenniumDevelopmentGoals(MDG).WashingtonDC:Gallup.Tourangeau,R.andYan,T.(2007)‘Sensitivequestionsinsurveyss’,PsychologicalBulletin133(5):859-883.Tourangeau,R.,Groves,R.andRedline,C.(2010)‘Sensitivetopicsandreluctantrespondents.Demonstratingalink

betweennonresponsebiasandmeasurementerror’,PublicOpinionQuarterly74(3):413-432.Traugott,M.W.(2003)‘Canwetrustthepolls?Italldepends’,TheBrookingsReview,Summer.UN(2014a)IntroductiontotheProposaloftheOpenWorkingGroupforSustainableDevelopmentGoals.NewYork:UN.UN(2014b)Theroadtodignityby2030:endingpoverty,transformingalllivesandprotectingtheplanet.Synthesis

ReportoftheSecretary-GeneralonthePost-2015Agenda.NewYork:UN.UNDP(2014)Buildingtransparency,accountabilityandanti-corruptionintothepost-2015developmentframework.

UNDPDiscussionPaper,September2014.NewYork:UN.UNDP(2010)Beyondthemidpoint:achievingtheMillenniumDevelopmentGoals.NewYork:UNDP.UNTaskTeam(onthePost-2015DevelopmentAgenda)(2013)Statisticsandindicatorsforthepost-2015development

agenda.NewYork:UN(http://bit.ly/12xWbHF).UNWomen(2013)Atransformativestand-alonegoalonachievinggenderequality,women’srightsandwomen’sempowerment:imperativesandkeycomponents.NewYork:UNWomen.

Veenhoven,R.(1988)‘Theutilityofhappiness?’,SocialIndicatorsResearch20:333-354.Veenhoven,R.(2004)Subjectivemeasuresofwell-being.UNU-WIDERDiscussionPaper2004/07(http://bit.ly/1KoTkmh).WorldBankandSecondMuse(2014)Bigdatainactionfordevelopment(http://bit.ly/1DEafwG).Wright,E.(2008)FindingsfromtheIndicatorsofPovertyandSocialExclusionproject:aprofileofpovertyusingthe

sociallyperceivednecessitiesapproach(http://bit.ly/1AKnmMY).Yirga,W.etal.(2012)‘Femalegenitalmutilation:prevalence,perceptionsandeffectonwomen’shealthinKersadistrict

ofEthiopia’,InternationalJournalofWomen’sHealth4:45-54.

This material has been funded by UK aid from the UK Government, however the views expressed do not necessarily re�ect the UK Government’s of�cial policies.

ODI is the UK’s leading independent think tank on international development and humanitarian issues. Readers are encouraged to reproduce material for their own publications, as long as they are not being sold commercially. As copyright holder, ODI requests due acknowledgement and a copy of the publication. For online use, we ask readers to link to the original resource on the ODI website. The views presented in this paper are those of the author(s) and do not necessarily represent the views of ODI or our partners.© Overseas Development Institute 2015. This work is licensed under a Creative Commons Attribution-NonCommercial Licence (CC BY-NC 3.0). ISSN: 2052-7209

Overseas Development Institute 203 Blackfriars Road London SE1 8NJTel +44 (0)20 7922 0300 Fax +44 (0)20 7922 [email protected]

http://www.stiglitz-sen-fitoussi.fr/documents/rapport_anglais.pdf

www.odi.org

mailto:[email protected]

using perceptions data to monitor the post-2015 … · using perceptions data to monitor the ......

Documents