3/1/18

1

Photo Credit Goes Here

IAPRI-MSU Technical Training

Difference-in-differences

FacilitatedbyNicoleM.MasonMichiganStateUniversity

DepartmentofAgricultural,Food,&ResourceEconomics

1March2018IndabaAgriculturalPolicyResearchInstitute

Lusaka,Zambia

Recall from July/September trainings on introduction to impact evaluation (1)

“Animpactevaluationassesseschangesinthewell-beingofindividuals,households,communitiesorfirmsthatcanbeattributedtoaparticularproject,programorpolicy”Source:WorldBank

3/1/18

2

Recall from July/September trainings on introduction to impact evaluation (2)

•  Wedidabriefoverviewofcommonmethodsofimpactevaluation(IE)•  Randomizedevaluation•  PropensityScoreMatching•  Difference-in-Differences•  InstrumentalVariables•  RegressionDiscontinuity

•  Todaywe’llfocusondifference-in-differences–  Reminderonbasicconcepts/theory–  ApplicationsinStata

Learning objectives •  Bytheendoftoday’ssession,youshouldbeableto:

1.  UnderstandthekeyassumptionofDIDmodels2.  WritedowntheregressionequationforaDID

model3.  Identifywhichparameterintheregression

equationrepresentsthecausaleffectofinterest4.  EstimateaDIDmodelinStataandinterpretthe

results

3/1/18

3

References Thetrainingmaterialsforthissession(developedbyNicoleMason)drawheavilyon:•  Khandker,S.R.,Koolwal,G.B.andSamad,H.A.,2010.

Handbookonimpactevaluation:quantitativemethodsandpractices.Washington,DC:WorldBankPublications.Availablehere.

Othersuggestedreadingsandreferences-DIDsectionsin:•  Angrist,J.D.,&Pischke,J.S.(2009).Mostlyharmless

econometrics:Anempiricist'scompanion.PrincetonUniversityPress.

•  Angrist,J.D.,&Pischke,J.S.(2015).Mastering‘metrics:Thepathfromcausetoeffect.PrincetonUniversityPress.

•  Imbens,G.W.,&Wooldridge,J.(2007).What’snewineconometrics?Difference-indifferencesestimation,Lecturenotes10forNBERsummer2007.

•  Ravallion,M.(2008).Evaluatinganti-povertyprograms.Handbookofdevelopmenteconomics,4,3787-3846.

REVIEW: With vs. Without

•  ThekeycomparisonwewanttomakeinIEisbetweenoutcomesWITHVS.WITHOUTtheintervention(project/program/policy)

•  Impact=“With”outcome–“without”outcome

3/1/18

4

Impact=With-Without

Source:Khandkeretal.(2010)

“Thekeychallengeinimpactevaluationisfindingagroupofpeoplewhodidnotparticipate,butcloselyresembletheparticipantshadthoseparticipantsnotreceivedtheprogram.Measuringoutcomesinthiscomparisongroupisascloseaswecangettomeasuring‘howparticipantswouldhavebeenotherwise.”J-PALIntroductiontoEvaluations

Participants’incomeWITHtheprogram?•  Y4Participants’incomeWITHOUTtheprogram(counterfactualincome)?•  Y2Programimpact?•  Y4-Y2

Introducing Difference-in-Differences (DID) •  WhohasusedDIDbeforeandwhatwereyoustudying?•  WhatistheDIDapproachtoconstructingacomparisongroup/

approximatingthecounterfactual,andhowistheDIDtreatmenteffectcalculated?–  “TheDIDestimatorreliesonacomparisonofparticipantsandnon-participantsbeforeandaftertheintervention”(Khandkeretal.2010,p.72)

–  DIDImpact=(avg.ΔYparticipants)-(avg.ΔYnon-participants)•  (avg.YTafter-avg.YTbefore)–(avg.Ycafter-avg.Ycbefore)•  àwhyit’scalleddifference-in-differencesordoubledifference

3/1/18

5

DID – visual representation DIDimpact=(avg.YTafter-avg.YTbefore)–(avg.Ycafter-avg.Ycbefore)

Source:Khandkeretal.(2010)

Changeinparticipants’income?•  Y4-Y0Changeinnon-participants’(control)income?•  Y3-Y1DIDimpact?•  (Y4-Y0)-(Y3-Y1)

DID key assumption: parallel (common) trends Paralleltrends=“unobservedcharacteristicsaffectingprogramparticipationdonotvaryovertimewithtreatmentstatus”(Khandkeretal.2010,p.73)•  I.e.,trendsintheoutcomevariablewouldbethesameinthetwogroups

withouttreatment(Angrist&Pischke2009);or•  “…treatmentandcontroloutcomesmoveinparallelintheabsenceof

treatments”(Angrist&Pischke(2015,p.178)•  Implies(Y1-Y0)=(Y3-Y2)

Source:Khandkeretal.(2010)

Changeinparticipants’income(after–before)?•  Y4-Y0Changeinnon-participants’(control)income(after–before)?•  Y3-Y1DIDimpact?•  (Y4-Y0)-(Y3-Y1)•  =Y4-Y0-Y3+Y1=Y4-Y3+Y1-Y0Substitutein(Y1-Y0)=(Y3-Y2)(paralleltrendassumption):•  =Y4-Y3+Y3-Y2=Y4-Y2Sameaswithvs.without!

3/1/18

6

What happens if trends are not parallel? •  DIDestimateisbiased•  àSeenext2slidesandhandoutbasedonfiguresinRavallion(2008)forillustration

KeyassumptionforDID:Paralleltrends

Source: Ravallion (2008)

Selection bias

Same

DID estimate = Treatment effect

Treatment effect

Legend:-Blue=controlgroup-Gray=treatmentgroup-Striped=counterfactualfortreatmentgroup

Parallel

3/1/18

7

Selection bias

Source: Ravallion (2008)

Different

Non-paralleltrendscauseDIDtobebiased

Treatment effect

DID estimate ≠ Treatment effect (here DID estimate < Treatment effect )

Legend:-Blue=controlgroup-Gray=treatmentgroup-Striped=counterfactualfortreatmentgroup

NOTparallel

Can partially test for parallel trends if have multiple pre-treatment waves of data

EX)Masonetal.(2017)studyoneffectsofKenya’sNAAIAPinputsubsidyprogramonHHbehaviorandwelfare•  Use3wavesofdata:

–  2wavespriortoimplementationofNAAIAP(2004&2007TAPRAsurveys)–  1waveafter(during)implementation(2010TAPRAsurvey)

•  Regresschangeinoutcomevariable(2007minus2004)ondummyforifHHwasNAAIAPparticipantin2010(“Treated”)andbaseline(2004)controls

•  Whilenoguaranteethattrendswouldhavebeenparallel2007to2010inabsenceoftreatment,if“Treated”dummyisn’tstat.sig.above,itprovidessomeevidenceinsupportofparalleltrendsassumption

3/1/18

8

DID simple numerical example

Source:Khandkeretal.(2010)

DIDimpact=(avg.YTafter-avg.YTbefore)–(avg.Ycafter-avg.Ycbefore)

DID data requirements •  Repeatedcross-sections

– Separaterandomsamplesfromtherelevantpopulationbeforeandaftertheproject/program/policychange

OR•  Paneldata

– Randomsamplefromtherelevantpopulationanddatafromthesamecross-sectionalunitsbeforeandaftertheproject/program/policychange

3/1/18

9

Regression DID – Basic Setup Yit=α+γTreatedi+λAftert+δ(Treatedi×Aftert)+εit

Where:•  iindexesthecross-sectionalunitandtindexestime•  Treated=1ifunitisultimatelytreated(exposedtoproject/program/policychange),

=0o.w.(specifiedasatime-constantvariable)•  After=1iftimeperiodisaftertheproject/program/policychange,

=0before(changesovertimebutnotacrossunitsinthedataset)•  Treated×Afteristheinteractionofthesetwovariables•  *Note:Notationaboveisforwhen“treatment”ortheproject/program/policychange

isatthesamelevelastheoutcomevariable.We’lllookathigherlevelchangesnext.•  Whichparameterrepresentsthecausaleffectofinterest(assumingthekey

assumptionshold)? δ(theparameterontheTreatedXAfterterm)

Regression DID – Basic Setup - with higher level project/program/policy change

Supposetheproject/program/policychangeisatthedistrict(d)levelbutyouhavedataatthehouseholdlevel(i).Thenthenotationwouldbe:

Yidt=α+γTreatedd+λAftert+δ(Treatedd×Aftert)+εidt

•  Whichparameterrepresentsthecausaleffectofinterest(assumingthekeyassumptionshold)?

•  ThisisthemorecommoninstanceinwhichDIDisused•  Wouldwanttoclusteryourstandarderrorsatthedistrictlevel

δ(theparameterontheTreatedXAfterterm)

3/1/18

10

Regression DID – Basic Setup – with covariates

Canalsocontrolforadditionalcovariates:

Yidt=α+γTreatedd+λAftert+δ(Treatedd×Aftert)+Xidtβ+εidt

•  Boldrepresentsvectors•  Whichparameterrepresentsthecausaleffectofinterest(assumingthe

keyassumptionshold)? δ(theparameterontheTreatedXAfterterm)

EX)2periods:Yit=α+λAftert+δTreatedit+ci+uit

•  Firstdifferencetoremoveci(orestimateviaFE)ΔYi=λ+δΔTreatedi+Δui

•  Whichparameteristhecausaleffectofinterest(assumingthekeyassumptionshold)?•  δ

Panel FE setup without control variables Source:Imbens&Wooldridge(2007)withminorchangestonotation

3/1/18

11

EX)2ormoreperiodsYit=α+Yeartλ+δTreatedit+Xitβ+ci+uit

•  WhereYearisavectorofyeardummies•  EstimateviaFE

•  Whichparameteristhecausaleffectofinterest(assumingthekeyassumptionshold)?•  δ

Panel FE setup with control variables Source:Imbens&Wooldridge(2007)withminorchangestonotation

Examples & tweaking the variable names/notation to fit your particular situation (1)

General:Yit=α+γTreatedi+λAftert+δ(Treatedi×Aftert)+εitExample:Wooldridge(2002)-newgarbageincineratorandeffectsonhousingvaluesinacityinMassachusetts•  Newgarbageincineratorconstructionbeganin1981•  Havehousingvaluedatafrom1978and1981plusinfoondistancefromhouse

tonewincinerator(let“rprice”bethehomevalueinrealUS$)•  Let“nearinc”=1ifhouseisnear(within3miles/4.83km)incinerator,=0o.w.

(farfromincinerator)•  Let“y81”=1ifyearis1981,and=0o.w.(yearis1978)•  HowwouldyouwritetheDIDregressionequationinthiscase?Whatisthekey

parameterofinterest?

Zambiaexample:Proximitytofarmblock&landvalues

3/1/18

12

Examples & tweaking the variable names/notation to fit your particular situation (1)

General:Yit=α+γTreatedi+λAftert+δ(Treatedi×Aftert)+εitExample:Wooldridge(2002)-newgarbageincineratorandeffectsonhousingvaluesinacityinMassachusettsSpecific:rpriceit=α+γnearinci+λy81t+δ(nearinci×y81t)+εit•  Where

–  “rprice”isthehomevalueinrealUS$–  “nearinc”=1ifhouseisnear(within3miles/4.83km)incinerator,=0o.w.

(farfromincinerator)–  “y81”=1ifyearis1981,and=0o.w.(yearis1978)

Whichparametercapturestheeffectofinterest? δ!

Examples & tweaking the variable names/notation to fit your particular situation (2)

General:Yit=α+γTreatedi+λAftert+δ(Treatedi×Aftert)+εitExample:Angrist&Pischke(2009)–understandingcholeratransmissioninmid-1800s•  Havedistrict-leveldatafromLondonondeathratesandwatercompanyin1849&

1852.Let:–  “deathr”bethedeathrateindistrictd–  “Lambeth”=1ifdistrictdgetsitswaterfromtheLambethCompany(which

moveditswaterworkstoacleanersectionoftheThamesRiverin1852),=0ifthedistrictgetsitswaterfromtheSouthwarkandVauxhallCompany

–  “y1852”=1ifyearis1852,and=0o.w.(yearis1849)•  WritedowntheDIDequationforthisscenarioandusingthesevariablesSpecific:deathrdt=α+γLambethd+λy1852t+δ(Lambethd×y1852t)+εdt

JohnSnow(Britishphysician)believedtobethepioneeroftheDIDidea!

3/1/18

13

Examples & tweaking the variable names/notation to fit your particular situation (3)

General:Yidt=α+γTreatedd+λAftert+δ(Treatedd×Aftert)+εidtExample:Angrist&Pischke(2009)–effectofémin.wageonfastfoodemployment•  Havedatafromneighboringstates(NJ&PA).Bothstateshave$4.25minimumwagein

early1992butNJraisesminimumwageto$5.05inApril1992.Youhaveemploymentdatafromindividualfastfoodrestaurants(i)ineachstate(s)before(Feb.1992)andafter(Nov.1992)thepolicychange.Let:–  “employ”betheemploymentlevelofeachrestaurant–  “NJ”=1ifstateisNJ,=0ifstateisPA–  “Nov92”=1iftimeisNovember1992,and=0o.w.(timeisFebruary1992)

•  WritedowntheDIDequationforthisscenarioandusingthesevariables.Thinkcarefullyaboutwhichsubscriptstoputoneachvariable.

Specific:employist=α+γNJs+λNov92t+δ(NJs×Nov92t)+εist

DID good to consider if have natural experiment EX)Dillon,B.(2016).Sellingcropsearlytopayforschool:Alarge-scalenaturalexperimentinMalawi.WorkingPaperNo.243.Abidjan,Côted’Ivoire:AfricanDevelopmentBank.•  Bigpicturequestion:whydomanyHHsselllow,buyhigh(w.r.t.cropprices)?•  Hypothesis:“short-termexpenditureneedsforcepoorhouseholdstosell

cropsearly,whenoutputpricesarewellbelowtheirpeak”(p.4).I.e.,“farminghouseholdsthatarecredit-constrainedsellcropsearlytofinanceimmediateneeds”(p.12)

•  Naturalexperiment:Malawichangedprimaryschoolcalendar–  2009:startinDecember. 2010:startinSeptember.à HHshavetomakeschool-relatedexpendituresmuchearlierin2010than

2009.HHswithschool-agedchildrenarethemainonesweexpecttochangetheirbehaviorinresponsetotheschoolcalendarchange.

3/1/18

14

DID & natural experiment example (cont’d) Dillon,B.(2016).Sellingcropsearlytopayforschool:Alarge-scalenaturalexperimentinMalawi.WorkingPaperNo.243.Abidjan,Côted’Ivoire:AfricanDevelopmentBank.

DIDregression:General:Yit=α+γTreatedi+λAftert+δ(Treatedi×Aftert)+εitSpecific:Cropsalesit=α+γChildreni+λy2010t+δ(Childreni×y2010t)+εitWhere•  Cropsales=cumulativevalueofHHcropsalesthroughAugustofyeart•  Children=#ofchildreninprimaryschool(0,1,2,3,etc.).Orcoulddo0/1•  y2010=1ifyearis2010;=0ifyearis2009•  EstimateseparatelyforHHsabovevs.belowthepovertyline

What other situations/examples appropriate for DID can you think of?

•  AndhowwouldyousetupyourDIDregressionequation?•  General:Y=α+γTreated+λAfter+δ(Treated×After)+ε

3/1/18

15

DID vs. other methods •  KeydifferencebetweenPSMandDID:PSMassumes

selectiononobservablesonly,DIDallowsselectiontobeafunctionoftime-constantunobservedfactors(a.k.a.timeinvariantunobservedheterogeneity)–  Wherehaveyouheardthistermbefore?–  Whatifselectionisafunctionoftime-varyingunobservables?

•  Anotherkeydifference:–  Randomizedevaluations&PSM–cross-sectionaldatasufficient(althoughpaneldatabetter–baseline/endline)

–  NeedrepeatedcrosssectionsorpaneldataforDID

ParaphrasedfromKhandkeretal.(2010)

Theory wrap-up (Source: Angrist & Pischke 2015, pp. 203-204)

“MasterStevefu:Wrapitupforme,Grasshopper.Grasshopper:Treatmentandcontrolgroupsmaydifferintheabsenceoftreatment,yetmoveinparallel.ThispatternopensthedoortoDDestimationofcausaleffects.MasterStevefu:WhyisDDbetterthansimplytwo-groupcomparisons?Grasshopper:Comparingchangesinsteadoflevels,weeliminatefixeddifferencesbetweengroupsthatmightotherwisegenerateomittedvariablesbias.…MasterStevefu:OnwhatdoesthefateofDDestimatesturn?Grasshopper:Paralleltrends,theclaimthatintheabsenceoftreatment,treatmentandcontrolgroupoutcomeswouldindeedmoveinparallel.”

3/1/18

16

In Stata – DID (panel or pooled cross-sections) General:Yit=α+γTreatedi+λAftert+δ(Treatedi×Aftert)+εitInStata?regYi.Treatedi.Afteri.Treated#i.After(where“Treated”hereistime-constant)Forpaneldataorifhaverepeatedcross-sectionsandpolicychangeisathigherlevelthandata,considerclusterings.e.’s(seeAngrist&Pischke2015DIDchapterfordetails)

WhichcoefficientistheDIDestimateofthecausaleffectofinterest(assumingthekeyassumptionshold)?•  Theoneonthei.Treated#i.Aftervariable

In Stata – FE (panel data) General:Yit=α+λAftert+δTreatedit+ci+uitInStata?xtregYi.Afteri.Treated,fe(where“Treated”hereistime-varying)Considerclusterings.e.’s(seeAngrist&Pischke2015DIDchapterfordetails)

WhichcoefficientistheFEestimateofthecausaleffectofinterest(assumingthekeyassumptionshold)?•  Theoneonthei.Treatedvariable

3/1/18

17

Stata exercises 1.   Wooldridge(2002)newgarbageincinerator&

housingvaluesa.  UseKIELMC.DTAin“data”foldertoestimate:

rpriceit=α+γnearinci+λy81t+δ(nearinci×y81t)+εitRecall:regYi.Treatedi.Afteri.Treated#i.After

b.  Interpretthekeycoefficientofinterest

Stata exercises 2a.Khandkeretal.(2010)exampleoneffectsofamicrocredit

programonHHwelfare(expenditure)–DIDDatafromBangladeshin1991&1998onlogtotalHHexpenditurepercapita(lexptot)&participationinmicrocreditprogramin1998(dfmfd98).Assumenomicrocreditprogramin1991.

a.  Usehh_9198.dtain“data/Khandkeretal2010datafiles”foldertoestimate:lexptotit=α+γdfmfd98i+λyeart+δ(dfmfd98i×yeart)+εit(whereyear=1if1998,=0if1991)Recall:regYi.Treatedi.Afteri.Treated#i.After

b.  Interpretthekeycoefficientofinterest

3/1/18

18

Stata exercises 2b.Khandkeretal.(2010)exampleoneffectsofamicrocredit

programonHHwelfare(expenditure)–FEThesedataareactuallyHHpanelsurveydata.NowestimateviaFEinstead.Time-varyingvariableforparticipationinmicrocreditprogramisdfmfdyr.RecallforHHpaneldata,canwriteFEmodelas:

Yit=α+λAftert+δTreatedit+ci+uita.  ContinueusingthesamedatasetandestimateviaFE:

lexptotit=α+λyeart+δdfmfdyrit+ci+uitRecall:xtregYi.Afteri.Treated,fe

b.  ComparethekeycoefficientestimatesbetweenDID&FE

Thank you for your attention & participation! NicoleM.Mason(masonn@msu.edu)AssistantProfessorDepartmentofAgricultural,Food,&ResourceEconomicsMichiganStateUniversity

3/1/18

19

Acknowledgements

ThistrainingwassupportedbyfundingfromtheU.S.AgencyforInternationalDevelopmentthroughthe

ZambiaMissionBuy-IntotheInnovationLabforFoodSecurityPolicy.

www.feedthefuture.gov