EffectivenessofInteractiveDistanceInstruction:ExperimentalEvidencefromGhanaianPrimarySchools
JAM I E J O HN S TON – S TAN FO RD UN I V E R S I T Y
CHR I S TO PH E R K SO L L – MATH EMAT I C A P O L I C Y R E S E A RCH
GhanaEducationEvidenceSummit2017
PhotoCredit:VarkeyFoundation
Howcanwedeliverhigh-qualityteachingtoruralstudents?
• Studentsinruralareassignificantlyunderperform(OECD2013,GES2012,WorldBank2012)
• Inremoteruralareas,shortageoftrainedprimaryteachers(WorldBank2012)
• Teachingqualitymattersforstudentlearning(Glewwe etal.2011,McEwan2015)
• Structuredpedagogyinterventionsappeartohavelargest,mostconsistentpositiveeffectsonlearning(Ganimian &Murnane 2016,Snilstveitetal2015,Kremeretal.,2013)
• Littleknownabouteffectivenessofinteractivedistanceinstructionmodels
VideoCredit:VarkeyFoundation
MGCubedSatelliteClassProgram• Varkey FoundationMGCubedProgram
• Equipped72schoolswithtechnologypackage
• Broadcastslivelessonsdailytoremoteclassrooms§ 6trainedstudioteachers,eachbroadcastingto12classesatatime§ 1hourmathand1hourEnglish,5daysaweek
• In-personteacherstrainedas“facilitators”
• Targetsmostmarginalizedstudentswithinschools§ Grades2-5§ Girlsandboys§ Identifiedaccordingtocriteria:distancetoschool,numberofsiblings,truancy,age
PhotoCredit:VarkeyFoundation
Girls’AfterSchoolProgram• Providesafter-schoolsessionstoin-schoolgirlsandout-of-schoolgirlsincommunity
(whohaveleftschool)
• 1 hourperday,4daysaweek
• Engagegirlsintopicsincluding§ Earlypregnancy§ Earlymarriage§ Girls’rights§ Financialliteracy
• Providesaccesstoadultfemalerolemodels
• Boys’monthlyafter-schoolprogramstartedinresponsetodemandforsimilarprogramPhotoCredit:VarkeyFoundation
TheoryofChangeImprovequantityandqualityofprimaryschooleducation
Increaseinstructionaltime-on-task:§ Reduceabsenteeism§ Increaseclassroomtime-on-task§ “Double”instructorpresence
Improveteachingpedagogy:§ Leveragestudioteachercapacity§ Improvingfacilitatorcapacity
Improvecurricularcontent:§ Moreengaging§ Moreinteractivity
In-Schoo
l
Gainsinnumeracyandliteracy
ResearchQuestions:
1) Istheprogrameffectiveatincreasingskills?
2) Whatarepotentialmechanismsofchange?
EvaluationApproach
cohortreceivesintervention
ClusterRandomizedControlledTrial(RCT)§ 70treatmentand77controlschools,assignedatbaseline§ Followedcohortofstudents(grades2-4atbaseline)fortwofullyears
Interventionbegins Year1 Year2
Jan2014baseline
Jun-Jul2014monitoring1
Year0
Feb-Mar2015monitoring2
Jun-Jul2016endline
Jan-Feb2017postendline
Jun-Jul2015midline
Mar-Apr2016monitoring3
Cohort:Grade2 Grade3 Grade4
Cohort:Grade3 Grade4 Grade5
Cohort:Grade4 Grade5
Cohort:Grade5(excludedfromRCT)
datacollections
2yearimpact
Grade6
1yearimpact
cohortnotreceivingintervention
Data1) Skillsassessments
§ Baseline,midline,andendline§ Literacy:EarlyGradeReadingAssessment(EGRA)§ Numeracy:EarlyGradeMathematicsAssessment(EGMA)
2) Studentandteacherdata§ Studentsurveysatbaseline,midlineandendline§ TeachersurveyinYear2§ Attendancemeasures(recordsandspotchecks)inYears1and2§ Qualitativeinterviewsandfocusgroupdiscussionswithstudents,teachersandadministrators
3) Classroomobservations§ Year2§ AdaptedStallingsObservationProtocol§ Snapshotsyieldingproportionofclasstimeon(1)activeinstruction,(2)passiveinstruction,(3)classroom
management,and(4)offtaskactivities
MeasuringLearning:EGRA&EGMA• EarlyGradeReadingAssessment(EGRA)
§ Oraltestmeasuringsubtasks:§ Lettersperminute§ Wordsperminute(WPM)<- focalmeasure§ Inventedwordsperminute§ Listeningcomprehension§ Oralvocabulary§ Readingcomprehension
§ BaselineWPM:11.5§ Baselinestandarddeviation:21.2
• EarlyGradeMathematicsAssessment(EGMA)§ Oraltest,measuringsubtasks:
1. Numberidentification(20%)2. Quantitydiscrimination(20%)3. Missingnumberpatterns(20%)4. Addition/subtraction/wordproblems(40%)
§ Baselinemeanscore:46.0percentcorrect§ Baselinestandarddeviation:20.1
focalmeasure:combinedtotal
SOURCE:RTIInternational,2009
EGRAWPMExamplePassage:
• N=5485atbaseline- 60%girls- Averageage=10.9- ~8householdmembers- 50%mothersattendedschool- 60%fathersattendedschool
• Treatmentandcontrolbalancechecks- Balanceonobservablesatbaseline,midline,andendline- Midlineattrition=13%- Endlineattrition=16%- Nodifferenceinattritionbetweentreatmentandcontrolgroups
SampleCharacteristics
MainResultsWhatistheeffectofsatelliteprogramonliteracyandnumeracyskills?
◦Classessignificantlyimprovednumeracy◦ 0.22standarddeviationimprovementinyear1◦ 0.26standarddeviationimprovementinyear2
◦Gainsinmathscoreslasted oneyearaftertheprogramended◦ 0.23standarddeviationgainsforstudentstestedoneyearafterprogramended
◦Aftertwoyears,marginallysignificantimprovementinliteracy◦ 0.07standarddeviationgainsaftertwoyears
GainsinNumeracyandLiteracy
◦ After2yearsofclasses,largeandsignificantimprovementinfoundationalskills◦ 0.82standarddeviationgaininlettersperminute◦ 0.17standarddeviationgainininventedwordsperminute◦ 0.17standarddeviationgaininoralvocabulary
LiteracySubtasks
0.07
0.82
0.17 0.17
0.04 0.000
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Words per minute Letters per minute Invented words per minute
Oral vocabulary Listening comprehension
Reading comprehension
Stan
dard
dev
iatio
n ga
ins
Two years of classes***p<0.01,**p<0.05,*p<0.1
***
****
**
◦ Afteroneyearofclasses,significantimprovementinallnumeracysubtasks◦ Aftertwoyearsofclasses,greatergainsinnumberidentityand missingnumbertasks
NumeracySubtasks
0.2
0.150.17
0.19
0.34
0.12
0.250.21
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
Number identification Quantity discrimination Missing number Addition/subtraction
Stan
dard
dev
iatio
n ga
ins
One year of classes Two years of classes***p<0.01,**p<0.05,*p<0.1
***
******
***
***
**
******
*** *
MechanismsWhatarethepotentialdriversoflearninggains?
TheoryofChangeImprovequantityandqualityofprimaryschooleducation
Increaseinstructionaltime-on-task:§ Reduceabsenteeism§ Increaseclassroomtime-on-task§ “Double”instructorpresence
Improveteachingpedagogy:§ Leveragestudioteachercapacity§ Improvingfacilitatorcapacity
Improvecurricularcontent:§ Moreengaging§ Moreinteractivity
In-Schoo
l
Gainsinnumeracyandliteracy
§ ->Noevidenceofeffectsonattendance§ ->Noevidenceofeffectsontime-on-task§ ->BUT:In-classroomfacilitatorsengagedinon-task
activities(includingactiveinstruction)morethanhalfofthetimethatthevideowasbeingshown,essentiallyincreasingclassrooms’teacher-studentratio
TheoryofChangeImprovequantityandqualityofprimaryschooleducation
Increaseinstructionaltime-on-task:§ Reduceabsenteeism§ Increaseclassroomtime-on-task§ “Double”instructorpresence
Improveteachingpedagogy:§ Leveragestudioteachercapacity§ Improvingfacilitatorcapacity
Improvecurricularcontent:§ Moreengaging§ Moreinteractivity
In-Schoo
l
Gainsinnumeracyandliteracy
§ ->Noquantitativeevidenceofimprovedteachingmethods§ ->Qualitativeevidencesuggestsimprovements
Mechanisms:TeachingPedagogy• Studentfocusgroups/interviewssuggestimprovementsinlearningexperience
“Sometimesteachersleaveyoubehindifyoudon’tunderstandandevencaneus.MGCubedteacherstrytheirbesttomakeusunderstandlessons.”
• Facilitatorsseenaseffectiveteachingintermediaries“ThefacilitatorintheMGCubedclassexplainswhatthestudioteacherteachesifwearefindingitdifficulttounderstandthestudioteacher.”
• Newteachingmethodsandapproacheslearnedaspartoffacilitatorrole“Andtheytaughtuswayswecanusetoteachthem...Thoughweweretaught‘thinkpairshare’atthetrainingcollegeIwasnotusingitsowhenwehadtheprogrammerecentlyIwentbackandamnowusingit.”
TheoryofChangeImprovequantityandqualityofprimaryschooleducation
Increaseinstructionaltime-on-task:§ Reduceabsenteeism§ Increaseclassroomtime-on-task§ “Double”instructorpresence
Improveteachingpedagogy:§ Leveragestudioteachercapacity§ Improvingfacilitatorcapacity
Improvecurricularcontent:§ Moreengaging§ Moreinteractivity
In-Schoo
l
Gainsinnumeracyandliteracy
§ ->Stillinvestigating
PolicyImplicationsandNextSteps• Interactivedistancemodeleffectiveatincreasingliteracyandnumeracyskills
• Increasedinstructorpresenceinformofdistanceand in-personinstructoranimportantlevertoexplore– aretherewaystomakerolesmorecomplementary?
• Forthcomingisacosteffectivenessanalysistodeterminehowinterventioncomparestootherheavier-handedstructuredpedagogyprograms
• Relevanceofinteractivemodelincreasingascostsofinternetanddistanceinstructiondecreasing andinstitutionsincreaseuseofonline/blendedinstruction
LinktoWorkingPapercepa.stanford.edu/sites/default/files/wp17-08-v201708.pdf
ThankYouQuestionsandfeedbackwelcome!