philip c. abrami, anne wade, larysa lysenko jonathon marsh ......elementary school students: a brief...
TRANSCRIPT
TheEffectsofABRACADABRAEarlyLiteracySoftwareontheLearningofKenyan
ElementarySchoolStudents:ABriefReportonthe2015Study123
PhilipC.Abrami,AnneWade,LarysaLysenkoCentrefortheStudyofLearning&Performance
ConcordiaUniversity,Montreal,Canada
JonathonMarsh,AgaKhanAcademiesUnit,Paris,France&GiokoMaina,AgaKhanAcademy,Mombasa
REVISED:February17,2017Background
In2012,theCentrefortheStudyofLearningandPerformance(CSLP)joinedwiththeAgaKhanAcademiesunitandtheAgaKhanAcademy,Mombasa(AKA)andtheytogetherembarkedonapreliminaryinvestigationtoexplorethefeasibilityandimpactofusingABRACADABRA(ABRA),earlyliteracysoftwareongrade-onestudentsinMombasa,Kenya.ChildrenfromlocalschoolswerebussedtotheAgaKhanAcademy,MombasaandusedtheAcademycomputerlabtoaccessABRA.Foraperiodofseveralmonthsaboutonceperweekforuptotwohourstheyusedthesoftwareunderthecarefulsupervisionoftrainedteachersandstaff.Controlstudentsreceivedtraditionalinstructionfollowingthenationalcurriculum.Attheconclusionoftheprogram,ABRAstudentsshowedmoresignificantgainsonGRADE,astandardizedtestofreadingskills,thantheirpeersincontrolclasses.ABRAstudentsalsoscoredhigheronseveralKenyancoresubjectexaminationincludingEnglish,Math,ScienceandSocialStudies.In2013,thecontrolstudentsusedABRAandtheyalsoshowedimprovementsinliteracyskills.
In2014,theresearchwasextendedbutwithanimportantdifference.InsteadofbringingthechildrentotheAcademylabstouseABRAunderthesupervisionofstaff,theprojecttookplaceinschoolcomputerlabs.TheexperimentalteachersweretrainedintheuseABRAandlearnedstrategiesonhowtointegratethesoftwareintotheirEnglishLanguageinstruction.Theywerealsoprovidedwithongoingsupportbymeansofbi-weeklyplanningsessionsandregularschoolvisits.ABRAambassadors,seasonedmasterteacherswhohad
1ThisresearchwasmadepossiblebyagranttotheauthorsfromtheSESEAproject,AgaKhanFoundationCanadaandtheSSHRCPartnershipDevelopmentprogram.2TheauthorsexpresstheirappreciationtotheKenyancoordinatorsandambassadorswhoworkedextensivelytotrainandsupporttheteacherswhoimplementedABRACADABRAintheirclassrooms.TheauthorsarealsogratefultotheCAMARAorganizationwhooutfittedtheschoolcomputerlabsandprovidedregularmaintenanceduringtheABRAintervention.3ForadditionalinformationaboutABRAoranyofthesoftwaretoolsintheLearningToolkitpleasecontactAnneWadeatwada@education.concordia.ca.
ABRA2015Study
2
startedusingABRAasearlyas2012,facilitatedthesesupportactivities.ThistimetheexperimentalparticipantsusedtheABRAsoftwareoverseveralmonths,withthecontrolparticipantsnotusingABRA.TheGRADEreadingtestthatthestudentstookwhentheinterventionwascompletedalsoshowedthattheABRAstudentsoutperformedtheirpeerswhodidnotuseABRA.The2015Experiment
The2015study,whichwearebrieflyreportinghere,wasalsoconductedinpublicschoolsinMombasaandNairobi.Similarto2014,teachersimplementedtheABRAinterventionintheirclassroomsforseveralmonths.Butkeydifferencesincludedtestingallstudentsatthebaseline,beforetheyusedABRAaswellastestingthemattheconclusionoftheintervention.Inaddition,wewereabletoestablishpretestequivalenceoftheABRAstudentsversusthecontrolstudents.Inotherwords,the2015studywasastrongertestofABRAconductedundertheauthenticconditionsofKenyanclassroomuse.Thisstudywasapretest-posttestcontrolgroupquasi-experimentaldesignwherewestatisticallycontrolledfortheinitialdifferencesbetweenABRAandcontrolstudents.Therewereatotalof22teachers(11ABRAexperimentalteachers,11controlteachers)andtheir749students(307ABRAstudents,442controlstudents)Studentswerefromgrades1,2&3.SomehadusedABRAthepreviousyear,whileotherswerefirsttimeusers.Wealsorecordedstudentgendersothatwecouldcomparechangesinliteracyscoresfor366boysversus383girls.InadditiontocollectingGRADEstandardreadingtotalscoreandsubtestscores,wealsocollectedandanalyzedKenyannationalexaminationresults.Finally,weusedanumberofdifferentstatisticalstrategiestoanalyzethescores.Theresultswereconsistentacrossalltheseanalyses.ResultsFirst,wecomparedABRA(N=307)andcontrol(N=442)groupsatthepretest.ThistestofgroupdifferencesdidnotrevealanysignificantdifferencesbetweentheABRAandcontrolgroupsontheGRADEsubtestsandtotaltestscores(pvocabulary>.09;pcomprehension>.26;ptotaltest>.7;plisteningcomprehension>.06).Overall,theseresultslendcredibilitytothenotionthattheexperimentalandcontrolstudentsweremostlyequivalentinliteracyskillsattheoutsetoftheABRAintervention.AsummaryoftheANOVAresultsofthedifferencescoresonallthesubscalesoftheGRADEtestsisreportedinTable1.Table1.GRADEmeans,standarddeviations,changescoresanddifferencebetweenABRAandcontrolstudents
GRADEscalesABRAmeans(N=307) Controlmeans(N=442)
Fvalueandsignificance(***p<0.00)Post Pre Change Post Pre Change
VocabularyComposite 44.42 41.77 2.65 40.86 42.98 -2.11 53.08***
StandardDeviation 9.88 10.86 7.50 9.46 8.88 9.61
ReadingComprehension 22.56 17.48 5.08 17.21 16.83 0.38 62.71***
StandardDeviation 9.25 7.75 8.94 8.01 7.67 7.28
ABRA2015Study
3
TheresultsshowthatonGRADEsubtestsandTotalTest,studentsinABRAclassesimprovedtheirscoresatasignificantlyhigherratethanstudentsincontrolclasses.Theconsiderableeffectsizeof0.58fortheGRADETotalTestindicatedthatafterhavingusedABRA,averagestudentsintheABRAgroupimprovedby22percentilepointsovertheirpeersinthecontrolgroupwhowereexposedtotraditionalinstruction.Figure1belowrepresentsthedifferenceinmeanchangescoresbetweenthetwogroupsontheGRADETotalscoreovertime.
Figure1.GRADEtestTotalReadingAverageScores(changeovertime)
GenderInadditiontothemainanalysis,weexaminedifABRAeffectsdifferedforboysandgirls.Table2andFigure2splittheGRADETotalTestscoresforboysandgirlsintheABRAandcontrolclasses.ThedatasuggestthatgirlsandboysbenefitedfromABRAwithABRAgirlsdemonstratinghighergainsthanABRAboys.BothgendersintheABRAclassesshowedenhancedperformanceontheGRADEcomparedtostudentslearningtoreadinthetraditionalmanner,withafocusonteacher-directedandrecitationinstruction.
2015 Change in GRADE(Total Test) scores: experimental vs. control
50
55
60
65
70
75
Pretest(Winter,2015) Pos4est(Fall,2015)
Control(N=442)
ABRA(N=307)
TotalTest 66.98 59.54 7.70 58.07 59.73 -1.65 81.58***StandardDeviation 17.74 15.58 13.35 15.92 14.83 14.33 ListeningComprehension 10.91 8.5 2.40 9.48 9.09 0.38 36.60***StandardDeviation 2.89 2.78 3.41 2.98 4.85 5.13
ABRA2015Study
4
Table2.GRADETotalTestMeanScoresbyGender
Figure2.ABRAEffectsSeparatedbyGender
KenyaExamResultsToexamineiftheABRAeffectswereassociatedwithstudents’learninginthecorecurriculumsubjects,weusedtheresultsofend-of-termexamsprovidedbyKenyateachers.Therewasconsiderabledropinthesamplesizebecauseexamresultsofonly464studentshavebeenfurnishedforanalyses.WecomparedthescoresinEnglish,Math,ScienceandSocialStudiesofcontrolandABRAstudentscollectedattheendoftermthreeafterstatisticallyadjustingthesescoresusingtheGRADEpre-testtotalscoreasthecovariate.TheanalyseswereperformedforEnglish,Math,ScienceandSocialStudiesexamscores.TheassociationbetweenGRADEpretestscoresandtheexamscoreswasstatisticallysignificantandrangedfrommodestof.28(Math)to.6(English).
2015 Change in GRADE(Total Test) scores: Gender
50
55
60
65
70
75
Pretest(Winter,2015) Pos4est(Fall,2015)
Controlboys(N=230)
Controlgirls(N=212)
ABRAboys(N=136)
ABRAgirls(N=171)
GRADEscores
ABRA Control
Girls(N=171) Boys(N=136) Girls(N=212) Boys(N=230)
Post Pre Post Pre Post Pre Post Pre
TotalTest 70.53 60.13 62.53 58.21 58.85 60.6 57.36 58.92
StandardDeviation 16.20 16.48 18.63 16.79 16.13 15.04 15.73 14.12
ABRA2015Study
5
ThetestsfortheeffectofABRAonEnglish,Math,ScienceandSocialStudiesexamscoresshowedthedifferencebetweenthegroupswasstatisticallysignificant(p<0.001).ThestatisticssummarizedinTable3andillustratedinFigure3showthatstudentsintheABRAgroupperformedhigherthancontrolstudentsoneachofthefourcoreexams.Table3.CoreexamsresultsforABRAandcontrolclasses(adjustedmeans,andstandarderrorvalues)
1Adjustedmeansforpost-testscoresandstandarderrorcalculatedinthemodel.GRADEpre-testcovariatescoresmodelwereevaluatedat58.18.
Coresubjectexams ABRAMeans1(N=149)
ControlMeans1(N=315)
English 85.86 74.03StandardError 1.34 0.91Mathematics 77.40 68.31StandardError 1.58 1.08Science 86.39 68.87StandardError 1.23 0.84SocialStudies 82.25 65.14StandardError 1.20 0.83
ABRA2015Study
6
Figure3.KenyaEndofYearExaminationResultsbySubjectGradeLevelStudentgradeleveldidnotlessentheeffectsofABRA.StudentsinGrades1,2&3allbenefitedfromABRAregardlessofwhethertheyhadusedABRAinthepreviousschoolyear.Table4andFigure4allowonetofollowhowthemeanGRADETotalTestscoreschangedfrompre-topost-testbygradeinABRAandcontrolgroups.AsexpectedthehighestincreaseisnoticeableforABRAstudentsingrades2and3asrespectively23%and85%ofstudentsinthesegradeshadbeenexposedtoABRAsince2014for1ormoreyears.Table4.GRADETotalTestMeanScoresbyGradeLevel
Difference between the groups: Core end-of-year 2015 exams
10
20
30
40
50
60
70
80
90
English Socialscience Mathema;cs Science
Control(N=315)
ABRA(N=149)
GRADEscores
ABRA ControlGrade1(N=83)
Grade2(N=124)
Grade3(N=100)
Grade1(N=98)
Grade2(N=151)
Grade3(N=193)
Post Pre Post Pre Post Pre Post Pre Post Pre Post Pre
TotalTest 47.58 44.01 69.85 59.6 79.54 71.56 47.37 50.93 51.79 57.46 68.43 65.96
StandardDeviation 14.38 14.95 12.12 13.33 11.57 9.93 8.97 12.26 13.05 14.63 14.53 13.39
ABRA2015Study
7
Figure4.ResultsbyStudentGradeLevelTeacherExperienceWealsoexploredwhetherthereweredifferencesamongteachersbasedonyearsofexperiencesusingABRA.Becauseofinitialtrainingandfollow-upsupportaswellasin-classexperienceoveroneormoreyears,wedidnotexpectthatnoviceuserswouldhavelowereffectscomparedtoexperiencedusers.However,wewereconcernedthatiftherewerenoveltyeffectstheywoulddissipateovertime.However,asFigure5shows,therewerenodecrementsinstudentlearninglinkedtoteacherexperience.
Difference between the groups on GRADE(Total Test) post-test scores: grade level
20
30
40
50
60
70
grade1(Ncontrol=98;NAbra=83)
grade2(Ncontrol=151;NAbra=124)
grade3(Ncontrol=193;NAbra=100)
Control
ABRA
ABRA2015Study
8
Figure5.YearsofTeacherExperienceandABRAEffectsImplications
Theseremarkableresultsareatestamenttotheevidence-baseddesignofABRAandqualityteacherimplementationintheauthenticcontextoftheirschools.ABRAisnotintendedtosupplementclassroominstruction,butinsteadshouldbeintegratedintoclassroominstruction.ThetrainingandsupportprovidedbytheABRAcoordinatorsandambassadorscoupledwithteacherswillingtoworktoimproveteachingarewhatmadethedifference.
SeveralotherimprovementsshouldmaketheefficacyofABRAevenstrongerinyearstocome.First,weexpandedandimprovedourteacherprofessionaldevelopmentresourcesandassembledthemtogetherinTheLearningToolkitTeachersGuide:KenyaEdition.Second,toenhancefurtherstudentfluencyandcomprehensionskills,wedevelopedREADS,afreelyavailabledigitalrepositoryofhundredsofbooksandstoriesinmultiplelanguagesincludingEnglishandKiswahili.READSfitseasilyonaUSBkeyandmeansnoKenyanchildwilllackengagingreadingmaterialtopracticeandhonetheirliteracyskills.Third,wemorecompletelymappedABRAactivitiesandstoriesontotheKenyancurriculumsothatteacherswillfindintegratingABRAintotheirlessonsespeciallyeasy.Fourth,wetackledthechallengeoflargeclasssizesandtherelianceonfrontalteachingbyemphasizingcooperativesmallgrouptechniquesduringABRAlessons.Finally,to
Change in GRADE(Total Test) scores: teacher experience with ABRA
50
55
60
65
70
75
Pretest(Winter,2015) Pretest(Fall,2015)
Oneyear(N=68,grades1,and3)
Twoyears(N=157,grades1,2,and3)
Threeyearsandmore(N=68,grades2and3)
ABRA2015Study
9
complementgender-balancedstoriesandactivitiesinABRA,weofferedsupportmaterialstoteacherstohelpthemadequatelycultivategenderequalityintheirclassrooms.
WeareexcitedaboutthelatestABRAresearchandhopeyouaretoo.Wethinkwe’vetakenanothersteptocreatinganationofreadersinKenyaandbeyond.