exploratory analysis of students’ mathematics achievement ......and their associated students (14...
TRANSCRIPT
Exploratory Analysis of Students’ Mathematics Achievement After Using Freckle
Written in partnership with WestEd
Freckle and Student Mathematics Achievement
ii
Table of Contents
Abstract..........................................................................................................................................................................1
Executive Summary...................................................................................................................................................2
Introduction..................................................................................................................................................................3
Method.........................................................................................................................................................................4
Participants....................................................................................................................................................4
Data Analysis................................................................................................................................................6
Results...........................................................................................................................................................................6
Baseline Equivalence..................................................................................................................................6
Post-Score Analysis………………………………………………………………………………………....……………………………………........7
Discussion.........................................................................................……………………….....................................................8
Study Limitations...................................................………………………...............................................................8
Conclusions....................................................................................……………………….........................................8
References................................................................................................……………………….............................................9
Appendix A........................................................................................……………………….................................................10
Appendix B...........................................................................………………………...............................................................13
Freckle and Student Mathematics Achievement
iii
List of Figures
Figure 1. Sample Report Card data view available for teachers in Freckle Math..............................3
Figure 2. Means for students’ beginning and end of year MAP scores by grade and condition..........5
Figure 3. Covariate-adjusted means for each group’s end of year MAP scores.......................................7
List of Tables Table 1. School Demographic Information..........................................................................................................4
Table 2. Teacher and Student Counts by Grade Included in Final Analytic Sample..............................5
Table 3. Growth norms observed in MAP scores throughout the year.........................................................6
Table 4. Random Effects.........................................................................................................................................11
Table 5. Fixed Effects...............................................................................................................................................11
Freckle and Student Mathematics Achievement
1
ABSTRACTWestEd—anonpartisan,nonprofitresearch,development,andserviceagency—partneredwithFreckletoexplorewhetherstudentswhouseFreckleexhibitdifferentmathematicsachievementoutcomesthanstudentswhodonotuseFreckle.AruralschoolshareddataonFreckleadoptionandstudentscoresontheNorthwestEvaluationAssociation’s(NWEA)MeasureofAcademicProgress(MAP)inmathematicswithFreckle.WestEdusedthesedatatoconductanindependentanalysistodeterminewhetherornotFreckleusewasassociatedwithhigherNWEAMAPtestscores.ResultsareconsistentwiththehypothesisthatusingFreckleintheclassroomresultsinhigherachievementinmathematics.Limitationstothestudyandfutureresearchdirectionsarealsodiscussed.
Freckle and Student Mathematics Achievement
2
EXECUTIVESUMMARYFreckleisacomputerizedmathandreadingcomprehensionprogramthatprovidesadaptivefeedbackandpracticeopportunitiesforK-8students.Freckleishypothesizedtoimprovestudents’readingandmathematicsoutcomesineducation.Inthepresentstudy,WestEdexploredthepotentialofFreckletosupportstudents’mathematicsachievementinKindergarten,Grade1,andGrade2classrooms.Studyparticipantsconsistedof466Kindergarten,Grade1,andGrade2studentsand25teachersinaruralelementaryschoolduringthe2014-2015academicyear.TeacherschosetouseFreckleorchosenottouseFreckleduringthecourseoftheschoolyear.Approximatelyhalfoftheteachersandtheirassociatedstudents(14teachers,263students)usedFreckleintheirclassrooms,whereastheotherhalf(11teachers,203students)participatedinbusiness-as-usualclassroomactivities.ThisstudyexploreswhetherresultsareconsistentwiththepredictionsthatFrecklesupportsstudentlearninginmathematics.Towardsthisend,students’endofyearmathematicsscoresontheMeasureofAcademicProgress(MAP)assessmentwascollectedforeachstudent.PreliminaryanalysesindicatedthatFrecklestudentsexhibitedhigherbeginningofyearMAPscoresthanthecomparisonstudents.However,ananalysisthatattemptedtocontrolforbeginningofyearMAPscoredifferencesaswellasgradelevelindicatedthatstudentsintheFreckleconditionscoredapproximately4.6pointshigherontheirendofyearMAPscoresrelativetostudentswhodidnotuseFreckle,astatisticallysignificantdifference(p<.001).TheresultsofthisexploratorystudyareconsistentwiththehypothesisthatFrecklepositivelyimpactsstudents’mathematicsachievementinK-2Grades.However,becauseteacherswerenotrandomlyassignedtouseFreckle,theresultsshouldnotbeusedtoclaimthatFrecklecausedhigherstudentachievement.Inaddition,thestudyreliesonadatasetwithlimitedinformation:TeacherschosetouseFreckleornotforreasonsunknowntothestudyteam.Inaddition,potentiallyimportantdemographicinformationonteachersandstudentsaremissing,therebylimitingthegeneralizabilityofthefindings.FuturestudiescanbeconductedthatutilizerandomassignmenttogroupsandthatcollectinformationonfactorsthatmaypotentiallyinfluencetheeffectivenessofFreckle.SuchstudieswillincreaseconfidencethatFreckleuseleadstoimprovedmathematicsoutcomes.
Freckle and Student Mathematics Achievement
3
INTRODUCTIONFreckleisanadaptivemathandreadingcomprehensionprogramthatensureseverystudentreceiveshigh-quality,individualizedmathandreadingpractice.Byaddressingstudentweaknessesandbuildingoffofstrengths,Freckleisdesignedtoincreaseconfidenceandpromotegrowthinallstudents,regardlessoftheirabilitylevel.WhenstartingonFreckle,studentstakeadiagnostictodeterminewhattheyalreadyknowandareasthattheycouldusemorepracticein.Then,astheycontinuepracticing,theprogramadaptstoprovideindividualizedinstructiontailoredforeachstudent’sskilllevel.Freckleisdesignedtobeusedmultipletimesaweekintheclassroomtobuildstudents’mathfluency,andprovidesteachersdataforeachstudentsuchasthereportshownbelow:
Figure1.SampleReportCarddataviewavailableforteachersinFreckleMath.Thisstudyaddressedthequestion:DostudentswithteacherswhousedFreckleexhibithigherendofyearmathematicsscoresthanstudentsfromteacherswhodidnotuseFreckle?Towardsthisend,WestEd,anonprofitresearch,development,andserviceorganization,analyzeddatathatwasprovidedbytheparticipatingschool,whichincludedendofyearmathematicsscoresontheMeasureofAcademicProgress(MAP)assessmentforallKindergarten,Grade1,andGrade2studentsattheschool.TheMAPassessmentisadministeredbytheNorthwestEvaluationAssociation(NWEA)andisanadaptivemeasureofstudents’mathematicsachievement.ItwaspredictedthatifFrecklesupportsstudents’mathematicsachievement,studentsofteacherswhousedFrecklewouldexhibithigherendofyearscoresontheMAPassessmentrelativetostudentsofteacherswhodidnotuseFreckle.
Freckle and Student Mathematics Achievement
4
METHODThisstudyisnotarandomizedcontroltrialstudy(i.e.,studentswerenotrandomlyassignedtoconditions).Becauseofmanythreatstointernalvalidityunderthisdesign,thisstudydoesnotattempttomakecausalclaimsontheimpactofFreckleinstudentscores.Rather,itexploreswhethertheresultsareconsistentwiththehypothesizedpredictionsrelatedtotheimpactofFreckleonstudentachievement.
PARTICIPANTSParticipantsinthisstudywerefromarural,publicelementaryschoolintheSouthernUnitedStates(seeTable1).Participantsconsistedof25teachers(14teachersintheFrecklegroupand11teachersintheComparisongroup)and466students1(263studentsintheFrecklegroupand203studentsintheComparisongroup).The25teacherscomprisedalloftheteachersofKindergarten,Grade1,andGrade2attheparticipatingschool.SomeoftheteachersusedFrecklewiththeirstudentswhereassomeoftheteachersdidnotuseFreckle(henceforth,thesegroupsarereferredtoastheFreckleandComparisongroups,respectively).Theschooldidnotprovideinformationonteachers’reasonsforoptingtouseFreckle.StudentsinbothgroupstooktheMAPMathassessmentatthebeginningandendofthe2014-2015academicyear(i.e.,theFallandSpring,respectively).Attrition.Thirty-fivestudents(16studentsfromtheFreckleconditionand19studentsfromtheComparisoncondition)didnothavecompletepreandpost-testdataandthereforeareexcludedfromallanalyses.Thecompleteanalyticsamplethenconsistedof431students(247intheFreckleconditionand184studentsintheComparisoncondition)(seeTable2).Table1.SchoolDemographicInformation
Type %Free/ReducedLunch Enrollment %White %Black %Hispanic %Asian/PI
Public,TitleI 69% 572 72% 21% 2.6% 0.3%
1Onestudentidappearedintwoclassroomsinthedatasetsprovidedandisnotconsideredintheseanalyses.
Freckle and Student Mathematics Achievement
5
Table2.TeacherandStudentCountsbyGradeIncludedinFinalAnalyticSample Kindergarten FirstGrade SecondGrade Total
Students Teachers Students Teachers Students Teachers Students Teachers
Freckle 66 4 86 5 95 5 247 14
Comparison 63 4 82 5 39 2 184 11
Total 129 8 168 10 134 7 431 25
TheaveragebeginningandendofyearMAPscoresbygradeandconditionareprovidedinFigure2.
Figure2.Meansforstudents’beginningandendofyearMAPscoresbygradeandcondition.
Freckle and Student Mathematics Achievement
6
Table3showstheobservedandexpectedgrowthforstudentsforeachparticipatinggradelevelandcondition.MAPgrowthnormsarebasedonthebeginningtoendofyeargrowthontheMAPmathematicsassessment(NorthwestEvaluationAssociation,2015).Table3.Growth(andstandarddeviations)observedinthestudyrelativetothegrowthnormsfromthebeginningtoendoftheacademicschoolyearontheMAPmathematicsassessmentforeachgradelevel.
Kindergarten Grade1 Grade2
FreckleGrowth 25.71(7.51)
25.52(7.73)
20.68(8.19)
ComparisonGrowth 23.08(7.17)
22.56(7.16)
12.79(5.11)
MAPGrowthNorms 19.1(7.59)
18.4(7.45)
15.2(7.11)
DATAANALYSISStudentachievementdatafromtheMAPassessmentwereanalyzedbyWestEd.TheprimaryanalyticstrategyforassessingstudentachievementscoresinvolvedfittingHierarchicalLinearModels(HLM).Thisanalysiswaschosenbecauseitaccountsforthenestedstructureofthedata(i.e.,studentsnestedwithinteachers;Raudenbush&Bryk,2002)(seeAppendixA).FrecklealsoconductedinterviewswithasubsetofFreckleteacherstogaininsightintotheperceivedvalueofFreckle(asummaryoftheteacherinterviewsisincludedinAppendixB).
RESULTSBASELINEEQUIVALENCEAnanalysiswasfirstconductedtodeterminewhetherstudentswerestatisticallyequivalentontheirbeginningofyearMAPscores.ThisanalysisindicatedthatstudentsintheFreckleconditionscoredhigher(MFR=161.44,SEFR=4.22),butnotstatisticallyhigher,onthepre-MAPassessmentthanstudentsintheComparisoncondition(MComp=156.05,SEComp=4.77;p=.41).Tomeasuretherelativesizeofthebaselinedifferencesbetweenthegroups,aneffectsizestatistic(i.e.,hedgesg)wascalculated.Theeffectsizestatisticisameasureofthesizeofthedifferencesbetweenthegroupsinstandarddeviations.TheeffectsizedifferencebetweentheFreckleandComparisongroups’baselinescoreswas.31,whichisconsideredsubstantive(WhatWorksClearinghouse,2014).Thedifferenceinbeginning-of-yearperformanceislargeenoughthatthetwogroupsshouldbeconsiderednon-equivalent(WhatWorksClearinghouse,2014).
Freckle and Student Mathematics Achievement
7
POST-SCOREANALYSISToexaminestudentachievementforstudentswhoseteachersusedFrecklerelativetostudentswhodidnot,weusedanHLMmodelthatattemptedtocontrolforstudents’gradelevelandtheirbeginningofyearscores(seeAppendixA).ThisanalysisindicatedthatstudentsintheFrecklegroupexhibitanendofyearMAPscorethatisanestimated4.64pointshigherthanstudentsintheComparisongroup.Thisdifferenceisconsideredsignificant.Theeffectsizeforthisdifferenceis.29(seeFigure2formeansofeachgroups’endofyearMAPscores)2.
Figure3.Covariate-adjustedmeansforeachGroup’sendofyearMAPscores.Errorbarsrepresentstandarderrorsofthemeans.
Tointerprettheeffectsizeestimateof.29inamoremeaningfulway,weconvertedthisstatisticusingpropertiesofthenormaldistribution.ThisanalysissuggestedthatifstudentsintheComparisongroupwereatthe50thpercentileofanormedsample,theFrecklegroupstudentswouldbeplacedatthe59th--60thpercentileofanormedsample(i.e.,approximately9.86percentagepointshigherthanthecomparisonstudents).WealsocomputedaneffectsizeforeachgradelevelandcomparedthiseffectsizetothetypicalexpectedMAPgrowthinmathematics(NorthwestEvaluationAssociation,2015).ThisanalysissuggestedthattheFrecklegroupstudentsareapproximatelytwomonthsaheadofComparisongroupstudentsinmathematicsachievement(seeAppendixA).
2ResultsweresimilarwhenexcludingtheGrade2students.
Freckle and Student Mathematics Achievement
8
DISCUSSIONStudentsintheFrecklegroupexhibitedhigherendofyearachievementscoresthanstudentsintheComparisongroup.Thiseffectheldwhencontrollingforbeginningofyearscoresandgradelevel.Thus,findingsfromthisstudyareatleastconsistentwiththehypothesisthatusingFreckleintheclassroomimprovesstudentachievementoutcomes.STUDYLIMITATIONSBecausethisstudyreliedondatacollectedpost-hocfromanaturalisticimplementation,therearemanyissuesthatmaylimittheconfidenceinconclusions.First,thedatasetprovidedbytheschoolwaslimitedinthatitcontainedlittlebackgroundinformationaboutparticipatingstudentsandteachers,includingreasonswhyteacherschosetouseFreckleattheschool.Thisinformationwouldhavebeenusefulforinferringthegeneralizabilityoffindings,aswellasforexploringfactorsthatmaymoderatetheeffectivenessofFreckle.Second,becausethisstudydidnotutilizerandomassignment,itispossiblethatvariablesotherthanFreckleuse-suchasmoreeffectiveteachersoptingtouseFreckle-contributedtheincreaseinstudentscores.Third,studentsintheFrecklegrouphadbaselineMAPscoresthatweresubstantivelyhigherthantheComparisongroupandthereforecannotbeconsideredequivalentpriortotheintervention.Finally,sincethestudywasconductedonarelativelyhomogeneousandnarrowpopulation(participantswithinasingleschool),theeffectsmaybelargerthanwouldbeobservedifthestudywereconductedonadiversepopulation(e.g.,multipleschoolsindifferentgeographicregions)(Lipseyetal.2012).CONCLUSIONSThisstudyisexploratoryinnatureandrepresentsaninitialstepinunderstandingtheeffectivenessofFreckleonstudentachievementoutcomes.ThesefindingssupportthehypothesisthatFreckleimprovesstudents’mathematicsoutcomes.However,futurestudiescanutilizerandomassignmenttogroupsandcollectbaselineandimplementationinformationtomoreconclusivelydetermineFreckle’simpactonstudentoutcomes.
Freckle and Student Mathematics Achievement
9
REFERENCES Bates,D.,Maechler,M.,Bolker,B.,&Walker,S.(2015).FittingLinearMixed-EffectsModelsUsing
lme4.JournalofStatisticalSoftware,67(1),1-48. Lipsey,M.W.,Puzio,K.,Yun,C.,Hebert,M.A.,Steinka-Fry,K.,Cole,M.W.,Roberts,M.,Anthony,K.S.,
Busick,M.D.(2012).TranslatingtheStatisticalRepresentationoftheEffectsofEducationInterventionsintoMoreReadilyInterpretableForms.(NCSER2013-3000).Washington,DC:NationalCenterforSpecialEducationResearch,InstituteofEducationSciences,U.S.DepartmentofEducation.ThisreportisavailableontheIESwebsiteathttp://ies.ed.gov/ncser/
NorthwestEvaluationAssociation(2015).2015NWEAMeasuresofAcademicProgressNormative
Data.https://www.nwea.org/content/uploads/2015/08/2015-MAP-Normative-Data-NOV15.pdf
RCoreTeam(2015).R:Alanguageandenvironmentforstatisticalcomputing.RFoundationfor
StatisticalComputing,Vienna,Austria.ISBN3-900051-07-0,URLhttp://www.R-project.org/Raudenbush,S.W.,&Bryk,A.S.(2002).Hierarchicallinearmodels:Applicationsanddataanalysis
methods(2nded.).ThousandOaks,CA:SagePublications.U.S.DepartmentofEducation,InstituteofEducationSciences,WhatWorksClearinghouse.(2014,
March).WhatWorksClearinghouse:ProceduresandStandardsHandbook(Version3.0).Retrievedfromhttp://whatworks.ed.gov
Freckle and Student Mathematics Achievement
10
AppendixA.Statisticalandmethodologicaldetailsofthemethodsandanalysesconducted.
DATAANALYSISStudentachievementdatafromtheMAPassessmentwereanalyzedbyWestEd.TheprimaryanalyticstrategyforassessingstudentachievementscoresinvolvedfittingHierarchicalLinearModels(HLM)witharandomeffecttermforteachers(Raudenbush&Bryk2002).Thisanalysiswaschosenasitaccountsforthenestedstructureofthedata(i.e.,studentsnestedwithinteachers).ModelswerefitusingRestrictedMaximumLikelihoodEstimation(REML).Fixedeffectsinthemodelincludetreatmentgroup,students’baselineachievement,andgradelevel.WeusedRstatisticalsoftware(RCoreTeam,2015)version3.1.2andthelme4package(Batesetal.2015)version1.1-7toconducttheHLManalysesonstudentachievement.
QUANTITATIVEANALYSESWestEdanalyzedstudentMAPscoresusingHierarchicalLinearModeling(Raudenbush&Bryk2002).PreliminarymodeltestingindicatedthatHLMisasuitablechoice,astheteacherintraclasscorrelationwassizeable(ICC=.69)intheunconditionalmodel.BaselineEquivalence.AnanalysiswasfirstconductedtodeterminewhetherstudentswerestatisticallyequivalentatbaselineonthepreMAPscore.ToexaminewhetherstudentsexhibitedsimilarpreMAPscores,studentpreMAPScoreswereregressedontothegroupingvariable(FreckleorComparison)whileincludingarandomeffecttermforteachers.ThisanalysisindicatedthatstudentsintheFreckleconditionscoredhigher(MFR=161.44,SEFR=4.22),butnotstatisticallysignificantlyhigher,onthepre-MAPassessmentthanstudentsintheComparisoncondition(MComp=156.05,SEComp=4.77;p=.41).Theeffectsizeforthisdifferencewashedgesg=.31,whichisconsideredsubstantive.TheWhatWorksClearinghouse(2013)standardsconsidersthislevelofbaselinedifferencetobenon-equivalent.Post-ScoreAnalysis.ToexaminestudentachievementforstudentswhoseteachersusedFrecklerelativetostudentswhodidnot,weusedanHLMmodelwhichincludedfixedeffectstermsforgradeandbeginningofyearscores,aswellasarandomaeffecttermtoaccountforthenestingofstudentswithinteachers.TheHLMmodelthatwasusedispresentedbelow:
𝑃𝑜𝑠𝑡𝑆𝑐𝑜𝑟𝑒)* = 𝛽.. + 𝛽.0𝐹𝑟𝑒𝑐𝑘𝑙𝑒 + 𝛽0.𝑃𝑟𝑒𝑆𝑐𝑜𝑟𝑒)* + 𝛽.4𝐹𝑖𝑟𝑠𝑡𝐺𝑟𝑎𝑑𝑒* + 𝛽.9𝑆𝑒𝑐𝑜𝑛𝑑𝐺𝑟𝑎𝑑𝑒* +𝜉.*
+ 𝜖)*where𝑃𝑜𝑠𝑡𝑆𝑐𝑜𝑟𝑒)*istheendofyearmathematicsscoreontheMAPassessmentforthei-thstudentofthej-thteacher,𝛽..isthegrandmeanofscores,𝛽0.𝑃𝑟𝑒𝑆𝑐𝑜𝑟𝑒)*representsstudents’beginningofyearMAPscore,and𝛽.4𝐹𝑖𝑟𝑠𝑡𝐺𝑟𝑎𝑑𝑒*and𝛽.9𝑆𝑒𝑐𝑜𝑛𝑑𝐺𝑟𝑎𝑑𝑒*representdummyvariablesfor1stand2ndgradeclassrooms,respectively(withKindergartenservingasthebaseline).𝜉.* isarandom
Freckle and Student Mathematics Achievement
11
effecttermforteachersandand𝜖)*isarandomerrorterm.ThedifferencebetweenscoresofstudentsofteachersusingFrecklevs.studentsofteachersintheComparisongroupiscapturedby𝛽.0𝐹𝑟𝑒𝑐𝑘𝑙𝑒.AsummaryoftherandomandfixedeffectsofthismodelarepresentedinTables4and5,respectively.
Table4.RandomEffectsTable.
Variance St.Dev.
Teacher 5.62 2.37
Residual 46.08 6.79Table5.FixedEffectsTable.
Coefficient St.Error p-value
Intercept 54.62 4.89 <.001***
PreScore 0.77 0.03 <.001***
FirstGrade 5.05 1.57 .003**
SecondGrade 0.66 1.86 0.72
Freckle 4.64 1.19 <.001***Note.**p<.01,***p<.001IfthehypothesisaboutFreckleimprovingstudents’mathematicsscoresiscorrect,theoutputinTable4suggeststhatthestudentsintheFrecklegroupwouldhavegrownanestimated4.64pointshigherthanstudentsintheComparisongrouphadthegroupsbeenfullyequivalentpriortotreatment.Thisdifferenceissignificantatthe⍺<.001significancelevel.Theeffectsizeforthisdifferenceishedge’sg=.29.Thecovariate-adjustedmeansfortheFreckleandComparisongroupsare184.52and179.88,respectively,andtheunadjustedstandarddeviationsforthesegroupsare16.12and16.45,respectively.BecauseGrade2hadsubstantiallydifferentsamplesizesintheFrecklegrouprelativetotheControlgroup,wererantheabovemodelexcludingGrade2students.ThemodelcoefficientforFreckletermusingthissubsetteddatawasstillpositive,andsignificant(B=3.93,p=.007).
Freckle and Student Mathematics Achievement
12
Tointerprettheeffectsizeestimateof.29inamoremeaningfulway,weconvertedthisstatisticusingpropertiesofthenormaldistribution.Inthenormaldistribution,a1standarddeviationincreasecorrespondstoa34pointpercentileincrease.Thus,inanormedsample,aneffectsizeof.29correspondstoa9.86pointpercentileincrease(i.e.,34x.29=9.86).Therefore,theresultssuggestthatifstudentsintheComparisongroupwereatthe50thpercentileofthenormedsample,theFrecklegroupstudentswouldbeplacedatthe59th-60thpercentileofthenormedsample(i.e.,50+9.86=59.86).WealsocomputedaneffectsizeforeachgradelevelandcomparedthiseffectsizetothetypicalexpectedMAPgrowthinmathematics.Tothisend,wereranthemodelpresentedaboveseparatelyforeachgradelevel(i.e.,weexcludedthegrade-leveltermsfromeachmodel).TheeffectsizesfortheFrecklevariablefromthesemodelsare.29,.38,and.34forKindergarten,Grade1,andGrade2,respectively.BasedonthebeginningandendofyearstudentstatusnormsontheMAPmathematicsassessment(NorthwestEvaluationAssociation,2015),theestimatedbeginningtoendofyeareffectsizesforMAPmathematicsare1.33,1.39,1.14foreachofthesegrades.ComparingtheeffectsizesobservedinthepresentstudytotheeffectsizesoftheaveragegrowthfortheMAPassessmentsuggeststhattheobservedeffectsizesarenoteworthy,correspondingto22%,27%,and30%oftheexpectedgainoverthecourseofaschool-yearinKindergarten,Grade1,andGrade2,respectively(e.g.,forKindergarten,.29/1.33=.22,etc.).Multiplyingtheaverageschool-yeargainforallgrade-levels(.26)by9months(thetypicalnumberofmonthsinaschoolyear),weestimatethattheFrecklegroupstudentsareapproximatelytwomonthsaheadofComparisongroupstudentsinmathematicsachievement.
Freckle and Student Mathematics Achievement
13
AppendixB.DescriptionofteacherinterviewsconductedbyFreckle.
TEACHERINTERVIEWSTeachers’perceivedvalueoftheprogramwasdefinedastheextenttowhichFrecklecouldbeusedeffectivelyandefficientlybyclassroomteachers.TogainanunderstandingoftheperceivedvalueofFreckle,FreckleconductedinterviewswiththreeteacherswhousedFreckleabouttheirexperiencewiththeprogram.Theinterviewswereconductedoverthephoneinthefallof2015,andconsistedofquestionsabouthowFreckleusageimpactedstudentengagementandoutcomesinmath.TheteachersinterviewedhadusedFreckleinthe2014-2015schoolyearandwerewillingtodiscusstheirexperienceswiththeprogram.QuestionsincludedhowtheyfeltFreckleimpactedgrowthintheirclassroom,students’engagementinmath,andtheireffectivenessasteachers.Interviewslastedapproximately30minuteseach.Ingeneral,Frecklewasusedinmathcenterswithsmallgroupsofstudentsforabout30minutesaday,afewtimesaweek.AlltheteachersinterviewedsaidtheywouldrecommendFreckle,citingitsindividualizedpracticeasthemainbenefitoftheprogram.TheinterviewedteachersreportedthattheysawtremendousgrowthfromusingFreckle.Forinstance,oneteachersaid,“atthebeginningoftheyearlastyear,Ihadkidswhoweresignificantlybelowgradelevel.Bytheendoftheyear,outofthekidsIhad,only1or2werenotongradelevelorabove”.TeachersalsoappreciatedthewayFreckledeepenedstudents’understandingofmathconcepts.Oneteacherremarkedthat“itbuilttheirconfidence,becausetheproblemswerehuge,andoncetheycouldbreakitdownintoitsparts,theygotit.Theyusedstrategiestosolvetheproblem.Ibelieveitexposedthemtothinkingoutsidethebox”.AnotherteacherreportedthatstudentsenjoyeditsomuchthattheywantedtouseFreckleathome.TeachersreportedthatFreckleallowedthemtobettermeettheneedsofeverychildintheirclassroom,givingthemtheabilitytofocusonsmallgroupsofstudentsatatime.OneteacherstatedthatFreckle“allowsmetoseewhatIdon’tneedtowastetimeon.I’mabletobettermanagestudents’instructionandmanagemytimebetter”.Overall,theinterviewedteachersviewedFreckleasaneffectivetool,withonesaying“onceIfoundoutaboutit,Ibasicallyuseditallthetime”.InterviewresponsessuggestthatFrecklecanbeavaluableclassroomtool.ThoughonlyasmallsubsetofteachersthatusedFreckleparticipatedininterviews,theseteacherssuggestedthatFrecklewasfluidlyintegratedintotheirclassroominstruction,andthattheywereabletouseclasstimemoreeffectivelytosupportstudentmathematicslearning.