evaluang a province-wide early grade mathemacs intervenon...

Evalua'ngaprovince-wideearlygrademathema'csinterven'on:

ThecaseofR-Maths

7September2018NickyRobertsandEleanorHazell

Presenta'onoverview

1.  R-Mathsprojectdesign2.  R-MathsevaluaEondesign3.  Testdesign4.  FindingsfromtesEng:

•  Subjectadvisors•  Teachers•  Learners

5.  WhatarereasonableexpectaEonsforshiPsintestaQainment?

6.  Andso?2

Spoileralert1:Stuffweknewin2007

Spoileralert2!Morestuffweknewin2007:

1.   Schoolchangetakes'me.Ittakesaboutthreeyearstoseechangesinaprimaryschool,andfiveyearstoseechangesinasecondaryschool,dependingonsizeandcomplexity.

2.   Thecoreofeduca'onistheteacherintheirclassroom.Thisisthehardestparttochange.Tomakerealchangesthiscoremustbereached.

3.  Thereisnoprovenwayofchangingthedynamicsofindividualschoolsotherthanworkingcloselywiththem.

4

Paraphrased from Christie, Butler and Pottering (2007) Schools That Work, Ministerial Report to the Minister of Education, South Africa

1.R-Mathsprojectdesign

5

R-Maths

•  LedbytheWesternCapeEduca'onDepartment

•  IncollaboraEonwiththeUCTSchoolsDevUnit(SDU)

•  Programmetraining&materialsdevelopedbytheSDUincollaboraEonwiththeWCED

•  Fundedandsupportedbydonors

6

Goal: Improve the conceptual understanding

and Mathematical skills of Grade R learners in the Western Cape

R-Maths‘Form’:Modifiedcascade

TheR-Mathstraining–includingcoursematerialsandclusternotes–isbackboneofR-MathsSA’sreceive30hrsblocktraining,UCT/SACEassessment/PoE/accreditaEon,+/-30hourssupport(‘dryruns’)Teachersreceive14hrsclusterw’shops,30hrstraining,SACEaccreditaEonprovidedbytheSA’s,reflecEonworkshop,PLConR-Maths.FidelityofimplementaEonissupportedthroughfullsetofteacherresources

7

Process theory

Impact theory

Development of teacher training &

resources

Capacitation of district officials

Training of Grade R teacher/

practitioners

Resource use for teacher training & classroom teaching

SAs: Enhanced knowledge & skills to support Grade R

teacher/ practitioners

Teacher/practitioners: Enhanced knowledge & skills to

teach Grade R maths

Improved learner performance in mathematics

(readiness for Grade 1)

Effective teaching

Short-term

M

edium-

term

Long-term

(impact)

Process theory Impact theory Outcomes

Support to Grade R teacher/

practitioners

2016 2017 2018

Training and Materials Development:

70 SA training (30 hrs) UCT/

SACE accredited

J F M A M J J A S O N D

1500 Teacher training (30 hrs) SACE

accredited

1500 Teacher training (30 hrs) SACE

accredited

Cluster w

’shop (2 hrs)

2x Cluster

w’shop (2 hrs)

Cluster w

’shop (2 hrs)

Cluster w

’shop (2 hrs)

Cluster w

’shop (2 hrs)

Cluster w

’shop (2 hrs)

Cluster w

’shop (2 hrs)

Cluster w

’shop (2 hrs)

PLC

Reflection

Reflection

PLC

Dry run

Dry run

R-Maths‘Dosage’

Cluster w

’shop (2 hrs)

Cluster w

’shop (2 hrs)

Cluster w

’shop (2 hrs)

Cluster w

’shop (2 hrs)

Cluster w

’shop (2 hrs)

R-Mathstrainingandsupporttosubjectadvisors “dry-runs”

priortoclusterworkshops(2hoursx7)&blocktraining

Blocktraining:5days/30hoursx1

R-Mathsteachermaterials

11

R-Maths:‘substance’

•  ExplicittheoryofGradeRlearning:PlayinmathemaEcs•  7principlesofR-Mathslearning•  ExplicitintegraEonofmathsindailyprogramme•  AddedtoandexpandedCAPSperterm

•  Integratednumberintoalltheothertopics•  Focusedon1topicperweek•  Specificguidanceontopicfortheweekandsmallgroupwork,

repeatedover1week•  Explicitdevelopmentalframeworkfornumberwork•  AfewpowerfulrepresentaEons,consistentlyapplied(egdot

cards,numeralcards,structuredbeadstring

CAPSreferencetoplayinmathema'cs

TheapproachtolearningMathemaEcsshouldbebasedontheprinciplesofintegra'onandplay-basedlearning.Theteachershouldbepro-ac've,amediatorratherthanafacilitator.MakemostoflearningopportuniEesthatarisespontaneouslyduringarangeofchild-centredac'vi'es:-  freeplayinthefantasycornerorblockconstrucEonsite,sandandwaterplay

acEviEes-  teacher-guidedacEviEesthatfocusonmathemaEcalconceptssuchascounEng,

numberconceptdevelopment,spaceandshape,paQerns,EmeandotheremergentmathemaEcsacEviEes

Duringfreeplaytheteachercanpromoteemergentmathema'csthroughtheappropriatestructuringofthefree-playarea(CAPSGradeRMathemaEcs,page14;Grade1–3,page13)

7PrinciplesofR-Maths

1.  Context2.  AcEvity3.  Level4.  InteracEon5.  Guidance6.  Inclusivity7.  PracEce

14

CAPSMathema'cs:Fivetopics

18

Data handling

Measurement

Shape and space (geometry)

Patterns, functions and algebra

Number operations and relationships

•  TrajectoriesforLearning(Sarama&Clements,2009;TreffersvanDenHeuvel-Panhuizened,2012;vanDenHeuvel-Panhuizen,Kuhne,Lombard,2014)

•  SubiEzing(Kaufmanetal.in1949;StarkeyandCooper,1995;Clements,1999)•  CounEngprinciples(Gelman&Gallistel,1978)•  MulEplerepresentaEons(Leinhardtetal.,1991;Hiebert&Carpenter,1992;

Greeno&Hall,1997)•  UniEzing(Dewey,vonGlasersfeld&Richards,1983;vonGlasersfeld,1981,1995)

TheoriesunderpinningR-Maths

All young children have the same implicit understanding of maths but to make that understanding explicit, they need to hear the language of maths and get the chance to think about it and use it when they speak (Clements, 2016).

•  Numberwordlist(ordinality)-acousEc/oral,stories,songs,rhymes,‘washingline’•  Cardinality(howmanyinaset)-resultaEvecounEng,perceptual-conceptualsubiEzing,comparethencounttofind‘howmanymore/fewer’•  One-to-onecorrespondence-extendaccuracytolargersets•  Numbersymbols-represenEngnumbers:concrete,iconic,symbolic

2.R-Mathsevalua'ondesign

1.  Focusedon‘productandprocess’aswellas‘outcomesandimpact’

2.  Purposivelyselected2districts:1urbanand1district,andcasestudyschoolswithinthosedistricts

3.  PhasedimplementaEonover2yearsallowedforacounter-factuallearnersinthesamedistrict(notpossibleforSAs)

4.  Mixedmethodstogettodifferentlevelsofthesystem

21

Evalua'onques'ons

22

Productandprocessevalua'onques'ons1.WhatdoestheR-MathsProjectentail?2.Whatisthecontext(e.g.rural/urban,socio-economicstatusoftheschoolcommunity,numberofteachersandlearners,backgroundofGradeRteacher/pracEEoners,etc.)withinwhichtheimplementaEonistakingplace?3.Istheprojectbeingimplementedasplanned:inputs,outputsandtargets?4.Whatisthenatureandqualityofthetraining,supportandresourcesprovidedtoFPSubjectAdvisorsandteacher/pracEEoners?5.WhatisthelevelofadopEonbyFPSubjectAdvisorsandteacher/pracEEoners?6.Whatarethechallenges/barrierstoimplementaEonbydistrictsandschools?7.Whatarethesuccessesandchallengesofthepartnershipmodelthroughwhichthisprojectisbeingimplemented?8.Whatarethekeymechanismsofchangeinthecasestudycontexts?

Outcomeandimpactevalua'onques'ons9.WhatistheimpactoftheprojectonFPSubjectAdvisors,GradeRteacher/pracEEonersandGradeRteacher/pracEEoners’teachingpracEce?10.DoestheR-MathsProjecthaveanimpactonGradeRlearners’MathemaEcalknowledgeandskills?11.WhatarethesuccessesofandbarrierstoscalabilityandembeddednessintotheWCED?12.Hastheprojectmetitsintendedoutcomesoutlinedinthetheoryofchangeandlogicmodel?

10.WhatistheimpactofR-MathsontheSubjectAdvisorsandGradeRteacher/prac''oners’andteachingpracEce?11.DoestheR-MathsprojectimpactonGradeRlearnersmathemaEcalknowledgeandskills?

Datacollec'onmethodsandsources

23

Ac'vi'es:

Level

Project/

province

District/

Subject

Advisor

HoD Teacher Learner

Keystakeholderinterviews x x

SubjectAdvisortest x x

Teachertest x x

Training & training dry-run

observa'on

x x

CT&dry-runobserva'on x

Monitoring fidelity, tracking

“dosage”

x x x

Casestudies x x x

Learnertest x x

3.Testdesign

Howdoweassesstheknowledgeof:•  SubjectAdvisors•  Teachers•  LearnersforsupporEng,teachingandlearningGradeRmathemaEcs?…andin3differentlanguages?

24

SubjectAdvisortest

Whatisourexpected‘knowledgeforteachingGradeRmathemaEcs”,forSubjectAdvisors?•  Nocommonstandards…•  Noagreed/commonassessmentsitems…•  Novalidatedinstruments…•  Comparabilityofanytestinstrumentacrosslanguages?So,(30min)testdevelopedbasedontrainingguidesandteacherconceptguide(drawingonexpertknowledge):-  MixofWCEDpolicyrequirements-  SubjectmaQerknowledge-  PedagogicalcontentknowledgeaboutGradeR(atthelevelofwhatwasbeingtaught) 25

ExampleSAtestitems

26

Teachertest

1.  ThesameissuesasforSAtest2.  Shorterandwithfeweritems.3.  QuesEonedre-phrasedasfromperspecEveofateacher

–  YouobserveaGradeRteacher’smathslesson….to–  InyourGradeRmathslesson…

27

Learnertest

28

AdaptaEonofGermantest48itemone-to-oneoralinterviewValidatedinGautengforEnglish,AfrikaansandisiZulu(+2)HLlearners

Learnertest

29

1.  Testconductedorallyinschools,onechildataEme,administeredbytrainedtestadministrators

2.  ChildrencompletedthetestintheirLoLT3.  BaselineinFeb/Mar2017;endlineinOct/Nov20174.  IniEalsimplerandomsampleoflearnersin2districts:-  168ineachoftheintervenEonandcomparisongroups,-  168incomparisongroup-  DuetoaQriEonetc.,therewasatotalof622matchedtests

4.Testfindings

Whatdidwefindabouttheknowledgeof:•  SubjectAdvisors•  Teachers•  LearnersforsupporEng,teachingandlearningGradeRmathemaEcs?

30

SAtestresultsPre-TestResults(n=47)•  Meanscoreof48.6%.•  Scoresrangedfrom20%to80%

31

Post-TestResults(n=47)•  Meanscoreof66.1%.•  Scoresrangedfrom46%to92%•  Increaseof17.5pppretopost

test=significantat95%confidencelevel

•  Largeeffect(1.44sd).

Teacher/prac''onertestresultsPreTestresults(n=157)•  Meanscoreof51.4%.•  Scoresrangedfrom16%to88%.

32

PostTestresults(n=157)•  Meanscoreof68.1%.•  Scoresrangedfrom28%to96%.•  Increaseof16.7pppre-posttest•  Significant,largeeffect(1.37sd).

Learnerfindings:UrbanDistrict1

33

•  Mostlearners(≈70-80%)wereassessedinEnglish.•  AsmallminorityofthesewereELLs(i.e.wereassessedinEnglish

butwerenotEnglishHL)•  AlmostallremaininglearnerswerelearningMathemaEcsintheir

HLofisiXhosa.

Notsignificant

Learnerlevelfindings:RuralDistrict2

34

•  Mostlearners(≈75%intheintervenEongroup,and100%inthecomparisongroup)wereassessedinAfrikaans.

•  AllremaininglearnerswerelearningMathemaEcsintheirHLofisiXhosa.

Significant,withasmalleffectsize.

Learnerlevelfindings:Generallinearmodelofwholesample–setup

Fixedeffects:-  Baselinetestscores,-  District(urban,rural),and-  Group(intervenEon,comparison)

Dependentvariables:-  Marko-Dendlinetotalscores,and-  Marko-Dendlinescores,bylevel(L1,L2,L3,L4,L5)

Factorsincluded:-  Gender(Male;Female),-  QuinEle(1;2;3;4;5),-  LoLT(E,A,X),and-  Age

35

Generallinearmodelonwholeresult:findings1

1.  Greatest(medium)effectsonMarko-Dperformancewere:a.   LoLT(isiXhosa-andAfrikaans-speakersimprovedbest;

English-speakersleast)–foralllevelsandtotal,andb.   District(urbanlearnersimprovedmore,atL3toL5,and

total)2.  Weaker(small)effectonMarko-Dperformancewere:

a.   Group(intervenEongrouplearnersperformedbeQeratL2,L3andL5andonwholetest),and

b.   Age(olderlearnersperformedbeQeratL2,L3andL4)3.  NosignificanteffectsonMarko-Dperformancefor:

a.  Genderb.  QuinEle

36

Generallinearmodelonwholeresult:findings2ImpactofgroupInterven'ongrouplearnersperformedbeQeronwholetestandatL2,L3andL5

37

Generallinearmodelonwholeresult:conclusions•  BiggesteffectswereonLevels2and3oftheMarko-D.

•  Grade R children in the intervenEon groupwere performingsimilarlytothoseGradeRlearnersinthecomparisonschoolswhoweresixmonthsolder(whenassessedatLevels2and3ofMarko-D).

•  ExpecEnglargedifferencesinimprovementsinlearnerscoresinashortspaceofEmeisunrealisEc.

•  Overall,therefore,thefactthattheR-MathsintervenEonhada generally small but posi've effect on the MathemaEcsresults of childrenwhose teachers had been exposed to theintervenEonisencouraging.

38

5.Whenisaneffectonlearneroutcomes,goodenough?

IsastaEsEcallysignificantdifferencebetweenintervenEonandcomparisonenough?

Isitenoughwhenthedifferencesareonlyafewpp?IsitenoughwheneffectisaboutafiihofaSD?Is‘6monthsgain’,relaEveto‘ageadvantage’,good?Isitgoodover10monthsinGradeR?

WhataboutwashoutoverEme?

Meta-analysesofimpactofinterven'ons•  EducaEonintervenEonsinGr1-3haveaverageeffectsizeof

0.18SD(Hill,Bloom,BlackandLipsey,2008)

•  Sub-SaharanAfricanmetaanalysis(Conn,2017):•  OveralleffectsizeforalleducaEonintervenEontypes:0.18SD•  EducaEonintervenEonsfocusingonpedagogy:0.92SD

ComparisonswithfindingsfromotherSAstudies

1.ELOM(50-59and60-69month)Differencebetween>5y.oand<5y.onormsforemergentMathsisonly2.5pp

2.Readingcatch-upresearchproject(Grades1-3)ThestudyshowedanoverallimprovementfrombothcontrolandintervenEonschools,withintervenEonschoolsshowingaslightlybelerperformanceparEcularlyinspecificareasofreading(spellingandgrammar)

3.WCEDLitNumGrade1Numeracymean:4pp:27%(2009)to31%(2012)

4.E-Lit(2016lit/langinterven'oninGradeRacrossGrR)6monthsaPertheintervenEontrainingbegan(Mid-GradeR):0.41SDEndofGradeR:0.24SDEndofGrade1:Nosignificantdifference.

41

6.Sowhat?

WasR-Maths“successful”?WhatdoesthisR-MathsexamplehighlightaboutoureducaEonresearchlandscape?

42

43

Achieved

Partially Achieved

Not achieved

Issuesthathinder

1.  QualitaEveandquanEtaEverealmsareatEmesdifferentuniverses

(cold&clinicalnumbersvswarm&fizzypeople)2.   Enoughofpilots.Wecaneasilychangeafewschools.Butweneedto

getchange“atscale”.Butthen,wegobig.Immediately.Withouttakingintoaccountthestability/robustnessofourtreatment(whichissEllbeingconceptualisedand/ortheorised)

3.  Changeisurgent.Thingsaredire.Sowemustimplementnow.Thebudgetisforthisfinancialyear…onlyinthisadministraEveterm…

4.   Tomeasure:RCTis“goldstandard”(irrespecEveofthephaseofintervenEondesign…)

5.   TeachingCAPSandnotchildren.Tightlymonitoringcomplianceonform(“supporEng”)daily/weekly/termlycurriculumpaceand‘coverage’.

44

Whereourresearchisimproving

45

Bogstandard(now…):•  Enrolment(grossandneQ),drop-outs,throughput,NCS

aQainment•  Totalcohort(withGHSandDBEdata)•  Inequality(class:school,household,child;race,gender)

Increasingly:•  Theoriesofchange,•  IntervenEoninput:level,purposeand‘form’,(egcurriculum

coverage,dosage,fidelityof“uptake”)•  Learneroutcomes:curriculum-basedandinternaEonal

assessments,preandpost-intervenEonorchangeoverEmeSeldom:

•  Delayedposttests,or•  TrackingofmorethanoneacademicyearforSAs,teachersand

learners

Whatdoweares'llnotdoingwell–notmeasuringordescribing:

•  Joined-upfamilyservicesinECD•  Levelofthesystemandwhy•  Costs&costeffecEveness•  DifferenEaEon/remediaEonatschoollevel•  Sensiblesamplingforsystem-widefeedbackonlearning•  OurinstrumentaEonforstandardisedmeasurementforlearning

outcomes/knowledge•  Validityoftheinstrumentsweuse•  Standard/commonassessmentscrossstudies•  Comparabilityofourinstrumentsacrosslanguages

•  The‘substance’andnotjusttheform,ofourintervenEons46

Whatisthe‘substance’?

Ourtheoryoflearning:•  Howourchildrenofthisagelearnx?•  Ourchildren’sexpecteddevelopmentaltrajectoryforx?Ourpedagogictheory:•  OurenvisagedandactualroleandcapacityofourSubject

Advisorsforx(whatdotheydo,knowandfeelaboutx?)•  DiQoforHODs:Whatdotheydo,knowandfeelaboutx?•  DiQoforteachers:Whatdotheydoknowandfeelaboutx?OurREALISTICexpectedlearningoutcomesforchildrenandcaringadults:•  Knowledgeforx–eg“teachingmathstoGradeRs(inthis

schoolcontext)”

Thankyou

NickyRobertsUniversityofJohannesburgnroberts@uj.ac.zanicky@kelello.org@Kelello_EleanorHazellJETEducaEonServicesehazell@jet.org.za