Impact of two teacher training programmes on pupils’ development of literacy and numeracy ability: a randomised trial

Jack WorthNational Foundation for Educational Research

9th September 2015

RCTs in the Social Sciences, York

Overview

• Introduction• Methods• Results• Discussion

Introduction

• Randomised trial commissioned by the Education Endowment Foundation

• Motivations:• General identify interventions that raise the

attainment of pupils, particularly in literacy and numeracy, and for disadvantaged pupils

• Specific evaluate the impact of University of Oxford’s Improving Numeracy and Literacy programme on numeracy and literacy

Two interventions

Mathematics and Reasoningdevelop children’s understanding of the logical principles underlying mathematics

Literacy and Morphemesimprove spelling and reading comprehension by teaching children about sentence structure and morphemes

• Aimed at teachers of Year 2 pupils (age 7)• One day of teacher training and a researcher visit• Lesson materials for 10 whole-class lessons• Accompanying computer games for in-class/home use by pupils• Previous researcher-delivered interventions had effect sizes 0.3 - 1.2

Trial design

• School-randomised controlled trial• Three arms to evaluate two interventions• Eligibility

• State-funded mainstream infant and primary schools• (Teachers of) Year 2 pupils eligible to participate• Opt-out consent offered to parents

Outcomes

Primary outcome

Raw score in Progress in Maths / Progress in English tests

Test administered by NFER test administrator, blind to group allocation and instructed not to discuss group allocation or the intervention with teachers

Secondary outcomes• Free school meal sub-group, and interaction analysis• Prior attainment interaction analysis• English as an additional language interaction analysis• Sub-domain scores in relevant areas• Key Stage 1 teacher assessment levels (maths and reading/writing)• Transfer effects (numeracy intervention on literacy, and v.v.)

Sample size

• Design• Aimed to recruit 60 schools, 20 in each group• Average 45 pupils per school• Intra-cluster correlation = 0.15• Correlation between pre- and post-test = 0.8• Two-tailed significance = 5%, power = 80%

Minimum detectable effect size = 0.22

Sample size

0

0.2

0.4

0.6

0.8

1

0 0.1 0.2 0.3

Po

we

r

Effect size

Design

Actual

▼ Number of schools = 55 (60)

▼ Pupils per school = 35 (45)

▼ Intra-cluster correlation = 0.09 (0.15)

▲ Correlation between pre- and post-test = 0.81 (0.80)

▼ Actual MDES = 0.18

Randomisation

• Simple randomisation by NFER• First block of 51 schools randomised (17 each)• Block of three late-recruited schools randomised • Block of two late-recruited schools randomised to two

randomly selected groups

• School told of group allocation after its pre-test researcher visit• One randomised school dropped out before testing,

but randomisation outcomes retained

Participants

S = 55

171919 Allocation Allocation

171919 Follow-up Follow-up

171919 Analysis Analysis

972 652 593

890 594 537

517 (87%)577 (88%)850 (87%)

P = 2217

Control Literacy Numeracy

Participants

School-level averages Control group

Literacy group

Numeracy group

Pre-test literacy score 23.1 23.8 23.6

(8.9) (8.9) (9.2)

Pre-test numeracy score 20.2 21.1 21.1

(4.7) (4.8) (4.8)

School-level averages Control group

Literacy group

Numeracy group

Female (%) 47.9 49.7 46.3

Free school meals (%) 21.0 14.0 10.1

Special educational needs (%) 16.4 15.6 12.2

English as an additional language (%) 16.2 21.0 14.1

Participants

Multilevel model of pre-test imbalance Effect size 95% CI

Pre-test literacy score 0.08 -0.12 – 0.27

Pre-test numeracy score 0.20 -0.03 – 0.43

School-level averages Control group

Literacy group

Numeracy group

Pre-test literacy score 23.1 23.8 23.6

(8.9) (8.9) (9.2)

Pre-test numeracy score 20.2 21.1 21.1

(4.7) (4.8) (4.8)

Analysis

• Two-level multilevel model (school, pupil)• Post-test raw score as dependent variable• Pre-test raw score as covariate• Literacy/ Numeracy group indicators (forced into final models)• Additional covariates (backwards selection)

• Age in months at post-test• Gender• Randomisation block

• Hedges’ g effect size• calculated by: group indicator coefficient

pupil-level standard deviation from MLM

Results

Effect size 95% confidence interval

Numeracy intervention 0.20 0.02 – 0.37

Literacy intervention -0.05 -0.18 – 0.08

Numeracy intervention (FSM) 0.14 -0.08 – 0.37

Literacy intervention (FSM) 0.10 -0.13 – 0.34

• No significant impact on Key Stage 1 teacher assessments• No significant transfer effects• No significant differential impact by prior ability, or for EAL pupils• No significant differential impact on sub-domain scores

Further results

22.0

22.5

23.0

23.5

24.0

24.5

25.0

Non-FSM pupils FSM pupils

Lit

era

cy

po

st-

tes

t sc

ore

Control group

Literacy group

FSM interaction analysis

Effect size 95% confidence interval

Literacy intervention (FSM sub-group) 0.10 -0.13 – 0.34

Literacy intervention (FSM interaction) 0.26 0.08 – 0.43

Further results

Effect size 95% confidence interval

Total number of computer games played (numeracy)

0.19per 20 games

0.10 – 0.27

Total number of computer games played (literacy)

0.05per 15 games

-0.01 – 0.11

Note: 15 and 20 games played were the interquartile ranges for literacy and numeracy respectively. The effect size shows the effect of moving from the 25th percentile to the 75th percentile in terms of the number of games played.

Computer games “on-treatment” analysis

Discussion

• Numeracy intervention has shown promise at raising attainment. Literacy intervention has not.

• Both interventions are very low cost• £10 per pupil per year over 3 years• Plus a day of teacher training

• Will it work at a larger scale?• This was an efficacy trial to demonstrate impact in ideal conditions• Now an effectiveness trial with a scalable delivery model

NFER provides evidence for excellence

through its independence and insights, the

breadth of its work, its connections, and a

focus on outcomes.

National Foundation for Educational Research

The Mere, Upton Park

Slough, Berks SL1 2DQ

T: 01753 574123

F: 01753 691632

E: enquiries@nfer.ac.uk

www.nfer.ac.uk

Worth, J., Sizmur, J., Ager, R. and Styles, B. (2015). Improving Numeracy and Literacy: Evaluation Report and Executive Summary. London: Education Endowment Foundation

https://educationendowmentfoundation.org.uk/projects/improving-numeracy-and-literacy/

Any questions?