Grace Grima

Associate Professor & Director of Research – Pearson grace.grima@pearson.com

Jennie Golding

Associate Professor in Mathematics Education, UCL j.golding@ucl.ac.uk

1

Assessment to support the development of

problem-solving goals in mathematics curricula 5-16

Pearson: The Efficacy Agenda

evaluating the impact on improving lives through

learning

Delivering what we promise At Pearson, efficacy means making sure that our products and services work the way we say they do. It is at the very heart of what we do. Our efficacy strategy helps us create and deliver learning products, platforms, and services across global markets—all with the aim of improving learner outcomes. The Efficacy Agenda focuses on measuring the impact on improving lives through learning. Our research focuses on the value of measuring impact on learners and the value-added to schools. 3 What is efficacy at Pearson?

Image by Jose Vidal Carrasco

Defining our outcomes

We define learner outcomes for our products in these categories:

ACCESS

The learner’s access and experience

COMPLETION ACHIEVEMENT PROGRESSION

The timeliness of completion

The standard of achievement

or level of competence

Where the learner goes next: study,

training, or employment

4 Making progress

Product or service has well-defined

outcomes and measures and a

supporting theoretical evidence

base appropriate to its purpose

Usage of the product or service is designed

and achieves appropriate levels of

satisfaction

Evidence that the

product or service is shifting or influencing

leading indicators and/or evidence of a relation to outcomes

Evidence isolates the impact of the product

or service on the target outcome(s) with

educational and statistical significance

Evidence that the

impact of the product or service on the

target outcome(s) is replicated in

multiple situations

JUSTIFY ENGAGE CORRELATE CAUSE EXPLAIN

How much impact?

The Pearson Levels of Evidence1 allow us to apply rigour

to evaluating our impact on improving learner outcomes. LEVEL

EVIDENCE

5 Making progress

1Based on nesta.org.uk/sites/ default/files/standards_of_evidence.pdf

Benefits Research on learning in schools enables us to: • Improve our services by bringing in real data from the end users • Measure the impact of services and qualifications on learners • Have learner and teacher voices at the centre of our work • Evaluate and understand implementation / classroom processes • Understand the complexities of learning in the school context

6

Assessment to

support mathematical

problem-solving

8

The policy context

• Recent curriculum changes in England - renewed focus on

mathematical reasoning and problem solving.

• New curricula for 5-16 year olds rolled out from 2014: first ‘GCSE’ assessment for all 16-year-olds in 2017 first large scale A-level assessment for new curriculum- 2019

• GCSE and A-level assessment are high stakes and via four

Awarding Organisations, including Pearson.

• A coherent curriculum offer (Schmidt and Prawat 2006) depends on achieved alignment across the system: enacted curriculum, resources, assessments and teacher CPD / capacity to deliver.

• Pearson offers resources, assessment and teacher CPD.

.

9

We asked how the resources and assessments produced by Pearson impacted student and teacher experiences for the range of student outcomes. In particular: • How is mathematical problem-solving being enacted in classrooms?

• How do the issues of teacher capacity, resources, and assessments affect the teaching of

problem solving?

Research questions

10

The studies

Study When? Size

A Primary: 5-11 2016-18 (y1-2; y 5-6)

9 schools and mathematics coordinators,

18 classes and teachers

B Secondary: 11-16 2016-18

(y7-8 or y 8-9; 10-11)

15 schools and Heads of Mathematics (HoM),

32 classes and teachers

C GCSE Mathematics / progression: 15-17 2016-Nov 18

15 schools and HoMs, 30 GCSE classes and

teachers, 32 post-16 student

groups and teachers

D A level Mathematics: 16-18 2017-2020

12 schools and HoMs, 48 A level classes and

teachers

11

Methodology for Study B: an example

Schools using Pearson resources recruited. 2 classes per school.

2016 HoM and class teacher telephone interviews Student baseline assessments

Spring 2017 Spring 2018

Lesson observations: focus on new aspects of curriculum

Teacher post-lesson interviews

Student focus groups

Summer 2018

Class teacher surveys Student surveys

External assessment outcome date

Autumn 2018 Final reflections with HoMs

Data analysis using Research question themes leading to a grounded approach (Charmaz, 2014)

to developing subthemes

12

Profile of problem-solving in assessment:

Assessment Objectives

GCSE Foundation

tier

GCSE Higher tier

A Level Mathematics

AO1 Use and apply

standard mathematical techniques*

50% 40% 50%

AO2 Reason, interpret and

communicate mathematically

25% 30% 25%

AO3 Solve problems within

mathematics and in other contexts

25% 30% 25%

13

AO3: Solve problems within mathematics and other contexts Students should be able to: • Translate problems in mathematical or non-mathematical contexts into a

process or series of mathematical processes. • Make and use connections between different parts of mathematics. • Interpret results in the context of a given problem. • Evaluate methods used and results obtained. • Evaluate solutions to identify how they may have been affected by

assumptions made.

What is meant by problem solving in AO3?

14

Challenges: What is mathematical problem solving?

Mathematical ‘problem solving’ is a contested term in the literature. Problems are those tasks to which there is no familiar approach to a solution. Teaching for problem-solving is complex because it is likely to draw on deep conceptual understanding, mathematical reasoning and well-developed communication (e.g. Schoenfeld, 2007). It therefore makes substantial demands on the capacity for change of teachers coming new to it (Golding, 2017). In England, we have little historical experience of validly assessing mathematical problem solving, despite its presence in the intended curriculum*

*but note World Class Tests

A pattern is made from four identical squares. The sides of the squares are parallel to the axes. Point A has coordinates (6, 7) Point B has coordinates (38, 36) Point C is marked on the diagram. Work out the coordinates of C.

Is this problem solving?

15

ABCDEFGH is a cuboid. AB = 7.3 cm CH = 8.1 cm Angle BCA = 48° Find the size of the angle between AH and the plane ABCD. Give your answer correct to 1 decimal place.

Is this problem solving?

16

Is this problem solving?

17

Gavin, Harry and Isabel each earn the same monthly salary. Each month, • Gavin saves 28% of his salary and spends the rest of his salary.

• Harry spends 34

of his salary and saves the rest of his salary. • The amount of salary Isabel saves : the amount of salary she spends = 3 : 7

Work out who saves the most of their salary each month. You must show how you get your answer.

The principle: Teachers, students and end-users are supportive of students learning to problem-solve. They felt Pearson resources supported that well.

In-class experiences: Problem solving was initially interpreted as solving worded exercises and only for the most able.

CPD: Pearson provide paid-for and within-resource CPD. The former is rarely used (time and cost); the latter, infrequently (time, and low perception of need) – yet many lesson observations showed teachers ill-prepared for curriculum problem-solving intentions.

Assessment: National assessments at 11, 16, 18 appeared uppermost in teachers’ thinking and framed their understanding of the meaning of ‘problem-solving’.

18

Key findings from the research projects

Evidence from the classroom

(Study D, initial teacher interview 6)

19

One of the biggest issues industry has is finding people who can solve problems. It’s not even the direct application of mathematical topics. … It’s vital. So for me it’s a no-brainer in terms of tackling those kinds of things.

I think the nice thing about the challenge is that, if we continue teaching like this, …we can say to students …we believe that you can do this. I think it must be to the benefit of the students because it's just so much more interesting and useful. Obviously it's going to impact the way you teach, and it’s early days still…but I think that's a really good thing.

(Study B, HoM 13 Summer 2018)

I think the final set of mock papers probably gave us a closer indication to what the real paper is like. But even then, we still, we didn’t expect the kind of questions that we had.

(Study C, HoM19, Autumn 2017)

There were (in first live GCSE papers) lots of problems on top of problems and layered topics which … made it very difficult, so (problem solving) has to be a real focus.

(Study C, HoM 16, Autumn 2017).

Evidence from the classroom

(Study C, HoM11, Spring 2017)

20 (Study A, Y6 teacher 5, Spring 2018).

(Study A, Y2 LO 9)

(Study A, y6 teacher 1, Summer 2018);

(The teacher) did lots of modelling and breaking down of the questions for the children…. This meant that children, in fact, did not engage very much with identifying the steps/approaches/skills needed to solve the problems.

I was thinking if we're really struggling to work out what to do and to think, what chance have the students got when they're, you know, under pressure and not so experienced with that.

‘I, and the pupils, are becoming more confident as we are regularly exposed more to open, contextualised, multi-step problems’

‘Maybe I just need things a bit more highlighted for me, but that would be useful to have that as this is a problem solving and reasoning activity or this is an avenue, if you want to extend, to go off down here. …I don’t feel overly confident with problem solving’

21

• Teachers in all phases are not clear what is meant by ‘problem solving’. • They look first to assessments to clarify what is needed, but early assessments

have had variable demands. • Teachers often lack experience of, and skills and knowledge for, teaching

problem solving, yet might not realise that. • When good use is made of ‘educative’ resources (Davis and Krajcik, 2005)

including integral PD materials, teachers can become much more effective at teaching problem-solving – but this takes time and effort.

• Ofqual has now done some work about aspects of problem solving that correlate with demand (https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/668587/Summer_2017_GCSE_Maths_assessments_review.pdf; see also https://www.tandfonline.com/doi/pdf/10.1080/14794802.2017.1334576

Threats to valid classroom enactment

22

Summary

• Mathematical problem solving is *hard* to teach, especially for teachers coming new to it.

• And *hard* to assess validly, fairly and reliably. • Good assessment of related learning requires shared

understanding of terms used, and of student learning. • Teachers of mathematics in England require focused

and substantial CPD. • Classroom enactment is highly framed by high-stakes

assessments: a coherent curriculum system supports the enactment of problem solving.

• Pearson / UCL research projects to be completed in 2018 and 2019 and will be reported on separately.