Ofqual’s work on quality of marking

October 2014

Introduction

% of GCSE and A level papers marked that received an EAR, 2011 - 2014

The proportion of GCSE and A/AS level papers where an EAR was requested has continually increased:

0

0.5

1

1.5

2

2.5

3

3.5

4

2011 2012 2013 2014

% o

f scr

ipts

mar

ked

that

rece

ived

an

enq

uiry

GCSE A level

Total

Introduction

% of GCSE and A level qualification results involved in enquiries that had a grade change, summer exam series 2011 - 2014

There is a similar picture of grade changes resulting from EAR

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

2011 2012 2013 2014

% o

f tot

al q

ualif

icat

ions

aw

arde

d re

sulti

ng in

a q

ualif

icat

ion

grad

e ch

ange

GCSE A level

Steps we are taking to improve the quality of marking

1. Analysing this year’s Enquiries About Results data and requiring more information from exam boards

2. Improving the Enquiries About Results (EAR) and Appeals system to make it more simple, transparent and fair

3. Developing better ways for exam boards to measure and report on quality of marking in future

4. Identifying best practice in mark scheme design5. Requiring exam boards to upgrade action plans on quality of

marking 6. Requiring exam boards to improve their monitoring of markers

as they mark7. Evaluating the effectiveness of marker training.

1. Analysing this year’s Enquiries About Results data and requiring more information from exam boardsIssue Provisional data shows increased EAR

requests and more grade changes Increased EAR may reflect teacher

anxiety around changes to qualifications and school performance measures

Grade changes may reflect small changes to marks just below a grade boundary…or may reflect more significant mark changes

In more subjective subjects, small mark changes may reflect legitimate differences of opinion

We need to understand whether there has been deterioration in the quality of marking this summer

What we’re going to do November: exam boards submit final

EAR data to us, including: Average size of mark adjustment for

each paper Detailed breakdown of the causes

of any grade change December: we will publish a report of

the EAR data and analyses provided by exam boards

March 2015: deadline for all boards to have detailed quality of marking action plans based on their analyses

Summer 2015: boards must report on the impact of these plans on the summer exam series

2. Improving the Enquiries About Results (EAR) and Appeals system to make it more simple, transparent and fair

Issue Some teachers lack confidence in the

EAR and Appeals system The EAR process fails to distinguish

between true marking errors and legitimate differences of opinion

Exam boards have varying practice for the re-standardisation of markers

The basis upon which exam boards decide to authorise an extended review of marking is unclear and creates suspicion of unfairness

Teachers don’t think appeal panel hearings are sufficiently independent of the exam boards

What we’re going to do December: launch consultation on our

proposals to overhaul the EAR and Appeals system, including: All markers to be appropriately re-

standardised Boards to review and publish their

principles for extending a review of marking

Boards to have procedures to demonstrate how they meet Appeals’ independence requirement

Summer 2015: we think some of our proposals can be put in place ready for the summer exam series, and the remainder by summer 2016

3. Developing better ways for exam boards to measure and report on quality of marking in future

Issue EAR and Appeals data is not a good

indicator of quality of marking The current Appeals data lumps

together small mark changes in subjectively marked subjects and large mark changes in objectively marked subjects…the latter are far more concerning than the former

There is poor visibility of the quality of marking. Teachers cannot make informed choices about exam boards on the basis of the quality of their marking

What we’re going to do Create standardised quality of marking

indicators across exam boards to give a more accurate picture of marking quality

Summer 2017: indicators will be in place for the reformed GCSEs and revised A levels examined for the first time in 2017

4. Identifying best practice in mark scheme design

Issue Well designed mark schemes are

important determinants of marking quality, especially in more subjective subjects

Levels based mark schemes are often used in subjects associated with poorer consistency of marking

While a lot is known about how to write levels based mark schemes, there are still aspects about which there is little evidence

What we’re going to do We are conducting research studies into

levels based mark schemes so we can require boards to follow best practice

Summer 2015: we will begin to report findings from this long term piece of work

5. Requiring exam boards to upgrade action plans on quality of marking

Issue Before the summer we wrote to exam

boards to check what actions they were taking in response to the issues raised in our Quality of Marking report, published in February 2014

Exam boards wrote back during the summer with action plans which we have reviewed

 

 

What we’re going to do We have identified the need for more

detailed plans to be produced to address outstanding areas of concern raised in our original Quality of Marking report

We expect revised action plans to be with us by early November for review.  We will then monitor their implementation as part of our regular audit activity.

6. Requiring exam boards to improve their monitoring of markers as they mark

Issue Over 50,000 markers are involved in

GCSE and A level marking: the vast majority behave with utmost professionalism

But some rogue markers may need to be weeded out

Online marking opens the possibility of marker probity systems: real time data analysis to identify patterns that would trigger investigation and action

What we’re going to do 2015: require exam boards to:

Tell us how they currently monitor marker probity

Consider how they could improve these systems

Consider implementing new, more robust systems

Summer 2015: some improvements to marker monitoring will be in place

Summer 2016: noticeable improvements to quality control systems will have been made

7. Evaluating the effectiveness of marker training

Issue Standardisation of markers is

increasingly being carried out online Online standardisation has cost and

speed advantages and can be as good as face-to-face training

But online standardisation is unpopular with some markers

And exam boards vary greatly in how they deliver it – the quality of online standardisation is not always as high as it could be

What we’re going to do Conduct an extensive evaluation of the

impact of online standardisation on marking quality: Differences between exam boards Where does it work Where does it not work, and why

Audit exam boards’ standardisation processes in summer 2015

Carry out a research evaluation of the impact of online standardisation on marking quality

Investigate how boards monitor and improve their standardisation processes over time

Summary of actions

Action Nov Dec Jan 2015

Feb Mar Summer2015

Summer 2016

Summer 2017

Exam boards submit detailed EAR data analysis We publish our report of the boards’ EAR data and analyses We launch consultation on our proposals to overhaul the EAR and Appeals process

Deadline for boards to have detailed QoM action plans in place Boards start reporting on the impact of their QoM plans on the summer exam series

We audit exam boards’ standardisation processes Some improvements to examiner monitoring are in place Some reforms to the EAR and Appeals process are in place We begin to report findings from our research into levels based mark schemes

All reforms to EAR and Appeals process are implemented Noticeable improvements to exam boards’ quality control systems are in place

Standardised QoM indicators are in place for reformed GCSEs and A levels