cambridge english language assessment: principles of...

© UCLES 2013

Cambridge English Language Assessment: Principles of good practice for test development and quality management

Dr Angeliki Salamoura, Research and Validation GroupGraeme Harrison, Head of Assessment Services, Southern Cone & Andes

© UCLES 2013

Overview• Cambridge English Language Assessment:

who we are & what we offer• Principles of test development• Quality management:

Question Paper Production process• Cambridge English: Advanced (CAE) • Teaching Knowledge Test (TKT)

© UCLES 2013

Cambridge English Language

Assessment

University of Cambridge

Cambridge University

Press

Cambridge English

A not-for-profit university department

© UCLES 2013

Language Policy & strategy

Analysis & diagnostics

Curriculum reform

Materials & resources

Teacher Development

Assessment &

certification

Programme implementation

& evaluation

Cambridge English

educationalvalue chain

© UCLES 2013

Language Policy & strategy

Analysis & diagnostics

Curriculum reform

Materials & resources

Teacher Development

Assessment &

certification

Programme implementation

& evaluation

Cambridge English

educationalvalue chain

© UCLES 2013

English language

exams since 1913 One

of the largest dedicated research

teams

4.5 millioncandidates

per yearAccepted by

12,500organisations

globally

2,700 test centres in

130 countries

Over 50,000preparation

centres worldwide

Assessment and Certification

© UCLES 2013

© UCLES 2013

© UCLES 2013

© UCLES 2013

1913: The first Certificate of Proficiency in English (CPE)

The 1913 UCLES Regulations:

‘The Certificate of Proficiency in English is designed for Foreign Students who desire a satisfactory proof of their knowledge of the language with a view to teaching it in foreign schools.’

© UCLES 2013

1913: The first CPE

Candidates: 3 Exam centres: 2 Fee: £3 (≈ 2,500 CLP) Length (time):

© UCLES 2013

1913: The first CPE

Candidates: 3 Exam centres: 2 Fee: £3 (≈ 2,500 CLP) Length (time): 12 hours!

© UCLES 2013

1913: The first CPE

(ii) Oral: (f) Dictation: ½ hour.(g) Reading aloud and Conversation: ½ hour

© UCLES 2013

1913: The first CPE

(ii) Oral: (f) Dictation: ½ hour.(g) Reading aloud and Conversation: ½ hour

Pass rate: ?

© UCLES 2013

1913: The first CPE

(ii) Oral: (f) Dictation: ½ hour.(g) Reading aloud and Conversation: ½ hour

Pass rate: 0%

© UCLES 2013

1913: The first CPE

The 1913 examiner report on Phonetics:

‘None of the candidates [had] given the subject sufficient attention. The transcriptions were distinctly poor and suggested that the candidates had had but little practice…’

© UCLES 2013

Cambridge English exams and the CEFRCPEC2

1913

FCEB2

1939

CAEC1

1991

PETB1

1981

KETA2

1988

FlyersC2Movers

StartersBeg-A2

Effective Operational Proficiency

level

Council of EuropeVantage

level

Council of Europe

Threshold level

Council of Europe

Waystage level

Mastery level

Breakthrough Level

© UCLES 2013

Maintaining standards over time -an item banking approach using IRT

Probabilistic models:Estimating the probability that a candidate of known ability will

succeed on an item of known difficulty

Item Response models and Rasch analysis provide the statistical approach for building a measurement scale and doing item banking

© UCLES 2013

TEST

A test is a specific (and necessarily limited) sample of performance, from which users seek to make broader generalisations about the knowledge, skills, or abilities a person has.

© UCLES 2013

Test qualities…

Validity Reliability

Impact Practicality

© UCLES 2013

Balancing test qualities…

Validity Reliability

Impact Practicality

© UCLES 2013

The longer the test…

…the more it is.

…the less it is.

valid reliable practical

© UCLES 2013

A Fine Balance

© UCLES 2013

Multiple Choice Test

© UCLES 2013

Short Response

© UCLES 2013

Speaking Test

© UCLES 2013

Speaking Test

Describe and explain why…• Evaluate performance on a scale

Repeat the phrase you hear in the recording

• Accurately repeated = 1• Not accurately repeated = 0

© UCLES 2013

Balancing test qualities…

Validity Reliability

Impact Practicality

© UCLES 2013

How do we ensure reliability?

© UCLES 2013

Speaking and WritingRecruitment

Certification

Training

Standardisation &

RecertificationMonitoring

Examiner Administration

© UCLES 2013

How do we ensure test validity &

fitness for purpose?

© UCLES 2013

Socio-cognitive framework for test validation(Weir 2005)

© UCLES 2013

Socio-cognitive framework for test validation(Weir 2005)

How are the characteristics of

test takers catered for by this exam?

© UCLES 2013

Socio-cognitive framework for test validation(Weir 2005)

Are the characteristics of the test tasks and the test administration fair to the test takers?

© UCLES 2013

Socio-cognitive framework for test validation(Weir 2005)

Are the cognitive processes required to complete the

task appropriate?

© UCLES 2013

Socio-cognitive framework for test validation(Weir 2005)

How far can we depend on the scores of the test?

© UCLES 2013

Socio-cognitive framework for test validation(Weir 2005)

What effects does the test have on its various stakeholders?

© UCLES 2013

Socio-cognitive framework for test validation(Weir 2005)

What external evidence is there outside of the test

scores themselves that the test is doing a good job?

© UCLES 2013

Does the test measure the right things?

Research Notes:quarterly online publication

Studies in Language Testing:Examining Writing (2007)Examining Reading (2009) Examining Speaking (2011) Examining Listening (2013)

© UCLES 2013

© UCLES 2013

External Specialists

•Assessment Managers•Assessment Administrators•Operations Administrators •Validation Officers•Clerical Markers

• Chair (writing team leader)

• Item Writing Team• Content Vetters• Proofreaders• Examiners

Cambridge English Language Assessment

Stakeholders•Pretest centres•Pretest candidates

© UCLES 2013

The Recipe for Question Paper Production

Rejection

Editing

Pre-editing

Commissioning of material

Revision

© UCLES 2013

Pre-editing and Editing – the cake mix

ParticipantsAssessment ManagerChair (writing team

leader)Writers

AimsTopicTopicalityLevel of language Suitability for the task LengthFocus StyleAccuracy

© UCLES 2013

The Recipe for Question Paper Production

Rejection Pretesting

Editing

Pre-editing

Commissioning of material

Revision

© UCLES 2013

Aims of Pretesting – the taste test

• Generate statistical data• Difficulty• Discrimination

• Generate qualitative feedback• Check the quality and fairness of items and

tasks

© UCLES 2013

The Recipe for Question Paper Production

Rejection

Pretest Review

Pretesting & Trialling

Editing

Pre-editing

Commissioning of material

Revision

© UCLES 2013

Pretest Review – the proof of the pudding…

ParticipantsAssessment ManagerChair (writing team leader)Experienced writerAimsevaluate resultsfinalise key

© UCLES 2013

The Recipe for Question Paper Production

Test Construction

Pretest Review

Pretesting & Trialling

Editing

Pre-editing

Commissioning of material

© UCLES 2013

Test Construction

Participants: Assessment ManagerChair (writing team leader)Experienced writerAims:Construct question papersEnsure correct level of difficultyCoverage of skills and appropriate content

© UCLES 2013

The Recipe for Question Paper Production

Test Construction

Pretest Review

Pretesting & Trialling

Editing

Pre-editing

Commissioning of material

Paper checking before sign-off

© UCLES 2013

© UCLES 2013

What is Cambridge English: Advanced (CAE)?

Established and high-quality test of EnglishTaken by hundreds of thousands of students globallyFocused on the level needed for university study (CEFR C1)Proven validity and reliability

© UCLES 2013

Accepted by 3,000 organisations worldwide

Higher education

UK, Australia, North America

Immigration

UK Border AgencyDIAC - Australia

Employers

Accenture, Bayer,Ernst & Young,

Dell

Teachers

Mexico,Finland

State schools

Romania,Germany

© UCLES 2013

Why is CAE different?

CAETherm

om

eter test

More than just a snapshot of English skills

© UCLES 2013

Helps students improve their English

Positive impact

Students take a language improvement course

Focus on communicative language skills

Focus on grammar and vocabulary

© UCLES 2013

Test content

12345

Reading – 75 min

Writing - 90 min

Use of English – 60 min

Listening - 40 min

Speaking - 15 min

© UCLES 2013

Test content

1234

Reading and Use of English - 90 min

Writing – 90 min

Listening - 40 min

Speaking - 15 min

© UCLES 2013

Enhanced securityTest day photosStatistical analysis of results

before releaseOnline results verification

service

© UCLES 2013

Results and Certificate

© UCLES 2013

Why take Cambridge English: Advanced (CAE)?

Established – recognised worldwide

Positive impact – focus on language skills

High quality – test content; administration; results

© UCLES 2013

Teaching Knowledge Test (TKT)

• For both experienced and new teachers• Flexible, modular structure

© UCLES 2013

Module 2: Planning for language teaching

TKT - OverviewC

ore

Module 1: Background to language teaching

Module 3: Classroom management

TKT: Young Learners

TKT: Practical

TKT: CLIL

TKT: Knowledge about LanguageS

pecialist

© UCLES 2013

Summary

Quality Assurance through the QPP process

Cambridge English: Advanced for C1 certification

Teaching Knowledge Test for teacher development

© UCLES 2013

Further information– Learn more about Cambridge English teaching

qualifications – www.cambridgeenglish.org/sector/teaching

– For more information on Cambridge English: Advanced– http://www.cambridgeenglish.org/exams-and-

qualifications/advanced/

– Harrison.G@CambridgeEnglish.org

© UCLES 2013

Thank you!

Any questions?

© UCLES 2013

A range of fit for purpose exams

© UCLES 2013

1913: The first CPE

items about grammar & lexis:‘Give the past tense and past participle of… dividing them into strong and weak; add explanations:tell, wake, buy’

items on grammar & lexical usage:‘Embody each of the following words into a sentence in such a way as to show that you clearly apprehend its meaning:commence, comment, commend…’

© UCLES 2013

What can students do with CAE?

Listening/Speaking Reading Writing

CAN follow abstract argumentation, for example the balancing of alternatives and the drawing of a conclusion.

CAN read quickly enough to cope with the demands of an academic course.

CAN write an essay which shows ability to communicate, giving few difficulties for the reader.

Examples CEFR Level C1 abilities

© UCLES 2013

Typical abilities at CEFR Level C1

React appropriately in different cultural and social situations

Express yourself with a high level of fluency

Participate effectively in meetings or academic tutorials and seminars

Deal confidently with living in an English-speaking environment

Communicate effectively at managerial and professional level

Follow an academic course at university level

7

5

6

4

3

2

1

Carry out complex and challenging research

© UCLES 2013

What is the impact if students are below this level?

Increased requirement for supportStudents fail to reach academic potential

Dissatisfaction amongst home students

© UCLES 2013

Test content

12345

Reading - 1 hour 15 minutes

Writing - 1 hour 30 minutes

Use of English - 1 hour

Listening - 40 minutes

Speaking - 15 minutes

© UCLES 2013

ReadingPart 1 Part 2 Part 3 Part 4Multiple choice6 questions

Gapped text6 questions

Multiple choice7 questions

Multiple matching15 questions

© UCLES 2013

Reading text example

© UCLES 2013

WritingPart 1 Part 2

Compulsory questionWrite one of the following: article, report, proposal or letter.

180-220 words

1 task from choice of 5One of the following: article, competition entry, contribution to a longer piece, essay, information sheet, a letter, a proposal, a report, a review

220–260 words

© UCLES 2013

Writing – sample of students workA book review – The Pelican Brief

© UCLES 2013

Use of EnglishPart 1 Part 2 Part 3 Part 4 Part 5

Multiple-choicecloze12 questions

Open cloze15 questions

Word formation10 questions

Gapped sentences5 questions

Key word transformat-ions8 questions

© UCLES 2013

Use of English - example

© UCLES 2013

ListeningPart 1 Part 2 Part 3 Part 4

Multiple choice6 questions

Sentence completion8 questions

Multiple choice6 questions

Multiple Matching10 questions

© UCLES 2013

Listening

© UCLES 2013

SpeakingPart 1 Part 2 Part 3 Part 4

Conversation betweenthe interlocutor and each candidate (spoken questions)

3 minutes

Individual ‘long turn’ foreach candidate with brief response from the second candidate. In turn, the candidates are given three pictures to talk about

A 1-minute ‘long turn’ for each candidate,plus a 30-second response from the second candidate

A two-way conversation between the candidates. The candidates are given spoken instructions with written and visual stimuli, which are used in a decision-making task

4 Minutes

A discussion on topics related to the collaborative task (spoken questions)

4 minutes

© UCLES 2013

Speaking

http://www.cambridgeesol.org/assets/wmv/univ/c1-speaking.wmv