Download - TOEFL iBT Test Score Requirements for Entrance to - …baleap.qmlanguagecentre.on-rev.com/pdf/Nissan_slides.pdfEstablishing TOEFL iBT Test Score Requirements for Entrance to Universities

Copyright 2010 by Educational Testing Service. All rights reserved. 14962


Establishing TOEFL iBT Test Score Requirements for Entrance to Universities

Susan Nissan [email protected]

Educational Testing Service

April 12, 2011


TOEFL iBT : Establishing score requirements

OutlineOverview of the testDefinition of standard settingThe dilemmaStandard setting methodologiesSample judgmentsAdditional considerations

2


Overview of the TOEFL iBT test

Four sections: Reading Listening 10 minute break Speaking Writing

Test time: 4.0 hours Note taking allowed throughout the test

3


TOEFL iBT Reading

3 - 5 reading passages from academic texts Reading passages approximately 700 words long 12 to 14 questions about each reading passage

Purposes of reading: - Reading for information - Reading to Learn (by categorizing information by filling in chart or

completing a summary or by Paraphrasing) Glossary feature Review feature helps test taker to track questions not

answered

4


TOEFL iBT Listening

4 - 6 lectures; some with classroom discussion About 5 minutes each Each has 6 questions

2 - 3 conversations About 3 minutes each Each has 5 questions

Questions measure: Basic understanding Connecting information Pragmatic understanding

5


TOEFL iBT Speaking

2 independent tasks about familiar topics Responses based on personal experience

4 integrated tasks 2 Reading/Listening/Speaking tasks 2 Listening/Speaking tasks Responses based on what was read and heard must

include accurate content for highest score Academic and campus-based material Short listening and reading material

6


TOEFL iBT Speaking Section - Rating and Scores

Each task is rated by a trained rater on a scale of 0 4. Responses are rated on topic development, delivery, and language use.

The scores for the 6 tasks are then converted to a scaled score of 0 30.

ETS-trained raters evaluate responses via ETS Online Scoring Network, re-calibrate before each rating session.

7


TOEFL iBT Writing

1 integrated task: Reading-Listening-Writing Short academic listening and reading material Response based on what was read and heard,

must include accurate content for highest score 1 independent task

Response based on personal experience

8


Sample Integrated Writing task

9



10



11


TOEFL iBT Writing Section - Rating and Scores

Scoring guidelines for Independent writing task Development of ideas Organization Quality and accuracy of language used

Guidelines for Integrated writing task Quality of writing Completeness and accuracy of response

ETS-trained human raters evaluate responses via ETS Online Scoring Network

Ratings are 0 - 5 Then converted to a scaled score of 0 30

12


TOEFL iBT score reportsUniversities receive: Four skill scores

Reading 0 - 30 Listening 0 - 30 Speaking 0 - 30 Writing 0 - 30

A total score: 0 120

Students receive a total score, section scores, and performance descriptors.

13


Importance of section scores

Which student with a total score of 80 has sufficient language proficiency to succeed at your university?

Consider the use of section scores as well as the total score!

14

Reading Listening Writing Speaking TotalStudent 1 17 20 24 19 80Student 2 20 20 20 20 80Student 3 25 25 16 14 80


Standard setting - DefinitionStandard setting is a procedure for determining a recommended score for a specific purpose.

Also referred to as a cut score, score requirement, performance standard.Standard setting: Is a judgment process Relies on the concept of a student with minimally acceptable skills

(not a student with average skills) Focuses on a student for whom English is not a first language but

whose English proficiency is good enough to cope in academia

15


The dilemma

How high to set the score requirement?

If the score requirement is too high: International students may need to go elsewhere for

their studies and not to your university. If the score requirement is too low:

International students who do not have sufficient language proficiency to succeed will be accepted.

16


Methodology

Modified Angoff method: relies on concept of a minimally able examinee

For Speaking and Writing: Decide whether a test takers responses meet those criteria of acceptable performance

For Listening and Reading: Decide whether a person who has that defined level of minimal ability will answer each item correctly

17


Panel members Panelists need to be:

Familiar with the NNS population at the institution Familiar with target language tasks Familiar with ESL, at least some panelists

Typically, a panel has 12-20 participants, including University professors from a variety of disciplines University administrators (eg, admissions staff, international student

advisors) ESL instructors

18


Overview of process for Writing (and Speaking)

Define language tasks of first year students E.g., writing assigned papers, synthesize information from

multiple sources Define the minimally acceptable language user and skills

Describe strong writer and weak writer, and use those descriptions to come up with minimally acceptable writer

Panel needs to reach consensus about the definition of a minimally accepter writer

19


Sample description of minimal Speaking ability *

Accent doesnt hinder communication Appropriate use of vocabulary, but not without some errors Has some flow, connected dialogue Has semblance of organization May mix formal and informal speech but meaning is clear Reasonable pace

* From university panel, not ETS standards

20


Sample description of minimal Writing ability *

Organized and logical, with clear evidence of chunking of ideas Generally cohesive, though some sentences may be choppy Register is generally appropriate for written discourse Uses mostly accurate vocabulary (meaning is clear, not distorted) Uses mostly accurate grammar (grammatical errors do not distort

meaning)

* From university panel, not ETS standards

21



Continued: Become familiar with the assessment tasks and the

scoring guidelines Two writing tasks Scoring guidelines for independent task and for

integrated task

22



Scoring guidelines for Integrated task at Level 5 A response at this level successfully selects the important

information from the lecture and coherently and accurately presents this information in relation to the relevant information presented in the reading. The response is well organized, and occasional language errors that are present do not result in inaccurate or imprecise presentation of content or connections.

23


Scoring guidelines for Integrated task at Level 3

A response at this level contains some important information from the lecture and conveys some relevant connection to the reading, but it is marked by one or more of the following:

Although the overall response is definitely oriented to the task, it conveys only vague, global, unclear, or somewhat imprecise connection of the points made in the lecture to points made in the reading.

The response may omit one major key point made in the lecture. Some key points made in the lecture or the reading, or connections between the

two, may be incomplete, inaccurate, or imprecise. Errors of usage and/or grammar may be more frequent or may result in noticeably

vague expressions or obscured meanings in conveying ideas and connections.

24



Continued:

Practice making judgments, by answering this question about a students responses:

Has this student demonstrated the minimally acceptable writing skills adequate for first-year studies at our university?"

25



Continued: Each panelist reads student responses to both writing tasks and

individually makes Round 1 judgments for each student. Each judgment is based on this question:

Has this student demonstrated the minimally acceptable writing skills adequate for first-year studies at our university?" Discuss Round 1 judgments, entire panel Make Round 2 judgments, individually

26


Overview of process for Reading and Listening

Begin with definition of minimally able student, with just enough language skills to succeed at university

Become familiar with the test tasks Each panelist decides, for each test item, whether a

student with minimally acceptable proficiency will answer the correctly.

Discussion of initial judgements, then round 2.

27


Final recommendation for score requirementsOther information that informs decision: Percentile data (based on 2009 testing year)

http://www.ets.org/research/policy_research_reports/toefl-sum-09TOEFL Score Percentile (2009 data)

80 45 100 81 Score requirements of similar institutions

28

http://www.ets.org/research/policy_research_reports/toefl-sum-09


Questions?

Thank you for attending

29

Slide 1TOEFL iBT : Establishing score requirements Overview of the TOEFL iBT testTOEFL iBT ReadingTOEFL iBT ListeningTOEFL iBT Speaking TOEFL iBT Speaking Section - Rating and ScoresTOEFL iBT WritingSample Integrated Writing taskSample Integrated Writing taskSample Integrated Writing taskTOEFL iBT Writing Section - Rating and ScoresTOEFL iBT score reportsImportance of section scoresStandard setting - DefinitionThe dilemma MethodologyPanel membersOverview of process for Writing (and Speaking)Sample description of minimal Speaking ability *Sample description of minimal Writing ability *Overview of process for Writing (and Speaking)Overview of process for Writing (and Speaking)Scoring guidelines for Integrated task at Level 3

Overview of process for Writing (and Speaking)Overview of process for Writing (and Speaking)Overview of process for Reading and ListeningFinal recommendation for score requirementsSlide 29

Download - TOEFL iBT Test Score Requirements for Entrance to - …baleap.qmlanguagecentre.on-rev.com/pdf/Nissan_slides.pdfEstablishing TOEFL iBT Test Score Requirements for Entrance to Universities

Top Related