neas conference 2016: understanding & engaging with ... · pdf fileneas conference 2016:...
TRANSCRIPT
NEAS Conference 2016:
Understanding & Engaging With Computer-Based Testing Scoring Engines Louise FitzGerald
Presentation overview
1. Automated testing
2. PTE Academic - selected items
3. Scoring and scoring engines
4. Questions
Pedagogic disruption
High stakes tests – Study and immigration
Automated testing
PTE Academic
PTE Academic - Item types
Speaking and Writing
Personal introduction • Read aloud • Repeat sentence • Describe image • Re-tell lecture • Answer short question • Summarize written text • Write essay
Reading
• Multiple-choice, choose single answer • Multiple-choice, choose multiple answers • Re-order paragraphs • Reading: Fill in the blanks • Reading & writing: Fill in the blanks
Listening
• Summarize spoken text • Multiple choice, choose multiple
answers • Fill in the blanks • Highlight correct summary • Multiple choice, choose single answer • Select missing word • Highlight incorrect words • Write from dictation
PTE Academic – Scored practice tests
PTE Academic – Scored practice tests Offer
[S] Item 3: Describe image
No. of tasks: 6-7 Preparation time: 25 seconds Speaking time: 40 seconds
Scoring – Describe image
Enabling Skills and Traits
Content 5 Describes all elements of the image and their relationships, possible developments, conclusions or implications 4 Describes all the key elements of the image and their relations, referring to their implications or conclusions 3 Deals with most key elements of the image and refers to their implications or conclusions 2 Deals with only one key element in the image and refers to an implication or conclusion. Shows basic understanding of several core elements of the image 1 Describes some basic elements of the image but does not make clear their interrelations or implications 0 Mentions some disjointed elements of the presentation
Pronunciation 5 Native-like, 4 Advanced, 3 Good, 2 Intermediate, 1 Intrusive, 0 Non-English
Fluency 5 Native-like, 4 Advanced, 3 Good, 2 Intermediate, 1 Limited, 0 Disfluent
Communicative Skills: Speaking
Enabling Skills and Traits
Content 5 Describes all elements of the image and their relationships, possible developments, conclusions or implications 4 Describes all the key elements of the image and their relations, referring to their implications or conclusions 3 Deals with most key elements of the image and refers to their implications or conclusions 2 Deals with only one key element in the image and refers to an implication or conclusion. Shows basic understanding of several core elements of the image 1 Describes some basic elements of the image but does not make clear their interrelations or implications 0 Mentions some disjointed elements of the presentation
Pronunciation 5 Native-like, 4 Advanced, 3 Good, 2 Intermediate, 1 Intrusive, 0 Non-English
Fluency 5 Native-like, 4 Advanced, 3 Good, 2 Intermediate, 1 Limited, 0 Disfluent
[W] Item 6: Summarize written text
No. of tasks: 2-3 Length of text: Up to 300 words Response word limit: 5-75 words Writing time: 10 minutes
Scoring – Summarize written text
Enabling Skills and Traits
Content 2 Provides a good summary of the text. All relevant aspects mentioned 1 Provides a fair summary of the text but misses one or two aspects 0 Omits or misrepresents the main aspects of the text
Formal requirements
1 Is written in one, single, complete sentence 0 Not written in one, single, complete sentence or contains less than 5 or more than 75 words. Summary is written in capital letters
Grammar
2 Has correct grammatical structure 1 Contains grammatical errors with no hindrance to communication 0 Has defective grammatical structure which could hinder communication
Vocabulary 2 Has appropriate choice of words 1 Contains lexical errors with no hindrance to communication 0 Has defective word choice which could hinder communication
Communicative Skills: Reading and Writing
Scoring – Summarize written text
Enabling Skills and Traits
Content 2 Provides a good summary of the text. All relevant aspects mentioned 1 Provides a fair summary of the text but misses one or two aspects 0 Omits or misrepresents the main aspects of the text
Formal requirements
1 Is written in one, single, complete sentence 0 Not written in one, single, complete sentence or contains less than 5 or more than 75 words. Summary is written in capital letters.
Grammar
2 Has correct grammatical structure 1 Contains grammatical errors with no hindrance to communication 0 Has defective grammatical structure which could hinder communication
Vocabulary 2 Has appropriate choice of words 1 Contains lexical errors with no hindrance to communication 0 Has defective word choice which could hinder communication
Communicative Skills: Reading and Writing
Summarize written text
Test takers need to use just one compound or, preferably, complex sentence to summarize the main point and briefly mention the supporting detail.
Test takers need to check they have only written one sentence of between 5 and 75 words, with a capital letter at the beginning and only one full stop at the end.
1. ‘Just-in-time’ production relates goods produced to demand for goods and reduces the amount of wasted time, costs and materials. (19 words)
2. ‘Just-in-time’ manufacturing minimises production time and costs and stored stock is kept low because each stage is finished just before the next one is started. (25 words)
3. ‘Just-in-time’ is a method of manufacturing, which minimises time, costs and stock kept in factories, as each stage of production is completed before the next is started. (27 words)
[W] Item 7: Write essay
No. of tasks: 1-2 Input text : Up to 4 sentences Word limit: 200-300 words Writing time: 20 minutes
[W] Item 7: Write essay
Subskills tested
• Writing for a purpose (to learn, to inform, to persuade)
• supporting an opinion with details, examples and explanations
• organizing sentences and paragraphs in a logical way
• developing complex ideas within a complete essay
• using words and phrases appropriate to the context
• using correct grammar
• using correct spelling
• using correct mechanics
• writing under timed conditions
Scoring
Communication skill:
Writing Enabling skills:
Content
Form
Development, structure and coherence
Grammar
General linguistic range
Vocabulary range
Spelling
Write essay
Students need to analyze the essay prompt so they know how to answer the question. Note key words, limiting words and task words, e.g. discuss.
Practise producing plans for a variety of persuasive essay topics.
Make sure they’ve practiced typing within the time and word limits. In the test, they’ll lose marks if they don’t!
Make sure your students know to leave a few minutes at the end to edit their work. If they know their common errors, they can just concentrate on those. They won’t have time to much more than that.
[L] Item 19: Highlight incorrect words
No. of tasks: 2-3 Length of recording: 15-50 seconds No. of errors: up to 7
Scoring – Highlight incorrect words
Communicative Skills: Listening and Reading
Partial credit scoring * Points deducted for incorrect options chosen 1 Each correct word -1 Each incorrect word 0 Minimum score
Scoring and scoring engines
Score alignment with IELTS
IELTS PTE Academic
4.5 30
5 36
5.5 42
6 50
6.5 58
7 65
7.5 73
8 79
Score alignment with TOEFL iBT
Item Scoring
PTE Academic Score Guide
http://pearsonpte.com/test-takers/results/
Item Scoring Example: Write essay
+ Content (if 0, no further scoring)
+ Form (if 0, no further scoring) + Other traits
Enabling skills scores
+ Vocabulary Vocabulary
+ Spelling Spelling
+ Grammar Grammar
+ Development, structure and coherence Written discourse
+ General linguistic range
= Total item score
Communicative skills score: Writing
Overall score
Scoring and Results
Automated scoring: objective and reliable
Automated Scoring: Writing YouTube - Automated Scoring FAQs: how the machines are trained
Automated Scoring: Writing YouTube – Latent Semantic Analysis
Automated Scoring: Speaking YouTube - Automated Scoring FAQs: how the system is trained
Fascinating Facts: Scoring Engines No.2 Individual,
discrete features of a response can
be analysed. independently.
No.1 The system is not
‘distracted’ by language-
irrelevant factors, such as poor handwriting.
No.3 The scoring engine
for writing is trained to score any traits that humans can
reliably score.
No.4 The scoring engine has been trained for
every new item on the test by human raters before it goes live.
No.5 All standard
conventions for spelling are
accepted, but test takes must use the same convention throughout their
response.
Automated scoring
The automated system acts like a human rater when assessing test takers’ language skills, but does so with the precision, consistency, and objectivity of a machine.
Automated scoring
For more information on Automated Scoring: White paper: http://pearsonpte.com/institutions/scores/ YouTube: https://www.youtube.com/user/PearsonPTE/featured
Official Guide to PTE Academic with CD-ROM
References https://www.ets.org/Media/Tests/TOEFL/pdf/10855_ellguide.pdf
https://www.ets.org/s/toefl/pdf/toefl_ibt_insight_s1v6.pdf https://www.ets.org/Media/Tests/TOEFL/pdf/SampleQuestions.pdf https://www.ets.org/toefl/ibt/scores/get/ https://www.ets.org/Media/Tests/TOEFL/pdf/SampleQuestions.pdf https://www.ets.org/s/toefl/pdf/toefl_ibt_research_insight.pdf http://www.hagitegas.gr/2013/05/%CE%B7-%CF%80%CF%81%CE%BF%CF%83%CE%B5%CF%85%CF%87%CE%AE-%CF%84%CF%89%CE%BD-%CE%B5%CE%BE%CE%B5%CF%84%CE%AC%CF%83%CE%B5%CF%89%CE%BD/ http://www.startupsmart.com.au/advice/growth/why-disruption-is-about-experiences-not-technology/#.Vx72QHMU8v0.linkedin
35
Questions?
Learn more at future webinars and PTEA teacher training events.