writing and correcting communicative exams joann miller editorial macmillan [email protected]...
TRANSCRIPT
Writing and Correcting
Communicative Exams
JoAnn Miller
Editorial Macmillan
Overview
• Introduction– Alternative testing– What to test– Types of tests
• Communicative testing• Content Analysis• Written tests
– Balance– Formats– Correction
Alternative Testingo Not paper and pencil
o Constant not punctual
Definition:
“the ongoing process involving the student and teacher in making judgments about the student’s progress in language using non-conventional strategies
“techniques that can be used within the context of instruction and can be easily incorporated into the daily activities of the school or classroom.”
The Reflective Portfolio: Two Case Studies from the United Arab Emirates, Christine Coombe and Lisa Barlow, Forum Online, http://exchanges.state.gov/forum/vols/vol42/no1/p18.htm
Portfolios
• a collection of student production over time
• shows the stages in the learning process
• and the stages of the student’s growth.
The Reflective Portfolio: Two Case Studies from the United Arab Emirates, Christine Coombe and Lisa Barlow, Forum Online, http://exchanges.state.gov/forum/vols/vol42/no1/p18.htm
Portfolios in Mexico?
o more subjective
o acceptance
o physical limits
Why written tests?
• Not the only way and maybe not the fairest.
• But easiest with large numbers of students
• More objective• Accepted by institutions, parents,
students
Exam Banks
• A collection of exams for classroom use maintained by the institution itself.
• Written by the teachers themselves or a special committee
• Following institutional guidelines• Could be various “cycles” covering
the same material
The benefits of an exam bank
• Less work for teachers• More standardization in large one-
campus schools and in multi-campus schools– Criteria, instructions and grading
• Face validity– Format
• Uniform length
What are your exams going to be like?
• Many variables:– Institution– Students– Teachers– Text– Time
Teachers
• Level of English• Lack of mathematical skills• Time factor• Ease of grading• Answer key
Students• Younger students
– More images– Shorter exams– Humor
• Older students– Professionalism– Humor
How many points for each skill?
• If institution tells you, just follow through
• If not, base exam on the textbook (the common denominator)– General text analysis– How much time is spent on each skill– Count exercises in a few units, determine
percent
• Keep institutional goals in mind
Reasons for Testing• Placement tests
– student’s suitability to take a specific course – based on specific textbook
• Proficiency tests– check students’ progress in general – TOEFL, First Certificate, etc.
• Achievement tests– check how much a student has learned– based on what a student has studied in a
specific course
Communicative testing
We teach “communicatively” but we test “traditionally”.
What IS communicative testing?
Communicative testing means testing in context.
Grammar?
What will you test?
Which version? Why?Circle the correct answer
1. Do you like __________? a.swimming b. to swum c. swim
2. Where ________ live? a.does she b. she does c. she
3. I _________ speak French. a.no speak b. doesn’t c. don’t
4. What __________? a. does he do b. does he c. he does do
Write the correct forms of the words in parentheses.
Alice: Where (1)______ you _________ (live)?
Bart: Acapulco.
Alice: My brother (2)___________ (go) there every summer on vacation, but he (3)_________(not speak) Spanish.
Bart: Acapulco (4)_________ (attract) tourists from all over the world. Many people there (5)___________(speak) English very well. What about you, (6)______ you ________(speak) Spanish?
Alice: A little.
What are you testing?
• Memorization of grammar rules?
• Recognition of correct answer?
• Limited language knowledge?
Traditional testing
• Ability to use rules in a realistic context?
• Ability to produce accurate language?
• Ability to improvise language to communicate ideas clearly?
Communicative testing
Vocabulary?
What will you test?
Which version? Why?Match the letters (a to e)
with the numbers (1 to 5).
1. Your mother’s husband is your___.
2. Your mother’s father is your___.
3. Your mother’s brother is your___.
4. Your uncle’s son is your___.5. Your father’s sister is your
___.
(a) uncle(b) cousin(c) aunt (d) father(e) grandfather
Underline the word in each pair that completes the conversation correctly.
My (1)[ uncle / aunt ] likes
(2) [ playing / going to
] movies. He is my father’s
(3)[ sister / brother]. He’s
(4)[ heavy / average ] and
he has (5)[ blue / brown ]
hair. His birthday is on
October (6)[twelve / twelfth
].
Functions?
What will you test?
What is a function?
• The communicative purpose of the users of the language.
• How language is used.
• Usually expressed as gerunds: introducing, apologizing, asking directions, requesting
Examples of a Functional Cycle
Function: Requesting(1) Open the window, please.(2) Would you open the window?(3) Could you please open the window?(4) Would you mind opening the window?(5) I was wondering if you would mind opening
the window.(6) I’d be grateful if you opened the window.
Each time the difference in register is emphasized.
Where can you find them in your
textbook?
How to test? Complete the conversation.
Complete the conversation logically. Use the words in parentheses. Miriam: Tell me about your new apartment.Mary: (1)____________________(big / living room).Miriam: (2)___________________(how / bedrooms)?Mary: There are two, but (3)________(any furniture) in one of them.
Or: Miriam: Tell me about your new apartment.Mary: (1)_______________________(living room).
Or:Miriam: Tell me about your new apartment.Mary: (1)________________________________.
Content Validity
Assessment should be based on a content-analysis of the text being used
Content Analysis
• You must test only material students have seen
• The only common denominator is the textbook
• Analysis of percent of time spent on each topic (grammar structure, vocabulary item, function, etc.)
Content Analysis: Information from the contents
Functions (10 points):Talking about imitation products Talking about food and food festivalsDiscussing the movie industry Making a business plan
Grammar (5 points):Nouns in groupsIndefinite Pronouns
Vocabulary (10 points):Food Business language
///// //////// ////////// /////////
////////
////////
8109532
35
8
53
8
8 / 32 =__%
25%31%28%16%
25% X 10 = ___pts
2.5 pts3 pts3 pts1.5 pts
37% 2 pts 63% 3 pts
63% 6 pts 37% 4
pts
(P1) 1st Parcial (I: 1, 2 , 3, 4)Grammar BE: all persons, IWQ, Y/N, short ans,, contractions 3 points
Subject and object pronouns, possessive adjectives 2 points Simple present 4 points
Demonstratives, one/ones, which 3 points 15 pts How much 1 point
Comparative 1 point What kind? 1 point
Functions Introducing self/ someone; Ask about someone: name,place of origin, exchange personal info, spelling)
5 points
Describing work/school; asking for and giving opinionabout jobs
2 points
Talking about daily schedule 1 points Talking about and giving opinion about prices 2 points
Talking about preferences, comparing, buying andselling
3 points 20 pts
Talking about likes and dislikes 2 points Giving opinions 2 points
Making invitations; accepting/rejecting; giving excuses 1 pointVocabulary Names and titles 1 point
Time (expressions 1 point Jobs and workplaces 2 points 5 pts
Collocations and materials 1 pointReading True/false, comprehension questions , ads and pic-
tures, charts, choose the title5 points 5 pts
Listening Charts 5 points 5 pts
Formats
PracticePractice vs Testingvs TestingIn class practice During exam
Goals
Content
Learner activity
Teacher activity
Class-room climate
learning feedback on learning
process oriented product orientedopen ended close ended
ss know material students might not knowsuccess-oriented success/failure orientedpeer teaching no peer teaching
helps performance gives tasks
cooperative competitiverelaxed tenseintrinsic motivation extrinsic motivation
Balance
Ideally an exam will balance:– Accuracy and fluency– Production and recognition– Objective and subjective sections
Accuracy and Fluency• Fluency
– The ability to produce written and / or spoken language with ease
– Communicate ideas effectively– The ability to use vocabulary chunks
(phrases) to facilitate communication
• Accuracy– Ability to produce grammatically correct
sentences
Production and Recognition
• Production– Student writes more than one word– Can be creative / involves more “mental”
work– More than one answer may be possible
• Recognition– Student recognizes correct answer– Not creative– Only one correct answer
Objective and Subjective Sections
• Subjective– There is more than one possible answer– Corrector must be trained and experienced– There can be surprises– Students can protest grading
• Objective– There is only one answer– Anyone can correct the exam– No surprises– No argument from students
• Grammar / Vocabulary / Functions?
•Accuracy or fluency?
• Production or recognition?
• Subjective or objective?
III. They continue talking. Circle the correct sentences to complete the conversation. ( 3 pts, .5 each)Caesar: Where do you work?Cleopatra: I work for the people of Egypt at the palace.
Caesar: (1) (a) Really? (b) OK. (c) No, you don’t.
(2) (a) Where do you work? (b) What do you do there? (c) Where do you do?Cleopatra: I help people and I look beautiful.
Caesar: (3) (a) How do you work? (b) What do you live? (c) How do you spend your day?Cleopatra: I work in the palace.
Caesar: (4) (a) What do you like? (b) How do you like your job? (c) How do you do?
Cleopatra: (5) (a) I love it. (b) I don’t do it. (c) It’s at work.
(6) (a) It’s boring. (b) It’s great. (c) It’s late.
• Grammar / Vocabulary / Functions?
•Accuracy or fluency?
• Production or recognition?
• Subjective or objective?
• Grammar / Vocabulary / Functions?
•Accuracy or fluency?
• Production or recognition?
• Subjective or objective?
• Grammar / Vocabulary / Functions?
•Accuracy or fluency?
• Production or recognition?
• Subjective or objective?
• Grammar / Vocabulary / Functions?
•Accuracy or fluency?
• Production or recognition?
• Subjective or objective?
• Grammar / Vocabulary / Functions?
•Accuracy or fluency?
• Production or recognition?
• Subjective or objective?
What is an exam section?
• A certain number of items testing the same skill / aspect
• To be communicative, they should be written as a conversation, note, letter, or some “real” type of discourse
• All items in a section should be worth the same number of points and test a similar skill (all grammar, all vocabulary, all functions, etc.)
Determining sections
• You can combine point values within the same aspect.– Put two grammar structures in the same
section
• You can divide point values between two sections– Divide the points from one structure and
put them in two sections
Combining point values within the same aspect.
Vocabulary (8 points)• Family relationships 2.5• Describing people 0.5• say tell ask 1• Phrasal verbs 2• Every day expressions
2
Dividing point values between two sections
Vocabulary (8 points)• Family relationships 2.5• Describing people 0.5• say tell ask 1• Phrasal verbs 2• Every day expressions
2
1.5
1
Formats
Multiple Choice
• Parts of question– Stem– Options– Distractors
1. XXXXXXXXXXXXXXXXXX?
a. YYYYYYYYY
b. ZZZZZZZZ
c. AAAAAAAA{{
Correct answer
Stems1. Before writing, identify the one point to be
tested by that item. 2. The stem should either be an incomplete
statement or a direct question3. Don’t include words that do not contribute to
the basis for choosing among the options. For example, The American flag has three colors. One of them is (1) red (2) green (3) black
vs. One of the colors of the American flag is (1) red
(2) green (3) black
Kehoe, Jerard. Writing Multiple-Choice Test Items. ERIC/AE Digest Series EDO-TM-95-3, October 1995. http://www.ericdigests.org/1997-1/test.html
4. Include as much information in the stem and as little in the options as possible.
5. Restrict the use of negatives in the stem. Negatives in the stem usually require that the answer be a false statement.
6. Avoid irrelevant clues to the correct option.
Grammatical construction
Options (Kehoe)
1. Use three or four options. 2. Construct distractors that are
comparable in lengthcomplexity grammatical form
3. After the options are written, vary the location of the answer randomly.
Ordering Multiple Choice Items
Numerical a. 1939b. 1940c. 1941d. 1942
Burton, Steven J. Richard R. Sudweeks, Paul F. Merrill, Bud Wood. How to Prepare Better Multiple-Choice Test Items: Guidelines for University Faculty, Brigham Young University Testing Services and The Department of Instructional Science. 1991. http://testing.byu.edu/info/handbooks/betteritems.pdf
Sequential a. Heating ice from -100°C to
0°C. b. Melting ice at 0°C. c. Heating water from 0°C to
100°C. d. Evaporating water at
100°C. e. Heating steam from 100°C
to 200°C.
Sequential a. Heating ice from -100°C to
0°C. b. Melting ice at 0°C. c. Heating water from 0°C to
100°C. d. Evaporating water at
100°C. e. Heating steam from 100°C
to 200°C.Alphabetical a. Changing a from .01 to .05.b. Decreasing the degrees of freedom.c. Increasing the spread of the exam scores.d. Reducing the size of the treatment effect.
True / FalseAdvantages: • Can test large amounts of content
– Students can answer 3-4 questions per minute
Disadvantages: • They are easy
– Students have a 50-50 chance of getting the right answer by guessing
– It is difficult to discriminate between students that know the material and students who don't
• Need a large number of items for high reliability
Designing Test Questions, Grayson H. Walker Teaching Resource Center, The University of Tennessee at Chattanooga, http://www.utc.edu/Administration/WalkerTeachingResourceCenter/FacultyDevelopment/Assessment/test-questions.html
Tips for Writing Good True/False items
• Avoid double negatives.• Avoid long/complex sentences.• Use specific determinants with caution:
– never, only, all, none, always, could, might, can, may, sometimes, generally, some, few.
• Use only one central idea in each item; don't emphasize the trivial.
• Don't lift items straight from the text.• Make more false than true (60/40).
– (Students are more likely to answer true.)
How to “save” a T/F section…• Add a third option
– “Not mentioned”
• OR
• Have student correct F answers
• But only if students have practiced this version in the textbook
Cloze-type sections
• Write words or phrases from a box
• Write the correct forms of verbs, comparatives, etc.
• Even Complete the Conversation
True Cloze
A cloze test is a special type of fill-in exercise where, for example, every 5th word in a paragraph of about 150 words is deleted. (It could be every 6th word, or every 7th word, and so on.)
Boxes• In these sections, students are given a
text with certain words omitted • The omitted words / phrases (and
perhaps some distractors) are put in a box at the top or side of the text
• More difficult if extra options are supplied
• The student completes the “Cloze” exercise with the words presented
• Can be used with grammar, vocabulary or functions (chunks)
Fill in the blankA set of sentences or a text which has blanks in it
for the students to complete with the correct or appropriate word.
Example:He walked _____ school.
He ______ (walk) to school.
Fill-in-the blank exercises are a good way of reinforcing new grammar and vocabulary.
Also called: fill-in the gap, fill-in
Ordering: text and sentence
• Writing a text in order (paragraph, story, conversation) – evaluates the student’s ability to recognize
discourse cues (pronouns, connectors, chunks, etc.)
• Writing a sentence in the correct order – evaluates a students knowledge of syntax,
which is considered part of grammar.
cream / I / ice / like
Complete the conversation• These sections evaluate a student’s
ability to communicate ideas if they are corrected for communication and not for accuracy.
• They can be written with different degrees of cueing.
Complete the conversation.
Complete the conversation logically. Use the words in parentheses. Miriam: Tell me about your new apartment.Mary: (1)____________________(big / living room).Miriam: (2)___________________(how / bedrooms)?Mary: There are two, but (3)________(any furniture) in one of them.
Or: Miriam: Tell me about your new apartment.Mary: (1)_______________________(living room).
Or:Miriam: Tell me about your new apartment.Mary: (1)________________________________.
Point Values• Give more points to…
– Production items• Give fewer points to…
– Recognition items• Give partial credit in
– Fluency / Production sections• Use fractions only if your teachers
are mathematical
Instructions
• Keep them simple• You can use Spanish in lower levels• You can translate them into Spanish
whenever necessary.• Use the same wording in all your exams• Use examples whenever necessary
– But be careful they don’t give away the pattern
• Remember: The instructions are NOT part of the exam…
Correcting communicative sections
Correcting
Grammar, Reading, Vocabulary, Listening
In general these sections are all right or all wrong.
We rarely give partial credit.
These sections test accuracy.
Communicative sections
You can give partial credit
These sections test fluency.
Ask yourself if the S’s answer communicates what the S wants to say.
Examples of partial credit
Correct answer: What’s your name?Student writes: What you name?
Correct answer: If you invited me, I’d go.Student writes: If you invite me, I go.
Correct answer: I went to the movies yesterday.
Student writes: I go to the movies yesterday. I go to the movies.
IV. The clerk knows Cleopatra. Caesar asks the clerk about Cleo. Complete the conversation. Use the words in parentheses. ( 4 points, .5 each)
Clerk: Yes, I know her.
Julius: (1) _______________________________________ (work) ?
Clerk: (2) ________________________________ ( palace downtown).
Julius: (3) _______________________________________ ( do) ?
Clerk: (4) ___________________________________ ( help people).
Julius: (5) ___________________________________ ( close friend)?
Clerk: Yes, (6) ____________________________________ (funny).
Julius: (7) _______________________________________ ( sports)?
Clerk: Yes, (8) ___________________________________ ( tennis ).
Actual Student Responses on the worksheet