how much does it help to score language on speaking activities? ben taylor the virginia tech...

HOW MUCH DOES IT HELP TO SCORE LANGUAGE ON SPEAKING ACTIVITIES?

Ben Taylor

The Virginia Tech Language and Culture Institute

Background• 8-week, 10 hr/week Grammar, Listening and Speaking

(GLS) class in an Intensive English Program• Advancement is not pass-fail for speaking: it is largely the

accumulation of in-class speaking assignment grades (70% cutoff) – i.e. the grades matter.

• In-class speaking assessed by a standard rubric

Overview• Current LCI speaking rubric• My questions• Literature

---• Methodology• Results

---• Discussion and implications• Future research• Questions

Standard LCI formative speaking rubric

Content and Delivery(20 pts)

[teacher puts particular requirements of the assignment here]

Fluency(20 pts)

pauses, filled pauses (“um), and self-corrections are limited, natural, and do not interfere with communication

rate of speech is mostly natural and does not interfere with communication

Pronunciation(20 pts)

uses many intonation groups and intonation groups are used similarly to a native speaker

words are pronounced clearly and similarly to a native speaker

Vocabulary(20 pts)

uses a large variety of vocabulary, including academic and advanced vocabulary

Grammatical Accuracy(20 pts)

uses complex and combined verb phrases in sentences; sentences are long

accurately uses articles, subject/verb agreement, and plural markings of nouns

80% of grade weighted to linguistic elements of speaking

(Quick discussion)• How is speaking assessed and graded in your

classrooms?

• How does linguistic feedback (vocabulary, fluency, grammar, pronunciation, etc.) factor into the grade/assessment?

My questions (Pedagogical angst)• In class (not placement), should we penalize students for

being at a stage of interlanguage?• Do problems such as limited vocabulary, halted fluency,

and imperfect grammar represent a lack of effort? Should they be graded just the same way we grade a lack of preparation for an assignment?

• Is there a confusion or conflation between academic and linguistic assessment in our classes?

• Would focusing the grade more on content and less on form allow more communicative freedom/risk taking and thus better language development?

• For the communicative classroom, does providing mostly linguistic feedback convey an incorrect priority?

• In class (not placement), should we penalize students for being at a stage of interlanguage?

• Do problems such as limited vocabulary, halted fluency, and imperfect grammar represent a lack of effort? Should they be graded just the same way we grade a lack of preparation for an assignment?

• Is there a confusion or conflation between academic and linguistic assessment in our program?

• Would focusing grade on content instead of form allow more communicative freedom/risk taking and thus better language development?

• For the communicative classroom, does providing mostly linguistic feedback convey an incorrect priority?

My questions (Pedagogical angst)

Student: “How am I supposed to ‘work on my accent’?”

Literature on correction and motivation

• Numerous studies have been conducted discussing language anxiety (MacIntyre & Gardner, 1991).

• Fear of being corrected, or even penalized, by a professor for incorrect grammar or the inability to understand a certain language concept could keep them from trying to speak in the language at all (Capone, 2013).

• According to Krashen’s (1982) affective filter hypothesis, anxiety, including that derived from the premature or excessive correction of student errors, interferes with the process of acquiring a second language.

• Zoltán Dörnyei discusses student motivation in many books and articles.

Literature on correction and motivation

• There is much research on how motivation affects class grades, but very little on how class grades affect motivation.

• I began a pilot study in my classes in hopes to open the door for institutionally based research.

Research question• Does associating a grade with feedback on linguistic

factors of speaking performance in the L2 classroom help to improve speaking proficiency?

METHODOLOGY

Participants• 25 students in two high intermediate Grammar Listening and

Speaking courses (GLS 450) at the Virginia Tech Language and Culture Institute• Saudi Arabia(12)• China (11)• Panama (1)• Kuwait (1)• All ages 17-40, majority 18-23

• Signed IRB consent forms• Randomly assigned to test/control groups• Blocked so that ½ students in both classes (8-10a/1-3p) were

test and ½ were control• Taught with identical speaking assignments but given feedback

by different rubrics

Standard Rubric

Content and Delivery(20 pts)

Content addresses all questions/parts of the assignment topic demonstrates preparation through complete and detailed content presents information in a logical order with good organizationPresentation style captures and keeps the audience’s attention uses body language and eye contact to engage with audience uses appropriate media (if applicable)Time frame within the required time frame

Fluency(20 pts)

pauses, filled pauses (“um), and self-corrections are limited, natural, and do not interfere with communication

rate of speech is mostly natural and does not interfere with communication

Pronunciation(20 pts)

uses many intonation groups and intonation groups are used similarly to a native speaker words are pronounced clearly and similarly to a native speaker

Vocabulary(20 pts)

uses a large variety of vocabulary, including academic and advanced vocabulary

Grammatical Accuracy(20 pts)

uses complex and combined verb phrases in sentences; sentences are long accurately uses articles, subject/verb agreement, and plural markings of nouns

Test Rubric

Content and Delivery(100 pts)

Content (40 pts) addresses all questions/parts of the assignment topic demonstrates preparation through complete and detailed content presents information in a logical order with good organizationPresentation style (40 pts) captures and keeps the audience’s attention uses body language and eye contact to engage with audience uses appropriate media (if applicable)Time frame (20 pts) within the required time frame

Fluency(20 pts)

pauses, filled pauses (“um), and self-corrections are limited, natural, and do not interfere with communication

rate of speech is mostly natural and does not interfere with communication

Pronunciation(20 pts)

uses many intonation groups and intonation groups are used similarly to a native speaker words are pronounced clearly and similarly to a native speaker

Vocabulary(20 pts)

uses a large variety of vocabulary, including academic and advanced vocabulary

Grammatical Accuracy(20 pts)

uses complex and combined verb phrases in sentences; sentences are long accurately uses articles, subject/verb agreement, and plural markings of nouns

Procedure

• Pre-test: March 25 2013• Post-test: May 13 2013• Both times, participants used a computer-based

recorder with a headphone microphone• Recorded four speaking tasks of increasing complexities• Task description visible for 45 seconds/recorder invisible• Task disappears and recorder appears• No time limit

Pre/post recordings of speaking were used to determine proficiency increase

Tasks• The tasks contrasted the following three dimensions,

following De Jong et al (2012):• complexity (complex versus simple topic)• formality (informal versus formal setting)• discourse type (descriptive versus argumentative)

• Tasks varied slightly between pre-post to reduce direct recall but task dimensions remained constant

Tasks (pre)1. (simple, informal, descriptive): You are speaking on the phone to a

friend. Describe your apartment in Blacksburg to him/her.

2. (simple, formal, descriptive): A few days ago, you saw a car accident. Because you are a witness to the accident, you have been called to a courtroom to describe it to the judge. Tell the judge what happened.

3. (complex, informal, argumentative): Your friend told you she is planning to take a vacation to Africa. However, you think it is a bad idea. Talk to your friend and try to convince them NOT to go on the vacation.

4. (complex, formal, argumentative): You are the owner of a large supermarket in your town. You are trying to purchase some public land to build a parking lot. Your town is having a public meeting to decide if they should allow your company to purchase the land, and you allowed to give a speech at the meeting. Explain your reasons why the town should allow the supermarket to buy the land.

Tasks (post)1. (simple, informal, descriptive): You are speaking on the phone to a friend.

Describe the house that your family lives in, in your home country.

2. (simple, formal, descriptive): A few days ago, you saw a fight between two people. Because you are a witness to the fight, you have been called to a courtroom to describe it to the judge. Tell the judge what happened.

3. (complex, informal, argumentative): Your friend was planning to take a vacation to Africa. However, now they are saying that maybe they don't want to go. Talk to your friend and try to convince them to go on the vacation.

4. (complex, formal, argumentative): The owner of a large supermarket in your town is trying to purchase some public land to build a parking lot. Your town is having a public meeting to decide if they should allow the company to purchase the land. You are a citizen of the town and you allowed to give a speech at the meeting. Explain your reasons why the town should NOT allow the supermarket to buy the land.

Rating speech samples• Pre-post recording pairs were blindly rated by seven

volunteer LCI teachers on scales of 1-10 for vocabulary, grammar, fluency, and pronunciation (the 4 categories on the rubric).

• An equal number of test and control samples were given to each rater to eliminate the effect of teacher bias.

Rating speech samples• Distribution of samples

Test Control

P.ID 1 9 16 19 21 22 24 4 6 7 8 11 17 20 23 25

Rater

1   1   1       1         1 1

2   2 2         2   2       2

  1   4   1 1 1     4        

  3   3   3   3     1         3

  4   1   2             4 2 4  

    3   3   3   3   3   3      

    4   1   2   2       2 4    

Rating speech samples• Rater rubric

Student (S) number: ________ Task (T) number: ________ PRE Score POST Score

Fluency

pauses, filled pauses (“um”s), and self-corrections are limited, natural, and do not interfere with communication

rate of speech is mostly natural and does not interfere with communication

1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10

Pronunciation

uses many intonation groups and intonation groups are used similarly to a native speaker

words are pronounced clearly and similarly to a native speaker

1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10

Vocabulary uses a large variety of vocabulary, including

academic and advanced vocabulary 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10

Grammatical Accuracy

uses complex and combined verb phrases in sentences; sentences are long

accurately uses articles, subject/verb agreement, and plural markings of nouns

1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10

Comments?  

RESULTS

Data• Lost significant amounts of data

• Class transfers• Students missed pre-test• Students missed post-test• Students recorded different tasks successfully in pre- and post-

tests• Students had technical difficulty / submission errors• Unequal number of control/test samples for a given rater

• 2516 participants

Results• 60% of the possible data is not present, so results must

be taken with caution. (Pilot Study)• With the data collected, we could not statistically interpret

the overall effect of the rubric treatment on the proficiency improvement (independent of task type).

• Surprisingly we were able to interpret differences compared by task type.

According to our analysis, it seems as if there is a downward trend in the test students in terms of their improvement from pretest to posttest, as they move from task to task, compared with the control group.

“simple, descriptive”

“complex, argumentative”

Formality was an element of complexity that added difficulty to

both groups.

Both groups improved more with the simpler tasks than they did with the more difficult tasks.

Control students (normal rubric) reflected more stability of improvement depending on type of task.

Test students (test rubric) improved more than control students on the simple tasks, but their improvement decreased noticeably as task difficulty increased.

Results• We can tentatively observe that students who received

the test rubric (no linguistic feedback associated with grade) performed better on simpler tasks than their partners who received graded language feedback, whereas they performed worse on more difficult tasks.

DISCUSSION AND IMPLICATIONS

Discussion and implications• The study seems to indicate that associating linguistic

feedback with class grade suppresses fluency (slower improvement on easy tasks) but elevates accuracy (faster improvement on difficult tasks). In other words it affects the type of improvement more than overall improvement.

• Limitations of the study• Possibility of teacher bias (rating was blind but not teaching)• Rubrics’ different formatting probably affected the types of

comments and level of linguistic feedback I made (independent of whether it was graded).

• Small sample sizes and amount of data

Future research• Repeated, expanded study to confirm

• Larger sample sizes, >100 students (so the main effects can be measured independent of task type)

• Identical format of feedback• 4 levels of grade weight to linguistic sections (20, 40, 60, 80)• Test results for a variety of different proficiency levels• Get large enough results to judge the overall effect of rubric

independent of task type

• What other aspects of Intensive English program curriculum/assessment affect motivation and language proficiency development?

• Inter-program collaboration with a larger-scale follow-up study is welcome!

The big questions

What kinds of proficiency improvement (fluency/accuracy) align with our class goals? And how does that affect our assessment and feedback?

How can our assessment and feedback maximize student motivation?

Questions? Feedback?

Thank you!

Ben TaylorVirginia Tech Language and Culture Institute

bft@vt.edu