university of jyvÄskylÄ l2 learners performance across l2 writing tasks: comparing tasks and...
TRANSCRIPT
UNIVERSITY OF JYVÄSKYLÄ
L2 learners’ performance across L2 writing tasks: comparing tasks and language
proficiency through CEFR scales
Riikka [email protected]
Centre for Applied Language StudiesUniversity of Jyväskylä, Finland
UNIVERSITY OF JYVÄSKYLÄ
2
Background
CEFLING (http://www.jyu.fi/cefling ), funded by the Academy of Finland (2007-2009)
Part of the SLATE network Integrating SLA and language testing research perspectives
CEFLING: Language testing and assessment: Ari Huhta, Mirja Tarnanen + others from CALS; SLA: Maisa Martin (project leader) L2 Finnish, Riikka Alanen, Florencia Franceschina, Paula Kalaja, Katja Mäntylä L2 English.
Aim of the project is to describe the linguistic features of L2 Finnish and L2 English at various levels of language proficiency as described by the scales of the Common European Framework (CEFR).
UNIVERSITY OF JYVÄSKYLÄ
3
Task in CEFLING
Operational definition adopted by CEFLING:– “an activity which requires learners to use language, with
emphasis on meaning, to attain an objective” (Bygate, Skehan & Swain, 2001, p.11).
A unit of analysis whereby we can analyze and compare data on a number of social and cognitive aspects involved in language learning
In short, our talk is about analysing our unit of analysis
What kind of L2 performance – in terms of CEFR proficiency scales -- did the tasks used in CEFLING measure?
UNIVERSITY OF JYVÄSKYLÄ
4
Tasks in CEFLING
A set of communicative tasks were used to collect written L2 data from 7th to 9th graders (aged 12-16) studying English and Finnish as L2 at school– Tasks carefully designed (see Alanen, Huhta, Tarnanen in
preparation)
(1) Communicative L2 writing tasks: functions, text types and register taken into account: – From informal and formal email messages (a complaint to an Internet
company) to argumentative and narrative texts.
(2) Proficiency level: Aimed at A1 to B2 level Tasks piloted In the end, pupils asked to do five of the following tasks: Tasks 1
and 2 were alternatives, so that pupils either did Task 1, 3 - 6 or Task 2, 3 - 6.
UNIVERSITY OF JYVÄSKYLÄ
5
Tasks in CEFLING
Task 1: Informal email message to a friend Task 2: Informal email message to the teacher Task 3: Formal message to an Internet store Task 4: Opinion Task 5: Story (Task 6: Translation of a letter from Finnish into
English for the grandmother)– Excluded from the study
UNIVERSITY OF JYVÄSKYLÄ
6
What kind of L2 performance did the tasks used in CEFLING measure?
Q1. How do CEFLING L2 tasks compare with L2 teaching and testing materials used in Finland?
Q2. What was the raters’ perception of the CEFR level of the tasks before rating?
Q3. How did L2 students perform on CEFR proficiency scales across various task types? Is there any systematic variation in students’ performance that is task-related?
UNIVERSITY OF JYVÄSKYLÄ
7
Q1. How do CEFLING L2 tasks compare with L2 teaching and testing materials used in
Finland? Both textbooks and the National Certificate examination system
used task types that involved messages (cf. Tasks 1 -3)– Stylistic variation: informal – formal
– Functional variation: greeting – inquiry -- complaint – reclamation etc.
– in the NC, the informal and functionally less demanding types, were mostly found at the Basic level (i.e., intended for A1 and A2 learners)
Both textbooks and the National Certificate examination system used task types that involved expressing opinion (cf. Task 4)– In the NC, opinion pieces were usually -- although not exclusively --
intended for Intermediate and Advanced learners (B1 – C2 learners)
Only textbooks used task types that involved telling a story (narrative) (cf. Task 5)
UNIVERSITY OF JYVÄSKYLÄ
8
Rating Tasks and Scripts
At the same time, the assessment criteria and procedures necessary for the reliable and valid rating were prepared by Ari Huhta and Mirja Tarnanen
A team of raters for L2 Finnish and L2 English was selected and trained for the project– Assessment criteria: CEFR scales for written interaction,
correspondence and notes, messages and forms, creative writing and thematic development, and overall written production.
– Benchmarks: From the pilot performances; Benchmarks published by the Council of Europe for L2 English (no such benchmarks exist for Finnish)
Before rating the scripts, the raters were asked for their perceptions of the CEFR level of the tasks In this talk, this is referred to as Perceived Task Difficulty for short
UNIVERSITY OF JYVÄSKYLÄ
9
Collecting and Analysing the Scripts
3427 performances were collected from 7th to 9th graders (aged 12-16) from schools; 1789 scripts were selected for assessment by a team of trained raters (N=9 for English, N = 11 for Finnish).– Each script was rated by four (L2 English) or three (L2
Finnish) raters chosen from the team • Only those scripts were selected for the analysis of linguistic
features on which the raters were in sufficient agreement (three out of four raters for L2 English, two out of three for L2 Finnish). Final corpus for L2 English contained 562 scripts and for L2 Finnish 825 scripts.
• CHAT & CLAN (CHILDES) http://childes.psy.cmu.edu/ adapted for transcription and preliminary coding
UNIVERSITY OF JYVÄSKYLÄ
10
Task by Task Analysis
Q2. What was the raters’ perception of the CEFR level of the tasks before rating?– Raters were asked which CEFR level the task seemed best
suited for; they were also asked to indicate the second best level as well as the lowest and highest potential level only the raters’ perceptions about the most suitable level will be reported here
Q3. How did L2 students perform on CEFR proficiency scales across various task types, i.e., how did the raters rate the scripts? Is there any systematic variation in students’ performance that is task-related?
UNIVERSITY OF JYVÄSKYLÄ
11
Good to Know Before Task by Task Analysis
National Core Curriculum Objectives for L2 English Writing
By the end of 6th grade: A1.3 Functional elementary language proficiency for ”good performance”
By the end of 9th grade: A2.2 Developing basic language proficiency for ”good performance”
National Core Curriculum Objectives for L2 Finnish
By the end of 6th grade:
Not set
By the end of 9th grade: B1.1 to B1.2 Functional or fluent basic proficiency (Global proficiency -- not set separately for L2 writing)
A2
Figure 1. Median of (median) Ratings for L2 English
Table 1a. National Core Curriculum Objectives for L2 English Writing
Table 1b. National Core Curriculum Objectives for L2 Finnish
UNIVERSITY OF JYVÄSKYLÄ
12
Rubrics
• L2 English and L2 Finnish tasks very similar although not identical
• Language of instructions: L1 or L2 Finnish
• Slight differences in wording e.g in
– Task 2: No mention of English teacher to L2 Finnish learners (instead only ”teacher”)
– Task 3: syntax not as complex in L2 Finnish task
– Task 4: the topics of the opinion pieces slightly different and in Finnish for L2 Finnish learners
• Both L2 Finnish and English learners did two tasks in 2 45 minute lessons (plus Task 6, some other tasks for L2 English), specified for English 10-15 minutes per communicative task
UNIVERSITY OF JYVÄSKYLÄ
13
Task 1 Note to a friend
You’ve set up a meeting with your English-speaking friend at a café. However, something has come up and you have other things to do. Send an email message to your friend.
• Explain why you can’t come.• Suggest a new time and place.
Remember to begin and end the message in appropriately.
UNIVERSITY OF JYVÄSKYLÄ
14
Perceived Difficulty Level and L2 English Learner Performance for
Task 1
Figure 2. Task 1 Difficulty Perceived by Trained Raters (N=12)
0
2
4
6
8
10
12
A1
A2
B1
B2
C1
Figure 3. L2 English Ratings (N=474)
0,00 %
10,00 %
20,00 %
30,00 %
40,00 %
50,00 %
60,00 %
A1
A2
B1
B2
C1
UNIVERSITY OF JYVÄSKYLÄ
15
Perceived Difficulty Level and L2 Finnish Learner Performance for Task 1
Figure 5. L2 Finnish Ratings (N=318)
0,00 %
10,00 %
20,00 %
30,00 %
40,00 %
50,00 %
60,00 %
A1
A2
B1
B2
C1
Figure 4. Task 1 Difficulty Perceived by Trained Raters (N=12)
0
2
4
6
8
10
12
A1
A2
B1
B2
C1
UNIVERSITY OF JYVÄSKYLÄ
16
Task 2 Message to your teacher
You’ve been away from school for a week. Soon you’ll have an English exam. Your teacher, Mary Brown, speaks only English. Send an email message to the teacher.
• Tell her why you’ve been away.• Ask two things about the exam. • Ask two things about the English lessons that
were held during the week.
Remember to begin and end the message appropriately.
UNIVERSITY OF JYVÄSKYLÄ
17
Perceived Difficulty Level and L2 English Learner Performance for Task 2
Figure 6. Task 2 Difficulty Perceived by Trained Raters (N=12)
0
2
4
6
8
10
12
A1
A2
B1
B2
C1
Figure 7. L2 English Ratings (N=492)
0,00 %
5,00 %
10,00 %
15,00 %
20,00 %
25,00 %
30,00 %
35,00 %
40,00 %
45,00 %
50,00 %
A0
A1
A2
B1
B2
C1
UNIVERSITY OF JYVÄSKYLÄ
18
Perceived Difficulty Level and L2 Finnish Learner Performance for Task 2
Figure 8. Task 2 Difficulty Perceived by Trained Raters (N=12)
0
2
4
6
8
10
12
A1
A2
B1
B2
C1
0.00%
5.00%
10.00%
15.00%
20.00%
25.00%
30.00%
35.00%
40.00%
45.00%
50.00%
Figure 9. L2 Finnish Ratings (N=357)
A1
A2
B1
B2
C1
UNIVERSITY OF JYVÄSKYLÄ
19
Task 3 Message to an internet store
Your parents have ordered a PC game for you from a British internet store. When you get the game you notice that it doesn’t work properly. You get upset and decide to write an email message to the internet store. In the message, say
• who you are• what your parents ordered • why you’re unhappy (mention at least two
defects/problems) • how you would like them to take care of the matter • give your contact information
Remember to begin and end the message appropriately.
UNIVERSITY OF JYVÄSKYLÄ
20
Perceived Difficulty Level and L2 English Learner Performance for Task 3
Figure 10. Task 3 Difficulty Perceived by Trained Raters (N=12)
0
2
4
6
8
10
12
A1
A2
B1
B2
C1
Figure 11. L2 English Ratings (N=850)
0,00 %
5,00 %
10,00 %
15,00 %
20,00 %
25,00 %
30,00 %
35,00 %
40,00 %
45,00 %
A0A1A2B1B2
UNIVERSITY OF JYVÄSKYLÄ
21
Perceived Difficulty Level and L2 Finnish Learner Performance for Task 3
Figure 12. Task 3 Difficulty Perceived by Trained Raters (N=12)
0
2
4
6
8
10
12
A1
A2
B1
B2
C1
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
Figure 13. L2 Finnish Ratings (N=676)
A0
A1
A2
B1
B2
C1
UNIVERSITY OF JYVÄSKYLÄ
22
Task 4 Opinion
Choose one of the topics and write about what you think about the matter. Give reasons for your opinion.
1. Boys and girls should go to different classes at
school.
2. No mobile phones at school!
UNIVERSITY OF JYVÄSKYLÄ
23
Perceived Difficulty Level and L2 English Learner Performance for Task 4
Figure 14. Task 4 Difficulty Perceived by Trained Raters (N=12)
0
1
2
3
4
5
6
7
8
A1
A2
B1
B2
C1
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
Figure 15. L2 English Ratings (N=854)
A0
A1
A2
B1
B2
UNIVERSITY OF JYVÄSKYLÄ
24
Perceived Difficulty Level and L2 Finnish Learner Performance for Task 4
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%Figure 17. L2 Finnish Ratings (N=669)
A0
A1
A2
B1
B2
C1
Figure 16. Task 4 Difficulty Perceived by Trained Raters (N=12)
0
1
2
3
4
5
6
7
8
A1
A2
B1
B2
C1
UNIVERSITY OF JYVÄSKYLÄ
25
Task 5 Narrative
Tell about the scariest / funniest / greatest experience in your life. Choose one.
• Tell what happened (what, where, when, and so on).
• Tell why the experience was scary / funny / great.
Write in English in clear characters in the space below (continues on the reverse side).
UNIVERSITY OF JYVÄSKYLÄ
26
Perceived Difficulty Level and L2 English Learner Performance for Task 5
Figure 18. Task 5 Difficulty Perceived by Trained Raters (N=12)
0
1
2
3
4
5
6
7
A1
A2
B1
B2
Figure 19. L2 English Ratings (N=838)
0,00 %
10,00 %
20,00 %
30,00 %
40,00 %
50,00 %
60,00 %
A0
A1
A2
B1
B2
C1
UNIVERSITY OF JYVÄSKYLÄ
27
Perceived Difficulty Level and L2 Finnish Learner Performance for Task 5
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
Figure 21. L2 Finnish Ratings (N=666)
A1
A2
B1
B2
C1
Figure 20. Task 5 Difficulty Perceived by Trained Raters (N=12)
0
1
2
3
4
5
6
7
A1
A2
B1
B2
UNIVERSITY OF JYVÄSKYLÄ
28
Results of Rasch Analyses
FACETS used to run Rasch analyses: – Ordinal data transformed into interval data– Takes into account L2 learners’ proficiency
and the relative toughness of their raters Makes possible to analyse task difficulty
UNIVERSITY OF JYVÄSKYLÄ
29
Rasch Analyses
Zero stands for the center of the range of item difficulties (default origin) Order of task difficulty in L2 English and L2 Finnish, from the easiest
down For L2 English, there was .65 logit points difference between the tasks,
with Task 3 being the easiest For L2 Finnish, there was .75 logit points difference between the tasks,
with Task 2 being the easiest
Measure L2 Finnish Task
.29
.25
.20-.29-.46
Task 2 Task 1 Task 3 Task 5 Task 4
Measure L2 English Task
.38
.00
.02-.10-.27
Task 3 Task 5 Task1 Task2 Task4
Table 2a. Order of Task Difficulty for L2 English
Table 2b. Order of Task Difficulty for L2 Finnish
UNIVERSITY OF JYVÄSKYLÄ
30
Summary
L2 English– Tasks fall within half a level of each other, both sides of A2
and B1 on CEFR scale
• Task 4 was the most difficult, Task 3 easiest L2 Finnish
– Tasks fall within half a level of each other, within A2 just under B1 on CEFR scale
• Task 4 was again the most difficult, Task 2 easiest
Task 4 was quite difficult as expected for both languages
UNIVERSITY OF JYVÄSKYLÄ
31
Discussion
Task 3 was easier than expected– Perceived as a B1 task in both languages– Learner performance was a solid A2
• Slightly skewed towards A1 for L2 English• Slightly skewed towards B1 for L2 Finnish
Task 2 slightly more difficult for L2 English – Perceived as a B1 task in both languages, but:– In L2 English, 34.6% of the scripts were A1 (cf. Task 4:
28.5% of the scripts were A1) greater number of A1 performances may explain the result?
– In L2 Finnish, 44% of the scripts were A2 (31.7% were B1)
UNIVERSITY OF JYVÄSKYLÄ
32
Conclusion
Tasks were designed to elicit data from A1 to B2 learners– Tasks appeared suitable for the target group which
ranged mostly from A1 to B1 – Tasks were capable of eliciting performances from
CEFR levels ranging from A0-C1 (depending on the task)
FACETS used to assign difficulty levels to tasks some variation among the tasks but not much
UNIVERSITY OF JYVÄSKYLÄ
33
Further Study Needed
Task 2 : why did L2 English learners perform lower than expected on this task?
Unlike the other tasks, in Task 2, pupils were asked to make at least four questions Task complexity?
Question formation in L2 English linked to the mastery of certain grammatical structures (like fronting and inversion), while in L1 Finnish, Stage 5 English questions follow SVO pattern (with wh-fronting) : Mistä hän tulee? [lit.] ’Where+from he comes?’
UNIVERSITY OF JYVÄSKYLÄ
34
Further Study Needed
In a separate study (Alanen & Kalaja 2009), we analysed L2 English question formation across proficiency levels and task types in CEFLING L2 English data – Task 2 elicited a much greater number of questions than other tasks
(hardly surprising)
Overall, tasks elicited questions from all stages across proficiency levels A1 to B1, but B1 scripts contained hardly any non target-like, ”incorrect”, Stage 3 questions like *where we go now?, unlike A1 and A2 scripts B1 scripts characterized by the absence of non target-like
questions from a particular stage of development (i.e., lack of errors)?
– Other features must have played a role as well in task assessment → closer analysis needed
UNIVERSITY OF JYVÄSKYLÄ
35
Further Study Needed
Do tasks which require L2 learners to ask questions place different demands on them depending on their L2 and L1? – e.g. is asking questions in L2 Finnish less complex than in L2 English?
We need more research on the relationship between task difficulty (task features) and test takers/learners’ characteristics– For example, interview studies, think aloud protocol studies
on how A1 learners experience a B2 task or vice versa
UNIVERSITY OF JYVÄSKYLÄ
References
Alanen, R. & Kalaja, P. (2009). ”Where the teacher get the phones?”– The Emergence of L2 English Questions in the Communicative Tasks Written by L1 Finnish Learners. Paper presented at BAAL 2009, Newcastle, 3-5 September 2009.
Alanen, R., Huhta, A. & Tarnanen, M. (MS in preparation) Developing tasks and CEFR-based assessment procedures for SLA research purposes.
Bygate, M., Skehan, P. & Swain, M. (2001). Introduction. In M. Bygate, P. Skehan & M. Swain (Eds.), Researching pedagogic tasks. Second langauge learning, teaching and testing (pp. 1-20). Harlow, England: Longman/Pearson Education.