washbackof the center listening test

Japan Society of English Language Education

NII-Electronic Library Service

JapanSociety ofEnglish Language Education

Washbackof the Center Listening Test

on Learners'Listening Skills and Attitudes

Akiyo MRAIU>iivenyity of 7lsukuba

Ryoko FUJITAGraduate SchooL Uhiversdy of71sukuba

Makiko ITOAikoku Gakzten Ityugasaki Slenior H7igh Schoot

Toshihide O'KIHbkuoh Cl}iiversity

Abstraet

In 2006, an English listening test was added to the National Center 'Ibst

fbr UniversityAdmissions (NCT) in Japan. We investigated whether including the Center listening test on the

NCT has had positive washback on students' listening skills and attitudes toward studying

listening. In Study 1 , test sceres ofuniversity freshmen dating back to 2002 were analyzed to

investigate the improvement in students' listening skills. In Study 2, a questionnaire survey was

conducted to investigate students' motivational aspects. The results revealed three findings. First,

the test scores did not show a marked improvement in listening skills after the Center listening test

was introduced. Second, the Center listening test might influence students in different majors to a

different degree. Third, the majority ofstudents favored the introduction ofthe Center listening

test, and most were encouraged to study listening and felt their preparation had a positive efTect on

their score.

1. Introduction

As Japanese society becomes more globalizcd, people have more opportunities to use

English as a means ofcommunication in the international arena, However, they realized a

discrepancy existed between the English education they had at junior high school and above for

more than six years and their cornrriand efEnglish. Thus, to cope with the globalization ofthe

31




world, the Ministry ofEducation, Culture, Sports, Science, and 'Fechnology

(MEXT) emphasizedthe importance ofcultivating students' practieal communication skilis in the Course ofStudy fbr

Senior I-Iigh Schools, which was issued in 2000 and fu11y implemented in 20e3. In addition, theMEXT issued an Action Plan in 2003 to cultivate

C`Japanese with English language abilities'i to be

able to cemmunicate internationally in the language. One ofthe plans in the Action Plan was to

add a listening test to the English subject ofthe National Center 'fest

fbr University Admissions

(NCT), as the NCT has had a great impact on improving teaching methods and learners'motivation (MEXT, 2003),

The NCT is a nationwide university entrance examination administered once a year in Japan

and it is a high-stakes test fbr many students since the NCT scores are given significant weight in

the admissions process at many schools, As of2012, 685 universities and colleges in Japan use the

NCT for admissions, Accordingly, more than 520,OOO applicants take the NCT every year in

January (National Center For University Entrance Examinations, 201 2). rlb

this widely used test, a

30-minute English listening test with a total score of50 points was added to the 80-minute written

English test with a tota1 score of200 points in Jariuary 2006 (MEXI) n. d.). The purpose ofthis

paper is to investigate whether including the listening test on the NCT has influenced Japanesestudents' attitudes toward studying listening and the improvement in their listening skills,

As the MEXT aims to nurture students' comrriunication ability by introducing the Center

listening test, it is important to overview studies that examine whether the content ofthe test

reflects communicative intention. Tanaka and Sage (2006) examined the usefulness ofthe test by

comparing it with the [fest ofEnglish as a Fereign Language (TOEFL) iBT listening section. They

pointed out the passages of the Center listening test lack academic content and do not reflect

real-life language use compared with the TOEFL iBT, Thus, Tanaka and Sage suggested that, to

envision what the MEXT Action PIan advocates, the Center listening test should include test tasks

that measure listening as an interactive, integrative communicative ski11s in terrns ofacademic and

conversational situations, both of which occur in English language classes at Japanese universities.

Although the TOEFL and the NCT are primarily fbr students who wish to enter university,

comparing them is dificult. The TOEFL is a proficiency test, whereas the NCT {s an achievement

test with the content and vocabulary size limited to the range specified by the Course ofStudy, or

high school textbooks.

Research that compared the Center listening test with the Course of Study was conducted by

Yanagawa (20l2). He examined the validity ofthe Center listening test within the framework of

sociocognitive validity, which concerns whether the test reflects the listeners' cegnitive processand contextual parameters in real-life ianguage. Ylinagawa claimed that the Center listening test

lacks the sociocognitive validity specified in the Course of Study. He pointed out three major

problems: (a) the iistening test does not cover enough phonological ehange such as assimilation

and elision, <b) few questiQns ask the speakers' implicit intention, and (c) the speakers tend to

speak only with a Nonh American accent.

32




The main issue the MEXT concerns is whether the Center listening test has had positivewashback on teaching and learning. The term washback means

"a part ofthe impact a test may

have on leamers and teachers, on educational systems in general, and on society at large" (Hughes,2003, p, 53). According to Messick (1996), washback is a censequential aspect ofconstruct

validity, and should be demonstrated by collecting outcomes after implementing the test.

Watanabe (2004) conceptualized washback with five dimensions: specificity (general or

specific), intensity (strong er weak), length (short or long period), intentionality (unintended or

intended), and value tpositive or negative), Using these dimensions, the Center listening test

should have Lspecific' but `weak'

washback for a `short'

period, since the test relates to only one

specific skill ofEnglish ability (i.e., listening), might affect only a part ofthe classroom events at

high school, and might not infiuence students after they enter university, `Intended'

washback is`positive'

as thc tcst started aiming to enhance students' communication ability and motivation forstudying listening, while

`unintended'

washback can be `negative'

or `positive.'

[Ib date, few washback studies on the Center listening test have been conducted, For

example, Takeuchi and Kozuka (201e) compared the 2005 and 20e6 rlest

ofEnglish 'fbr

International Communication (TOEIC) scores of university fireslmen who majored in English,

Intemational Cultural Studies, or Early Childhood. fllatkeuchi

and Kozuka fbund that the 2006

scorcs ofthe International Cultural Studies and English majors were higher than those in 2005,

though only the International Cultural Studies majors' improvement was significant. The authors

explained that English majors may net have been affected by the Center listening test because

regardless ofthe Centertest, they had studied listening fbr independent listening tests anyway,

which are required by some universities. However, these results were net convincing since the

authors compared only two years oftest scores.

Most ofthe other washback studies used questionnaires to investigate students' attitude

toward and motivation for studying listening. Oguri (2009) revealed that students with high

listening skills recognized the importance ofthe Center listening test more than those with lowlistening skills. In addition, many students want to study listening earlier instead ofjust preparingfbr the Center listening test in their third year ofhigh school. Saida (2009) reported thatapproximately ene third ofthe freshmen at her university felt the Center listening test had a

favorable influence on their study method and their attitude toward English leaming. However,

negative effects were also raised by particularly lower proficiency students, who were anxious

about the listening test and felt overwhelmed by preparing fbr it, Furthermore, the listening test

has also infiuenced the high school curriculum. Researchers (Oguri, 2009; Saida, 2009; Sugino &'Ibkuda,

2008) have reported that after the listening section was introduced on the NCZ more

listening instruction has been included at many schools.

Overall, more positive than negative washback ofthe test has been reported for students'

listening skills and motivation, the school cuniculum, and the teachers, However, no studies havc

compared fbr a longer period the listening skills of students who entered university before and

33




after the Center listening test was introduced. Thus. to fill this research gap, we conducted txN'o

studies to answer the fbllowing research questions (RQs):

RQ1.

RQ2.RQ3.

Are there differences in students' listening skills befbre and afier the Center listeningtest was introduced?

Has the Center listening test influenced students in difll:rent majors differently?

What are the students' attitudes toward and motivation for studying listening after the

Center listening test was introduced?

2. Study 1

rlb

answer RQ1 and RQ2, Study 1 was conducted to investigate whether students' listening

skills have improved ever the past 1 O years (2002-201 1 ) including the years before arid after the

Center listening test was introduced,

2.1 Method

Pbrticipants. Participants were freshmen at a Japanese national university who took the

English placement test in April just after they entered the university. Approximately 1 ,600

freshnen each year from 2002 through 2011 totaled 15,727 students. Those who had entered the

university in Apri1 2006 or later had taken the Center listening test and the placement test, so that

it is possible to examine the change in listening skills by comparing the students who experienced

the Center listening test and those who did not. Thus, these 15,727 students were analyzed forRQl.

Among them, students from two humanities (Hl , n =

959; and H2, n =

1 201 ) and two

science (S1, n = 632; and S2, n

= 1577) departments were selected fbr RQ2. Dept S1 fbilowed by

Dept H1 requires students to gain higher proficiency in English in admissions than the ether

departments. Furthermore, in Dept H1 , some ofthose who specialized in language or cultures

were assumed to have more interest in gaining good comrriand ofspoken and written English. In

contrast, in admitting students, Dept H2 emphasizes skills and knowledge related to specialized

areas other than English, In total, 4,369 students' listening scores were analyzed fbr RQ2.

MateriatandAnaCFtsis. The participants' placement test scores from 2002 to 201 1 were used.

Similar to the TOEFL-ITR the placement test consists ofthree sections: listening (approximately23 short and long dialogue er monologue items), reading (1e vocabulary and about 13 reading

comprehension items), and grammar (about 1O grammar and 1O error identification items)

sections. The student scores spread approximately frem 240 to 700 in all the sections ofthe test,

The correlation between the placement test and the actual TOEFL-ITP was reported at .79 in a

validation study conducted in 2005, which was suthciently high to show concurrent validity. The

34



JapanSociety ofEnglishLanguage Education

test scores across years were comparable even though serne items used on the placement test in

each year difiered. Most ofthe items used in the tests were dra"n from an item bank in which al1

the items had been calibrated on the same scale by using the Rasch model.

First, to generalize the result ofthe change in students' listening scores over 1 O years, their

listening scores were compared with their scores in other skills each year. A two-way mixed

analysis ofvariance (ANONA) with a between-subjects factor, 1lear [2002- 20 1 1 ], and a

within-subjects factor, Skill [Listening, Reading, Grammar], was conducted. Next, another

two-way mixed ANOMK with a between-subj ects factor, Year [2002- 201 1], and a within-sub.jects

factor, Major [Depts }Il, H2, S1 , and S2], was used to examine whether students in diffl]rent

majors were infiuenced by the Center listening test in a different degree.

2.2 Results and Discussion

IVaeement 12zst Sboms Over the feat:s. "lable

1 and Figure 1 show the placement test scores

for the three skills fbr 1O years. One noticeable phenomenon is the trend in the listening scores,

which declined in 2004 and 2005.

[lable 1Mean and Standnrel Deviations ofthe 77zreeSections qf'the Placetnent 7lest.fi"om 2002 to 2011

Year2002200320042005200620072008200920102011

Sectien n 1408163l16S51642163815S3l5571616l5671537

ListeningM 511.92 509.54 496.00

SD 45.32 30.83 28.56500.64

508,79 509.32 514.66 513.96 511.55 512.87

3426 37,43 3e.50 34.81 34.71 33.89 34.73

M 516.00 50724 506.58 5192] 501.76 S09.14 511.46 510,51 513.15 515,89Reading SD 42.78 36.08 30.97 38.67 35.67 36.13 35.48 36.07 36.22 36,29

M 515.12 517.17 509.19 521.99 515.45 502.95 510.64 520,61 510.28 511.39Grammar SD 48.85 41,79 41.3S 42,04 44.22 4e.e2 40.00 42,69 42.60 41.57

Tota] ,l4 514.34 511.32 50392513.95 50g.67 507.14 512.26 515.03 511.66 513.38

35




5SO540

530

520

e 51og sooco

490480470

ttttttttttttttt' tttt

tttttttttt'

tttttwwt.

ttttwwtt-tttt-tt--...iLsts

N tN

--x

-L.-.-...---

r>f= .---".t-t--t.mtt

Nr"lvttt.......tt --tttttttttttt.t..........-.-

ttttttttt

ttnvtmettttttttttttttt.ttttttttttttt.

rr.....:....----=--------rr--11TT.tttttL'-w----･/

+Listening

.-.Reading

- -Grammar

2002 2003 2004 2005 2006 2007 2008 2009 2010 2011

Year

lilgttre 1. Placement test scores for the three skills frorn 2002 to 20]1.

The two-way ANOM`X showed a significant interaction between Yeair and Skill, fl1 1 7,08,29970.37) = 49,03,p < .OO1, n,2 =i .027, and the significant main effects ofYear, I7<9, 15794) =

25.1 7,p < .OO1, n,2 = .O14, and Skill, fi<1,90, 29970.37) == 85.79,p < .OO1, n,2= ,005, The result of

the fo11owing multiple comparisons revealed that the 2004 and 2005 listening scores were

significantly lower than the scores of the other ski11s in the same years (P < ,OO1 ). Iri addition,diffbrent from the two other ski11s, the 2004 and 2005 listening scores were significantly lower

than the scores in 2006 and later years Cp < .OO1).

However, it is dithcult to conclude that the improvement in the 2006 listening scores and the

fbllowing years was due to the influence ofthe Center listening test, since the 2002 and 2003

scores were as high as the 2008 scores. Apossible cause ofthe decline in 2004 and 2005 is the

negative influence ofa relaxed educational policB oryutori kyoiku, implemented in 2002 for

junior high schools and in 2003 for senior high schools, which aimed at giving students more free

time (MEX[iL n.d.), Under this policy, no classes are held on Saturdays, and the number ofEnglish

class hours has been decreased at junior and senior high schools. Students who entered a

university in 2004 or 2005 are assumed to have experienced the relaxed education since they fe11

in the transition period for implementing this policy. More specifically, manyjunior high schools

began to adopt this policy in 2000, a year after the MEXT announced it, when the 2004 university

freshnen were in their 3rd year ofjunior high school and the 2005 freshnen were in their 2"d year.This means that a large part oftheir six-year secondary education might have been affected by the

lenient educational policy, though we cannot go beyond speculation on this point in this study,

Listening Scores bj, Duterent Departments. Next, te examine whether the Center listening

test has influenced students in different majors differently, the placement listening scores were

36




classified according to the students' departments (i,e,, H1, H2, S1, and S2), The mean listening

scorcs fbr the four departments in each year are shown in [Ilable 2 and Figure 2, ilmong them,

Dept S 1 was outstandingly high, and the score changes were the targest. In particular, the 2002

scores were the highest and the 2004 and 20e5 scores plummeted. Similarly, Dept H2 showed the

highest scores in 2002 and 2003, but they decreased in the fbllowing two years. Therefbre, the

overall tendency ofthe relatively high scores in 2002 and 2003 and lower sceres in 2004 and 2005

as seen in Figure 1 was partly caused by Dept S 1 and Dept H2. However, it is hard to figure out

why these two departments in particular were more severely affected by the yutori education, ifit

was the primary cause.

'Iable 2Listening

S2:oTesfor the thur Departmenty,from 2002 to 2011

Year 2002200320042eos200620072008200920102011

Deptn39939240039345545]46348I473399

HliV 510,79 512.09 501.95 506,90 5]4,96 S12.05 521.18 522.33 516.S4 516.65

SD 36,25 30.00 23.74 34.91 33.61 26.76 30.73 27.63 30.30 31.04

H2 ,W 5[6,63 509.S8 494.41 494,72 503.56 S07,03 S04.33 506,5I 499,84 504.38

SZ) 54.00 29.03 27.57 37,35 30,75 27,21 29.47 29.76 33.47 26.51

Sl M 555.42 536.2 514.68 52121 536,S8

SD 52.37 34.06 25.59 31.88 41.30530.7

542.33 546.03 530.86 552.59

31.76 3320 35.33 33.89 37.69

S2 M 50626 503.63 497.95 501.78 506,14 501.18 504.03 503,l 503.63 '509.54

SD 35,Il 26.08 2S.99 32.28 36.18 33,06 33.2t 29,88 28.67 32.57

Total M 5]7.32 512.13 500.89 504.90 5]1.03 50922 512.50 513,92 5e8.99 515,53

S60ssoS40

K･--si'--

-

,... rx"'.tt

tt ttttttttt It trttt

'xN-

s3o --------x----------!-- ---N]Li

""mmrmrm.."...mm.rrxm.r+

...... -A.".-K 1 ･x ･･x, xl

y S208m

SIOsoo490

X./-'- --"-'---t"'-'"'-M''-

1.x'''NI'J']'i-..-.t'

"---"

-.=1.r,,:..S....T..ve.,..--t---.--N-."...--'x]N-A- ."-'h.-.-'--.....-Li-

' .t' . -s J)'-"--rt

480/470

'TTT-"r""-r7'--T""m'T'-"'t'-----t- -- ･/･- t- -t-- /

2002 2003 2004 200S 200S 2007 2008 2009 2010 2011

Year

-.- Dept Hl

----DeptH2

-iL-DeptSl

->(- DeptS2

Eigure 2. Listening placement scores fbr fbur departments from 2002 to 201 1 ,

37




The results ofthe two-wa>r (Year by Major) ANOVA fbllowed by multiple comparisons

revealed an interesting phenomenon. 'I'hat

is, Dept H1 did not significantly outperform Depts S2

and H2 from 2002 to 2005, when the Center listening test had not been introduced yet, but has

resulted significarrtly higher scores than Dept H2 since 2e06 (except 2007), and higher scores than

Dept S2 since 2007 (except 20 1 1). Furthermore, only Dept Hl showed a significant improvement

frorn 2002 to 2009, with a much smaller decline in 2004. This tendency may be due to the nature

ofDept H1 , in which some students majoring in languagcs or cultures may have been moremotivated to study aural aspects of English. Therefbre, it seems that the Center listening test had a

stronger impact on those who felt they needed higher scores in their entry to the university and

those who were more interested in communicative English.

3. Study 2

In Study 2, a questionnaire survey was conducted to investigate students' affective aspects

fbr RQ3.

3.1 Method

Ptirticipants. Out of93 university students who answered the questionnaire in 201 1 , the data

of87 students who had taken the Center listening test were analyzed (see Q1 in [Ilable 3), The

students' majors were quite diverse, such as Humanities, Comparative Culture, International

Studies, Arts and Design, Social Sciences, Information Sciences, Knowledge and Library Sciences,

Chemistry, Health and Physical Education, and Medical Sciences. The levels oftheir English

proficiency were also diverse.

MateriaL The questionnaire designed fbr this study (see fable 3) asked about: if students

had taken the Center listening test (Q1), their perception of the dienculty of the test (Q2), their

preparation fbr the test and its effect (Q3 to Q9), and the pros and cons ofthe test (Q1e to Q13).

3.2 Results and Diseussion

The results fbr most'ofthe questiens on the questionnaire are shown in rfable

3, Additionally,the results for the rest ofthe questions are surnmarized in th¢ contingency tables (fables 4, 5, and

6), to show the relationships between the perceived dienculty ofthe test and test preparation (i,e,,Q2 and Q3), between the frequency oftest preparation and the effectiveness of the test (Q6 aridQ I O), and between the hours of study and the effectiveness ofthe test preparation (Q7 and Q1 O).

38




[lable 3guestionnaire

Resbllts

Questions Count %

Q] , Have you ever taken the Center ristening test?

O yes@ No

876946

Q2. What do you think ofthe ditheulty level ofthe Center listening test? (See fable 3)

Q3. Did you prepare for the Center 1istening test? (See rlable

4)

Q4, (For those who answered "No"

to Q3), why didn't you study for the Center listening test?a

(D BecauseIfelt[azy.

@ Because I thought studying would not help.

@ Because I did not know how to study for thetest.

@ BecauseIdidnothavetime.

@ BecauseIwas confident thatJwould succeed on the test without studying,

@ Because they did not teach listening at school.

(IZ) Otherreasons

52234e22811Il1722o}1

Q5. (For those who answered "Noi'

to Q3), what were your results on the test?

(D I did not do as well as I had expected.

@ Idid not do ]ess well thanIhad expected,

@ I did as wel] as I had expected,

(D IdidbetterthanIhadexpected,

3191 227647

Q6. How often did you prepare fbr the test? (See Tlable 5)

Q7. How longdid you study in astudy session? (See fable 6)

Q8. Where did you study for the test? Give the percentage,

(D Athighschool( %)

@ Atcramschool( 9i6)

@ Athome ( %) 'fota1:]O09k

61831

Q9, Which textbeoks did you use to study for t]ie test?a

(D Listening test preparation textbook on the market

@ Listeningtextbookenthernarket

@ ListeningtextbookentheWeb

(D 1lextbook used at cram school and preparatory schoo[

@ rfextbook

used at high school

@ Others

619312406477210335

Q] O, Do you think preparing for the test helped you succeed on tlie test? (See [lables 5 and 6)

Q1 1 . (For everyone) Do yeu agree with the introduction of the Center listening test?

O Agree

@ Neitheragreenordisagree

@ Disagree

66l79 721810

39

NII-Electronic Mbrary




Questions Count %

Q12. For those who answercd "Agree"

for Q1 1, what isfare your reason(s)?"

(D I can improve my listening skil[s.

(2) Icanimprovemycommunicationskills,

@ Reading and grammar tests are net enough to measure English ability,

@ r.istening is thc most essentia] skill for undersrandjng people's speech.

@ Others

29103924427937234

Q]3, (For those who answered `dDisagree"

forQl 1), what is!are your reason(s)?a

(D lt is no use taking the listening section ofthe NCT as it is toe easy.

O It is not necessary as each university has an independent listeningtest.

@ It is time-consuming to explain how to use the listening device at the test.

0 It is possible to examine Engljsh ability without listenjng tests,

@ I spent Iesstime studying reading and grammar than on listening,

@ Others

1l2115 99189945

?Vbte, eMu]tip}e-response

question.

PerceivedD(tiieudy ofthe 7lest and ity ]Rreparation, First, the perceived test difficulty

(see Q2 in Table 4) was examined. Fifty-six percent ofthe students felt that the Centerlistening test was

"Moderate,"

and 30% felt it "Easy"

or "Very

easy," which implies that thetest was not particularly dithcult fbr most students. Even so, as shown in Q3 (Table 4), asmany as 899,6 prepared fbr it.

The analysis of the chi-square test fbr independence did not detect an association

between the perceived dithculty of the test (Q2) and student preparation for the test (Q3), x2(cij'== 4, N= 87) =1.73,p

=

.785, Q =

.14. This implies that regardless ofthe difficulty ofthe

test, most students prepared as the test was required for adrnission to the university.

The remaining 1 1% of the students did not study fbr the test mainly because the test wasrelatively easy for them (Q4 in [lable 3). In fact, the majority (i.e., 64 + 7 == 71%) reported that

they did well on the test (see Q5 in fable 3).

rfable 4Relationship

between 7lest Pnqparation and PereeivedDCfficulty ofthe fest

Q2. What do you think ofthe diMculty level ofthe Center listening

test?

very diMcultDicacultModerate EasyVery easy

Total[%]

Q3YesNo1 [1.1]o [o.o]

10 [ll.5]

1 [1,]]44

[50.6] 5 [5,7]20

[23.0] 3 [3.4]

2 [2.3]1 [1.1]77

[88.5]1O[11.5]

Total 1 [1.1] 11 []2.6]49 [56,3]23 [26.4]3 [3.4]87 [1OO,O]Note. Q3

== Did you prepare for the Center ]istening test?

40

NII-Electronic




Amount of 1lest Rreparation and its Iif/llact To get a good score on the test, most of the

students studied listening once or at most three times a week (see Q6 in [fable 5), and the

length of study in one session was around 1 hour or less (see Q7 in Table 6). This amount of

study may not be large but fair regarding the relative easiness ef the test and smaller point

distribution compared with the Center written test (i.e., 50 vs. 200 points, respectively),

Table 5Relationship

between F>"equenay qf'7Zist Preparation and its Efaci

Q1O, Do you think preparing for the test helped you succeed on it?

Strongly

agreeAgree Disagree

Do not

remember

Total[%]

Q6

Every day

Oncef2 or3 days

Oncelweek

Several times

6 [7,8]16 [20.8]

12 [15,6]4 [5.2]

5 [6.5]13 [16,9]

10 [13.0]6 I7,8]

o

o4

[52] o

o

o

o1

El.3]

11 [14.3]29 [37.7]26 [33.8]11 [14,3]

Total38 [49,4]34 [44,2] 4 [52] 1 [1.3] 77 []oe,o]Note, Q6

; How often did you prepare fbr the test?

'Ibble 6Relationship

between the Amount qfSZudy and its E29Zict

Q1O. Do you think preparing for the test helped you succeed on it?

Strongly

agreeAgree Disagree

Do not

remember

Total[%]

Q7

2 or 3 hours

1 to 2 hours

Less than 1 hour

1 [1,3]18 [23,4]19 [24.7]

1[1,3]9[11.7]

24 [3 1,2]

o1

[1.3]3 [3.9]

o

o1

[] .3]

2 [2,6]28 [36.4]47 [6].O]

Total38 [49.4]34 [44.2] 4[52] 1 [1,3] 77 [100.0]Note, Q7 = How long did you study in a study session?

Taking a closer look at [table 5, all students who studied every day or once every few

days (Q6) admitted a preparation effect (Q1O), whereas five students who prepared only once

a week or severai times denied a preparation effect, In [lable 6, seemingly, students who

studied more in one session (Q7) also tended to feel the effect (QlO), The chi-square test

revealed a marginally significant tendency regarding the relationship between the frequency

of the preparation (Q6) and its effect (Q1O), x2 (`ij'= 9, Ai = 77) = 15,OO,p == .091, p = .44.

However, the latter association (i.e., between Q7 and QlO) was not significant, x2 (clf'= 6, N =

77) == 4.48, p

=.613,

ap =

.24.

41




Thus, we fbund that students who studied listening more frequently tended to feel the

effect of their studying and that the frequency of the study may be more important than the

amount oftime spent studying in one session for the Center listening test. However, since the

frequency effect was marginal, the Center listening test was not as successfu1 in motivating

students to study listening.

IhLfluence on lbachers. Concerning where the students preparcd fbr the test (Q8 in'I}Lble

3), 61 9/6 ofthe students prepared at school, and most students used either prcp books fbrthe test or listening exercise books provided by their high schools (see Q9). Thus, the Center

listening test has also exerted influence on school. curriculum and teachers as reported in other

studies mentioned earlier (e.g., Oguri, 2009).

Rros and Cons ofthe introduction ofthe Center Llstening 7lesL Last, whether students

agreed with including the listening test on the NCT was examined with Q1 1, Ql2, and Q13(see Table 3). The majority (729t6) agreed mainly for the fbllowing three reasons:

"Reading

and grarnmar tests are not enough to measure English ability (37%)," "I

can improve myIistening skills (27%)," and ,"Listening is the most essential skiir fbr understanding people'sspeech (239x6)." However, the reason

"I can improve my communication skills" had a

relatively small response rate (99'6), This may indicate that many students realized that

training only receptive ski11s such as listening and reading is not enough to be able to use

English fbr communication. In fact, some wrote such opinions as `Cthe

listening section needs

revising in order to measure communication ability more appropriately" in Others in Q12, In centrast, only nine students (1O%) disagreed with including the listening test on the

NCT. As in Q13, the reasons varied (e.g., C`It

is not necessary as each university has an

indePendent listening test" or "It

is time-consuming to explain how to use the listening device

at the test"), yet fbur out of nine students gave concerns about mechanical problems as a

reason.

4. General Discussion and Conclusion

The NCT is a high-stakes entrance examination implemented in a large scale in Japan.Assuming that the NCT has a great impact on applicants and English education in Japan, the

MEXT decided to add an English listening test to the NCT in 2006 to encourage students to

improve their communication skills. This paper investigated the washback ofthe Center

listening test by exploring three research questions, For RQ1 (Are there diffbrences in students' listening skills befbre and after the Center

listening test was introduced?), the result fbr Study 1 was not conclusive since the

improvement in listening skills was not clear-cut. As seen in Figure 1, an increase in

42




freshmen's placement test scores in 2006 and later years was observed, but the 2002 and 2003

scores, the years befbre the Center listening test was introduced, were also as high as the 2008

scores,

This inconclusive result seems to be caused by factors other than the Center listening

test, For one thing, the decline in the placement listening scores in 2004 and 2005 (see Figure1) might be due to the relaxed education policy that had been implemented at manyjunior and

senior high schools since 2000, Most 2004 and 2005 university freshmen were those who first

had taken most oftheir secondary education under this policy. As a consequence ofthe

reduction in the number of classes, class time for subject areas not tested on university

examinations was decreased at many high schools. Listening might have been reduced during

these two years. Later, this relaxed policy was severely criticized due to the negative effect on

students' scholastic ability. In other words, the listening levels of the 2002 and 2003 students

were as high as the level ofthe 2006 students because of the old educatienal policy before

relaxed education was implemented. Thus, such educational policies or other social

environmentai factors might be interwoven with students' listening skills and make

interpreting the results dithcult.

In regard to RQ2 (Has the Center listening test influenced students in different majors

differently?), analyzing students' scores based on their maj ors showed an interesting result.

The scQres for Dept H1, where some students majored in languages er cultures, have been

significantly higher than the two other departments since 2006 (see Figure 2), In addition, the

decline in the 2004 and 2005 scores was the smallest among the fbur departments. These

favorable trends might indicate that students in Dept H1 were interested in fbreign languages

and cultures and thus were more motivated to study listening than students in other

departments, In this regard, the Center listening test rnay influence students in different

ma.iors to a difTerent degree, which agrees with 'Ilakeuchi

and Kozuka's (201O) finding,

Regarding RQ3 (What are the students' attitudes toward and motivation fbr studying

listening after the Center listening test was introduced?), the results of the questionnaire

revealed two positive attitudes toward the test. First, 89% ofthe students studied for the

Center listening test, and there was a slight tendency that students who studied more

frequently felt the preparation effect more strongly, However, considering the relatively small

or fair arnount ofpreparation (i.e,, a few times per week fbr less than an heur per session), this

intended washback was weak, Hence, as students study depending on the content and

difficulty ofthe test, adding more challenging tasks to the test is neeessary to bring about a

stronger washback effect on students.

Second, most ofthe students favored the introduction ofthe test because they recognizc

the impertance oflistening skills net only as a means of passing exams but also of

communication. Therefbre, teachers are responsible for teaching listening to fu1fi11 the

students' need. The questionnaire revealed that high school teachers at least tried to have

43




students study listening at school (Q8), providing them with exercise or prep books (Q9),However, according to the Listening Test Validation Group (2012) report, increasing theallocation of listening time in class, in turn, decreased the time available for other skills such

as speaking and writing in some schools, since the total amount ofclass time did not change.

Thus, the Center listening test could influence teaching and school curriculum not oniy

positively but also negatively.

Although we tried to compare on a large scale students' listening skills befbre and after

the Center listening test was introduced, the study has some seTious limitations. Foremost, like

other studies efthis type, establishing a causal Iink between the improvement in learners'

listening skills and the influence ofthe Center Iistening test is dithcult because other factorsmay interact over the years. By the same token, an exact comparison is hard regarding the

arnount of listening study before and after the Center listening test was introduced, Moreover,how much we can generalize the results is unciear since Studies 1 and 2 targeted only students

at a national university. If we include students with more diverse proficiency levels, the

relationship between the amount ofpreparation and its effect might be more clearly observed

For this reason, students at private universities should be investigated in future research.

Despite these limitations, we fbund that many students recognized the significance of

including the Center listening test on the NCT and feIt the effect oftheir listening study.

However, this relatively easy receptive test had limited washback on the actual improvementin listening skills and is far from suencient jfthe MEXT's goal is to enhance students'

communication skills. Considering that the scale of washback ofa test depends on itsdifliculty and what it measures, we first need to define what aspects ofcommunication skills

and to what extent we want students to acquire those skills, For example, if thecommunication ability entails an aspect ofproduction skills such as writing and speaking,

then the NCT needs to include items that measure those aspects and needs adjustment in thedifficulty ofthe items depending on the goals we want students to achieve, Without thosecarefuI test specifications, the test cannot bring about the expected washback.

Acknowledgment

We are gratefu1 to the chief and stafl'mernbers ofthe Foreign Language Center at theUniversity of'1'sukuba for allowing us to use all the placement data fbr Study l. Witheut them,

the studies would not have been possible. We would also like to thank anonymous reviewers

for their valuable comments.

References

Hughes,A. (2003). 7lestingfor language teachers (2"d ed.), Cambridge: Cambridge University

44




Press.

Listening fllest

Validation Group. (201 2). Listening test noJ'isshikekka ya seikatou wo

kennshou shi, sono haizen wo haharutame no chosakendyu ni kansuru hokokusho [A repert on the research fbr the Center listening test through the examination of the results

of the test]. Tokyo: National Center for University Entrance Examinations.

Messick, S, (l996). Validity and washback in language testing. Langorage Zesiing 13,

241-256.

MEXT (Ministry ofEducation, Culture, Sports, Science and "fechnology).

(2003). T7ieAction

Ptan to cultivate 7bpanese with English language abilities. Retrieved from

http:!lwarp,ndl.go.jp!infb:ndljplpidf286794!www.mext.go.jpformenulhoudoul15!03f030

33102,pdf

National Center For University Entrance Examinations (20l2). 7bkushokuaru `laigaku

izyuushi no tameni: He isei 25 nenndo IVational Center lixamination [For distinctive

university entrance examination: The year 2013 National Center Examination].

Retrievedfromhttp:!!www,dnc.ac,jplmoduleslcenterTexamlcontentOO1O.html

Oguri, Yl (2009). Washback effects from the listening section ofthe National Center Test:

Suggestions fbr high schools classes. S7iiga Kbnritsu Daigaku Kbkusai ?<youiku Center

Kenkyuu Klyou, 14, 35-44.

Saida, C, (2009). Washback effect ofthe Nationa} Center listening test on English learning

and curricular innovation at Ibaraki University. ,L4 CETZk7nkokutaikai Ktyou, 48,

195-196.

Sugino, K,, & Tokuda, M, (2008). Center test no iistening doupzyuu to koukoueigokyouiku:

iVbgunoken Nbzawakitakoutougakkou no rei o chuusin to site [Introduction ofthe Nationai

Center 'Ilest

Listening and high school education: Case study ofNozawa-kita high

school in Nagano], Jinbunkogakuronshuu Bunka Communicationgakkahen, 42, 95-1l 1.

[Iakeuchi, T., & Kozuka, Y (201O). An analysis ofTOEIC scores before and afier the

introduction oflistening test o the National Center fbr university entrance examinations:

A case study ofinternational cultural studies majors, English majors and early childheod

majors. Aichi 1<youiku Daigaku K),ouiku Jlr'ssen Sougou Center Ktyou, 13, 127-13l.

Tanaka, N., & Sage, K. (2006). How authentic is the English listening section ofthe NCT fbr

the EFL context in Japan?: Development based on the TOEFL iBT English listening

section. Mngenbunka Ronsou, 9, 113-129.

Watanabe, Y (2004), Methodology in washback studies. In L. Cheng, & Y Watanabe (with Curtis, A), (Eds,), PVlrshbaek in language testing (pp, 19-36), Mahwah, NJ: Lawrence

ErlbaumAssociates.

Yanagawa, K. (2012), A partial validLztion ofthe contextual vaiiddy ofthe Centre listening

test in .llipan (Unpublished docteral dissertation). University ofBedfordshire,

Bedfbrdshire.

45

washbackof the center listening test

Documents