alderson diagnosis c.1

8/4/2019 Alderson Diagnosis C.1

http://slidepdf.com/reader/full/alderson-diagnosis-c1 1/11

DiagnosingForeign Language Proficiency

T he In ter face be tween Learning an d Assessment

J. Charles Alderson

continuum



Cont inuum

T h e T o w e r B u i l d i n g , 11 Y o r k R o a d , L o n d o n S E 1 7 N X

15 East 26th St ree t , N ew Y o r k , N Y 10010

© J. C h a r l e s A l d e r s o n 2 0 0 5

A ll r igh ts reserved. N o pa r t o f t h i s pub l i ca t i on ma y be r e p r o d u c e d o r t r a n s m i t t e d in

a n y form or by any means, e lec t ronic o r mecha n i ca l , i nc lud ing pho tocopy ing , r ecord ing ,

o r a n y i n f o r m a t i o n s t o r a g e o r retr ieval system, without pr io r pe rm iss ion in w r i t i n g from

th e pub l i she r s .

First pub l i shed 2005

Bri t ish Library Catalog uing- in-Pub l icat ion Data

A c a t a l o g u e r e c o r d fo r t h i s b o o k is a v a i l a b l e from th e Br i t i sh L ibrary.

ISBN: 0-8264-8503-0 ( ha rdback )

L ibrary o f C o n g r e s s C a t a l o g i n g - i n - P u b l ic a t i o n D a ta

A ca ta log r ecord fo r t h i s b o o k is a v a i l a b l e from th e Library o f C o n g r e s s .

Typeset by A a r o n t y p e Limi ted, Ea ston, Br is to l

Pr in ted a n d b o u n d in Grea t Br i t a i n b y



Chapter 1: Diagnosing foreign language

proficiency: a teaching/testinginterface

It is a commonplace to claim the importance of assessment in language

teaching and learning. Teachers need to know what learners already

know, what they have learned in the course of instruction over a longer

or shorter period and where their strengths and weaknesses are, so that

they can plan their instruction appropriately, guide learners on where

they need to improve and give feedback to learners. Unfortunately,

when teachers give students tests, it usually takes a few days, if not

longer, for learners to receive their results, and so the feedback lacks

immediate relevance. Moreover, tests made by teachers are often of

poor quality, and the insight they could offer into achievement, pro-

gress, strengths and weaknesses is usually very limited indeed. In the

case of national and international examinations and proficiency tests,

feedback may be delayed by several months, and the results are irrele-vant to learning needs, with little or no information to help learners

understand what they need to do in order to improve.

The type of test that comes closest to being central to learning is the

diagnostic test.

What is adiagnostic test?

Language testing handbooks frequently distinguish diagnostic tests fromplacement, progress, achievement and proficiency tests. The ALTE

multilingual glossary defines a diagnostic test thus: 'A test which is used

for the purpose of discovering a learner's specific strengths or weaknesses. The results

m ay be used in making decisions on future training, learning or teaching1(ALTE,

1998). However, diagnostic tests are frequently confused with place-

ment tests. The same multilingual glossary defines 'placement test' as

follows: 'A test administered in order to place students in a group or class at a level

appropriate to their degree of knowledge and ability' (op.cit.). According to this,

Introduction



both diagnostic and placement tests appear to be designed to identify

what a learner knows in order to decide on future teaching or learning.

Placement tests, after all, are intended to group learners in homoge-

neous groups in order to have a suitable basis for further teachingand learning.

The Davies et al. dictionary of language testing describes the use of

diagnostic tests as follows:

information obtained from such (diagnostic) tests is useful at the beginning of a

language course, fo r example, fo r placement purposes (assigning students to

appropriate classes), fo r selection (deciding which students to admit to a parti-

cular course), for planning of courses of instruction or fo r identifying areas where

remedial instruction is necessary. (Davies etal., 1999)

In other words, diagnostic tests are used for placement purposes — and

thus appear to be identical to placement tests. Indeed, Davies et al. con-

tinue: 'It is common fo r educational institutions (e.g., universities) to administer

diagnostic language tests to incoming students, in order to establish whether or not

they need or would benefit from support in the language of instruction used' (op.cit.}.

In short, there appears to be no distinction between a diagnostic test and

a placement test, at least according to these authors.It is arguable whether it is indeed the case that 'it is common for

universities to administer diagnostic language tests'. Much more com-

mon is the administration of some form of post-admissions placement

test (for an account of such a placement test and the problems of validat-

ing it, see Wall etal., 1996). I have never heard any university claim that

what it is doing is diagnosing students' strengths and weaknesses with

its placement procedures, since typically what is decided is who needs

some form of in-sessional or pre-sessional assistance, and who does not.

At most, what is decided is whether a student should take a readingclass or a grammar class, a writing class or a listening class or, in the

case of US institutions, whether a student should take only a fractional

load of academic courses rather than a full academic load. This is rarely

known as diagnosis.

Bachman (1990) asserts that

virtually any language test has some potentialfo r providing diagnostic informa-

tion ...Information from language tests can be used for diagnosing students'

areas of strength and weakness in order to determine a ppropriate types and levels

of teaching and learning activities...A placement test can be regarded as a broad-

band diagnostic test in that it distinguishes relatively weak from relatively strong

students so that they can be provided learning activities at the appro priate level.

Similarly, a readiness test differentiates students who are ready for instruction

from those who are not.A detailed analysis of student responses to the questions

on placement and readiness tests can also provide more specific information about

particular areas of weakness, (p. 60)

D I A G N O S I N G F O R E I G N L A N G U A G E P R O F I C I E N C Y 5



Alderson etal. (1995) define diagnostic tests similarly:

Diagnostic tests seek to identify those areas in which a s tudent needs fur ther help.

These tests can b e air ly general, and s h o w , for exam ple, whether a student needspart icular help with one of the f o u r main language ski l ls; or they can be more

specific, seeking perhaps to identify weaknesses in a student 's use of g r a m m a r .

These more s p e c i f i c diagnostic tests are not easy to design since it is d i f f i c u l t to

diagnose precisely strengths and w eakn esses in the complexities of language ab i l -i ty . For this reason there are very few purely diagnostic tests . However, achieve-

men t and proficiency tests are themselves requently used, albeit unsystem atically ,

fo r diagnost ic purposes , (p. 12)

Thus it would appear that even achievement and proficiency tests can

perform diagnostic funct ions . Indeed, Davies e t al. claim that 'relatively

few tests a re designed specifically fo r diagnostic purposes ' . They explain this asfollows: 'It is d i f f i c u l t and t ime-consuming to construct a test which provides detailed

diagnostic in formation ' .Yet it is somewhat odd to say that few diagnostic

tests exist because they are time-consuming to construct — proficienc

tests are also difficult and time-consuming to construct, but many exist.

It would appear that we have a problem here: diagnosis is useful, most

language tests can be used for diagnosis in some sense, it is commonfor universities to administer diagnostic tests, and yet diagnostic tests

are rare!

To summarize: there is considerable confusion in the literature

between placement tests and diagnostic tests. Furthermore, it is fre-

quently claimed that achievement and proficiency tests can be used for

diagnostic purposes. Yet it is said that diagnostic tests are rare, and are

very difficult to construct. Moreover, there are f r equen t referencesin the

language testing research literature to, and investigationsof, proficiency

tests, achievement tests, placement tests and even aptitude tests butdiagnostic tests are very rarely referred to or investigated.

The reason for this is not immediately apparent. Clearly it would be

useful to have tests which enable teachers and learners to diagnose their

strengths and weaknesses. However, linguistic diagnosis might not be as

well developed as is desirable because of the lack of attention paid to the

subjec t . High-stakes tests that directly affect people's lives, like univer-

sity entrance tests or proficiency tests for citizenship or employment,

have more obvious potentially negative consequences, and so the qual-

ity of such instruments is of paramount importance. Inadequate diagno-

sis in the context of language education is unlikely to be life-threatening,

unlike inadequate medical diagnosis. And so much less attention has

been devoted to ensuring the validity and reliability of diagnostic tests

in the foreign language field.

The aim of this book is to describe and discuss the nature of diagnostic

tests, in order to understand better how they might be constructed and

validated.

D I A G N O S I N G F O R E I G N L A N G U A G E P R O F I C I E N C Y



Whatmight diagnostic testscontain?

Bachman (1990) o f f e r s the f o l lowing thoughts o n what is usually conside r e d to be suitable content f o r diagnostic tes ts :

When we speak of a diagnostic test...we aregenerally referring to a test that has

been designed and developed specifically to provide detailed information about th e

specific content domains that are covered in a given program or that are part of a

general theory of language proficiency. Thus, diagnostic tests may be either theory

or syllabus-based, (p. 60)

Acco rd ing to Bachman, a diagnostic test might contain important

aspects o f the content o f a specific programme o r , alternatively, itmight be based o n a specific theory o f language pro f ic iency. However,th e f o r m e r is usually regarded as an achievement test, and the latter as apro f ic iency test. W e appear t o be little nearer specifying the content o fdiagnos t ic tes ts per se.

M o u s s a v i ( 2 0 0 2 ) has quite a long section o n diagnostic testing, goingin to more detail about h o w achievement an d pro f ic iency tes ts can beu s ed f o r diagnosis, with ideas o n what might be suitable content:

If diagnostic testing is defined as providing feedback to teachers and students

regarding their strengths and weaknesses, then almost any test would be diagnos-

tic. Diagnostic tests are an essential part of individualized instruction pro-

grammes. In this case, the students take the test for each unit when they feel

ready fo r them. The tests help them to see whether or not they are ready to move

on to the next unit, assignment, or passage, and enable the teacher to develop a

cumulative grade or credit. The diagnostic test tries to answer the question: How

well have the students learnt thisparticular material? Since it relates to particular

elements in the course which havejust been taught,for example, 'type III condi-tional sentencewith if or' asking permission', theassessment will give immediate

feedback to the student. If his learning has been successful, the results will give a

considerable lift to the student's morale and he is likely to approach th e next learn-

ing tasks withfresh enthusiasm. The degree to which a test is diagnostic depends

not so much on the purpose of th e test, but on the way in which scores are analysed.

Let us consider TOEFL for a moment: TOEFL is usually considered a

PROFICIENCY TEST, and when its total score is considered by an admis-

sions o f f i c e r , it can quite rightly be so classified. However, if one looks at th e five

part-scores for reading comprehension, vocabulary, and so on, the test is serving a

diagnostic purpose in that information about an individual's particular strengths

and/or weaknesses is obtained. That is, we have specific information not on 'Eng-

lish', but on certain abilities or skills. As information from these tests can beused

for diagnosing students' areas of strength and weakness in order to determine

appropriate types and levels of teaching and learning activities, therefore virtually

any language test has somepotentialfor providing diagnostic information. Diag-

nostic tests are the reverse side of ACHIEVEMENT TESTS in the sense that




while the interest of the achievement test is in success, the interest in the diagnostic

test is in failure, what has gone wrong, in order to develop remedies.

Like Bachman, however, Moussavi has not clearly identified the differ-ence in content between diagnostic tests on the one hand, and achieve-

ment and proficiency tests on the other hand.

Bachman and Palmer (1996) have the following to add to this discus-

sion of diagnosis:

Diagnosis involves identifying specific areas of strength or weakness in language

ability so as to assign students to specific courses or learning activities. For

example, if a language program included three different courses, one ocused on the

editing of sentence level grammar and punctuation errors, a second focused on

revising the organisation of essays, and a thirdfocused on logic of argumentation

in writing, a teacher might use a test that included all these different language use

activities as a basis for deciding which course would be most appropriate fo r

students to take. (p. 98)

Thus, a diagnostic test would appear to be much more specific, focused

and related to particular (and different) language programmes. In con-

trast, their description of a placement test merely talks about assigningstudents to different 'levels' of a language course.

This apparent distinction is not, however, sustained throughout the

Bachman and Palmer volume. Later, they present an extended descrip-

tion of the development of a 'syllabus-based diagnostic achievement

test' for students in an ESP programme. In fact, this 'partially developed

example' is rather similar to other examples in the book, the main dif-

ference being that 'we will now control more tightly the exact structures that we

want the test takers to use'. Curiously, however, the specific purpose of the

test is to determine whether students have mastered specific course con-tent, and will provide diagnostic feedback to those who have not, as well

as to course designers and teachers, who can then tailor the course more

closely to the needs of the students. It is not clear how this is different

from an achievement test.

Detailed content specifications are given, focusing on prepositions,

participial modifiers, personal and relative pronouns, verb forms, infini-

tives, adverbs, quantifiers, auxiliary verb forms and coordinating and

subordinating conjunctions, as well as punctuation marks like comma,

semicolon, period and question mark. Authenticity is acknowledged to

be limited (might this be a feature ofdiagnostic tests?), the test method is

limited and it is suggested that 'there are no obvious affective barriers' that

would prevent students performing at their best. This might indeed be

a feature of diagnostic tests, since they are sometimes said to be low-

stakes. Unfortunately, although feedback to students, teachers and

course designers is said to be important, no indication is given as to

how the test results will be reported, However, for a diagnostic test

D I A G N O S I N G F O R E I G N L A N G U A G E P R O F I C I E N C Y



to be truly useful , profiles of performance are surely needed, and verydetailed information on the performance across the various componentsspecified in the content specifications is highly desirable.

Hughes (1989) considers that the diagnosis of linguistic strengths andweaknesses is very diff icul t :

// is not so easy to obtain a de ta i led ana l ys is of a student's c o m m a n d of g r a m m a -t ical s tructures, some thing wh ich wo uld tel l us , or exam ple , w he ther she or he had

ma stered the presen t perfect/past tense dist inction in English.In order to be sure of

th is , w e would need a num b er of ex a m p l e s of th e choice th e s tudent m ade be twee n

th e tw o structures in every d i f f e r e n t context which w e thought w a s significantly

d i f f e r e n t a nd important enough to warrant obta ining informat ion on. A single

example of each would not be enough, s ince a s tudent might give th e correct

response by c h a nc e . A s a resul t , a comprehensive diagnostic test of English gram-m a r would be v a s t (think of wha t would be invo l ved in testing th e modal v erbs ,

for ins tance) . The size o f such a tes t wo uld ma ke it impract ical to adminis ter in a

routine a s h i o n . For this reason , very few tes ts are constructed for purely d iagnos-

ti c purposes , a nd those tha t there are do not provide very deta i l ed informat ion.

The lack o f good diagn ostic tests is unfor tunate . They could be extremely useful

for individu a l ised instruction or sel f-instruction. Le a rners wo uld be show n wh ere

gaps exis t in the ir command of the l anguage , a nd could be directed to sources ofin format ion, exemplification a nd pract ice . Happi ly , th e ready avai labi l i ty o f rela-

t ively inexpens ive computer s with very large m em or ies m a y change th e situation.

Well-wri t ten computer program s wou ld ensure that the l earner spent no more time

than w a s absolutely necessary to obtain th e des ired informa t ion, a nd without th e

need for a test ad min istrato r. Te sts of this kind will still need a t r emendous

a m o u n t of work to produce. Whether or not they become genera l ly av a i lab l e willdepend on the will ingness of indiv iduals to write them a nd o f publishers to distri-bute t h em , (pp. 13-14)

Whilst repeating what Davies e t a l. say about placement, Hughes goesf u r the r to discuss why it might be diff icul t to design tests which couldprovide detailed diagnoses, and to suggest that computer-based testingmight offer a solution. (Interestingly, Hughes predicts the advent ofcomputer-based diagnostic testing, eight years before the DIALANGProject began.) He does not, however, address the issue of the construct:what exactly should a diagnostic test of grammar, say, actually contain?And one of reading, or speaking?

Interestingly, in the second edition (Hughes, 2003) there are more,albeit somewhat brief, references to diagnostic tests as follows:

Diagnost ic tes ts of g r a m m a r .. . will t end to be discrete-point, (p. 19)

// m a y also be thought worthwhile tes t ing lowe r lev el l is tening skills in a diagnos-

tic tes t , s ince problem s with these t end to pe rsist longer than they do in reading.

These might include:




T he usefulness (and indeed the feas ibi l i ty ) o f a general diagno s t ic tes t o f v o c abu-

lary i s n o t readily apparent. Asfar as placement tes ts are c o nc er ned ,w e w o u l d n o tnormal ly require , o r expec t , a part icular set o f lexical i tems to be a prerequisi te fo r

a par t icu lar language c lass . Al l we w o u l d be l o o k ing for is som e general indica-

tion of the adequacy of the s tudents ' vocabulary. T he l earning o f s p e c i f i c lexical

i tems in class will rarely depend o n pr ev io us k no w l edge o f other , s p e c i f i e d i t e m s .(p . 179)

It is encouraging for the f u tu re of diagnostic testing that Hughes hasbegun to address the topic in more depth than other authors. However,his views on the relative ease of diagnosing language use skills and thedifficulty of diagnosing grammatical and lexical knowledge remainmere suppositions. It is far from clear why i t should be possible to diag-nose strengths and weaknesses in reading or listening, for example. The

warning about the need to test grammatical structures in a comprehen-

sive range of contexts must surely, a p r io r i , also apply to a need to ascer-tain a learner's ability to process a whole range of different texts across a

wide variety of topics at varying levels of linguistic difficulty. Conver-sely, until we have developed tests of grammatical knowledge acrossthe range of contexts which Hughes asserts are essential, we will not

know whether it is indeed essential to include so much variation, orwhether many of the items/contexts are redundant. However specula-tive and uninformed by empirical evidence though these commentsmay be, they are helpful in constituting a set of hypotheses which can

be tested empirically. Thus can diagnostic tests be explored, and ourunderstanding enhanced.

In summary, the language testing literature offers very little guidanceon how diagnosis might appropriately be conducted, what content diag-nostic tests might have, what theoretical basis they might rest on, and

how their use might be validated. Diagnostic testing is virtually ignoredin the literature, and is rarely, if ever, problematized. Nevertheless, we

have seen some sporadic indications in the literature that diagnostic testsmight test aspects of the control of the linguistic elements, especially formorphosyntax and phonemes. What might be tested in detail in the

diagnosis of strengths and weaknesses in language u se skills is muchless clear.

Towardsacharacterizationofdiagnostic tests

Despite the lack of detailed discussion in the language testing literature,I suggest that it might indeed be possible to begin to identify some

D I A G N O S I N G F O R E I G N L A N G U A G E P R O F I C I E N C Y0

discr iminate between vowel phonemes

di sc r im inate be tw een c o ns o n an t ph o nem es

interpret in tonat ion pat terns (recogni t ion o f sarcasm, ques t ion s in dec larat ive

f o r m , etc.,interpretation o f sentence s t re s s ) . (p . 162)



fea tures of diagnosis that might dist inguish diagnostic tests f rom othe rtypes of test, even if other tes ts cont inue to be used for diagnostic pur-poses — or pseu do-diagnostic purposes like placem ent. I list be low a set

of hypothetical fea tures of diagno stic tests, as a synthesis of the l i te ra turereviewed above, which might guide fu r ther thinking about, design ofand research into diagnostic tests . It should be pointed out , however ,that some of these ' f ea tu res ' contradict o ther fea tures , and, m ore impor-tantly, that they const i tu te a potential agenda fo r research ra the r than aset of definitive s ta tements abou t wha t is necessary and possible.

1 Diagnostic tests are designed to identify s t rengths and weaknessesin a learner ' s kn ow ledge and use of langua ge.

2 Diagnostic tests are more likely to focus on weaknesses than onstrengths.

3 Diagn ostic tests sho uld lead to remedia t ion in f u r the r ins t ruct ion.4 Diagn ostic tests sho uld enab le a detailed analysis and re por t of

responses to i tems or tasks.5 Diagnostic tests thu s give detai led fee dba ck which can be acted

u p on .6 Diagnostic tests provide immediate resul ts , or resu l ts as little

delayed as possible af ter test- taking.7 Diagnostic tests are typically low-stakes or no-stakes.8 Because diagnostic tests are not high-stakes they can be expected to

involve l i t t le an xiety or other affective barr ie rs to op t imum per fo r -mance .

9 Diagnostic tests are based on conten t which has been covered ininstruct ion, or which will be covered short ly.

OR

10 Diagnostic tests are based on some theory of language develop-me nt , p referably a detai led theory rather than a global theory.

11 Thu s diagnostic tests need to be in fo rmed by SLA research, or mor ebro ad ly by applied linguistic the or y as well as resea rch.

12 Diagnostic tests are likely to be less 'authentic' than proficiency orother tests .

13 Diagnostic tests are more likely to be discrete-point than integra-tive, or mor e focused on specific elements than on global abilities.

14 Diagn ostic tests are m ore likely to focus on langua ge than on lan-guage skills.

15 Diagnostic tests are more likely to focus on ' low-level ' languageskills (like phonem e discr iminat ion in l istening tests) than higher-o rde r skills which are mo re integra ted.

16 Diagnostic tests of vocabulary knowledge and use are less likely tobe useful than diagnostic tests of grammat ica l knowedge and theability to use that knowledge in context .




17 Tests of detai led grammatical knowledge and use are difficult toconstruct because of the need to cover a ra nge of contexts and tomeet the demands of reliability.

18 Diagno stic tests of language use skills like speaking, listening, re ad-ing and w riting are (said to be) easier to co ns tru ct than tests of lan-guage knowledge and use. Therefo re the results of such tests may beinterpretable for remediat ion or instruct ion.

1 9 Diagnostic testing is likely to be enhanced by being computer-based.

To summarize and con clude this chap ter, I have argued that the litera-tu re on diagnostic testing in second and foreign languages is inadeq uate,contradictory and confused. N o clear distinction is made between diag-nostic tests and placement tests: it is claimed by some that any test canbe used for diagnostic purposes, and by others that it is very difficultto write diagnostic tests. It is argued that very few diagnostic tests offoreign langu age proficiency exist, and the reason for that may in part

be because th e very concepts of diagnosis and the nature of foreignlanguage proficiency are undeveloped, unproblemat ized and under-theorize d. The lack of diagnostic tests m ay also be due to mo re practical

matters, namely that to date there has been no demand for , and littlefunding to support , the development of such tests.However, with the advent of computer-based tests, especially those

delivered over the Internet , it is now relatively straightforward to givelearners immediate feedback on their performance, thereby making i tpossible for that feedback to have maximum impact , because learnersmay still recall their reasons for responding the way they did and bemore receptive to the feedback and able to incorporate it into theirdeveloping interlanguage. Thus feedback becomes, potentially, maxi-

mally inform ative and relevant.If tests are inform ed by an adeq uate theory of language use, language

deve lopm ent and lan gu age learning, and if learners can receive feed-back on their performance and their abili ty immediately, then the pos-sibility for the incorporat ion of assessment into language learningbecomes apparent, indeed urgent .

B ut first, it is impor tan t to look outside the field of second and foreignlangu age testing to see w heth er diagnosis is as poorly d eveloped a field inother content areas as it appears to be in the field of foreign languagetesting. In the next chapter, I shall exam ine the use of diagnostic testsin first language development and in learning to read.

D I A G N O S I N G F O R E I G N L A N G U A G E P R O F I C I E N C Y2

alderson diagnosis c.1

Documents