Report on theCEFR /ACTFL / ILR & STANAG

“Alignment Conference”

June 30 - July 03, 2010Leipzig, Germany

Julie J. Dubeau, M.A.BILC Secretary

Varna, BulgariaOctober 14, 2010

CEFR / ACTFL / ILR /STANAG “Alignment Conference”

• Goal (s) of Conference

• Some Perspectives Presented

• Some Preliminary Questions

• Some Preliminary Conclusions

About the ConferenceTaken from http://www.uni-leipzig.de/actflcefr2010

• The goal of the ACTFL / CEFR Alignment Conference 2010 is to bring together some 45 leaders in the field from both Europe and North America to explore a crosswalk between the ACTFL Proficiency Guidelines and the Common European Framework of Reference for Languages (CEFR) and to establish equivalencies on theoretical and empirical grounds.

About the Conference (cont’d)

• Both sets of scales claim to measure the same construct: proficiency. The successful establishment of equivalencies would support the validity of both scales.

• Problems in establishing equivalencies would point to the need for further research and development.

Further very ambitious goals included the following:

• to present and discuss:• empirical studies on the validity and reliability of

tests based on either framework;• theoretical studies of the construct validity of

either framework; and empirical studies comparing both frameworks and/or tests based on both frameworks;

• to present and critically discuss:• standardized tests (test systems) based on

either framework; and for different target groups (age, education, professional purposes, etc.);

Further very ambitious goals included the following:

• to develop guidelines for developing tests that can be rated according to both scales; and

• to develop guidelines for developing proficiency tests, their administration, and evaluation.

• These goals will be accomplished by combining general session presentations with break-out discussion groups.  Oh My!!

The ‘Alignment Conference’

• 4 parallel workshops

• Opening addresses

• Opening plenary presentations:– It’s easier to malign tests than to align tests

• Ray Clifford

– The CEFR: An evolving framework of reference• Nick Saville See abstracts on website

The ‘Alignment Conference’

12 papers organized by topics:

• Topic 1: Conceptual issues in a crosswalk• Topic 2: Technical issues across domains• Topic 3: Research and empirical issues in a

crosswalk

• 3 breakout sessions each with focus either on purposes and benefits, skills and domains, and on issues such as intercultural competence, implications for younger learners and language policy and curricular issues.

• http://www.uni-leipzig.de/actflcefr2010/abstracts.html

Test Equivalence, Equating, and Linking: The Issue of Validity

Slides used with permission

Prof. Olaf BärenfängerUniversität Leipzig

Olaf BärenfängerUniversität Leipzigbaerenfaenger@uni-leipzig.de

Test Equating and Linking

1. Some Conceptual Clarifications

2. Validity as Core of Equivalence

3. Conclusions

Equating adjusts for differences in difficulty, not differences in content. <... > in most cases, (different) tests clearly measure very different content/constructs. We refer generically to a relationship between scores on such tests as linking.

<...> it is virtually certain that score differences are attributable to construct differences as well as to errors of measurement, either or both of which could be quite large.

With equal force, however, the adequacy of the linking may be highly suspect depending on the nature of the decisions made based on the linking.

(Kolen & Brennan 20042: 423 f.)

Olaf Bärenfänger

Does Linking Really Only Mean to Correlate Test Scores?

1. Some Conceptual Clarifications

2. Validity as Core of Equivalence

3. Conclusions

Linking = Comparing the validity of two different test and adjusting for difficulty

„Validity is an integrated evaluative judgement of the degree to which empirical evidence and theoretical rationales support the adequacy and appropriateness and actions based on test scores or other modes of assessment“. (Messick 1989: 13)

„even for purposes of applied decision making, reliance on criterion validity or content coverage is not enough“. (Messick 1989: 17)

Olaf Bärenfänger

Suggestions for a Linking Argument

1. Some Conceptual Clarifications

2. Validity as Core of Equivalence

3. Conclusions

Step 1: Gather and compare all kind of evidential information about the two tests, e.g.

Constructs Internal validity (e.g. difficulty, discrimination, estimation of reliability, SEM, factor analysis, qualitative analyses, G Theory,IRT) External validity (e.g. correlation/regression studies, anchoring, test calibration with IRT, linking through experts‘ judgements) Content relevance (quality of relation between test domain and real life domain) Content representativeness (quantity of relation between test domain and real life domain) Process analyses

Test Interpretation Test Use

Evidential Basis Construct validity Construct validity +Relevance/utility

Consequential Basis Value implications Social consequences

Olaf Bärenfänger

Some Conclusions

1. Some Conceptual Clarifications

2. Validity as Core of Equivalence

3. Conclusions

Things are far more complicated than assumed initially .

When the goal is to link two tests, we need to be aware that

linking is more than mere concurrent validity.

Linking is essentially an issue of validity.

A venue might be to make use of an equivalence argument as

suggested.

In order to pursue this goal, more collaboration between

researchers and test institutions is needed as well as an

agreement on the details of an equivalence argument.

It is probably still a long way until we have safe crosswalks

between different test systems.

Olaf Bärenfänger

Pardee Lowe, Jr.

U.S. Gov. Interagency Language Roundtable

Slides used with permission

CAN ACTFL/ILR AND CEFR BE ALIGNED FOR SPEAKING

ASSESSMENT?

POSSIBILITIES

• COMPARE?

• RELATE?

• ALIGN?

• EQUATE?

• For the last two: Does one need the same “construct”?

Pardee Lowe, Jr.

QUESTIONS?• THE FOLLOWING DISCUSSES SIMILARITIES

AND DIFFERENCES BETWEEN ACTFL/ILR AND CEFR

• PRESENTED ARE THOSE ASSESSMENT FEATURES WHICH ALLOW ACTFL/ILR TO BE USED FOR EXAMINATIONS IN SPEAKING

• THE QUESTION TO CEFR ADEPTS?–HOW MANY OF THESE FEATURES OCCUR IN

CEFR?– IN WHAT WAY(S)?

Pardee Lowe, Jr.

SIMILARITIES

• WORK IN PROGRESS

• PROSE DESCRIPTIONS

• HIERARCHICAL

• CRITERION-REFERENCED

• CAN-DO STATEMENTS

• LEVELS VS BANDS

• PLUS LEVELS VS PLUS BANDS

Pardee Lowe, Jr.

CEFR BANDS• HOW MANY BANDS ARE THERE?• WHAT DOES “BAND” MEAN?• BASKET ANALOGY?• IS IT A RANGE?• DOES IT HAVE HEIGHT? DEPTH?• HOW ARE QUALITY AND QUANTITY ACCOUNTED

FOR WITHIN A BAND?• HOW MANY TASKS IN EACH BAND?• HOW ARE “BANDS” ASSIGNED?

– PARTICULARLY “PLUS BANDS”??

Pardee Lowe, Jr.

MAJOR PARAMETERS FOR ACTFL/ILR & CEFR ALIGNMENT

ACTFL

RELATIONAL FRAME• FIXED

RELATIONAL MODEL• NATIVE SPEAKER

FEATURES• A CORE PER LEVEL

BOUNDARIES• DELINEATED

CEFR

• FLEXIBLE

• MODEL LEARNER?

• RICH NUMBER

• BASKETS

Pardee Lowe, Jr.

QUESTIONS?

• HOW MANY OF THESE ACTFL/ILR FEATURES OCCUR IN CEFR?

• ARE THEY EMPLOYED IN THE SAME WAY?

• IF NOT, HOW WOULD ACTFL/ILR AND/OR CEFR HAVE TO BE ALTERED TO ACHIEVE ALIGNMENT?

Pardee Lowe, Jr.

Framing research to develop guidelines for developing tests that can be rated according to both scales: The case of writing

Liz Hamp-Lyons

University of Nottingham, UK/University of Hong Kong

Slides used with permission

What the CEFR knows about assessing high level English writing

Liz Hamp-Lyons

What the CEFR knows about assessing high level English writing

Liz Hamp-Lyons

What the CEFR knows about assessing high level English writing

Liz Hamp-Lyons

What ACTFL knows about assessing high level English writing

“It must be noted that the Superior level encompasses levels 3, 4, and 5 of the ILR

scale. However, the abilities at the Superior level described in these

guidelines are baseline abilities for performance at that level rather than a complete description of the full range of

Superior.”

Liz Hamp-Lyons

ACTFL Superior Level writing (ILR Levels 3 and above)SUPERIORWriters at the Superior level are able to produce most kinds of formal and informal correspondence, complex summaries, precis, reports, and research papers on a variety of practical, social, academic, or professional topics treated both abstractly and concretely. They use a variety of sentence structures, syntax, and vocabulary to direct their writing to specific audiences, and they demonstrate an ability to alter style, tone, and format according to the specific requirements of the discourse. These writers demonstrate a strong awareness of writing for the other and not for the self.

See p.24 of handout booklet for full description.

Liz Hamp-Lyons

What the ILR knows about high level English writing

Writing 5 (Functionally Native Proficiency):Has writing proficiency equal to that of a well educated native. Without non-native errors of structure, spelling, style or vocabulary can write and edit both formal and informal correspondence, official reports and documents, and professional/ educational articles including writing for special purposes which might include legal, technical, educational, literary and colloquial writing. In addition to being clear, explicit and informative, the writing and the ideas are also imaginative. The writer employs a very wide range of stylistic devices.

Liz Hamp-Lyons

Are the two frameworks intended to be, or indeed claimed to be, equivalent?

They are stylistically differentThey are strikingly different in lengthACTFL descriptors are a mix of ‘can-do’s’, personal attributes (“they are able to…”), text characteristics…CEFR descriptors are superficially ‘can-do’s’ (“Can express him/herself with clarity and precision relating to the addressee flexibly and effectively.”) but in fact are far too vague to be useable as they stand

Liz Hamp-Lyons

“The successful establishment of equivalencies would support the validity of both scales.”

OK…

We are not there yet

We may not be on the track to get there

What CAN we do? Liz Hamp-Lyons

Framing… to develop guidelines for assessing writing at the higher levels

Guidelines shouldBegin from a construct

The construct needsA theory of second language acquisition

A theory of learning

A theory of written language mastery trajectories

An empirical model of written language use at different levels

An argument (a) language use argument (a) assessment use argument that will weave all these dimensions together appropriately for different audiences/clients Liz Hamp-Lyons

Framing… to develop guidelines for assessing writing at the higher levels

Guidelines shouldContinue by stipulating the need to obtain perceptions/judgements from a range of stakeholders

Language testing specialistsTeachers of the area under studyStudents of the area under studyScore usersTest developers/item writers

Liz Hamp-Lyons

Framing… to develop guidelines for assessing writing at the higher levels

Guidelines should alsoPropose a specification study comparing/ contrasting tasks, input texts, level descriptors across all components of each test (for which the “same construct” is being claimed(

Propose a text linguistic study such as corpus analysis of tasks for difficulty specification; or discourse analysis of what persons scoring specified levels on each test can do within the domain.

Liz Hamp-Lyons

Framing… to develop guidelines for assessing writing at the higher levels

Guidelines also need toStipulate the need for rigorous quantitative studies to test hypotheses deriving from the construct definition and qualitative elicitation stages

Propose consequential validity processes to check the impact of any emerging conclusions on teachers, learners and other stakeholders outside the testing and research enterprises

Liz Hamp-Lyons

A safe crosswalk???Where we are TODAY!

Considerations

• Can BILC Study groups tackle these complex and complicated issues?

• Is the issue about comparing/aligning STANAG/CEFR scales?

• Can we reconcile scale orientations?• Each test would have to be linked to a test

derived from the other scale!• Generalizations cannot be made today. Will

they ever be reliable?

The Way Ahead?

• Another conference is being planned for next year.

• Setting a research agenda will be critical

• BILC will be monitoring & reporting back and continue to encourage dialogue & research among our constituents

• What can you do????

Communicative proficiency and linguistic development: intersections between SLA and language testing research

• http://eurosla.org/monographs/EM01/EM01home.html

• It is an edited volume by the SLATE network (Second Language Acquisition and Testing in Europe), a group of SLA researchers and language testers which aims to explicate the CEFR in various languages informed by SLA and language assessment research.  http://www.slategroup.eu/

• The introductory chapter by Hulstijn et al., in particular, offers a useful summary of the origins of the CEFR and raises a number of issues also discussed in the “Leipzig’ conference (e.g. the origins of the CEFR in the Threshold series as well as in some North American scales including ACTFL, FSI and ILR, see p. 14; the suitability of the CEFR for language assessment etc.).

• 2010 ACTFL Convention and World Languages Expo at the Hynes Convention Center in Boston, MA from November 19-21, 2010 (Pre-convention workshops, Thursday, November 18). www.actfl.org