the international legal english certificate issues in developing a test of english for specific...
TRANSCRIPT
The International Legal English Certificate
Issues in Developing a Test of English for Specific Purposes
David Thighe, Cambridge ESOLEALTA ConferenceKrakow, May 2006
2
Outline
What is ILEC? Features of ESP tests
Continuum of Specificity
Evidence from the Development Practice Evidence from the Trialling stages Future Directions
3
What is ILEC?
A test for law students & newly qualified lawyers A test that assesses language skills in a legal
context Set at CEFR levels B2 & C1 Recognised by leading associations of Lawyers
4
…. One in which test content and methods are derived from an analysis of a specific purpose target language use situation (TLU), so that test tasks ….are authentically representative of tasks in the target situation, allowing for an interaction between the test taker’s language ability and specific purpose content knowledge, on one hand, and the test tasks on the other.
Douglas (2000: 19)
A definition of specific purpose assessment
5
Revisiting Douglas’ definition of LSP tests All tests can be seen as lying on a specificity
continuum, between the highly specific and the general purpose.
This continuum is multi-componential and includes the twin aspects of authenticity – situational and interactional.
O’Sullivan (2006)
6
Specificity – a continuum
PET, FCE, CAE etc
BULATSBEC
ILEC?
Generalisability
Increase in test of content knowledge
A test of English for air-traffic controllers
7
Where does ILEC go on this continuum?
Test purpose Needs Analysis/Viability Study
Development practice Working together with subject experts at the planning,
design and development stages
Evidence from Candidate Performance the trialling stage on authenticity and specificity
CLAIM
JUSTIFICATION OF CLAIM
SUBSTANTIATING CLAIM
8
Test Purpose
To assess candidate’s ability in English in an international legal environment
To allow candidates to demonstrate to employers their ability to communicate in English with clients and colleagues
To allow candidates to demonstrate to educational institutions their ability to follow a course of legal study in English
9
Cambridge ESOL
External contentexperts
Item Writers
and Chair
Development practiceCambridge ESOL and External Content Experts
Sources
Tests for vetting Test construction
Items produced and vetted
10
Development PracticeTest Content
Reading, Listening, Writing and Speaking components
Tasks based on law-related texts and topics Set in the context of International commercial
law
11
Evidence from ILEC TrialSpecific Issues
Authenticity of components Specificity of ILEC components
Reliability issues Practical constraints e.g. time,
compulsory/optional parts, length of paper etc
12
Sample and Instruments
320 candidates from 12 centres worldwide Candidates completed
ILEC Reading, Writing and Listening papers General Placement Test Information Sheets Questionnaires
Instructors questionnaires
13
Trial Findings: Authenticity
1 strongly disagree 0.5
2 disagree 5.9
3 no opinion 28.9
4 agree 38.7
5 strongly agree 26
total 100
Table 6 Q7 Candidate Questionnaire: On the Reading paper the topics are authentic (I or my colleagues may be expected to cover these topics in our legal profession).
Table 7. Q8 Candidate Questionnaire: On the Reading paper the texts and language are authentic ( I or my colleagues may be expected to read similar texts in our legal profession).
1 strongly disagree 1.5
2 disagree 7.3
3 no opinion 25.9
4 agree 34.1
5 strongly agree 31.2
total 100
•Similar results for other papers
•Similar results for instructors
14
Trial Findings: SpecificityCandidate Performance
Table of Comparison of scores of candidates with or without professional experience in GPT, Reading, Listening and Writing
Without professional experience
With professional experience
GPT 37.4 40.5
Reading 35.6 46.2
Listening 9.7 13.5
Writing 19.4 20.9
15
Trial Findings: Reliability
Cronbach’s Alpha = 0.81 to 0.89
16
Trial Findings: Time Constraints
Candidate Questionnaire• Approx 45% responded that they strongly agree or agree
that there is enough time, 33% strongly disagree or disagree, 22% indicated no opinion
Shadowing of candidates taking Reading and Writing components
• All candidates shadowed completed components, • 30% writing candidates used full time allowed.• 70% candidates completed tests in order of sequence in
paper
17
ILEC Writing – Compulsory or Optional Part 2
P1 mean P1 sd P2 mean P2 sd N CandsV1
(compulsory Part 2)5.64 1.86 5.95 1.65 56
V2(optional Part 2)
5.71 1.99 5.82 2.08 48
Table: comparison of mean scores in both versions of writing trial paper
The ILEC Writing Test was offered to a group of trial candidates as two versions. Version 1: Both parts compulsory (56 cands) Version 2: Part 2 gives a choice of 3 options (48 cands)
18
ILEC: Future Directions Evidence from Live Data on specificity and the role of content knowledge
in the test Performance of those with and without work experience
Using expert judgement to provide further evidence of Task Specificity
19
In Summary Outline of ILEC Defining features of ESP tests Purpose from Viability Study Test Development Process with External Expert Empirical Evidence from ILEC Trial Future Research
http://www.cambridgeesol.org/exams/ilec.htm
20
References Douglas, D. 2000. Assessing Languages for Specific
Purposes. Cambridge: Cambridge University Press.
O’Sullivan, B. 2006. Issues in Testing Business English: the revision of the Cambridge Business English Certificates SILT Volume 17. Cambridge: Cambridge University Press.
Weir, C. 2005. Language Testing and Validation: an Evidence-based Approach. Hampshire: Macmillan