technical report ch 01

8/9/2019 Technical Report Ch 01

1/18

!"#$%'( *"+,-.

!"#$% '(#)*+,- .(#/+0 )-

1,-2(,2+ 3)4*+%+-5+$


2/18

ii

First European Survey on Language Competences: Technical Report

Technical Report authorsThe table below presents the key authors of the Technical Report.

The following people were instrumental in the reviewing, proofreading and formatting

chapters of this report: Erna Gille, Johanna Kordes, Robert Manchin, Agnes Ilyes,Peter Husztik, Anna Chan, Michaela Perlman-Balme, Julia Guess, Danilo Rini,

Guiliana Bolli, Sylvie Lepage, Roselyne Marty, Heidi Endres, Inma Borrego, Joost

Schotten, Remco Feskens, Rebecca Stevens and John Savage.

Key authors Institutions Position

Neil Jones ESOL Project Director

Karen Ashton ESOL Project Manager, Field Operations Lead

Gunter Maris Cito Data Analysis Lead

Sanneke Schouwstra Cito Questionnaires development Lead includingframework and indices

Norman Verhelst Cito Standard Setting Lead

Ivailo Partchev Cito Weighing Lead, Data Management Lead

Jesse Koops Cito Data management Co-Lead

Martin Robinson ESOL Language Testing Team Lead

Manas

ChattopadhyayGallup Sampling and base weights Lead

Gergely Hideg Gallup Sampling Co-Lead

Jostein Ryssevik Gallup Software systems Lead


3/18

iii


Abbreviations and codes used in this report

The following educational system and language codes are used throughout this report.

Participatingeducationalsystem

Educationalsystem code

Questionnairelanguage(s)

Language code

Flemish Community of

BelgiumBE nl Dutch Nl

French Community of

BelgiumBE fr French Fr

German Community of

BelgiumBE de German/French de, fr

Bulgaria BG Bulgarian Bg

Croatia HR Croatian Hr

England UK-ENG English En

Estonia EE Estonian; Russian et, er

France FR French Fr

Greece EL Greek El

Malta MT English En

Netherlands NL Dutch Nl

Poland PL Polish Pl

Portugal PT Portuguese Pt

Slovenia SI Slovene Sl

Spain ES

Spanish, Basque,

Catalan, Galician,

Valencian

es, Spanish-Basque

Spanish-Catalan, Spanish-

Galician, Spanish-Valencian

Sweden SE Swedish Sv

The following abbreviations are used in this report.

Abbreviations In full

BoW Body of Work method

CB Computer-based

CD Compact Disc

CEFR Common European Framework of Reference

CFI Comparative Fit Index


4/18

iv


Abbreviations In full

CLIL Content and Language Integrated Learning

COGN Cognitive

CML Conditional Maximum Likelihood

CMOS Cumulative Measure of Size

DIF Differential Item Functioning

DVD Digital Versatile Disc

EC European Commission

EILC European Indicator of Language Competences

ENR Enrolment

ESCS Economic, social and cultural status

ESLC European Survey on Language Competences

FL Foreign Language

Gb Gigabyte

HISEI Parental Occupation

HOMEPOS Home possessions

ICT Information and Communication Technologies

ID Identification

ILO International Labour Organisation

INES OECD Indicators of Education Systems

INT International

IRT Item Response Theory

ISO International Organization for Standardization

ISCED International Standard Classification of Education

ISCO International Standard Classification of Occupations

ISCO_F International Standard Classification of Occupation Father

ISCO_M International Standard Classification of Occupation Mother

ISEI International Socioeconomic Index

MM Multiple Marking

MOS Measure of Size

NFI Normed Fit Index

NNFI Non-Normed Fit Index

NRC National Research Coordinator


5/18


6/18

vi


TABLE OF CONTENTS

1 INTRODUCTION ............................................................................................................... 2

1.1 KEY ELEMENTS OF THE ESLC ........................................................................................ 4

1.2 THIS TECHNICAL REPORT............................................................................................... 7

1.3 REFERENCES................................................................................................................... 8

2 INSTRUMENT DEVELOPMENT - LANGUAGE TESTS .......................................... 10

2.1 DEVELOPMENT OF THE LANGUAGE TESTING FRAMEWORK.......................................... 11

2.2 DEVELOPMENT OF THE LANGUAGE TESTS.................................................................... 27

2.3 TEST DEVELOPMENT PROCESS...................................................................................... 34

2.4 MARKING..................................................................................................................... 41

2.5 FINAL TEST DESIGN...................................................................................................... 44

2.6 REFERENCES................................................................................................................. 49

3 INSTRUMENT DEVELOPMENT - QUESTIONNAIRES .......................................... 51

3.1 CONCEPTUALISATION................................................................................................... 51

3.2 OPERATIONALISATION................................................................................................. 74

3.3 REFERENCES................................................................................................................. 98

4 OPERATIONS - SAMPLING ........................................................................................ 105

4.1 TARGET POPULATION AND OVERVIEW OF THE SAMPLING DESIGN............................. 105

4.2 POPULATION COVERAGE AND SCHOOL AND STUDENT PARTICIPATION RATE

STANDARDS........................................................................................................................... 107

4.3 COVERAGE OF THE INTERNATIONAL TARGET POPULATION....................................... 107

4.4 ACCURACY AND PRECISION........................................................................................ 109

4.5 RESPONSE RATES........................................................................................................ 110

4.6 ESTABLISHING THE NATIONAL TARGET POPULATION................................................ 111

4.7 SAMPLING IMPLEMENTATIONTEST LANGUAGES.................................................... 111

4.8 TESTING GRADES........................................................................................................ 112

4.9 SCHOOL SAMPLING FRAME......................................................................................... 116

4.10 STRATIFICATION..................................................................................................... 117

4.11 ASSIGNING A MEASURE OF SIZE TO EACH SCHOOL................................................. 122

4.12 SORTING THE SAMPLING FRAME............................................................................. 122

4.13 SCHOOL SAMPLE ALLOCATION ACROSS EXPLICIT STRATA..................................... 123


7/18

vii


4.14 PROBABILITY PROPORTIONAL TO SIZE SAMPLING.................................................. 123

4.15 IDENTIFYING REPLACEMENT SCHOOLS................................................................... 125

4.16 STUDENT SAMPLING............................................................................................... 126

4.17 SELECTING THE SCHOOL SAMPLE PERSONNEL........................................................ 129

4.18 SAMPLING FORMS................................................................................................... 130

5 OPERATIONS - TRANSLATION ................................................................................ 133

5.1 INTRODUCTION........................................................................................................... 133

5.2 OVERVIEW OF TRANSLATION SYSTEM,SUPPORT AND TRAINING............................... 134

5.3 DOCUMENTATION NEEDING TRANSLATION AND THE TRANSLATION PROCESS.......... 135

5.4 SURVEYLANG TRANSLATION GUIDELINES................................................................. 140

5.5 QUESTIONNAIRE LANGUAGE,LOCALISATIONS AND AMENDMENTS TO STANDARD

PROCESS................................................................................................................................ 141

5.6 DEVELOPMENT OF SOURCE VERSIONS........................................................................ 143

5.7 FIELD TRIAL AND MAIN STUDY TRANSLATION PROCESSES....................................... 144

5.8 RECRUITMENT GUIDELINES FOR TRANSLATORS........................................................ 145

5.9 REFERENCES............................................................................................................... 145

6 OPERATIONS - THE SURVEYLANG SOFTWARE PLATFORM ........................ 148

6.1 INTRODUCTION........................................................................................................... 148

6.2 REQUIREMENTS.......................................................................................................... 148

6.3 ARCHITECTURE.......................................................................................................... 150

6.4 TEST-ITEM AUTHORING TOOL.................................................................................... 151

6.5 TEST-ITEM DATABANK............................................................................................... 154

6.6 TRANSLATION MANAGEMENT.................................................................................... 156

6.7 TEST ASSEMBLY......................................................................................................... 157

6.8 A:TEST ASSEMBLY.................................................................................................... 157

6.9 B:ALLOCATION.......................................................................................................... 160

6.10 TEST MATERIALS PRODUCTION............................................................................... 162

6.11 THE USBMEMORY STICK PRODUCTION UNIT......................................................... 163

6.12 TEST RENDERING.................................................................................................... 164

6.13 THE USB-BASED TEST RENDERING OPERATING ENVIRONMENT............................ 166

6.14 DATA UPLOAD SERVICE.......................................................................................... 166

6.15 ADDITIONAL UTILITIES........................................................................................... 167

6.16 SOFTWARE QUALITY AND TESTING........................................................................ 167


8/18

viii


6.17 PERFORMANCE........................................................................................................ 168

7 FIELD OPERATIONS ................................................................................................... 170

7.1 OVERVIEW OF ROLES AND RESPONSIBILITIES............................................................ 170

7.2 KEY NATIONAL RESEARCH COORDINATOR TASKS.................................................... 170

7.3 COMMUNICATIONS BETWEEN SURVEYLANG AND NRCS.......................................... 173

7.4 STAFF SELECTION AND STAFF TRAINING.................................................................... 174

7.5 NRCSAMPLING TASKS............................................................................................... 175

7.6 NRCPRE-ADMINISTRATION TESTING TASKS.............................................................. 176

7.7 NRCTEST MATERIALS MANAGEMENT TASKS............................................................ 176

7.8 KEY SCHOOL COORDINATOR TASKS.......................................................................... 178

7.9 KEY TEST ADMINISTRATOR TASKS............................................................................ 1817.10 KEY TECHNICAL SUPPORT PERSON TASKS (IF CBTESTING) .................................. 183

7.11 RECEIPT OF MATERIALS AT THE NRCAFTER TESTING........................................... 183

7.12 DATA ENTRY TASKS................................................................................................ 183

7.13 MARKING OF WRITING........................................................................................... 185

7.14 DATA SUBMISSION.................................................................................................. 188

7.15 DATA CHECKING..................................................................................................... 189

7.16 CODING OF STUDENT QUESTIONNAIRES................................................................ 189

7.17 MAIN STUDY:A REVIEW......................................................................................... 191

7.18 REFERENCES........................................................................................................... 191

8 OPERATIONS - QUALITY MONITORING .............................................................. 193

8.1 AN INTRODUCTION..................................................................................................... 193

8.2 SUPPORT FOR NRCS IN QUALITY MONITORING.......................................................... 194

8.3 IMPLEMENTATION OF QUALITY MONITORING PROCEDURES...................................... 196

8.4 ESLCQUALITY MONITORS......................................................................................... 197

8.5 QUALITY MONITORING DATA..................................................................................... 200

9 DATA PROCESSING - WEIGHTING ......................................................................... 203

9.1 MOTIVATION AND OVERVIEW.................................................................................... 203

9.2 BASE WEIGHTS........................................................................................................... 205

9.3 ADJUSTING WEIGHTS FOR NON-RESPONSE................................................................. 207

9.4 VARIANCE ESTIMATION.............................................................................................. 214

9.5 REFERENCES............................................................................................................... 215


9/18

ix


10 DATA PROCESSING - QUESTIONNAIRE INDICES .......................................... 217

10.1 TYPE OF INDICES..................................................................................................... 217

10.2 TESTING THE STRUCTURE OF LATENT VARIABLES................................................. 217

10.3 DATA PREPARATION............................................................................................... 218

10.4 STUDENT QUESTIONNAIRE ..................................................................................... 219

10.5 TEACHER QUESTIONNAIRE..................................................................................... 247

10.6 PRINCIPAL QUESTIONNAIRE................................................................................... 261

10.7 REFERENCES........................................................................................................... 273

11 SETTING STANDARDS IN RELATION TO THE CEFR .................................... 275

11.1 OUTLINE OF THE STANDARD SETTING CONFERENCE (SEPT 26-302011) ................ 276

11.2 STANDARD SETTING METHODOLOGY..................................................................... 277

11.3 THE STANDARD SETTING CONFERENCE -RESULTS................................................. 283

11.4 THE WRITING ALIGNMENT STUDY (AUGUST 2011) ................................................ 289

11.5 WRITING ALIGNMENT STUDY OUTCOMES............................................................... 291

11.6 THE STATUS OF THE STANDARDS............................................................................ 294

11.7 REFERENCES........................................................................................................... 295

12 ANALYSES .................................................................................................................. 298

12.1 INTRODUCTION....................................................................................................... 298

12.2 ITEM RESPONSE THEORY........................................................................................ 300

12.3 LINKING LANGUAGE PROFICIENCY TO KEY POLICY INDICATORS:LATENT

REGRESSION.......................................................................................................................... 308

12.4 APPENDIX:GENERATING PLAUSIBLE VALUES........................................................ 310

12.5 REFERENCES........................................................................................................... 320

13 DATA PROCESSING - DATA SETS ........................................................................ 332

13.1 THE STUDENT QUESTIONNAIRE AND PERFORMANCE DATA FILE........................... 332

13.2 LANGUAGE ASSESSMENT ITEMS DATA FILES.......................................................... 332

13.3 TEACHER QUESTIONNAIRE DATA FILE................................................................... 333

13.4 SCHOOL QUESTIONNAIRE DATA FILES.................................................................... 333

13.5 RECORDS IN THE DATA SETS................................................................................... 334

13.6 RECORDS EXCLUDED FROM THE DATASETS........................................................... 334

13.7 WEIGHTS IN THE DATASETS.................................................................................... 334

13.8 REPRESENTING MISSING DATA............................................................................... 335


10/18

x


13.9 IDENTIFICATION OF RESPONDENTS,SCHOOLS AND MARKERS................................ 335

14 APPENDICES .............................................................................................................. 338

14.1 EXAMPLE LANGUAGE TEST TASK TYPES................................................................. 338

14.2 QUESTIONNAIRES ................................................................................................... 373

14.3 SAMPLING FORMS................................................................................................... 553

14.4 ESLCTECHNICAL STANDARDS.............................................................................. 630

14.5 DEFINITIONS........................................................................................................... 641

14.6 MULTIPLE MARKING............................................................................................... 630

14.7 MANAGING AND IMPLEMENTING THE ESLC .......................................................... 630


11/18

1


!"#$%&' )*

+,%'-./0%1-,


12/18

2


1 Introduction

The European Survey on Language Competences (ESLC), the first survey of its kind,is designed to collect information about the foreign language proficiency of students in

the last year of lower secondary education (ISCED2) or the second year of upper

secondary education (ISCED3) in participating countries or country communities

(referred to herein as

language competences but a survey that should be able to provide information about

language learning, teaching methods and curricula. (European Commission 2007a).

by questionnaires to teachers and pupils to gather contextual informatio

The ESLC is a collaborative effort among the 16 participating educational systems and

SurveyLang partners to measure the language proficiency of approximately 53000

students across Europe, to assist the European Commission in establishing a

European Indicator of Language Competence to monitor progress against the March

2002 Barcelona European Council conclusions. These conclusions

improve the mastery of basic skills, in particular by teaching at least two foreign

languages from a v linguistic

competence indicator 2005). As the Commission (European

Commission 2005) states, the decision to launch the ESLC arose from the current

lack of data on actual language skills of people in the European Union and the need

ESLC was thereforeinitiated by the Commission with the aim that

establishment of a European Indicator of Language Competence and will provide

reliable information on language learning and on the language competences of young

a)

policy makers, teachers and learners in all surveyed

of contextual information in the background questionnaires (European Commission

2007b).

Each educational system tested students in two languages; the two most widely taught

of the five most widely taught European languages: English, French, German, Italian

and Spanish. This effectively meant that there were two separate samples within each

educational system, one for the first test language, and one for the second. Each

sampled student was therefore tested in one language only.

assessed in two of the three skills of Listening, Reading and Writing.

understand spoken or written texts, or to express themselves in writing. Their observed

language proficiency is described in terms of the levels of the Common European

Framework of Reference (CEFR) (Council of Europe 2001), to enable comparison

across participating educational systems. The data collected by the ESLC will allow


13/18

3


participating educational systems

weaknesses across the tested language skills, and to share good practice with other

participating educational systems.

To facilitate a more productive comparison of language policies, and language

teaching methods (European Commission 2005:5) context questionnaires, covering

the 13 policy issues detailed below, were administered to the students tested, their

teachers of foreign languages, and their institution principals. In addition, system-wide

information was collected through the National Research Coordinators.

Early language learning is explored through questions on the onset of foreign

language learning, and the weekly amount of time for target and foreign

language learning (lesson time and homework).

The diversity and order of foreign language teaching is explored through

questions to principals and students on the number of foreign and ancient

languages provided (schools) and learned (students).The language friendly living environment explores the number of students' first

languages, languages used at home, and parents' target language

knowledge; also the ways in which students use the target language: at

home, in the living environment, through visits abroad or through the media.

The concept of the language friendly school looks at the degree of language

specialisation, for example, whether content and language integrated

learning (CLIL) is practised.

A set of indices related to the use of ICT to enhance foreign language learning

and teaching.

Intercultural exchanges arising from school trips, visits or language projects are

explored from the perspective of students, teachers, principals andeducational systems.

The impact of teachers from other language communities is explored.

Language learning for all looks at provision for immigrant students of the first

and second generation.

Under approaches to foreign language teaching a large number of indices

explore, for example, the relative emphasis teachers put on teaching the

different skills, emphasis placed on similarities between the target language

and other known languages, and use of the target language during lessons

by teachers and students - all these from the perspective of teachers and

students.

language: their perception of its usefulness, of how difficult it is to learn andof how they evaluate the lessons, teacher and textbooks.

Teacher initial and in-service training includes indices for teacher qualifications

and competences. Questions to teachers and principals explore financial

and other incentives for in-service training, how much training teachers

attend, and whether the focus of training is on language teaching.

A period of work or study in another country addresses questions to teachers

and principals on the number of such stays, financial incentives, and

availability of funding for exchange visits or stays abroad.


14/18

4


Several questions on the use of existing European language assessment tools

explore uptake of the CEFR and a language portfolio: is use of the CEFR

language portfolio, and how do they use it?

for example on

the target language and other languages and

the number of languages taught over the past five years.

The ESLC data adds significantly to the knowledge base that was previously available

at European level or from official national statistics. The data should prove a valuable

resource for researchers, policy makers, educators, parents and students and will

enable them to review progress towards achieving the March 2002 Barcelona

European Council conclusions of learning two foreign languages from an early age.

SurveyLang recognises the contribution of all of its partners and National Research

Coordinators (NRCs) in the delivery of the survey. The ESLC is methodologicallycomplex and its implementation has required a considerable collaborative effort by the

participating educational systems with SurveyLang. The in-country administration of

the survey was the responsibility of the representatives of each educational system

(NRCs). Implementing the ESLC depended not only on this collaboration but also on

pooling the expertise of SurveyLang partners to develop and exploit innovative

methodologies, test instruments and technologies. This Technical Report describes

these methodologies, together with other aspects of the methodology that have

enabled the ESLC to provide data to support the European Commission in this area of

policy. The descriptions are provided at a level that will enable review of the

implemented procedures and the solutions adopted for the challenges faced.

This report contains a description of the theoretical underpinning of the complex

techniques used for the ESLC and to create the ESLC data sets, which contain data

on approximately 50000 students from 151educational systems. The data sets include

not only information on student performance in two of the three language skill areas of

Listening, Reading and Writing, but also their responses to the Student Questionnaire

that they completed as part of the administration. Data from the school principals and

language teachers of participating schools teaching at the eligible ISCED level are also

included in the data sets.

1.1 Key elements of the ESLC

Elements central to the design of the ESLC are outlined in brief below. The remainder

of this report describes these elements, and the associated procedures and

methodology, in more detail.

1As England participated in the Main Study later than other adjudicated entities, at this stage data from England is notincluded in the data sets.


15/18

5


Sample size: Approximately 53000 students enrolled in schools in 16 participating

educational systems were assessed in the ESLC Main Study 2011.

Tested education level: Students were tested at the last year of lower secondary

education (ISCED2) or the second year of upper secondary education (ISCED3) in

participating educational systems.

Tests: The tests covered three language skills: Listening, Reading and Writing in five

test languages: English, French, German, Italian and Spanish. Each student was

assessed in two out of these three skills in one test language and also completed a

contextual questionnaire. The language tests measure achievement of levels A1 to B2

of the Common European Framework of Reference (CEFR) (Council of Europe, 2001).

The pre-A1 level which is also reported indicates failure to achieve A1. Language

teachers and school principals at sampled schools also completed a contextual

questionnaire.

Testing mode: The ESLC was administered in both paper and computer-based

formats. The Teacher and Principal Questionnaires were administered through an

internet-based system.

Testing duration: Students had either 30 minutes or 45 minutes to complete each

test. All Listening and Reading tests were set at 30 minutes. The low and intermediate

Writing tests were set at 30 minutes, while the high level Writing test and Student

Questionnaires (including a CEFR self-assessment) were set at 45 minutes. The total

testing time for a student, including the questionnaire, was thus 105 or 120 minutes.

Summary of tested languages, levels and testing mode across participatingeducational systems: The tables below provide a summary of the tested languages,

levels and testing mode of each educational system. Further details on the tested

languages and levels can be found in Chapter 4 on sampling.


16/18


17/18

7


Table 2 Tested languages summary

Language Number of countries testing

language as first most widelytaught language

Number of countries testing language

as second most widely taught language

English 13 2

French 3 3

German 0 8

Italian 0 1

Spanish 0 2

Table 3 Tested levels summary

Number of countries testing

ISCED 2

Number of countries testing ISCED 3

First most widely taught

language13 3

Second most widely

taught language11 5

Outcomesthe ESLC delivers the following outcomes:

A profile of the language proficiency of sampled students. Contextual

indicators providing a broad range of information on the context of foreignlanguage teaching policies and foreign language learning at student, teacher

and school level.

Information on the relationship between language proficiency and the

contextual indicators.

A resource and knowledge base for policy analysis and research.

1.2 This technical report

This technical report is concerned with the technical aspects of the ESLC whereas the

Final Report is concerned with the results of the ESLC. Policy recommendations areoutlined in the conclusions of the Final Report and not discussed in this report.

This technical report describes the methodologies and procedures adppted to enable

the ESLC to provide high quality data to support the European Commission in this

area of policy. The descriptions are provided at a level that will enable review of the

implemented procedures and solutions to the challenges faced.

The report covers the following areas:


18/18

8


Instrument design: Chapters 2 and 3 describe the development of the

language tests to produce measures comparable across languages and

interpretable in relation to the CEFR, and the questionnaires, to address a

range of European language policy issues.

Operations: Chapter 4 describes the sampling procedures, Chapter 5 the

translation of the questionnaires, Chapter 6 the innovative software platform

developed for the ESLC to support both paper-based and computer-based

administration, Chapter 7 the field operations and Chapter 8 the approach

taken to quality monitoring.

Data processing, scale construction and data products: Chapter 9 describes

the handling of sampling weights, Chapter 10 design of the questionnaire

indices, Chapter 11 the approach to setting CEFR-related standards for the

five languages, Chapter 12 the analyses, and Chapter 13 the development

and the contents of the data sets.

Appendices: Examples of the language test task types, the complete set ofMain Study Questionnaires, the sampling forms, the Technical Standards and

a comprehensive report on multiple marking of Writing.

1.3 References

Council of Europe (2001) Common European Framework of Reference for Languages:

Learning, Teaching, Assessment, Cambridge: Cambridge University Press.

European Commission (2005) Commission Communication of 1 August 2005 - The

European Indicator of Language Competence [COM(2005) 356 final - Not

published in the Official Journal], retrieved 18 January 2012, from

http://europa.eu/legislation_summaries/education_training_youth/lifelong_learni

ng/c11083_en.htm

European Commission (2007a) Communication from the Commission to the Council of

13 April 2007 entitled Framework for the European survey on language

competences[COM (2007) 184 finalNot published in the Official Journal]

European Commission (2007b) Terms of Reference: Tender no. 21 European Survey

on Language Competences, Contracting Authority: European Commission.

technical report ch 01

Documents