l anguage t esting s ervices john h.a.l. de jong 2006 1 assessing reading literacy in the language...
TRANSCRIPT
LANGUAGE TESTING SERVICES John H.A.L. de Jong 20061
Assessing Reading Literacy in the Language of Instruction
John H.A.L. de Jong
Intergovernmental Conference: Languages of Schoolingtowards a Framework for Europe Strasbourg, 16-18 October 2006
LANGUAGE TESTING SERVICES John H.A.L. de Jong 20062
Overview
• Background of the PISA studies
• Design of the PISA studies
• PISA approach to Reading Literacy
• Some central findings of PISA 2000
• Extensions for PISA 2009
• Relationships between PISA & CEF
• Applying CEF in Upper Secondary Vocational
LANGUAGE TESTING SERVICES John H.A.L. de Jong 20063
Background of PISA study
OECD: Organisation for Economic Co-operation and Development
• 30 member countries committed to democracy and market economy
• Provide comparative data and analyses• For countries
– to compare policy experience; – seek answers to common problems; – identify good practice;– co-ordinate policies.
• To support economic growth, boost employment, raise living standards.
LANGUAGE TESTING SERVICES John H.A.L. de Jong 20064
History OECD• 1948 Organisation for European Economic Co-operation
(OEEC), set up with support from the United States and Canada to co-ordinate the Marshall Plan for the reconstruction of Europe after World War II (www.oecd.org/history).
• 1961 Created as an economic counterpart to NATO, the OECD took over from the OEEC
• Early recognition of importance of education of its Citizens
• 1985 Gather indicators on investment in education
• Education at a glance yearly publication
• 2000 first PISA
LANGUAGE TESTING SERVICES John H.A.L. de Jong 20065
PISA Goals
After studying resources & investments in education for a number of consecutive years:Wish to know the yield of education:
at the end of compulsory education how well are students prepared to continue further education or to start working life?
LANGUAGE TESTING SERVICES John H.A.L. de Jong 20066
2000 2003 2006 2009 2012
Reading Reading Reading Reading Reading
Maths Maths Maths Maths Maths
Science Science Science Science Science
PISA assessment schedule
LANGUAGE TESTING SERVICES John H.A.L. de Jong 20067
2000 2003 2006 2009 2012
Reading Reading Reading Reading Reading
Maths Maths Maths Maths Maths
Science Science Science Science Science
PISA assessment schedule
LANGUAGE TESTING SERVICES John H.A.L. de Jong 20068
Participation in OECD/PISA
PISA Year OECD Partner
2000 28 15
2003 30 10
2006 30 28
LANGUAGE TESTING SERVICES John H.A.L. de Jong 20069
Definition
Reading literacy is understanding, using and reflecting on written texts, in order to achieve one’s goals, to develop one’s knowledge and potential and to participate in society.
LANGUAGE TESTING SERVICES John H.A.L. de Jong 200610
Retrieving information
Interpreting text
Reflection & Evaluation
LANGUAGE TESTING SERVICES John H.A.L. de Jong 200611
Let’s look at ReadingIn the activity of reading we can distinguish four main elements:
1 there is a reader
2 there is a text (the author is part of the text, but not present in the activity)
Both of these are observable.
3 the reader has a goal (a self-chosen reason for reading the text)
4 the reading activity will have a result
(a change in the reader: more knowledge, satisfaction, etc )
These are not observable.
LANGUAGE TESTING SERVICES John H.A.L. de Jong 200612
Text - Reader - Goal - Result
Assessing Reading:
- Response- Test taker - TaskText
Reading in ‘real’ life
- Scoring rules - Score
Not chosen, but imposed by
testMust be
observable
Conform to the task
Reflects difference between task and observed response
End point of process
LANGUAGE TESTING SERVICES John H.A.L. de Jong 200613
These elements should be relevant to the intended traitThese elements
should interfere
the least
possible
• refers to (part of) a text
• sets task pertaining to the text, requiring particular subskill(s)
• specifies format for overt response demonstrating application of subskill
• specifies rules for scoring response
A Reading item
• may require to link information from the text to prior knowledge and experience
Only then are score differences likely to reflect differences in true ability on the intended trait: i.e. reliable and valid
LANGUAGE TESTING SERVICES John H.A.L. de Jong 200614
Difficulty of comprehension depends on text characteristics
Reader
Aspects of item difficulty
TextComprehension is the interaction between text &
reader ▪ Familiarity▪ # of elements
▪Ling.Characteristics▪Complexity
Difficulty of task depends on prior
knowledge & experience of
candidate
– P
erso
nal /
Pri
vate
– F
orm
al /
Pub
lic
+ +Evaluate / Hypothesize
Find / Connect /Assess
Comprehension
Difficulty of item depends on
scoring rules
Difficulty of item depends on skills
required to perform task
Scoring Guide
Specificity
Difficulty of item depends on
response format Response
Format
LANGUAGE TESTING SERVICES John H.A.L. de Jong 200615
LANGUAGE TESTING SERVICES John H.A.L. de Jong 200616
LANGUAGE TESTING SERVICES John H.A.L. de Jong 200617
PISA 2000: Relation SES & Reading
SES READING
High 540
Medium 500
Low 460
LANGUAGE TESTING SERVICES John H.A.L. de Jong 200618
LANGUAGE TESTING SERVICES John H.A.L. de Jong 200619
Mean PISA 2003 scores per educational track in the Netherlands
LANGUAGE TESTING SERVICES John H.A.L. de Jong 200620
Mean Reading Literacy score for Dutch-born students and for immigrant students
LANGUAGE TESTING SERVICES John H.A.L. de Jong 200621
Language proficiency in upper secondary vocational education
CINOP Study using CEF
• what is the actual language proficiency of students?
• is there a discrepancy between actual and required language proficiency for further education and professional occupation?
LANGUAGE TESTING SERVICES John H.A.L. de Jong 200622
Research design
- written questionnaires for language teachers and vocational teachers, N = 210
- self assessment on language proficiency for students, N= 345
- followed by a test for reading, N= 328
- oral interviews with teachers, staff members and management, N= 40
LANGUAGE TESTING SERVICES John H.A.L. de Jong 200623
Teachers’ views
• 60 to 70 % estimate language proficiency (reading A2-B1) insufficient for school
• 10 % estimate the language proficiency on a higher level than needed for school
• 80 % estimate the language proficiency insufficient for professional occupation
LANGUAGE TESTING SERVICES John H.A.L. de Jong 200624
Self assessment by students
• > 2/3 of students estimate their reading proficiency level at B2
• In reality: 7 % at A1 24 % at A2 52 % at A2-B1 17 % at B1-B2
LANGUAGE TESTING SERVICES John H.A.L. de Jong 200625
CEF appropriate for description of language of instruction• Coherent and workable structure for teachers
(contributes to improvement of their professional competences)
• Competence based descriptions more suitable for language of instruction than linguistic descriptions of L1
• Descriptors applicable
• More transparency and motivating for students (specially in combination with a language portfolio)
• Offers opportunity for reflection on language learning by students
LANGUAGE TESTING SERVICES John H.A.L. de Jong 200626
Shortcomings of CEF in describing language of instruction in vocational education
• Can do statements are geared at academic learning routes: not well adjusted to describing specific tasks in vocational education and specific tasks in the workplace
• More situations and examples have need to be offered for a range of vocational sectors
LANGUAGE TESTING SERVICES John H.A.L. de Jong 200627
“Milestones”
• 2000 1st PISA Cycle: Reading Major Domain
• 2002 Publication Reading for Change
• 2004 CINOP Dutch in Vocational Education
• 2006 ALARM: Language skills at risk