earli 2009, amsterdam roel van steensel (sco-kohnstamm instituut – uva) project 4: reading and...

EARLI 2009, AmsterdamRoel van Steensel (SCO-Kohnstamm Instituut – UvA)

http://salsa.socsci.uva.nl

Project 4: Reading and writing development

Assessing the reading and writing proficiency of at-risk adolescents

in multilingual contexts

Roel van Steensel, Ron Oostdam, Amos van Gelderen and Mirjam Trapman

Study into Adolescent Literacy of Students At-risk

SALSA

2Roel van Steensel, EARLI 2009

Project SALSA

• Large groups of students in secondary education lag behind in literacy skillsOECD (2001)

• Relatively little is known about the factors that influence at-risk students’ literacy developmentCurtis (2002)

• Hence: Project SALSA


Project SALSA

Project 4Reading and writing

development

Environment Student

Project 1Literacy context

in schoolDe Milliano

Project 2Literacy context outside schoolVan Kruistum

Project 3Literacy-related

skills and attitudesTrapman


Project SALSA

• Central research questionWhich factors contribute to/impede the reading and writing development of at-risk adolescents in multilingual contexts?

• Sample– Lower tracks of prevocational secondary education (vmbo): basic

and middle-management program– Grades 7, 8 and 9 (age range: appr. 12-15)– 30% lowest scoring on CITO End of Primary School Test– Year 1 (2007/08): N=63– 32 monolinguals, 31 bilinguals (based on student questionnaire)

• International contextParallel projects in Canada en Switzerland


The test

• Newly developed literacy test:– Reading: text comprehension– Writing: text production

• This presentation: (construct) validity of the reading test


The test

• 9 reading tasks

• Text selection on the basis of:

– Text type: narrative, expository, instructive, argumentative

– Medium: school books, newspapers/magazines, internet, ‘official documents’ (house rules)

– Subject: context-neutral, relevant for the target group


The test

Narrative Expository Instructive Argumentative

School books NAR1 EXP1

Newspapers/ magazines

NAR2 EXP2

EXP3

ARG1

ARG2

Official documents

INS1

Internet INS2


The test

• Item construction on the basis of hypothesized subskills:

– Retrieving: retrieving relevant details from the text

– Interpreting: making inferences from shorter passages in the text (e.g. cause-effect relationships)

– Reflecting: reflecting on larger passages/the text as a whole

Davis (1968); Rosenshine (1980); Goldman & Durán (1988); Alderson (2000); OECD (2003); Song

(2008); Cerdán et al. (2009)


The test

Text comprehension

Local level/microstructure

Retrieving

Interpreting

Global level/macrostructure

Reflecting


Validity study

• Messick (1989) - aspects of (construct) validity:

– Construct (ir-)relevance: does the test only reflect aspects of the construct to be measured? Or are the test results influenced by variables irrelevant to the construct?

– Internal structure: can (theoretical) expectations about relations between items be confirmed?

– External structure: can (theoretical) expectations about relations with other measures be confirmed?


Validity study

• Test administered to 200 students in the lower tracks of prevocational secondary education (school year 2007-08)

• Additional tests administered to a subsample of 63 students


Construct (ir-)relevance

• Assumption: difficulty of the tasks is determined by difficulty of the texts

• Comparison between:– Text difficulty measured by Flesch/Douma readability

index– Task difficulty indicated by mean task score as % of the

maximum task score



Douma Rank Rank Mean/max*100%

NAR2 89.7 1 1 79

NAR1 83.8 2 3 65

EXP2 70.8 3 5 60

INS2 67.9 4 9 51

EXP1 62.2 5 5 60

ARG1 57.2 6 4 64

ARG2 56.3 7 7 57

EXP3 49.7 8 8 52

INS1 48.3 9 2 68



• INS1 en INS2 are both discontinuous texts:– INS1 is a set of house rules

discontinuity makes the task easier

– INS2 is a website

discontinuity makes the task more difficult


Internal structure

Text comprehension

Local level/microstructure

Retrieving

Interpreting

Global level/macrostructure

Reflecting


Internal structure

• Do the data show a:– 1-factor model?– 3-factor model?– 2-factor model?

• Confirmatory factor analysis using Structural Equation Modeling (EQS 6.1)

• Individual test items were combined into 6 parcels (RTR1, RTR2, INT1, INT2, RFL1, RFL2)


Internal structure

1-factor model

Text comprehension

RTR1

RTR2

INT1

INT2

RFL1

RFL2

Excellent fit! χ2=6.27, df=9, p=.713, N=175, SRMR=.026, RMSEA=.000, 90% confidence interval=.000-.060.


Internal structure

3-factor model

Text comprehension

RTR1

RTR2

INT1

INT2

RFL1

RFL2No improvement!Difference score 1-factor/3-factor model: χ2=3.36, df=3, p>.05.

retrieving

interpreting

reflecting


Internal structure

2-factor model

Text comprehension

RTR1

RTR2

INT1

INT2

RFL1

RFL2No improvement!Difference score 1-factor/2-factormodel: χ2=2.41, df=1, p>.05.

local level skills

global level skills


External structure

• Assumption: text comprehension is facilitated by:

– Well-developed decoding skills

– A rich vocabulary

– Well-developed morpho-syntactic skills

– A good working memory

– (Metacognitive) knowledge of reading and writing

• And: text comprehension is associated with IQ


External structure

Variable Correlation with SALSA reading score (N=63)

Decoding skills .22

Receptive vocabulary .59**

Morpho-syntactic knowledge .66**

Working memory .42**

Metacognitive knowledge .41**

Nonverbal IQ .36**

** p<.01


Conclusions: support for validity?

• Construct (ir-)relevance:

Task difficulty largely corresponds with text difficulty– Exceptions (INS1, INS2) are explainable

• Internal structure:

No support for the presence of subskills

Data do show that the test measures one construct

• External structure:

Substantial correlations with other skills– Exception (decoding skills) is explainable


• Thank you for your attention!

• Contact?– [email protected]– Or visit our website: http://salsa.socsci.uva.nl/

earli 2009, amsterdam roel van steensel (sco-kohnstamm instituut – uva) project 4: reading and...

Documents

reading test slide

project salsa slide

project salsa project

school van kruistum

developed literacy test

literacy context

test results

amos van gelderen