the use of technology in the development of better reading tests:eye
TRANSCRIPT
The use of technology in the
development of reading tests:
Eye-tracking and automated text
analysis
Stephen Bax
Professor in Applied Linguistics
CRELLA
www.beds.ac.uk/crella May 2014
Teaching since 1981
CRELLA 2014
Background
Research into reading, discourse, assessment and
ICT in education
2011 2013
Plan of the talk
1. Cognitive validity in testing reading
2. How can eye tracking help?
3. Research project – IELTS reading tasks
4. Implications for testing, teaching and learning
5. Automated text analysis: TextInspector.com
TESOL Distinguished Research award
www.beds.ac.uk/crella © 2014
The research reported in this article/talk was supported in
part by the ELT Research Award scheme funded by the
British Council to promote innovation in English language
teaching research.
The views expressed are not necessarily those of the
British Council.
1. Cognitive validity
Cognitive validity: “we need to find out if the mental
processes that a test elicits from a candidate resemble
the processes that he/she would employ in non-test
conditions” (Field 2012, emphasis in original).
“Are the processes adopted during a test sufficiently
similar to those which would be employed in the
target context? “
“Or do candidates adopt additional processes that are
a by-product of facets of the test, rather than part of
the normal operations associated with the construct
being tested? “
Model of cognitive processes in reading
Kh
alif
a &
Wei
r (2
00
9)
Exa
min
ing R
ead
ing
, C
UP
Levels of cognitive processing in reading tests (adapted from Khalifa and Weir 2009)
Previous research
Weir, C. J., Hawkey, R. Green, T., Devi, S. (2009) ‘The
cognitive processes underlying the academic reading
construct as measured by IELTS’, British Council/IDP
Australia Research Reports Volume 9: 157-189.
Procedure: Readers completed IELTS reading test items
and then reported back using questionnaires and
interviews.
Examining their cognitive processes
Can eye-tracking help?
Eye tracking
www.beds.ac.uk/crella © 2014
Long history of eye tracking research into reading
www.tobii.com
2. The contribution of eye-
tracking to test validation
Eye tracking can potentially help us to infer cognitive
processes. For example:
• Observe student X’s eye movements
• Student gets correct answer
• Student reports on behaviour
• Contrast with other students
We can therefore infer that this reader has probably used
high-order inferencing strategies of the kind described by
Khalifa and Weir (2009).
Types of reading
Most research has been carried out with L1, on what has
been termed the ‘default’ mode of reading i.e.
“when comprehension is proceeding without difficulty and
the eyes are continuing to move forward along a line of
text” (Reichle et al. 2009:9)
But L2 reading is different
Test reading is different
3. Research project: who
Malaysian first and second-year undergraduates (n=71)
studying at a UK university
an onscreen test
eye movements of a random sample of participants (n=38)
were tracked
further sample (n=20) then completed a Stimulated Recall
interview procedure
3. Research project: RQs
Can eye tracking technology shed light on the cognitive
processing of participants completing onscreen reading test
(IELTS) items?
To what extent and in what ways are successful readers
differentiated from less successful readers in terms of their
cognitive processing on different items?
Eye tracking
Eye tracking
www.beds.ac.uk/crella © 2014
Samples
Eye tracking: successful reader
Eye tracking: unsuccessful reader
Eye tracking
Eye tracking
3. Results
3. Research project: RQs
Can eye tracking technology shed light on the cognitive
processing of participants completing onscreen reading test
(IELTS) items?
YES, with some items
To what extent and in what ways are successful readers
differentiated from less successful readers in terms of their
cognitive and metacognitive processing?
See detailed discussion in article
4. Implications for testing
It is useful to:
1. target each level of cognitive activity systematically, according to Khalifa and Weir’s framework
2. pilot each item, preferably using eye tracking, to see if it is targeting that cognitive process
3. pilot each item, preferably using eye tracking, to see if it distinguishes weak from strong candidates
4. Implications for teaching
It is useful for teachers to
1. practise reading at different levels of Khalifa and Weir’s framework – e.g. word matching, grammar, inferencing, across sentences, across paragraphs etc.
2. practise and develop expeditious reading skills
3. get students to read the question / item closely
4. develop reading holistically as well
Lexis in reading
Lextutor
Many valuable features
Transparent and comprehensive
Issue of ‘word families’ in some lists
e.g. work, working, workman, workmanship
Issue of homographs
Cohmetrix
Many valuable features
Huge range of indices
But: lack of transparency, which hides:
Some odd analysis
Some weak wordlists, e.g. concreteness
TextInspector.com
Caveat
Eye-tracking can give insights
TextInspector can give insights
But of course neither can give a full picture, so….
…. we need to beware of treating technology as a panacea or ‘silver bullet’
