ltrc 2007 messick address slide 1 june 9, 2007 toward a test theory for the interactionalist era...
Post on 21-Dec-2015
214 views
TRANSCRIPT
LTRC 2007 Messick Address Slide 1June 9, 2007
Toward a Test Theory for the Interactionalist Era
Robert J. MislevyUniversity of Maryland
Samuel J. Messick Memorial LecturePresented at the Language Testing Research Colloquium, Barcelona, Spain, June 9, 2007.
LTRC 2007 Messick Address Slide 2June 9, 2007
Chi, Feltovich, & Glaser, 1981
LTRC 2007 Messick Address Slide 3June 9, 2007
Key Ideas
Generative patterns from the domain organized experts’ perceptions, understanding, and actions
Experts assembled pieces from patterns to fit particulars of context and purpose» F=MA: Conventional v. situated meaning
Critical role of narrative layer» Integrates principles & equations with context » Locus for understanding, planning, & action
LTRC 2007 Messick Address Slide 4June 9, 2007
Why Am I Telling You This?
Situative / sociocultural / interactionist perspective is frontier in all domains
Analogues between model-based reasoning and using language
Test theory is model-based reasoning» Interactionalist perspective on test theory
» Bottleneck the narrative layer
LTRC 2007 Messick Address Slide 5June 9, 2007
Outline
Messick on assessment arguments Interactionalist perspective
» Re language, comprehension, cultural meaning
A narrative space / metaphor for assessment in this light» Attention to senses and roles of context
Implications for building and using measurement models
LTRC 2007 Messick Address Slide 6June 9, 2007
The Assessment Argument What complex of knowledge, skills, or other
attributes should be assessed? What behaviors or performances should
reveal those constructs? What tasks or situations should elicit those
behaviors? Messick (1994) “The interplay of evidence and consequences in
the validation of performance assessments”
LTRC 2007 Messick Address Slide 7June 9, 2007
Perspectives on the L2 Construct
What is foregrounded? Trait
» Focus on underlying abilities of individuals that are called upon in a wide variety of situations.
Behaviorist» Focus on context, from external point of view—
success of action in specified classes of situations.
Interactionalist …
LTRC 2007 Messick Address Slide 8June 9, 2007
An Interactionalist Perspective
Concern with language used in specific discursive practices rather than on language ability independent of context.
Focus on the co-construction of discursive practices by all participants ...
A set of general interactional resources that participants draw upon in specific ways in order to co-construct a discursive practice.
(Young, 2000, p. 5)
LTRC 2007 Messick Address Slide 9June 9, 2007
An Interactionalist Perspective
successful interaction presupposes not only a shared knowledge of the world, …
but also the construction of a shared internal context … that is built through the collaborative efforts of the interactional partners.
Kramsch ( 1986, p. 367)
LTRC 2007 Messick Address Slide 10June 9, 2007
Challenges for Assessment Amending the construct of individual ability
to accommodate [how] language use in a communicative event reflects dynamic discourse, which is co-constructed among participants; and …
reconciling [the notion that language ability is local] with the need for assessments to yield scores to generalize across contextual boundaries.
Chalhoub-Deville (2003, p. 373)
LTRC 2007 Messick Address Slide 11June 9, 2007
A Narrative Frame
Themes from, e.g., cognitive psychology, literacy, neuroscience, anthropology: » Connectionist metaphor, Associative memory
Situated cognition & information processing» Construction-Integration (CI) theory of
comprehension (Kintsch and others)
Individual Sociocultural perspectives» A cognitive theory of cultural meaning
(Strauss & Quinn, 1997)
LTRC 2007 Messick Address Slide 12June 9, 2007
A Cognitive Theory of Cultural Meaning “Interactional Resources”
External: knowledge “out there”, patterns that exist in use by groups, in tools & processes» Cultural models: What ‘being sick’ means,
restaurant script, Newton’s laws, complaints» Linguistic: Grammar, conventions, frames
Interactional: enable the co-construction of new shared meanings
Internal: patterns in individuals attuned to shared external patterns
LTRC 2007 Messick Address Slide 13June 9, 2007
Inheritance from Schema Theory
Knowledge as organized patterns, at many levels…
Assembled to understand & to create particular
situations in the world
Developed, strengthened, modified by use Associations of all kinds, including applicability,
affordances, procedures, strategies, affect
» “The user’s knowledge of the language rules is interlocked
with his knowledge of when, where, and with whom to use
them” (Ellis, 1985)
LTRC 2007 Messick Address Slide 14June 9, 2007
B Inside BInside AContext
A
A la Kintsch: “Conventional” meaning, or propositional content of text / speech…
A la Kintsch: “Conventional” meaning, or propositional content of text / speech…
and all aspects of context…and all aspects of context…
LTRC 2007 Messick Address Slide 15June 9, 2007
B Inside BInside AContext
A
The C in CI theory, Construction:Activation of both relevant and irrelevant …Linguistic models, e.g.
•Conventions, Rhetorical frames Cultural models, e.g.,
•Equilibrium, Human motivation
The C in CI theory, Construction:Activation of both relevant and irrelevant …Linguistic models, e.g.
•Conventions, Rhetorical frames Cultural models, e.g.,
•Equilibrium, Human motivation
•If B hasn’t developed a given pattern in past experience, it can’t be activated (although it may get constructed in the interaction).•Relevant patterns from LTM may be activated in some contexts but not others (e.g., physics models; question formation (Tarone)).
•If B hasn’t developed a given pattern in past experience, it can’t be activated (although it may get constructed in the interaction).•Relevant patterns from LTM may be activated in some contexts but not others (e.g., physics models; question formation (Tarone)).
LTRC 2007 Messick Address Slide 16June 9, 2007
B Inside BInside AContext
A
The I in CI theory, Integration:•Resulting synthesis of reinforced activated cultural / linguistic / situational patterns:•the Situation model•Akin to ‘stories’ in Larkin’s physics study•Situation model is the understanding
The I in CI theory, Integration:•Resulting synthesis of reinforced activated cultural / linguistic / situational patterns:•the Situation model•Akin to ‘stories’ in Larkin’s physics study•Situation model is the understanding
LTRC 2007 Messick Address Slide 17June 9, 2007
B Inside BInside AContext
A
Situation model is also the basis of planning and action.
Situation model is also the basis of planning and action.
LTRC 2007 Messick Address Slide 18June 9, 2007
B Inside BInside AContext
Context
Context
Context
A
LTRC 2007 Messick Address Slide 19June 9, 2007
A B Inside BInside AContext
Context
Context
Context
Ideally, participants activate cultural &
linguistic models that are compatible in relevant ways…
Ideally, participants activate cultural &
linguistic models that are compatible in relevant ways…
LTRC 2007 Messick Address Slide 20June 9, 2007
A B Inside BInside AContext
Context
Context
Context
To lead to co-constructed meaning / sufficiently (?)
shared understanding
To lead to co-constructed meaning / sufficiently (?)
shared understanding
Kramsch’s "shared internal context"
Kramsch’s "shared internal context"
LTRC 2007 Messick Address Slide 21June 9, 2007
A B Inside BInside AContext
Context
Context
Context
Preceeding overheads highlight the importance of a common narrative space for thinking about assessment:A = LinguistB = Psychometrician
Preceeding overheads highlight the importance of a common narrative space for thinking about assessment:A = LinguistB = Psychometrician
“shared internal context” re nature & use of knowledge would help ground compatible views of assessment purpose, design, analysis, and use for the job at hand.
“shared internal context” re nature & use of knowledge would help ground compatible views of assessment purpose, design, analysis, and use for the job at hand.
LTRC 2007 Messick Address Slide 22June 9, 2007
Senses and roles of “context”A B Inside BInside A
Context
Context
Context
Context
Can distinguish external and internal aspects of context (e.g., Douglas, 2000)
Can distinguish external and internal aspects of context (e.g., Douglas, 2000)
Some external aspects of context are public & objective, e.g.,
•Setting•Physical attributes•Directives
Some external aspects of context are public & objective, e.g.,
•Setting•Physical attributes•Directives
Target language use (TLU) featuresAssessment task features(Bachman & Palmer)
Target language use (TLU) featuresAssessment task features(Bachman & Palmer)
LTRC 2007 Messick Address Slide 23June 9, 2007
A B Inside BInside AContext
Context
Context
Context
Some aspects of context can be interpreted by an external observer through the lens of targeted cultural/linguistic models. e.g., apology situation. In assessment, we can often arrange for this to be the case.
Some aspects of context can be interpreted by an external observer through the lens of targeted cultural/linguistic models. e.g., apology situation. In assessment, we can often arrange for this to be the case.
The question at issue in assessment is whether the examinee activates targeted compatible l/c models, then constructs and acts accordingly through a corresponding situation model.
The question at issue in assessment is whether the examinee activates targeted compatible l/c models, then constructs and acts accordingly through a corresponding situation model. Note the need to activate many other l/c
models in order to construct a situation model, plan, and carry out action.•Many places to slip, but others to compensate.•“Alternative explanations” in assessment argument.
Note the need to activate many other l/c models in order to construct a situation model, plan, and carry out action.•Many places to slip, but others to compensate.•“Alternative explanations” in assessment argument.
LTRC 2007 Messick Address Slide 24June 9, 2007
A B Inside BInside AContext
Context
Context
Context
Some aspects of context can be interpreted by an external observer through the lens of targeted cultural/linguistic models. e.g., apology situation. In assessment, we can sometimes arrange for this to be the case; sometimes watch for it to happen.
Some aspects of context can be interpreted by an external observer through the lens of targeted cultural/linguistic models. e.g., apology situation. In assessment, we can sometimes arrange for this to be the case; sometimes watch for it to happen.
As assessment designers, we use these situations that call upon targeted linguistic/ cultural models to determine what examinee actions would signal recognition, comprehension, action through them.
As assessment designers, we use these situations that call upon targeted linguistic/ cultural models to determine what examinee actions would signal recognition, comprehension, action through them.This sense of context plays a key role in
•Evaluation of performance, hence •Observable variables that go into a measurement model.
This sense of context plays a key role in•Evaluation of performance, hence •Observable variables that go into a measurement model.
LTRC 2007 Messick Address Slide 25June 9, 2007
What can we say about individuals?
Use of interactional resources in appropriate contexts in appropriate ways; i.e.,
Attunement to targeted cultural/linguistic patterns: Recognize markers of externally-viewed patterns? Construct internal meanings in their light? Act in ways appropriate to targeted c/l models in the
assessment contexts? What are the ranges and circumstances of activation?
(variation of performance across contexts)
LTRC 2007 Messick Address Slide 26June 9, 2007
Implications for measurement models
Basic form: Probability of aspects of performance Xij given
parameters for person i and situation j
(all could be vector-valued)
jiij ,XProb
• Way too simple • No explicit connection with CI comprehension model, interaction processes, etc.• Apparent separation of person and situation characteristics
These are indeed properties of the conventional meaning of the measurement model and parameters.
These are indeed properties of the conventional meaning of the measurement model and parameters.
LTRC 2007 Messick Address Slide 27June 9, 2007
An Interactionalist Perspective: Instantiation in a Context
Xs result from particular persons calling upon resources in particular contexts (or not, or how)
Mechanically s simply accumulate info across situations
Our chosing situations and what to observe drives their situated meaning.
Situated meaning of s are tendencies toward these actions in these situations that call for certain interactional resources, via l/c models.
LTRC 2007 Messick Address Slide 28June 9, 2007
An Interactionalist Perspective: Instantiation in a Context
Inference to criterion contexts (TLU) depends on analysis of what l/c models are called upon in assessment use argument…
What is similar, what is different, re the resources task & criterion situations call for?
To what degree does activation and success in task context correspond to activation and success in criterion context?
(e.g., Bachman, Chalhoub-Deville, Douglas, Chapelle)
LTRC 2007 Messick Address Slide 29June 9, 2007
An Interactionalist Perspective: Instantiation in a Context
What demands do we minimize via task design, so needn’t model?
What resources do we already know examinees can draw upon, so tasks can require them but we needn’t model?» “Hidden” but essential to meaning» Occupational English Test (McNamara)» Analogous to ‘focus on forms’ learning
LTRC 2007 Messick Address Slide 30June 9, 2007
An Interactionalist Perspective: Instantiation in a Context
How to model inconsistent performance? If “unmotivated,” it’s noise; via probability model Promising direction: Model individual’s degree or
pattern in variation in terms of context features If “motivated”: Model in terms of s
» Divide & Conquer: Multiple unidimensional tests (OET)
» Exploratory multidimensional: Discover patterns in data. » Controlled: Structured multidimensional models
(e.g., Embretson, Adams & Wilson, von Davier)
LTRC 2007 Messick Address Slide 31June 9, 2007
Structured Multidimensional Models
Still way too simple, but …» purposeful modeling of motivated variation in
complex tasks when persons differ in targeted ways» exploit what is known about examinees but not
modeled
Controlled mixes of demand features» E.g., in OET-like situations, wrt medical knowledge,
complexity of stimulus language, complexity of language to be produced.
“Throwing the data over the wall” won’t work
LTRC 2007 Messick Address Slide 32June 9, 2007
Structured Multidimensional Models
Probabilities modeled in terms of task features:» Which dimensions are relevant for which observables
from which tasks? (Robinson’s “difficulty” features)» Task parameters modeled in terms of Robinson’s
“complexity” features.
Hence a priori structure of patterns to interpret » Can organize s in terms of traits or context features
Coordinated task design and measurement model» Create tasks within task models
LTRC 2007 Messick Address Slide 33June 9, 2007
How much can testing gain from modern cognitive psychology?
So long as testing is viewed as something that takes place in a few hours, out of the context of instruction, and for the purpose of predicting a vaguely stated criterion,
then the gains to be made are minimal. Buzz Hunt (1986)
Conclusion
LTRC 2007 Messick Address Slide 34June 9, 2007
I have argued that we need to capitalize on [method effects] by designing tests for specific populations -- tests that contain instructions, content , genre, and language directed toward that population.
The goal is to produce tests … that would provide information interpretable as evidence of communicative competence in context.
Douglas (1998)
Conclusion
LTRC 2007 Messick Address Slide 35June 9, 2007
Conclusion
Interactionalist view of test theory… for assembling, analyzing, and interpreting
assessments, for arguments in interactionalist view of language Methods and exemplars needed,
but more pressing need is narrative frame … » To connect view of language proficiency with the
machinery of test theory, » Toward modeling purposeful variations in a coherent
design space.