art graesser department of psychology and the institute for intelligent systems assessments of...
Post on 18-Dec-2015
218 Views
Preview:
TRANSCRIPT
Art Graesser
Department of Psychology and the Institute for Intelligent Systems
Assessments of Reading at Deeper Levels of Comprehension
Main Messages
• We need assessments of reading comprehension that systematically tap deeper levels of comprehension.
• Good deep measures integrate reading skills/strategies with world knowledge and nonlinguistic components.
Levels of Language & Discourse
• Graphemes and phonemes• Morphemes and words• Syntactic composition • Linguistic style and dialect• Explicit propositions• Referents of referring expressions• Common ground • Discourse focus versus
presuppositions• Situation models & inferences• Embedded dialog• Configuration of multiple agents• Genre, registers, rhetorical
structures• Plot configurations • Local and global coherence• Point of message -- theme• Goals and attitude of author
Five major categories (Kintsch, 1998; Graesser, Millis, & Zwaan, 1997)
1)Surface code
2)Propositional textbase
3)Situation model
4)Text genre and rhetorical structure
5)Pragmatic communication
What do we mean by deep comprehension?
Cognitive Processes Bloom (1956)
Recognition
Recall
Comprehension (Inference)
Application Analysis
Synthesis
Evaluation
Abstractness of Content (Mosenthal, 1996)
(1) Most concrete: concrete person, action, thing
(2) Highly concrete: observable attributes (action, types, attributes, amounts)
(3) Intermediate: Procedures, goals, and manner
(4) Highly abstract: cause, effect, reason, evidence
(5) Most abstract: inferred theories & themes
Gates-MacGinitie Reading Test
0
0.1
0.2
0.3
0.4
0.5
Levels 1 & 2 Levels 4 & 5
Good Questions that Tap Pragmatic Communication
(Questioning the Author , Beck, McKeown, Hamilton, & Kucan, 1997)
• What is the author trying to tell you?
• Why is the author telling you that?
• Does the author say it clearly?
• How could the author have said things more clearly?
• What would you say instead?
Domain KnowledgeWisher & Graesser (2007)
People(Agents)
Taxonomies
Spatial Layout
Compositional Structure
Procedures & Plans
Causal Networks
Other
Deep-Level Reasoning Questions (Graesser and Person,1994)
LEVEL 1: SIMPLE or SHALLOW 1. Verification Is X true or false? Did an event occur?2. Disjunctive Is X, Y, or Z the case?3. Concept completion Who? What? When? Where?4. Example What is an example or instance of a category?
LEVEL 2: INTERMEDIATE 5. Feature specification What qualitative properties does entity X have?6. Quantification What is the value of a quantitative variable? How much? 7. Definition questions What does X mean?8. Comparison How is X similar to Y? How is X different from Y?
LEVEL 3: COMPLEX or DEEP9. Interpretation What concept/claim can be inferred from a pattern of data?10. Causal antecedent Why did an event occur? 11. Causal consequence What are the consequences of an event or state? 12. Goal orientation What are the motives or goals behind an agent’s action?13. Instrumental/procedural What plan or instrument allows an agent to accomplish a goal? 14. Enablement What object or resource allows an agent to accomplish a goal?15. Expectation Why did some expected event not occur?16. Judgmental What value does the answerer place on an idea or advice?
Analysis of 120 Multiple Choice Questions in Cognitive Psychology
• 120 MCQ items of cognitive psychology textbooks – 4 test banks– 30 items per textbook
• Items coded on the level in Graesser & Person taxonomy.
0
0.1
0.2
0.3
0.4
0.5
Shallow level Deep Level
Challenges
• How can we promote more diversity in the landscape of questions?
• How can we encourage deeper questions?
• How do we change:– Textbook writers– Professors &
teachers– Students
Cognitive and Behavioral Tests in Experimental Psychology
OFF-LINERecall SummarizationRetrospective think aloudRecognition testsCloze procedure Multiple choice testsSentence verificationQuestion asking &
answeringResponse signal
paradigmRatingsWord sorting
ON-LINEImmediate think aloud Self-paced reading timesEye trackingWord naming latencies Lexical decision latencies RSVP-SOAPhysiological recordings
fMRI Evoked potential
Open-ended Questions
The sun exerts a gravitational force on the earth as the earth moves in its orbit around the sun. Does the earth pull equally on the sun? Explain why?
EXPECTATIONSThe sun exerts a gravitational force on the earth. The earth exerts a gravitational force on the sun. The two forces are a third-law pair.The magnitudes of the two forces are the same.
MISCONCEPTIONSOnly the larger object exerts a force. The force of earth on sun is less than that of the sun on earth.
Virtues of the Multiple Choice Format
• Familiar to the educational enterprise– School systems– Textbook companies– Psychometric community
• Easy to score
• Objectively and systematically scored
• Distracters add complexity
Analytical Schemes
• Categories of distractors– Near miss with pedagogical point (versus obscurity)– Thematic versus unrelated distracters
• Relation to associated text– Local versus global– Explicit versus inference– Main rhetorical content versus details
• Qualitative causal analysis – How does a change in A affect B?– Increase, decrease, no change, versus indeterminate
What happens to the pins when the key is turned to unlock the door? A) they drop B) they rise C) they remain stationary
(Graesser & Olde, 2003)
Closing Comments
• Rand Reading Study Group (Snow, 2002) emphasized comprehension– Text– Reader– Activities– Sociocultural context
• Training effective strategies of reading comprehension requires analysis of world knowledge and deeper levels of processing (edited by McNamara, 2007)
?
top related