margje van de wielfaculty of psychology, universiteit maastricht thinking-aloud & protocol...
TRANSCRIPT
Margje van de Wiel Faculty of Psychology, Universiteit Maastricht
Thinking-Aloud &
Protocol Analysis
Margje van de WielDepartment of Experimental Psychology
Universiteit Maastricht
Unit 3.7.1Macro-Ergonomics: Evaluating human
work
Margje van de Wiel Faculty of Psychology, Universiteit Maastricht
Overview
• Research on cognitive processes underlying cognitive task performance:– Problem solving – Knowledge representation
• Focus on the think-aloud method– Collecting think-aloud protocols– Analyzing think-aloud protocols
Margje van de Wiel Faculty of Psychology, Universiteit Maastricht
Applications of cognitive psy-chological research methods• Fundamental psychological research
– Explorative– Testing theoretical hypotheses– Compare tasks or groups
• Cognitive task analysis• Building knowledge based systems• Usability research
Margje van de Wiel Faculty of Psychology, Universiteit Maastricht
Usability research
• How easy is it for people to set the time and alarm on a clock without a manual?
• How easy is it for them with the manual?• Do people understand the manual?• Do they use the manual?
• Video, computer program, web site etc. Compare conditions or groups
Margje van de Wiel Faculty of Psychology, Universiteit Maastricht
Problem solvingTower of HanoiThe objective of this game is to move all of the rings to the right most peg.You may only move one ring at a time; you must never allow a large ring to rest on a smaller ring.
Margje van de Wiel Faculty of Psychology, Universiteit Maastricht
Problem space (Newell & Simon, 1972)
Initial state
Goal state
Subgoalstate
Subgoalstate
Inter-mediate
states
Inter-mediate
states
Inter-mediate
states
Inter-mediate
states
Margje van de Wiel Faculty of Psychology, Universiteit Maastricht
Research on problem solving• Product evaluation• Observation and registration of
behavior (real-life, video, log files)
But Gives no direct information on cognitive processes
Margje van de Wiel Faculty of Psychology, Universiteit Maastricht
Research on problem solving• Introspection= Direct observation of mind
ButParticipants may give their interpretation of their cognitive processes when they prefer during task performance
Margje van de Wiel Faculty of Psychology, Universiteit Maastricht
Research on problem solving• Retrospection
ButParticipants may give their interpretation of their cognitive processes after task performance
Margje van de Wiel Faculty of Psychology, Universiteit Maastricht
Research on problem solving• Free recall • Stimulated recall
ButParticipants give their memories of the thoughts they had during problem solving after task performance
Margje van de Wiel Faculty of Psychology, Universiteit Maastricht
Instructions
• Retrospection– How did you proceed in solving the
problem?– What did you think during solving the
problem?
• Stimulated / Free recall– Report everything you can remember
about your thoughts during solving the problem
Margje van de Wiel Faculty of Psychology, Universiteit Maastricht
Research on problem solving• Questioning and prompting
ButParticipants may give their interpretation of their cognitive processes when prompted so that natural task performance may be disturbed
Margje van de Wiel Faculty of Psychology, Universiteit Maastricht
Research on problem solving• Dialogues in explanation and collaboration
ButThe cognitive processes might be different than in natural task performance
Margje van de Wiel Faculty of Psychology, Universiteit Maastricht
Research on problem solving• Thinking aloud=Saying everything that comes to mind
Advantages:• Concurrent verbalization• Close to natural task performance• Discourages interpretation and
explanation
Margje van de Wiel Faculty of Psychology, Universiteit Maastricht
Theoretical framework / verbalization theory (Ericsson & Simon, 1993)
=the information processing model
LTM
Verbal report
STM Taskperformance
Inputinfo
Margje van de Wiel Faculty of Psychology, Universiteit Maastricht
Theoretical framework
Verbal report
STM Taskperformance
Verbal encoding
VocalizationType of task- modality- verbalizability- instruction
Margje van de Wiel Faculty of Psychology, Universiteit Maastricht
The type of problem and the role of knowledge
Well-defined problems Ill-defined problems
Weak problem solving methods
Strong problem solving methods
Novices Experts
Knowledge lean task Knowledge rich task
Margje van de Wiel Faculty of Psychology, Universiteit Maastricht
Focus on knowledge representationThe nature and organization of knowledge is central in problem solving and not the strategies used (e.g. Chi, 1997)
The representation of a problem determines the problem solving process
Margje van de Wiel Faculty of Psychology, Universiteit Maastricht
Research on knowledge representation• Same methods as for investigating
cognitive processes, but analysis of the knowledge used
• Free recall– Report everything you remember about ...
=Reflection of the representation of (prior) knowledge, a text or a problem in the mind
Margje van de Wiel Faculty of Psychology, Universiteit Maastricht
The think-aloud method
• Collecting think-aloud protocols• Transcribing the verbal protocols Verbal reports as data• Segmenting the transcribed
protocols• Analyzing the transcribed protocols
Margje van de Wiel Faculty of Psychology, Universiteit Maastricht
Research questions
Gaining insight in cognitive processes and knowledge structures
• Explorative• Testing hypotheses
• Comparing types of problems• Comparing groups
Margje van de Wiel Faculty of Psychology, Universiteit Maastricht
Collecting think-aloud protocolsWhat do you need?• Experimenter• Participant• Problem• Tape-recorder and tape• Instructions
Margje van de Wiel Faculty of Psychology, Universiteit Maastricht
Instructions
• Make participant feel comfortable• Explain the purpose of the research• Explain the task• Explain thinking aloud• Emphasize that it is no test situation • Emphasize that the data are confidential• Practice problem(s) and talking out loud• Give feedback
Margje van de Wiel Faculty of Psychology, Universiteit Maastricht
Transcribing the protocols
• Type out the protocols as literally as possible, including – Natural punctuation– Eh, ehs– Thinking pauses; with dots or even better
to measure length– Indicate when something is said mumbling
so you don’t understand well
• Indicate interventions of experimenter
Margje van de Wiel Faculty of Psychology, Universiteit Maastricht
Example Tower of Hanoi (Anzai & Simon, 1979)
I’m not sure, but first I’ll take 1 from A and place it on B. And I’ll take 2 from A and place it on C. And then, I take 1 from B and place it n C. Experimenter: If you can, tell me where you place it there?Because there was no place else to go, I had to place 1 from B and C. Then, next, I placed 3 from A to B. …..
Margje van de Wiel Faculty of Psychology, Universiteit Maastricht
Segmenting the protocol
• Use natural pauses indicated by punctuation
• Indicate each segment by a line number
• No interpretation
Margje van de Wiel Faculty of Psychology, Universiteit Maastricht
Example Tower of Hanoi (Anzai & Simon, 1979)
1. I’m not sure, 2. but first I’ll take 1 from A and place it on B. 3. And I’ll take 2 from A and place it on C. 4. And then, I take 1 from B and place it on C. Experimenter: If you can, tell me where you place it there?5. Because there was no place else to go,.. …..
Margje van de Wiel Faculty of Psychology, Universiteit Maastricht
Analyzing verbal protocols
(Transcribed) and segmentedverbal protocols
Interpretation
RAW DATA
DATA
Margje van de Wiel Faculty of Psychology, Universiteit Maastricht
Interpretation
• Qualitative analysis
• Analyzing qualitative data in an objective and quantifiable way
Compare experimental conditions– Groups– Tasks– Instructions
Margje van de Wiel Faculty of Psychology, Universiteit Maastricht
Research question --> qualitative analysis• What comprehension strategies do
skilled readers use in reading a text?
Thinking aloud while reading a text Analysis: What strategies are used?
– Exploratory: describe the strategies used by the participants
Margje van de Wiel Faculty of Psychology, Universiteit Maastricht
Research questions --> testing hypotheses (Rauenbush & Bereiter, 1991)• What strategies do skilled readers use in
dealing with degraded text?• Are these comparable to comprehension
strategies used in reading normal text? Thinking aloud while reading text with 3
types of instruction:– Just reading– + ‘Adults skim this type of material first’– Skim text for 60 sec and report the topic
Margje van de Wiel Faculty of Psychology, Universiteit Maastricht
Research questions --> testing hypotheses (Rauenbush & Bereiter, 1991) Analysis: what strategies are used?• Normal comprehension strategies
– Rereading – Reading ahead– Summing up
• Other comprehension strategies– (Letter generating strategy)– (Determining word type)
Margje van de Wiel Faculty of Psychology, Universiteit Maastricht
Analyzing verbal protocols 1
HypothesesCodingscheme
Verbalprotocols
Explore Test
Psychological theoryCognitive task analysis
Margje van de Wiel Faculty of Psychology, Universiteit Maastricht
Analyzing verbal protocols 1
HypothesesCodingscheme
Verbalprotocols
Problem solving <--> Knowledge representationprocesses (E&S, 93) (Chi, 97)
Margje van de Wiel Faculty of Psychology, Universiteit Maastricht
Cognitive task analysis
• What knowledge is available in the problem?
• What knowledge is necessary for problem solving?
• What steps need to be taken to solve the problem?
• What is the most efficient problem solving process?
• What strategies can be used?
Margje van de Wiel Faculty of Psychology, Universiteit Maastricht
Coding scheme
• Operationalizing hypotheses in coding categories
• Clearly defining coding categories and providing examples from pilot protocols
• Categories may differ in level of detail– Segments– Aggregations of segments
Margje van de Wiel Faculty of Psychology, Universiteit Maastricht
Analyzing verbal protocols 2• Code the verbal protocols• Determine interrater reliability (% of
correspondences, kappa, correlation)• Depict the results• Seek and interpret patterns in results
Eventually• Repeat process
Margje van de Wiel Faculty of Psychology, Universiteit Maastricht
Example medical diagnosis• Clinical case
Margje van de Wiel Faculty of Psychology, Universiteit Maastricht
Example of a case
A 70-year old female is admitted into hospital because of increasing shortness of breath. She has been very tired lately and tolerates her food badly. Sometimes she has chest pain, especially after dinner.Et cetera
Margje van de Wiel Faculty of Psychology, Universiteit Maastricht
Example medical diagnosis• Clinical case• Thinking aloud while diagnosing
the case– Problem solving process– The nature of the knowledge used
Margje van de Wiel Faculty of Psychology, Universiteit Maastricht
Analysis of think-aloud protocol
Problem solving process• Data examination• Data exploration• Data explanation• Hypothesis generation• Hypothesis evaluation• Discrepancy
processing• Meta-reasoning• Summarization
Nature of knowledge used
• Data interpretation• Clinical reasoning• Biomedical reasoning• Hypotheses (C or B)• (Repetitions)• Plans for actions• Metacognitive remarks
Margje van de Wiel Faculty of Psychology, Universiteit Maastricht
Example medical diagnosis• Clinical case• Thinking aloud while diagnosing
the case– Problem solving process– The nature of the knowledge used
• Free recall– Case representation
Margje van de Wiel Faculty of Psychology, Universiteit Maastricht
Analysis of free recall protocolsProposition analysis• Number of correct propositions• Number of inferences
(summarizing two or more propositions)
Margje van de Wiel Faculty of Psychology, Universiteit Maastricht
Proposition analysis
Proposition = single idea unit
Concept ConceptQualifier
CausalConditionalAttributeSpecificationIs a
Margje van de Wiel Faculty of Psychology, Universiteit Maastricht
Propositions in the case
Female - 70-year old (attribute)
- is admitted into hospital (attribute) - (because of) shortness of breath (attribute)
- increasing (specification)
- very tired (attribute)
- lately (specification)
- tolerates her food badly (attribute)
Margje van de Wiel Faculty of Psychology, Universiteit Maastricht
Other literature
• Chi, M.T.H. (1997). Quantifying qualitative analyses of verbal data: A practical guide. The journal of the learning sciences, 6(3), 271-315. SGT 3179
• Ericcsson, K. A. & Simon, H. A. (1993). Protocol analysis: verbal reports as data. Cambridge, MA: MIT Press (ch 6 & 7). SL BF 180
Margje van de Wiel Faculty of Psychology, Universiteit Maastricht
Other literature
• Reitman Olson, J. & Biolsi, K. J. (1991). Techniques for representing expert knowledge. In: K. A. Ericsson & J. Smith (Eds.) Toward a general theory of expertise: Prospects and Limits (pp. 240-285). Cambridge: Cambridge University Press. ME EAM 374 / Margje