analyzing verbal protocols: thinking aloud during reading

Analyzing Verbal Protocols:

Thinking Aloud During Reading in

Cognitive and Educational

Psychology

© 2019 SAGE Publications, Ltd. All Rights Reserved.

This PDF has been generated from SAGE Research Methods Datasets.

Analyzing Verbal Protocols:

Thinking Aloud During Reading in

Cognitive and Educational

Psychology

Student Guide

Introduction

This guide illustrates how to code and analyze verbal protocol data for participants

who are thinking aloud about a text, with the goal of better understanding the

processes involved in quality reading comprehension. In think-aloud tasks,

participants either verbalize or write their thoughts as they read a text or engage

in a task. The data represent a sample from Bohn-Gettler and Kendeou (2014),

who examined how reading is multifaceted and influenced by many interacting

variables. Their study examined three variables. The first was expository text

structure, which refers to the formats of non-narrative texts, including

compare–contrast, problem–response, descriptive, or chronological formats. The

second variable was reader goals, which refers to whether readers were asked

to read for study or entertainment. The final variable was working memory, which

refers to how much content an individual can mentally process at one time.

The study documented the importance of studying the individual and interactive

influences of multiple variables on text comprehension, as comprehension is a

complex process occurring in multidimensional settings and with diverse readers.

Importantly, Bohn-Gettler and Kendeou (2014) utilized the think-aloud approach to

understand the processes associated with quality comprehension. This is a form

SAGE

2019 SAGE Publications, Ltd. All Rights Reserved.

SAGE Research Methods Datasets Part

2

of 13 Analyzing Verbal Protocols: Thinking Aloud During Reading in Cognitive

and Educational Psychology

of verbal protocol research in which participants state their thoughts out loud as

they read a text. The participants’ verbal statements were transcribed and coded

for different cognitive processes that commonly occur during reading, such as

paraphrasing, filling in missing information, making predictions, incorporating prior

knowledge, stating opinions, making loose associations, reflecting on one’s own

understanding, and more. The primary focus of this guide is to consider how to

effectively code think-aloud data to better understand the processes involved in

quality reading comprehension.

Think-Alouds

Think-alouds are a form of verbal protocol (Ericsson & Simon, 1993) useful for

examining the mental strategies (e.g., processes) in which participants engage.

The rich data can reveal direct thoughts, beliefs, strategies, and more. In classic

think-alouds, participants say everything that comes to their mind while reading

or engaging in a learning task. Because this can sometimes result in stream-

of-consciousness responses, researchers sometimes ask readers to engage in

a specific strategy while thinking aloud. Think-alouds are believed to reveal the

spontaneous processes in which readers engage, that is, what participants are

thinking at each moment, in a relatively unfiltered manner. (Note that, although

think-alouds can be written instead of verbalized, participants may not write

unspoken strategies.) For more information about think-alouds, see Bohn-Gettler

(2018).

Researchers use think-alouds with many age groups and across a variety of

arenas, such as education, psychology, marketing, human factors, and more. The

only requirement is the ability to speak. Collecting think-aloud data is relatively

simple: An experimenter demonstrates thinking aloud with a practice text, and

the participant practices before engaging in the experimental task. Verbal think-

alouds are typically recorded and transcribed. To reduce biasing the participants’

SAGE



2



responses, the researcher should be careful to not provide feedback other than

encouragement to verbalize thoughts at the prompted points. Think-alouds are

flexible such that the frequency with which participants think-aloud can vary. For

example, Bohn-Gettler and Kendeou (2014) asked participants to think aloud after

every sentence, but other studies require thinking aloud at specific points (e.g.,

Bohn-Gettler & McCrudden, 2018).

Thinking Aloud With Expository Texts With College Students

For this data exemplar, an original, anonymized think-aloud transcript from Bohn-

Gettler and Kendeou (2014) is provided. Recall that Bohn-Gettler and Kendeou

studied the strategies and processes in which readers engaged during the

moment-by-moment reading of expository (e.g., non-narratives) texts with

different formats. The goal was to determine what factors led to higher quality

comprehension. To accomplish this, they examined how reading is multifaceted,

such that text format can interact with reading goals and working memory to

influence the quality of comprehension strategies. This type of research question

is particularly important because reading and learning occur in complex settings,

yet many studies do not account for the numerous factors that can influence

comprehension. Thinking aloud was the best method for exploring the

spontaneous strategies in which readers engage. Because the researchers were

interested in the processes associated with quality comprehension, they coded

participants’ think-alouds for the strategies associated with successful versus

unsuccessful comprehension.

The participants were college students at a Midwestern university. This population

was of particular interest because college students must often interpret non-

narrative texts in varying formats in order to learn content. In addition, the

strategies college students use when engaging with texts may vary, which can

influence learning. The think-aloud samples contain the transcriptions of

SAGE



2



participants’ verbalizations while reading non-narrative texts of different formats

(compare–contrast, problem–response, chronological, and descriptive). The study

occurred in a laboratory setting, such that the participant and a researcher were

the only individuals in the room during the reading experience.

Analysis: Coding and Analyzing Think-Alouds

Once the think-aloud data are transcribed, researchers may begin coding,

meaning applying labels to the verbalizations that align with categories of interest.

This section describes how to determine the coding scheme in alignment with

research goals, considerations of parsing participant responses, training and inter-

rater reliability, and analyses.

Stage 1: Determining the Coding Scheme

Prior to engaging in coding, the researcher must determine whether they want to

use an inductive or deductive method for establishing coding categories. In an

inductive method, codes are decided based on a thorough reading of transcripts,

with no preconceived notions of what the categories might be. A deductive method

is when the researcher has a preestablished plan for coding based on the

research goal. Bohn-Gettler and Kendeou (2014) were interested in strategies

associated with successful comprehension and therefore opted to use a deductive

method based on prior research examining which reading processes result in

higher quality comprehension of text (Bohn-Gettler & Rapp, 2011; van den Broek,

Lorch, Linderholm, & Gustafson, 2001). (See Table 1 for a full description of the

coding scheme, along with aligning responses.) Note that coding schemes can

and should vary depending on the research goals.

Table 1: Think-Aloud Codes, Definitions, and Examples.

Code Definition Text excerpt Participant response

SAGE



2



Paraphrases Restating the sentence in different

words

Colleges and universities could develop

programs aimed at training female

politicians.

Colleges could create

training for female

politicians.

Connecting

inferences

Referring to textual information

presented in the immediately

preceding sentence

Sentence 2: Instead of fumbling for change,

they drive right through.

Sentence 3: The device sends a radio signal

that records their passage.

So that people don’t

just drive through

without paying…

Reinstatement

inferences

Referring to textual information

presented earlier than the

immediately preceding sentence

Sentence 12: Airplanes regularly carry

several hundred passengers.

Sentence 15: They usually carry only two to

five passengers.

It’s smaller than an

airplane.

Elaborations Utilizing prior knowledge to help

explain what was just read Virginia has reduced its minimum age to 14.

… and they wouldn’t

be able to punish

them, unless they were

tried as adults.

Predictions Anticipating what will occur next They voted to remove South Carolina from

the United States of America.

Uh sounds like the

confederacy will soon

be formed.

Opinions Providing an opinion about what

was read Virginia has reduced its minimum age to 14.

Uh, that seems really

low to be tried as an

adult.

Statements of

uncertainty

Statements reflecting not

understanding or not knowing

On December 23, 1947, three scientists at

Bell Telephone Laboratories demonstrated

their new invention of the point contact

transistor amplifier.

I don’t know what that

is.

Associations

Stating information and/or prior

knowledge and experiences that

do not serve to increase

understanding of the text

President Eisenhower sent federal troops

and the National Guard to intervene on

behalf of the students.

Um, this randomly

reminded me of

Remember the Titans.

Other Responses that do not fit any other

category, and non-responses

In September, formerly all-white Central

High School learned that integration is

easier said than done.

Integration … so yeah.

Invalid Statements reflecting a In 1950, President Truman ordered the January is 1850 …

SAGE



2



statement

misunderstanding of the text.

(Coded in addition to at least one

of the other codes above.)

Atomic Energy Commission to make the

hydrogen bomb.

Stage 2: Parsing

Participants may respond to a sentence with multiple thoughts and ideas, making

it important to determine how to parse participants’ statements. In this context,

parsing refers to when a researcher splits a participant’s response into shorter

statements to enable greater specificity when assigning codes. For example,

consider the following participant response:

Text Excerpt: On December 1, 1955, NAACP member Rosa Parks

refused to give up her seat at the front of the “colored section” of a bus

to a white passenger, defying a southern custom of the time.

Participant’s Response: Rosa Parks was …. Sick of the black treatment,

/ and she was going to do something about it. / She was tired from

cleaning all day, / and, um, this was her way of defying. /

A researcher could assign codes in multiple ways. (1) The researcher could

code the entire response with just one code that represents the majority of the

response, which is an attempt to get at the dominant process that a participant

used in each response. (2) The researcher could code only the first idea of the

response, which would address the first, initial thoughts of the participants. Or, (3)

the researcher could code each idea in the response separately (i.e., parse the

response into multiple shorter statements), to obtain a more detailed picture of

all the processes that occurred. Bohn-Gettler and Kendeou (2014) parsed each

response into ideas or events, which usually represented subject–verb phrases.

In the example above, a “/” indicates where parsing occurred. Please note that the

data exemplar contains a fully coded transcript.

SAGE



2



Stage 3: Training, Inter-Rater Reliability, and Independent Coding

Once the coding scheme has been established, the actual process of coding

can begin. (Note, however, that the coding scheme can be revised as needed

depending on the nature of the data.) It is important to engage in training (or

practice) with the coding scheme, as well as to establish inter-rater reliability. Inter-

rater reliability is a method for ensuring that all raters are coding in a consistent

manner.

During the training phase in Bohn-Gettler and Kendeou (2014), all raters coded

several transcripts together as a group to have a chance to practice, talk through

the codes, and ensure consistent application and understanding of each code.

Next, the raters individually and independently coded transcripts. The raters

coded the exact same transcripts, so that they could compare their independent

codes. When comparing the codes for the same transcript, they kept track of

the location and frequency of agreements versus disagreements. When

disagreements occurred, the raters would discuss and decide upon which code

was most appropriate. From that point, raters then calculated inter-rater

agreement based on the percentage of times agreements versus disagreements

occurred. This process continued until at least 20% of the transcripts were coded,

and agreement rates were consistently at or above 90%.

After the training phase, raters entered into a phase in which transcripts could be

divided among raters, but at least 20% of the remaining transcripts were still coded

by multiple raters to ensure inter-rater reliability. This was established by setting

up a weekly schedule in which each rater received four to five transcripts to code.

At least one or two of those transcripts would be assigned to multiple raters, while

the remaining transcripts were only assigned to one rater. To ensure the same

care and attention was given for every transcript, the raters were not aware of

which transcript(s) would be coded in common with another rater until their coding

SAGE



2



meeting. In the coding meetings, raters met once a week to compare their codes

on the in-common transcripts and discuss challenges or questions encountered.

As in the training phase, raters kept track of the number of agreements and

disagreements to calculate percentage agreement. In addition, the original codes

were always monitored to establish whether any rater displayed a consistent

coding bias or pattern.

Stage 4: Analyses

To analyze such think-aloud data, researchers are confronted with a few

decisions. First, each participant can provide a different number of sentences

and ideas. For example, one participant may have provided a one-word answer,

whereas another may have stated five to six sentences. In addition, participants

read multiple and different texts, which varied in length. To enable comparisons

between participants, Bohn-Gettler and Kendeou (2014) calculated the total

number of each code that the participants provided and divided that by the total

number of codes in the entire text. This resulted in percentages of how often each

participant engaged in each process. One challenge was that the distributions

were skewed (meaning that, when creating histograms of the variables, the

shapes did not reflect a normal bell curve). This is common with proportion data,

and hence the arcsine transformation was applied. (Mathematically, arcsine is

the inverse of the sine function.) In an arcsine transformation, each data point

is multiplied by the formula below. These transformations help to make the data

more normally distributed for statistical tests but do not change the validity of the

data (Howell, 2002). In the equation below, p represents the proportion with which

each participant engaged in each process.

arcsine(√p)

Statistical analyses can be performed on each category separately, or related

categories can be combined. To examine whether particular categories should be

SAGE



2



combined, Bohn-Gettler and Kendeou (2014) considered whether the codes were

conceptually similar and ran a correlation analysis. A correlation analysis indicates

whether variables are associated with one another. For example, connecting and

reinstatement inferences were related. This made the results easier to interpret

and reduced the number of analyses.

The larger categories Bohn-Gettler and Kendeou (2014) created included

paraphrases, which can help with memory for text. The next category was text-

based inferences, which included connecting and reinstatement inferences.

These inferences utilize prior text information to help understand what the

participant was reading. The third category was knowledge-based inferences,

which included elaborative and predictive inferences. These involved activating

prior knowledge to help understand the text. The fourth category was non-

coherence processes. These were processes that were less likely to help a

participant understand the text, such as stating uncertainty, providing

unelaborated opinions, and making loose associations. Finally, the “other”

category was dropped because it contained too little data.

These categories then allowed Bohn-Gettler and Kendeou (2014) to engage in

meaningful statistical analyses to determine how reading goals, text format, and

working memory individually and interactively influenced the quality of reading

processes and strategies.

Reflective Questions

1. What factors should researchers be cautious of when using the think-

aloud methodology? Why? How could you design your study to

minimize some of the challenges associated with think-alouds?

2. Why might it be difficult to distinguish between elaborations versus

associations, or elaborations versus paraphrases? What other coding

SAGE



2



categories present difficulties, and how can you clearly distinguish

between them?

3. This guide described one deductive method of coding think-alouds for

the cognitive processes in which readers engage. Does this coding

method align with your research interests and goals? Why or why not?

If not, what kind of coding scheme would better fit your research goals?

4. Why is training and inter-rater reliability important? Based on your

study, design a plan for training and establishing inter-rater reliability.

Further Readings

Afflerbach, P. (2000). Verbal reports and protocol analysis. In M. L. Kamil, P. B.

Mosenthal, P. D. Pearson, & R. Barr (Eds.), Handbook of literacy research (pp.

163–179). Mahwah, NJ: Lawrence Erlbaum.

Bohn-Gettler, C. M. (2018). The complexities of comprehension: Conducting

think-aloud studies with multiple interacting variables. SAGE Research Methods

Cases. doi:http://dx.doi.org/10.4135/9781526439550

Bohn-Gettler, C. M., & Kendeou, P. (2014). The interplay of reader goals,

working memory, and text structure during reading. Contemporary Educational

Psychology, 39(3), 206–219. doi:http://dx.doi.org/10.1016/

j.cedpsych.2014.05.003

Crutcher, R. J. (1994). Telling what we know: The use of verbal report

methodologies in psychological research. Psychological Science, 5, 241–244.

Leighton, J. P. (2017). Using think-aloud interviews and cognitive labs in

educational research. New York, NY: Oxford University Press.

Magliano, J. P., Trabasso, T., & Graesser, A. C. (1999). Strategic processing

during comprehension. Journal of Educational Psychology, 91, 615–629.

SAGE



2



http://dx.doi.org/10.4135/9781526439550

http://dx.doi.org/10.1016/j.cedpsych.2014.05.003


Pressley, M., & Afflerbach, P. (1995). Verbal protocols of reading: The nature of

constructively responsive reading. Hillsdale, NJ: Erlbaum.

References

Bohn-Gettler, C. M. (2018). The complexities of comprehension: Conducting

think-aloud studies with multiple interacting variables. SAGE Research Methods

Cases. doi:http://dx.doi.org/10.4135/9781526439550

Bohn-Gettler, C. M., & Kendeou, P. (2014). The interplay of reader goals,

working memory, and text structure during reading. Contemporary Educational

Psychology, 39(3), 206–219. doi:http://dx.doi.org/10.1016/

j.cedpsych.2014.05.003

Bohn-Gettler, C. M., & McCrudden, M. T. (2018). The effects of task instructions

and topic beliefs on reading processes and memory. Discourse Processes, 55(4),

410–431. doi:http://dx.doi.org/10.1080/0163853X.2017.1292824

Bohn-Gettler, C. M., & Rapp, D. N. (2011). Depending on my mood: Mood-driven

influences on text comprehension. Journal of Educational Psychology, 103(3),

562–577. doi:http://dx.doi.org/10.1037/a0023458

Education Database Online. (2006). Nuclear history timeline. Retrieved from

http://www.onlineeducation.net/resources/nuclear-history-timeline

Ericsson, K. A., & Simon, H. A. (1993). Protocol analysis: Verbal reports as data.

Cambridge, MA: MIT Press.

Howell, D. C. (2002). Statistical methods for psychology. Pacific Grove, CA:

Wadsworth Cengage Learning.

Kelley, W. J. (Ed.). (2002). Strategy and structure: Short readings for composition

(3rd ed.). New York, NY: Longman.

SAGE



2



http://dx.doi.org/10.4135/9781526439550



http://dx.doi.org/10.1080/0163853X.2017.1292824

http://dx.doi.org/10.1037/a0023458

http://www.onlineeducation.net/resources/nuclear-history-timeline

Scribd. (2009, September). Online social publishing community. Retrieved from

http://www.scribd.com

Turner, R. (1998). Women in parliament. Retrieved from

http://www.bilkent.edu.tr/’fast/women.html

van den Broek, P., Lorch, R. F., Linderholm, T., & Gustafson, M. (2001). The

effects of readers’ goals on inference generation and memory for texts. Memory &

Cognition, 29, 1081–1087. doi:http://dx.doi.org/10.3758/BF03206376

SAGE



2



http://www.scribd.com/

http://www.bilkent.edu.tr/%E2%80%99fast/women.html

http://dx.doi.org/10.3758/BF03206376

analyzing verbal protocols: thinking aloud during reading

Documents