Think Aloud Protocol (TAP)

-A Reflection

ByDebaleena Chattopadhyay

Research Paper: Janni Nielsen, Torkil Clemmensen, and Carsten Yssing. 2002. Getting access to what goes on in people's heads?: Reflections on the think-aloud technique. In Proceedings of the second Nordic conference on Human-computer interaction (NordiCHI '02). ACM, New York, NY.

2

History• What is TAP?

– TAP is a method to gather data in usability testing in product design and development.*

– A usability testing protocol popularly used for its simplicity and effectiveness.

• Who introduced TAP in HCI?– Clayton Lewis while at IBM in 1982. **

• What do TAPs involve?– TAPs involve participants thinking aloud as they perform a set of specified tasks.

Users are asked to say whatever they are looking at, thinking, doing, and feeling, as they go about their task.*

• What are the different types of TAPs?– Concurrent TAP (collected during the task)*– Retrospective Thinking (collected after the task)*

* From Wikipedia** Presented in the technical report: Task-Centered User Interface Design: A Practical Introduction by C. Lewis and J. Rieman.

I563, Fall 2012 3

Getting access to what goes on in people’s heads?

• Thinking aloud technique dates back to the works of experimental psychology and was first ever described by Karl Duncker (1945) while he studied productive thinking.

• TAP is popularly used by usability researchers today. But what do researchers think they get from TAP? Is it right to assume that there is a one-to-one mapping between verbal protocols and ‘pure data’?

• TAP adds cognitive load on users and can hinder primary tasks (Preece, 1994). So how do the users experience it? What do users think of it?

I563, Fall 2012 4

Some applications of TAP• Study search strategies and navigation behavior (Van Waes,

1998)

• Understand mental processes in connection with writing programs (Bringham, John & Lewis, 1991)

• Understand user’s reasoning while learning Smalltalk (Koenemann-Belliveau et al. 1994)

• Study students’ writing and reading processes

• Investigate the cognitive processes involved in problem solving.

Researchers argue that TAP allows them access to the cognitive processes and mental behavior (Karsenty, 2001).

I563, Fall 2012 5

What do we get access to when asking users to think aloud?

• Does TAP really give us access to what goes on in people’s heads?Boren and Ramey(2000) has questioned the technique and its theoretical underpinnings. They studied how practitioners actually carried out TAP sessions and discussed the practice in relation to the classical work by Ericsson and Simon (1984) on vocalization and verbalization of thoughts. They argue that it is necessary to have a firm theoretical grounding and a unified practice before the technique can be called a method.

• The classical model of Ericsson and Simon (1984)on verbal report as data*

This work wanted to reinstate verbal data as a valid resource for understanding human cognitive processes to make it (a) possible to use verbal data to verify, not only discover, phenomena of interest, provided (b) that verbal data was interpreted within a theoretical framework.Such frameworks are necessary to investigate the construct validity, if there exists a one-to-one mapping between useful information and introspective data.

* Protocol Analysis

I563, Fall 2012 6

TAP in the light of Information Processing Theory

• Everything we know has, at some point, gone through our short-term memory (STM) and we have been conscious of it.

• We can verbalize what we are perceiving while in the process of perceiving, and we can verbalize what we were conscious of if questioned shortly after the process has taken place. This is because it is still retained in our short-term memory.

• However, if there is a time span between perceiving and the request to recall, we will produce descriptions and explanations - not a report of our immediate thoughts, because the information from STM is lost.

• Using this assumption, we can distinguish between (classical) introspection, retrospective reports and communication to the experimenter on the one hand, and on the other verbalization of currently “heeded” thoughts (thoughts reflecting current attention)

I563, Fall 2012 7

Ericsson and Simon (1984) Model

• To study task directed cognitive processes – only concurrent and certain kinds of retrospective verbalizing will address the information employed while performing a given task.

• This model identifies and analyses these verbalizations and distinguishes them into three kinds:

- Vocalizations of thoughts that are already encoded in the verbal form (talk aloud).- Verbalization of a sequence of thought that are held in memory in some other form, e.g. visually (think aloud).- Other verbalizations (retrospective reports on thoughts not held in memory).

I563, Fall 2012 8

Reflection• The authors in the paper report expert users’ experience with TAP–

“Teaching graduate students in Informatics to think aloud and asking them to reflect on their experience with using the technique have raised a number of issues. Students complain that they think faster than they can speak, that their thought processes are much more complex than they can verbalize, and that thinking aloud interferes with their interaction with the interfaces and the task.”

• It is interesting to note that one particular consequence of Ericsson and Simon’s use of their model of verbalization is their advice to instruct the subject to “keep talking”. Ericsson and Simon argue that– “In their model, verbalization will always lack behind thinking in time, except perhaps in the execution of very new tasks. This is because thinking in already encoded verbal forms is fast, the activation of “old thoughts” somewhat slower and only the generation of “new thoughts” is really slow.”

• As two cognitive processes are competing, the process of thinking and the process of verbalizing, hence the delay.

• The model finally makes the assumption that only introspections, which are verbalizations of currently heeded thoughts, will enhance investigation into task directed cognitive processes lead to an information processing paradigm.

I563, Fall 2012 9

Conclusion• Thinking is much more that what can be explicitly expressed in words.

• To get access to human cognitive processes, a way forward may be to develop a practice of introspection; to expand our knowledge about the reflective activity of the user in the expert-guided think aloud session.

• The authors argue that access to subjective experience is possible in terms of introspection where user has to become a participant in the analysis of his or her own cognitive processes.

• The paper suggests that use of think aloud should have, as a prerequisite, explicit descriptions of design, test procedure and framework for analysis.