Cued Retrospective Think-aloud Interviewing to Capture Thinking

During Cognitively Demanding Tasks

Ginger Shultz gshultz@umich.eduTwitter: @GingCER and @ShultzGroupCER

Megan Connor mcarole@umich.eduTwitter: @megan_connor10

What is your Interviewing Experience?

Select the item that best describes your research experience:

A. I have conducted a think-aloud interview

B. I have never conducted a think-aloud but have used other interview methods

C. I have never conducted any type of interview

Overview of Interview Methods: How deep are you diving?

Structured Narrow focusRigid set of questionsNo latitude

Overview of Interview Methods: How deep are you diving?

Structured

Semi-structured

NarrowRigid set of questionsNo latitude

Broader focusWell formed questionsLatitude to pursue lines of inquiry

Overview of Interview Methods: How deep are you diving?

Structured

Semi-structured

Unstructured

NarrowRigid set of questionsNo latitude

Broader focusWell formed questionsLatitude to pursue lines of inquiry

Broad focus A few questions to get you startedYou aren’t sure from the outset what will be important

Think-aloud: Combines interviewing with observation

Participant • Articulates their thinking out loud while completing a task

• Vocalizes what they are doing

Interviewer• Observes and notes interesting comments and behavior

• May prompt if participant is not vocal

What is a “Cognitively Demanding Task”?

When the complexity of a task exceeds one’s level of cognitive development, resulting in a cognitive overload that leaves ”little to no capacity left for verbal thought processes”When the complexity of a task exceeds one’s level of cognitive development, resulting in a cognitive overload that leaves ”little to no capacity left for verbal thought processes”

Branch, Jennifer L. "Junior high students and Think Alouds: Generating information-seeking process data using concurrent verbal protocols." Library & Information Science Research23.2 (2001): 107-122.Hyrskykari, Aulikki, et al. "Gaze path stimulation in retrospective think-aloud." (2008).

Participant is unable to complete the task and attend to other things at the same time

Participant is able to complete the task almost automatically while attending to other things

Participant DependentNovice Expert

NMR Spectral Interpretation: A Cognitively Demanding Task

Decode chemical shift, splitting, integration, number of peaks

Connect spectral information to structural features and implicit properties

Novice may find each individual item challenging Expert does many components automaticallyCan also easily select which info to pay attention to

Think-aloud interviewing – temporal considerationsTechnique Pros Cons

Concurrent Think-Aloud: Participant verbalizes all thoughts while completing a task

> Widely-employed > Minimal fabrication> Rich data > Provides action info

>Possible cognitive overload> Potential to interfere

Retrospective Think-Aloud: Participant articulates their thinking immediately after engaging in a task

> Minimal cognitive overload

> Less rich data > Possible fabrication> No action info

Cued RTA: Participant articulates their thinking after engaging in a task and is cued by a record of observations

> Minimal cognitive overload > Rich data

> Possible fabrication

Van Gog, Tamara, et al. Journal of Experimental Psychology: Applied 11.4 (2005): 237.

Think-aloud interviewing – temporal considerationsTechnique Pros Cons

Concurrent Think-Aloud: Participant verbalizes all thoughts while completing a task

> Widely-employed > Minimal fabrication> Rich data > Provides action info

>Possible cognitive overload> Potential to interfere

Retrospective Think-Aloud: Participant articulates their thinking immediately after engaging in a task

> Minimal cognitive overload

> Less rich data > Possible fabrication> No action info

Cued RTA: Participant articulates their thinking after engaging in a task and is cued by a record of observations

> Minimal cognitive overload > Rich data

> Possible fabrication

Van Gog, Tamara, et al. Journal of Experimental Psychology: Applied 11.4 (2005): 237.

Moderate task: Concurrent TA ≈ Cued RTA Cognitively demanding task: Cued RTA > Concurrent RTAModerate task: Concurrent TA ≈ Cued RTA Cognitively demanding task: Cued RTA > Concurrent RTA

Break out session

• Consider the following three scenarios. Discuss whether concurrentor retrospective think-aloud interview would be best. Identify concrete reasons to support your decision. Select one person to report out for your group. You have 5 minutes.

1) Investigating how students connect ideas about reactivity while writing an essay about the Aldol Condensation reaction mechanism

2) Investigating how students translate between a tactile molecular model and a drawn Lewis structure

3) Investigating how students find the endpoint during an acid-base titration.

Report out in the chat

What factors did your group consider when deciding whether a concurrent or retrospective think-aloud interview would be best?

Tobii Pro X3-120 screen-based eye tracker

1H NMR spectrumInitial research question:How do undergraduates solve 1H NMR problems, and how

do these approaches differ between successful and unsuccessful problem solvers?

o NMR spectra are among most common visualizations used by practicing chemists

o We had an eye tracker, a leading tool for investigating processing of visual information

o Initial plan was to conduct a predominantly quantitative eye-tracking study

Connor et al. (2019), Chem. Educ. Res. Pract., 20, 522-541.Kozma and Russell (2006), in Visualizations in Science Education, 121-146.Havanki and VandenPlas (2014), in Tools of Chemistry Education Research, ACS Symposium Series.

Using cued RTA interviewing to capture students’ thinking during NMR spectral interpretation

Interviewing Techniques Pros Cons

Concurrent Think-AloudWidely-employed Minimal fabrication

Possible cognitive overloadPossible distortion of eye-tracking data

Retrospective Think-AloudMinimal cognitive overload Minimal distortion of eye-tracking data

Less rich data Possible fabrication

Cued RTAMinimal cognitive overload Minimal distortion of eye-tracking dataRich data

Possible fabrication

Choosing cued RTA interviewing as a data collection method

o Eye-tracking data is most informative when triangulated with additional qualitative data

o Needed to combine eye tracking with an interviewing technique

o Eye tracking allows for cued RTA interviewing

23

1

Triangulation of eye-tracking and interview data

Hyrskykari (2008), J. Eye Mov. Res., 2, 1–18.

Developing interpretation tasks

o We would display tasks on 24-inch monitor using Tobii Pro Studio

o Second-semester organic chemistry students had to complete tasks without the ability to make annotations or notes

o Participants could use keyboard to move between multiple pages per task

Tobii Technology (2020), Eye Tracking for Research, https://www.tobiipro.com/.

Developing interpretation tasks

Developed three tasks incorporating:

o Literature on day-to-day problems of practicing chemists

o Difficult features identified through faculty interviews

o Cognitive load considerations

o Content knowledge considerations

Chemists conducted a series of reactions to synthesize N-(2-hydroxyethyl)-propanamide. The chemists then analyzed the final product spectroscopically to determine if the synthesis was successful. Based on the spectroscopic data of the final

product below (IR and 1H NMR spectra), did the chemists successfully synthesize N-(2-hydroxyethyl)-propanamide?

0

5001000

100

150030004000

-1

50

124567 038

2000

O

NH

OH

N-(2-hydroxyethyl)-Propanamide

1, singlet

2, triplet1, singlet

2, quartet

2, quartet

3, triplet

Raker and Towns (2012), Chem. Educ. Res. Pract., 13, 179-185.

*Font size enlarged here

Developing interpretation tasks

Developed three tasks incorporating:

o Literature on day-to-day problems of practicing chemists

o Difficult features identified through faculty interviews

o Cognitive load considerations

o Content knowledge considerations

Chemists conducted a series of reactions to synthesize N-(2-hydroxyethyl)-propanamide. The chemists then analyzed the final product spectroscopically to determine if the synthesis was successful. Based on the spectroscopic data of the final

product below (IR and 1H NMR spectra), did the chemists successfully synthesize N-(2-hydroxyethyl)-propanamide?

0

5001000

100

150030004000

-1

50

124567 038

2000

O

NH

OH

N-(2-hydroxyethyl)-Propanamide

1, singlet

2, triplet1, singlet

2, quartet

2, quartet

3, triplet

Raker and Towns (2012), Chem. Educ. Res. Pract., 13, 179-185.

Developing the eye tracking and cued RTA interview protocol

Introduction to study• Consent and experience survey• Three IR and 1H NMR problems• Sample problem walkthrough• Brief response process validity

Preparation for data collection• Explanation of eye tracking• Eye calibration • Explanation of cued RTA interview

Sample problem walkthrough

Page 1 Page 2 Response selection page

Developing the eye tracking and cued RTA interview protocol

Chemists conducted a series of reactions to synthesize N-(2-hydroxyethyl)-propanamide. The chemists then analyzed the final product spectroscopically to determine if the synthesis was successful. Based on the spectroscopic data of the final product below (IR and 1H NMR spectra), did the chemists successfully synthesize N-(2-hydroxyethyl)-propanamide?

0

5001000

100

150030004000

Wavenumber (cm -1)

50

124567

ppm (δ)

038

1H NMR spectrum

2000

% t

ran

smit

tan

ce

O

NH

OH

N-(2-hydroxyethyl)-Propanamide

IR spectrum

1, singlet

2, triplet1, singlet

2, quartet

2, quartet

3, triplet

Preparation fordata collection

Eye movement data collection Cued RTA data collection

Task 1

Task 2

Task 3

Data Collection

• Undergraduates recruited from second-semester organic chemistry laboratory

• Each participant completed one 30 to 60-minute session in accordance with protocol

• Recruited and collected data until saturation achieved, resulting in 18 total participants

Data sources

• Survey data on experience with IR and NMR • Eye tracking data (approx. 250,000 datapoints total)• Audio-video recordings of RTA interviews (approx. 400 minutes total)

Audio-video recording of RTA interviews

Data Collection

• Undergraduates recruited from second-semester organic chemistry laboratory

• Each participant completed one 30 to 60-minute session in accordance with protocol

• Recruited and collected data until saturation achieved, resulting in 18 total participants

Data sources

• Survey data on experience with IR and NMR • Eye tracking data (approx. 250,000 datapoints total)• Audio-video recordings of RTA interviews (approx. 400 minutes total)

Qualitative analysis of cued RTA interviews

o RTA data appeared to provide rich insight into reasoning, so shifted from eye tracking study

o Transcribed RTA audio verbatim

o Used RTA audio-video data to clarify ambiguous references to spectral data in NVivo

o Inductively coded for chemical assumptions to investigate reasoning

RTA audio-video recording Interview transcription in NVivo, with clarifications in boxes

Qualitative analysis of cued RTA interviews o Expanded data collection and analysis to include seven PhD student participantso 83 total chemical assumptions, five themes, and four levels of sophistication

Foundational Assumptions

Assumptions about deviations from the N+1 rule

Lo

wer

Up

per

Inte

rmed

iate

IIn

term

edia

te I

I

Invalid assumptions that the N+1 rule should apply in contexts where it fails.

Valid assumptions reflecting doubt about whether the N+1 rule applies in

contexts where it fails.

Valid assumptions reflecting knowledge of specific contexts in

which the N+1 rule does not apply. Knowledge of implicit chemical

features is not expressed.

Valid assumptions reflecting knowledge of implicit chemical

features resulting in deviations from the N+1 rule, including some invalid

ideas about complex implicit chemical features affecting splitting.

Assumptions about variability of spectral data Visuospatial Assumptions Practical Assumptions

Valid assumptions reflecting an understanding of basic fundamental principles that underly spectroscopic data

Invalid assumptions reflecting a misunderstanding of basic fundamental principles that underly spectroscopic data

Invalid assumptions that spectral data should not vary in contexts

where such variability is acceptable.

Invalid assumptions that spectral data can vary in contexts where such

variability is unacceptable, including invalid ideas about implicit chemical

features affecting variability.

Valid assumptions reflecting knowledge of specific contexts in

which spectral data can vary. Knowledge of implicit chemical

features is not expressed.

Valid assumptions reflecting knowledge of implicit chemical features and their effect on the

variability of spectral data.

Invalid assumptions reflecting difficulty with identifying planes of symmetry and implicit hydrogens

within structural formulae.

Valid assumptions reflecting doubt or uncertainty regarding visuospatial

features, in particular when identifying planes of symmetry.

Valid assumptions reflecting an ability to identify planes of symmetry,

chemically equivalent protons, and implicit hydrogens within structural formulae, along with relevant but inconspicuous spectral features.

Valid assumptions reflecting recognition of characteristic NMR

splitting patterns involving more than one peak, as well as a developing

ability to evaluate the stereotopicity of implicit hydrogens.

Invalid assumptions reflecting limited familiarity with additional peak

characteristics, including the misidentification of salient IR peaks.

No assumptions relating to experimental variables.

Valid assumptions reflecting doubt or uncertainty about additional peak

characteristics, specifically when identifying salient IR peaks. No

assumptions relating to experimental variables.

Valid assumptions reflecting general familiarity with additional peak

characteristics, in particular when identifying salient IR peaks. Some

assumptions relating to basic experimental variables.

Valid assumptions reflecting familiarity with specific aspects of additional peak characteristics and

developed knowledge of experimental variables influencing the appearance

of spectral data.

Connor et al. (2020), Mapping the development of expertise in spectral interpretation, under review.

Resources• Interviewing

Herrington, Deborah G., and Patrick L. Daubenmire. "Using interviews in CER projects: Options, considerations, and limitations." Tools of chemistry education research. American Chemical Society, 2014. 31-59.

Roulston, Kathryn, Kathleen DeMarrais, and Jamie B. Lewis. "Learning to interview in the social sciences." Qualitative inquiry 9.4 (2003): 643-668.

• Technologies for think-aloud

• Eye-tracking (consider a collaborator)Havanki, Katherine L., and Jessica R. VandenPlas. "Eye tracking methodology for chemistry education research." Tools of chemistry education research. American Chemical Society, 2014. 191-218.

• Video, audio, or keystroke playback

• Livescribe pen

• Pen and paper

Shultz Group

Dr. Solaire Finkenstaedt-Quinn

Benjamin Glass

Q & A: please type your questions in the Chat